Using Cleanlab to check data extraction fidelity.

cleanlab-legal.py

import warnings
warnings.filterwarnings('ignore')

from rich import print

import outlines
from cleanlab_tlm import TLM
import os
from pydantic import BaseModel
from transformers import AutoTokenizer

os.environ["CLEANLAB_TLM_API_KEY"] = "<CLEANLAB_TLM_API_KEY>"

model_id = "microsoft/Phi-3.5-mini-Instruct"
# model_id = "HuggingFaceTB/SmolLM2-1.7B-Instruct"
model = outlines.models.transformers(model_id, device="cuda")
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Some data to extract information from
prompt = """
SETTLEMENT AGREEMENT AND RELEASE

CASE NO.: CV-2025-04789-JRT
UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF CALIFORNIA

PLAINTIFF: GREENTECH SUSTAINABLE SOLUTIONS, LLC
v.
DEFENDANT: NEXUS INDUSTRIAL SYSTEMS, INC.

THIS SETTLEMENT AGREEMENT AND RELEASE (hereinafter referred to as the "Agreement") is made and entered into on this 17th day of April, 2025, by and between GREENTECH SUSTAINABLE SOLUTIONS, LLC, a Delaware limited liability company with its principal place of business at 4578 Innovation Way, Palo Alto, California 94304 (hereinafter referred to as "PLAINTIFF" or "GREENTECH"), and NEXUS INDUSTRIAL SYSTEMS, INC., a corporation organized under the laws of the State of Nevada with its principal place of business at 8901 Corporate Boulevard, Suite 300, Henderson, Nevada 89074 (hereinafter referred to as "DEFENDANT" or "NEXUS").

WHEREAS, PLAINTIFF initiated legal proceedings against DEFENDANT in the United States District Court for the Northern District of California (Case No. CV-2025-04789-JRT) on January 12, 2025, alleging patent infringement of U.S. Patent Nos. 9,845,721, 10,124,553, and 11,278,966 relating to advanced sustainable manufacturing processes (hereinafter referred to as the "PATENTS-IN-SUIT"); and

WHEREAS, DEFENDANT has denied and continues to deny any wrongdoing or liability with respect to any of the claims asserted in the aforementioned lawsuit; and

WHEREAS, to avoid the uncertainty, burden, and expense of protracted litigation, the PARTIES desire to settle and resolve their disputes on the terms set forth herein;

NOW, THEREFORE, in consideration of the mutual promises, covenants, and conditions contained herein, and for other good and valuable consideration, the receipt and sufficiency of which are hereby acknowledged, the PARTIES hereby agree as follows:

ARTICLE I: DEFINITIONS
1.1 "EFFECTIVE DATE" shall mean the date upon which this Agreement is executed by all PARTIES hereto.
1.2 "PARTIES" shall mean GREENTECH and NEXUS, collectively.
1.3 "LICENSED PRODUCTS" shall mean any product, process, or service that, but for the license granted herein, would infringe one or more valid claims of the PATENTS-IN-SUIT.
1.4 "AFFILIATE" shall mean any entity that, directly or indirectly, controls, is controlled by, or is under common control with a PARTY.

ARTICLE II: CONSIDERATION AND PAYMENT TERMS
2.1 SETTLEMENT PAYMENT. DEFENDANT shall pay to PLAINTIFF the total sum of Seventeen Million Eight Hundred Fifty-Three Thousand Four Hundred Twenty-Five Dollars and Sixty-Seven Cents ($17,853,425.67) (the "SETTLEMENT PAYMENT") within thirty (30) calendar days of the EFFECTIVE DATE.
2.2 PAYMENT METHOD. The SETTLEMENT PAYMENT shall be made by wire transfer to the following account:
   Financial Institution: Pacific Western Bank
   ABA Routing Number: 121042882
   Account Number: 87420195366
   Account Name: Greentech Sustainable Solutions, LLC
   Reference: Settlement CV-2025-04789-JRT
2.3 ROYALTY PAYMENTS. In addition to the SETTLEMENT PAYMENT, DEFENDANT shall pay to PLAINTIFF a running royalty of three and one-quarter percent (3.25%) of Net Sales of LICENSED PRODUCTS sold by DEFENDANT or its AFFILIATES during the ROYALTY TERM. "Net Sales" shall mean the gross amount invoiced by DEFENDANT or its AFFILIATES for sales of LICENSED PRODUCTS, less the following deductions to the extent actually allowed and taken:
   (a) trade, quantity, and cash discounts;
   (b) credits or allowances given for rejected or returned LICENSED PRODUCTS;
   (c) any tax or government charge levied on the sale of LICENSED PRODUCTS;
   (d) any charges for freight, insurance, or other transportation costs.
2.4 ROYALTY TERM. The obligation to pay royalties shall commence on the EFFECTIVE DATE and shall continue until the expiration of the last-to-expire of the PATENTS-IN-SUIT.
2.5 ROYALTY REPORTS AND PAYMENTS. DEFENDANT shall provide to PLAINTIFF, within forty-five (45) days after the end of each calendar quarter, a written report setting forth:
   (a) the quantity of LICENSED PRODUCTS sold during the applicable quarter;
   (b) the gross sales of such LICENSED PRODUCTS;
   (c) applicable deductions from gross sales;
   (d) Net Sales of LICENSED PRODUCTS; and
   (e) royalties due.
   Each such report shall be accompanied by payment of the royalties due for the reported quarter.
2.6 MINIMUM ANNUAL ROYALTY. Notwithstanding Section 2.3, DEFENDANT shall pay to PLAINTIFF a minimum annual royalty of Five Hundred Thousand Dollars ($500,000.00) for each calendar year during the ROYALTY TERM, prorated for any partial year.
2.7 AUDIT RIGHTS. PLAINTIFF shall have the right, at its own expense and no more than once per calendar year, to audit DEFENDANT's books and records relating to the calculation of royalties due under this Agreement.

ARTICLE III: LICENSE GRANT
3.1 PLAINTIFF hereby grants to DEFENDANT and its AFFILIATES a non-exclusive, non-transferable (except as provided in Section 7.3), worldwide license under the PATENTS-IN-SUIT to make, have made, use, offer to sell, sell, and import LICENSED PRODUCTS during the term of this Agreement.
3.2 The license granted herein may not be sublicensed by DEFENDANT except to its AFFILIATES.

ARTICLE IV: MUTUAL RELEASES
4.1 PLAINTIFF RELEASE. PLAINTIFF, on behalf of itself and its AFFILIATES, hereby releases, acquits, and forever discharges DEFENDANT and its AFFILIATES from any and all claims, demands, actions, causes of action, suits, damages, costs, and expenses of every kind and description, whether known or unknown, that PLAINTIFF or its AFFILIATES had, now have, or may hereafter have, arising out of or relating to the PATENTS-IN-SUIT or the allegations in the lawsuit through the EFFECTIVE DATE.
4.2 DEFENDANT RELEASE. DEFENDANT, on behalf of itself and its AFFILIATES, hereby releases, acquits, and forever discharges PLAINTIFF and its AFFILIATES from any and all claims, demands, actions, causes of action, suits, damages, costs, and expenses of every kind and description, whether known or unknown, that DEFENDANT or its AFFILIATES had, now have, or may hereafter have, arising out of or relating to the PATENTS-IN-SUIT or the allegations in the lawsuit through the EFFECTIVE DATE.
4.3 WAIVER OF UNKNOWN CLAIMS. The PARTIES understand and agree that the releases set forth above extend to all claims of every nature and kind, known or unknown, suspected or unsuspected, past or present, and all rights under Section 1542 of the California Civil Code are hereby expressly waived. Section 1542 provides as follows:
"A GENERAL RELEASE DOES NOT EXTEND TO CLAIMS THAT THE CREDITOR OR RELEASING PARTY DOES NOT KNOW OR SUSPECT TO EXIST IN HIS OR HER FAVOR AT THE TIME OF EXECUTING THE RELEASE AND THAT, IF KNOWN BY HIM OR HER, WOULD HAVE MATERIALLY AFFECTED HIS OR HER SETTLEMENT WITH THE DEBTOR OR RELEASED PARTY."

ARTICLE V: DISMISSAL OF LITIGATION
5.1 Within five (5) business days of receipt of the SETTLEMENT PAYMENT, the PARTIES shall file a stipulation of dismissal with prejudice pursuant to Fed. R. Civ. P. 41(a)(1)(A)(ii) of all claims and counterclaims in the lawsuit, with each PARTY to bear its own costs, expenses, and attorneys' fees.

ARTICLE VI: CONFIDENTIALITY
6.1 CONFIDENTIALITY OF AGREEMENT. The PARTIES agree that the terms and conditions of this Agreement, but not its existence, shall be maintained in confidence and shall not be disclosed to any third party except:
   (a) with the prior written consent of the other PARTY;
   (b) to the extent necessary to comply with applicable laws, regulations, or valid court orders;
   (c) to a PARTY's legal counsel, accountants, tax advisors, and other professional advisors under an obligation of confidentiality;
   (d) to a PARTY's AFFILIATES under an obligation of confidentiality;
   (e) in connection with a merger, acquisition, or financing transaction, provided that any such disclosure is made under an obligation of confidentiality; or
   (f) to enforce rights under this Agreement.
6.2 PUBLIC STATEMENTS. Neither PARTY shall issue any press release or make any public statement regarding this Agreement without the prior written consent of the other PARTY, except as required by law or regulations. In the event of any required disclosure, the disclosing PARTY shall provide the other PARTY with reasonable advance notice and an opportunity to review and comment on any such disclosure.

ARTICLE VII: MISCELLANEOUS PROVISIONS
7.1 ENTIRE AGREEMENT. This Agreement constitutes the entire agreement between the PARTIES with respect to the subject matter hereof and supersedes all prior agreements, understandings, negotiations, and discussions, whether oral or written.
7.2 GOVERNING LAW. This Agreement shall be governed by and construed in accordance with the laws of the State of California, without regard to its conflict of laws principles.
7.3 ASSIGNMENT. Neither PARTY may assign this Agreement without the prior written consent of the other PARTY, except that either PARTY may assign this Agreement without consent to a successor in interest in connection with a merger, acquisition, or sale of all or substantially all of its assets.
7.4 SEVERABILITY. If any provision of this Agreement is held to be invalid or unenforceable, the remaining provisions shall continue in full force and effect, and the PARTIES shall substitute for the invalid or unenforceable provision a valid and enforceable provision that most closely approximates the economic effect and intent of the invalid or unenforceable provision.
7.5 COUNTERPARTS. This Agreement may be executed in counterparts, each of which shall be deemed an original, but all of which together shall constitute one and the same instrument. Facsimile or electronic signatures shall be deemed original signatures.
7.6 NOTICES. All notices required or permitted under this Agreement shall be in writing and shall be delivered personally, by certified mail (return receipt requested), or by overnight courier to the addresses set forth below:

If to PLAINTIFF:
Greentech Sustainable Solutions, LLC
Attn: Legal Department
4578 Innovation Way
Palo Alto, CA 94304
With a copy to: [email protected]

If to DEFENDANT:
Nexus Industrial Systems, Inc.
Attn: General Counsel
8901 Corporate Boulevard, Suite 300
Henderson, NV 89074
With a copy to: [email protected]

7.7 MODIFICATION. This Agreement may not be modified or amended except by a written instrument executed by both PARTIES.
7.8 NO WAIVER. The failure of either PARTY to enforce any provision of this Agreement shall not be construed as a waiver of such provision or the right of such PARTY thereafter to enforce such provision or any other provision of this Agreement.
7.9 INTERPRETATION. This Agreement has been negotiated by the PARTIES and shall be interpreted fairly and not more strictly against one PARTY than the other.
7.10 FORCE MAJEURE. Neither PARTY shall be liable for any failure or delay in performance under this Agreement due to causes beyond its reasonable control.

IN WITNESS WHEREOF, the PARTIES have executed this Agreement as of the date first written above.

GREENTECH SUSTAINABLE SOLUTIONS, LLC

By: /s/ Eleanor J. Ramirez
Name: Eleanor J. Ramirez
Title: Chief Executive Officer
Date: April 17, 2025

NEXUS INDUSTRIAL SYSTEMS, INC.

By: /s/ Marcus T. Williams
Name: Marcus T. Williams
Title: President and Chief Operating Officer
Date: April 17, 2025

APPROVED AS TO FORM AND CONTENT:

MORGAN, LEWIS & BOCKIUS LLP

By: /s/ David Chen
Name: David Chen
Title: Partner
Attorneys for PLAINTIFF

WILSON SONSINI GOODRICH & ROSATI, P.C.

By: /s/ Sarah J. Kennedy
Name: Sarah J. Kennedy
Title: Partner
Attorneys for DEFENDANT

The task is to extract the full settlement payment information between Greentech Sustainable Solutions, LLC and Nexus Industrial Systems, Inc. This should include the initial settlement payment amount, the royalty percentage, and the minimum annual royalty amount, formatted as a JSON object with the following fields:

- initial_settlement_amount (exact dollar amount including cents)
- royalty_percentage (as a decimal)
- minimum_annual_royalty (dollar amount)
- payment_deadline (number of days after effective date)
- royalty_term_end (description of when royalties end)
"""

# Apply chat templating to the prompt
prompt = tokenizer.apply_chat_template(
    [
        {"role": "user", "content": prompt},
    ],
    tokenize=False,
    add_generation_prompt=True,
)

# Define the shape of the data to extract
class SettlementData(BaseModel):
    initial_settlement_amount: float
    royalty_percentage: float
    minimum_annual_royalty: float
    payment_deadline: int
    royalty_term_end: str

# Construct a data extractor function
generator = outlines.generate.json(model, SettlementData)

# Call the LLM and extract the data
output = generator(prompt)

print("Output:")
print(output)

# Initialize the TLM client
cleanlab_tlm = TLM()
# cleanlab_tlm = TLM(options={"log": ["explanation"]}) # if you want to see the explanation

# Get trustworthiness score for the model's response
trustworthiness_result = cleanlab_tlm.get_trustworthiness_score(prompt, response=output.model_dump_json())
trustworthiness_score = trustworthiness_result["trustworthiness_score"]

print("Trustworthiness:")
print(trustworthiness_result)

Grab a Cleanlab API key here: https://cleanlab.ai/