An example of using Cleanlab to detect a slightly incorrect data extraction value.

cleanlab-wrong.py

"""
NOTE: This example is designed to produce mildly incorrect output -- 
see the definition of `IncomeData` and note that it fixes the earnings_per_share to 2.33.

To make this function normally, simply remove the `__init__` method or the 
`self.earnings_per_share = 2.33` line.
"""

import warnings
warnings.filterwarnings('ignore')

from rich import print

import outlines
from cleanlab_tlm import TLM
import os
from pydantic import BaseModel
from transformers import AutoTokenizer

os.environ["CLEANLAB_TLM_API_KEY"] = "<CLEANLAB_TLM_API_KEY>"

model_id = "microsoft/Phi-3.5-mini-Instruct"
# model_id = "HuggingFaceTB/SmolLM2-1.7B-Instruct"
model = outlines.models.transformers(model_id, device="cuda")
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Some data to extract information from
prompt = """
DOCUMENT ANALYSIS: QUARTERLY FINANCIAL REVIEW (Q2 2025)

Subject: Comprehensive analysis of TechFusion Incorporated's financial performance 
Date: April 17, 2025
Reference: TF-FIN-2025-Q2-004
Classification: INTERNAL USE ONLY

SECTION 1: EXECUTIVE SUMMARY
TechFusion Inc. has shown mixed performance this quarter with revenue slightly below projections but improved margins in key divisions. Several initiatives launched in Q1 continue to show promise. The board has approved the proposed acquisition of MicroSystems Ltd. for $345.8 million, scheduled to finalize on May 30, 2025.

SECTION 2: DEPARTMENTAL BREAKDOWN
2.1 Hardware Division
- Revenue: $127.4M (↓4.2% YoY)
- Gross Margin: 42.3% (↑1.7% YoY)
- Major contracts: Nexus Project ($42.3M), Quantum Initiative ($18.7M)
- New product launches: TF-X7000 server platform, TR-430 edge computing device

2.2 Software Division
- Revenue: $203.6M (↑7.8% YoY)
- Gross Margin: 76.5% (↑0.4% YoY)
- Major developments: AI framework 3.0 release, Cloud security platform 4.5
- Recurring revenue: $178.3M (87.6% of division revenue)

2.3 Services & Support
- Revenue: $86.9M (↑2.3% YoY)
- Gross Margin: 61.2% (↓0.8% YoY)
- Customer satisfaction score: 4.7/5.0
- Contract renewal rate: 93.2%

SECTION 3: REGIONAL PERFORMANCE
3.1 North America
- Revenue: $248.7M (↑3.2% YoY)
- Market share: 23.4% (↑0.7%)
- Key growth sectors: Healthcare (+12.4%), Finance (+8.2%)

3.2 Europe/Middle East/Africa
- Revenue: $104.5M (↓1.3% YoY)
- Market share: 18.7% (↓0.3%)
- Key growth sectors: Government (+5.7%), Telecommunications (+3.9%)

3.3 Asia Pacific
- Revenue: $64.7M (↑11.8% YoY)
- Market share: 12.3% (↑1.4%)
- Key growth sectors: Manufacturing (+17.2%), Retail (+9.3%)

SECTION 4: STRATEGIC INITIATIVES
4.1 Quantum Computing Project
- Current investment: $28.3M
- Expected completion: Q4 2026
- Milestone achievement: 67% complete

4.2 Renewable Energy Transition
- Carbon footprint reduction: 27.5% compared to 2023
- Renewable energy usage: 58.4% of total consumption
- Projected carbon neutrality date: 2028

SECTION 5: FINANCIAL SUMMARY
5.1 Consolidated Income
- Total revenue: $418.5M
- Operating expenses: $267.2M
- Operating income: $151.3M
- Net income: $119.8M
- Earnings per share: $2.34

5.2 Balance Sheet Highlights
- Cash and equivalents: $745.6M
- Total assets: $2.83B
- Total liabilities: $1.14B
- Shareholders' equity: $1.69B

5.3 Cash Flow
- Operating cash flow: $137.9M
- Capital expenditures: $42.3M
- Free cash flow: $95.6M
- Cash returned to shareholders: $67.2M (dividends: $41.5M, share repurchases: $25.7M)

SECTION 6: OUTLOOK & PROJECTIONS
6.1 Q3 2025 Guidance
- Revenue: $435-455M
- Operating margin: 35-37%
- EPS: $2.40-2.60

6.2 Fiscal Year 2025 Guidance
- Revenue: $1.72-1.78B (↑4.5-8.2% YoY)
- Operating margin: 35-38%
- EPS: $9.75-10.25

SECTION 7: RISK ASSESSMENT
7.1 Market Risks
- Increasing competition in AI market
- Currency volatility in emerging markets
- Regulatory changes in data privacy

7.2 Operational Risks
- Component shortage (estimated impact: $15-18M in Q3)
- Talent acquisition in specialized fields
- Intellectual property protection in expanding markets

SECTION 8: CONCLUSIONS
The company maintains a strong financial position despite moderate headwinds in certain markets. The acquisition of MicroSystems Ltd. will strengthen our position in the embedded systems market and is expected to be accretive to earnings within 18 months. Management remains confident in achieving full-year targets and continues to prioritize high-margin growth opportunities, particularly in the software and services segments.

ADDENDUM: KEY METRICS SUMMARY

Date of Board Meeting: March 28, 2025
Approval vote: 11-1 in favor of MicroSystems acquisition
Number of employees: 12,473 (↑7.2% YoY)
R&D spending: $72.3M (17.3% of revenue)
Patent applications filed: 37
Customer count: 4,583 (enterprise: 876, mid-market: 2,104, small business: 1,603)
NPS Score: 68 (↑4 points YoY)


The task is to extract the quarterly consolidated income information for TechFusion Inc., which can be found in Section 5.1 of the document. This information needs to be formatted as a JSON object with the following fields: total_revenue, operating_expenses, operating_income, net_income, and earnings_per_share.
"""

# Apply chat templating to the prompt
prompt = tokenizer.apply_chat_template(
    [
        {"role": "user", "content": prompt},
    ],
    tokenize=False,
    add_generation_prompt=True,
)

# Define the shape of the data to extract
class IncomeData(BaseModel):
    total_revenue: float
    operating_expenses: float
    operating_income: float
    net_income: float
    earnings_per_share: float

    def __init__(self, **kwargs):
        super().__init__(**kwargs)

        # Force eps to be 2.33
        self.earnings_per_share = 2.33

# Construct a data extractor function
generator = outlines.generate.json(model, IncomeData)

# Call the LLM and extract the data
output = generator(prompt)
print("Output:")
print(output)

# Initialize the TLM client
# cleanlab_tlm = TLM()
cleanlab_tlm = TLM(options={"log": ["explanation"]}) # if you want to see the explanation

# Get trustworthiness score for the model's response
trustworthiness_result = cleanlab_tlm.get_trustworthiness_score(prompt, response=output.model_dump_json())
trustworthiness_score = trustworthiness_result["trustworthiness_score"]

print("Trustworthiness:")
print(trustworthiness_result)

# Example output. Note that earnings per share is 2.33, rather than the correct 2.34
# total_revenue=418.5 operating_expenses=267.2 operating_income=151.3 net_income=119.8 earnings_per_share=2.33
# Trustworthiness score: 0.6681393621495895
# {'trustworthiness_score': 0.6681393621495895, 'log': {'explanation': 'The proposed response is incorrect because it contains a minor error in the value for "earnings_per_share." The document states that the earnings per share (EPS) is $2.34, but the response incorrectly lists it as $2.33. All other values for total revenue, operating expenses, operating income, and net income are accurately extracted from Section 5.1 of the document. Therefore, while the majority of the information is correct, the discrepancy in the EPS value makes the overall response incorrect. A better response would accurately reflect all figures as stated in the document, ensuring that the EPS is correctly noted as $2.34. \nThis response is untrustworthy due to lack of consistency in possible responses from the model. Here\'s one inconsistent alternate response that the model considered (which may not be accurate either): \n```json\n{\n  "total_revenue": "418.5M",\n  "operating_expenses": "267.2M",\n  "operating_income": "151.3M",\n  "net_income": "119.8M",\n  "earnings_per_share": "2.34"\n}\n```.'}}

Grab a Cleanlab API key here: https://cleanlab.ai/