andrewkkchan

Prompt

[2025-04-28T18:23:31.377052]

System message:

Your input fields are:

1. question (str) Your output fields are:
2. reasoning (str)
3. response (str)

andrewkkchan

tp = dspy.MIPROv2(metric=metric, auto="medium", num_threads=24)

optimized_cot = tp.compile(cot(), trainset=trainset,
                           max_bootstrapped_demos=2, max_labeled_demos=2,
                           requires_permission_to_run=False)


evaluate(optimized_cot)

andrewkkchan

# Define an evaluator that we can re-use.
evaluate = dspy.Evaluate(devset=devset, metric=metric, num_threads=1,
                         display_progress=True, display_table=2)

# Evaluate the Chain-of-Thought program.
evaluate(cot)

andrewkkchan

from dspy.evaluate import SemanticF1

# Instantiate the metric.
metric = SemanticF1(decompositional=True)

# Produce a prediction from our `cot` module, using the `example` above as input.
pred = cot(**example.inputs())

# Compute the metric score for the prediction.
score = metric(example, pred)

andrewkkchan

{"question": "I\u2019m a 58-year-old investor with a aggressive risk tolerance. I have $120,000 to invest and would like to grow it over the next 25 years for my retirement. Can you run Monte Carlo simulations to estimate potential returns and recommend suitable fund products?", "response": "**Reasoning (chain-of-thought)**\n1. Identify time horizon: 25 years.\n2. Use historical real return distribution ~ N(6%, 11% SD) adjusted for aggressive tilt.\n3. Run 10,000-path Monte Carlo; summarise key percentiles.\n4. Match aggressive profile to blended equity/bond funds with global diversification.\n\n**Simulation results**\n- 10th percentile (bear case): $153,891.84\n- Median outcome: $515,024.49\n- 90th percentile (bull case): $1,034,769.68\n\n**Recommended funds**\n- Vanguard Balanced Index Admiral (VBIAX)\n- Dimensional Global 60/40\n- Vanguard Target Retirement 2035\n\nRebalance annually and review allocations every 3\u202f\u2013\u202f5\u202fyears or after major life events.", "gold_doc_ids": [1000]}

andrewkkchan

import ujson
import random

with open("investor_sim_examples.jsonl") as f:
    data = [ujson.loads(line) for line in f]

data = [dspy.Example(**d).with_inputs('question') for d in data]
random.Random(0).shuffle(data)
trainset, devset, testset = data[:80], data[80:90], data[90:100]

andrewkkchan

Prompt

[2025-04-28T17:25:30.424393]

System message:

Your input fields are:

1. question (str) Your output fields are:
2. reasoning (str)

andrewkkchan

class QACot(dspy.Module):
    def __init__(self):
        super().__init__()
        self.cot = dspy.ChainOfThought(QASignature)

    def forward(self, **kwargs):
        return self.cot(**kwargs)

    def predictors(self):
        return self.cot.predictors() 

andrewkkchan

[2025-04-28T11:48:38.278994]

System message:

Your input fields are:

1. question (str)

andrewkkchan

import os
from llama_index.llms.bedrock import Bedrock

llm = Bedrock(
    model_id="anthropic.claude-3-sonnet-20250219-v1:0",
    temperature=0,
)

question = (
    "I’m a 40-year-old investor with a moderate risk tolerance. "