Skip to content

Instantly share code, notes, and snippets.

@slopp

slopp/ultra-profile-chatnvidia-patches.md

Last active July 3, 2026 23:51
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save slopp/d314c5da6421a2ee3e59d10074f13e06 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save slopp/d314c5da6421a2ee3e59d10074f13e06 to your computer and use it in GitHub Desktop.
Download ZIP
Nemotron Ultra Deep Agents ChatNVIDIA patches
Raw
ultra-profile-chatnvidia-patches.md

Nemotron Ultra Deep Agents: ChatNVIDIA patches used in addition to the harness profile

This note documents the client/runtime patches we used in addition to the downloaded Nemotron Ultra Deep Agents harness profile.

The harness profile itself is registered with:

--harness-profile nemotron-ultra-downloaded

That profile contributes the system-prompt suffix, read_file tool-description override, and middleware stack, including:

  • NemotronProgressBudgetMiddleware
  • DomainToolPreferenceMiddleware
  • FilesystemRequestNudgeMiddleware
  • ConversationTransitionNudgeMiddleware
  • ActionCommitNudgeMiddleware
  • ToolChainCompletionNudgeMiddleware
  • DomainToolCompletionNudgeMiddleware
  • ReadFileContinuationNoticeMiddleware
  • ToolRetryMiddleware
  • NemotronToolCallShim
  • ModelRateLimitRetryMiddleware
  • NemotronTextToolCallParser
  • FollowupDisciplineMiddleware
  • EntityResolutionGuardMiddleware
  • FinalAnswerGuardMiddleware

The patches below are separate compatibility fixes for the ChatNVIDIA / NVIDIA Inference Hub client path.

Why patches were needed

The external LC profile runs used a different OpenAI-compatible client/provider path. Our local TME runs used ChatNVIDIA, which had a few behavior differences that affected Deep Agents evals:

  1. Standard LangChain AIMessage.tool_calls could be present without equivalent OpenAI-format additional_kwargs["tool_calls"], so ChatNVIDIA could fail to send prior assistant tool calls back in the expected shape.
  2. ToolMessage.name was not preserved in the serialized payload.
  3. Non-assistant messages with content=None could reach the NVIDIA client validators/API in an invalid shape.
  4. The Ultra model factory defaulted to max_tokens=32768; for this eval path we wanted no explicit output cap from ChatNVIDIA and relied on the model/profile context settings instead.
  5. Transient provider errors from the Inference Hub path, especially 504 Gateway Time-out from judge/user-simulator calls, could fail a test before the agent under evaluation did useful work.
  6. The Inference Hub aliases require a LiteLLM virtual key with the sk- prefix. The launcher now fails fast if the shell is accidentally using a non-sk- NVIDIA/NGC key.

Patch behavior

The guarded launcher applies these patches before pytest starts:

from langchain_core.messages import AIMessage, ToolMessage
from langchain_core.messages.utils import _convert_to_openai_tool_calls
import langchain_nvidia_ai_endpoints._utils as nvidia_utils
import langchain_nvidia_ai_endpoints.chat_models as nvidia_chat_models
from tests.evals import nvidiahub_models

original_convert = nvidia_utils.convert_message_to_dict

def patched_convert_message_to_dict(message):
    message_dict = original_convert(message)

    # Preserve ToolMessage.name for tool-result messages.
    if isinstance(message, ToolMessage) and getattr(message, "name", None):
        message_dict.setdefault("name", message.name)

    # Preserve standard LangChain AIMessage.tool_calls by converting to
    # OpenAI-compatible tool_calls when additional_kwargs did not already carry it.
    if (
        isinstance(message, AIMessage)
        and "tool_calls" not in message_dict
        and getattr(message, "tool_calls", None)
    ):
        message_dict["tool_calls"] = _convert_to_openai_tool_calls(message.tool_calls)
        if message_dict.get("content") == "":
            message_dict["content"] = None

    # NVIDIA/OpenAI-compatible APIs do not accept null content for non-assistant messages.
    if message_dict.get("role") != "assistant" and message_dict.get("content") is None:
        message_dict["content"] = "null"

    return message_dict

nvidia_utils.convert_message_to_dict = patched_convert_message_to_dict
nvidia_chat_models.convert_message_to_dict = patched_convert_message_to_dict

original_make_model = nvidiahub_models.make_nvidiahub_chat_model

def patched_make_nvidiahub_chat_model(alias):
    model = original_make_model(alias)
    if alias == "ultra-tme":
        model.max_tokens = None
    return model

nvidiahub_models.make_nvidiahub_chat_model = patched_make_nvidiahub_chat_model

We also wrap ChatNVIDIA._generate with a bounded retry for transient provider failures:

from langchain_nvidia_ai_endpoints import ChatNVIDIA
import time

original_generate = ChatNVIDIA._generate

def _is_transient_nvidia_error(exc: BaseException) -> bool:
    text = str(exc)
    return any(
        marker in text
        for marker in (
            "[504]",
            "Gateway Time-out",
            "[500]",
            "APIConnectionError",
            "Connection error",
            "No fallback model group found",
            "[429]",
            "rate limit",
            "temporarily unavailable",
        )
    )

def patched_generate(self, *args, **kwargs):
    for attempt in range(1, 5):
        try:
            return original_generate(self, *args, **kwargs)
        except Exception as exc:
            if attempt == 4 or not _is_transient_nvidia_error(exc):
                raise
            time.sleep(min(2 ** (attempt - 1), 8))

ChatNVIDIA._generate = patched_generate

Guardrail before expensive runs

We created a guarded launcher:

scripts/run_ultra_profile_patched.py

The launcher runs a preflight in the same Python process that will execute pytest. It refuses to run unless all of these are true:

  • nemotron-ultra-downloaded is registered for the ChatNVIDIA Ultra model keys.
  • The profile contains the downloaded read_file override.
  • The profile middleware includes ReadFileContinuationNoticeMiddleware, ToolRetryMiddleware, NemotronToolCallShim, NemotronTextToolCallParser, ModelRateLimitRetryMiddleware, and FinalAnswerGuardMiddleware.
  • ChatNVIDIA serializer patch is installed in both _utils and chat_models modules.
  • ToolMessage.name survives serialization.
  • standard AIMessage.tool_calls serializes to OpenAI-format tool_calls.
  • non-assistant content=None is converted to "null".
  • Ultra model construction changes max_tokens from 32768 to None.
  • temperature=1.0, top_p=0.95, and max_input_tokens=256000 are present.
  • the ChatNVIDIA transient retry wrapper is installed with 4 attempts.
  • LANGSMITH_ENDPOINT is the API base, not the UI root: https://langsmith.prd.astra.nvidia.com/api/v1.
  • the judge and tau2 user simulator model aliases are set to the intended Inference Hub aliases.
  • NVIDIA_API_KEY starts with sk-; if NVIDIA_INFERENCE_HUB_API_KEY is set, it must also start with sk-.

Cheap verification command:

uv run --group test python scripts/run_ultra_profile_patched.py \
  --preflight-only \
  --proof-file /tmp/ultra_profile_chatnvidia_preflight.json

A successful proof starts with:

PROFILE_PATCH_PREFLIGHT_OK

and includes:

{
  "chatnvidia_patches": {
    "ai_tool_calls_openai_serialized": true,
    "generate_retry_attempts": 4,
    "generate_retry_installed": true,
    "max_input_tokens": 256000,
    "non_assistant_none_content_to_null": true,
    "serializer_patch_shared": true,
    "temperature": 1.0,
    "tool_message_name_preserved": true,
    "top_p": 0.95,
    "ultra_max_tokens_after": null,
    "ultra_max_tokens_before": 32768
  },
  "expected_harness_profile_arg": "nemotron-ultra-downloaded",
  "expected_judge_model": "nvidiahub:judge-sonnet-inference-hub",
  "expected_model_arg": "nvidiahub:ultra-tme",
  "expected_tau2_user_sim_model": "nvidiahub:tau2-user-gpt-4.1-mini",
  "environment": {
    "DEEPAGENTS_LLM_JUDGE_MODEL": "nvidiahub:judge-sonnet-inference-hub",
    "DEEPAGENTS_TAU2_USER_SIM_MODEL": "nvidiahub:tau2-user-gpt-4.1-mini",
    "NVIDIA_API_KEY_present": true,
    "NVIDIA_API_KEY_startswith_sk": true
  },
  "status": "ok"
}

Running the eval

Use the guarded launcher, not raw pytest:

export LANGSMITH_TRACING=true
export LANGSMITH_ENDPOINT=https://langsmith.prd.astra.nvidia.com/api/v1
export NVIDIA_API_KEY=sk-...
export NVIDIA_INFERENCE_HUB_API_KEY=sk-...
export DEEPAGENTS_LLM_JUDGE_MODEL=nvidiahub:judge-sonnet-inference-hub
export DEEPAGENTS_TAU2_USER_SIM_MODEL=nvidiahub:tau2-user-gpt-4.1-mini
export DEEPAGENTS_NVIDIAHUB_ULTRA_TME_TEMPERATURE=1.0
export DEEPAGENTS_NVIDIAHUB_ULTRA_TME_TOP_P=0.95
export DEEPAGENTS_NVIDIAHUB_ULTRA_TME_MAX_INPUT_TOKENS=256000
export DEEPAGENTS_NVIDIAHUB_JUDGE_SONNET_INFERENCE_HUB_MODEL=aws/anthropic/bedrock-claude-sonnet-4-6
export DEEPAGENTS_NVIDIAHUB_TAU2_USER_GPT_4_1_MINI_MODEL=azure/openai/gpt-4.1-mini
export DEEPAGENTS_EVALS_REPORT_FILE=./report.json

uv run --group test python scripts/run_ultra_profile_patched.py \
  --proof-file ./preflight_proof.json \
  tests/evals \
  --eval-category-exclude memory \
  --model nvidiahub:ultra-tme \
  --harness-profile nemotron-ultra-downloaded

The launcher validates profile + patches before pytest can make model calls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment