Graders

A grader is the verify method of the environment server. It reads the model’s answer and the task row’s grader fields, and it returns a reward from 0 to 1. The eval score and the training reward both come from this one number, and monte env add pins the grader code by hash.

Grader Output

Class	Extends	Contents
Verify request	`BaseVerifyRequest`	The model’s answer as `response`, plus the task row’s grader fields
Verify response	`BaseVerifyResponse`	`reward`, plus extra fields that explain it

The extra fields do not change the reward. They show in each trace why a score is low.

Choose Checks in Code or a Judge Model

	Checks in code	Judge model
How it scores	Code tests the answer: a string match, a parser, a unit test, a tool call	A second model rates the answer against a rubric
Best for	Facts, formats, tool calls, code that must pass tests	Tone, style, helpfulness
Same answer, same score	Always	Only while the judge model, its prompt, and its sampling settings stay the same
Cost per attempt	None	One model call

A grader can mix both. The support_escalation grader checks the route, the category, and the facts in code. In the training environment, it also adds a tone score from the open-weight judge.

Write a Grader

This compact verify gates the reward on the route. A resolved ticket then scores the share of expected facts that the reply states. An escalated ticket scores 1 divided by the number of categories it names, if the right one is among them, so naming all six scores 1/6.

import json

from nemo_gym.base_resources_server import BaseVerifyRequest, BaseVerifyResponse, SimpleResourcesServer

CATEGORIES = ("money", "legal", "dataloss", "cancel", "angry", "kbgap")


class SupportVerifyRequest(BaseVerifyRequest):
    # Grader fields from the task row. The model never sees them.
    should_escalate: bool
    expected_facts: list[str] = []
    kb_entry: int = -1
    expected_category: str = ""


class SupportVerifyResponse(BaseVerifyResponse):
    routed_correctly: bool
    fact_recall: float
    category_match: float


def reply_text(response) -> str:
    return "\n".join(
        chunk.text
        for item in response.output if item.type == "message"
        for chunk in item.content if getattr(chunk, "text", None)
    )


def escalation_reasons(response) -> list[str]:
    return [
        json.loads(item.arguments or "{}").get("reason", "")
        for item in response.output
        if item.type == "function_call" and item.name == "escalate_to_human"
    ]


class SupportServer(SimpleResourcesServer):
    async def verify(self, body: SupportVerifyRequest) -> SupportVerifyResponse:
        reasons = escalation_reasons(body.response)
        escalated = bool(reasons)
        routed = escalated == body.should_escalate
        text = reply_text(body.response).lower()

        found = [f for f in body.expected_facts if f.lower() in text]
        recall = len(found) / len(body.expected_facts) if body.expected_facts else 1.0

        named = {c for c in CATEGORIES if any(c in r.lower() for r in reasons)}
        category = 1 / len(named) if body.expected_category in named else 0.0

        reward = 0.0 if not routed else category if escalated else recall
        return SupportVerifyResponse(
            **body.model_dump(exclude={"should_escalate", "expected_facts", "kb_entry", "expected_category"}),
            reward=reward,
            routed_correctly=routed,
            fact_recall=recall,
            category_match=category,
        )

The full support_escalation grader also scores fact precision against kb_entry, so a reply that recites the whole knowledge base does not score high. In training, it also adds the judge’s tone score.

Choose a Judge Model

A judge model is a model that verify calls. An eval environment can use an open-weight or a closed-weight judge. A training environment uses an open-weight judge only.

	Open-weight model	Closed-weight model
Name	A deployment, such as `qwen3-235b` for `Qwen/Qwen3-235B-A22B`	`<provider>/<model>`, such as `anthropic/claude-sonnet-4-5`
Where it runs	In Monte Cloud, or in the team’s AWS, GCP, or Azure account with BYOC	Behind the provider’s API
What goes to the provider	Nothing	Each task text and answer that it judges
Secret	A Monte inference key, as `JUDGE_API_KEY`	The provider key, such as `ANTHROPIC_API_KEY`
Environments	Eval and training	Eval only

Set the judge model’s name in the server config, so the environment version records it. A closed-weight judge runs with the team’s own key under that provider’s terms, and Monte refuses it in a training environment: monte env add --training stops with E_REFUSAL. The tone score of the support_escalation training grader comes from the open-weight judge. An eval grader can make the same call to a closed-weight judge:

Open-weight
Closed-weight (eval only)

import os

from openai import AsyncOpenAI


class SupportConfig(BaseResourcesServerConfig):
    judge_model: str = "qwen3-235b"  # The deployment name
    judge_url: str = "https://<deployment-endpoint>/v1"


async def rate_tone(config: SupportConfig, reply: str) -> float:
    judge = AsyncOpenAI(base_url=config.judge_url, api_key=os.environ["JUDGE_API_KEY"])
    result = await judge.chat.completions.create(
        model=config.judge_model,
        temperature=0,
        max_tokens=5,
        messages=[{"role": "user", "content": TONE_PROMPT.format(reply=reply)}],
    )
    text = result.choices[0].message.content.strip()
    return int(text) / 5 if text in {"1", "2", "3", "4", "5"} else 0.0  # TONE_PROMPT asks for 1 to 5

import os

from anthropic import AsyncAnthropic


class SupportConfig(BaseResourcesServerConfig):
    judge_model: str = "anthropic/claude-sonnet-4-5"


async def rate_tone(config: SupportConfig, reply: str) -> float:
    judge = AsyncAnthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
    result = await judge.messages.create(
        model=config.judge_model.split("/", 1)[1],  # The model ID without the provider
        temperature=0,
        max_tokens=5,
        messages=[{"role": "user", "content": TONE_PROMPT.format(reply=reply)}],
    )
    text = result.content[0].text.strip()
    return int(text) / 5 if text in {"1", "2", "3", "4", "5"} else 0.0  # TONE_PROMPT asks for 1 to 5

Security describes what a closed-weight judge sends out.

Grade Open-Ended Text

A task about style or tone has no one correct answer. For such a task, the grader fields hold a style_guide or a reference reply, and an open-weight judge scores the reply against it.

STYLE_PROMPT = """Rate how well the reply follows the style guide, from 1 to 5.
Answer with the number only.

Style guide:
{style_guide}

Reply:
{reply}"""


class StyleVerifyRequest(BaseVerifyRequest):
    style_guide: str  # A grader field from the task row


async def rate_style(config: SupportConfig, style_guide: str, reply: str) -> float:
    judge = AsyncOpenAI(base_url=config.judge_url, api_key=os.environ["JUDGE_API_KEY"])
    result = await judge.chat.completions.create(
        model=config.judge_model,
        temperature=0,
        max_tokens=5,
        messages=[{"role": "user", "content": STYLE_PROMPT.format(style_guide=style_guide, reply=reply)}],
    )
    text = result.choices[0].message.content.strip()
    return int(text) / 5 if text in {"1", "2", "3", "4", "5"} else 0.0  # Any other answer scores 0

verify returns this score as the reward, or mixes it with checks in code.

Give the Grader a Secret

Store a judge key as a team secret. Every job of the team, including each eval, training run, and regrade, reads it as an environment variable with the same name.

monte secret set JUDGE_API_KEY

The environment version does not record the value, so a new key does not change the fingerprint. Run history and logs never show it. Setup Commands lists the flags.

Fix a Grader and Regrade

A fixed grader is a new environment version, so it needs a new experiment. monte grade sends the stored replies of an old eval run to the new grader, without serving the model again:

monte env add --eval support_escalation_dev --as support-escalation-dev      # makes @2
monte experiment create support-triage-v2 --held-out support-escalation --dev support-escalation-dev@2
monte grade support-triage/eval-01 --into support-triage-v2
monte grade support-triage/eval-02 --into support-triage-v2

Each monte grade writes a new eval run in support-triage-v2, and the regraded baselines are the baselines there. The command refuses with E_NOT_COMPARABLE when the target eval has other tasks or other eval settings. A grader that reads the state of the attempt cannot regrade. Evaluation Commands lists the flags.

Get Started

Training

Inference

Datasets

Environments

Evaluation

Observability

Security

Grader Output

Choose Checks in Code or a Judge Model

Write a Grader

Choose a Judge Model

Grade Open-Ended Text

Give the Grader a Secret

Fix a Grader and Regrade

​Grader Output

​Choose Checks in Code or a Judge Model

​Write a Grader

​Choose a Judge Model

​Grade Open-Ended Text

​Give the Grader a Secret

​Fix a Grader and Regrade

Grader Output

Choose Checks in Code or a Judge Model

Write a Grader

Choose a Judge Model

Grade Open-Ended Text

Give the Grader a Secret

Fix a Grader and Regrade