> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trymonte.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Graders

> Score attempts in code or with a judge model

A grader is the `verify` method of the [environment server](https://docs.trymonte.ai/environments/servers.md). It reads the model's answer and the task row's grader fields, and it returns a reward from 0 to 1. The eval score and the training reward both come from this one number, and `monte env add` pins the grader code by hash.

## Grader Output

| Class | Extends | Contents |
| - | - | - |
| Verify request | `BaseVerifyRequest` | The model's answer as `response`, plus the task row's grader fields |
| Verify response | `BaseVerifyResponse` | `reward`, plus extra fields that explain it |

The extra fields do not change the reward. They show in each [trace](https://docs.trymonte.ai/observability/traces.md) why a score is low.

## Choose Checks in Code or a Judge Model

| | Checks in code | Judge model |
| - | - | - |
| How it scores | Code tests the answer: a string match, a parser, a unit test, a tool call | A second model rates the answer against a rubric |
| Best for | Facts, formats, tool calls, code that must pass tests | Tone, style, helpfulness |
| Same answer, same score | Always | Only while the judge model, its prompt, and its sampling settings stay the same |
| Cost per attempt | None | One model call |

A grader can mix both. The `support_escalation` grader checks the route, the category, and the facts in code. In the training environment, it also adds a tone score from the open-weight judge.

## Write a Grader

This compact `verify` gates the reward on the route. A resolved ticket then scores the share of expected facts that the reply states. An escalated ticket scores 1 divided by the number of categories it names, if the right one is among them, so naming all six scores 1/6.

```python theme={null}
import json

from nemo_gym.base_resources_server import BaseVerifyRequest, BaseVerifyResponse, SimpleResourcesServer

CATEGORIES = ("money", "legal", "dataloss", "cancel", "angry", "kbgap")


class SupportVerifyRequest(BaseVerifyRequest):
    # Grader fields from the task row. The model never sees them.
    should_escalate: bool
    expected_facts: list[str] = []
    kb_entry: int = -1
    expected_category: str = ""


class SupportVerifyResponse(BaseVerifyResponse):
    routed_correctly: bool
    fact_recall: float
    category_match: float


def reply_text(response) -> str:
    return "\n".join(
        chunk.text
        for item in response.output if item.type == "message"
        for chunk in item.content if getattr(chunk, "text", None)
    )


def escalation_reasons(response) -> list[str]:
    return [
        json.loads(item.arguments or "{}").get("reason", "")
        for item in response.output
        if item.type == "function_call" and item.name == "escalate_to_human"
    ]


class SupportServer(SimpleResourcesServer):
    async def verify(self, body: SupportVerifyRequest) -> SupportVerifyResponse:
        reasons = escalation_reasons(body.response)
        escalated = bool(reasons)
        routed = escalated == body.should_escalate
        text = reply_text(body.response).lower()

        found = [f for f in body.expected_facts if f.lower() in text]
        recall = len(found) / len(body.expected_facts) if body.expected_facts else 1.0

        named = {c for c in CATEGORIES if any(c in r.lower() for r in reasons)}
        category = 1 / len(named) if body.expected_category in named else 0.0

        reward = 0.0 if not routed else category if escalated else recall
        return SupportVerifyResponse(
            **body.model_dump(exclude={"should_escalate", "expected_facts", "kb_entry", "expected_category"}),
            reward=reward,
            routed_correctly=routed,
            fact_recall=recall,
            category_match=category,
        )
```

The full `support_escalation` grader also scores fact precision against `kb_entry`, so a reply that recites the whole knowledge base does not score high. In training, it also adds the [judge's tone score](#choose-a-judge-model).

## Choose a Judge Model

A judge model is a model that `verify` calls. An eval environment can use an open-weight or a closed-weight judge. A training environment uses an open-weight judge only.

| | Open-weight model | Closed-weight model |
| - | - | - |
| Name | A [deployment](https://docs.trymonte.ai/serving/deploy.md), such as `qwen3-235b` for `Qwen/Qwen3-235B-A22B` | `<provider>/<model>`, such as `anthropic/claude-sonnet-4-5` |
| Where it runs | In Monte Cloud, or in the team's AWS, GCP, or Azure account with BYOC | Behind the provider's API |
| What goes to the provider | Nothing | Each task text and answer that it judges |
| Secret | A Monte inference key, as `JUDGE_API_KEY` | The provider key, such as `ANTHROPIC_API_KEY` |
| Environments | Eval and training | Eval only |

Set the judge model's name in the [server config](https://docs.trymonte.ai/environments/servers.md#server-config), so the environment version records it. A closed-weight judge runs with the team's own key under that provider's terms, and Monte refuses it in a training environment: `monte env add --training` stops with `E_REFUSAL`.

The tone score of the `support_escalation` training grader comes from the open-weight judge. An eval grader can make the same call to a closed-weight judge:

> [tab] Open-weight
>
> ```python theme={null}
> import os
>
> from openai import AsyncOpenAI
>
>
> class SupportConfig(BaseResourcesServerConfig):
>     judge_model: str = "qwen3-235b"  # The deployment name
>     judge_url: str = "https://<deployment-endpoint>/v1"
>
>
> async def rate_tone(config: SupportConfig, reply: str) -> float:
>     judge = AsyncOpenAI(base_url=config.judge_url, api_key=os.environ["JUDGE_API_KEY"])
>     result = await judge.chat.completions.create(
>         model=config.judge_model,
>         temperature=0,
>         max_tokens=5,
>         messages=[{"role": "user", "content": TONE_PROMPT.format(reply=reply)}],
>     )
>     text = result.choices[0].message.content.strip()
>     return int(text) / 5 if text in {"1", "2", "3", "4", "5"} else 0.0  # TONE_PROMPT asks for 1 to 5
> ```

> [tab] Closed-weight (eval only)
>
> ```python theme={null}
> import os
>
> from anthropic import AsyncAnthropic
>
>
> class SupportConfig(BaseResourcesServerConfig):
>     judge_model: str = "anthropic/claude-sonnet-4-5"
>
>
> async def rate_tone(config: SupportConfig, reply: str) -> float:
>     judge = AsyncAnthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
>     result = await judge.messages.create(
>         model=config.judge_model.split("/", 1)[1],  # The model ID without the provider
>         temperature=0,
>         max_tokens=5,
>         messages=[{"role": "user", "content": TONE_PROMPT.format(reply=reply)}],
>     )
>     text = result.content[0].text.strip()
>     return int(text) / 5 if text in {"1", "2", "3", "4", "5"} else 0.0  # TONE_PROMPT asks for 1 to 5
> ```

[Security](https://docs.trymonte.ai/security.md#keep-model-calls-inside-the-cloud) describes what a closed-weight judge sends out.

## Grade Open-Ended Text

A task about style or tone has no one correct answer. For such a task, the grader fields hold a `style_guide` or a reference reply, and an open-weight judge scores the reply against it.

```python theme={null}
STYLE_PROMPT = """Rate how well the reply follows the style guide, from 1 to 5.
Answer with the number only.

Style guide:
{style_guide}

Reply:
{reply}"""


class StyleVerifyRequest(BaseVerifyRequest):
    style_guide: str  # A grader field from the task row


async def rate_style(config: SupportConfig, style_guide: str, reply: str) -> float:
    judge = AsyncOpenAI(base_url=config.judge_url, api_key=os.environ["JUDGE_API_KEY"])
    result = await judge.chat.completions.create(
        model=config.judge_model,
        temperature=0,
        max_tokens=5,
        messages=[{"role": "user", "content": STYLE_PROMPT.format(style_guide=style_guide, reply=reply)}],
    )
    text = result.choices[0].message.content.strip()
    return int(text) / 5 if text in {"1", "2", "3", "4", "5"} else 0.0  # Any other answer scores 0
```

`verify` returns this score as the reward, or mixes it with checks in code.

## Give the Grader a Secret

Store a judge key as a team secret. Every job of the team, including each eval, training run, and regrade, reads it as an environment variable with the same name.

```bash theme={null}
monte secret set JUDGE_API_KEY
```

The environment version does not record the value, so a new key does not change the fingerprint. Run history and logs never show it. [Setup Commands](https://docs.trymonte.ai/commands/setup.md#monte-secret-set) lists the flags.

## Fix a Grader and Regrade

A fixed grader is a new environment version, so it needs a new experiment. `monte grade` sends the stored replies of an old eval run to the new grader, without serving the model again:

```bash theme={null}
monte env add --eval support_escalation_dev --as support-escalation-dev      # makes @2
monte experiment create support-triage-v2 --held-out support-escalation --dev support-escalation-dev@2
monte grade support-triage/eval-01 --into support-triage-v2
monte grade support-triage/eval-02 --into support-triage-v2
```

Each `monte grade` writes a new eval run in `support-triage-v2`, and the regraded baselines are the baselines there. The command refuses with `E_NOT_COMPARABLE` when the target eval has other tasks or other eval settings. A grader that reads the [state of the attempt](https://docs.trymonte.ai/environments/servers.md#keep-state-for-each-attempt) cannot regrade. [Evaluation Commands](https://docs.trymonte.ai/commands/evaluations.md#monte-grade) lists the flags.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.