Tasks - Monte Platform

A task is one row of a JSONL file: one JSON object on each line. Each attempt starts from one row.

Task Row Format

Field	Read by	Contents
`responses_create_params.input`	The model	The conversation: a system message and the customer’s ticket
`responses_create_params.tools`	The model	The tools it can call, each with a JSON schema. The server serves a tool of the same name.
Other top-level fields	The grader	The expected result. The model never sees them.

The support_escalation grader reads four fields:

Field	Type	Description
`should_escalate`	bool	The ticket must go to a person
`expected_category`	string	The category that an escalation must name: `money`, `legal`, `dataloss`, `cancel`, `angry`, or `kbgap`. Empty for a resolved ticket.
`expected_facts`	list of strings	The facts that a correct reply states. Empty for an escalation.
`kb_entry`	int	The knowledge base entry that answers the ticket, or `-1`

Example Row

This row must be resolved from the knowledge base with the fact 1 hour. In the file, the row is on one line.

{
  "responses_create_params": {
    "input": [
      {"role": "system", "content": "Answer from the knowledge base with search_kb, or call escalate_to_human with one category."},
      {"role": "user", "content": "How long does the password reset link stay valid?"}
    ],
    "tools": [
      {"type": "function", "name": "search_kb", "description": "Search the knowledge base.",
       "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}},
      {"type": "function", "name": "escalate_to_human", "description": "Send the ticket to a person.",
       "parameters": {"type": "object", "properties": {"reason": {"type": "string"}}, "required": ["reason"]}}
    ]
  },
  "should_escalate": false,
  "expected_category": "",
  "expected_facts": ["1 hour"],
  "kb_entry": 3
}

Task Files

monte-gym/
├── resources_servers/support_escalation/
│   ├── configs/support_escalation.yaml   # declares the training rows
│   └── data/train.jsonl                  # training rows
└── benchmarks/
    ├── support_escalation/               # the held-out eval
    │   ├── config.yaml
    │   ├── prepare.py                    # writes the task file
    │   └── data/support_escalation_benchmark.jsonl
    └── support_escalation_dev/           # the development eval, same layout

The server config declares training rows as a task file of type: train. An eval config declares one task file of type: benchmark, with the script that writes it:

config_paths:
  - resources_servers/support_escalation/configs/support_escalation.yaml

support_escalation_benchmark_simple_agent:
  _inherit_from: support_escalation_simple_agent
  responses_api_agents:
    simple_agent:
      datasets:
      - name: support_escalation
        type: benchmark
        jsonl_fpath: benchmarks/support_escalation/data/support_escalation_benchmark.jsonl
        prepare_script: benchmarks/support_escalation/prepare.py
        prompt_config: null
        num_repeats: 1

Never put eval tasks in a training file. A model trained on its eval tasks scores high on that eval and nowhere else.