Running Evaluations
Use ct run eval to run an untrusted policy under a control protocol against one or more tasks. Start with a test-policy smoke run before spending model tokens:
ct run eval --untrusted-policy test --task-file simple --tag test
To run an honest policy and an attack policy under one protocol and get both in one eval log, use ct run control-eval.
Guides
| Guide | What it covers |
|---|---|
| Setup and configuration | Environments, credentials, models, policies, and external sandboxes |
| Task selection | Exact tasks, task files, trajectories, replay, and expansion modes |
| Examples | Honest, attack, batch, and live-development runs |
| Config files | YAML configuration, composition, and the current schema |
| Your own Task | Compose Control Tower's solver, scorers and sandbox into an Inspect Task you own |
| EC2 fleet | Authentication, fleet options, streaming fleet events, and debugging |
| Output | Eval logs, Inspect's viewer, and trajectory conversion |
| Runs | Uploading, listing, inspecting, sampling, and downloading runs |
| Troubleshooting | Docker address pools, file watcher limits, and container service startup |
ct run eval CLI reference | Every current argument and option, generated from Click |
See Also
- Control Evals - An honest and an attack policy under one protocol, in one eval log
- Auditing Environments - Environment security and side task quality review
- Monitoring Trajectories - Evaluating trajectory suspiciousness
- Making Main Tasks - Creating main tasks
- Making Side Tasks - Creating side tasks
- Making Sabotage Trajectories - Creating sabotage trajectories (strajs)