Skip to content
BoringStack
Star

Agent tooling evaluation

3 min read

agent:eval is a maintainer tool for one concrete Projects feature. It does not start Claude or Cursor, call a model provider, or establish a general agent benchmark. Everyday feature work uses agent:verify.

Validate the evaluator

Section titled “Validate the evaluator”

With dependencies installed and Docker available, run from the repository root:

Terminal window
bun run agent:check
bun run agent:check:docker
bun run agent:eval --deterministic

The deterministic run creates a docs-free fixture, generates Projects, starts isolated services and installs acceptance tests from the reviewing checkout. It checks a known-good implementation and deliberately introduces missing tenant predicates, missing role enforcement, client-selected ownership, an extra delete operation, stale cache invalidation and a missing migration.

An evaluator check passes when the observed named outcomes match expectations. For an intentionally defective variant, that includes the intended assertion failures and passing controls. A crashed fixture is not evidence that a defect was detected.

The run also checks generated API/UI code, exercises Projects through a browser and applies a description migration over existing data. The example UI and acceptance templates are test fixtures, not starter features installed in the application.

Review an independently implemented Projects feature

Section titled “Review an independently implemented Projects feature”
Terminal window
bun run agent:eval -- --candidate=/absolute/path/to/checkout

The reviewer copies only declared production integration files and appended Drizzle artifacts into its fixture. Existing migration history must match. Candidate tests, scripts, CI and evaluator code are excluded; missing SQL is not generated for the candidate. Files must be regular files inside the selected checkout.

The same named API/UI/browser acceptance and app static checks run against that submission. This first judge does not support arbitrary resource names or task schemas. Security and release verification remain separate requirements.

Candidate execution runs automatically inside a non-root Docker container. Its root filesystem is read-only; writable temporary storage is bounded. It has no host bind mounts, Docker socket, inherited host credentials, published ports or outbound network. Disposable Postgres and Valkey share only the runner’s private loopback namespace. CPU, memory and process limits apply.

The trusted image is built from the reviewer’s source and locked dependencies before candidate files are copied in. Candidate scripts, dependencies and configuration never control that build. Docker must be available; there is no fallback to executing a candidate on the host. The controller removes its containers and image tag on completion or failure. If cleanup fails, the command reports the run’s resource names for inspection.

This boundary protects the host. It does not make assertions running in the same process as arbitrary application code tamper-proof, and containers still share their Docker host’s kernel. Continue reviewing candidate code and evidence. agent:check:docker includes real isolation probes and a complete positive candidate control.

Measure agents separately

Section titled “Measure agents separately”

Task prompts under tools/agent-evals/tasks describe Projects, a description migration and a tenant fix. They are inputs for future measured agent runs. Deterministic success only validates this judge against its known examples.

For each live run, retain the agent/model version, prompt, resulting patch, verifier revision, human interventions and complete outcomes. No live-model benchmark result is claimed by this tooling. Selecting agent configurations, executing repeated runs and evaluating tasks beyond Projects are separate work.

Section titled “Related”