How Scoring Works

A score is not produced by asking a model what it thinks of the code. It comes from three layers, in order.

1. Deterministic grading

The submission is executed in a clean environment and run through public tests, hidden tests, integration and end-to-end tests, lint, static analysis, coverage, performance checks, security checks, and database or concurrency assertions where the task defines them.

This is objective and reproducible. It produces most of the correctness score and much of testing and security.

2. Behavioural telemetry

The full event log of the session: prompts, agent responses, tool calls, code authored by hand, code generated, diffs, reverts, terminal commands, tests run, suggestions accepted and rejected, time spent, tokens used, iterations, errors.

This is where AI collaboration and efficiency come from — and where attribution is calculated.

3. Reviewer agent

Only after the first two layers does a reviewer agent run. It receives the task, the hidden rubric, the final code, the grader results, the complete conversation, the event history, and the attribution data.

It produces the subjective dimensions — architecture, code quality, engineering judgement — and every finding it makes must cite evidence from the event log.

Why efficiency is not token count

A senior engineer often uses more tokens: they ask the agent to review its own work, explore an alternative, or threat-model the design. That is good practice and would be punished by a raw count.

Efficiency measures outcome relative to effort. The question is whether the candidate used the agent deliberately, not whether they used it sparingly.

Consistency

Deterministic grading is identical every time. The reviewer layer is model-driven and can vary slightly between runs on borderline judgements.

Two consequences worth knowing: treat small differences between candidates as noise, and treat the evidence as the real output. The evidence does not drift — it is a record of what happened.