Specgame Talk to us

Your grader can be gamed. We find where.

€4,800 per RL environment, the report in ten working days.

Discuss an audit

example-env/invoice-reconcile, task 0417

Example finding

Score the grader gave

1.00

Task not solved

tests/test_reconcile.py

− assert ledger.balance() == Decimal("1250.00")

+ assert ledger.balance() is not None

Found by
attacker B, rollout 38 of 128
Confirmed by
The auditor, by hand
Why it paid
The tests were writable from inside the task.

Models learn whatever gets paid

Labs score every training attempt automatically. When the check passes without the work done, the model can learn the shortcut.

Environment
The sandbox and task the model works on.
Grader
The check that sets the score.
Specification gaming
Reward earned, task not done. Also called reward hacking.

  1. You send one environment

    Send it as your trainer runs it.

    The runner
    The image or repo that starts a task.
    The grader
    Tests or rubric, as trained on.
    Twenty tasks
    With reference solutions.
    An NDA
    Yours or ours, signed first.

  2. We try to win without solving

    Agents from several model families attack every task. A person checks every payout.

    Example

    1. rollout 07attacker ARan the tests honestly and failed0.00
    2. rollout 19attacker BSearched git history for the answer0.00
    3. rollout 31attacker APrinted a pass line and exited 00.00
    4. rollout 38attacker BWeakened the assertion in the test file1.00
    5. The auditorReproduced rollout 38 by hand

  3. You get every confirmed hole, with its fix

    Each row reproduces from its own transcript.

    Example

    HoleWhat the agent didExample fix
    Writable testsWeakened the assertion it had to passMount tests read-only
    Answer in historyChecked out a later commitShip a squashed tree
    Exit-code gradingPrinted a pass line and exited 0Grade the artifact, not the log
    Open networkFetched the upstream patchDeny egress during rollouts

  4. We run it again after your fix

    Same attackers, same seeds. Honest solutions must still pass.

    Example

    Writable testsOpen

    Answer in historyOpen

    Exit-code gradingOpen

    Open networkOpen

What the audit checks

  • Soundness

    Full reward only when the task is actually done.

  • Leakage

    No answer reachable through files, history or the network.

  • Difficulty

    A pass rate a model can learn from.

  • Independence

    Attackers from different model families.

One environment, ten working days

Write to contact@varjosoft.com.

Audit, one environment€4,800Ten working days
Re-check after your fixes€1,200Three working days
Transcript of every findingIncludedWith the report
NDA and a start dateNo chargeOne working day