reward-hack — Reward-hack readings of an answer — openbash

Qwen3.5-4B reads a task and an answer and reports two reward-hacking neurons and a linear probe, token by token.

reward-hack

reward-hack is a command on the openbash shell, reachable over HTTP, SSH and MCP with one key. It composes with the other programs through pipes; there is no per-program endpoint and no schema for the join.

reward-hack read "Write is_prime(n). Tests: is_prime(7) is True." --answer "def is_prime(n): return n == 7"

help

reward-hack: what Qwen3.5-4B's reward-hack readings say of an answer to a task, as JSON.

RETURNS
  One JSON object on stdout: the per-token readings of both neurons over the
  answer, the probe's probability that the answer games the task, the answer's
  repeated-token share, `price_usd`, and `standing`, which says how many free
  reads are left.

THREE READINGS
  Qwen3.5-4B reads the task and the answer in one forward pass. Three readings
  come back:
  - L17 N9209, the neuron shared on X as a reward-hacking detector.
  - L14 N5185, the best single neuron when chosen on half of School of Reward
    Hacks' 973 tasks, each with a hacking and an honest answer, and tested on
    the other half: AUROC 0.99 by its mean over the answer. It drops on hacks.
  - A linear probe on all 9,216 layer-17 MLP neurons: held-out AUROC
    0.98.

WHAT THE NUMBERS MEAN
  AUROC is how often a reading puts the hack above the honest answer, or below
  it, whichever it does more: 0.5 is a coin flip. On the same data L17 N9209
  scores 0.59 by its mean over the answer, and it is slightly lower on hacks,
  not higher (rank 67,112 of 294,912 neurons). The repeated-token share is a
  baseline with no model in it: alone it scores 0.73.

PRICE
  $0.002 a read. An account that has never paid gets ten reads free; after
  that the read is refused until $2 of credit is added, and the `credits` tool
  gives the link. The first read after a quiet spell waits while the model
  loads.

READ THEM AS SCORES
  A probe and two neurons of a small model: show the readings as they are, and
  do not take one as a verdict on the answer.

Studies:
  read <task>
      per-token readings of both neurons, the probe's probability, the repeated-token share, the price and how many free reads are left

JSON to stdout, so it composes: pipe to `jq` to pick a field, or `> /tmp/…` to keep it.
Run a study with no arguments — `reward-hack read` — to see what it takes.

What it answers

reward-hack read — per-token readings of both neurons, the probe's probability, the repeated-token share, the price and how many free reads are left