A public experiment in answer-key-free AI evaluation

When there is no answer key, can strategic AI advice still be tested?

Parallax asks a narrower, observable question: when controlled evidence changes, does a model’s stated assessment move with it—and does that assessment stay put when only decision-irrelevant wording changes?

Open the Situation Room Read the 90-second brief Inspect the evidence

Current local evidence 1,600 scheduled runs

No usable response delivered 777 / 1,600

Experiment 3 Claude / Gemini result None

The interactive evaluation

Enter the Situation Room.

Follow the experiment from question to evidence to observable model response. Explore controlled evidence doses, matched wording variants, missing responses, historical transfer, and the exact limits on what Parallax has earned.

Open the research release How the test works

Why this matters in government

Before relying on AI advice, test what moves it.

Strategic questions often have no single answer key. That makes ordinary accuracy scoring a poor test of whether an assessment responds to the evidence or to the way the evidence is presented.

Parallax gives a decision-maker a narrower, inspectable signal: does the observable assessment change when controlled evidence changes, and does it remain comparatively stable when only decision-irrelevant wording changes?

The 90-second brief

The failure is part of the result.

A fluent answer is not enough for high-stakes policy advice. Parallax makes evidence sensitivity, wording sensitivity, refusal, and missingness separately visible—and keeps the negative findings.

The question

Can strategic assessments be evaluated when no objective policy answer key exists?

Why policymakers should care

Advice that reacts more to phrasing than evidence can look useful while remaining presentation-fragile.

What failed first

Experiment 1 failed its frozen rule: relevant evidence moved responses, but a nuisance paraphrase also moved one.

What changed

The project rebuilt around five ordered evidence doses, explicit refusal, common-denominator missingness, and fail-closed analysis.

What real models did

The governing SR-8R1 campaign produced bounded Vela and Afghanistan profiles: 823 rows reached a non-transport terminal state, comprising 822 substantive assessments and one measurement-unavailable row; 777 rows were transport-unavailable. Earlier 4,800-row Qwen evidence remains historical and non-governing.

What remains unproven

No Experiment 3 Claude/Gemini science exists. The designed 556,200-call confirmatory study remains unfunded and unexecuted.

Research lineage

Test. Fail. Rebuild. Attack again.

Parallax did not appear fully formed. Each experiment preserved the previous failure and narrowed what the next design had to solve.

Experiment 1 · live

The first claim failed.

Not supported

Three synthetic strategic scenarios, two provider configurations, and 210 confirmatory calls. Relevant evidence moved three complete units; a nuisance paraphrase moved one. Refusal and observability were uneven. The frozen rule returned NOT SUPPORTED, and the result was not rescued post hoc.

See Experiment 1 evidence →

Experiment 2 · live rebuild

More data did not fix identification.

Insufficient evidence

A larger two-provider campaign resolved all 888 planned coordinates, but only 66 of 288 matched triples were fully valid. Both frozen provider decisions were INSUFFICIENT_EVIDENCE. Offline qualification established custody and replay—not a scientific win.

See Experiment 2 evidence →

Experiment 3 · mature successor

The instrument became conservative by design.

Provider science unexecuted

Five evidence doses, refusal outside the ordinal scale, joint partial identification, 6.7 million synthetic checks, and a governing 1,600-row SR-8R1 local campaign. The earlier 4,800-row Qwen campaign remains predecessor evidence. Production paths were qualified separately; the full scientific provider study has not run.

Open the governing Situation Room →

See Experiment 3 evidence →

What has actually run

Real evidence, separated by role.

Synthetic validation tests software. Local models stress the instrument. Provider qualification tests transport. None is allowed to impersonate confirmatory evidence.

6,674,400

Known-ground-truth synthetic coordinate evaluations across 12 worlds; all matched independent oracle dispositions.

1,600

Scheduled rows in the governing SR-8R1 two-family, full-bundle local campaign.

777

Transport-unavailable governing rows, retained on the denominator and described as infrastructure unavailability—not model behavior.

Live Anthropic/Google path-qualification calls using no scientific Parallax prompt and producing no scientific provider result.

Earlier surrogate stress evidence

The earlier Qwen models moved—but not cleanly.

These 4,800-run plots predate and do not govern the current Situation Room result. They remain accepted descriptive stress evidence, are never pooled with SR-8R1, and are never Claude/Gemini evidence.

Figure A — Local model evidence-response profile. Across three scenarios and four local Qwen2.5 strata, response distributions change with evidence dose, but not in a uniformly monotone or clean pattern. Every bar uses the all-scheduled denominator, so unusable output remains visible. This is descriptive local evidence only; it does not establish provider behavior, belief revision, mechanism, or a confirmatory result.

Figure data · MATLAB source · Hash receipt

Figure B — Nuisance instability matrix. Each cell reports the maximum absolute A–B shift across four all-scheduled outcome shares. Seventeen of 60 blinded pair-by-stratum comparisons were nonzero. Printed values make the result non-color-dependent. This is not a provider nuisance-equivalence test and does not identify a causal capacity or decoding effect.

Figure data · MATLAB source · Hash receipt

Read the local evidence summary · Open the raw aggregate JSON

Why run the full study?

The frontier question is still worth asking because every outcome is informative.

Claude and Gemini were studied in Experiments 1 and 2, but not with the mature Experiment 3 instrument. The local campaign has already shown why this matters: evidence-response profiles, wording sensitivity, refusal, schema failure, transport unavailability, and conservative accounting all remain visible.

A future provider run could reveal evidence-responsive and wording-stable behavior, evidence-responsive but presentation-fragile behavior, stable but insensitive behavior, high refusal or unavailability, insufficient identification, or provider differences. Parallax does not assume a favorable result.

Evidence boundary

What this work establishes—and what it does not.

The strongest current claim is methodological: the instrument can expose failure while preserving custody, missingness, and negative evidence. It is not a score of hidden cognition or policy wisdom.

Supported now

The first two live studies returned negative or insufficient results under frozen rules. The full Experiment 3 software geometry matched independent synthetic oracles. The 4,800-row local campaign produced real descriptive pathology. Current Anthropic and Google transport paths passed six non-scientific qualification probes.

Not established

Parallax does not measure hidden belief, truth, internal reasoning, general strategic competence, or correct policy. Claude and Gemini have no Experiment 3 scientific result. Qwen is not confirmatory evidence. Larger model capacity is not a demonstrated causal factor. Surface-cue concordance does not identify mechanism.

Still designed, unfunded, and unexecuted

The reference confirmatory plan separates two providers across 30 cells each, with N=9270 per cell, M=210 simultaneous scalar supports, 278,100 calls per provider, and 556,200 calls total. It makes no full-joint power claim. Funding and account capacity are prerequisites for execution, not evidence that execution occurred.

Read what the evidence supports—and does not support · Inspect the technical claim/evidence matrix

The public research release

Explore the evidence now; inspect the exact method when you want it.

The Situation Room is the guided reader experience: cases, evidence doses, observable response profiles, unavailability, nuisance sensitivity, validation, and limitations.

The repository is the audit layer: exact data, contracts, verifiers, hashes, historical records, and reproduction instructions.

Open the Situation Room Read the full research narrative

Reproduce the public release

No model, GPU, credential, or network call required.

The verifier checks the public evidence boundary, accepted constants, unchanged scientific files, links, privacy, and claim limits from the published tree.

python reproducibility/verify_policy_reader_presentation.py

Read the complete reproduction instructions

What exists today

A public evaluation release, with the scientific boundary intact.

The instrument, local evidence, failures, and reproduction path are public. The frontier-provider study remains designed but unexecuted.

Scientific record

Experiment 1 was not supported. Experiment 2 returned insufficient evidence. Experiment 3 has governing local evidence but no frontier-provider scientific result.

Public evidence

The Situation Room is live. Its frozen predecessor remains byte-preserved, and the current presentation adds no scientific observation or classification.

Frontier boundary

Claude and Gemini have no Experiment 3 scientific result. Their paths were qualified separately; the 556,200-call reference study remains unexecuted.