No answer key: stress-testing strategic AI advice
What happens when presidents and policymakers ask AI questions with no answer key—whether to reassure an ally, escalate, or wait? Standard benchmarks cannot tell us whether strategic advice responds to evidence or merely shifts with phrasing.
Parallax is a public, reproducible research project that asks a narrower behavioral question: as controlled evidence moves from strongly against to strongly toward a hypothesis, does the model’s stated assessment move too, while staying stable under decision-irrelevant rewrites?
The story is unusually falsifiable. Our first live two-provider experiment (210 calls) failed its frozen rule: relevant evidence moved some responses, but a nuisance paraphrase also moved one, and refusals made observability uneven. We did not rescue the result.
We rebuilt the instrument, ran a larger second live campaign that still returned insufficient evidence under its frozen rule, then attacked the mature design with 6.7 million known-ground-truth synthetic evaluations and 4,800 real local Qwen2.5 generations.
The Qwen models responded to evidence, but not cleanly: dose paths were often non-monotone, nuisance pairs were unstable, and 256 outputs violated the response schema. These are local surrogate findings, not claims about Chinese models or Claude/Gemini.
The finished ChinaTalk piece would combine a reported essay with an interactive evidence explorer, showing why answer-key-free AI evaluation matters for national-security decision support, what repeated failure taught us, and what a genuinely confirmatory Claude/Gemini study—designed but still unfunded—would need to establish.
A commission would support reporting and interactive refinement; full confirmatory execution would require separate API funding and provider capacity.