Legal Reasoning · Coming soon

Can frontier models pick the winning legal strategy from the complaint alone?

Give an AI model only the complaint from a real lawsuit and ask it, as the defense lawyer, what to file. Does it pick the argument that went on to win?

1 · WHAT THE AI GETS2 · HIDDEN FROM THE AIComplaint filedNames redactedAI'S VIEW STOPS HEREMotion filedJudge rules3 · THE AI ANSWERSWhich motion? Which lead argument?SCORED AGAINSTThe argument the judge accepted
The model sees the case only up to the cutoff line, so its pick is scored against what the judge actually ruled
Setup79 federal motions to dismiss from D.C., all decided in 2026. Claude Opus 4.8, Gemini 3.1 Pro and GPT-5.6 each got only the complaint
MeasureScored against the ruling: did the model lead with the ground the judge dismissed on
ChecksAny model that could name the parties in a case was dropped for that case. Three graders, one from each AI company, agreed on 96% of the winning-ground calls
Results · October 2026
71 to 77%of the time the model's lead used the Rule the judge dismissed on
66 to 72%if it had just guessed 12(b)(6) every time, on the same cases
45 to 50%of the time it led with the exact ground the judge used

FindingNo model is shown to beat always guessing 12(b)(6), and how you ask drives whether it says file