Computer scientists say they have developed a method for pulling back the curtain on the private “chain of thought” that frontier AI systems use to reason through hard problems. The technique, laid out in a new paper, does not offer conclusive proof that any company broke the rules, but the researchers argue it produces suggestive evidence that at least one Chinese model may have been trained by borrowing the reasoning of leading US systems. The findings also exposed a privacy hole that could spill passwords and API keys before it was patched.
How the Reasoning-Extraction Trick Works
Modern chatbots rarely show their work. When a model like Claude, GPT, or Gemini tackles a puzzle, it generates an internal monologue of reasoning steps. Providers encrypt that text and ship it to the user’s machine to offload computation, while keeping the plain version secret so rivals cannot copy it. The new attack exploits a simple quirk: companies also ship smaller, cheaper variants of the same model, and those lighter siblings share the decryption key but receive less safety training.
What the Hidden Traces Revealed About Chinese Models
Feeding the encrypted reasoning into a weaker model version caused it to decrypt and reveal the hidden thinking. Across a battery of ninety questions, the team noticed that an open-weight Chinese model produced answers strikingly close to the captured traces of a major US model.
Kimi K3 and the Suspicious Similarities
The standout case was Moonshot AI’s Kimi K3, whose reasoning for certain prompts mirrored a US frontier model almost line for line. The authors are careful to note they cannot prove causation, and two other open models showed no such pattern. Still, they say the gap between what could be distilled and what was assumed safe is now far wider.
A Privacy Leak Hiding in Plain Sight
Beyond competitive copying, the same flaw surfaced real secrets. Reasoning traces captured from a user’s computer sometimes contained API keys and passwords embedded mid-thought. The vendors were notified last month and have adjusted their interfaces so private data can no longer be pulled this way.
Why Smaller Models Gave Up Their Secrets
The core insight is that a model’s alignment is uneven across its size tiers. A flagship refuses to expose its inner life; a lightweight edition, built for cost savings, is far more talkative once handed the right key. That asymmetry turned a convenience feature into an extraction channel.
ADVERTISEMENT
Distillation Becomes a Geopolitical Flashpoint
Model distillation has always been a standard engineering shortcut, but it has hardened into a sovereignty dispute. US lawmakers have already heard claims that Chinese labs copied American models to build their own, while some open-source advocates warn that banning the practice would only cede ground.
What Companies and Regulators Should Do Next
Closing the leak entirely would mean reworking how APIs deliver reasoning, not a quick patch. Until then, expect more surprising disclosures as the method is tried against other closed systems.
Outlook: The episode shows that “private” model reasoning was never as sealed as advertised. As Washington and Beijing spar over AI supremacy, techniques like this will keep testing the boundary between fair learning and quiet copying, and users should assume anything typed into a prompt could briefly live inside the model’s hidden notes.
Source: Original report. Rewrite for Your News Website.
Really helpful article, thank you – looking forward to more like this!
Really helpful guide, thank you – looking forward to more like this!
Great read, I learned something new. Thanks for sharing!