Name: The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort - Aleksandr Churilov, Independent Researcher
Start: 2026-10-06T11:02:00+0200
End: 2026-10-06T11:20:00+0200

The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort - Aleksandr Churilov, Independent Researcher

Tuesday October 6, 2026 11:02 - 11:20 CEST

South Hall 3A

Spracklen et al. (USENIX Security '25) showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm at rates ranging from 5.2% on commercial models to 21.7% on open-source models, creating an attack surface for slopsquatting—the registration of malicious packages under hallucinated names. We replicate their methodology on five frontier code-capable LLMs released between October 2025 and March 2026: Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2. Across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists, we measure overall hallucination rates between 4.62% (Claude Haiku 4.5) and 6.10% (GPT-5.4-mini)—an order-of-magnitude compression of the inter-model spread observed by Spracklen, but not a retirement of the threat. Beyond replication, we identify a set of 127 package names (109 on PyPI, 18 on npm) that all five evaluated models invent identically; following coordinated disclosure with PyPI Security and Socket.dev, 53 of these (41 on PyPI, 12 on npm) remain registrable by an attacker after each registry's existing defenses, constituting a model-agnostic supply-chain attack surface that no single-model study can reveal. We further document a Python-over-JavaScript hallucination asymmetry that inverts Spracklen's 2024 finding, identify a Haiku-below-Sonnet inversion within the Anthropic family, and observe a Jaccard-similarity peak between DeepSeek V3.2 and GPT-5.4-mini (J = 0.343) suggestive of shared training-data origins.

Speakers

Aleksandr Churilov

Independent researcher

Aleksandr Churilov is an independent security researcher working on the safety of LLM-based systems, with a focus on the supply-chain and agent-security failure modes that appear as models move into real developer workflows. His recent work replicated and extended the measurement... Read More →

Tuesday October 6, 2026 11:02 - 11:20 CEST
South Hall 3A

SCORED Sessions, AI and SoK

OpenSSF Community Day Europe 2026

Aleksandr Churilov

Get help with the event

OpenSSF Community Day Europe 2026

Aleksandr Churilov

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Get help with the event