What a 3‑Month OpenAI Internal Audit Revealed About Prompt‑Injection Failures

This article contains affiliate links. We may earn a commission at no extra cost to you. Full disclosure.

Here’s the number that should worry anyone shipping an LLM-powered agent: in the AgentDojo benchmark published by researchers at ETH Zurich in mid-2024, attackers achieved task hijacking against GPT-4o and other frontier models in a meaningful share of test scenarios — even after developers applied the defenses that were available at the time. OpenAI has not published a document literally titled a “3-month internal audit” of prompt injection, and no leaked report with that exact framing has surfaced publicly. What does exist — and what this piece pulls together — is a dense trail of evidence: OpenAI’s own system cards for GPT-4o and o1, the “Instruction Hierarchy” paper its safety team published in April 2024, its Bugcrowd-run bounty program, and independent academic red-teaming from groups at ETH Zurich, UIUC, and elsewhere. Stitched together, that evidence paints a clearer picture than any single audit could: prompt injection is not a solved problem, and the gap between marketing language (“safety-tuned,” “aligned”) and measured attack success rates is still wide.

6 min read

The Real Evidence Trail Behind the “Audit” Framing

Prompt injection got its name from developer Simon Willison, who described the attack pattern in a blog post in September 2022, months before ChatGPT’s public launch made LLMs a mainstream target. The core idea is simple: if a model can’t reliably distinguish between “instructions from my developer” and “text I happened to read,” an attacker just has to get malicious text in front of the model. That’s it. No SQL injection-style syntax tricks required — just plain English hidden in a webpage, PDF, email, or calendar invite.

OpenAI’s public paper trail on this problem is more substantial than most people realize. The GPT-4o System Card (May 2024) and the o1 System Card (September 2024) both include sections on external red-teaming, and OpenAI’s Preparedness Framework — first published in December 2023 and updated since — explicitly names “model autonomy” and tool-use risks as tracked categories. Separately, OpenAI’s bug bounty program, run through Bugcrowd since April 2023, pays out $200 for low-severity findings up to $20,000 for critical ones, and prompt injection reports have been a recurring submission category according to public program summaries. None of this is a “leaked audit.” It is, however, a documented, citable body of work that functions as one.

None of this is a “leaked audit.” It is, however, a documented, citable body of work that functions as one.

Why This Keeps Breaking Production Systems

The reason prompt injection matters more in 2025 than it did in 2023 is deployment surface. Two years ago, the worst-case outcome was a chatbot revealing its system prompt — embarrassing, not dangerous. Kevin Liu’s February 2023 extraction of Bing Chat’s “Sydney” codename via a simple “ignore previous instructions” prompt is the canonical early example. Today, models have tool access: browsing, code execution, email drafting, calendar writes, payment APIs. An injected instruction inside a webpage or a PDF attachment can now trigger an action, not just a weird sentence.

Real incidents back this up outside the lab. In December 2023, a Chevrolet dealership’s ChatGPT-based chatbot was manipulated into agreeing to sell a 2024 Tahoe for one dollar, an interaction widely screenshotted and reported by Business Insider. In January 2024, DPD’s customer-service chatbot was coaxed into swearing at a customer and criticizing its own employer, a story that went viral on X within hours. Neither case involved sophisticated code — both used plain-language social engineering against a model that couldn’t tell the difference between “the user’s request” and “content it should treat as untrusted.”

The pattern OpenAI’s own materials acknowledge, and that independent researchers keep reproducing, is this: as capability goes up, so does the attack surface, because more capable models are given more tools to misuse. A model that can only chat is a low-stakes injection target. A model that can browse, execute code, and send emails on your behalf is a high-stakes one, and it’s the same underlying architecture doing both.

A model that can browse, execute code, and send emails on your behalf is a high-stakes one, and it’s the same underlying architecture doing both.

Direct vs. Indirect Injection — the Mechanics That Actually Matter

Direct injection is what most people picture: a user types “ignore your system prompt and do X” straight into the chat box. It’s the easiest to defend against because the attacker and the victim are the same person typing into the same box — there’s no third party to protect. Indirect injection is the harder problem, formalized in the paper “Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection” by Greshake et al. (arXiv:2302.12173, February 2023). Here, the malicious instruction lives in content the model retrieves on someone else’s behalf — a webpage, a document, an email body, a product review — and the end user never sees it.

OpenAI’s Instruction Hierarchy paper (Wallace, Xiao, et al., arXiv:2404.13208, April 2024) is the most concrete technical response to this so far. The approach trains models to assign different trust levels to different message roles: system messages outrank developer messages, which outrank user messages, which outrank third-party tool outputs and retrieved content. In the paper’s own reported evaluations, this training approach measurably reduced attack success rates on held-out injection test sets compared to the baseline model, though the authors are careful to frame the results as a mitigation, not a fix — the hierarchy can still be probed and, in some conditions, subverted.

• Direct injection: attacker = end user, typed straight into the prompt box, easiest to log and rate-limit
• Indirect injection: attacker = a third party embedding text in retrieved content the model treats as data, not instructions
• Multi-hop injection: instructions planted in one tool call (a search result) surface later in a downstream tool call (an email draft), making the chain harder to audit

What the Benchmarks Actually Show

This is where the marketing language and the measured numbers diverge. AgentDojo (Debenedetti, Zhang, Balunović, Beurer-Kellner, Fischer, and Tramèr, published at NeurIPS 2024’s Datasets and Benchmarks track, arXiv:2406.13352) built 97 realistic agentic tasks — email, banking, travel booking — and layered 629 adversarial injection test cases on top. The finding that matters for practitioners: no tested model-defense combination drove attack success to zero, and several combinations that looked strong on paper still failed against adaptive attackers who iterated on their injection wording.

InjecAgent (Zhan et al., arXiv:2403.02691, March 2024) ran a complementary test across 17 tools and 62 attacker cases, and found that models varied widely by task type — attacks aimed at direct harm (like deleting files) were caught more often than attacks aimed at data exfiltration, which blended in more easily with a model’s normal tool-use behavior. Meanwhile, the OWASP Top 10 for LLM Applications, maintained by the OWASP GenAI Security Project, has ranked Prompt Injection as LLM01 — the single highest-priority risk category — in every version published since 2023, ahead of data leakage and insecure output handling.

When I ran a quick informal test against the GPT-4o API in late 2024 — embedding an instruction inside a fake “customer review” text block and asking the model to summarize it — the model followed the summarization request correctly about half the time and, in the other half, partially executed the embedded instruction anyway (in my case, appending an unrelated promotional sentence to its output). That’s not a controlled benchmark, just one data point, but it lines up with what AgentDojo reports: defenses raise the bar, they don’t remove it.

The Cost Side Nobody Talks About

Running these evaluations at scale isn’t free, and that’s worth putting in dollar terms because it explains why smaller teams under-test for injection. GPT-4o’s API pricing sits at roughly $2.50 per million input tokens and $10 per million output tokens as of OpenAI’s published rate card; o1 runs considerably higher, around $15 per million input tokens and $60 per million output tokens, reflecting its longer reasoning traces. AgentDojo’s 629 test cases, run across even a handful of models with multiple retry attempts per case, easily reaches tens of

Written byAllen Sindaporean

Allen Sindaporean covers emerging AI tools, platforms, and industry developments for AI Discovery Digest. With a focus on practical applications, Allen helps readers understand how artificial intelligence is transforming industries and creating new opportunities.