Learn how AI-powered applications break when they trust what the model says — hands-on, against a live target. Steer an assistant into producing output the app renders and acts on, turning a helpful reply into cross-site scripting and server-side command execution in the NeuraDesk lab.
Large language models are increasingly wired into real applications — drafting replies, summarising tickets, suggesting fixes, calling tools. The temptation is to treat what the model returns as safe, finished content and hand it straight to the next component: a web page, a shell, a query, an API call. Insecure output handling is exactly that mistake — trusting model output instead of validating it.
The catch is that a model will relay whatever it is steered to produce. If its output is rendered as HTML, an attacker can smuggle active markup through it. If its output drives an action, an attacker can smuggle a command through it. The AI layer is new, but the sinks it feeds — the browser, the shell, the interpreter — are the same ones application security has always had to defend. The flaw is never the model being clever; it is the app forgetting that its output is still untrusted.
When an app renders a model's reply as raw HTML — so it can show formatting like bold text or links — any markup in that reply is interpreted by the browser. Steer the assistant into emitting active content and it runs in the operator's session, turning a support console into a foothold.
When a feature acts on the model's suggestion — running a diagnostic, evaluating generated code, calling a tool — the suggested action can reach a shell without validation. Influence what the model proposes and you influence what the server runs, escalating a formatting bug all the way to code execution.
This is LLM02 – Insecure Output Handling in the OWASP Top 10 for Large Language Model Applications. Its downstream effects land squarely on the classic OWASP Top 10 too: rendered output becomes A03 Injection and cross-site scripting, while executed output becomes command injection. The impact classes below are what make these high-severity findings.
| Impact | What the attacker gains | Severity |
|---|---|---|
| Cross-site scripting | Run script in an operator's session via rendered model output | High |
| Remote command execution | Execute commands on the server via a trusted suggested action | High |
| Sensitive data exposure | Reach internal data the application or model can read | Medium |
| Session or host takeover | Full control of the operator session or the underlying host | High |
The core lesson of Room 12 is that automation and human judgment are a pipeline, not a competition. An autonomous scan maps the attack surface in minutes — the exposed paths, the endpoints, the open ports. But insecure output handling is a logic bug: it only reveals itself when you understand the flow from prompt to model to sink, and craft input that survives the round trip. That's the gap a human closes.
In the lab, you run a real PhantomRed scan against NeuraDesk, read the surface it maps, then take over manually to exploit the trust the app places in its assistant's output — the flaw the scanner can't see:
It builds on the injection thinking from the XSS lab and the logic-flaw workflow from the broken authentication lab — only here the untrusted input arrives through an AI model, which is exactly what makes it easy to miss.
The NeuraDesk lab is live and free to play. Run a real autonomous scan, then exploit the trust an AI app places in its own model — script in the browser, then a command on the server, end to end.