Note
An Honest Security Self-Audit of an AI App, P0 to P3
What does an honest security self-audit of an AI application look like, and how should findings be ranked P0 to P3?
Review your own system as an outsider would, write every finding down graded by consequence — P0 for security and data exposure, P1 for crashes, P2 for correctness and scale, P3 for polish — lead with the worst, and keep the document next to the fixes. For AI apps, the P0s are usually the same three: no authentication, prompt injection through user-controlled message history, and an unmetered paid model endpoint.
The grading scale
- P0 — Critical: security loopholes and data exposure. Fix before anyone real touches it.
- P1 — High: crashes and robustness — malformed input, unhandled errors, error bodies rendered as data.
- P2 — Medium: correctness and scale — wrong results at the edges, work recomputed on every request, missing pagination.
- P3 — Low: polish — leftover scaffolding, misleading labels, platform-specific scripts.
The three P0s that show up in AI apps
In the audit Shreyansh wrote against Sarthi — his AI relationship-manager copilot for banking — the worst findings were exactly these:
- No authentication on any endpoint, with sequential record IDs: the whole (synthetic) customer book could be scraped with a for-loop.
- Prompt injection via chat history: the API accepted a caller-supplied 'system' role in the history array and forwarded it to the model — a full policy override from one request. The fix is to whitelist roles (anything not 'assistant' becomes 'user') and cap history length.
- No rate limit in front of a paid LLM API: one script could run up the bill or exhaust the key. The fix is a per-IP limit with a stricter cap on the chat route, plus a hard budget.
What the lower tiers catch
The P1s were the classic prototype crash paths: clients that never check the response status before parsing it, POST routes that throw on malformed JSON, and inputs that are type-cast instead of validated — even though a validation library was already installed. P2 caught an analytics route recomputing every score on every request (some twice), a hard-coded cap that silently skipped records, and a chart built on Math.random().
One P3 is worth singling out for AI work: an 'explainable AI' view whose contributions didn't actually sum to the prediction. Fine as a visual; wrong to call it SHAP in a compliance setting. Honest labelling is part of the audit.
Publish it, and say what you deferred
The audit stayed in the repository alongside four batches of fixes. Just as important, it names what was deliberately not done — per-user authorization and a move off in-memory data — because a prototype on synthetic data doesn't need them, and a production system would. That sentence is what separates a hackathon build from something you could put in front of real customers.