LLM and AI Application Security: A Practical Guide

Why LLM applications need their own security thinking

In conventional software, code and data are separate. A web form field is data; it is never executed as an instruction unless you make a mistake. In an LLM application, instructions and data arrive in the same channel: plain text. A customer message, a PDF the assistant summarises, an email it triages and a web page it browses can all contain text that the model may treat as an instruction.

That is why prompt injection has no complete fix at the model level, and why the reliable defences sit around the model rather than inside it. The useful design assumption is simple: whatever the model can read, an attacker may be able to influence, and whatever the model can do, an attacker may be able to trigger.

Prompt injection, direct and indirect

Direct prompt injection is a user typing instructions designed to override yours: ignore your rules, reveal your instructions, pretend to be a different assistant. Jailbreaks are a form of it. They matter most when the model has access to something worth stealing or an action worth abusing.

Indirect prompt injection is more dangerous for business systems. The malicious instruction is planted in content the model reads on someone else's behalf: a support ticket, a CV, a shared document, a web page, an email. When a staff member asks the assistant to summarise it, the hidden instruction runs with that staff member's access.

Input filters and stronger system prompts reduce casual attacks but will not stop a determined one. What works is limiting the blast radius: give the model the least data and the fewest tools that the task needs, keep a human in the loop for actions that move money, send messages or change records, and treat every model output as untrusted input to the next system.

Securing RAG: retrieval is an access control problem

Retrieval-augmented generation (RAG) lets a model answer from your own documents by searching a knowledge base and placing the most relevant passages in its context. The common failure is that the search index is built from everything, and the model is then asked to only show each user what they are allowed to see. A model cannot enforce permissions reliably.

Enforce access before retrieval, not after. Carry each document's permissions into the index, filter results by the requesting user's identity, and only then pass passages to the model. Control who can add or edit content in the knowledge base, because anyone who can write to it can plant instructions or false information that the assistant will repeat to everyone else.

AI agents and excessive agency

An agent is a model that can call tools: query a database, send an email, call an API, run code, browse the web. Every tool turns a successful prompt injection from a wrong answer into a real action. Most serious AI incidents involve an agent doing something it was able to do but should never have been allowed to do.

The controls are the same least-privilege principles used everywhere else in security, applied deliberately:

• Give each agent only the tools its task requires, and prefer narrow tools (look up an order) over general ones (run any SQL).
• Use scoped, short-lived credentials tied to the end user, not a shared administrator key.
• Require human approval for consequential actions: payments, external messages, deletions, permission changes.
• Block server-side request forgery: an agent that can fetch URLs must not be able to reach internal addresses or cloud metadata endpoints.
• Log every tool call with its inputs, so you can reconstruct what happened.

Data protection and AI in Kenya

Kenya's Data Protection Act, 2019 applies to personal data whether a human or a model processes it. Three obligations come up repeatedly in AI projects. First, sending personal data to a model hosted outside Kenya is a cross-border transfer, which needs a valid basis under the Act. Second, the Act gives data subjects rights in relation to decisions based solely on automated processing that significantly affect them, which matters for AI used in lending, hiring or claims. Third, new technology applied to personal data is a classic trigger for a data protection impact assessment.

In practice: know which model provider processes your data and where, read their data retention and training terms, minimise the personal data that reaches the model, and run a DPIA before launch rather than after.

AI governance: the minimum an organisation needs

Governance is what stops AI security from depending on one careful engineer. It does not need to be heavy. Recognised frameworks such as the OWASP LLM AI Cybersecurity and Governance Checklist, the NIST AI Risk Management Framework and ISO/IEC 42001 for AI management systems all cover the same core:

• An inventory of every AI system and feature in use, including staff use of public AI tools.
• An acceptable use policy that says what data may and may not be put into AI tools.
• A named owner for each AI system, accountable for its risks.
• Risk assessment and DPIA before launch, and again on significant change.
• Security testing of AI features, including prompt injection and agent abuse, before go-live.
• Monitoring, logging and an incident response plan that covers AI misuse.
• Vendor due diligence on model providers: data handling, retention, location and training use.

Testing an AI application before launch

Conventional penetration tests do not cover LLM behaviour unless they are scoped to. An AI security assessment should attempt direct and indirect prompt injection, system prompt extraction, data leakage across users, retrieval permission bypass, tool and agent abuse, server-side request forgery through agent tools, and cost exhaustion. It should also review the architecture: what the model can reach, with what credentials, and where a human approves actions.

Use the free self-assessment checklist below to find gaps quickly, then test the high-risk paths properly before the feature reaches customers.

Key takeaways

• An LLM cannot reliably separate your instructions from instructions hidden in content it reads, so prompt injection has no complete model-level fix.
• The defences that work sit around the model: least privilege, scoped credentials, human approval for consequential actions, and treating output as untrusted.
• In RAG systems, enforce permissions before retrieval; never rely on the model to hide documents.
• Personal data sent to an AI model is still governed by the Kenya Data Protection Act, including cross-border transfer and DPIA obligations.
• Governance can be light but must exist: an inventory, an acceptable use policy, owners, testing before launch and monitoring after.

Frequently asked questions