AI Security: Looking at Information Flows

This is the second blog post in a series about AI security that explores the topic in detail, demystifies it and provides practical guidance for analysing the security of AI systems.

In the first post, we provided a brief history of AI security and traced the changes in technology and practices that have led us to where we are today. In this post, we introduce general principles that greatly simplify the problem of analysing the security of a system with AI components. We focus specifically on the security considerations that arise from the presence of AI components within a system and do not consider conventional cybersecurity practices with which the reader is likely already familiar.

In particular, both model training and inference are mediated by code and configuration as well as the hardware, firmware and software of the training and inference hosts; and, if a training or inference host is remote, there might be communication channels that attackers could interfere with but, again, we consider that to be a matter of conventional cybersecurity rather than AI security specifically and hence we do not discuss it further.

Information flow-based AI security analysis#

The most important of these principles are as follows:

• attackers must gain access through information inflows and hence tracing all possible flows of information to an AI component reveals all the ways in which it can be attacked – all of the entry points and all of the means of attack;
• attack effects must be manifest through information outflows and hence tracing all possible flows of information from an AI component reveals all of the effects that can be produced by a successful attack;
• the type of the component determines what attacks it is vulnerable to and how likely attacks are to succeed;

Types of information inflow#

In the case of LLMs (large language models) and LRMs (large reasoning models), which lie at the heart of most modern AI systems – including the rapidly expanding ecosystem of intelligent agents – information flows into them in essentially two ways:

• through the model context – the instructions and data that are consumed by the model during inference; and
• through the model parameters – the data that represent the weights of the trained model.

The former is fast and highly dynamic in the sense that it can change with every inference and mediates attacks such as prompt injection, and the latter is slow and relatively static in the sense that it changes only when the model itself is updated and mediates attacks such as model poisoning. When model parameters are sourced from a third party or they result from training or finetuning on data sourced from a third party, they provide opportunities for model supply chain attacks.

When tracing information flows, particular attention should be paid to flows that:

• cross privilege boundaries; and
• form feedback loops.

Privilege boundaries#

Information flows across privilege boundaries are particularly significant in AI security because successful prompt injection attacks can allow low privilege attackers to force an LLM to act on their behalf in a much higher privilege environment than they could ever hope to access directly – effecting virtual privilege escalation or privilege escalation by proxy. Privilege boundaries often occur in or around tools, but it is important to remember that LLMs can also transport information from their inputs to their outputs and hence boundaries may implicitly be crossed within the LLMs themselves.

Feedback loops#

Information flows from model outflows back to model inflows form feedback loops that typically come in two types:

• internal information loops, where the flow is wholly contained within the system, such as a conversation history or memory mechanism; and
• external information loops, where the flow is at least partially outside of the system, perhaps mediated by third parties, and might not obviously form a loop at all.

Internal information loops can be exploited by attackers to create high reliability persistence, as was the case with spAIware [Rehburger:24] whereas external information loops are more likely to be used to automatically optimise attacks, as was first noted in the early days of email spam filtering [Lowd:05].

Simple principles#

We now have a set of simple principles that we can use to guide us through the process of analysing an AI system:

• identify each AI component and, for each one:
  • determine its type,
  • determine known vulnerabilities of the type,
  • trace all information inflows and outflows with respect to it:
    • inflows consist primarily of
      • everything that can affect the model context; and
      • everything that can affect the model parameters
  • note information flows that cross privilege boundaries
  • note information flows that form internal or external loops

Conclusion#

In this post, we have attempted to simplify the complex and rapidly evolving field of AI security by teasing out some fundamental principles and properties of AI systems and describing how they can be used to analyse the systems’ security. In the next post in this series, we’ll examine information inflows in more detail and look at ways in which they can be defended.

References#

[Lowd:05] Good Word Attacks on Statistical Spam Filters, Lowd et al., 2005. https://www.ceas.cc/papers-2005/125.pdf

[Rehburger:24] Spyware Injection Into Your ChatGPT’s Long-Term Memory (spAIware), Rehburger, 2024. https://embracethered.com/blog/posts/2024/chatgpt-macos-app-persistent-data-exfiltration/