Blog Archive

In Lab 5, we turned the lights on. We instrumented our agent with OpenTelemetry and visualised the execution traces in .NET Aspire. We can see what happened.

But in a production system, "seeing what happened" isn't enough. You need to know if the system is healthy. In traditional software engineering, we rely on Google's SRE Golden Signals: Latency, Traffic, Errors, and Saturation.

Do these apply to Stochastic Parrots? Yes, but they require translation. In this lab, we will define the operational dimensions of an LLM Agent and implement custom metrics to track them.

Read More

You've built an AI agent. It's deployed. It's answering questions and processing requests. But how do you know if it's working well? Traditional application monitoring gives you some signals, but AI systems introduce unique challenges that require us to rethink what we measure.

In this post, we'll define the operational metrics that truly matter for LLMs and agentic workflows, grounded in the industry-standard SRE Golden Signals framework.

Read More

In previous labs, we built the "brains" of the machine. We explored embeddings, set up a vector database, and even constructed a functional RAG pipeline. But there is a lurking problem with LLM development: The Black Box.

You send a prompt, you wait (sometimes an agonising amount of time) and then you get a response. But what is happening inside the LLM?

• How many tokens did that specific step use?
• Why did the latency spike?
• Did the model actually see the system prompt correctly?

In this lab, we'll explore the foundation of AI Observability by instrumenting your first LLM with OpenTelemetry (OTel). This serves as the bedrock for observing AI systems, turning that "invisible" processing time into structured, analysable data.

Read More

As we previously went through, a common pattern when implementing models into systems is to use RAG (retrieval augmented generation) by using domain specific data with the GenAI models. But what happens if the data source is compromised or poisoned? In this post we'll explore RAG poisoning attacks, their real-world implications and mitigation strategies to secure your AI implementations.

Read More

In this lab, I'm going to move away from security for a little while and look at how we can augment our LLM using a technique called Retrieval Augmented Generation (RAG).

The idea is that we can use a pre-trained LLM to answer questions about a specific dataset. We can then augment the pre-trained model with additional information from a knowledge base. To do this we usually convert the data into a vector space and storing this information in a vector database, like ChromaDB. In this we'll cover:

• Store text embeddings in ChromaDB (an open source vector database)
• Retrieve relevant knowledge dynamically
• Use a local model to generate answers

Read More

The main interface to the models is a chat interface. When you present a questions or insutrction, how do the models "know" what you mean? How does it understand the relationship between words?

The answer is embeddings!

An embedding is a high-dimensional vector representation of words, phrases or concepts. Instead of memorising words models map them into a numerical space where similar meanings are close together.

Read More

Previosuly I've looked at the use of white hat attacks. These test the models as you're training them and should be considered as part of the development process. A prompt injection attack is a type of attack that is used to manipulate the output of a model and is usually an attack of a deployed model.

Large language models generate responses based on user inputs and hidden system instructions. A prompt injection attack exploits this by tricking the AI into ignoring its original constraints, leading to unsafe, unintended, or malicious outputs.

Read More

In my previous post we explored the use of FGSM a powerful yet simple attack method for generating adversarial examples for LLMs. What happens if we need something more subtle and sophisticated?

Enter the Carlini and Wagner (C&W) attack — a method that iteratively optimises the perturbation to the input to maximise the loss function which generates adversarial examples with minimal distortion.

Read More

When working with Large Language Models (LLMs) like GPT, the core mathematical structures you're dealing with are scalars, vectors, and tensors. Just like Dorothy braving the forest, I'm going to follow the yellow brick road and break down these concepts so that I have a better understanding of what they mean and maybe... just maybe I'll be in Kansas again.

Read More

In my first lab I'm going to explore the use of FGSM (fast gradient signed method) to generate adversarial examples for a LLM.

What is FGSM?

FGSM is one of the most famours adversarial attack methods. It is designed to trick a neural network by adding small, carefully-crafted noise (a perturbation) to the input with the goal of having the model misclassify the input.

Read More

Hello! My name is Steve, and I'm a DevOps engineer with a passion for security. I've been in the industry for around 15 years now, and I've seen a lot of changes. I've seen the rise of cloud computing, the adoption of DevOps practices, and the increasing importance of security. I've always dabbled in security, but I never really thought of it as a career path.

If I'm honest I have had some fear and anxiety around the LLMs and AI. I've seen the headlines, I've seen the hype, and I've seen the potential. I've also seen the risks, and I'm not sure how to navigate them. One thing is clear is that they're here to stay, and I need to find a way to move from fear to excitement.

Read More