What Are AI Agents? A Practical Guide to LLMs, Tokens, and Memory
The term “AI agent” is often used interchangeably with large language models (LLMs), but that’s not accurate. It’s time to put proper naming conventions in place and clarify what each component actually does. This is why DAC.digital CTO, Krzysztof Radecki, recorded a video that introduces AI agents using a simple hands-on chat tool interacting with data. This practical approach will give you a clear understanding of how modern AI systems actually work. Let’s watch it.
For those of you who prefer reading texts to watching videos, here is a transcript-based description of the recorded tutorial. Read it if you want to learn how Large Language Models (LLMs) operate, why tokens are critical for both memory and cost, and how we transform a simple text predictor into a fully functional AI agent.
1. What is an LLM?
To use the example from the video, typing “The quick brown fox…” does not make the model “think” about animals. It simply calculates the statistically most probable continuation (“jumps over the fence”). This does not mean the model understands animals or actions. It simply predicts the most statistically probable continuation based on patterns learned during training. LLMs are prediction systems and that affects how we will use them in AI agents.
2. LLMs operate on tokens
Tokens are the basic currency of how LLMs are measured and they also form the backbone of how LLMs operate. You can think of them as fragments of text that a model can process and store in memory. While tokens often resemble words, they are not exactly the same thing. Simple words often become single tokens. For example, the word “fox” is one token. However, more complex or uncommon words may be split into smaller parts. A rough rule of thumb is that one token equals about three-quarters of a word in English text.
How tokenization works
To illustrate this, consider a tokenization tool provided by OpenAI. If we input the sentence: “The quick brown fox” Each word is treated as a separate token. This means the model has seen these words frequently enough during training to store them as whole units. Now consider a more specialized phrase: “Hyperbaric chamber” The word hyperbaric is less common, so the model may split it into smaller tokens: “Hyper” “bar” “ic” This process is called tokenization.
Understanding how tokenization works is important, mainly for two reasons: memory capacity and cost. A model’s memory is determined by how many tokens it can hold in its context window. For example, a model that supports 200,000 tokens can consider that much information at once. Tokens are also the unit used for billing. Users are charged for both the tokens they send to the model and the tokens it generates in response. Understanding how tokenization works helps you design interactions that are both more efficient and more cost-effective.
3. Raw LLMs simulate memory rather than have it
There’s another important limitation. Models only know what they were trained on. They don’t know current news, events or internal company data.
Let’s look at a simple example. I built a basic agent that acts as a wrapper around an LLM using OpenAI’s API. It works like a chat interface. When I say: “Hi” The model replies politely. When I say: “My name is Krzysztof,” it responds appropriately. But when I ask: “What’s my name?” The model says it doesn’t know. Why? Because each prompt is processed independently. There is no memory and no conversation history attached. Every interaction is isolated. This is how raw LLM systems behave.
How memory is simulated in LLMs?
To make models appear conversational, we must send previous messages along with each new prompt. All prior conversation is placed into the context window, which acts as working memory. When you hear that a model supports a large number of tokens, it means that many tokens can fit into this context window. The model doesn’t “remember.” It simply reprocesses the entire conversation each time.
4. AI agents extend LLMs with tools, memory, data, and execution logic
To make LLMs useful in real-world applications, we must build systems around them. These systems are what we call AI agents. AI agents extend the model with additional capabilities that allow them to operate reliably, interact with their environment and execute tasks.
These capabilities typically include:
- Tools that allow the agent to perform actions (e.g., calling APIs, running code, querying databases);
- Context and short-term memory that provide awareness of the current interaction;
- Persistent memory that enables continuity across sessions;
- Access to external data sources to supplement the model’s training knowledge;
- Execution logic that determines how tasks are planned and completed;
- Guardrails and system prompts that enforce safe and reliable behavior.
5. What are common considerations when using AI agents?
Using AI agents involves several key considerations. Organizations must account for data privacy when relying on cloud-hosted models, understand how conversational memory is managed across sessions, and decide whether interactions should be temporary or persist over time. Additionally, because language models generate responses probabilistically, agents require clear instructions and safeguards to ensure accurate, reliable, and policy-compliant outputs.
Privacy and data considerations
When using cloud-hosted models, your data is sent to external servers. Depending on licensing terms: Free tiers may allow providers to use your data for training Enterprise licenses typically restrict training usage However, the data is still processed in the cloud. If privacy regulations such as GDPR are a concern, you should consider: Data anonymization Running models locally within your infrastructure Understanding where your data goes is critical for secure deployments.
Adding conversational history
One of the most important agent capabilities is conversational context. After enabling session history, the model receives previous messages together with each new prompt. For example, if a user says, “Hi, my name is Krzysztof,” and later asks, “What’s my name?”, the model can now respond correctly. This is not because the model remembers information on its own, but because earlier messages are included in the current context window.
Session memory
Session memory is temporary. If the conversation ends and the session is closed, the stored context is lost. When a new session begins, the model starts without knowledge of previous interactions. This behavior is similar to having a conversation, leaving the room, and returning later with no recollection of what was discussed. In privacy-sensitive applications, this design can be beneficial because user data is not retained longer than necessary.
Persistent memory
For applications that require continuity, conversation history can be stored externally and reloaded when a user returns. By saving prior interactions in a database and reinserting them into the context window, the agent can appear to retain long-term memory. In practice, the model is restoring earlier exchanges so they can be processed again. This mechanism is comparable to reviewing meeting notes before speaking with a client.
Preventing hallucinations
Because LLMs generate responses based on statistical prediction rather than reasoning, they can sometimes produce incorrect or fabricated information, commonly known as hallucinations. To reduce this risk, agents rely on system prompts and guardrails that define strict behavioral rules. These instructions may include directives such as:
- Do not fabricate answers;
- Use only the data provided in the prompt or connected systems;
Designing prompts is therefore less like traditional programming and more like giving precise, unambiguous instructions. Clear boundaries and explicit guidance are essential for reliable outputs.
What do you want your agentic AI to do?
6. What an AI agent really is again?
An AI agent is not just a language model. It is a structured system that combines: an LLM for reasoning and generation, memory systems for context, tools for interacting with external systems, guardrails for safety and reliability. AI agents transform prediction engines into practical, task-oriented systems. They are carefully engineered systems designed to make language models useful, reliable, and safe for real-world applications.
Contact us!
Send us an email: [email protected]