How to Choose the Right LLM for Your Application

Define the Problem Before Choosing an LLM

Choosing a model should never be the starting point. Before comparing providers, benchmarks or capabilities, the first step is to clearly define what the system is expected to do. Without that, any model selection becomes guesswork.

A useful way to approach this is to break the problem down into a few key dimensions.

If you want a more hands-on take on this topic, see how DAC’s engineers approach this decision in How Do AI Engineers Choose an LLM for Agentic AI?

1. Task Complexity and Model Capabilities

The complexity of the task should directly influence the type of model you choose.

A large and expensive model is often unnecessary for simple use cases, such as text summarisation, classification or entity extraction. And sometimes, LLMs are not even needed at all. Traditional approaches such as rule-based logic, named entity recognition, or regular expressions may provide more predictable results with lower cost and effort.

The need for LLMs with more advanced capabilities becomes more justified, once the LLM is expected go beyond simple text transformation, particularly when involving:

• reasoning
• planning and breaking problems into steps
• generating structured outputs
• interacting with external tools

However, even in these cases, the largest model is not always the best choice.

If a task relies heavily on structured problem-solving, it is more effective to use a system trained specifically for that purpose rather than a larger general-purpose option. The same applies to tools’ usage – solutions with native support for tool calling tend to be more reliable in such scenarios.

2. Tool Usage and System Interaction

Some applications require the model to interact with external systems, e.g.: calling APIs, executing functions (e.g. calculators) or generating and running SQL queries. In these cases, the model does not only need to produce correct outputs, but also decide when to use a tool, which is a different type of capability.

A model that has been trained for tool calling will likely perform better at knowing when a given tool should be used thus reducing the need for complex prompt engineering, and improve overall system reliability.

Without this capability, the model may attempt to generate answers instead of calling an appropriate function, leading to undesired behaviour.

3. Bounded vs Open-Ended Problems

Another important distinction is whether a given engineering problem is bounded or open-ended.

Bounded problems:

• have clearly defined outputs
• are easy to evaluate
• can be measured objectively

Examples include data extraction or classification, such as extracting a telephone number from text or spam-vs-non-spam email classification.

In such cases, it is often best to start with the smallest model that performs adequately, prioritizing determinism and cost efficiency, and using it as a benchmark for future improvements.

Open-ended problems:

• involve subjective outputs
• are difficult to evaluate at scale
• often require creativity or context awareness

Examples include content generation or storytelling. By nature, tasks such as generating a “funny song’s lyrics” are harder to unanimously evaluate or even explain exactly what a success would exactly look like. Consequently, more advanced models are typically chosen, as they provide a higher chance of obtaining satisfactory results, given more complex descriptions and prompts.

4. Latency and Response Time Considerations

Latency is often overlooked but it can be as critical as other criteria, especially when systems’ responsivity is critical to success. 

For real-time applications, such as chatbots, response speed is important as users expect immediate answers for experiencing a natural conversation. In such cases, smaller models are usually preferred as they process requests faster.

For batch processing tasks, such as data cleaning or offline analysis, longer processing times may be acceptable. Whether to pick a simple or a more complex LLM is likely to depend on other selection criteria for as long as the processing time is manageable.

5. Domain and Data Alignment

The closer the model’s training data is to your domain, the better the results.

General-purpose tasks such as paraphrasing or intent recognition usually don’t require more than a generic model. However, when operating in specialised domains such as legal, medical, or software, it is important to know the kind of “material” that was used to train an LLM. The material, being essentially the training data, allows an LLM to be more specific regarding phrasing and using correct vocabulary when facing a task. Of course, equipping an LLM with a curated knowledge base or RAG (retrieval augmentation generation) pipeline is likely to increase the model’s success rate; it comes with an additional latency cost as it becomes a multistep process. Therefore, whenever a model that was trained on specific domain data is available, at least trying it is usually a very good first step.

6. Language and Cultural Context in LLMs

In a similar way to the last point, many models are primarily trained on English data, as English is by now the most popular language on the Internet. However, LLM applications are obviously not limited to English. If your application operates in another language or requires cultural awareness, it can affect performance significantly by twisting words, phrases, misunderstanding idioms or even completely misinterpret or mistranslate your queries

In such cases, models trained on language-specific datasets may not only better handle grammatical and lexical nuances, but also act “smoother” and more appropriately whenever cultural context is imperative to the performanceWhy Public LLM Benchmarks Fall Short

When having to select an LLM, it is natural for an engineer to look for some sort of “condensed figures” that could single-handedly help with picking the right model. Within the LLM realm, there exist figures of merit that are indeed standardized to a certain extent and proposed by various academic boards. However, as the development in AI happens at an unprecedented case, the so-called public benchmarks pose significant challenges themselves.

First of all, most often they are designed to serve academic purposes, where publishing papers is primal to academic success. Consequently, they are designed to facilitate research and often test and compare LLMs under specific isolated tasks and aggregate outputs from different attempts. For example, this can be finding solutions to match problems, specific puzzles or extracting specific pieces of information. While these tests are necessary and useful, they are often less informative regarding the usage of such models in production, where they are naturally exposed to different requests and conditions.

Secondly, it is not uncommon that, again, given the research pace, that models become so good at scoring high on some benchmarks, that the benchmarks themselves become uninformative and simply obsolete. What happens next is that new, more sophisticated benchmarks are introduced to provide ground for comparison, which for a system or AI engineer imposes a dual challenge. Not only an LLM needs to be chosen, but also a specific metric needs to be found and understood, often leading to engineers proposing their own evaluation criteria. 

And this brings us to…

Focus on Your Own Evaluation Criteria

A more effective approach is to define evaluation criteria based on your specific use case, and the best way to start is clearly defining the goal of the model. If the objective is not measurable, it will be difficult to determine whether the model performs well.

One of the most practical metrics is completion rate, i.e. how often the model successfully completes the intended task. It may require creating custom test cases, but provides a much more accurate view of performance in your context.

Key Aspects to Measure

When evaluating models, focus on factors that directly impact your system. Here are a few examples:

• Task completion rate – how often does the model produce the expected result?
• Tool usage accuracy – does the model correctly decide when to call external functions or APIs?
• Structured output reliability – how often does the model return valid formats (e.g. JSON)?
• Error handling and recovery – how does the system behave when the model fails or produces invalid output?
• Latency in real conditions – how long does the model take to respond under realistic load?
• Cost per task – how many tokens are consumed per successful operation?

Setting up this kind of evaluation infrastructure in production, including prompt versioning, tracing, and experiment tracking, is the core of an LLMOps practice.

API vs Self-Hosted LLMs: Choose the Right Hosting Strategy

Choosing a model is only part of the decision. What is equally important is how it will be deployed and accessed.

The choice between using a cloud API and hosting models yourself has direct implications for cost, performance, control, and compliance. Although there is no single correct approach, the right option depends on your scale, constraints, and technical maturity.

Start with a Practical Constraint

Before considering deployment options, one thing should be clear:

Training large language models from scratch is not a realistic option, as they require significant computational resources, long training cycles, and very high financial investment.

Fine-tuning may be possible in some cases, but building a model from the ground up is out of scope for most organisations. The real decision is how to use the existing solutions.

Cloud API: Fast and Accessible

Using a cloud API is the most straightforward option, especially at the beginning. It allows teams to start quickly without infrastructure setup, access advanced models immediately, and focus on application logic rather than operations

Although it seems like a natural choice for prototyping and early-stage development, this convenience comes with trade-offs. Data sent to the model leaves your environment, which raises privacy and compliance concerns. Latency can vary depending on the provider. And most importantly, costs scale directly with usage. It all becomes particularly relevant once the system moves to production and traffic increases.

Self-Hosting: Control and Predictability

Hosting models within your own infrastructure provides a different set of benefits like full control over data, greater predictability of cost at scale, and flexibility in how models are configured and deployed.

This approach is often preferred when handling sensitive data, operating under strict regulatory requirements, or running high-volume workloads. At the same time, it introduces additional responsibilities, including managing infrastructure and scaling, handling updates and potential performance degradation, and ensuring overall operational reliability.

When Does It Make Sense to Switch?

A practical way to approach this decision is to consider usage scale over time. In the early stages, during development or at lower volumes, using an API is typically more cost-effective and easier to manage. However, as usage grows, token-related costs can increase rapidly. At that point, investing in your own infrastructure may start to make more economic sense. There is no fixed threshold for when this shift should happen, which is why it is important to monitor usage from the beginning and understand how it evolves as the system scales.

A Middle Ground

There are also intermediary options that combine aspects of both approaches. Some platforms provide access to models without tying you to a single vendor, while still reducing the need for full infrastructure management. It can lower operational overhead, provide more flexibility than a single-provider API, and reduce dependency risks. However, data is still processed externally, so privacy considerations remain.

Avoid Vendor Lock-In in LLM Systems

One risk that is often overlooked is tight coupling to a specific provider. If your system depends heavily on a single vendor, you may run into challenges over time, whether due to pricing changes, model deprecation, or evolving policies around data usage.

To mitigate this risk, it is advisable to design your system with abstraction layers that separate the application logic from the underlying model provider. Using standardised APIs where possible also helps reduce dependency on a specific implementation. Additionally, ensuring that models can be replaced without major refactoring makes it easier to adapt when conditions change.

This approach improves long-term flexibility and makes the system more resilient to external changes.

Why Tokens Are Only Part of the Cost

Cost is often reduced to a single metric: tokens. While their usage is an important factor, it represents only part of the overall expense. In practice, the total price of running an LLM-based system depends on multiple components, many of which are not immediately visible at the beginning. The key challenge is that costs tend to remain manageable during development, but increase rapidly once the system is exposed to real traffic.

What Actually Drives Cost

Each interaction with a model involves input tokens, which you send, and output tokens, which the model generates. In most pricings, output tokens are more expensive, so the way prompts and responses are structured has a direct impact on cost. Short, well-defined outputs are typically more efficient than long, loosely structured responses.

However, tokens are only part of the picture. In more complex systems, additional factors start to contribute, including infrastructure costs (especially in self-hosted setups), memory and compute usage, network traffic, and orchestration between multiple models. These elements become increasingly relevant as the system grows.

Hidden Costs in LLM Architectures (RAG, Guardrails, Observability)

As systems evolve, they often include additional components that introduce indirect costs. For example, in architectures that use RAG, the process typically involves generating embeddings, querying a database, and re-ranking results. Each of these steps adds both computational overhead and token usage.

Similarly, guardrails and safety layers require additional prompts, and observability tools introduce further processing and logging. These costs are easy to overlook because they are distributed across the system rather than tied to a single model call.

The Context Window Trade-Off

Another important factor is the size of the context window. Sending more context to the model does not necessarily improve output quality, but it will always increase token usage. This often leads to inefficiencies, where large inputs add cost without delivering proportional value. Managing context effectively is therefore an architectural decision, not just a model capability.

Scaling Is Not Linear

As systems grow, cost does not increase in a simple, predictable way. More data leads to more interactions, and more interactions trigger additional processing across multiple components. In many cases, this results in a faster-than-linear increase in cost, particularly when using larger models or multi-step workflows.

Practical Ways to Control Cost

There are several practical techniques that help keep costs under control.

Reducing prompt size while maintaining clarity can significantly lower token usage. Caching allows the system to reuse previous results instead of calling the model repeatedly, which reduces both cost and latency. It is also worth assigning simpler tasks to smaller models, which are typically cheaper and faster, without compromising overall system performance.

These optimisations are most effective when considered early in the design process, rather than introduced later as a response to rising costs.

How to Make LLMs Reliable in Production

Large language models are inherently non-deterministic. The same prompt can produce different outputs, and small variations in input can lead to entirely different responses. This is a fundamental characteristic, not a flaw that can be fully eliminated. The goal is therefore not to achieve perfection, but to design a system that remains reliable despite this behaviour.

Invest in Prompt Design

One of the simplest and most effective ways to improve reliability is to focus on prompt quality. There is a practical cost implication here as well. Input tokens are typically cheaper than output tokens, which means it is more efficient to invest effort in structuring the input than dealing with long, unstructured responses.

Clear prompts should define the expected format of the output, reduce ambiguity, and limit unnecessary verbosity. A vague prompt on the other hand, often results in long and inconsistent outputs, which increases both cost and post-processing effort.

Manage Error Propagation in Multi-Step Systems

As systems grow more complex, they often involve multiple model calls or agent-like workflows. In such setups, errors do not stay isolated. A small mistake in one step can propagate through the entire chain, affecting subsequent outputs and making the final result unreliable.

You need visibility into where errors occur, how often they happen, and which components are responsible. Without this, it becomes difficult to identify weak points in the system.

Combine Probabilistic and Deterministic Components

LLMs are probabilistic, but your system should not rely on probability alone. In most applications, model outputs need to be consumed by deterministic systems like APIs, databases, or business logic. This requires consistency and structure. For this reason, it is important to prefer models that can generate structured outputs (e.g. JSON) and enforce output formats where possible. It reduces the need for complex parsing and lowers the risk of integration errors.

Handle Inconsistency Explicitly

The same input may produce different outputs, and not all of them will be correct. In non-critical scenarios, this may be acceptable. In critical workflows, it is not.

For sensitive operations, such as financial decisions, it is advisable to introduce human validation or additional control mechanisms. Systems should not assume that the model will always produce the correct answer.

Use Retry Logic

Model responses are not always stable on the first attempt, so, in some cases, repeating the same request can produce a correct result. Implementing retry mechanisms is therefore a simple but effective way to improve reliability, especially when working with external APIs.

Be Aware of Context Limitations

As interactions grow longer, earlier information may be pushed out of the model’s context, which can lead to situations where the model “forgets” previously provided data. Expanding the context window is not always the best solution, as it can raise costs without guaranteeing better results.

Practical Techniques to Improve Stability

Instead of relying on larger models, it is often more effective to adjust the system design. For example, routing different types of requests to different models allows you to better match capabilities to tasks. Caching responses reduces repeated processing and helps maintain consistency. Compressing prompts keeps inputs focused and reduces unnecessary token usage. These techniques improve both reliability and cost efficiency, while giving you more control over how the system behaves.

Legal, Privacy and Compliance Constraints in LLM Projects

Technical and financial considerations are only part of the decision. In many cases, legal and compliance requirements will define what is actually possible. These constraints can significantly narrow down the range of models and deployment options available from the very beginning.

Data Privacy and Control

Every interaction with a model involves sending data. When using external APIs, this data leaves your environment and is processed on third-party infrastructure. Depending on the provider and configuration, it may also be used for model improvement or training, which creates a risk, especially when working with sensitive information.

In regulated industries such as healthcare, finance, or defence, this alone may rule out the use of external model providers. In such cases, organisations often need to make sure that all data remains within their own infrastructure.

Regulatory Constraints

In regions such as the European Union, data protection regulations impose strict rules on how personal data can be processed, defining what can be shared, where it can be stored, and how it must be handled. As a result, certain architectures, particularly those relying on external APIs, may not be compliant.

This is not a purely technical decision. It often requires legal validation, especially when dealing with personal or confidential data.

Deployment Implications

If data cannot leave your environment, self-hosting becomes the default option. In some cases, organisations may use controlled cloud environments, such as virtual private clouds, where data handling is more restricted and isolated. These setups can offer a compromise, but they still need to be evaluated carefully against regulatory requirements.

Language and Cultural Fit

Another important factor is language. Many models are trained primarily on English data, which can affect performance in other languages. Applications that rely on local language, cultural context, or domain-specific terminology may require models trained on more relevant datasets. It can have a direct impact on output quality, especially in use cases where nuance and accuracy matter.

Example Use Cases and Model Selection Scenarios

The best way to understand how different factors influence model selection is to look at practical scenarios. The priorities change depending on the type of application, the scale of the workload, and the operational constraints.

Simple Internal Automation

A relatively simple example would be an internal assistant used for talking to data, summarising emails, or handling basic classification tasks. In this case, the workload is usually limited, there may only be a single user, and advanced reasoning capabilities are unnecessary.

Under these circumstances, lightweight models accessed through external APIs can often be a perfectly reasonable choice, especially when no sensitive data is involved and the goal is to deploy something quickly.

Customer-Facing Chatbots

The situation changes when building customer-facing systems such as e-commerce chatbots, product assistants, or customer support tools.

Here, priorities shift towards scalability, uptime, and low latency. Users expect immediate responses, which makes response speed a critical factor. At the same time, these applications may still not require advanced reasoning, meaning that smaller cloud-hosted models can remain sufficient.

Large-Scale Data Processing

Another scenario involves large-scale internal processing, such as data cleaning or warehouse operations.

In these cases, latency is less important because tasks can run asynchronously. Data control, however, plays a much bigger role. If information needs to remain within the organisation’s environment, self-hosting is often a more suitable option.

Solutions such as LLaMA or Qwen, deployed through tools like Ollama, may offer a practical balance between capability and control.

There are also intermediary options, such as Together AI or Replicate, which reduce infrastructure overhead while still offering more flexibility than a single-provider API.

Multi-Agent Architectures

The most demanding example is multi-agent architecture.

In such systems, one agent may act as a planner or coordinator while others perform narrower operational tasks. This often requires combining different models with different capabilities. Smaller models may be assigned to specialised tasks, while more advanced reasoning-focused systems handle orchestration and planning.

As communication between agents increases, hidden costs and operational complexity also grow. In these situations, achieving reliable outcomes becomes more important than minimising every token cost.

Language and Cultural Context

Language can also become a deciding factor.

Applications operating primarily in Polish or other non-English languages may benefit from models trained on local datasets or fine-tuned on language-specific data. This can improve grammar handling, terminology accuracy, and cultural understanding.

Conclusion: How to Choose the Right LLM

Choosing the right large language model starts with asking the right questions:

• What exactly do you expect from the model?
• Does the task involve reasoning?
• Does it require tool calling?
• Is the problem bounded or open-ended?
• Is response time critical, or can delays be tolerated?
• Do you need a domain-specific model or a general-purpose one?
• Does language or cultural context matter?

At the same time, consider your constraints. If you are working with sensitive data, using external APIs may not be an option, which will immediately limit your choices.

Once selected, the model should be continuously observed and evaluated. Define your own tests to understand where it works well and where it does not. This will help you improve results through prompt design, fine-tuning, or by choosing a different model.

There is no single correct answer. The right model is the one that fits your use case and performs reliably in practice.