1. 01The basics
  2. 02Intermediate
  3. 03Advanced
  4. 04Expert

Phase 3 · Advanced

Advanced: efficiency and economics

Where the money actually goes in an agentic workload, and the three habits that cut it: send less, cache what repeats, and match the model to the task.

Large models versus small ones

Most people pick one model and use it for everything. That is like hiring a senior architect to reformat CSV files — it works, and you are paying enormously for capability you are not using.

Frontier models are the large, expensive, slow ones — the top tiers of Claude, ChatGPT, and Gemini. They are worth their price on genuinely hard reasoning: unfamiliar architecture decisions, subtle debugging, anything where being wrong is costly and the path is not obvious.

Each of those families also ships cheaper, faster tiers — a mid-sized and a small variant alongside the flagship. Moving down one tier within the same family is usually the easiest saving available, and most people never try it because they picked the best model on day one and never revisited the choice.

Small language models — the Llama, Mistral, Gemma, and Phi families, run locally through something like Ollama or served with vLLM — are dramatically cheaper and faster still, and your data never leaves your machine. On narrow, well-specified, high-volume work they are frequently indistinguishable from the expensive option.

A note on names

Specific model names and prices date faster than anything else in this book, so this phase deliberately talks in categories. Check the provider's own pricing page before making a decision on numbers.

Choosing per task

The useful question is not "which model is best" but "what is the cost of being wrong here". Three rough tiers:

  • Mechanical work — classification, extraction, reformatting, tagging, routing, generating embeddings. High volume, easily checked, cheap to redo. Use the smallest thing that passes your tests.
  • Ordinary generation — drafting, summarising, routine code, tests for existing behaviour. A mid-tier model, and measure before assuming you need more.
  • Hard reasoning — architecture, non-obvious bugs, security-relevant decisions, anything touching money or user data. Use the best model available. This is where the price is actually justified.

A pipeline that routes between tiers beats any single choice. The classifier deciding which tier a request belongs to can itself be a small, cheap model.

What you are actually paying for

You are billed per token, separately for input and output, and input is where agentic workloads spend nearly everything. A long autonomous session sends its whole accumulated context on every step — so a fifty-step task pays for that context fifty times.

That is why context discipline is a cost strategy and not just a quality one. Halving what you send does not halve one bill; it halves every step of every loop, forever.

Prompt caching in practice

Caching stores the processed form of your prompt's prefix, so an identical prefix on the next request skips the computation. Reads are heavily discounted and noticeably faster.

The two major providers approach it differently, and the difference matters for agents:

  • Anthropic — explicit. You mark cache breakpoints yourself. There is a surcharge to write to the cache, and reads are discounted around ninety percent within the lifetime window. You control it, so it is predictable, which is what a production loop wants.
  • OpenAI — automatic. Long prefixes are cached without any code change, at a smaller discount, on a best-effort basis depending on routing. Less to think about, less to rely on.

Both require a minimum prefix length before caching engages at all, and both eventually expire. Treat the specific figures as things to verify in the docs rather than memorise — they change.

The practical rule

Static content first, volatile content last, and never edit the front of the prompt casually. Everything else is tuning.

Hybrid RAG

Retrieval-augmented generation means finding the few relevant pieces of your own documents and putting only those in front of the model, rather than hoping it already knows or stuffing everything in.

The hybrid arrangement splits the job by cost. A small local embedding model converts your documents into vectors once, and converts each incoming question into a vector to find the closest matches. Only those matched snippets — a few thousand tokens, not a few hundred thousand — go to the expensive model, which does the actual reasoning.

  1. Chunk your documents into passages.
  2. Embed each passage locally and store the vectors.
  3. At question time, embed the question and retrieve the closest passages.
  4. Send only those passages plus the question to the frontier model.

The retrieval step costs almost nothing and runs on your own hardware. You pay frontier prices only for the reasoning, and the model sees a short, relevant context instead of a long, noisy one — which usually improves the answer as well as the bill.

Cost at scale

Three habits, in order of how much they save:

  1. Send less. Progressive disclosure and retrieval beat every other optimisation, because the cheapest token is the one you never send.
  2. Cache what repeats. Stable prefix at the front, and the savings compound across every step of every session.
  3. Match the model to the task. Route mechanical work to small models and keep the expensive one for decisions that deserve it.

And measure before optimising. Log tokens per request, split input from output, and find your actual largest line item — it is very often something dull, like a tool definition resent on every call, rather than the thing you assumed.