LLMs
7 posts
Evaluating LLM Apps: Treat Prompts Like Pipelines
A data pipeline without tests breaks quietly. An LLM feature without evaluations breaks quietly and changes behaviour whenever someone edits a prompt or the provider ships a new model version. The fix is the same discipline data engineers already use: a fixed dataset, automated checks and a number you watch. 1. Build a golden dataset Collect 50–200 real inputs: actual user questions, real documents, messy edge cases. For each, write down what a good output looks like. That could be an exact label, a required fact, or a short rubric. Version it in git next to the prompts. Start small. Thirty well-chosen examples beat none. 2. Use the cheapest check that works In order of preference: 1. Deterministic checks. Valid JSON? Matches the schema? Contains the required field? Correct label? These are fast, free and unambiguous. 2. Reference comparisons. Does the extracted amount equal the labelled amount? Did retrieval return the labelled document? 3. LLM-as-judge. A second model grades the output against a rubric ("Is every claim supported by the sources? yes/no, and why"). Useful for open-ended text, but spot-check the judge against human ratings before you trust it. 3. Score the parts separately For a RAG app, a single "quality" number hides the cause. Track retrieval (did the right chunk come back?), faithfulness (does the answer stick to the sources?) and answer quality on their own. For an agent, track task success, steps and cost per task. 4. Run evals like CI text prompt or model change │ ▼ run the golden set ──► scores vs. last release │ ▼ regression? ── yes ──► block the change │ no ▼ ship it Every prompt edit, model upgrade or retrieval change runs the suite. A drop in any metric beyond a small tolerance blocks the change, just as a failing dbt test blocks a deploy. 5. Keep watching in production - Log inputs, outputs, latency, token cost and the model version for every call. - Sample production traffic for review each week and add interesting failures to the golden set. - Collect cheap user signals such as thumbs up or down, and whether the user retried or edited the answer. 6. Don't forget cost and latency Quality is one axis. A change that improves accuracy by two points but doubles latency or cost per request is a trade-off, not a win. Put all three on the same dashboard. The teams that ship reliable GenAI features aren't the ones with the cleverest prompts. They're the ones that can tell, within minutes, whether a change made things better or worse. Try it: a five-question eval that fails Here's a tiny golden set with cheap graders (substring, exact match, regex) run against a 1.7B local model: bash curl -O https://www.yashank.site/files/demos/ollamaclient.py curl -O https://www.yashank.site/files/demos/evaldemo.py python evaldemo.py This is the eval working, not breaking. It caught a model inventing a port number, contradicting a basic SQL fact, and running past the length budget. Swap in a bigger model or a better prompt, rerun, and you have a number to compare instead of a feeling.
How LLMs Work: A Mental Model for Engineers
You don't need to train a large language model to build good things with one. You do need an accurate picture of what it does, because most bad LLM features come from expecting it to behave like a database or a search engine. It predicts the next token An LLM reads a sequence of tokens (chunks of text, often parts of words) and predicts a probability for every possible next token. It picks one, appends it, and repeats. Everything else (answering, summarising, writing SQL) emerges from doing that very well. Two consequences: - It generates plausible text, not retrieved facts. If the right answer wasn't in its training data or in the prompt, it will still produce something that sounds right. That's a hallucination, and it's the default behaviour, not a bug. - Cost and speed scale with tokens. You pay for input and output tokens, and long outputs are slow because they're generated one token at a time. The context window is its only working memory The model knows nothing about your conversation, your company or your data except what's in the context window for this request: the system prompt, the chat history, any documents you pasted. Close the request and it's gone. That's why "chat with your docs" products don't retrain anything. They find the relevant text and put it in the context. That pattern has a name: retrieval-augmented generation, or RAG. Sampling controls variety - Temperature near 0 makes the output close to deterministic. Use it for extraction, classification and SQL. - Higher temperature gives more varied wording. Use it for brainstorming and creative text. Even at temperature 0, don't assume byte-identical output across runs or model versions. Validate the result instead. Embeddings: meaning as numbers An embedding model turns text into a vector, a list of numbers, where texts with similar meaning land close together. Search "getting fit" and you can find a post about a workout routine that never uses those words. Embeddings power semantic search, deduplication, clustering and the retrieval step of RAG. Ask for structure When the output feeds code, ask for JSON that matches a schema, and validate it: python from pydantic import BaseModel class Claim(BaseModel): claimid: str amountinr: float category: str raw = llmcall(prompt) your provider's SDK claim = Claim.modelvalidatejson(raw) fails loudly on bad output Most providers now offer a structured-output or JSON mode. Use it, and still validate. Where LLMs fit in a data stack They're good at messy-text-to-structure jobs: classifying free-text claim notes, extracting fields from PDFs, writing a first draft of a SQL query, summarising incident logs. They're bad at arithmetic over large tables and at being the source of truth. Let the warehouse compute; let the model read and write language. Try it with a local model Everything above is easy to see on a laptop with Ollama. This script asks qwen3:1.7b for a product name four times at two temperatures, then asks for a log classification that has to match a JSON schema: bash ollama pull qwen3:1.7b curl -O https://www.yashank.site/files/demos/ollamaclient.py curl -O https://www.yashank.site/files/demos/samplingdemo.py curl -O https://www.yashank.site/files/demos/structureddemo.py python samplingdemo.py && python structureddemo.py The token counts are worth a look too: a 20-word prompt came to 32 tokens once the chat template was added. That's the number you pay for and the number that fills the context window.
Embeddings Explained: Meaning as Vectors
Most people meet embeddings as the retrieval step in a RAG pipeline and never look further. That's a shame, because they're one of the most useful tools in applied machine learning, and they're far simpler than the models that generate text. What an embedding is An embedding is a list of numbers (a vector) that represents a piece of input, such as a sentence, a product description or an image. An embedding model is trained so that inputs with similar meaning end up with vectors that point in similar directions. "How do I reset my password?" and "I forgot my login" share almost no words, but a good text embedding model places them close together. "How do I reset my router?" lands somewhere nearby but noticeably further away. That's the whole trick. Once meaning is geometry, you can compare, group and search things using basic linear algebra. Cosine similarity The standard way to compare two embeddings is cosine similarity: the cosine of the angle between the vectors. text cos(a, b) = (a · b) / (|a| |b|) It ranges from -1 to 1. Close to 1 means the vectors point the same way, close to 0 means they're unrelated. In practice, with most text embedding models, scores cluster in a narrower band, so "0.8" doesn't have a universal meaning. You learn what a good score looks like for your model and your data. Cosine similarity ignores vector length and only looks at direction. That's usually what you want, since length often reflects things like input size rather than meaning. Normalisation If you normalise every vector to unit length (divide by its norm), cosine similarity becomes a plain dot product. That's faster to compute and it's what many vector databases assume. Some embedding APIs return normalised vectors already. Check the documentation rather than assuming. Normalising an already normalised vector is harmless, so when in doubt, do it once at write time and store the result. A small runnable example Real embeddings have hundreds or thousands of dimensions. These toy vectors have four, hand-made so the dimensions loosely mean "account", "networking", "billing" and "food". The maths is identical. python import numpy as np docs = { "reset my password": [0.9, 0.1, 0.0, 0.0], "forgot my login": [0.8, 0.2, 0.1, 0.0], "router keeps disconnecting": [0.1, 0.9, 0.0, 0.0], "wifi is very slow": [0.0, 0.8, 0.1, 0.1], "charged twice this month": [0.1, 0.0, 0.9, 0.0], "refund for my invoice": [0.2, 0.0, 0.8, 0.1], "best biryani nearby": [0.0, 0.0, 0.1, 0.9], } names = list(docs) X = np.array([docs[n] for n in names], dtype=float) X = X / np.linalg.norm(X, axis=1, keepdims=True) normalise once def search(queryvec, k=3): q = np.array(queryvec, dtype=float) q = q / np.linalg.norm(q) scores = X @ q cosine via dot product top = np.argsort(-scores)[:k] return [(names[i], round(float(scores[i]), 3)) for i in top] print("Search 'can't sign in':") for name, score in search([0.85, 0.15, 0.05, 0.0]): print(f" {score:.3f} {name}") Near-duplicate detection: pairs above a threshold sim = X @ X.T print("\nLikely duplicates ( 0.95):") for i in range(len(names)): for j in range(i + 1, len(names)): if sim[i, j] 0.95: print(f" {sim[i, j]:.3f} {names[i]!r} ~ {names[j]!r}") Classification by nearest labelled centroid labels = {"account": [0, 1], "network": [2, 3], "billing": [4, 5]} centroids = {k: X[idx].mean(axis=0) for k, idx in labels.items()} centroids = {k: v / np.linalg.norm(v) for k, v in centroids.items()} ticket = np.array([0.15, 0.05, 0.85, 0.05]) ticket = ticket / np.linalg.norm(ticket) best = max(centroids, key=lambda k: centroids[k] @ ticket) print(f"\nNew ticket classified as: {best}") Swap the toy vectors for output from a real embedding model and the same code works for search, deduplication and classification. Uses beyond RAG - Semantic search. Find documents by meaning rather than keywords. Often best combined with keyword search (hybrid search), because embeddings can miss exact terms like product codes. - Deduplication. Near-identical support tickets, product listings or records from different sources score very high against each other. A threshold plus human review of the borderline band goes a long way. - Clustering. Run k-means or a density-based method on embeddings to find themes in thousands of free-text responses without labelling anything first. - Classification. Train a simple logistic regression on embeddings, or use nearest centroid as above. With a few hundred labelled examples this is often competitive with much heavier approaches, and far cheaper to run. - Anomaly detection. Items far from every cluster are worth a look. Choosing dimensions More dimensions can capture more nuance, but cost more to store and compare. A million 1,536-dimensional float32 vectors is roughly 6 GB before any index overhead. Some newer models are trained so you can truncate vectors to fewer dimensions with modest quality loss. If yours supports that, test a smaller size on your own retrieval eval before defaulting to the largest. Re-normalise after truncating. Pitfalls - Mixing models. Vectors from different embedding models (or different versions of the same one) are not comparable. Changing models means re-embedding everything. Store the model name alongside each vector. - Assuming a universal threshold. A similarity cut-off that works for one model and domain won't transfer. Calibrate on labelled pairs. - Long inputs. Models have input limits and may truncate silently. Embedding a whole document into one vector also blurs its topics together. Chunk sensibly. - Domain mismatch. A general model may not separate terms that matter in your field, like insurance policy types. Test on your own data. - Exact matches. Embeddings are weak at IDs, numbers and rare names. Keep keyword search around. Where to start - Pick one embedding model, record its name with every vector, and normalise at write time. - Build a small labelled set of similar and dissimilar pairs from your own data, and use it to choose thresholds and dimensions. - Try deduplication or clustering before RAG. They're simpler, easier to evaluate and often more immediately useful. - Use hybrid search when exact terms matter.
Prompting, RAG or Fine-Tuning? Picking the Right Lever
When an LLM doesn't do what you want, there are three main levers: change the prompt, give it better context with retrieval, or change the model itself with fine-tuning. They get discussed as if they compete. They don't. They fix different problems, and picking the wrong one is an expensive way to not solve yours. The one distinction that matters Before anything else, ask whether the problem is about knowledge or behaviour. - Knowledge: the model doesn't know something. Your internal docs, last week's policy change, a customer's order history. - Behaviour: the model knows enough but responds the wrong way. Wrong format, wrong tone, ignores a rule, too verbose, inconsistent at a narrow task. Retrieval mostly fixes knowledge. Fine-tuning mostly fixes behaviour. Prompting can help with both, a bit, and it's nearly free, so it always comes first. Prompting Prompting changes nothing about the model. You're just giving it better instructions for this request. What it's good at: - clear task definitions and constraints - output formats, especially combined with structured-output or JSON modes - a few worked examples (few-shot prompting) to show the pattern you want - small amounts of reference information pasted directly in Cost: basically your time, plus the extra input tokens. Data needed: a handful of examples. You can iterate in minutes. Where it runs out: when the information you need is too large or changes too often to paste in, or when you've written a page of rules and the model still drifts on some of them. RAG Retrieval-augmented generation keeps the model unchanged and adds a search step. At query time you find the most relevant chunks of your data and put them in the context window alongside the question. python def answer(question: str) - str: chunks = vectorstore.search(embed(question), k=5) retrieve context = "\n\n".join(c.text for c in chunks) prompt = ( "Answer using only the context below. " "If the answer isn't there, say so.\n\n" f"Context:\n{context}\n\nQuestion: {question}" ) return llmcall(prompt) generate What it changes: what the model knows for this request. Nothing is learned permanently. What it's good at: - large or frequently changing knowledge bases - answers that need citations, since you know exactly which chunks were used - access control, because you can filter retrieval by what the user is allowed to see - keeping information current: update the index, and the next answer reflects it Cost: an embedding model, a vector store (or a search engine you already run), and the engineering to chunk, index and keep data fresh. More input tokens per call. Data needed: your documents. No labelled examples required to get started, though you'll want some question–answer pairs to evaluate retrieval quality. Where it struggles: most RAG failures are retrieval failures. If the right chunk isn't found, the model can't use it. Bad chunking, weak queries and missing metadata cause more trouble than the choice of model. Hybrid search (keyword plus embeddings) and reranking help a lot. Fine-tuning Fine-tuning continues training the model on your own examples, so the weights change. What it changes: behaviour. How the model responds by default: format, style, terminology, how it handles a narrow task. It's a poor way to add facts. The model may pick some up, but unreliably, they go stale, and you can't cite them. Use retrieval for facts. What it's good at: - consistent output format or tone without a long prompt every time - narrow, repetitive tasks such as classification, extraction and routing, where a smaller tuned model can match a larger general one at lower cost and latency - shortening prompts, which cuts tokens on every call LoRA Full fine-tuning updates every weight, which needs a lot of GPU memory. LoRA (low-rank adaptation) freezes the original weights and trains small extra matrices added to certain layers. You end up with an adapter that's a small fraction of the model's size. Why it matters in practice: - far less memory to train, so it fits on modest hardware (QLoRA goes further by quantising the base model) - you can keep several adapters for different tasks on one base model - results are often close to full fine-tuning for task adaptation Cost: preparing data is usually the biggest cost, followed by training runs, evaluation and hosting the tuned model. Data needed: hundreds to thousands of good input–output examples, consistent and representative. Fifty sloppy examples will teach the model to be sloppy. The hidden cost is maintenance. When a better base model comes out, you retrain. When requirements change, you rebuild the dataset. They combine These aren't exclusive. A common mature setup is a model fine-tuned for format and domain behaviour, fed by RAG for current facts, with a solid prompt holding it together. But each layer should earn its place. A decision checklist Work through these in order: 1. Have I written a clear prompt with a few examples and tested it on real inputs? If not, do that first. 2. Is the failure about missing or changing information? Use RAG. 3. Do I need citations, freshness or per-user access control? Use RAG. 4. Is the failure about format, tone or consistency, after good prompting? Consider fine-tuning. 5. Do I have hundreds of high-quality examples, or can I build them? If not, fine-tuning isn't ready yet. 6. Is cost or latency the problem at high volume on a narrow task? A small model fine-tuned with LoRA may beat a large one. 7. Do I have an evaluation set to prove each change helped? Without it, none of these decisions are measurable. In short - Prompt first. It's cheap and fast, and often enough. - Knowledge problem: retrieve. Behaviour problem: fine-tune. - Spend effort on retrieval quality before blaming the model. - Fine-tune only with good data and an eval set, and prefer LoRA unless you have a reason not to.
Open-Weight vs Closed Models: How I Choose
Every LLM project I start hits the same fork early: call a hosted API, or run an open-weight model myself. It's a trade-off across a handful of dimensions, and once you list them the answer for a given project is usually obvious. What the two camps actually are Closed models are the ones you reach through a provider's API: the GPT, Claude and Gemini families. You send a request, you get tokens back, and you never see the weights. Open-weight models publish their weights so you can download and run them: the Llama, Qwen, Mistral and Gemma families, among others. "Open-weight" is the honest term. Most of them don't release training data or the full training recipe, so they aren't open source in the sense the term usually carries for software. Neither group is uniformly better. The best closed models tend to lead on the hardest reasoning tasks, but the gap on everyday work like classification, extraction and summarisation is often small, and a well-chosen open model can be the better tool. Privacy and data residency This is usually the first filter, and sometimes the only one that matters. With a closed API, your prompts leave your infrastructure. Providers offer enterprise terms, zero-retention options and regional hosting, and for many teams that's enough. For others, particularly anything touching financial, health or customer records, the policy answer is simply "data does not leave our network". In that case an open-weight model running on your own hardware or a private cloud deployment isn't a preference, it's the requirement. Cost The cost models are different in shape, not just size. - Closed APIs charge per token. Zero cost when idle, cost grows linearly with use. - Self-hosted open models cost roughly the same whether you send one request or a million, because you're paying for GPUs by the hour. Cheap per request at high, steady volume; expensive if the machine sits idle. A rough way to think about it: estimate monthly tokens, multiply by the API price, and compare that with the cost of the smallest GPU setup that can serve the model at the latency you need. At low volume the API almost always wins. At high, predictable volume, self-hosting starts to look attractive, provided someone is going to run it. Latency Hosted APIs are fast but you don't control them. You share capacity with everyone else, and response times can vary through the day. Self-hosting gives you control: you can put a small model next to your application, keep it warm, and get consistent latency. A small open model that answers in a fraction of a second can beat a much stronger remote model for anything interactive, like autocomplete or routing a request to the right handler. Quality Measure it on your task, not on a leaderboard. Build a small evaluation set (fifty to a few hundred real examples with known good answers) and run every candidate against it. What I usually find: - For open-ended reasoning, long multi-step instructions and tricky code, the top closed models are still the safest bet. - For narrow, well-defined jobs, a mid-sized open model often lands close enough, especially with a good prompt and a few examples. "Close enough" is a product decision. A 2% drop in accuracy may be fine for tagging support tickets and unacceptable for anything a customer reads directly. Licensing Open-weight doesn't mean do whatever you like. Licences vary across families: some are permissive (Apache 2.0 style), others are custom licences with acceptable-use policies, attribution rules or restrictions above a certain user count. Read the licence for the specific model before building a product on it, and check it again when you upgrade, because terms can change between releases. Closed APIs have their own terms too, usually around what you can use outputs for. Read those as well. Fine-tunability With open weights you can fine-tune however you want: full fine-tuning, or parameter-efficient methods like LoRA that train a small set of extra weights. You own the result and can run it anywhere. Some closed providers offer fine-tuning through their API, but on their terms: limited model choice, limited control over the process, and the tuned model lives on their platform. If adapting the model's behaviour deeply is central to the project, open weights give you far more room. Ops burden This is the dimension people underestimate. Running a model means: - provisioning and paying for GPUs - choosing and maintaining a serving stack (vLLM, TGI, llama.cpp or similar) - handling scaling, monitoring, failover and upgrades - managing quantisation trade-offs between memory and quality An API call is one line of code and someone else's pager. If there's no one on the team who wants to own inference infrastructure, that's a strong vote for closed. How it plays out A simplified version of my decision: text Must data stay in-house? - open-weight, self-hosted Prototype or low/spiky volume? - closed API Hardest reasoning, top quality? - closed API (check with evals) Narrow task, high steady volume? - open-weight, possibly fine-tuned Need deep behaviour changes? - open-weight + LoRA No one to run infra? - closed API, or a managed host for open models The last line matters: there's a middle path. Several providers host open-weight models behind an API, so you get the model choice and licence of open weights with per-token pricing and no GPUs to babysit. The practical bit - Start with a closed API to prove the feature is worth building. It's the fastest way to learn what "good" looks like. - Build an evaluation set from day one, so switching models is a measurement rather than a debate. - Keep the model behind a thin interface in your code. Swapping providers or moving to a self-hosted model should be a config change, not a rewrite. - Revisit the choice when volume, data sensitivity or the model landscape changes. It isn't a decision you make once.
Prompt Engineering Is Mostly Specification Writing
"Prompt engineering" sounds like a bag of tricks: magic phrases, threats, tipping the model. Some of those tricks worked once, on some model, for some task. What keeps working is far more boring. A good prompt is a specification: it says what the input is, what the output must look like, what's out of scope, and how you'll know it's right. If you've ever written a ticket that a stranger could pick up and finish without asking you anything, you already know how to write most of a good prompt. Treat the prompt as a contract The model has no idea what you're building. It sees text and predicts more text. Every gap in your instructions gets filled with whatever is most plausible, and plausible is not the same as what you wanted. So write the prompt the way you'd write an interface contract: - Inputs: what the model will receive and in what form. - Outputs: the exact shape of the response. - Constraints: what it must not do. - Edge cases: what to do when the input is empty, ambiguous or off-topic. Vague prompt: text Summarise this customer complaint. Specified prompt: text You will receive one customer complaint email for a food delivery app. Return a JSON object with: - "category": one of "late", "wrongitem", "refund", "appbug", "other" - "summary": one sentence, under 25 words, no customer names - "needshuman": true if the customer mentions legal action or a health issue If the email is not a complaint, return {"category": "other", "summary": "", "needshuman": false}. The second one is longer, but every extra line removes a decision the model would otherwise make for you. Roles are context, not magic "You are a senior data engineer" doesn't make a model smarter. What a role line does is set context: vocabulary, assumed audience, level of detail. That's useful, but it's a small lever. A role with no task description is still a vague prompt. I find it more reliable to describe the situation directly: who will read the output, what they already know, and what they'll do with it. "The reader is a support agent who has 30 seconds to decide whether to escalate" tells the model more than any job title. Constraints do the heavy lifting Most bad outputs aren't wrong, they're unbounded: too long, too chatty, padded with caveats, or inventing details. Constraints fix that. - Length: "under 25 words", "at most three bullet points". - Scope: "use only the information in the provided text". - Refusal path: "if the answer isn't in the text, say NOTFOUND". - Tone: "plain language, no marketing words". Giving the model an explicit way to say "I don't know" matters more than people expect. Without one, the most plausible continuation is usually a confident guess. Examples beat adjectives "Write a concise, professional summary" means different things to different people, and to models. One or two examples of input and expected output pin down what you mean far better than adjectives. A few things I've learned about examples: - Make them representative, including at least one awkward case. - Vary them. If every example has three bullet points, every output will too. - Keep them consistent with your stated rules. If the instructions say 25 words and the example has 40, you've written a contradiction and the model will pick one. Ask for a schema when code reads the output If the response goes into a program, don't parse prose. Ask for JSON with named fields and allowed values, and validate it on the way in: python from typing import Literal from pydantic import BaseModel, Field class Complaint(BaseModel): category: Literal["late", "wrongitem", "refund", "appbug", "other"] summary: str = Field(maxlength=200) needshuman: bool raw = callmodel(prompt, emailtext) result = Complaint.modelvalidatejson(raw) raises on bad output Many APIs now have a JSON or structured-output mode. Use it, and still validate. A schema in the prompt plus a schema in code is the same contract written down twice, which is exactly what you want. Specs need tests A specification you never check is a wish. The step most people skip is evaluation: running the prompt against a fixed set of inputs and checking the outputs, every time you change something. It doesn't need to be fancy: python cases = [ ("My order came 2 hours late and cold.", "late"), ("I got paneer instead of chicken.", "wrongitem"), ("Hi, what are your opening hours?", "other"), ] failures = [] for text, expected in cases: out = Complaint.modelvalidatejson(callmodel(prompt, text)) if out.category != expected: failures.append((text, expected, out.category)) print(f"{len(cases) - len(failures)}/{len(cases)} passed") Twenty or thirty hand-picked cases catch most regressions. When you find a new failure in the wild, add it to the set. This is ordinary test-driven thinking applied to a component that happens to be probabilistic. For outputs that can't be checked with equality, such as summaries, you can check properties instead: length limits, required fields, banned words, whether named entities in the output actually appear in the input. Iterate like you'd debug When a prompt fails, resist the urge to add another sentence in capital letters. Look at the failing case and ask what the spec left open. Usually it's one of these: - An edge case nobody described. - Two instructions that conflict. - An example that teaches the wrong pattern. - A task that's really two tasks and should be two calls. Change one thing, rerun the eval set, compare. It's slower than vibes but you'll know whether you made it better. What to keep in mind - Write the prompt so a capable stranger could do the task from it alone. - Define the output shape exactly, and validate it in code. - Give the model an explicit way to say it doesn't know. - Use a couple of varied examples instead of adjectives. - Keep a small test set and rerun it on every change. Most of what gets called prompt engineering is just being precise about what you want. That skill was useful before LLMs and it'll stay useful after the current tricks stop working.
Attention, Explained for Engineers
Every explanation of transformers I found while studying either stopped at "attention lets the model focus on relevant words" or jumped straight into a wall of matrix notation. The first is too vague to reason with. The second hides a fairly simple idea. This is the middle version: enough to understand why models behave the way they do, and why long prompts cost what they cost, with code you can run. Text becomes tokens A model never sees characters or words directly. A tokenizer splits text into tokens, chunks from a fixed vocabulary that are often whole common words, word pieces, or punctuation. "Unbelievably" might become un, believ, ably. Each token maps to an integer ID. This is why models are oddly bad at counting letters in a word: they never saw the letters, only the chunks. Tokens become embeddings Each token ID looks up a row in an embedding matrix, giving a vector of a few hundred to a few thousand numbers. These vectors are learned during training, and tokens used in similar contexts end up with similar vectors. So a sentence of n tokens becomes a matrix of shape (n, dmodel). Everything after this is arithmetic on that matrix. The problem attention solves A token's meaning depends on its neighbours. "Bank" in "river bank" and "bank account" should end up represented differently. Older recurrent networks passed information along the sequence one step at a time, which was slow to train and tended to lose track of distant words. Attention lets every token look at every other token directly, in one step, and decide how much each one matters to it. Queries, keys and values For each token, the model computes three vectors by multiplying its embedding with three learned weight matrices: - Query (Q): what this token is looking for. - Key (K): what this token offers to others looking. - Value (V): the information it actually passes along if chosen. The analogy I find useful is a fuzzy dictionary lookup. A normal dictionary matches a query to exactly one key and returns its value. Attention compares a query against every key, scores how well each matches, and returns a weighted blend of all the values. Scaled dot-product attention The scoring is a dot product: Q · K is large when the two vectors point in similar directions. For a whole sequence at once: Attention(Q, K, V) = softmax(Q Kᵀ / √dk) V Step by step: 1. Q Kᵀ gives an (n, n) matrix of scores: how much each token cares about each other token. 2. Divide by √dk (the key dimension). Without this, dot products grow with dimension, softmax saturates, and gradients become tiny. 3. Softmax each row, turning scores into weights that are positive and sum to 1. 4. Multiply by V, so each token's output is a weighted average of the value vectors. Here it is in NumPy: python import numpy as np def softmax(x, axis=-1): x = x - x.max(axis=axis, keepdims=True) for numerical stability e = np.exp(x) return e / e.sum(axis=axis, keepdims=True) def attention(Q, K, V, causal=False): dk = Q.shape[-1] scores = Q @ K.T / np.sqrt(dk) (seq, seq) if causal: mask = np.triu(np.oneslike(scores, dtype=bool), k=1) scores = np.where(mask, -np.inf, scores) weights = softmax(scores, axis=-1) each row sums to 1 return weights @ V, weights rng = np.random.defaultrng(0) seqlen, dmodel, dk = 4, 8, 8 X = rng.normal(size=(seqlen, dmodel)) 4 token embeddings Wq = rng.normal(size=(dmodel, dk)) Wk = rng.normal(size=(dmodel, dk)) Wv = rng.normal(size=(dmodel, dk)) out, w = attention(X @ Wq, X @ Wk, X @ Wv, causal=True) print(out.shape) (4, 8) print(w.round(2)) lower-triangular: no token sees the future The causal flag is what text-generating models use. Setting future positions to negative infinity before softmax makes their weights zero, so token 3 can attend to tokens 1 to 3 but not 4. That's what lets the model be trained to predict the next token without cheating. In a real model the weight matrices are learned, not random. Multi-head attention One attention pattern can only capture one kind of relationship at a time. Multi-head attention runs several attention operations in parallel, each with its own smaller Q, K and V projections. One head might learn to track which noun a pronoun refers to; another might focus on the previous token. The heads' outputs are concatenated and projected back to dmodel. A transformer block is multi-head attention followed by a small feed-forward network applied to each token separately, with residual connections and normalisation around both. A model stacks dozens of these blocks. Where does word order come from? Look at the formula again: nothing in it knows about position. Shuffle the input tokens and each token's output is the same, just shuffled. Attention on its own treats a sentence as a bag of tokens. So position has to be injected. The original transformer paper added fixed sine and cosine patterns to the embeddings. Some models learn a position embedding per slot. Many recent models use rotary position embeddings (RoPE), which rotate the query and key vectors by an angle that depends on position, so their dot product reflects relative distance. The details vary; the point is that order is added information, not something attention gets for free. Why context windows cost what they do The score matrix is n × n. Double the prompt length and you quadruple the number of query-key comparisons, per head, per layer. That quadratic cost is the main reason context windows were small for so long and why long prompts are slow and expensive. There's a second cost during generation. To avoid recomputing everything for each new token, models keep a KV cache: the keys and values for every previous token, at every layer. It grows linearly with context length and can eat a lot of GPU memory, which limits how many requests a server can handle at once. Plenty of engineering goes into softening this, from memory-efficient attention kernels to sparse and sliding-window attention patterns. But the basic shape holds: more context means more compute and memory, and providers price accordingly. The short version - Text becomes tokens, tokens become vectors. - Attention scores every token against every other with Q · K, softmaxes the scores, and blends the values. - Multiple heads learn different relationships; position has to be added separately. - The n × n score matrix and the KV cache are why long contexts are expensive. If you remember one thing, make it the fuzzy dictionary lookup. Most transformer behaviour, good and odd, makes more sense once you picture every token asking every other token "are you relevant to me?" at every layer.