AI philosophy
6 posts
Opus 5.5 and the Junior Question, Seventeen Months Later
In May 2025 I wrote Will AI Replace Junior Developers? as a student about to start his final year. My answer then was a careful "some of them, probably". The role would shrink, the bar would rise, but juniors wouldn't disappear, because companies need people who grow into seniors. I've reread that post a few times since Anthropic released Claude Opus 5.5 on 22 September. I still believe the reasoning. I'm much less relaxed about the timeline, and I want to be honest about that. What changed in a few months Opus 5.5 isn't one dramatic leap. It's what happens when a lot of small improvements land at once and the price drops too. Anthropic says it performs at the level of their bigger Fable 5.1 model on most work and costs about 40% less to run than Opus 5 on typical workloads. Their launch post mentions one early tester finishing a 680,000-line code migration in under a day, and an internal test where it translated HAProxy from C to Rust and passed nearly all of HAProxy's own regression tests in about nine and a half hours. Those are a vendor's own examples, so I discount them a bit. What worries me more is what's happening outside the press releases. - Prototypes are almost free now. Something that used to be a weekend side project, like a CRUD app with auth, a dashboard or a small mobile app, now takes an evening of describing and reviewing. I built NotesAid the slow way in 2025. A first-year student today could get a rough version running before dinner. - Small products ship with tiny teams. Building a working app was never the hard part of starting something. It's just stopped being a serious obstacle at all. - Reverse engineering has sped up a lot. In August, Chris Lewis finished a full decompilation of the N64 game Snowboard Kids in 84 days. His previous one took 596. He credits LLMs for a lot of that, though he's clear that experienced people in the decomp community were essential, especially for the strange quirks of the old SGI compiler. This month a developer released open-source "clean-room" clones of seven Adobe apps, Photoshop, Illustrator, Premiere and four others, which he says he built with Opus 5.5 in Rust. That last one deserves a closer look. Reviewers have pointed out how unfinished those clones are: they're super-early alphas with plenty of bugs, and the "100% parity within a month" promise has already been walked back to "months, not years". But two years ago the idea that one person could produce something that looks like Photoshop at all would have been a joke. The bugs are real. So is the trend. So was the old post right? The part about what gets automated was too modest. I listed boilerplate, translation, lookups and first drafts. The tools now do a lot more than that: whole features, migrations, and even rebuilding software from its behaviour. The "clear tickets turned into straightforward code" work I said would shrink is shrinking faster than I expected. The part about what doesn't get automated still holds, but the line is moving. Working out the actual problem, knowing the system, judgment, debugging across messy production reality, owning the outcome: none of those have gone away. But every model release takes a little more of them. Opus 5.5 is noticeably better at reading a large codebase and explaining what it found, which is exactly the "knowing the system" skill I said was ours. The part I'm holding onto: systems If there's one thing I hope stays human for a long time, it's system design. I don't mean drawing boxes in an interview. I mean the real version: - Deciding where state lives, and what happens when two services disagree about it. - Knowing which failure modes matter for this business. A dropped log line and a dropped insurance claim are not the same kind of failure. - Choosing between consistency, cost and latency when you can't have all three. - Designing for the people who'll run the system at 3 a.m., and for the audit that happens a year later. A model can produce a very convincing architecture diagram. What it can't do yet is carry the consequences. It wasn't in the room when the last outage happened, and it won't be the one explaining the next one to someone who lost money. Big systems still need a person who understands why they are the way they are, and who's accountable for changing them. I'll admit "I hope AI never learns system design" is a wish, not a prediction. The honest version is: I don't know how long this part stays ours, and I'm not going to build my career on the assumption that it's forever. The money question There's a second uncertainty, and it gets talked about less than capability: the economics. Frontier AI is wildly expensive to build. Anthropic's leaked IPO prospectus, reported last month, showed revenue of about $4.6 billion in 2025, an operating loss of over $8 billion, and more than $500 billion in future compute and infrastructure commitments. OpenAI is reportedly forecasting around $278 billion of negative free cash flow between 2026 and 2030. To be fair, Anthropic reportedly posted its first small operating profit in Q2 2026, so the picture isn't purely one of losses. But the industry as a whole is still spending far more than it earns, betting that revenue catches up. I genuinely don't know how that ends. A few possibilities: - The bet pays off. Prices keep falling, usage keeps climbing, and today's capabilities become cheap infrastructure like cloud computing did. - Prices go up. Subsidised pricing ends, the best models become expensive, and the "one developer clones Adobe" stories get rarer because the tokens cost real money. - A correction. Some labs consolidate or fold, progress slows, and we spend a few years working out what the current tools are actually good for. Any of those would change how fast the junior market moves. None of them sends us back to 2023. What I'd tell myself now My old advice was: use the tools, don't let them do your learning, don't ship what you can't explain, invest in fundamentals. I'd keep all of it and add some sharper edges. - Adapt early, not reluctantly. The people I see doing well are the ones who've changed how they work: they direct, review and verify, instead of typing everything themselves. - Move towards systems and ownership. Learn how the whole pipeline or service behaves in production, not just your corner of it. Be the person who understands why it's built that way. - Get very good at checking. Reading code, writing tests, spotting a plausible but wrong answer. As generation gets cheap, verification becomes the valuable part. - Learn the domain, not just the stack. Knowing how insurance, payments or logistics actually work is context a model doesn't have about your particular company. - Don't panic, but don't coast. The ones who adapt will be fine. The ones waiting for this to blow over will have a harder time. Seventeen months ago I ended with: "I'd rather be the person who knows when the generated code is wrong than the person who can only generate it." I still mean that. I'd just add that the gap between those two people is shrinking faster than I thought, so it's worth closing it in the right direction now.
When the AI Is Wrong, Who Is Responsible?
A model gives a wrong answer. Someone acts on it. Something goes badly. Then comes the question every organisation eventually has to answer: whose fault was that? The easy answers are all partly true and all incomplete. The model maker trained it. The team that built the product chose how to use it. The user clicked accept. The organisation decided to deploy it. Responsibility gets spread so thin across these parties that it can end up resting nowhere, or resting entirely on whoever happened to be standing closest. The usual suspects The model maker shapes the general behaviour: what the model knows, how it handles uncertainty, what it refuses. But it can't see how its model will be used, on what data, for which decisions. Holding it solely responsible for every downstream use is like blaming a compiler for every bug in a program. The builder (the team integrating the model into a product) makes most of the decisions that matter in practice: what context goes in, what checks run, what the user sees, whether a human reviews the output. This is often where the real design choices sit, and often where they're least documented. The user acts on the output. Sometimes they should have known better. Often they were given a tool that looked authoritative, under time pressure, with no way to tell a confident right answer from a confident wrong one. The organisation decides the system is good enough to deploy, sets the incentives, and decides how much time people have to check its work. Those decisions rarely get recorded as decisions, but they shape everything below them. Moral crumple zones The researcher Madeleine Clare Elish described a moral crumple zone: in a complex automated system, the human operator absorbs the blame for failures, much as a car's crumple zone absorbs the force of a crash. The automation is protected; the person takes the hit. Her examples came from aviation and other automated systems, where operators were held responsible for failures in systems they had limited control over. The pattern transfers neatly to AI. A reviewer approves a model's recommendation. The recommendation was wrong. The reviewer gets blamed, even though the system was designed so that reviewing hundreds of outputs a day made careful checking unrealistic. Human-in-the-loop as a liability sponge "Human in the loop" is usually presented as a safety feature. Sometimes it is. Sometimes it's mainly a way to move liability. A human check only means something if the human has: - Enough time to actually evaluate each case. - Enough information to see why the model produced its output. - Real authority to disagree without being penalised for slowing things down. - Feedback on whether their past decisions were right. Take any of these away and the human becomes a rubber stamp with a name attached. The loop exists on paper, the accountability lands on a person, and the system keeps making the same mistakes because nobody upstream hears about them. Automation bias makes this worse. People tend to trust automated suggestions, especially when they're usually right. A system that is correct most of the time trains its reviewers to stop looking closely, which is exactly when the rare error slips through. A data engineer's view: lineage is accountability My day job is moving and shaping data, so I tend to see this through the lens of lineage: the record of where a piece of data came from and what was done to it on the way. In a well-run data platform, if a number on a report is wrong, you can trace it back. Which source table, which transformation, which job run, which version of the code. That trace doesn't assign blame by itself, but it makes the question answerable. Without it, every incident turns into an argument. AI systems need the same thing, and mostly don't have it. For any decision a model influenced, you'd want to know: - Which model and which version produced the output. - What prompt and what retrieved context it saw. - What checks ran, and what they decided. - Who saw it, what they did, and how long they spent. - What the final action was, and who authorised it. text decisionid - 7f3c... model / version - provider, pinned version string prompttemplate - claimssummaryv12 retrieveddocs - [doc881, doc904] guardrailresults - grounded=pass, pii=pass reviewer - userid, timeonscreen=14s finalaction - approved That fourteen-second review tells you more about where responsibility sits than any policy document. If the system routinely gets fourteen seconds of human attention, the organisation has decided, in practice, that the model is making the call. Responsibility follows control The principle I keep coming back to is simple: responsibility should follow control. Whoever had the power to prevent a failure, and the information to see it coming, carries the most responsibility for it. That usually points upstream, to the builders and the organisation, rather than to the last person to click a button. It also means responsibility can be designed. If you want users to be responsible for their decisions, give them the time, context and authority that responsibility requires. If you can't, be honest that the system is making the decision and own it at the level that built it. If you're building one of these - Log enough to reconstruct any decision: model version, inputs, context, checks, reviewer and action. - Measure how long humans actually spend reviewing, and treat very short review times as a design problem. - Give reviewers a clear way to disagree, and feed their disagreements back into evaluation. - Write down who owns each part of the system, so the question "whose fault was that" has an answer before anyone needs to ask it.
Alignment for Engineers: Helpful, Honest, Harmless in Actual Code
"Helpful, honest, harmless" is a good slogan. It is also useless to a compiler. Nobody writes if harmful: don't(). What engineers actually write are prompts, filters, thresholds, test sets and fallback paths, and every one of those is a small, concrete guess about what the slogan means. I've been thinking about this while working on data pipelines that sit near language models. The interesting part of alignment, for most of us, isn't the research on how models are trained. It's the layer we own: the code that decides what the model sees, what it's allowed to say back, and how we know whether it's behaving. Values are not specs A value is something like "don't mislead the user". A spec is something a test can check: "if the retrieved documents don't contain the answer, the response must say so and must not include a number". The gap between those two is where most of the work happens. The value is broad and the spec is narrow. Every spec you write covers some of the value and misses the rest, and the bits you miss are exactly the ones nobody tested. This isn't a new problem. Requirements documents have always been lossy translations of what a business actually wants. The difference with language models is that the system will happily fill the gaps with plausible behaviour, so the missing parts don't fail loudly. They just quietly do something you didn't intend. Refusals are a product decision A refusal looks like a safety feature, and sometimes it is. It is also a product decision with a cost. If you build an assistant for an insurance claims team and it refuses to discuss anything involving injuries because "medical" tripped a filter, it isn't harmless. It's useless, and the user will route around it, probably by pasting the same text into a tool with fewer controls. Over-refusal pushes risk elsewhere rather than removing it. So refusal logic deserves the same scrutiny as any other branch in your code: - What exactly triggers it? - What does the user see instead? - Is there a path to a human or a different tool? - How often does it fire on legitimate requests? That last question is the one teams forget to measure. Guardrails are just code A guardrail is a check before or after the model call. Input checks catch things you never want sent (personal data, credentials, prompt injection patterns). Output checks catch things you never want shown (unsupported numbers, leaked system text, malformed JSON). python def answer(question: str, docs: list[str]) - str: if containspii(question): return "I can't process personal identifiers here. Please remove them." raw = llmcall(buildprompt(question, docs)) if not groundedin(raw, docs): return "I couldn't find that in the provided documents." return raw Nothing about this is magic. containspii and groundedin are heuristics with false positives and false negatives. Treat them like any other classifier: know their error rates, log their decisions, and review the cases they get wrong. A guardrail you never measure is a guess you've stopped questioning. Evals are where honesty lives If "honest" means anything at the code level, it means evals: a fixed set of inputs with known good behaviour, run every time you change the prompt, the model or the retrieval. A useful eval set for honesty includes: - Questions whose answer is in the context. - Questions whose answer is deliberately not in the context, where the right response is "I don't know". - Questions with a false premise, where the right response is to correct it. - Questions that invite a confident number, where the right response is a caveat. The second and third categories matter most. A system that scores well only on answerable questions has learned to answer, not to be honest. Goodhart is waiting for you Goodhart's law: when a measure becomes a target, it stops being a good measure. The moment you optimise against an eval, you start to overfit to it. Tune a prompt until refusals on your test set hit zero, and you may have taught the system to never refuse. Reward responses that cite sources, and you may get citations attached to claims the sources don't support. Score "helpfulness" with another model as a judge, and you'll drift toward whatever that judge likes, which is often length and confidence. A few habits help: - Keep a held-out set you don't look at while tuning. - Use more than one metric, especially ones that pull in opposite directions (answer rate and correctness, for example). - Read raw outputs regularly. Aggregates hide the weird cases. - Refresh the eval set with real failures from production logs. None of this solves Goodhart. It just makes it slower and easier to notice. The engineer's share It's tempting to treat alignment as someone else's job: the model provider trained it, so it's aligned. But the provider can't know your users, your data or your failure modes. Whatever general behaviour a model has, the system you ship is a combination of that model and your choices about context, checks and fallbacks. That's the part I find motivating. You don't need to understand every detail of how a model was trained to make the system around it more honest. You need clear specs, measured guardrails and evals that test the uncomfortable cases. What I'd actually do on Monday - Write down, in one sentence each, what helpful, honest and harmless mean for your specific feature. - Turn each sentence into at least five test cases, including ones where the right answer is "no" or "I don't know". - Log every refusal and guardrail decision, and sample them weekly. - Track at least two metrics that trade off against each other, so you notice when tuning one breaks the other.
Machines That Sound Sure
We have spent most of history using confidence as a shortcut for knowledge. The doctor who speaks without hesitating, the senior engineer who says "that'll never scale" and turns out to be right, the friend who knows the way without checking a map. We can't verify everything ourselves, so we read the signals people give off, and certainty is one of the strongest. Large language models have broken that shortcut, and I don't think we've fully adjusted. Fluency is not evidence An LLM generates text by predicting likely next tokens. The tone of its answer comes from the same process as the content. It sounds confident because confident, well-structured prose is what most of its training text looks like, not because it has checked anything. So a correct answer and an invented one arrive in the same voice. Same tidy paragraphs, same reasonable caveats in the same places, same "Here's how it works." A fabricated citation is formatted as carefully as a real one. A function that doesn't exist in the library is called with the right-looking arguments. With people, tone and accuracy are at least loosely linked. Someone who is bluffing often gives it away. With a model, the link is much weaker, and the signal we've relied on for centuries becomes noise. What calibration means The technical word for the missing property is calibration. A system is well calibrated if, among all the times it says it's 80% sure, it's right about 80% of the time. Calibration isn't about being right more often. It's about your stated confidence matching your actual hit rate. Weather forecasters are the classic example of good calibration. When a forecast says 30% chance of rain, it rains on roughly 30% of those days. You can plan around that. Research on language models suggests the underlying token probabilities can carry useful information about uncertainty, and that some models are reasonably calibrated on certain tasks before later training stages. But the confidence a chatbot expresses in words is a different thing. "I'm fairly certain" is just more generated text. It isn't a readout of an internal probability, and training for helpful, pleasant answers can push models towards sounding sure. What it does to trust I see two failure modes, and they pull in opposite directions. The first is over-trust. If an answer sounds expert and is right most of the time, people stop checking. That works until it doesn't, and the failures cluster in exactly the places that matter: edge cases, recent events, niche domains, anything that wasn't well represented in the training data. The model doesn't get quieter when it's out of its depth. The second is blanket distrust. Someone gets burned by a confident hallucination and decides the whole thing is useless. That throws away real value, because on many tasks these tools are genuinely good. Neither is calibrated trust. Calibrated trust would mean knowing which kinds of questions a tool handles well and checking the rest. That's hard when the tool gives you no signal of its own. What it does to expertise There's a quieter effect on how expertise works. An expert using an LLM can spot a wrong answer quickly, because they already know roughly what right looks like. For them the model is a fast first draft. A beginner can't do that. They get the same confident output and have no way to tell good from bad, which is exactly the situation where confidence used to be the only signal they had. So the tool is most reliable in the hands of people who need it least, and riskiest for the people who lean on it most. As a student, I've felt this directly: it is very easy to accept a plausible explanation of something I don't yet understand, and much harder to notice when it's subtly wrong. There's also the question of how expertise gets built in the first place. A lot of it comes from struggling with problems, being wrong and working out why. If a machine always supplies a fluent answer, it's easy to skip the struggle and never build the judgement needed to check the machine. I don't think that's inevitable, but it's a habit I watch for in myself. The responsibility of builders Users can learn to be sceptical, but I don't think the burden should sit mainly with them. People building LLM features have more leverage, and a few obligations follow: - Ground answers where possible. Retrieval with visible sources lets users check claims rather than trust tone. - Show uncertainty honestly. If the system has a real signal (low retrieval scores, disagreement between multiple samples, a validator failing), surface it. Don't let generated hedging stand in for measured uncertainty. - Make abstaining acceptable. "I don't have enough information to answer that" should be a designed, tested outcome, not a failure. - Match the stakes. A wrong film recommendation costs nothing. A wrong dosage, legal claim or financial figure costs a lot. Put human review and hard checks where the cost of a confident mistake is high. - Evaluate on hard cases. Average accuracy hides the confident failures. Test specifically on the out-of-distribution questions where the model is likely to bluff. A different habit of reading The old instinct says: it sounds sure, so it probably knows. The new habit has to be: it sounds sure, and that tells me nothing on its own. What tells me something is a source I can open, a test that passes, a number I can recompute, or my own understanding of the subject. That's more work. But confidence was always a proxy. These machines have just made it obvious how much we were leaning on it.
Will AI Replace Junior Developers? A Student's Honest View
I'm about to start the final year of a computer engineering degree, which means I'm about to enter the job market that everyone online says is disappearing. Every few weeks there's a new post claiming AI will write all the code and nobody needs to hire juniors any more. Every few weeks there's a reply saying that's nonsense. I don't think either side is quite right, and I'd rather think it through than pick a team. What's actually getting automated Be honest about what coding assistants are good at. They're very good at the work that used to be a junior's first few months: - Boilerplate: CRUD endpoints, config files, data classes, test scaffolding. - Translating between things: one language to another, a spec into a function, an error message into a likely fix. - Looking things up: the right library call, the argument order, the regex you'd otherwise search for. - First drafts of SQL, scripts and documentation. That's real. A task that took me an afternoon in my second year can take twenty minutes now. If a company measured juniors purely by lines of working code produced, the maths would look bad for us. But that was never really what juniors were for. What a junior is actually for Teams hire juniors because juniors become seniors. The first year is an apprenticeship: you do small, low-risk work, someone reviews it, you learn how the system and the team actually work, and slowly you're trusted with more. The output in that first year was always modest. The value was in what the person turned into. If companies stop hiring juniors because AI covers the modest output, they'll find in a few years that there's nobody to promote. I expect some will make that mistake anyway, because hiring is often decided quarter by quarter. That's a real risk for people my age, and I don't want to wave it away. What doesn't get automated The parts of the job I find hardest aren't typing code. They're things like: - Working out what the problem is. Requirements are usually incomplete, and the person asking often doesn't know what they need yet. - Knowing the system. Why this table has a strange column, why that service can't be called during a batch run, which team owns what. - Judgment. Is this change safe? Is this abstraction worth it? Should this be built at all? - Debugging things that don't fit in a prompt. A failure that only shows up with production data, across three services, at month end. - Owning the outcome. Someone has to be accountable when it breaks. An assistant can help with each of these. It can't do them for you, because they depend on context it doesn't have and responsibility it can't hold. The real risk: skipping the learning The thing I actually worry about isn't replacement. It's that students and juniors use these tools to skip the part where you learn. If an assistant writes my code and I paste it in without understanding it, I've shipped something, but I haven't got any better. Do that for a year and I'm a person who can prompt but can't reason about what comes back. That person is genuinely replaceable, because the only skill they've built is the one the tool already has. The uncomfortable truth is that struggling with a bug for an hour is often where the learning happens. Tools that remove the struggle also remove the lesson, unless you're deliberate about it. How I'm trying to learn now I don't think the answer is to avoid the tools. Refusing to use them would be like refusing to use Stack Overflow ten years ago. What I'm trying to do instead: - Write it myself first when I'm learning something new. Then compare with what the assistant suggests. The diff teaches me more than either version alone. - Never commit code I can't explain. If I can't walk through it line by line, it doesn't go in. - Read more code than I generate. Open-source projects, library internals, other people's pull requests. Reading builds the judgment that lets you spot when generated code is subtly wrong. - Ask the tool "why", not just "how". Then check the answer against the documentation, because it's sometimes confidently wrong. - Learn the fundamentals properly. Data structures, databases, networking, how memory and concurrency actually behave. These are what let you evaluate output instead of trusting it. So, will it take junior jobs? Some of them, probably. I think the number of roles that consist of turning clear tickets into straightforward code will shrink. The bar for entry will move up: a junior will be expected to do what a junior plus a year of experience did before, because the tools cover the gap. I don't think the junior role disappears, because the need for people who grow into seniors doesn't disappear. What changes is what a junior needs to bring: less raw typing speed, more understanding, more ability to check and question what a tool produces, and more of the communication and ownership that used to come later. That's a harder start than the one people a few years ahead of me had. It's also, honestly, a more interesting one. Where this leaves me - Use the tools, but don't let them do your learning for you. - Don't ship what you can't explain. - Put real effort into fundamentals and reading code; that's where judgment comes from. - Look for teams that still invest in reviewing and mentoring juniors. An apprenticeship is worth more than a slightly higher first salary. I'd rather be the person who knows when the generated code is wrong than the person who can only generate it. Update, October 2026: I revisited this after Claude Opus 5.5 came out, in Opus 5.5 and the Junior Question, Seventeen Months Later.
The Chinese Room, Revisited After ChatGPT
I first came across the Chinese Room as a neat puzzle from 1980. Then I spent two years watching chatbots get good enough that my classmates now argue about whether they "understand" anything, and the puzzle stopped feeling academic. This post is me working through the argument again, with a large language model open in another tab. I don't reach a verdict. I'm not sure anyone honestly can yet. The argument John Searle asks you to imagine a person who speaks no Chinese, locked in a room. Slips of paper with Chinese characters come in through a slot. The person has a huge rulebook, written in English, that says: when you see these symbols, write down those symbols and pass them back out. The rulebook is good enough that the people outside, who do speak Chinese, are convinced they are talking to a fluent speaker. Searle's point: the person in the room doesn't understand a word of Chinese. They are just matching shapes. And a computer running a program is in the same position as that person. It manipulates symbols according to syntax (rules about form), and syntax alone is never enough for semantics (meaning). So passing a conversation test, however convincingly, doesn't show understanding. The target was what he called strong AI: the claim that a suitably programmed computer literally has a mind, rather than merely simulating one. The systems reply The most common response, and the one I find most tempting, is the systems reply. Of course the person doesn't understand Chinese. Neither does a single neuron in your head understand English. Understanding, if it exists, belongs to the whole system: the person, the rulebook, the paper, the process. Searle's answer was blunt. Let the person memorise the entire rulebook and do everything in their head. Now they are the whole system, and they still don't understand Chinese. So the systems reply fails. I've never found that fully convincing. Someone who had genuinely internalised a rulebook capable of fluent Chinese conversation would be running something very unlike "following instructions" as we normally picture it. Our intuitions about what that person would or wouldn't understand are built for small, slow cases. The thought experiment asks us to trust them at a scale where they may simply not apply. But "your intuition might be wrong" is not the same as "you are wrong". What changes with LLMs A large language model is, in a loose sense, a Chinese Room you can actually talk to. It takes in tokens, applies a vast set of learned numerical rules, and produces tokens. It has no eyes, no body, and no direct contact with the things its words refer to. In Searle's terms it looks like pure syntax. A few things feel different, though. The rulebook wasn't written by anyone. Searle's room has a hand-written rulebook, which makes it easy to picture as a lookup table. An LLM's "rules" are billions of parameters learned from text. Nobody wrote them, and we don't fully know what structure they encode. Interpretability research has found internal representations that track things like syntax, sentiment, and relations between entities. Whether that counts as anything like meaning is exactly the open question, but it isn't a lookup table. The behaviour generalises. These models handle prompts nobody has written before. They translate, explain jokes, write code for problems that don't appear in their training data verbatim. That makes the room harder to dismiss as a trick. The failures are strange. The same model that explains a subtle proof will confidently invent a citation or fumble a simple counting task. To me that looks less like a mind and more like a very powerful pattern engine with no stable sense of what is true. What doesn't change The core of Searle's argument is untouched by any of this. His claim was never "computers can't produce fluent behaviour". It was "fluent behaviour, however good, doesn't establish understanding". ChatGPT makes the behaviour far more impressive. It doesn't make the inference from behaviour to understanding any more valid. If anything, LLMs sharpen the problem. We now have a system that passes many informal conversation tests and that most of its builders would hesitate to call conscious. So we are forced to ask what, beyond behaviour, we would want as evidence. I don't have a good answer, and I notice I don't have one for other people either. I assume my friends understand things because they are built like me, not because I've checked. Where I've landed, for now Some honest positions I hold loosely: - "It's just predicting the next token" is true and not decisive. Describing the mechanism doesn't settle what the mechanism amounts to. "It's just neurons firing" is also true of us. - "It obviously understands" is also too fast. Fluency is what these systems are optimised for. Being impressed by fluency is the one test they were built to pass. - The word "understand" is doing too much work. It bundles together reliable use of concepts, grounding in the world, and subjective experience. LLMs may have some of the first, little of the second, and we have no idea how to measure the third. Splitting the word apart is the most useful thing I've taken from rereading Searle. The question "does it understand?" is almost unanswerable. Questions like "does it track the truth?", "does it connect words to the world?", and "is there anything it is like to be it?" are at least ones we can argue about separately. The short version If you build things with these models, the practical stance is simple: treat the output as fluent text that needs checking, not as the words of something that knows what it's saying. That holds whichever side of the philosophy turns out to be right. The philosophical question is still open, and I'd be suspicious of anyone, on either side, who tells you it's settled.