ROB ISTA . ORG

Personal Thoughts And Misfires

You already have a private test for whether a machine is truly intelligent. You may not be able to state it, but you will know the instant a machine passes it, because at that moment you will quietly decide that the test was never the real one. This is not a flaw in your reasoning. It is one of the most human things about you, and it is the reason artificial general intelligence may be the first revolution in history that people refuse to notice while standing directly inside it.

Last year we argued that the true obstacle to AGI had stopped being technical and become philosophical. Every advance follows the same small ceremony. A machine does something we said required intelligence, and within weeks that something is demoted. Chess was reduced to brute-force calculation, language to statistics, Go to optimization, scientific discovery to pattern matching. The programs kept improving. What kept changing was us. The technology barely moved overnight. We changed the definition.

Pause on how strange that is. In every other science we fix the target before the test. A chemist does not declare a reaction meaningless because it went as predicted. Imagine a physicist announcing that gravity no longer counts now that the prediction came true. Yet in artificial intelligence we have made a habit of naming the goal, watching a machine reach it, and then explaining, with great confidence, why that goal never counted. When success is met by moving the finish line, we are no longer measuring the machine. We are managing our own discomfort.

And it is worth asking what that discomfort actually is, because we were never troubled when machines outran our bodies. No one felt diminished by the crane, humiliated by the telescope, or threatened by the pocket calculator. We handed those tasks over gratefully. The unease begins only when a machine reaches toward the one capacity we treated as proof that we were special. For most of human history, intelligence was the story we told about why we sat at the top. A rival to that story is not experienced as a technical event. It is experienced as a loss.

It does not help that nobody can say clearly what is being lost. Ask ten psychologists what intelligence is and you will get ten answers: reasoning, memory, creativity, planning, common sense, adaptability, self-awareness. We have argued about it for more than a century without agreement. Neuroscientists still cannot explain how billions of neurons produce a single conscious experience. We are asking machines to satisfy a definition we have never managed to establish for ourselves, and then treating their failure to satisfy it as decisive.

Human intelligence was never one thing anyway. The brain is not a single magnificent processor but a federation of specialists, with language, vision, memory and planning distributed across networks that still puzzle the people who study them. Modern AI is drifting toward a similar shape: specialised models joined through memory, search, planning and verification. The remarkable thing is not that intelligence emerges from such cooperation. The remarkable thing is that we insist an artificial mind must emerge in some entirely different way before we will consent to call it one.

Watch how the loss distorts our judgement, because the distortion is strikingly consistent. Consider how forgiving we are of each other. People lie every day. Politicians reshape the truth, experts defend positions they later abandon, witnesses remember events that never happened, honest friends misremember last week. We absorb all of it and never conclude that human beings cannot be trusted. We simply say people are people. Then a language model produces one fluent paragraph that turns out to be wrong, and we ask whether the whole technology can ever be relied upon.

We judge one another against the average human and we judge machines against perfection. No person alive could survive the standard we casually apply to software. The same reflex sits behind our fear of self-driving cars. Human drivers kill well over a million people a year, and we keep driving, because we have quietly accepted that people are tired, distracted and occasionally reckless. Yet a single autonomous crash can dominate the news for a week, even as the wider evidence increasingly suggests these systems can lower accident rates in many ordinary conditions. Perfection has quietly become the admission fee for machines. It has never been asked of us.

Creativity is where we retreat when the other defences fall. Surely, we tell ourselves, a machine can calculate but it cannot create. It is a comforting belief and a fragile one. Shakespeare borrowed his plots, Picasso borrowed his styles, and every scientist has stood on shoulders that stood on other shoulders still. Creativity has always been the art of recombining what already exists into something unexpectedly worth having. When a machine does something recognisably similar, we insist it does not count, and once again the objection reveals more about what we need to believe than about what the machine has done.

 

Alan Turing saw this trap three quarters of a century ago. He did not offer his test because imitation proves a mind. He offered it because we have no instrument that measures a mind directly, not even in each other. You infer that the people around you are conscious; you have never verified it. You extend them that generosity every day and think nothing of it, then withhold it, on principle, from anything else. Ray Kurzweil noticed a related pattern. Whatever one makes of his exact dates, the milestones experts called impossible keep arriving early, and yesterday's impossibility has a habit of becoming this year's ordinary tool.

So the honest question is no longer whether machines are changing. That argument is over. The question is what we will do as each remaining line is crossed. Watch where the definition of intelligence goes when it is pressed. It becomes consciousness, then wisdom, then embodiment, then mortality, and finally the possession of a soul. Each new requirement drifts a little further from anything we could test and a little deeper into what we simply wish to believe. There is nothing wrong with reaching for philosophy. But we should at least notice the moment we stopped doing science and began defending ourselves.

Perhaps AGI will never arrive with a headline. Perhaps it will be like electricity or the internet, doubted and belittled right up until the world had quietly rearranged itself around it, so that later generations wonder how we failed to see what was plainly happening. The real threshold will not be the day a machine out-reasons us. It will be the day when continuing to deny it takes more effort than admitting it. And if you feel the urge to move the goalposts one final time as you finish this sentence, sit with that urge for a moment. It may be the most revealing evidence in the whole argument, and it is not evidence about the machines. It is evidence about you.

 

Further reading

Alan Turing (1950), Computing Machinery and Intelligence — https://academic.oup.com/mind/article/LIX/236/433/986238

Ray Kurzweil (2005), The Singularity Is Near, and (2024) The Singularity Is Nearer — https://www.thesingularityisnear.com/

Daniel Kahneman (2011), Thinking, Fast and Slow — https://us.macmillan.com/books/9780374533557/thinkingfastandslow

Herbert Simon (1969), The Sciences of the Artificial — https://mitpress.mit.edu/9780262690232/the-sciences-of-the-artificial/

Douglas Hofstadter (1979), Gödel, Escher, Bach — https://www.basicbooks.com/titles/douglas-r-hofstadter/godel-escher-bach/9780465026562/

David Chalmers (1996), The Conscious Mind — https://global.oup.com/academic/product/the-conscious-mind-9780195117899

John Searle (1980), Minds, Brains and Programs (the Chinese Room) — https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/minds-brains-and-programs/DC644B47A4299C637C89772FACC2706A

François Chollet (2019), On the Measure of Intelligence (the ARC benchmark) — https://arxiv.org/abs/1911.01547

Silver, Hassabis et al., AlphaGo and AlphaZero — https://deepmind.google/research/breakthroughs/alphago/

Jumper et al. (2021), AlphaFold and protein structure prediction — https://www.nature.com/articles/s41586-021-03819-2