What it is
Modern AI is mostly machine learning: instead of writing rules, engineers show a model many examples and adjust its internal weights until its outputs improve. Neural networks, loosely inspired by neurons, are layers of these adjustable connections.
Two ideas carry most of today’s progress. Backpropagation (1986) made it practical to train deep networks, and the transformer architecture (2017) made it possible to train them on vast amounts of text. Language models such as Claude are transformers.
How a neural network passes a signal
A small network showing inputs, hidden layers and outputs, with a signal moving through the weights.
Source: Nature
A model to explain the idea; not drawn to scale.
Ideas to carry
- 1
Learning, not programming
The behaviour comes from data and training, not from hand-written rules.
- 2
Attention
Transformers let every word weigh every other word when building meaning.
- 3
Scale is not understanding
Bigger models do more, but how much they understand is still debated.
Established What we know
- Rumelhart, Hinton and Williams showed in 1986 how back-propagating errors can train multi-layer networks.[1]
- The transformer architecture was introduced in the 2017 paper “Attention Is All You Need”.[2]
- The 2024 Nobel Prize in Physics went to John Hopfield and Geoffrey Hinton for foundational discoveries enabling machine learning with artificial neural networks.[3]
Interpretation The older lens · Jewish folklore
Jewish folklore’s golem is a made servant that does exactly what it is told, which is why the instructions matter.
Reading
- 1986Learning representations by back-propagating errorsRumelhart, Hinton and Williams
- 2017Attention Is All You NeedVaswani et al.
- 2019Artificial Intelligence: A Guide for Thinking HumansMelanie Mitchell
Neural network after 10 billion examples
“Is this a cat?” — 97% sure it is a fluffy towel
Connected subjects, projects and writing
- EvolutionSubject · Life
Established Artificial neural networks were loosely inspired by neurons, and evolutionary search inspired optimisation methods.
- PerceptionSubject · Origin
Interpretation Predictive-processing accounts of the brain and machine learning both describe models that improve by reducing prediction error.
- ConsciousnessSubject · Origin
Perspective Whether systems that process information like this could ever have experience is an open question.
- Agents and automationSubject · Intelligence
Software that plans and acts with tools, and where a human should stay in the loop.
- Computing and informationSubject · Intelligence
From Turing’s machine and Shannon’s bit to the stack of chips, networks and code underneath everything.
- Private AI that runs at homeExperiment · Intelligence
Small open models running entirely on my own machines: an offline chat in the browser, and a home assistant that never leaves the network.
Sources
- Learning representations by back-propagating errors Nature, 1986. Checked 5 Nov 2026.
- Attention Is All You Need arXiv, 2017. Checked 5 Nov 2026.
- The Nobel Prize in Physics 2024 Nobel Prize, 2024. Checked 5 Nov 2026.