Your local study space

Understand AI.
By building it.

Hands-on exercises where you train models yourself, right in your browser – from your first tokens to transformers, AI safety and interpretability. Nothing to install, and no programming required.

Free start · no account · runs in your browser
See what you'll learn

0 tegn → 0 tokens

The same idea as in the first lab: the most frequent neighbouring pairs are merged, one at a time. This is a teaching model, not a fully trained tokenizer.

How it works

Error –

Touch the model

Change text, weights and settings, and watch the model respond instantly.

Step 0 · Error –

See what happens inside

Tokens, loss, attention and activations are shown while the model trains.

Keep what you discover

Write observations in your experiment log and compare your experiments.

Small steps. Real understanding.

From your first token to your own research.

14 labs in four tracks. You start for free and can unlock the rest whenever you like.

Start here

Build the foundations

Language models from the inside

AI safety & interpretability

Start for free. Continue when you're ready.

Free

€0
  • LLMs & tokenisation with your first next-character model
  • Vectors, neurons and your first trained model
  • Learning map and experiment log

Already have an access key?

You have full access.

All labs are unlocked in this browser. Save your access key if you want to use Learn the Machine on another device.

…

Good to know.

Do I need to know how to program?

No. Every lab runs in the browser, and you control it with buttons, sliders and your own text. The Python parts are optional, for when you want to go further.

What do I need?

A recent browser such as Chrome, Edge, Firefox or Safari. It works on a phone, but a computer or tablet gives the best experience in the larger labs.

Where is my data stored?

In your own browser. There is no account, and your notes and models are never sent to us. You can make a backup from the experiment log.

How do I use my access on another device?

After your purchase you get an access key. Redeem it on the new device under “Already have an access key?”.

Can I cancel my purchase?

You get access immediately. When you pay, you therefore agree that the right of withdrawal lapses once delivery has begun. Write to us if something doesn't work, and we'll find a solution.

Is Learn the Machine available in English?

Yes. The whole site is in English, including all the paid labs. Downloadable Python files, example data and some generated reports are still in Danish.

Built for understanding, not just answers. New labs and exercises are added regularly.

LLM / start here · about 20 min.

From text to tokens.

Before a language model can work with text, a tokenizer turns the text into a sequence of ID numbers. Let's find out how.

1 / See the idea · about 5 min.

How can text be split?

A token can be a word or part of a word. Press «Merge next pair» to join the most frequent pair, then try your own text. In this exercise a pair must occur at least twice to be merged. This is a teaching model, not a fully trained LLM tokenizer.

When you change the text, the merges start over. A model trained on the split also starts over.

Round 0

Your split · a dashed border marks what was formed in the latest round. ␠ shows a space, ↵ a line break.

One visible character can take up several bytes · optional

Try the two versions of é. They can look the same but are different sequences of Unicode code points and bytes. A tokenizer may normalise the text first; others keep the difference. In the next step you can compare the two forms with real tokenizers under «Explore languages, Python and token view».

Unicode code points

UTF-8 is one way of encoding text as bytes. Each byte is not necessarily one token; tokenizers can use different rules. Counting visible characters follows the browser's Unicode segmentation when it is available.

See the starting point: text split into characters · optional

␠ shows a space, ↵ a line break.

Choose a specific round · optional

The exercise starts with Unicode code points, which are called characters here. A visible character can consist of several code points. The exercise learns its rules from your text. A real model tokenizer uses rules it has already learned.

Try a new text and explore the rules · optional

Try the rules on a new text

Write a new text. We only use the rules from the training text above and the chosen rounds; the new text does not teach the tokenizer any more rules.

This simple model starts with the characters from the training text. Unknown characters are marked as out-of-vocabulary. The exercise does not compute token IDs; real tokenizers may handle unknown characters differently.

Follow the merges one by one

Choose a round to see how the text is split after that round. The list shows the rounds that are possible with your text, even when the slider is set lower. When counts are tied, the pair that appears first in the text is chosen.

    See the teaching tokenizer's vocabulary

    The vocabulary starts with the distinct characters in the text. Each chosen merge can add a new piece; earlier pieces are kept even if they no longer appear in the text's current split.

    Original characters

    New pieces from chosen merges

    This is a small teaching model without token IDs. A real model tokenizer has its own fixed vocabulary and may work differently.

    How are pairs counted and merged?

    In each round, pairs are counted even when they overlap. The pairs are then merged from left to right without overlap, so fewer may be merged than were counted. Here the merges are learned from exactly the text above. Real model tokenizers have already been trained on far larger amounts of data and apply their fixed vocabulary to new text.

    2 / Compare real tokenizers

    Same text. Different tokens.

    Compare two model tokenizers on the same text. A token ID belongs to the tokenizer's own vocabulary.

    More options · optional

    Qwen3.5 is newer than Qwen2.5, Pythia and GPT-2. BERT is an older WordPiece model with a different training objective from an autoregressive chat model.

    You can guess before comparing, see internal token pieces or try a guided model training.

    The guided example replaces the text above, compares the tokenizers and opens a micro model with a test text ready.

    Runs in your browser. The first time, the selected tokenizer files are loaded from this site. The default choice of Qwen3.5 and GPT-2 is about 14 MB uncompressed together; the actual download depends on compression and cache. The comparison only shows text, not the model's image input.

    Øvelsen klargøres. Hvis den ikke bliver klar, kan du prøve at genindlæse siden.

    The comparison does not automatically add the special tokens a model may use.

    Explore languages, Python and token view · optional
    Do the two versions of é give the same tokens?

    É as one code point and e + accent often look the same but have different bytes. Examine both forms with the two selected tokenizers.

    Compare IDs within the same tokenizer. Identical IDs can result from normalisation or loss of information, for example [UNK]; that does not show what a model understands. No special tokens are added automatically.

    Compare the same greeting in several languages

    Examine the two selected tokenizers on a short greeting in Danish, English and Japanese. Guess first: is the number of tokens the same across the languages?

    The greetings mean roughly the same, but do not have the same characters or length. The examples do not show a whole language's tokenisation or model quality. BERT may replace unknown text with [UNK], so fewer tokens can also mean lost information. A lower token count can affect how much text fits, but says nothing on its own about the quality of the answers.

    Want to run it yourself in Python too?
    1. Use the Python environment from the course, or install with python -m pip install transformers.
    2. Download sammenlign.py and run python sammenlign.py --tekst "Hello world!" in the same folder. The default is Qwen3.5 and GPT-2; the first run downloads tokenizer files, not model weights.
    3. Choose tokenizers-resultat.json below. Python does not read the choices above: use e.g. --foerste qwen --anden bert for other tokenizers. The options are qwen35, qwen, pythia, gpt2 and bert. Use --output another-name.json on the next run; existing files are not overwritten. --offline uses files that have already been downloaded.

    How are tokens and special tokens shown?

    For ByteLevel BPE, pieces are shown as readable text when their bytes alone form valid UTF-8; otherwise byte values are shown. In BERT's WordPiece, ## marks a continuation, and [UNK] can replace unknown text. A character can be split across several tokens. ␠ shows a space, ↵ a line break and ⇥ a tab. Turn on internal pieces to see symbols such as Ġ. The result leaves out automatically added special tokens; if you type a recognised special token yourself, it may still be recognised. The result is therefore not necessarily equal to the token usage of a finished chat prompt.

    Sources: Hugging Face: the parts of a tokenizer and OpenAI: tiktoken and BPE · Hugging Face: BERT and WordPiece.

    3 / Connect to the LLM

    What happens next?

    TextToken IDsEmbeddingsTransformerNext token

    1 / 5 · Text

    A text is the starting point. It could be your message to a language model, for example.

    Same example · text

    This diagram explains the process; it does not run the model itself.

    Optionally compare a text in the previous step to see its first token IDs here.

    See training pairs from the teaching tokenizer · optional

    Here we use the teaching tokenizer's split from step 1, not the model tokenizer from the comparison above. Choose a position: the context is the tokens before the position, and the answer is the next token.

    Context

    Next token · answer from the text

    ␠ shows a space; ↵ shows a line break. This is the teaching tokenizer's split, not the tokenizer from the model comparison. The figure above shows data. In the next step you also train a separate character model on other example texts. Real training windows may have limited context; the first token needs a starting context and is not shown as an answer here.

    Train a model on this split · optional

    This small table model uses only the latest token as input and guesses the next one. It starts with equal probabilities and is trained on roughly the first 75 % of the token pairs; the rest is used for measurement. The vocabulary and merges were made from the whole text, and pairs can repeat on both sides. The validation is therefore not an independent test. It is a bigram model, not a transformer.

    See the loss during training

    The curves show measurements before and after each press of the training button. Lower loss means higher probability for the actual next tokens. The validation pairs are not used to update the weights, but may resemble or repeat the training pairs.

    ━ Training   ┄ Validation · horizontal: training steps · vertical: loss in nats per pair, from zero.

    See the measured numbers
    StepTraining lossValidation loss
    How new are the validation pairs?

    A pair is input token → answer. We count occurrences, including repeats. A new pair can use an input token the model has already trained on.

    The group numbers are descriptive: the groups can have different content and few pairs. A new pair here also comes from the same text, and the tokenisation was made on the whole text. It is not an independent final test.

    The history only exists in this page view and is not included in the experiment file. When reopened, the chart starts with the restored model's current measurement.

    The bars show the distribution for the selected example's latest context token. The answer is marked. The model is reset when the split changes. Download an experiment file below if you want to reopen it. No text is sent away from the browser.

    What can the model see of the context?

    This model only sees the last token before the answer. If the same last token appears after different earlier tokens, the two examples therefore get the same distribution — even when the answer differs. That is a limitation of this table model; a transformer can use several earlier tokens within its context window.

    The bars show the probability of each example's answer on the same 0–100 % scale. Choose another example above, or change the text and the tokenisation. The comparison uses the current model weights.

    Save or reopen the experiment

    Download a file with your text, token split, model weights and number of steps. It is only stored on your device, contains your text and is not encrypted. It is not automatically included in Learn the Machine's other backup.

    When you choose Restore, the file replaces the current text, split and model. Download the current experiment file first if you want to keep it. Files can be edited; their origin cannot be verified.

    Try another text

    We apply the same learned merges and model weights to the text below. See how many of its token pairs also occurred in the model's training pairs. The text can be changed and reused, so it is not an untouched final test.

    A known token pair can be repeated several times and is not an independent observation. The model has no unknown token: if a token is not in its vocabulary, the whole text cannot be measured here. The text is processed locally.

    Check your understanding · optional

    Two occurrences of the same token ID have the same input embedding in the model. What can happen to their representations after a transformer layer?

    You can try again or continue; the answer is not recorded as a passed test.

    Explore embeddings, chat and attention · optional
    How does attention get positional information? · RoPE

    Here a fixed query and key are each rotated according to their position. First try moving both one position forward. Then change only the key position. Compare the dot product, which can feed into an attention score.

    Green, solid line: query. Orange, dashed: key. Each line starts in the centre and ends at the vector.

    The example uses two fixed two-dimensional unit vectors and 30° per position. If you move both equally far, their relative angle and dot product are preserved. If you move only one, it can change; the effect does not always decrease with distance. Also try changing the key's starting direction without moving the positions: the same distance can give a different dot product. The starting direction was chosen for the exercise, not learned from text. Neither softmax nor a causal mask is computed here, so the key can come after the query.

    RoPE rotates the query and key in attention. Real models have many component pairs and frequencies, often other variants too, and the query/key depend on content and context. A shared position shift therefore does not necessarily preserve the whole model's output. The small browser transformer in the transformer lab uses learned position vectors instead; this illustration does not train a model.

    See an actual component pair in Pythia-70M

    The included example uses layer 0, head 0 and the text “The capital of France is”. You can also open a local measurement with your own text and choices.

    Choose token positions visually

    Write the text and use Pythia's tokenizer to find the positions. The text is processed in the browser; the first time, the tokenizer files are downloaded from the site. Choose query and key, and use the numbers in the Python command below. This does not run the language model.

    Forhåndsvisningen klargøres. Hvis den ikke bliver klar, kan du genindlæse siden.

    Repeat locally with a pinned checkpoint: download rope_pythia.py and run python rope_pythia.py --output min-rope.json. The first run downloads the model; add --offline once it is cached. Before choosing positions, see the token numbers with python rope_pythia.py --text "The cat and the cat" --list-tokens. This command only uses the tokenizer and does not load the model weights. Then use the numbers in a measurement, e.g. python rope_pythia.py --text "The cat and the cat" --query 4 --key 1 --layer 1 --head 3 --pair 2 --output mit-forsøg.json. The text must give 2–32 tokens; positions are counted from 0, and the key must not come after the query. Layers are 0–5, heads and component pairs 0–7. Without chosen positions, the last query and a key up to two places earlier are used. The file contains the text, token IDs and measurements; only share it if you choose to. Open it above. Only the same setup is directly comparable with the default example. Files and hashes can be edited.

    Sources: Su et al.: RoFormer ↗ · Hugging Face: RoPE variants ↗.

    See real embeddings from Pythia-70M

    Choose two tokens from a real, open language model. Start with ␠cat and ␠dog, then choose the same token twice. What happens to the similarity?

    Want to see what happens through the model? Start a ready-made experiment with the text The cat and the cat.

    Examine one change in the context

    Here you compare two precomputed runs: A: The cat and the cat and B: The dog and the cat. Only the token at position 2 has changed. The last ␠cat has the same ID and position in both texts. Predict first: is its activation after layer 1 the same?

    Also try comparing the first token, The, in A and B. It comes before the change and cannot see later tokens in this model. The largest component difference summarises all 512 numbers; cosine measures direction. Two examples do not show which internal mechanism causes the difference.

    The example is precomputed from Pythia-70M and is only fetched when you press the button. No installation needed.

    The first 8 of 512 components. The direction of the bar shows the sign; both tokens use the same scale.

    ␠ means a space. ␠cat and cat are different tokens with different IDs. The figure shows an excerpt, not a projection of the whole vector; the similarity is computed on all 512 numbers. Cosine similarity measures how similar the angle is, from −1 to 1, and does not in itself say what the model understands or will answer. The input embedding is a weight before the transformer's context processing. A transformer layer, on the other hand, shows the activation at a position after the layer. It can be affected by earlier tokens and the position; cosine alone does not tell you why two activations differ.

    The six vectors were extracted in advance from Pythia-70M, pinned checkpoint; no model weights are trained or changed here. Token IDs and weights apply to this model, not to the other tokenizers. You can reproduce the extraction: download export_embeddings.py, install PyTorch and Transformers, and run python export_embeddings.py --output pythia-embedding-examples.js. The first run downloads the model. The file only contains the six chosen vectors.

    Examine your own text with Pythia
    1. Download the Python file above, and install PyTorch and Transformers in your Python environment.
    2. Open a terminal in the folder with the file and run python export_embeddings.py --tekst "The cat and the cat" --lag --output mine-embeddings.json. The first time, the model and tokenizer are downloaded. Later, choose your own short text; it must give 2–16 tokens. If the file name already exists, choose another name.
    3. Choose the created JSON file here. Choose the two ␠cat positions above. Compare the input first, then choose “After transformer layer 1”. What happens to the similarity?

    The file and text are processed locally on your computer and are not sent to Learn the Machine. The Python program downloads the model and tokenizer from Hugging Face the first time. The file contains token pieces, IDs and vectors from your text, including layer activations when you use --lag; only share it if you want to share that. Repeated IDs use the same input vector regardless of position. With --lag the text is also run through the model, and the vector after each of the six transformer layers is saved before the model's final normalisation. The model can only use the current and earlier text, not later tokens. Differences cannot be attributed to one particular word or one mechanism alone. The imported token pieces are the tokenizer's internal symbols; ␠ shows Ġ, while other byte symbols cannot necessarily be read as ordinary text. The file describes its origin, but the site cannot verify the authenticity of the weights.

    Try an embedding lookup

    Choose a made-up token ID and see its vector as a point. Move the components and watch how a small output computation changes the next-token distribution.

    120

    A small distribution over two possible next tokens:

    The numbers, token IDs and the two possible answers are constructed for the exercise. A fixed linear output computation and softmax give the distribution; the sliders change a vector by hand, they do not train it. There is no transformer or context from earlier tokens here. A real model has far more dimensions and uses the transformer between lookup and output.

    Want to see how a position can mix information from earlier positions? The next exercise uses other, constructed numbers for query, key and value.

    What does a chat look like to the model?

    A chat template turns messages with roles into a single token sequence. Try a limited example with Qwen2.5-0.5B-Instruct: one system message and one user message, without tools or earlier conversation.

    The tokenizer runs locally in the browser and is loaded the first time you use it.

    Sources: Qwen's pinned chat template · Hugging Face: chat templates.

    4 / Train a small language model · about 5 min.

    Train your first next-token guess

    Here every character is a token. The model sees one character and guesses the next; it is not a transformer. It has its own small vocabulary and does not change the tokenizer or the Pythia model from before.

    Try this: in the training data, k is always followed by a. How do the model's guesses change as you train? (The training sentences are in Danish.)

    Guess before training · optional

    Your guess is assessed after the next training. Less than 0.05 percentage points counts as almost unchanged.

    How does training data change the model? · optional

    Train two new character models from the same zero weights for 20 steps. Model A sees the original three sentences. Choose what kat should be replaced with for model B. Both use 57 training pairs, the same learning rate and the same measurement sentence. Several character pairs may change. The models are lookup tables with one character of context and have no attention heads.

    Guess before training · optional

    Less than 0.05 percentage points counts as almost the same. The guess is not saved and can be given after you have seen the result.

    Others groups the characters that are not shown individually. All observed training answers are shown individually. The percentages are rounded.

    0 training steps · training loss — · validation loss —

    Explore the model or choose another path · optional
    See all training texts · optional

    Training text: “en kat kan se en mus.” · “en mus kan se en kat.” · “kat og mus kan se.” (Danish for “a cat can see a mouse” etc.)

    Write the model's building blocks in Python · optional

    Write softmax and the gradient yourself. A local Python exercise checks your code and trains the same small next-character model for 20 steps. It needs Python 3 but no extra packages.

    New to Python?

    The exercise uses functions, lists and loops. You can take an introduction first and come back here.

    Show softmax and gradient with three characters

    A standalone example with one training pair and three possible next characters. It does not change the model above.

    Softmax converts scores into probabilities. For each character the gradient is probability minus answer: 1 for the correct character, 0 for the others.

    How can I check the gradient?

    Change one score very slightly up and down. The difference in loss divided by the distance gives a numerical approximation of the gradient.

    Here ±0.001 is used on one example. The Python checker uses another small change and more targets. The numbers are shown rounded. A numerical check is an approximation at specific inputs, not proof that the code is always correct.

    The example uses one gradient step with learning rate 0.5. The coding exercise uses more characters and the average over all training pairs. Subtract the largest score before exp for better numerical stability.

    1. Download and unzip the coding exercise ↓.
    2. Fill in the functions in min_model.py and run python tjek_og_traen.py in the folder.
    3. Open the saved naeste-tegn-resultat.json here and compare training and measurement texts with the browser's computation.

    The open checks are not a complete code test. The Python code runs on your computer; only run code you trust. This is one character of context, not a transformer.

    Repeat the data experiment in Python · optional

    Once your functions work, you can train with the same data change as in the browser experiment. The original result file is not overwritten.

    python tjek_og_traen.py --replacement mat

    Also choose kan or kat as an unchanged control. The program saves e.g. naeste-tegn-mat.json; open the file below. The browser recomputes the model for the given choice. The measurement texts overlap the training data and are not untouched final tests.

    More ways to examine the training · optional
    Choose another way forward · optional

    Train on the split of your own text in a separate table model.

    You can train a small text model in the browser or continue to the Python projects. Training a BPE tokenizer locally needs Python and the tokenizers library; the local transformer also needs PyTorch.

    Examine loss and validation · optional
    What is in the validation sentence?

    The validation sentence is used for measurement, not for weight updates. Choose a character pair and see whether the same pair also occurs in training.

    Blue marks the input; orange is the answer. The model only sees one character at a time. A pair appearing in both sets does not prove that the whole sentence was used for training; it shows that the measurement does not test a new character pair for this particular model.

    Measure another sentence

    The example has one new character pair. Measure the text, keep training, and see how the loss changes on the same text. You can edit it with characters from the model's vocabulary.

    Lower loss means the model does better overall on the measured text. The validation sentence shares words and character pairs with the training data; it is not a broad test of language ability.

    Explore training and generation · optional

    Choose a topic if you want to examine the model more closely. You can also continue directly.

    How are the loss and the bars computed?

    The loss is the average of −ln(probability of the observed next character); individual characters can still get a lower probability. The five largest probabilities are shown together with any training answers outside the top 5. When tied, characters are sorted in vocabulary order.

    See the training pairs for this character

    Each pair shows an input and the next character the model is trained to predict. Repeated pairs count each time; no pairs are made across sentences.

    See the next weight update · optional

    The model stores a logit for each possible next character. Here we follow the most common answer for the chosen input. The gradient shows how the total loss changes with a small change to this logit while the others are held fixed. Gradient descent adjusts all logits at once based on the whole training set; the learning rate is 4.

    Green: probability now · orange: after one step. This is a computed preview and does not change the model.

    See how the chosen input contributes to the loss

    Each training pair gives −ln(probability of its answer). Repeated pairs count each time. The table only shows pairs for the chosen input; the contribution is divided by all training pairs.

    Next characterNumber of pairsModel's probability−ln(p) · nats per pairContribution to total loss

    Watch the model write character by character

    Both start with the character chosen above and then pick the most likely next character 32 times. The model only uses the latest character as input.

    Before training:

    After 0 steps:

    ␠ shows a space. When several characters have the same probability, the first in the model's fixed character order is chosen; before training all characters are equally likely. No stop character is used, so the display always continues for 32 characters. The small model can repeat itself and is not a transformer.

    Draw the next character with temperature · optional

    Use the current model and the input chosen above. At temperature 1 and top-p 1, the bars are the model's own distribution. A lower temperature concentrates the distribution on the largest logits; a higher temperature flattens it. This changes the selection, not the model's weights.

    Limit candidates with top-p

    After temperature, characters are sorted by probability. We keep the most likely until their combined chance reaches top-p. The rest get 0 %, and the kept ones are renormalised to 100 %. When tied, we use the model's fixed character order. A top-p of 1 keeps all.

    The candidate set is recomputed for each new input. This does not change the model's weights. Source: Hugging Face on top-p ↗

    Green on top shows the chance after temperature but before top-p. Orange below shows the chance after top-p, which is used for the draw. The numbers follow the same order; the number after the dot shows observed draws. All draws use the same input and are separate — they do not form a connected text. Changing temperature, top-p, input or model resets the count. Up to 10,000 draws can be made per count.

    Generate a chain of characters

    Start with the input chosen above. Each drawn character becomes the input for the next step. Compare with the separate count above, where the input was the same every time.

    See each step
    StepInputDrawn characterChance at this step

    ✓ marks the character that was drawn. Green: probability after temperature and before top-p. Orange: probability after top-p. The distribution belongs to the chosen step in the chain.

    ␠ shows a space. The chain uses the current temperature and top-p and resets if temperature, top-p, input or model change. At most 32 characters are generated; a full stop does not stop the model. This is a small model that only sees one character at a time, not a chat model.

    How does a whole text become training data?

    See the input and answer in one of the text transformer's windows. The view uses characters like this exercise, but the transformer can use several earlier characters as context. You don't need Python to see it.

    Want to see first what an extra character of context changes? Train two small models and compare their guesses.

    Check your understanding · optional

    Try to answer without looking above. You can try again. When both answers are correct, the check is saved in this browser; it does not document overall competence.

    Extra · train a tokenizer · local Python

    Train your own tokenizers

    Train two real ByteLevel BPE tokenizers on the same training texts, but with different target vocabulary sizes. Then examine how they split validation text, which is not used to learn the tokenizer.

    The example comes from two tokenizers trained on small, artificial texts. You don't train anything yourself by opening it. Use the guide below to make your own result.

    You need an activated Python environment. The setup also installs PyTorch, which you will need for the model afterwards.

    1. Download the project package and unzip it. In the same terminal where the environment is activated, change to the unzipped folder: type cd "PATH_TO_FOLDER" and replace the text with the folder's actual path. Use the example file egne-tekster.json or your own file from the dataset builder.
    2. Install the library in your Python environment: python -m pip install "tokenizers>=0.20,<0.22".
    3. If you use the package's example data, run python train_bpe.py --data egne-tekster.json. If you downloaded mine-tekster.json from the dataset builder, move it to the project folder and run python train_bpe.py --data mine-tekster.json. The terminal shows Gemt: followed by the path to the new folder. Open bpe-result.json from that folder below.

    The first 256 slots cover byte values. Very little data and a requirement for repetitions can give an actual vocabulary below the target. A lower token count does not in itself mean a better language model. Tokenizer training is not training of model weights.

    Once you have trained a tokenizer, you can use it in the next experiment.

    Background: Hugging Face: train a BPE tokenizer.

    Foundations / 00 · about 40 min.

    From numbers to a neuron's answer.

    A neural network consists of many simple computations. Start with one: change the bias and watch how the neuron's answer changes.

    First experiment · about 3 min.

    Try your first neuron

    Start here

    A neuron adds up its weighted inputs. The bias is a number added to the sum. Move the bias from 0 to 1 and see what happens to the answer.

    The neuron's output—
    Predict: what happens when the bias goes from 0 to 1?

    First set the bias to 0, choose your guess, then move the bias to 1. You can also skip the prediction.

    Sigmoid turns the sum into a number between 0 and 100 %. That is not automatically a reliable probability.

    Want to try training a model now? You can skip the other neuron steps and come back to them later.

    Next experiment · about 3 min.

    What does a weight do?

    An input is a number the neuron receives. A weight decides the input's contribution: input × weight. Here we follow input x₂ and weight w₂.

    Move the weight from −1 to 0. What happens to the contribution and the neuron's output?

    Output at w₂ = −1—
    Output now—
    Explore more inputs and weights

    Change one number at a time. A vector here is a list of two numbers: the green arrow shows the input, and the orange one shows the weights.

    Input vector x Weight vector w

    See the whole computation step by step
    The computation · step by step

    Follow the numbers through the neuron

    First contribution · x₁ × w₁—
    Second contribution · x₂ × w₂—
    Dot product · x₁w₁ + x₂w₂—
    Add bias · z = x · w + b—
    Sigmoid · σ(z)—

    Sigmoid turns the sum into a number between 0 and 1. In some classification models it is interpreted as a probability estimate, but a number in this range is not automatically a reliable probability.

    From one neuron to a layer · about 15 min.

    Two neurons. One weight matrix.

    The same two inputs go to both neurons. Each connection has its own weight. The rows of the matrix belong to one neuron each; the columns belong to input 1 and 2.

    Try first
    x₁x₂z₁z₂

    Green solid line: positive weight · orange dashed line: negative weight. The thickness shows the weight's size. The numbers on the right are the sums before activation.

    W =
    × x + b = z

    Neuron 1: → ReLU:

    Neuron 2: → ReLU:

    Explore more inputs and weights
    Input
    Weight matrix w · row, column
    More experiments and the computation behind

    Also open «Explore more inputs and weights». Predict what happens to both sums when you change x₁. Reset, and change only w₁₂. Can you make z₁ negative and see what ReLU does?

    With the input as a column we write z = Wx + b. Here W is 2 × 2, x has 2 numbers and the bias is fixed at b = (0.5; −0.5). ReLU gives max(0, z) for each neuron. In PyTorch the batch form is written XWᵀ + b, because the examples are stored as rows.

    This is a separate, manually set forward computation; it is not trained. A real layer can have other dimensions and activation functions. Once the weights are fixed, the same weights are used for every example in a batch.

    Background: PyTorch: linear layer and dimensions ↗ · PyTorch: ReLU ↗ · reviewed 31 October 2026.

    A look backwards · about 10 min.

    How does a weight affect the error?

    Here you try a separate neuron with one input. A known answer is the target; the loss tells you how wrong the neuron's guess is. Choose a target and update the weight once.

    Forward →
    SUM—
    Sigmoid—
    LOG LOSS—

    Explore input, weight and the computation behind

    It computes z = x · w + b, then p = sigmoid(z) and a binary log loss for the target y. When y = 1, the loss is −ln(p); when y = 0, it is −ln(1 − p).

    ← Backward: local slopes
    ∂ loss / ∂ p—
    ∂ p / ∂ z—
    ∂ z / ∂ w—
    ∂ loss / ∂ w—

    Each local slope tells you how one step changes with a small change in its input. The chain rule multiplies them together: that is how we find how the loss changes when the weight changes a little. The numerical check tries small changes to the weight and compares the loss.

    Try it yourself: first choose y = 1 and then y = 0 with x = 1 and w = 0. How does the weight's gradient change sign? Then set x = 0: why can this weight not affect the loss for this particular example?

    The update uses learning rate 0.2 and only changes the weight for one example; the bias is held fixed. In a network with many layers and data points, several gradient contributions are computed and combined. The visualisation computes in the browser; PyTorch uses autograd to compute gradients automatically in the later exercise.

    Background: PyTorch: autograd and the chain rule ↗ · Google: backpropagation ↗ · reviewed 31 October 2026.

    From one computation to a model

    What do the numbers mean?

    01

    Input

    Two measurements or properties are each described by a number. Which properties are used depends on the task.

    02

    Weights and bias

    Weights decide the inputs' contribution to the sum. The bias shifts the sum, even when the input is zero.

    03

    Activation

    Sigmoid turns the sum into a number between 0 and 1. Other neurons can use other activation functions.

    A limited model: In the vector diagram above you set the numbers yourself. The separate backpropagation exercise updates one weight for one example. In the next exercise a model learns its parameters by measuring errors and updating them.

    Sources: Google: weighted sum and sigmoid ↗ · PyTorch: sigmoid ↗ · reviewed 31 October 2026.

    Laboratory / 01

    Watch a model learn.

    Train a small model to find a relationship in data. It really updates its two numbers based on its errors — directly on your device.

    Experiment 01 · real model

    Find the line in the data

    The model tries to find a line that fits the green points. What do you think happens to its error when you train?

    Pick a guess if you like, then train the model.

    Training steps0
    Training error—Green points · used to adjust the model
    Validation error—Orange points · used to measure the model

    Training data Validation data The model's current line

    The orange points are used for validation, not training. The errors are mean squared vertical distances between points and line.

    Explore settings and parameters

    A larger learning rate gives larger changes per step. A rate that is too large can make the error grow. Reset, try 0.05 and then 0.5, and compare the curves. Read about learning rate ↗

    CautiousLarge
    ŷ = 0.00x + 0.00

    ŷ is the model's guess. Weight and bias are the two numbers training changes. The validation points are not an untouched test set, since you can tune against them.

    What happened?

    A small model. A big idea.

    Which of the two errors was used to adjust the model? What can the other one tell you?

    Show a hint

    The model is adjusted using the green training points. The orange validation points measure how it handles other examples in this small dataset; they do not prove that it works on all new data.

    01

    A guess

    When you reset the model, the weight and bias are zero, and the line is horizontal.

    02

    Measure the error

    The distance between the model's guesses and the known answers is used to compute an error.

    03

    Update

    Gradient descent uses the error to choose the direction of a parameter change. A learning rate that is too large can still increase the error. Repeat and measure on validation data.

    This is linear regression on a small synthetic dataset, not a neural network. The same idea of optimising parameters is a building block of deep learning.

    Further reading: Google Machine Learning Crash Course: gradient descent ↗

    Laboratory / 02 · about 25 min.

    When the model learns too much.

    A model can fit the data it has seen and still fail on new examples. Compare three models and examine why.

    Full access

    Continue with the whole laboratory

    This laboratory is part of full access. Here you will work with:

    • Which curve generalises?
    • Try a final test
    • What do the curves tell you?
    • What hides behind the average?

    Foundations / probability · about 10 min.

    Same chance.
    Different results.

    A measurement is based on concrete examples. See why the result can vary even when the underlying chance is unchanged.

    Full access

    Continue with the whole laboratory

    This laboratory is part of full access. Here you will work with:

    • Chance and sample
    • How much can a measurement vary?
    • What does a 95 % interval mean?

    Laboratory / 03 · about 35 min.

    Train a small neural network.

    Can a model learn a relationship that a single straight line cannot separate? Train a network and examine it from the inside.

    Full access

    Continue with the whole laboratory

    This laboratory is part of full access. Here you will work with:

    • Four groups, two classes
    • Can the information be read out?
    • Three things to notice.
    • Do the probe and the model read the same thing?

    Laboratory / 04 · AI safety

    Examine model behaviour.

    Start with a concrete chat answer, then examine how a small model can learn a fragile shortcut.

    Full access

    Continue with the whole laboratory

    This laboratory is part of full access. Here you will work with:

    • What does a chat answer tell you?
    • Two signals. One hidden shortcut.
    • What errors hide in the score?
    • Can a small intervention change the answer?
    • When the reward favours a shortcut

    Foundations / Python · about 65–80 min.

    Read code.
    Try it yourself.

    You don't need to know programming beforehand. Learn the first building blocks, then write small functions similar to those in an ML project.

    Follow the code

    One function, three inputs

    Step-by-step view

    A function is a named recipe. return gives the answer back. A list holds several values, and a for loop repeats something for each element.

    First predict the result for x = −1. Then press Next pass. Change the weight and bias, and see both the code and the computation change.

    def predict(x, weight, bias):
        return x * weight + bias
    
    inputs = [-1, 0, 1]
    outputs = []
    for x in inputs:
        outputs.append(predict(x, 2, 1))
    1 / 4Start
    Inputs
    −101

    outputs = []
    Try it yourself with a new input

    What does predict(2, weight, bias) give with the sliders' current values?

    You can try again or skip the task. The answer is not recorded as a passed check.

    The numbers are recomputed in the browser's JavaScript when you change a slider; the browser does not run the Python code. The next exercise uses the same function with three arguments and lets you write and run real Python locally.

    Full access

    Continue with the whole laboratory

    This laboratory is part of full access. Here you will work with:

    • One function, three inputs

    Practical project / 05 · about 30–60 min.

    From browser to real code.

    Train a small neural network with Python and PyTorch. You get training measurements, a chart and the model's saved weights, which you can examine yourself.

    Full access

    Continue with the whole laboratory

    This laboratory is part of full access. Here you will work with:

    • An experiment you own
    • Write your own training step
    • Write a next-token step

    Laboratory / 06 · mechanistic interpretability · about 25 min.

    Move an activation.
    See what changes.

    Activation patching is a causal intervention: we run the same model on two inputs and replace an internal value from one run with a value from the other.

    Full access

    Continue with the whole laboratory

    This laboratory is part of full access. Here you will work with:

    • From input A to input B
    • What have you actually measured?
    • Does the effect hold across inputs?

    Project / 07 · transformers · about 135–185 min.

    Train a transformer.
    Examine attention.

    Build on the Python exercise: train a small causal transformer locally, and load its actual attention matrix here in the browser.

    Full access

    Continue with the whole laboratory

    This laboratory is part of full access. Here you will work with:

    • Train a model to guess the next character.
    • What does an extra character help?
    • What does attention mix together?
    • Which symbol came first?
    • Intervene in your trained transformer.

    LLM project · about 25–40 min.

    Train a small text model.

    Go from the next-token idea to an actual transformer that you train and examine yourself.

    Text project / 1 · browser or local Python

    Train a model on text

    Start with the two sentences and see how the model's guesses change. You can try your own texts and later train a small transformer in Python.

    Train on text in the browser

    The model sees two characters and guesses the next. Press Train to start; if you change the text, a new model is created at the next training.

    This is a small teaching model with one attention head and a fixed placeholder ?, not the Python transformer below or a full LLM. It only computes output at the placeholder and has no MLP or layer normalization.

    Use 4–160 characters per text and at most 24 different characters in training. The validation must use known characters. The texts and the model are only saved locally if you choose so below; they are not included in the backup.

    Full access

    Continue with the whole laboratory

    This laboratory is part of full access. Here you will work with:

    • Train a model on text
    • What did the text model learn?
    • Train with your own tokenizer
    • Fine-tune a real language model.

    Research exercise / open model · about 45–90 min.

    Probe a real language model.

    Take the method from the small experiments over to an open, pretrained checkpoint. Here you analyse the model locally; you don't train it from scratch.

    Full access

    Continue with the whole laboratory

    This laboratory is part of full access. Here you will work with:

    • Two prompts. One causal intervention.
    • Is the pattern stable across examples?
    • Where in the prompt does the result change?
    • Repetition in a real language model

    Research project / 08 · about 60–120 min.

    From exercise to research note.

    Formulate a hypothesis, repeat an experiment on several small models, and write what the results can and cannot tell you.

    Full access

    Continue with the whole laboratory

    This laboratory is part of full access. Here you will work with:

    • How stable is the patch effect?

    Your map

    A way into research.

    From your first LLM exercises to small research projects.

    Choose a direction

    What do you want to work towards?

    Everyone starts with LLMs. Each direction remembers your position, and you can switch along the way.

    The direction is saved locally when browser storage is available.

    See the route and jump to a topic

    The guided path: From tokens to attention and then onwards in the chosen direction.

      Foundations along the way: Later projects assume maths and programming. Take or refresh the topics here when you need them.

        The guided start goes from tokens to attention. The topic list also contains foundations you can take as needed; later projects build on them. Local projects require installation.

        Your progress

        0 / 12 understanding checks

        Small checks give feedback on your understanding. The results are not an exam.

        See all understanding checks
        One possible order

        Explore, step by step.

        Start with the first exercises, or choose a track by interest. Suggested times do not include local installation.

        See the long-term research direction
        The longer view

        The road ahead towards research.

        The tracks below show the road ahead. New topics are added continuously.

        An honest goal. The road to a research job requires deep practice, independent results and often collaboration with others. This map is a direction — not a promise of employment.
        Your work

        Your experiment log.

        Real researchers don't just write down results. They also write what they expected and what surprised them.

        Save an observation

        Only on this device

        Your notes

        Take your work with you

        Backup of your notes and progress

        Your notes, understanding checks, recorded steps, learning position, chosen direction and some saved browser models are stored only here. Save a copy if you want to move them or protect them against loss.

        Load and merge a copy

        Choose an earlier copy and review the changes. Notes and passed checks are merged; existing models, position and direction are kept by default.

        The file is not encrypted and may contain your own notes and any texts and safety draft you opted in. Store it as you would other private files. The regular saved browser models are included; the text model is only included if you opt in. Temporary measurements, generalisation models and locally run PyTorch projects are not included. The file is processed locally and is not sent to a server.