Error –
Touch the model
Change text, weights and settings, and watch the model respond instantly.
Hands-on exercises where you train models yourself, right in your browser – from your first tokens to transformers, AI safety and interpretability. Nothing to install, and no programming required.
0 tegn → 0 tokens
The same idea as in the first lab: the most frequent neighbouring pairs are merged, one at a time. This is a teaching model, not a fully trained tokenizer.
Error –
Change text, weights and settings, and watch the model respond instantly.
Step 0 · Error –
Tokens, loss, attention and activations are shown while the model trains.
Write observations in your experiment log and compare your experiments.
14 labs in four tracks. You start for free and can unlock the rest whenever you like.
Sikker betaling med Stripe
Already have an access key?
All labs are unlocked in this browser. Save your access key if you want to use Learn the Machine on another device.
…No. Every lab runs in the browser, and you control it with buttons, sliders and your own text. The Python parts are optional, for when you want to go further.
A recent browser such as Chrome, Edge, Firefox or Safari. It works on a phone, but a computer or tablet gives the best experience in the larger labs.
In your own browser. There is no account, and your notes and models are never sent to us. You can make a backup from the experiment log.
After your purchase you get an access key. Redeem it on the new device under “Already have an access key?”.
You get access immediately. When you pay, you therefore agree that the right of withdrawal lapses once delivery has begun. Write to us if something doesn't work, and we'll find a solution.
Yes. The whole site is in English, including all the paid labs. Downloadable Python files, example data and some generated reports are still in Danish.
Built for understanding, not just answers. New labs and exercises are added regularly.
Before a language model can work with text, a tokenizer turns the text into a sequence of ID numbers. Let's find out how.
A token can be a word or part of a word. Press «Merge next pair» to join the most frequent pair, then try your own text. In this exercise a pair must occur at least twice to be merged. This is a teaching model, not a fully trained LLM tokenizer.
When you change the text, the merges start over. A model trained on the split also starts over.
Round 0
Your split · a dashed border marks what was formed in the latest round. ␠ shows a space, ↵ a line break.
Try the two versions of é. They can look the same but are different sequences of Unicode code points and bytes. A tokenizer may normalise the text first; others keep the difference. In the next step you can compare the two forms with real tokenizers under «Explore languages, Python and token view».
Unicode code points
UTF-8 is one way of encoding text as bytes. Each byte is not necessarily one token; tokenizers can use different rules. Counting visible characters follows the browser's Unicode segmentation when it is available.
␠ shows a space, ↵ a line break.
The exercise starts with Unicode code points, which are called characters here. A visible character can consist of several code points. The exercise learns its rules from your text. A real model tokenizer uses rules it has already learned.
Write a new text. We only use the rules from the training text above and the chosen rounds; the new text does not teach the tokenizer any more rules.
This simple model starts with the characters from the training text. Unknown characters are marked as out-of-vocabulary. The exercise does not compute token IDs; real tokenizers may handle unknown characters differently.
Choose a round to see how the text is split after that round. The list shows the rounds that are possible with your text, even when the slider is set lower. When counts are tied, the pair that appears first in the text is chosen.
The vocabulary starts with the distinct characters in the text. Each chosen merge can add a new piece; earlier pieces are kept even if they no longer appear in the text's current split.
Original characters
New pieces from chosen merges
This is a small teaching model without token IDs. A real model tokenizer has its own fixed vocabulary and may work differently.
In each round, pairs are counted even when they overlap. The pairs are then merged from left to right without overlap, so fewer may be merged than were counted. Here the merges are learned from exactly the text above. Real model tokenizers have already been trained on far larger amounts of data and apply their fixed vocabulary to new text.
Compare two model tokenizers on the same text. A token ID belongs to the tokenizer's own vocabulary.
Qwen3.5 is newer than Qwen2.5, Pythia and GPT-2. BERT is an older WordPiece model with a different training objective from an autoregressive chat model.
You can guess before comparing, see internal token pieces or try a guided model training.
The guided example replaces the text above, compares the tokenizers and opens a micro model with a test text ready.
Runs in your browser. The first time, the selected tokenizer files are loaded from this site. The default choice of Qwen3.5 and GPT-2 is about 14 MB uncompressed together; the actual download depends on compression and cache. The comparison only shows text, not the model's image input.
Øvelsen klargøres. Hvis den ikke bliver klar, kan du prøve at genindlæse siden.
When retrying, the text and your tokenizer choices are stored temporarily in this tab's session storage and removed after reloading. Unsaved work in other exercises may be lost.
The comparison does not automatically add the special tokens a model may use.
É as one code point and e + accent often look the same but have different bytes. Examine both forms with the two selected tokenizers.
Compare IDs within the same tokenizer. Identical IDs can result from normalisation or loss of information, for example [UNK]; that does not show what a model understands. No special tokens are added automatically.
Examine the two selected tokenizers on a short greeting in Danish, English and Japanese. Guess first: is the number of tokens the same across the languages?
The greetings mean roughly the same, but do not have the same characters or length. The examples do not show a whole language's tokenisation or model quality. BERT may replace unknown text with [UNK], so fewer tokens can also mean lost information. A lower token count can affect how much text fits, but says nothing on its own about the quality of the answers.
python -m pip install transformers.python sammenlign.py --tekst "Hello world!" in the same folder. The default is Qwen3.5 and GPT-2; the first run downloads tokenizer files, not model weights.tokenizers-resultat.json below. Python does not read the choices above: use e.g. --foerste qwen --anden bert for other tokenizers. The options are qwen35, qwen, pythia, gpt2 and bert. Use --output another-name.json on the next run; existing files are not overwritten. --offline uses files that have already been downloaded.For ByteLevel BPE, pieces are shown as readable text when their bytes alone form valid UTF-8; otherwise byte values are shown. In BERT's WordPiece, ## marks a continuation, and [UNK] can replace unknown text. A character can be split across several tokens. ␠ shows a space, ↵ a line break and ⇥ a tab. Turn on internal pieces to see symbols such as Ġ. The result leaves out automatically added special tokens; if you type a recognised special token yourself, it may still be recognised. The result is therefore not necessarily equal to the token usage of a finished chat prompt.
Sources: Hugging Face: the parts of a tokenizer and OpenAI: tiktoken and BPE · Hugging Face: BERT and WordPiece.
1 / 5 · Text
A text is the starting point. It could be your message to a language model, for example.
GPT-2 and Pythia both give the text Hello one token, but with different IDs.
What does GPT-2's ID 15496 correspond to in Pythia's vocabulary?
In Pythia, ID 15496 is ␠counting, not Hello. The model looks the ID up in its own embedding table; it does not know the ID came from another tokenizer. Use the tokenizer that belongs to the model. ␠ shows a leading space.
Lookups in the fixed GPT-2 and Pythia vocabularies without automatically added special tokens. The example shows ID lookups, not a model run.
This diagram explains the process; it does not run the model itself.
You can try again or continue; the check is not saved as a passed test.
Two short English prompts have been run in advance through the small, open base model Pythia-70M. Choose a sentence:
Paris and Rome are each one token in this particular example. The rank counts the tokens with higher probability plus one. A low probability for a single next token does not by itself decide what a longer continuation would contain.
␠ shows a leading space. The five shown are each a possible next token; the rest is the sum of all other tokens, not one token. The distribution is before sampling. The model is small and not a chat assistant; the most likely token need not be the correct answer.
The examples come from Pythia-70M's pinned checkpoint, without automatically added special tokens. Softmax is computed over the whole vocabulary at temperature 1. Download the Python program to reproduce the result with PyTorch and Transformers. The first run downloads the model; the precomputed example above needs no installation.
Optionally compare a text in the previous step to see its first token IDs here.
Here we use the teaching tokenizer's split from step 1, not the model tokenizer from the comparison above. Choose a position: the context is the tokens before the position, and the answer is the next token.
Context
Next token · answer from the text
␠ shows a space; ↵ shows a line break. This is the teaching tokenizer's split, not the tokenizer from the model comparison. The figure above shows data. In the next step you also train a separate character model on other example texts. Real training windows may have limited context; the first token needs a starting context and is not shown as an answer here.
This small table model uses only the latest token as input and guesses the next one. It starts with equal probabilities and is trained on roughly the first 75 % of the token pairs; the rest is used for measurement. The vocabulary and merges were made from the whole text, and pairs can repeat on both sides. The validation is therefore not an independent test. It is a bigram model, not a transformer.
The curves show measurements before and after each press of the training button. Lower loss means higher probability for the actual next tokens. The validation pairs are not used to update the weights, but may resemble or repeat the training pairs.
━ Training ┄ Validation · horizontal: training steps · vertical: loss in nats per pair, from zero.
| Step | Training loss | Validation loss |
|---|
A pair is input token → answer. We count occurrences, including repeats. A new pair can use an input token the model has already trained on.
The group numbers are descriptive: the groups can have different content and few pairs. A new pair here also comes from the same text, and the tokenisation was made on the whole text. It is not an independent final test.
The history only exists in this page view and is not included in the experiment file. When reopened, the chart starts with the restored model's current measurement.
The bars show the distribution for the selected example's latest context token. The answer is marked. The model is reset when the split changes. Download an experiment file below if you want to reopen it. No text is sent away from the browser.
This model only sees the last token before the answer. If the same last token appears after different earlier tokens, the two examples therefore get the same distribution — even when the answer differs. That is a limitation of this table model; a transformer can use several earlier tokens within its context window.
The bars show the probability of each example's answer on the same 0–100 % scale. Choose another example above, or change the text and the tokenisation. The comparison uses the current model weights.
Download a file with your text, token split, model weights and number of steps. It is only stored on your device, contains your text and is not encrypted. It is not automatically included in Learn the Machine's other backup.
When you choose Restore, the file replaces the current text, split and model. Download the current experiment file first if you want to keep it. Files can be edited; their origin cannot be verified.
We apply the same learned merges and model weights to the text below. See how many of its token pairs also occurred in the model's training pairs. The text can be changed and reused, so it is not an untouched final test.
| Group | Number of pairs | Average loss · nats |
|---|
A lower average loss means a higher average log probability for the answer. The groups can contain repeats; different pairs make them not directly comparable as a controlled experiment.
| Input → answer | Seen in training? | Answer probability |
|---|
A known token pair can be repeated several times and is not an independent observation. The model has no unknown token: if a token is not in its vocabulary, the whole text cannot be measured here. The text is processed locally.
Two occurrences of the same token ID have the same input embedding in the model. What can happen to their representations after a transformer layer?
You can try again or continue; the answer is not recorded as a passed test.
Here a fixed query and key are each rotated according to their position. First try moving both one position forward. Then change only the key position. Compare the dot product, which can feed into an attention score.
Green, solid line: query. Orange, dashed: key. Each line starts in the centre and ends at the vector.
The example uses two fixed two-dimensional unit vectors and 30° per position. If you move both equally far, their relative angle and dot product are preserved. If you move only one, it can change; the effect does not always decrease with distance. Also try changing the key's starting direction without moving the positions: the same distance can give a different dot product. The starting direction was chosen for the exercise, not learned from text. Neither softmax nor a causal mask is computed here, so the key can come after the query.
RoPE rotates the query and key in attention. Real models have many component pairs and frequencies, often other variants too, and the query/key depend on content and context. A shared position shift therefore does not necessarily preserve the whole model's output. The small browser transformer in the transformer lab uses learned position vectors instead; this illustration does not train a model.
The included example uses layer 0, head 0 and the text “The capital of France is”. You can also open a local measurement with your own text and choices.
Write the text and use Pythia's tokenizer to find the positions. The text is processed in the browser; the first time, the tokenizer files are downloaded from the site. Choose query and key, and use the numbers in the Python command below. This does not run the language model.
Forhåndsvisningen klargøres. Hvis den ikke bliver klar, kan du genindlæse siden.
␠ shows a space; bytes in square brackets may be parts of a character. No automatic special tokens have been added.
QUERY marks the position that looks back; KEY marks the position it is compared with. The markers show positions in the text, not the query and key vectors themselves. In this causal experiment the key can only be at or before the query. The same position can have both roles.
Put the downloaded rope-tekst.txt in the same folder as rope_pythia.py and run the command from there. Use the latest version of the program via the link below. The command follows your chosen positions.
If the browser renames the text file, change the file name in the command accordingly. Choose a new output name if min-rope.json already exists. The text file and the result file contain your text and stay on your own device until you share them.
The selected component pair is only one contribution. Pythia rotates 16 of 64 components in this head; the rest pass through unchanged. The whole dot product before rotation is a computed comparison, not the score the model actually uses. The model scales the rotated score and applies softmax over all allowed keys, so the attention weight cannot be derived from this pair alone. The example is one text and one head, not evidence about all inputs.
Repeat locally with a pinned checkpoint: download rope_pythia.py and run python rope_pythia.py --output min-rope.json. The first run downloads the model; add --offline once it is cached. Before choosing positions, see the token numbers with python rope_pythia.py --text "The cat and the cat" --list-tokens. This command only uses the tokenizer and does not load the model weights. Then use the numbers in a measurement, e.g. python rope_pythia.py --text "The cat and the cat" --query 4 --key 1 --layer 1 --head 3 --pair 2 --output mit-forsøg.json. The text must give 2–32 tokens; positions are counted from 0, and the key must not come after the query. Layers are 0–5, heads and component pairs 0–7. Without chosen positions, the last query and a key up to two places earlier are used. The file contains the text, token IDs and measurements; only share it if you choose to. Open it above. Only the same setup is directly comparable with the default example. Files and hashes can be edited.
Sources: Su et al.: RoFormer ↗ · Hugging Face: RoPE variants ↗.
Choose two tokens from a real, open language model. Start with ␠cat and ␠dog, then choose the same token twice. What happens to the similarity?
Want to see what happens through the model? Start a ready-made experiment with the text The cat and the cat.
Here you compare two precomputed runs: A: The cat and the cat and B: The dog and the cat. Only the token at position 2 has changed. The last ␠cat has the same ID and position in both texts. Predict first: is its activation after layer 1 the same?
Also try comparing the first token, The, in A and B. It comes before the change and cannot see later tokens in this model. The largest component difference summarises all 512 numbers; cosine measures direction. Two examples do not show which internal mechanism causes the difference.
The example is precomputed from Pythia-70M and is only fetched when you press the button. No installation needed.
The first 8 of 512 components. The direction of the bar shows the sign; both tokens use the same scale.
␠ means a space. ␠cat and cat are different tokens with different IDs. The figure shows an excerpt, not a projection of the whole vector; the similarity is computed on all 512 numbers. Cosine similarity measures how similar the angle is, from −1 to 1, and does not in itself say what the model understands or will answer. The input embedding is a weight before the transformer's context processing. A transformer layer, on the other hand, shows the activation at a position after the layer. It can be affected by earlier tokens and the position; cosine alone does not tell you why two activations differ.
The six vectors were extracted in advance from Pythia-70M, pinned checkpoint; no model weights are trained or changed here. Token IDs and weights apply to this model, not to the other tokenizers. You can reproduce the extraction: download export_embeddings.py, install PyTorch and Transformers, and run python export_embeddings.py --output pythia-embedding-examples.js. The first run downloads the model. The file only contains the six chosen vectors.
python export_embeddings.py --tekst "The cat and the cat" --lag --output mine-embeddings.json. The first time, the model and tokenizer are downloaded. Later, choose your own short text; it must give 2–16 tokens. If the file name already exists, choose another name.The file and text are processed locally on your computer and are not sent to Learn the Machine. The Python program downloads the model and tokenizer from Hugging Face the first time. The file contains token pieces, IDs and vectors from your text, including layer activations when you use --lag; only share it if you want to share that. Repeated IDs use the same input vector regardless of position. With --lag the text is also run through the model, and the vector after each of the six transformer layers is saved before the model's final normalisation. The model can only use the current and earlier text, not later tokens. Differences cannot be attributed to one particular word or one mechanism alone. The imported token pieces are the tokenizer's internal symbols; ␠ shows Ġ, while other byte symbols cannot necessarily be read as ordinary text. The file describes its origin, but the site cannot verify the authenticity of the weights.
Choose a made-up token ID and see its vector as a point. Move the components and watch how a small output computation changes the next-token distribution.
A small distribution over two possible next tokens:
The numbers, token IDs and the two possible answers are constructed for the exercise. A fixed linear output computation and softmax give the distribution; the sliders change a vector by hand, they do not train it. There is no transformer or context from earlier tokens here. A real model has far more dimensions and uses the transformer between lookup and output.
Want to see how a position can mix information from earlier positions? The next exercise uses other, constructed numbers for query, key and value.
A chat template turns messages with roles into a single token sequence. Try a limited example with Qwen2.5-0.5B-Instruct: one system message and one user message, without tools or earlier conversation.
The tokenizer runs locally in the browser and is loaded the first time you use it.
Formatted text (line breaks are part of the format):
The markers <|im_start|> and <|im_end|> are special tokens; the role and line breaks are also part of the input. The token count for the whole chat includes the system message and the format. The difference cannot generally be attributed to the format alone, because splitting depends on context. Other models and chat products use other templates. This example leaves out tools and history and does not predict API usage.
Sources: Qwen's pinned chat template · Hugging Face: chat templates.
Here every character is a token. The model sees one character and guesses the next; it is not a transformer. It has its own small vocabulary and does not change the tokenizer or the Pythia model from before.
No training pairs have this character as input.
Your guess is assessed after the next training. Less than 0.05 percentage points counts as almost unchanged.
You have reached the limit of 1,000 steps. Save an observation if you want to keep it, and choose Start over to train again.
See the distribution below, or continue with attention.
Try an optional Python exercise with explanations and a comparable result. You can continue with attention afterwards.
Train two new character models from the same zero weights for 20 steps. Model A sees the original three sentences. Choose what kat should be replaced with for model B. Both use 57 training pairs, the same learning rate and the same measurement sentence. Several character pairs may change. The models are lookup tables with one character of context and have no attention heads.
Less than 0.05 percentage points counts as almost the same. The guess is not saved and can be given after you have seen the result.
The top bar is A, the bottom is B. All bars use the same scale; the numbers are counts, not probabilities. ␠ shows a space.
The measurement sentence shares character pairs with the training data and is used repeatedly. The experiment compares two small models; it does not measure how a head forms in Pythia.
Save data, settings and measurements in the Experiment log on this device. The model weights are not saved; the note can be included in your backup.
Others groups the characters that are not shown individually. All observed training answers are shown individually. The percentages are rounded.
See how the guesses moved. Green is before and orange is after the latest training. The bars use the same 0–100 % scale. Only two possible next characters are shown; the whole distribution sums to 100 %.
The weight is the model's score for this character pair, not a probability. The model converts all scores for the input character into probabilities that add up to 100 %. Training also adjusts other weights.
Here is the measurement from the latest training and your guess at the start of training, if any. It is saved with the model on this device when browser storage works, and can be included in the backup. Save it in the experiment log together with your optional comment. The guess does not document what you knew before you opened the exercise.
0 training steps · training loss — · validation loss —
Training text: “en kat kan se en mus.” · “en mus kan se en kat.” · “kat og mus kan se.” (Danish for “a cat can see a mouse” etc.)
Write softmax and the gradient yourself. A local Python exercise checks your code and trains the same small next-character model for 20 steps. It needs Python 3 but no extra packages.
The exercise uses functions, lists and loops. You can take an introduction first and come back here.
A standalone example with one training pair and three possible next characters. It does not change the model above.
Softmax converts scores into probabilities. For each character the gradient is probability minus answer: 1 for the correct character, 0 for the others.
Change one score very slightly up and down. The difference in loss divided by the distance gives a numerical approximation of the gradient.
Here ±0.001 is used on one example. The Python checker uses another small change and more targets. The numbers are shown rounded. A numerical check is an approximation at specific inputs, not proof that the code is always correct.
The example uses one gradient step with learning rate 0.5. The coding exercise uses more characters and the average over all training pairs. Subtract the largest score before exp for better numerical stability.
min_model.py and run python tjek_og_traen.py in the folder.naeste-tegn-resultat.json here and compare training and measurement texts with the browser's computation.The open checks are not a complete code test. The Python code runs on your computer; only run code you trust. This is one character of context, not a transformer.
Once your functions work, you can train with the same data change as in the browser experiment. The original result file is not overwritten.
python tjek_og_traen.py --replacement matAlso choose kan or kat as an unchanged control. The program saves e.g. naeste-tegn-mat.json; open the file below. The browser recomputes the model for the given choice. The measurement texts overlap the training data and are not untouched final tests.
P(a after k) after 20 steps · same scale from 0 to 100 %.
In each field: loss · probability of a after k.
| Step | Python | Browser |
|---|
None of the measurement texts change the weights. Compare before and after within the same text; differences between texts may be due to their content. After repeated use they are not untouched final tests.
Loss in nats per pair · Python / browser.
| Step | Validation | Second text |
|---|
The comparison recomputes a separate model from scratch in the browser. The numbers and hashes can be edited; agreement does not prove the file's origin or that the code works for all inputs.
Open another result with the same stated code hashes and measurement texts. Both files are read locally. Hashes and results can be edited and do not prove origin.
The probabilities use the 0–100 % scale. The loss bars use a shared scale up to the larger of the two losses; the two kinds of bars cannot be compared directly.
Both are measured on the same texts; overlap depends on the training data. The measurement texts are reused and not untouched final tests. One character of context does not show general language understanding.
Train on the split of your own text in a separate table model.
You can train a small text model in the browser or continue to the Python projects. Training a BPE tokenizer locally needs Python and the tokenizers library; the local transformer also needs PyTorch.
What did the training change? Green is before, orange is after the latest press. A shorter bar means lower loss; both rows use the same scale.
The validation sentence is not used for the weight update, but all its character pairs also occur in training. This is therefore a limited test of new texts. The bars are not saved on reload; the numbers can be seen under the observation above.
The validation sentence is used for measurement, not for weight updates. Choose a character pair and see whether the same pair also occurs in training.
Blue marks the input; orange is the answer. The model only sees one character at a time. A pair appearing in both sets does not prove that the whole sentence was used for training; it shows that the measurement does not test a new character pair for this particular model.
The example has one new character pair. Measure the text, keep training, and see how the loss changes on the same text. You can edit it with characters from the model's vocabulary.
Less than 0.05 percentage points counts as almost unchanged. Your guess applies to the next training here and is not saved on reload.
The model has reached the limit of 1,000 steps. Start over to make a new comparison.
Green shows the measurement when you pressed; orange shows the current model. A shorter bar means lower loss. Press Measure the text again to start a new comparison. The interactive comparison resets on reload; you can save a note below.
For each pair, the answer probability is shown at the first measurement → now. An orange outline marks pairs that do not occur in training. The text does not change the weights. The two texts have different compositions, so differences in loss can come from both known and new pairs. If you change the model after seeing the result, this text is no longer an untouched final test.
The whole text and the measurements are saved locally on this device and can be included in a backup. The model's weights are not saved in the log note.
Lower loss means the model does better overall on the measured text. The validation sentence shares words and character pairs with the training data; it is not a broad test of language ability.
Choose a topic if you want to examine the model more closely. You can also continue directly.
The loss is the average of −ln(probability of the observed next character); individual characters can still get a lower probability. The five largest probabilities are shown together with any training answers outside the top 5. When tied, characters are sorted in vocabulary order.
Each pair shows an input and the next character the model is trained to predict. Repeated pairs count each time; no pairs are made across sentences.
The model stores a logit for each possible next character. Here we follow the most common answer for the chosen input. The gradient shows how the total loss changes with a small change to this logit while the others are held fixed. Gradient descent adjusts all logits at once based on the whole training set; the learning rate is 4.
Green: probability now · orange: after one step. This is a computed preview and does not change the model.
Each training pair gives −ln(probability of its answer). Repeated pairs count each time. The table only shows pairs for the chosen input; the contribution is divided by all training pairs.
| Next character | Number of pairs | Model's probability | −ln(p) · nats per pair | Contribution to total loss |
|---|
Both start with the character chosen above and then pick the most likely next character 32 times. The model only uses the latest character as input.
Before training:
After 0 steps:
␠ shows a space. When several characters have the same probability, the first in the model's fixed character order is chosen; before training all characters are equally likely. No stop character is used, so the display always continues for 32 characters. The small model can repeat itself and is not a transformer.
Use the current model and the input chosen above. At temperature 1 and top-p 1, the bars are the model's own distribution. A lower temperature concentrates the distribution on the largest logits; a higher temperature flattens it. This changes the selection, not the model's weights.
After temperature, characters are sorted by probability. We keep the most likely until their combined chance reaches top-p. The rest get 0 %, and the kept ones are renormalised to 100 %. When tied, we use the model's fixed character order. A top-p of 1 keeps all.
The candidate set is recomputed for each new input. This does not change the model's weights. Source: Hugging Face on top-p ↗
Green on top shows the chance after temperature but before top-p. Orange below shows the chance after top-p, which is used for the draw. The numbers follow the same order; the number after the dot shows observed draws. All draws use the same input and are separate — they do not form a connected text. Changing temperature, top-p, input or model resets the count. Up to 10,000 draws can be made per count.
Start with the input chosen above. Each drawn character becomes the input for the next step. Compare with the separate count above, where the input was the same every time.
| Step | Input | Drawn character | Chance at this step |
|---|
✓ marks the character that was drawn. Green: probability after temperature and before top-p. Orange: probability after top-p. The distribution belongs to the chosen step in the chain.
␠ shows a space. The chain uses the current temperature and top-p and resets if temperature, top-p, input or model change. At most 32 characters are generated; a full stop does not stop the model. This is a small model that only sees one character at a time, not a chat model.
See the input and answer in one of the text transformer's windows. The view uses characters like this exercise, but the transformer can use several earlier characters as context. You don't need Python to see it.
Want to see first what an extra character of context changes? Train two small models and compare their guesses.
Try to answer without looking above. You can try again. When both answers are correct, the check is saved in this browser; it does not document overall competence.
Train two real ByteLevel BPE tokenizers on the same training texts, but with different target vocabulary sizes. Then examine how they split validation text, which is not used to learn the tokenizer.
The example comes from two tokenizers trained on small, artificial texts. You don't train anything yourself by opening it. Use the guide below to make your own result.
You need an activated Python environment. The setup also installs PyTorch, which you will need for the model afterwards.
cd "PATH_TO_FOLDER" and replace the text with the folder's actual path. Use the example file egne-tekster.json or your own file from the dataset builder.python -m pip install "tokenizers>=0.20,<0.22".python train_bpe.py --data egne-tekster.json. If you downloaded mine-tekster.json from the dataset builder, move it to the project folder and run python train_bpe.py --data mine-tekster.json. The terminal shows Gemt: followed by the path to the new folder. Open bpe-result.json from that folder below.Tokens in all validation texts:
First validation text:
The next small model uses up to eight tokens as context. Move the window for each tokenizer: eight tokens are the input, and the following token is the answer. The windows are controlled separately and do not necessarily show the same place in the text. The model is not trained here.
Tokenizers can cover different text with eight tokens and can therefore have different answers here. The comparison does not show which model is best.
The first 256 slots cover byte values. Very little data and a requirement for repetitions can give an actual vocabulary below the target. A lower token count does not in itself mean a better language model. Tokenizer training is not training of model weights.
Once you have trained a tokenizer, you can use it in the next experiment.
Background: Hugging Face: train a BPE tokenizer.
A neural network consists of many simple computations. Start with one: change the bias and watch how the neuron's answer changes.
A neuron adds up its weighted inputs. The bias is a number added to the sum. Move the bias from 0 to 1 and see what happens to the answer.
Sigmoid turns the sum into a number between 0 and 100 %. That is not automatically a reliable probability.
Want to try training a model now? You can skip the other neuron steps and come back to them later.
An input is a number the neuron receives. A weight decides the input's contribution: input × weight. Here we follow input x₂ and weight w₂.
Move the weight from −1 to 0. What happens to the contribution and the neuron's output?
Change one number at a time. A vector here is a list of two numbers: the green arrow shows the input, and the orange one shows the weights.
Sigmoid turns the sum into a number between 0 and 1. In some classification models it is interpreted as a probability estimate, but a number in this range is not automatically a reliable probability.
The same two inputs go to both neurons. Each connection has its own weight. The rows of the matrix belong to one neuron each; the columns belong to input 1 and 2.
Green solid line: positive weight · orange dashed line: negative weight. The thickness shows the weight's size. The numbers on the right are the sums before activation.
Neuron 1: → ReLU:
Neuron 2: → ReLU:
Also open «Explore more inputs and weights». Predict what happens to both sums when you change x₁. Reset, and change only w₁₂. Can you make z₁ negative and see what ReLU does?
With the input as a column we write z = Wx + b. Here W is 2 × 2, x has 2 numbers and the bias is fixed at b = (0.5; −0.5). ReLU gives max(0, z) for each neuron. In PyTorch the batch form is written XWᵀ + b, because the examples are stored as rows.
This is a separate, manually set forward computation; it is not trained. A real layer can have other dimensions and activation functions. Once the weights are fixed, the same weights are used for every example in a batch.
Background: PyTorch: linear layer and dimensions ↗ · PyTorch: ReLU ↗ · reviewed 31 October 2026.
Here you try a separate neuron with one input. A known answer is the target; the loss tells you how wrong the neuron's guess is. Choose a target and update the weight once.
It computes z = x · w + b, then p = sigmoid(z) and a binary log loss for the target y. When y = 1, the loss is −ln(p); when y = 0, it is −ln(1 − p).
Each local slope tells you how one step changes with a small change in its input. The chain rule multiplies them together: that is how we find how the loss changes when the weight changes a little. The numerical check tries small changes to the weight and compares the loss.
The update uses learning rate 0.2 and only changes the weight for one example; the bias is held fixed. In a network with many layers and data points, several gradient contributions are computed and combined. The visualisation computes in the browser; PyTorch uses autograd to compute gradients automatically in the later exercise.
Background: PyTorch: autograd and the chain rule ↗ · Google: backpropagation ↗ · reviewed 31 October 2026.
Two measurements or properties are each described by a number. Which properties are used depends on the task.
Weights decide the inputs' contribution to the sum. The bias shifts the sum, even when the input is zero.
Sigmoid turns the sum into a number between 0 and 1. Other neurons can use other activation functions.
Sources: Google: weighted sum and sigmoid ↗ · PyTorch: sigmoid ↗ · reviewed 31 October 2026.
Train a small model to find a relationship in data. It really updates its two numbers based on its errors — directly on your device.
The model tries to find a line that fits the green points. What do you think happens to its error when you train?
Pick a guess if you like, then train the model.
The ringed point has input x = 2.1 and known answer y = 2.1. The lines show the vertical difference between the model's guess and the point.
| Model | Guess | Guess − answer | Squared error |
|---|---|---|---|
| Before | |||
| Now |
Numbers are rounded to three decimals. This is one of 15 training points; the overall training error is the mean of all points' squared errors. One point does not tell you whether the model generalises.
The orange points are used for validation, not training. The errors are mean squared vertical distances between points and line.
The curve only shows the latest button press. The vertical axis is logarithmic so both small and large errors can be shown. The curve is not saved on reload.
Move the slider from start to end and see how the line and the error changed during the latest training.
Solid green: the selected step. Dashed orange: the end of the round. Lines may fall outside the view. The display does not change the trained model and is not saved on reload.
A larger learning rate gives larger changes per step. A rate that is too large can make the error grow. Reset, try 0.05 and then 0.5, and compare the curves. Read about learning rate ↗
ŷ is the model's guess. Weight and bias are the two numbers training changes. The validation points are not an untouched test set, since you can tune against them.
Which of the two errors was used to adjust the model? What can the other one tell you?
The model is adjusted using the green training points. The orange validation points measure how it handles other examples in this small dataset; they do not prove that it works on all new data.
When you reset the model, the weight and bias are zero, and the line is horizontal.
The distance between the model's guesses and the known answers is used to compute an error.
Gradient descent uses the error to choose the direction of a parameter change. A learning rate that is too large can still increase the error. Repeat and measure on validation data.
This is linear regression on a small synthetic dataset, not a neural network. The same idea of optimising parameters is a building block of deep learning.
Further reading: Google Machine Learning Crash Course: gradient descent ↗
A model can fit the data it has seen and still fail on new examples. Compare three models and examine why.
This laboratory is part of full access. Here you will work with:
A measurement is based on concrete examples. See why the result can vary even when the underlying chance is unchanged.
This laboratory is part of full access. Here you will work with:
Can a model learn a relationship that a single straight line cannot separate? Train a network and examine it from the inside.
This laboratory is part of full access. Here you will work with:
Start with a concrete chat answer, then examine how a small model can learn a fragile shortcut.
This laboratory is part of full access. Here you will work with:
You don't need to know programming beforehand. Learn the first building blocks, then write small functions similar to those in an ML project.
A function is a named recipe. return gives the answer back. A list holds several values, and a for loop repeats something for each element.
First predict the result for x = −1. Then press Next pass. Change the weight and bias, and see both the code and the computation change.
def predict(x, weight, bias):
return x * weight + bias
inputs = [-1, 0, 1]
outputs = []
for x in inputs:
outputs.append(predict(x, 2, 1))What does predict(2, weight, bias) give with the sliders' current values?
You can try again or skip the task. The answer is not recorded as a passed check.
The numbers are recomputed in the browser's JavaScript when you change a slider; the browser does not run the Python code. The next exercise uses the same function with three arguments and lets you write and run real Python locally.
This laboratory is part of full access. Here you will work with:
Train a small neural network with Python and PyTorch. You get training measurements, a chart and the model's saved weights, which you can examine yourself.
This laboratory is part of full access. Here you will work with:
Activation patching is a causal intervention: we run the same model on two inputs and replace an internal value from one run with a value from the other.
This laboratory is part of full access. Here you will work with:
Build on the Python exercise: train a small causal transformer locally, and load its actual attention matrix here in the browser.
This laboratory is part of full access. Here you will work with:
Go from the next-token idea to an actual transformer that you train and examine yourself.
Start with the two sentences and see how the model's guesses change. You can try your own texts and later train a small transformer in Python.
The model sees two characters and guesses the next. Press Train to start; if you change the text, a new model is created at the next training.
This is a small teaching model with one attention head and a fixed placeholder ?, not the Python transformer below or a full LLM. It only computes output at the placeholder and has no MLP or layer normalization.
Use 4–160 characters per text and at most 24 different characters in training. The validation must use known characters. The texts and the model are only saved locally if you choose so below; they are not included in the backup.
Temporary undo in this page view. It disappears on reload or when a new model has finished training.
This laboratory is part of full access. Here you will work with:
Take the method from the small experiments over to an open, pretrained checkpoint. Here you analyse the model locally; you don't train it from scratch.
This laboratory is part of full access. Here you will work with:
Formulate a hypothesis, repeat an experiment on several small models, and write what the results can and cannot tell you.
This laboratory is part of full access. Here you will work with:
From your first LLM exercises to small research projects.
Everyone starts with LLMs. Each direction remembers your position, and you can switch along the way.
The direction is saved locally when browser storage is available.
The guided path: From tokens to attention and then onwards in the chosen direction.
Foundations along the way: Later projects assume maths and programming. Take or refresh the topics here when you need them.
The guided start goes from tokens to attention. The topic list also contains foundations you can take as needed; later projects build on them. Local projects require installation.
Small checks give feedback on your understanding. The results are not an exam.
Start with the first exercises, or choose a track by interest. Suggested times do not include local installation.
The tracks below show the road ahead. New topics are added continuously.
Real researchers don't just write down results. They also write what they expected and what surprised them.
Your notes, understanding checks, recorded steps, learning position, chosen direction and some saved browser models are stored only here. Save a copy if you want to move them or protect them against loss.
Choose an earlier copy and review the changes. Notes and passed checks are merged; existing models, position and direction are kept by default.
The file is not encrypted and may contain your own notes and any texts and safety draft you opted in. Store it as you would other private files. The regular saved browser models are included; the text model is only included if you opt in. Temporary measurements, generalisation models and locally run PyTorch projects are not included. The file is processed locally and is not sent to a server.