THE BOTTLENECK
1,797 digits, 2 numbers each.

01  What it does

Give a neural network 1,797 pictures of handwritten digits and one instruction: copy the picture. No labels, no "this is a seven", nothing to annotate.

The catch is that the copy has to pass through a layer only two numbers wide. Sixty-four pixels go in, two numbers come out, and the second half of the network has to rebuild the whole image from that pair. Memorising is impossible at that width, so the network has to work out what actually distinguishes one digit from another — loops, slant, stroke weight — and spend its two numbers describing that.

The payoff is the map in section 03. Plot each image at the two numbers it was given, and the digits sort themselves into groups: all the zeros here, all the fours over there. Nobody told the network what a zero is. The colours are added afterwards, on this page, purely so a human can check that the groups it found line up with the real digits.

• This is how recommendation systems, semantic search and face recognition find structure in data nobody had the budget to label.
• It runs on a laptop CPU in about ten seconds, so the whole idea is inspectable end to end.
• The seed is fixed, so the map on this page is the map you get when you run it.

02  Architecture

Three components, one direction of travel. The Python side trains and exports; the web side only ever reads what was exported.

Inside the model

Two named keras.Sequential models, composed into a third. The encoder is the half you keep; the decoder exists to make the encoder honest.

• The 2-unit layer has no activation. The code must be free to live anywhere in the plane; a relu there would clamp the map into one quadrant.
• The output layer is sigmoid. The targets are the inputs and the inputs are pixels in [0, 1], so the output is squashed into the data's own range.
• Loss is MSE, optimiser is Adam, and there are no callbacks. The bottleneck alone is what forces the representation; anything else would muddy the attribution.
• Labels are structurally excluded. train(X) has no y parameter, so no label can reach the loss even by accident.

The spec for this exercise started at a 32-unit hidden layer for 60 epochs. At that size the map was a single smear, so — following the spec's own instruction to raise width and epochs until clusters are visible by eye — the built system uses 96 units for 240 epochs. Everything else is unchanged. The diagram above is the network as actually built.

03  Real output

Captured from an actual run on —. The scatter below is the live embedding; the terminal block underneath is the unedited stdout of that same run.

The learned space

Every dot is one handwritten digit, placed at the two numbers the bottleneck gave it. Toggle the classes off and on — that is how you see the clusters.

Hover a dot (or tap, on touch) to read its digit. Click a legend chip to hide or show that class.

These coordinates are the network's actual learned embedding from the run above — clusters were never labeled during training.

Through the bottleneck

Three captured images and what the decoder rebuilt after everything about them was reduced to a single pair of numbers. Both grids are 64-float arrays from the run, drawn here as 8×8 with an identical value-to-colour mapping.

Blurry is the honest outcome. Two numbers cannot carry which pixels were inked; they carry what kind of digit this is, and the decoder paints the average of that kind.

Terminal capture

Verbatim stdout, saved to docs/run-output.txt by python autoencoder.py --json. The ASCII map is the same embedding as the canvas above, rendered with the majority label per character cell.

data: 1797 digits x 64 pixels, range [0.00, 1.00]
training 240 epochs through a 2-unit bottleneck (labels are NOT used)...
final reconstruction MSE: 0.032450

THE LEARNED SPACE — 1797 digits, each drawn at its own 2 numbers.
Characters are the true labels, painted on AFTER training, purely so
you can see that the clusters the network found line up with them.

+--------------------------------------------------------------------+
|                                      2                             |
|                                      222                           |
|                                        22                          |
|                                     2222222                        |
|                        11 1         22222 2 2                      |
|                        111 1  1     22 2222                        |
|                        11118    1    222222211  1                  |
|                       11111111     22222111211                     |
|                     1  11111177    82222222                        |
|                   1 111111817718 1 82212212  3 3 33                |
|4      4   4       111112111177  88 6282222 3333333333  99          |
|     4 4 4   4     11111 6211748888888228333333333 3 9 3   9        |
|     44 44 444444  111122 8777285588288833333333939999999  9        |
|        4 4444444444411  2 877785888888339839999999999      6  66 66|
|          4 4 444444  499  87788558883888999999993 93 66 66  6 66   |
|             444444444   99777795553888599599 95  6 6666 666666666  |
|              44444444445 77779955555555585 566 666666666 666 66 6 6|
|                 44444444447779 55555 5666666 6666666666666666  6   |
|                  444 4 9977 7  5  5   6   06 6 6 66 666666 66      |
|                  4 4    99977    0  00000000000000000   0 0        |
|                           7         00 0000000000000000  00        |
|                                           00000 0000000000 0       |
+--------------------------------------------------------------------+

THROUGH THE BOTTLENECK — original vs. what 2 numbers can rebuild:

  digit 3: original     ->  reconstruction
      ==%%##..      ->      ++%%##--    
    ==##--%%::      ->    --%%++**##..  
    ....####        ->    ::--::##**..  
      ..%%**..      ->      ..**%%++..  
        ..####..    ->        ::####::  
          ..**==    ->          ..##==  
      ==::--%%++    ->      ==--==%%--  
      ==####++      ->      ++%%@@**..  

  digit 6: original     ->  reconstruction
        ####        ->      ..**##::    
      --@@==        ->      ++%%++..    
      ##@@::        ->    ..##**  ..    
      %%##          ->    ::%%##--::..  
      %%##==..      ->    ::%%%%##**..  
      ##@@##@@::    ->    ..%%**==##--  
      ==@@**%%==    ->      ++**++##==  
      ..++%%**::    ->      ..**%%**..  

  digit 8: original     ->  reconstruction
      ++%%==..      ->      ==##%%++    
      ##%%%%##      ->    ::%%++**##::  
      ++**  %%::    ->    ::++::==**..  
      ::@@##%%..    ->    ..--**##--..  
      ::@@@@..      ->      ..++%%==..  
    ::@@==**##..    ->      ....++%%..  
    ..%%..::@@==    ->      ====**##..  
      **@@%%**..    ->      ==%%##--..  

wrote results.json — 1797 real embedded points, 3 reconstruction demos

The 8s stay mixed into the middle of the map in both renderings. That is a real property of a 2-D code — an 8 shares its loops with 3s, 5s and 9s — not a rendering artefact, and it is left uncorrected.

04  Key decisions

The full log lives in docs/decisions.md. These are the ones that changed the product.

05  How to run it

CPU only, about ten seconds end to end. No GPU, no dataset download — load_digits ships inside scikit-learn.

python3.11 -m venv .venv
source .venv/bin/activate
pip install tensorflow scikit-learn

python autoencoder.py

python autoencoder.py --json      # writes results.json

python -m unittest discover -s tests -v

open site/index.html

After a re-train, re-embed the new data into this page with the snippet in site/README.md; the test suite fails if the page and results.json ever drift apart.

06  Code tour

Five files. The interesting parts of each, and why they are the interesting parts.

encoder = keras.Sequential([
    keras.Input(shape=(64,)),
    layers.Dense(HIDDEN_UNITS, activation="relu"),
    # THE BOTTLENECK. Two units. No activation, so the code is free to
    # live anywhere in the plane.
    layers.Dense(LATENT_DIM, name="bottleneck"),
], name="encoder")

decoder = keras.Sequential([
    keras.Input(shape=(LATENT_DIM,)),
    layers.Dense(HIDDEN_UNITS, activation="relu"),
    # sigmoid, because the targets ARE the inputs and the inputs are
    # pixels in [0, 1].
    layers.Dense(64, activation="sigmoid"),
], name="decoder")

autoencoder = keras.Sequential([encoder, decoder], name="autoencoder")

def train(X: np.ndarray) -> tuple[keras.Model, keras.Model, keras.Model, float]:
    keras.utils.set_random_seed(SEED)

    encoder, decoder, autoencoder = build_models()
    autoencoder.compile(optimizer="adam", loss="mse")

    autoencoder.fit(X, X, epochs=EPOCHS, batch_size=BATCH_SIZE, verbose=0)

    final_mse = float(autoencoder.evaluate(X, X, verbose=0))
    return encoder, decoder, autoencoder, final_mse

cells: dict[tuple[int, int], Counter] = {}
for x, row, label in zip(cx, cy, y):
    cells.setdefault((x, row), Counter())[int(label)] += 1

grid = [[" "] * MAP_W for _ in range(MAP_H)]
for (x, row), counter in cells.items():
    winner = counter.most_common(1)[0][0]
    grid[MAP_H - 1 - row][x] = str(winner)   # flip y so it reads like a plot

// Bounds are computed ONCE over every point, so toggling classes never
// rescales the space.
PTS.forEach(function (p) { /* min/max over x and y */ });

function sx(x) { return (x - bx.min) / (bx.max - bx.min) * W; }
function sy(y) { return H - (y - by.min) / (by.max - by.min) * H; }   // y up

canvas.addEventListener("pointerleave", function (ev) {
  // A tap ends with pointerleave; clearing there would make the label
  // flash and vanish on touch.
  if (ev.pointerType === "touch") return;
  hoverIdx = -1; tip.style.opacity = "0"; draw();
});

def test_fit_target_is_the_input(self):
    for call in re.findall(r"\.fit\(([^)]*)\)", self.src):
        args = [a.strip() for a in call.split(",")]
        self.assertEqual(args[0], "X")
        self.assertEqual(args[1], "X", "fit() must be fit(X, X) — never fit(X, y)")

def test_page_states_the_real_architecture(self):
    hidden = re.search(r"HIDDEN_UNITS = (\d+)", self.src).group(1)
    diagram = re.search(r'<div class="arch".*?<div class="archlabels"', self.html, re.S).group(0)
    self.assertIn(f'<div class="u">{hidden}</div>', diagram)

The whole file

"""
autoencoder.py — 64 pixels squeezed through 2 numbers.

A tiny, complete lesson in unsupervised representation learning with Keras.

THE WHOLE IDEA
--------------
An 8x8 handwritten digit is 64 numbers. We ask a network to reproduce that
image *after* forcing it through a layer that is only 2 units wide. Nothing
else may pass. To rebuild a 3 from two numbers, the network has no choice but
to discover what actually varies between digits — stroke thickness, slant,
loop-vs-line — and spend its two coordinates on that. Those two numbers are
the "embedding", and when you plot all 1797 of them, digits of the same class
land near each other.

>>> NOBODY EVER TOLD IT WHAT A "3" IS. <<<

Read the training call below: `autoencoder.fit(X, X)`. The target is the
input. `y` (the digit labels) is loaded in this script and is used for exactly
two things: colouring the ASCII map at the end, and labelling the JSON export.
It is NEVER passed to fit(), never used in the loss, never seen by a gradient.
Every cluster you see in the map is something the network found on its own.

Usage
-----
    python autoencoder.py            # train + print the ASCII map & recon demos
    python autoencoder.py --json     # ...and write results.json for the site
"""

from __future__ import annotations

import argparse
import datetime as _dt
import json
from collections import Counter

import numpy as np
import keras
from keras import layers
from sklearn.datasets import load_digits

# --------------------------------------------------------------------------
# Hyper-parameters.
#
# The spec's starting point was 60 epochs / width 32. At that setting the
# ASCII map was a single smear — the network had learned *a* 2-D code, but not
# a well-separated one. Following the "raise epochs/width until clusters are
# visible by eye" rule, these are the values that actually produce clusters
# you can read off the terminal. Everything else is unchanged.
# --------------------------------------------------------------------------
SEED = 42
EPOCHS = 240
BATCH_SIZE = 64
HIDDEN_UNITS = 96          # width of the one hidden layer on each side
LATENT_DIM = 2             # the bottleneck. The entire point of this file.

MAP_W, MAP_H = 68, 22      # ASCII scatter-plot grid
RAMP = " .:-=+*#%@"        # 10 shades, dark -> bright
EPS = 1e-9                 # guards the ptp() normalisation against zero range


# ==========================================================================
# 1. DATA
# ==========================================================================
def load_data() -> tuple[np.ndarray, np.ndarray]:
    """1797 handwritten digits, 8x8, greyscale 0..16 -> flattened floats 0..1."""
    digits = load_digits()
    X = (digits.data / 16.0).astype("float32")   # (1797, 64), pixels in [0, 1]

    # ------------------------------------------------------------------
    # LABELS ARE FOR DISPLAY ONLY.
    # y is loaded so the ASCII map can print a '7' where a 7 landed, and so
    # results.json can colour the scatter on the site. Search this file for
    # `y` and you will find it in exactly one more place: the map/JSON code.
    # It never touches the model, the loss, or fit().
    # ------------------------------------------------------------------
    y = digits.target.astype(int)

    return X, y


# ==========================================================================
# 2. ARCHITECTURE — two named Sequentials, composed
# ==========================================================================
def build_models() -> tuple[keras.Model, keras.Model, keras.Model]:
    """encoder: 64 -> 2.  decoder: 2 -> 64.  autoencoder = decoder(encoder(x))."""

    encoder = keras.Sequential(
        [
            keras.Input(shape=(64,)),
            layers.Dense(HIDDEN_UNITS, activation="relu"),
            # THE BOTTLENECK. Two units. No activation, so the code is free to
            # live anywhere in the plane — these two numbers become the (x, y)
            # of every point on the map. Everything the decoder will ever know
            # about this image has to squeeze through here.
            layers.Dense(LATENT_DIM, name="bottleneck"),
        ],
        name="encoder",
    )

    decoder = keras.Sequential(
        [
            keras.Input(shape=(LATENT_DIM,)),
            layers.Dense(HIDDEN_UNITS, activation="relu"),
            # sigmoid, because the targets ARE the inputs and the inputs are
            # pixels in [0, 1]. Squashing the output into the same range as the
            # data means the model never wastes capacity learning "don't emit
            # -3.2 for a black pixel".
            layers.Dense(64, activation="sigmoid"),
        ],
        name="decoder",
    )

    autoencoder = keras.Sequential([encoder, decoder], name="autoencoder")
    return encoder, decoder, autoencoder


# ==========================================================================
# 3. TRAIN
# ==========================================================================
def train(X: np.ndarray) -> tuple[keras.Model, keras.Model, keras.Model, float]:
    keras.utils.set_random_seed(SEED)

    encoder, decoder, autoencoder = build_models()
    autoencoder.compile(optimizer="adam", loss="mse")

    # ------------------------------------------------------------------
    # fit(X, X) IS THE ENTIRE TRICK.
    # Supervised learning is fit(inputs, answers). Here the answer *is* the
    # input: "give me back what I gave you, but you may only remember two
    # numbers about it". No labels, no annotation, no human in the loop —
    # the supervision signal is the data reconstructing itself.
    # ------------------------------------------------------------------
    autoencoder.fit(X, X, epochs=EPOCHS, batch_size=BATCH_SIZE, verbose=0)

    final_mse = float(autoencoder.evaluate(X, X, verbose=0))
    return encoder, decoder, autoencoder, final_mse


# ==========================================================================
# 4. TERMINAL OUTPUT (a): the ASCII map of the learned space
# ==========================================================================
def normalize_coords(Z: np.ndarray) -> np.ndarray:
    """Scale each latent axis into [0, 1]. ptp + eps so a flat axis can't /0."""
    lo = Z.min(axis=0)
    span = np.ptp(Z, axis=0) + EPS
    return (Z - lo) / span


def ascii_map(Z: np.ndarray, y: np.ndarray) -> str:
    """Plot every digit's 2-D code on a MAP_W x MAP_H grid, drawn as its label.

    Where several digits land in the same character cell we print the majority
    label for that cell — that is what makes the clusters legible instead of
    whichever point happened to be plotted last.
    """
    N = normalize_coords(Z)
    cx = np.clip((N[:, 0] * (MAP_W - 1)).round().astype(int), 0, MAP_W - 1)
    cy = np.clip((N[:, 1] * (MAP_H - 1)).round().astype(int), 0, MAP_H - 1)

    cells: dict[tuple[int, int], Counter] = {}
    for x, row, label in zip(cx, cy, y):
        cells.setdefault((x, row), Counter())[int(label)] += 1

    grid = [[" "] * MAP_W for _ in range(MAP_H)]
    for (x, row), counter in cells.items():
        winner = counter.most_common(1)[0][0]
        grid[MAP_H - 1 - row][x] = str(winner)   # flip y so it reads like a plot

    border = "+" + "-" * MAP_W + "+"
    body = "\n".join("|" + "".join(row) + "|" for row in grid)
    return f"{border}\n{body}\n{border}"


# ==========================================================================
# 5. TERMINAL OUTPUT (b): original vs reconstruction, side by side
# ==========================================================================
def ascii_digit(vec: np.ndarray) -> list[str]:
    """One 64-float image -> 8 rows of 16 chars (each pixel doubled for aspect)."""
    rows = []
    for r in range(8):
        line = ""
        for c in range(8):
            v = float(vec[r * 8 + c])
            idx = int(round(min(1.0, max(0.0, v)) * (len(RAMP) - 1)))  # clamped
            line += RAMP[idx] * 2                                       # doubled
        rows.append(line)
    return rows


def side_by_side(original: np.ndarray, recon: np.ndarray, label: int) -> str:
    left, right = ascii_digit(original), ascii_digit(recon)
    head = f"  digit {label}: original".ljust(22) + "  ->  " + "reconstruction"
    body = "\n".join(f"  {a}  ->  {b}" for a, b in zip(left, right))
    return f"{head}\n{body}"


def pick_demo_indices(y: np.ndarray, wanted=(3, 6, 8)) -> list[int]:
    """First occurrence of three visually distinct digits (display choice only)."""
    return [int(np.where(y == w)[0][0]) for w in wanted]


# ==========================================================================
# 6. MAIN
# ==========================================================================
def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument(
        "--json",
        action="store_true",
        help="write results.json (real coordinates from this run) for the site",
    )
    args = parser.parse_args()

    X, y = load_data()
    print(f"data: {X.shape[0]} digits x {X.shape[1]} pixels, range "
          f"[{X.min():.2f}, {X.max():.2f}]")
    print(f"training {EPOCHS} epochs through a {LATENT_DIM}-unit bottleneck "
          f"(labels are NOT used)...")

    encoder, decoder, autoencoder, final_mse = train(X)
    print(f"final reconstruction MSE: {final_mse:.6f}")

    # The learned space. This is the only forward pass that matters.
    Z = encoder.predict(X, verbose=0)
    recon = autoencoder.predict(X, verbose=0)

    print("\nTHE LEARNED SPACE — 1797 digits, each drawn at its own 2 numbers.")
    print("Characters are the true labels, painted on AFTER training, purely so")
    print("you can see that the clusters the network found line up with them.\n")
    print(ascii_map(Z, y))

    print("\nTHROUGH THE BOTTLENECK — original vs. what 2 numbers can rebuild:\n")
    for i in pick_demo_indices(y):
        print(side_by_side(X[i], recon[i], int(y[i])))
        print()

    if args.json:
        payload = {
            "run_date": _dt.datetime.now().astimezone().isoformat(timespec="seconds"),
            "keras_version": keras.__version__,
            "seed": SEED,
            "epochs": EPOCHS,
            "hidden_units": HIDDEN_UNITS,
            "final_mse": round(final_mse, 8),
            "points": [
                {"x": round(float(zx), 5), "y": round(float(zy), 5), "label": int(lbl)}
                for (zx, zy), lbl in zip(Z, y)
            ],
            "recon_demos": [
                {
                    "label": int(y[i]),
                    "original": [round(float(v), 5) for v in X[i]],
                    "reconstructed": [round(float(v), 5) for v in recon[i]],
                }
                for i in pick_demo_indices(y)
            ],
        }
        with open("results.json", "w") as fh:
            json.dump(payload, fh)
        print(f"wrote results.json — {len(payload['points'])} real embedded points, "
              f"{len(payload['recon_demos'])} reconstruction demos")


if __name__ == "__main__":
    main()