03Notes & essays

Long-form, low-frequency

Research dispatches, build logs, occasional manifestos against well-loved assumptions. Written in the order I figure things out - not in the order I would tell you to read them.

Published15 min

Note 001 · Architecture

Parallax: cognition without the token clock

What I have been working on for almost two years across three rewrites. A small recurrent core that thinks by settling, language treated as one of several sensory channels, and reasoning depth detached from the token loop. Notes from the v3 codebase and what I am still trying to figure out.

ParallaxCWSAttractor dynamicsDEQArchitecture

Read →

Published8 min

Note 002 · Parallax · K-DOF

The model stops and ponders

One observation from training the v3 CWS that I keep showing people. The model autonomously escalates compute on hard tokens, without ever being told what 'hard' means. Five to six times more reasoning iterations on harder lookups, paid for at constant memory.

ParallaxCWSK-DOFTest-time compute

Read →

Published13 min

Note 003 · Honest log

Parallax: what I got wrong three times

Parallax v3 is the third honest restart. v1 was a RAG replacement that hit a ceiling no projector architecture could lift. v2 was a dual-process system with a memory bank that worked best when it collapsed to a single vector. v3 is what I have now. Notes from getting the project wrong and learning more from each version than from any one of them being right.

ParallaxProcessHonest log

Read →

Published8 min

Note 004 · Surprises

Memory collapse is a feature

Halfway through training a memory-augmented architecture, I watched the entire memory bank collapse to a single vector. I spent a month trying to fix it. Then I looked at the numbers, realised the collapse was actually the optimum, and unlearned a thing I thought I knew about memory.

Parallaxv2SurprisesInductive bias

Read →

Published9 min

Essay · Open problems

Continual learning is the actual hard part

Static benchmarks made us forget that the interesting question was never whether a model can do a task. It was whether a model can learn a new task without forgetting the old one. Here is why I think this is the cliff modern ML is about to walk off, and what would have to change.

Continual learningAdaptive cognitionOpen problems

Read →

Published8 min

Note 002 · Representations

The geometry of grokking

Grokking is the strangest replicable phenomenon in modern ML. A model that has memorized a task for thousands of steps suddenly, with no apparent reason, generalizes. I think the reason is geometric, and I think it tells us something important about what 'understanding' actually is.

GrokkingRepresentationsGeometryTheory

Read →

Published10 min

Essay · World models

Language is just an interface

We treat language models as the substrate. They are not. They are an interface to an underlying world model that the model has been forced to construct in order to predict the next token. The substrate is the world model. Language is one way to query it.

World modelsLLMsMultimodalTheory

Read →

Published9 min

Research log · structural plasticity

Notes on the dendritic unit

An aside from the rest of the lab. I wanted to see what happens when you take the MLP's nonlinearity off the soma and put it inside the unit, where dendrites compete and coactivate. Open notes on what I tried and where it ends up surprising me.

Dendritic UnitStructural plasticityCompetitionMLP

Read →

Working on something adjacent? I like comparing notes.

Get in touch ↗