Shicheng Wen¹²†, Chufan Shi¹²†, Linghao Jin¹²†, Zhengzhong Liu², Eric Xing², Xuezhe Ma¹²
¹ USC-ISI ² MBZUAI-IFM
† Work done during an internship at MBZUAI-IFM.
<aside> 🌟
Attention has a read side and a write side, but we almost always talk only about the read side. The equation $O=AV$ tells us where each token reads from. It does not tell us what gets written into the context in the first place: whether a token's value should be stored as raw content, or as a correction to what the past already predicts. Standard attention never asks this write-side question. This post makes that question explicit, and shows why the resulting layer is not just another signed-attention trick.
</aside>
For a nonnegative addressing kernel, one row of the causal mixing matrix $A\in\mathbb{R}^{n\times n}$ has the familiar form
\frac{\kappa(q_t,k_j)} {\sum_{i\le t}\kappa(q_t,k_i)}, \qquad j\le t. $$
Each position builds a query, compares it against previous keys, and reads a weighted mixture of values. The matrix $A$ decides where to read from; the matrix $V$ decides what is available to be read.
Viewed this way, attention is already a memory system: tokens write address-content associations, and later queries retrieve from them. This associative-memory framing is useful for comparing softmax attention, linear attention, and delta-style updates (Zhong et al., 2025). These variants differ in how they form the addressing weights, but they share the same read template:
$$ \text{query} \longrightarrow \text{weights over keys} \longrightarrow \text{mixture of values}. $$
The read side is clean. The write side is almost implicit. In ordinary causal attention, each token contributes a key-value pair $(k_t,v_t)$ to the context, and later rows of $A$ decide how to read from the accumulated set. Once token $j$ produces $v_j$, later tokens can decide how strongly to attend to it, but the value itself is not rewritten relative to earlier content.
The attention row answers:
Where should I read?
It does not explicitly answer:
What part of this value is still new relative to the past?
That second question matters when the context contains repeated, overlapping, or partially explained information. Two previous tokens can carry near-identical evidence; a later query can weight them differently, but the read is still a mixture of the original value vectors. The goal is to keep the soft read while making it act on residual innovations instead of raw observations.
The delta rule answers the write-side question directly: before writing the current value, ask what the past already predicts, and write only the residual. DeltaNet makes this idea explicit for linear transformers: a finite-dimensional associative state predicts the value at the current key, and the update writes the prediction error back to the same key direction (Yang et al., 2024).
The same idea can be written directly in attention language. Keep the ordinary causal read matrix $A=[a_{tj}]$, and introduce a second matrix $B=[b_{tj}]$ used only for the residual-writing branch: