Anchor-Bounded Collaborative Attention (ABCA)
This study introduces Anchor tokens — special learned units that delimit zones of local collaborative attention within a sequence. Rather than computing full attention across all token pairs, tokens within a zone resolve their representations jointly, guided only by the two anchors bounding them. The approach reduces attention complexity from O(n²) toward O(k·m²) while preserving — and in some cases improving — semantic coherence across long sequences.
Motivation
Standard self-attention treats every token as equally capable of attending to every other token in the sequence. This is powerful but expensive: the cost scales with the square of the sequence length. A sequence of 4,096 tokens requires roughly 16 million attention computations per head per layer.
Existing sparse attention methods — Longformer, BigBird, Sliding Window Attention — address this by restricting attention to fixed-size local windows. The fundamental problem is that these windows are positional, not semantic. A window of size 64 has no knowledge of whether the content it covers is structurally coherent. It cuts wherever it was told to cut.
ABCA takes a different approach: boundaries are determined by meaning, not position. Anchor tokens act as semantic signposts, and the zones they define are the natural units of collaborative computation.
The Anchor Mechanism
An Anchor token is a special unit in the model's vocabulary — similar in spirit to [CLS] or [SEP] in BERT, but with a distinct role. Anchors are not manually inserted. They are predicted by the model as part of generation or encoding, placed at positions where semantic boundaries are detected.
Once anchors are placed, the sequence is divided into intervals. Each interval is bounded on the left and right by an anchor. The tokens within the interval — which we call interval tokens — only attend to each other and to their two bounding anchors.
The key property: anchors are shared between adjacent zones. Anchor B belongs to both Zone A–B and Zone B–C. This means that while interval tokens are locally isolated, information propagates across the full sequence through the chain of anchors — without requiring global attention.
Collaborative Resolution
The term "collaborative" refers to how the two bounding anchors interact to shape the zone. Before interval tokens begin attending, the two anchors exchange representations across the boundary they share. This exchange updates each anchor's representation to incorporate both what came before and what comes after.
Concretely: Anchor B, having participated in Zone A–B, has accumulated a representation of the left context. Before Zone B–C begins computing, B and C exchange a lightweight attention pass — a boundary handshake. C learns the left context from B; B learns the right context from C. Interval tokens in B–C then attend to these updated anchor representations.
This means the interval tokens, despite never seeing Anchor A or Zone A–B directly, receive an indirect signal of what came before — mediated through Anchor B's updated state.
Complexity Analysis
Let n be the total sequence length, k the number of anchor tokens, and m the average zone size (so that n ≈ k · m).
| Method | Attention complexity | Semantic boundary |
|---|---|---|
| Full self-attention | O(n²) |
None — global |
| Sliding window (Longformer) | O(n · w) |
Positional only |
| BigBird | O(n) |
Random + positional |
| ABCA (this study) | O(k · m²) |
Semantic — anchor-defined |
On a sequence of 4,096 tokens with 16 anchors and average zone size of 256 tokens:
- Full attention — ~16.7M operations
- ABCA — 16 × 256² ≈ ~1.05M operations
A theoretical reduction of approximately 16× on this configuration, with no global attention pass required. The boundary handshakes between adjacent anchors add a negligible O(k²) term.
Anchor Placement
How does the model decide where to place anchors? This is the most open question in ABCA and the primary subject of planned experiments. Three candidate approaches:
Learned placement
The model learns to predict anchor positions during pre-training, supervised by a placement signal derived from syntactic or semantic segmentation of the training corpus. Anchors emerge at clause boundaries, topic shifts, or entity introductions.
Explicit special tokens
Anchor tokens are added explicitly to the input, either by the user or a lightweight preprocessing model. This gives full control but requires an additional pass before encoding. Suitable for structured inputs (documents with sections, code with function boundaries).
Fixed-interval fallback
When no semantic boundary is detectable, anchors fall back to fixed positional placement. This degrades to a form of sliding window attention but ensures the system always has a valid segmentation.
Relation to Study 01 — CPTM
ABCA and CPTM are designed to compose. Super-Tokens from Study 01 are natural candidates for Anchor tokens: they represent semantically dense, high-confidence units that the model has already identified as structurally significant. A Super-Token is, by definition, a unit that carries more meaning per representation than an average token — precisely the property that makes a good anchor.
In a combined CPTM + ABCA architecture, Super-Tokens serve a dual role: they compress the sequence (CPTM) and they define the boundaries of collaborative attention zones (ABCA). The two mechanisms reinforce each other — fewer tokens means fewer zones, and semantically defined zones means cleaner Super-Token boundaries.
This combination is the first architectural stack planned for integration into Matheo AI.
Open Questions
- What is the optimal zone size distribution? Uniform zones or variable-length zones adapted to content density?
- How many boundary handshake passes are needed between adjacent anchors before zone computation begins?
- Can anchors be sparse — present only where needed — while maintaining a valid attention graph?
- Does ABCA degrade gracefully on short sequences where global attention would be equally cheap?
- How does anchor placement interact with tokenization — particularly with Super-Tokens from CPTM?
Status
ABCA is in the conceptual phase. The theoretical complexity analysis above is the primary current contribution. Empirical validation — comparing perplexity, throughput, and long-range coherence against full attention and Longformer baselines — is the planned next step. Results will be published as a supplement to this document.