# Positional encoding in transformer architectures

The advent of the transformer architecture revolutionized machine learning by replacing the sequential processing bottlenecks of Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks with highly parallelized attention mechanisms. However, the core mathematical operation that enables this massive parallelization—scaled dot-product self-attention—presents a fundamental architectural limitation. Attention mechanisms compute relationships between all tokens simultaneously, processing the input sequence not as an ordered progression of data, but as an unordered mathematical set. 

Without a mechanism to explicitly inject sequence order, a transformer is functionally blind to time and syntax, unable to distinguish between identical words in different structural arrangements. Positional encoding serves as the critical solution to this constraint, acting as the mechanism by which transformers comprehend the spatial and temporal relationships between tokens. The evolution of positional encoding—from fixed sinusoidal waves to dynamic, complex-plane rotations—mirrors the broader evolution of large language models (LLMs) toward expansive context windows, highly efficient inference, and robust out-of-distribution generalization.

## Permutation Equivariance and the Self-Attention Mechanism

The necessity of positional encoding stems directly from the algebraic properties of the self-attention mechanism. In a standard transformer, an input sequence of tokens is projected into three distinct vector spaces via learned weight matrices: Queries ($Q$), Keys ($K$), and Values ($V$) [cite: 1, 2, 3]. The attention output is subsequently computed using the following formulation:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

This operation is mathematically defined as permutation equivariant. If the rows of the input sequence matrix are shuffled, the resulting output matrix is shuffled in the exact same permutation, but the semantic representations of the individual tokens remain entirely unchanged [cite: 1, 2, 4]. Because the projection matrices $W_q$, $W_k$, and $W_v$ are applied identically across all sequence positions regardless of their sequential index, the attention mechanism inherently processes text as a "bag of words" [cite: 2, 5, 6, 7]. 

In practical terms, an unencoded model processes the sentence "the cat sat on the mat" identically to "the mat sat on the cat." The model recognizes the vocabulary and the semantic embeddings of the isolated words, but fails to compute the causal and syntactic relationships established by word order [cite: 6, 7, 8]. To break this symmetry, a unique mathematical signature indicating a token's absolute or relative position must be incorporated into the token's representation vector either before or during the attention calculation [cite: 2, 7, 9].

## Absolute Positional Embeddings

Early solutions to the permutation equivariance problem focused on Absolute Positional Embeddings (APE). In this paradigm, a unique vector representing the explicit position index (e.g., position 1, position 2, position $N$) is generated and added directly to the input token embedding prior to the first transformer layer [cite: 1, 7, 9, 10].

### Learned Absolute Positional Embeddings

The most straightforward computational approach, utilized by early architectures such as GPT-2, involves initializing a positional embedding matrix of shape $T \times d_{model}$, where $T$ is the maximum pre-defined context length and $d_{model}$ is the dimension of the semantic token embeddings [cite: 11, 12, 13]. During the training phase, the model treats this matrix as a set of standard network weights, using gradient descent and backpropagation to learn the optimal vector representation for each distinct absolute position [cite: 9, 12, 13].

While empirically effective within the bounds of the training data, learned embeddings suffer from a hard extrapolation limit. If a model is trained with a maximum context window of 1,024 tokens, no positional vector exists in the learned matrix for the 1,025th position [cite: 6, 9, 12]. Consequently, if the model is fed a sequence exceeding its pre-training maximum during inference, the mathematical operations break down, preventing the model from generalizing to longer documents [cite: 9, 10].

### Sinusoidal Positional Encoding

To resolve the constraints of strictly finite learned parameters, Vaswani et al. (2017) introduced Sinusoidal Positional Encoding in the original Transformer architecture. This deterministic, training-free approach assigns a unique positional vector using continuous, interlocking sine and cosine functions operating at varying frequencies [cite: 6, 7, 8].

For a token at position $pos$ and an embedding dimension $i$, the encoding is defined by the following equations:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$
$$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

This continuous formulation functions analogously to a high-dimensional clock or a smoothed binary counter [cite: 8, 14, 15]. Lower dimensions of the embedding vector (governed by high-frequency sine and cosine waves) oscillate rapidly, changing values almost every token, acting like the "second hand" of a clock to capture fine-grained, local positional relationships between adjacent words [cite: 8, 14, 16]. Conversely, higher dimensions (governed by low-frequency waves) oscillate very slowly, capturing global structural awareness and long-range dependencies across the entire sequence [cite: 8, 14, 16]. 

The inclusion of both sine and cosine functions is a deliberate geometric choice rooted in Fourier analysis. By pairing these functions, the model can theoretically represent the positional encoding of any offset position $pos + k$ as a direct linear transformation of the encoding at $pos$ [cite: 14, 17]. The transformation matrix required to shift an encoding by $k$ positions is independent of the absolute position, allowing the attention mechanism to implicitly learn relative distances across the sequence [cite: 14, 15, 17]. 

However, sinusoidal encodings exhibit several critical limitations. Because these positional vectors are added directly to the semantic token embeddings, the model is forced to disentangle "what the token means" from "where the token is" using the exact same vector space, leading to representational crowding and the pollution of semantic data [cite: 8, 18, 19]. Furthermore, empirical studies have demonstrated that standard sinusoidal encodings fail to extrapolate gracefully when evaluated on sequence lengths dramatically longer than those observed during training, as the attention mechanisms struggle to interpret extreme absolute coordinates [cite: 6, 10, 18].

## Relative Positional Embeddings

To address the limitations of absolute indexing, architectural research shifted toward Relative Positional Embeddings (RPE). The RPE paradigm posits that linguistic grammar and logical structures rely far more on the relative distance between two tokens than their absolute, fixed coordinates within a document [cite: 6, 10, 18]. 

A highly influential RPE implementation is the T5 Bias introduced by Raffel et al. (2020), which abandons input-level modifications entirely [cite: 20, 21]. Instead of altering the input embeddings, the model learns a discrete scalar bias for different relative distances and adds these biases directly to the attention logits prior to the softmax operation [cite: 20, 22, 23]. For inference tasks requiring sequences beyond the trained context window, the T5 architecture clamps or clips the relative distance to the maximum distance observed during training, reusing that maximum bias [cite: 9, 20, 22]. 

While T5 Bias and similar RPE mechanisms demonstrably improve long-sequence generalization compared to absolute methods, they introduce severe computational bottlenecks. Because RPE must calculate the precise offset between every pair of queries and keys dynamically across the sequence, it requires the materialization of the full $N \times N$ attention matrix [cite: 21, 24, 25]. This structural requirement renders standard RPE incompatible with modern, efficient, sub-quadratic attention implementations (such as FlashAttention), severely limiting the scalability of these models [cite: 21, 25].

## Rotary Position Embedding

Rotary Position Embedding (RoPE), introduced by Su et al. (2021), has become the de facto standard for modern open-source LLMs, including LLaMA, Qwen, and Mistral [cite: 25, 26, 27, 28]. RoPE elegantly unifies absolute and relative positioning without adding learnable parameters to the token embeddings, and without disrupting the efficiency of the attention calculation [cite: 13, 21].

### Mathematical Formulation and Complex Plane Rotation

Rather than adding a positional vector to the initial token embedding, RoPE modifies the Query ($Q$) and Key ($K$) vectors immediately before the attention dot product is computed [cite: 13, 19, 29].

[image delta #1, 0 bytes]

 It achieves this by splitting the $d$-dimensional embedding vector into $d/2$ discrete pairs. Each pair is treated as a coordinate $(x, y)$ in a two-dimensional complex plane [cite: 13, 18, 29]. 

RoPE then geometrically rotates these 2D vectors by an angle proportional to the token's absolute sequence position $m$ [cite: 21, 29]. The rotation angle relies on a frequency base, typically initialized to $\theta = 10000$ [cite: 26, 30]. The mathematical brilliance of RoPE emerges specifically during the dot product of a query vector at position $m$ and a key vector at position $n$. Because the inner product of two vectors in a 2D plane depends solely on the angle between them, the resulting attention score becomes entirely dependent on the relative distance $(m - n)$ [cite: 6, 21, 29, 30]. 

This property is formally expressed as:

$$\langle f_q(x_m, m), f_k(x_n, n) \rangle = g(x_m, x_n, m - n)$$

Through this transformation, the model computes self-attention with perfect relative awareness while maintaining the sub-quadratic computational profile of an absolute encoding method [cite: 12, 21]. Furthermore, RoPE scales cleanly to multidimensional data required for Vision-Language Models (VLMs) by applying independent rotational frequencies across different axes (e.g., width, height, and time), enabling spatial awareness without altering the core architecture [cite: 6, 28, 29]. 



### Implementation and Extrapolation Limits

In modern deep learning frameworks like PyTorch, RoPE is implemented efficiently using the "rotate-half trick," which negates and swaps pairs of embedding components before applying the sine and cosine transformations, bypassing the need for computationally heavy matrix multiplications [cite: 29]. 

Despite its elegance, vanilla RoPE faces extrapolation limits. If a model trained on 2,048 tokens receives an input of 8,192 tokens, the relative rotations for tokens beyond position 2,048 involve angles $\theta$ that the model has never optimized weights for [cite: 29, 30]. Relying strictly on these out-of-distribution rotation angles causes sequence coherence to degrade rapidly, necessitating the development of post-training context scaling mechanisms [cite: 29, 30].

## Attention with Linear Biases

Attention with Linear Biases (ALiBi), proposed by Press et al. (2021), represents a philosophy of radical architectural minimalism that completely sidesteps the complexities of input embedding alterations and complex-plane rotations [cite: 7, 19]. ALiBi abandons explicit positional embeddings entirely, injecting positional awareness strictly as an inductive recency bias [cite: 13, 19, 31].

### The ALiBi Distance Penalty Mechanism

In ALiBi, no positional parameters are added to the input sequence, and the Query and Key vectors are not rotated. Instead, after the attention matrix is computed via the standard dot product ($QK^T$), ALiBi subtracts a static, non-learned penalty from the resulting attention scores before they pass through the softmax function [cite: 7, 13]. This penalty grows linearly as the physical distance between the query token and the key token increases, effectively forcing the model to prioritize proximate context over distant context [cite: 13, 31]. 

To maintain the capacity for long-range sequence dependency, this distance penalty is scaled by a head-specific slope scalar $m$ [cite: 7, 13]. In a typical multi-head attention setup, this scalar is structured as a geometric sequence starting at $2^{-8/n}$. Consequently, some attention heads possess a steep slope—aggressively penalizing distant tokens to focus narrowly on local grammar—while other heads feature a highly gentle slope, allowing them to retain broad, long-range contextual awareness [cite: 7, 13].

### Extrapolation Capabilities and Known Pathologies

Because the ALiBi penalty is based purely on a continuous, linear relative distance penalty, it excels in zero-shot length extrapolation [cite: 10, 13, 20, 31]. The mathematical operation remains stable regardless of sequence length, allowing models to process text far beyond their training bounds without fine-tuning [cite: 10, 31]. Furthermore, ALiBi is slightly faster to train than RoPE due to the removal of trigonometric computations [cite: 7, 23].

However, the simplicity of ALiBi introduces distinct architectural pathologies. Empirical analyses of ALiBi-based models (such as BLOOM) reveal an "attention sink" phenomenon where the aggressive linear penalties compel a significant portion of attention heads to collapse [cite: 19, 32]. Studies indicate that 31% to 44% of attention heads in ALiBi networks may effectively shut down, directing almost all their probability mass indiscriminately to the Beginning of Sequence (BOS) token [cite: 19]. As the ecosystem evolved, the development of robust RoPE scaling methods ultimately superseded ALiBi's initial extrapolation advantages, establishing RoPE as the preferred standard [cite: 19].

### Taxonomic Comparison of Positional Encoding Frameworks

The selection of a positional encoding framework profoundly influences a transformer's memory footprint, training velocity, and context extrapolation ceiling. 

| Encoding Framework | Injection Point | Core Mathematical Mechanism | Learnable Parameters | Extrapolation Capability | Computational Footprint |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Absolute (Learned)** | Input embeddings | Static parameter lookup | Yes | None (Hard sequence limit) | Minimal |
| **Absolute (Sinusoidal)** | Input embeddings | Interlocking sine/cosine addition | None | Poor (OOD values) | Minimal |
| **Relative (T5 Bias)** | Attention logits | Learned distance-based scalar bias | Yes | Moderate (Clamped at max) | High (Requires $N \times N$ matrix) |
| **Rotary (RoPE)** | Query/Key vectors | 2D complex plane geometric rotation | None | Moderate (Requires scaling) | Minimal (FlashAttention compatible) |
| **Linear Bias (ALiBi)** | Attention logits | Static linear distance penalty | None | Excellent (Zero-shot) | Minimal |

## The Context Extrapolation Challenge

As the demand for processing massive documents, entire codebases, and long-form conversational histories has increased, standard LLMs encounter severe degradation when pushed past their pre-training context windows [cite: 6, 33, 34]. When evaluated on extended sequences, models frequently fail to retrieve information and hallucinate, driven by two primary mathematical breakdowns: Out-of-Distribution (OOD) positional values and Attention Entropy Collapse [cite: 34, 35, 36].

### Out-of-Distribution Positional Values

Transformers optimize their projection weights ($W_q, W_k, W_v$) based on the distribution of data observed during pre-training [cite: 18, 36]. If an LLM is trained on a context window of 4,096 tokens, it learns how to handle the specific rotation angles (in RoPE) or positional vectors (in APE) corresponding to indices 1 through 4,096 [cite: 18, 30, 37]. 

When an inference sequence extends to 10,000 tokens, the model encounters positional variables that lie entirely outside its training manifold [cite: 36, 38]. These out-of-distribution positional inputs generate catastrophic values in the attention dot product, destabilizing the sequence's internal representations and leading to immediate representational collapse [cite: 36, 37, 38, 39]. 

### Attention Dispersion and Entropy Collapse

Even if OOD values are mitigated, models face a secondary limitation embedded in the scaled dot-product attention formula itself. The softmax function normalizes attention scores across all available tokens, ensuring they sum to 1.0 [cite: 35, 40]. 

As the context length $N$ grows exponentially, the attention probability mass is fundamentally forced to spread across a vastly larger set of tokens [cite: 35, 41]. This causes "attention dispersion," where the softmax distribution flattens and approaches uniformity [cite: 35, 41]. Known formally as attention entropy collapse, this dilution prevents the model from sharply focusing on crucial tokens, burying highly relevant information beneath the accumulated mathematical noise of thousands of irrelevant tokens [cite: 35, 40, 41, 42]. Without adjustments, the information entropy of the attention distribution breaks the bounds optimized during the model's training phase [cite: 40, 43].

## Methods for Context Window Scaling

To avoid the prohibitive, multi-million-dollar computational costs required to pre-train LLMs from scratch on 100k+ token sequences, researchers developed a class of post-training scaling techniques. These methods manipulate RoPE frequencies to fit longer sequences into smaller, pre-trained domains with minimal required fine-tuning [cite: 6, 25, 37, 44].

### Position Interpolation and Linear Scaling

The earliest viable intervention was Position Interpolation (PI), which prevents token positions from extrapolating into unknown territory [cite: 45, 46, 47]. Rather than allowing inference to proceed to position 8,192 on a 4,096-trained model, PI linearly scales down the position indices by a scaling factor $s$ (e.g., $m/s$) [cite: 45, 47, 48]. If the target context is doubled, position 8,192 is mathematically compressed and processed as position 4,096 [cite: 30, 47]. 

While PI successfully bounds the positional inputs within the trained distribution, it compresses the entire frequency spectrum uniformly [cite: 30, 46]. This uniform stretching severely distorts the high-frequency RoPE dimensions, which are responsible for discerning precise local grammatical relationships, causing an immediate degradation in short-range reasoning and language modeling perplexity [cite: 6, 42, 46].

### Neural Tangent Kernel (NTK) Aware Scaling

NTK-Aware Scaling resolved the uniform compression flaw by applying non-uniform scaling grounded in signal processing and Neural Tangent Kernel theory [cite: 42, 46, 47]. NTK theory demonstrates that neural networks struggle to learn or recover high-frequency information if it is artificially compressed into low-dimensional spaces [cite: 42, 46]. 

Therefore, NTK-Aware scaling alters the base frequency ($\theta$, typically $10,000$) dynamically based on the scaling factor [cite: 26, 42, 49]. It leaves the fast-rotating, high-frequency dimensions largely uncompressed, preserving local grammatical fidelity. Simultaneously, it applies aggressive interpolation to the slow-rotating, low-frequency dimensions to accommodate the longer global sequence [cite: 42, 48, 49]. 

### YaRN (Piecewise Frequency Scaling)

The evolution of NTK-Aware scaling culminated in YaRN (Yet Another RoPE Extension) [cite: 34, 43]. YaRN systematically segments the RoPE dimensions into three distinct topological groups:
1. **Pre-critical (High Frequency):** Left completely unscaled, allowing the model to extrapolate perfectly for adjacent, high-resolution local patterns [cite: 38, 42, 49].
2. **Post-critical (Low Frequency):** Aggressively interpolated to force long-distance relationships into the bounds of the original context window [cite: 38, 42].
3. **Transition Zone:** Bridged using a smooth ramp function to prevent abrupt mathematical discontinuities between the extrapolated and interpolated dimensions [cite: 42, 43].

Crucially, YaRN simultaneously addresses the Attention Entropy Collapse problem [cite: 40, 42, 43]. YaRN introduces a pre-softmax temperature scaling parameter $t$ [cite: 40, 42]. By artificially altering the attention logits before normalization, YaRN prevents the attention probability mass from dispersing across hundreds of thousands of tokens, keeping the attention entropy in the regime the model was originally trained on [cite: 40, 42, 43]. This allows models to scale up to 128k contexts with fewer than 400 fine-tuning steps [cite: 38, 43, 46].

### LongRoPE: Scaling to 2 Million Tokens

Microsoft Research expanded the frontier of context extrapolation with LongRoPE, a methodology capable of extending context limits to an unprecedented 2 million tokens [cite: 25, 37, 39]. LongRoPE identifies that non-uniformities in positional embeddings exist not just across the embedding dimensions, but also across absolute token positions within the sequence (i.e., early tokens influence the model differently than late tokens) [cite: 37, 50, 51].

LongRoPE employs an efficient evolutionary search algorithm to dynamically locate optimal, highly specific rescaling factors for every RoPE dimension and position [cite: 37, 39, 50]. To stabilize the model at extreme lengths, it utilizes a progressive extension strategy: the model is first fine-tuned lightly at 256k tokens, after which a secondary evolutionary search interpolates the weights out to 2,048k tokens [cite: 37, 39, 51]. Finally, LongRoPE readjusts the scaling parameters on short, 8k sequences to recover any performance lost in standard, short-context tasks, maintaining near-lossless capability across all sequence lengths [cite: 39, 50, 51].

[image delta #2, 0 bytes]





## Systems Implications: KV Caching and Inference Latency

The choice of positional encoding has profound implications for hardware efficiency during autoregressive generation, particularly regarding the Key-Value (KV) cache [cite: 52, 53, 54].

### KV Cache Dynamics in LLM Inference

Autoregressive inference operates in two phases: the prefill phase (processing the input prompt in parallel) and the decode phase (generating new tokens sequentially) [cite: 52, 54]. To prevent the LLM from redundantly recalculating the attention states for historical tokens during the decode phase, the system stores previously computed Key ($K$) and Value ($V$) matrices in the GPU's memory as the KV cache [cite: 52, 53, 55]. This optimization reduces generation complexity from $O(n^2)$ to $O(n)$, drastically accelerating inter-token latency (ITL) [cite: 53, 54, 55]. For instance, decoding a 4,096-token sequence is functionally 20 to 40 times faster with KV caching enabled [cite: 54].

Absolute Positional Encodings (APE) are highly compatible with simple KV caching [cite: 54]. Because APE adds positional data to the token embedding before entering the attention layer, the $K$ and $V$ matrices inherently contain static positional information [cite: 54, 56]. As new tokens are generated, the absolute position of earlier tokens never changes, meaning the cached vectors remain permanently valid and addressable [cite: 56, 57].

### Positional Disruption in Cache Eviction

Conversely, relative methods, specifically RoPE, severely complicate cache management during extended generation and multi-turn conversations [cite: 54, 58]. RoPE dynamically rotates the $Q$ and $K$ vectors at query time based on their explicit sequence index [cite: 54]. 

When an LLM engages in an infinite-context scenario where the accumulated token count exceeds the physical bounds of the GPU VRAM, the serving engine must evict older tokens from the cache [cite: 53, 58, 59]. Common token eviction strategies (such as AttentionTop, which attempts to retain the top 99% of semantically important tokens) often delete non-contiguous tokens from the middle of the sequence [cite: 55, 59]. 

This eviction fundamentally alters the relative sequence indices for all subsequent tokens [cite: 59]. Because RoPE relies strictly on the linear distance $(m-n)$ to calculate attention, compacting the cache scrambles the rotational angles [cite: 59]. The model attempts to attend to a cached token using an outdated relative rotation, leading to catastrophic cache invalidation, model confusion, and degenerative text output [cite: 58, 59]. 

### Advanced Cache Decoupling and Compression

To mitigate caching constraints in RoPE-based models, engineers utilize advanced decoupling mechanisms. Frameworks like CachedAttention explicitly decouple the positional encoding from the KV caches when saving them to secondary storage, recalculating and re-embedding the RoPE rotations dynamically upon retrieval to prevent invalidation [cite: 58]. 

Furthermore, architectural optimizations like Multi-Head Latent Attention (MLA), deployed in models like DeepSeek-V2, apply low-rank compression to the KV cache while decoupling the rotary position embeddings from the head-specific KV storage, dramatically reducing memory footprint while maintaining the relative positional awareness critical for high-fidelity generation [cite: 60].

| Caching Metric | Baseline (Uncached) | Standard KV Cache | Impact of KV Eviction on RoPE |
| :--- | :--- | :--- | :--- |
| **Time to First Token (TTFT)** | 20-50ms (Prompt dominated) | 20-50ms (Unchanged) | N/A |
| **Inter-Token Latency (ITL)** | Scales linearly $O(n)$ | Constant time $O(1)$ | Spikes significantly upon invalidation |
| **VRAM Footprint** | Low (Discarded dynamically) | High (Grows unboundedly) | Fluctuates based on eviction policy |
| **Positional Stability** | Perfect (Recalculated every step) | Perfect for APE; Complex for RoPE | Severe degradation if non-contiguous |

## Out-of-Distribution Generalization and Algorithmic Reasoning

Positional encoding mechanisms also dictate an LLM's capacity for out-of-distribution (OOD) generalization, particularly in environments requiring algorithmic logic and symbolic reasoning [cite: 36, 61]. Standard transformers are generally capable of solving complex tasks within the data distributions they were trained on, but often fail when presented with variables or lengths outside that distribution [cite: 36, 62, 63].

Recent theoretical proofs demonstrate that decoupling attention weights from semantic values and relying heavily on positional attention—where attention is driven predominantly by positional encodings rather than the token content—significantly enhances a model's ability to extrapolate logical rules [cite: 63]. By utilizing fixed positional encodings across layers, transformers can effectively simulate arbitrary algorithms defined in parallel computation models, allowing them to solve synthetic algorithmic tasks (like sorting or copying) on OOD datasets [cite: 63]. 

This behavior is observable in "Induction Heads," which are specialized two-layer attention circuits within the transformer [cite: 32, 36]. The first layer utilizes positional encodings to shift its attention to an adjacent or previous token, storing historical context. The second layer uses that positional context to identify repeating patterns and predict the next token algorithmically, rather than relying strictly on semantic memory [cite: 32, 36].

## Architectural Alternatives to Explicit Encoding

Given the complexities of length extrapolation, rotation scaling, and KV cache invalidation, a growing body of research explores architectures that bypass explicit positional encodings entirely. 

### Transformers Without Positional Encoding (NoPE)

Studies evaluating decoder-only causal language models (such as the GPT family) have demonstrated that models trained entirely without explicit positional encodings (NoPE) can still generate coherent text and deduce structural order [cite: 3, 64, 65, 66]. 

This phenomenon occurs due to the causal attention mask, a standard component in autoregressive generation that prevents tokens from "looking forward" into the future to predict the current token [cite: 65, 66]. This mask creates an inherently asymmetric flow of information [cite: 65]. By implicitly analyzing the number of predecessor tokens available to attend to, the neural network derives a functional approximation of absolute position [cite: 64, 65, 66]. 

In certain synthetic length-extrapolation benchmarks, NoPE models have occasionally outperformed APE, RoPE, and ALiBi, largely because NoPE is not constrained by out-of-distribution mathematical values (like unknown rotation angles) [cite: 66]. However, this implicit positional awareness is generally insufficient for complex, broad-domain reasoning tasks, and explicit encodings remain mandatory for non-causal Encoder-Decoder architectures (like BERT or T5), which do not utilize causal masking [cite: 44].

### Selective State Space Models (Mamba)

The most aggressive departure from the positional encoding paradigm is found in modern selective State Space Models (SSMs) like Mamba [cite: 24, 67, 68]. Mamba architectures completely discard the $O(n^2)$ scaled dot-product self-attention mechanism, replacing it with a selective state-space update that processes sequence tokens in linear $O(n)$ time [cite: 24, 67]. 

Because Mamba sequentially updates a fixed-size hidden state vector (a Markovian representation of the past context), the architecture possesses an intrinsic, structural awareness of time and sequential order [cite: 67, 68]. Consequently, selective SSMs do not require explicit positional encodings, RoPE scaling tricks, or complex relative bias mechanisms [cite: 24, 67]. This fundamental architectural shift allows models like Mamba to seamlessly process sequences spanning millions of tokens—such as high-resolution genomics data or continuous multi-hour audio streams—bypassing the out-of-distribution extrapolation limits and KV-cache explosions that plague traditional transformer arrays [cite: 24, 68].

## Future Outlook

The trajectory of positional encoding underscores a central theme in deep learning architecture: the transition from rigid, absolute memorization to fluid, relational representation. The dominance of Rotary Position Embedding (RoPE) and Attention with Linear Biases (ALiBi) validates the hypothesis that language and logic are defined fundamentally by relative distances rather than absolute coordinates. 

Moving forward, the architectural bottleneck is no longer simply injecting positional information, but maintaining the thermodynamic stability of the attention matrix at extreme scales. As industry models push beyond multi-million token contexts, methodologies like YaRN and LongRoPE—which selectively manipulate frequencies to preserve local grammar while compressing global structure, combined with entropy-stabilizing temperature scaling—will dictate the immediate future of LLM context extension. Simultaneously, the systemic hardware challenges surrounding KV-cache invalidation and the rise of inherently sequential State Space Models suggest that the ultimate solution to sequence awareness may eventually involve abandoning the $O(n^2)$ attention paradigm entirely.

## Sources
1. [generativeai.pub](https://generativeai.pub/input-embeddings-positional-encoding-the-forgotten-foundations-of-transformers-e44e3f671f62)
2. [aclanthology.org](https://aclanthology.org/2021.emnlp-main.236.pdf)
3. [www.youtube.com](https://www.youtube.com/watch?v=6NPt3Vh7xRo)
4. [www.reddit.com](https://www.reddit.com/r/deeplearning/comments/1ew4dv4/transformers_without_positional_encodings/)
5. [arxiv.org](https://arxiv.org/html/2304.07735v2)
6. [galileo.ai](https://galileo.ai/blog/mamba-linear-scaling-transformers)
7. [athekunal.medium.com](https://athekunal.medium.com/mamba-and-state-space-models-explained-b1bf3cb3bb77)
8. [newsletter.maartengrootendorst.com](https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state)
9. [arxiv.org](https://arxiv.org/abs/2203.16634)
10. [huggingface.co](https://huggingface.co/papers/2203.16634)
11. [arxiv.org](https://arxiv.org/html/2503.08067v1)
12. [aclanthology.org](https://aclanthology.org/2025.emnlp-main.1545.pdf)
13. [ofir.io](https://ofir.io/The-Use-Case-for-Relative-Position-Embeddings/)
14. [sambanova.ai](https://sambanova.ai/blog/alibi-interpolation-vs-extrapolation)
15. [iclr-blogposts.github.io](https://iclr-blogposts.github.io/2025/blog/positional-embedding/)
16. [aclanthology.org](https://aclanthology.org/2024.lrec-main.1478.pdf)
17. [arxiv.org](https://arxiv.org/abs/2402.14202)
18. [towardsdatascience.com](https://towardsdatascience.com/positional-embeddings-in-transformers-a-math-guide-to-rope-alibi/)
19. [direct.mit.edu](https://direct.mit.edu/coli/article/48/3/733/111478/Position-Information-in-Transformers-An-Overview)
20. [azizbelaweid.substack.com](https://azizbelaweid.substack.com/p/complete-summary-of-absolute-relative)
21. [iaee.substack.com](https://iaee.substack.com/p/positional-encoding-intuitively-and)
22. [towardsdatascience.com](https://towardsdatascience.com/rope-clearly-explained/)
23. [medium.com](https://medium.com/@arturfarriols/rope-made-simple-6878c04c0fab)
24. [huggingface.co](https://huggingface.co/docs/transformers/internal/rope_utils)
25. [blog.eleuther.ai](https://blog.eleuther.ai/rotary-embeddings/)
26. [www.graphcore.ai](https://www.graphcore.ai/posts/longer-ropes-better-quantisation-papers-of-the-month-feb-2024)
27. [arxiv.org](https://arxiv.org/html/2504.12637v1)
28. [arxiv.org](https://arxiv.org/pdf/2403.00071)
29. [aclanthology.org](https://aclanthology.org/2024.findings-acl.32.pdf)
30. [medium.com](https://medium.com/@cenghanbayram35/yarn-extending-rope-without-breaking-it-ab07882b1581)
31. [towardsdatascience.com](https://towardsdatascience.com/extending-context-length-in-large-language-models-74e59201b51f/)
32. [medium.com](https://medium.com/generative-ai-playbook/generative-ai-foundations-attention-context-windows-and-llm-scaling-strategies-0f2768f2078c)
33. [arxiv.org](https://arxiv.org/html/2402.02244v2)
34. [lims.ac.uk](https://lims.ac.uk/documents/paper-beyond-attention-breaking-the-limits-of-transformer-context-length-with-recurrent-memory.pdf)
35. [arxiv.org](https://arxiv.org/html/2412.12465v1)
36. [cs.iit.edu](http://cs.iit.edu/~scs/assets/files/ye2025unboxkv_io.pdf)
37. [discuss.huggingface.co](https://discuss.huggingface.co/t/isnt-kv-cache-influenced-by-position-encoding-in-inference/63225)
38. [arxiv.org](https://arxiv.org/pdf/2403.19708?)
39. [www.reddit.com](https://www.reddit.com/r/learnmachinelearning/comments/1j8671r/how_positional_encoding_affects_kv_caching/)
40. [arxiv.org](https://arxiv.org/html/2412.19442v2)
41. [medium.com](https://medium.com/data-science/beyond-attention-how-advanced-positional-embedding-methods-improve-upon-the-original-transformers-90380b74d324)
42. [blog.gopenai.com](https://blog.gopenai.com/learned-vs-sinusoidal-vs-rope-vs-alibi-which-positional-embedding-method-works-best-a94ce5638e64)
43. [towardsdatascience.com](https://towardsdatascience.com/positional-embeddings-in-transformers-a-math-guide-to-rope-alibi/)
44. [iclr-blogposts.github.io](https://iclr-blogposts.github.io/2025/blog/positional-embedding/)
45. [arxiv.org](https://arxiv.org/html/2408.11382v1)
46. [medium.com](https://medium.com/@adnaan525/the-intuition-behind-sinusoidal-positional-encoding-in-transformers-ac13268a60b4)
47. [mesuvash.github.io](https://mesuvash.github.io/blog/2026/positional-encodings/)
48. [kazemnejad.com](https://kazemnejad.com/blog/transformer_architecture_positional_encoding/)
49. [www.youtube.com](https://www.youtube.com/watch?v=dWkm4nFikgM)
50. [www.youtube.com](https://www.youtube.com/watch?v=BkyEZwAf-Rw)
51. [arxiv.org](https://arxiv.org/html/2501.08570v2)
52. [xiaolishen.medium.com](https://xiaolishen.medium.com/hands-on-transformer-deep-dive-part-4-yarn-yet-another-rope-extension-8b7f769eff9e)
53. [www.emergentmind.com](https://www.emergentmind.com/topics/yarn-yet-another-rope-extension-method)
54. [arxiv.org](https://arxiv.org/html/2506.16640v2)
55. [openreview.net](https://openreview.net/forum?id=7SLtElfqCW)
56. [www.youtube.com](https://www.youtube.com/watch?v=PwB8rEcq4FE)
57. [blog.eleuther.ai](https://blog.eleuther.ai/yarn/)
58. [medium.com](https://medium.com/@zaiinn440/linear-rope-vs-ntk-vs-yarn-vs-cope-d33587ddfd35)
59. [saraswatmks.github.io](https://saraswatmks.github.io/2025/12/rope-scaling-llms.html)
60. [pmc.ncbi.nlm.nih.gov](https://pmc.ncbi.nlm.nih.gov/articles/PMC11831214/)
61. [www.researchgate.net](https://www.researchgate.net/figure/a-Attention-distribution-is-composed-of-the-summation-of-cross-attention-on-the_fig1_356283498)
62. [www.reddit.com](https://www.reddit.com/r/MachineLearning/comments/1b1he3r/d_isnt_the_idea_of_generalizing_outside_of_the/)
63. [proceedings.mlr.press](https://proceedings.mlr.press/v139/krueger21a/krueger21a.pdf)
64. [wujcan.github.io](https://wujcan.github.io/papers/tkdd24-cal+.pdf)
65. [www.emergentmind.com](https://www.emergentmind.com/topics/qwen-2-5-models)
66. [thesalt.substack.com](https://thesalt.substack.com/p/longrope-towards-unlimited-context)
67. [arxiv.org](https://arxiv.org/html/2510.23095v1)
68. [amaarora.github.io](https://amaarora.github.io/posts/2025-09-21-rope-context-extension.html)
69. [www.microsoft.com](https://www.microsoft.com/en-us/research/publication/longrope-extending-llm-context-window-beyond-2-million-tokens/)
70. [syncedreview.com](https://syncedreview.com/2024/02/25/microsofts-longrope-breaks-the-limit-of-context-window-of-llms-extents-it-to-2-million-tokens/)
71. [icml.cc](https://icml.cc/media/icml-2024/Slides/34166.pdf)
72. [github.com](https://github.com/microsoft/LongRoPE)
73. [raw.githubusercontent.com](https://raw.githubusercontent.com/mlresearch/v235/main/assets/ding24i/ding24i.pdf)
74. [www.google.com](https://www.google.com/search?q=time+in+CN)
75. [pub.towardsai.net](https://pub.towardsai.net/kv-cache-the-key-to-efficient-llm-inference-7260a504efed)
76. [www.reddit.com](https://www.reddit.com/r/learnmachinelearning/comments/1j8671r/how_positional_encoding_affects_kv_caching/)
77. [arxiv.org](https://arxiv.org/html/2511.04686v1)
78. [openreview.net](https://openreview.net/pdf?id=ZpgyPxdxiL)
79. [pyimagesearch.com](https://pyimagesearch.com/2025/10/13/kv-cache-optimization-via-multi-head-latent-attention/)
80. [pmc.ncbi.nlm.nih.gov](https://pmc.ncbi.nlm.nih.gov/articles/PMC11831214/)
81. [openreview.net](https://openreview.net/forum?id=NmFt9dIrSi)
82. [data-processing.club](https://data-processing.club/heads/)
83. [proceedings.mlr.press](https://proceedings.mlr.press/v139/krueger21a/krueger21a.pdf)
84. [proceedings.neurips.cc](https://proceedings.neurips.cc/paper_files/paper/2023/file/46d943bc6a15a57c923829efc0db7c7a-Paper-Conference.pdf)
85. [medium.com](https://medium.com/@wasowski.jarek/s01e08-one-formula-that-powers-90-of-models-rope-and-alibi-bb025588caee)
86. [blog.gopenai.com](https://blog.gopenai.com/learned-vs-sinusoidal-vs-rope-vs-alibi-which-positional-embedding-method-works-best-a94ce5638e64)
87. [towardsdatascience.com](https://towardsdatascience.com/positional-embeddings-in-transformers-a-math-guide-to-rope-alibi/)
88. [iclr-blogposts.github.io](https://iclr-blogposts.github.io/2025/blog/positional-embedding/)
89. [openreview.net](https://openreview.net/forum?id=Drrl2gcjzl)

**Sources:**
1. [aclanthology.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEvpt0Q0yVxTGMZR6kYkGWzVtuJGxkbigBbws2zW8a9Z2fkMbDbRb5g7_UCnEO7mSQ_qtZ9El1iZAoPueKznKJaAhcocVfFL9pl9d5qmtNGzGqC9O_kTmwItz36I4EYrH7mKQCOHHU=)
2. [reddit.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEiE7vQ1AzTvLuc8pGmJ3kVRUXWxPnjAs43c0ibWqKaZ24uMRb-9MMTlURiK0CSeThqJzbVUGvb6ReoHkz1Gi4CEhWpXW-cC2I5EKSvWuBTYdW_RLW9BUE8Cn4m4ewxQUjFEY1eOMQ_KByQMs__tW4aASqvh9z_VuZ-etjZBfh4sDeSymdgK984CgygqABFch76RBez1XVL)
3. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQH3FXec7wIYBgpIL3OG8hVurt84R3YKY7ZL73FWSv487TpqHA_yzsgUY9FUan4duISd3l5Aj9eCF4whxFSCaiCzDAPs5aRCwg9DpsgvMoUOeAWVm07kTN9lkg==)
4. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHVXJLyLwbXIq5vSoMzPwhNDr357n7oe8y96QGAkja5fbS_B64ccWG2KQRYMTuq1gebymuNOu4OZfh9EZFDIiNElQn9Bf5Lf8gVwZxAWXW-sQYpnS7nI3q4hQ==)
5. [generativeai.pub](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGu6EUJNeKxUUWSrSlEq5HCkORlHfRNIQMvvAH8-pHY4LwvolwbqGSoIBtirU_T7gFq6aY1DFccxt4Rbm1fFSaEcWDHU0qHOhi5rRJLTn4P9sppgZfiWfJucdiLNcuLNv6AKCzFKFnE-7axJQKSppgg0vRaYEjNEkk-CmUujVsChDHCdAwQt5miBH8BkO6vK7rVXt__IF7WryisCktcl1LQIHOvxXdXXSU3sw==)
6. [youtube.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG7UTA1hKauyH5g4i2_gFAZ_jfjQuWru49W4pWMQUp4mcKiccfDggDsb7noSpJn384F1hr1j8HWUByjeALQhsVBK4TIGt3FGBSf-JSnSz2ymRcLvJRvU_tW9F4VglXWnhp0)
7. [towardsdatascience.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHXiKmCXI2TaE7cbPydBJDkOpgPruE-0qoC3DzcuKTIJebOECQZug3AtrWKd1m4f8y6bo_co3s5griDvatc-7xDaVxCxU0DtCA2P2-HrvwejazqjbsWOzceF8Tp4VLmWaUNxUYhA-TmI4pLTTk7ezwLq6bg4TmpEFdmPPD6PL2dLdTpuGwy-5EylqU_eNo57tK6qMOzxIU=)
8. [github.io](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFFPHyW2At75MfF6I399n3iRbd5y1IX90V7TMCKelYqj_LLIDEckL2VI8ohoUdjheHs4rY77jPetvoYCibkxHqZVpMs7LJe59lF2kajyTMz_NXfuWZMhqJEZX8R6fyPQjjrcgedOcnmMxl6CL6oYjtF)
9. [substack.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFqbtdQy0b2tU3vXGz-ThOGO9QK_ATb_nITIhVt3256gGM6EG2w2eLS4pFsFUTGoJw1wCTfHq-8n_L7QGqNxBMk-2SE3DGWCAi06lGAPUYNAnXdTq9jtNslc2Gjz16RFvi3ODBCt9YTAyCuz1GDrHxzryBOwGba4S2lPw5g6yD6)
10. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFvmNK8O6qp_kX2iRFfOyTqd2CqRn7Z-_l6tuo5Dkr1mK0HuSfaTuQMcPvm3VnEXpxYgxO-jb-ea_jmaeNcXf4OGxfpJuEL-2FJpWDBjKhl7owu-1nP6G__NVOLzhV9HbN5vNqiZcT-5F-OGLqvVW-atAG-hvS9EPjxgs6tv_tf8XpP4bAnSy72BLtTNvgZBpS5v724qGZQnWZhi8l9_hb6R5BHJd0g7mIb-Oj4erN_trJoHzi902rvJfzSuWLeS0kYyMaY7w==)
11. [sambanova.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGmz39WBHLkLL2j7sZKDUbWdvzfU1KHwqNDvIlIgvcgg8O-kqFZka1hnjzrJjN_aQ7_cOXgzOMIrCevfN4UCAk0UqKTq5I_dXhYeFKXsxMoxj86iP7Etj17wgt_wAGPLpOU1ScZ-BDpCyjCUGVOu9F_cEzsOg==)
12. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEd1eCNui4WxjKJJdVV7u_5O4jG4a8rp5LEHS-E9K7WJaK2K3pWdLxnG6ekaCmyMW2gmhtskdw3GeDc_R8IXYFM3dkKpoX46RvLs9-QJihWv5tjrCh9pSxKJQXwyT9ESQL4y4VYx0AoX0UzbSwm9pKtwjHWnjM=)
13. [gopenai.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGu4lbRdzQIAp4e74I_8oyfCpaT0QrjgFrdLrE5TwOrHLna8dnGZAmV5Wfov9S1LaZmh9BumnxzKipidparlwuc0cUeM8ftyf2FExHladt3k2qhhYeBpa1aFhg3x2NSXO8_H4w2z0njqUPZ93uHe-Z0u9JAG8rilJO1mI14cc3yj-nmxte0Q1soMbb9msgvdOtYUWoNSj7WVRuUE19rM3SFCzim1C9kMzO3ME50TD8W)
14. [youtube.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHRPvfKa_9INgLeJYpTYHCA44dJjxh4YXekdPbDTkDQYbYVmGnYoGBdyabxAX5tKGt6kTTNKD0DccLxEv6wIysmf26G8fap-j9OGCv3noMVcuYu3krcua0SvOo9PvVQq7aD)
15. [youtube.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG8eZ5hJthsUuEK2dRfWr94eqNeC3UGpYlgWbeVaq1Nk6eAeUB17m_hvyYBj9qFsWSuOqIxQ16D_ZohrTfqKdjmO38A4R3ynQZE5G44ssyaqFGR8Aznwq_LYe1yC_K6ibYn)
16. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHsopA-Wjj-d8ISshqs5msZhzWRos4hY9dpDjkCN-HxShryaECx4jhCFaiwuxqRr8MT4BXiVcL0U2hjCrCgjO1mGPiNhWfmuml-jRRZlIe_ZZxA6u-G5kyc1ZKnVFn5pqplKwEpfjyJ4zZr5BBZuGJ4BMpN_TWjT6XKR6oZU9lLMiX-OHWH4RHkTyQiMKpc-iIWC91Q1tssUr4oSxvFsZnlf0-09w==)
17. [kazemnejad.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGsfmiLoHK46orTcTYI0IRdRpERWLuMVbcOaDFgvS0ZTc997pvufzuwknxKQPj_NQa7VBHQgcE9LJv-qsyPQCma280EKrunB4EP_uwSWLwvRsVhMuHUsam0Dgz7vwnmeS2MttnsoHetVLwv4_Anza3HVLX0UOM4ooPI-Tmvi56_)
18. [towardsdatascience.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEdauGhW4IAFl-V2MqI2kcyfVrvf3QLZOHjEbHXsbHyl_aToTo3_XwxU9qLQInlZ4WrQGs1u5jX2M85GK2Ps2YBvuERRztvatRoJGpZhAUI-MVXjuJ_WZ4-TuI_ZaCcNKbEvdBI0dVS0hebtSs=)
19. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEVEF5YPiFvLuenW6FYD0I4yj7InNW6V5qi-D1HU2gQ2br-5HtskPQXeCgI4nh764JUkZqQBDJ3PQISmAaUiSOVDoOMxbnYyWNImeDCAwjMkaFAUzbpGgBPxGRxVhbtc4OgMWwvbJLAdUq3W9DLs9AYjLExU_17pMU-cnqIk6NOSB3_41ZforqSjUeYB2fckmvC0_nr2PPe4TzRinS29wTQ)
20. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHevsv2FMYBGBZxP3Rkl6LmpD52qXwjJ7hb8qkIASHNfgVgq7piKZO0zekamy8HBTS53JjjIJceeSemQDeHagCteb3rM68XaWTMCqdJOwfOa3vOC7DaC0FAUw==)
21. [eleuther.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFDRITVQDLc4fL5esLMjVvIlijVC--cIvDJ00SFHIhONDa_NRjYouEocl3BUSZmn_oQ7fxipXVjmPkz5QCG8Gvr3tcK7-aRPR7BwZRn8LHe58j2U4fIQPIzeVNytr5rtF35)
22. [aclanthology.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEYrUA50kEfZJB1jpGWbdn-r8oCPm1FfaSIilnoWa4WtoIYhV5XPY8JXIKYK5hcAGl1-zdLRhhuyjkx84Vhg7Ml4l-rjHe4Ypgalu6qMALy01ziiiV8o7K2Nwe1G_53K3wadYOs7kVn)
23. [ofir.io](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHgkgEbtPsS8y13bxCUwkqyi0AdIg5EZT4uwRxr0jVL1NvDkcsVTZI5TRvoT-KwecVhDW-J6C0z-KT9iAALoPe2BB9YKoYyUDYoqlCA2zgZuh2-kZdwknLgUgpz0xkvUzgvgA_GRDRzMThvykPL8DuELpaO-g==)
24. [galileo.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF9vwf8kPic89UtL-45X5SiCzYnuMRm4b9CCaw-M_1WMT8FPilOQERYGYLAG8N5gXNtKtVYx_K72mNkbVasGBQr6feeSpQBi1CPUEBsRJqmMmmEbDQ_Qtm3WH7ll9Z_tNsyivs-CjyBlBY9bsx7PgI=)
25. [substack.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFyI4iDJNipYlfSImus3nSFBH4lRGO0DtZwf0ufyOw1kPuDzlGOZ1xpw_LuYp-8H190_0pqbRSwABro1jq0U_vk51WU0IGE6sASPIpVSU8XEwajxHJ6POPZB9Fp79T8DYBhRYgEb8W5RT2av0Tal0S3n4QM1Nvsxg==)
26. [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG0qX6Otc23rCAB3uPYIdQhpzVyXpg-S1vVF7nxzKeG3peyTUA3yXrgqyN9pCtMj3A2r0bYPhQKTRJ9MfKI_kEJx-QXc6eg8TO3pbWzIL3me3O_uFFyBHFC3MT0l5iL6s8mDgV4TZZr7ySfATZKu0Tlcv8=)
27. [emergentmind.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG0lyoSzk0xGSXLRG_z1J_gV4cjrMHxK6JU2FI-gdKbk03Np2DNL9fN_jRIoYQWV9bT_BDfFDu4ASBbvlCX5gIjWWMafqTOYnMk3W02HpJGtg7DVDnTWpNq67wR0ZgbgaDhuCzFpd6tmCo=)
28. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFhnLp3APzLaK4erbguoSanfWeFQLOfPIIX3wiCdE5eI6rL9E_qNI36yB7HTNGBKYLvkFt_2nkm8jbuBxg0BpyYg_tRFHcvZ4T9xEgzjQzzuLwcRuzuoj6x-A==)
29. [youtube.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHG2dAGCZwShJfgeQCWQ8uRxwQ0yaLA60XtYqOOS6AGkr94qEwVnp9A-cOS9t59hWeEcQHIogk84pNgNEHEhob-wPMRPpblKuQFsEwMpV2yhv4WEHJ01zAj-NvruEfrP-o0)
30. [github.io](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQE58ywnNhPorRECdIU0ya_A1vAvsgO5f3tuerfLrggnv_X-69mNEEzEhXV5cyBdgpnTaXY_jvJEtK9QzPvKGZMywmJKP2n8YNJOse4FN-U17-RNbxOkg-ehgCHrqhadKKbN5mN2_GurJfbZd7OtqxDuRm4=)
31. [github.io](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEUUQ8rfLeRwWRCo-22Kt1k9z4WfKkpi_4beUNxLGrCmA-3NrIFkrVLJdjiCChkViNhY5mHmgplhy2xaosjpZWsAJ08p9wtdwH2QWfLh-cz1qZHsX_jv1YZ4ZHlYJBqXi6mqPCIJhBi2wXARwO8gXCV9eTcSQk9)
32. [data-processing.club](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF7-Dlg1Rz0ooBpGwtxq4FWZCrtz_sg5UjBTuAcVOeQIJl44jprQUEDEjsyeYI4Ku2iX6gsBSvTYIVtTJWglMkxhq7Jo3KNiBWedZ4OAckfZlDrWu-5Mg3kEg==)
33. [towardsdatascience.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEla5nse7jkxfHsF57a6ddXvzqlfhUb-Dx724YyFK5WTs8_Y7Z3ktYB9GOh3xzkOMuSpfFuXmfa4gIMSG88zC7x0OKlKSYSpMJR6Gh9pCayg-SSlUYrr65karJXdqoiMJv6o3p7FmGjeMb6CUJnu-wlWx_pkHEhHhTq6Xste5pEFbIT39csm40P3gSJ6-t8URwltK6a)
34. [github.io](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGMMvkwYZCJZlj_SmmWSumfIyQ0PkpsXzXEi6xSBz0s9xaLTlT3oUjPkDgYShHtF5JEfYSCtg3GSAwN15SbOQYUJeKbhOX2T9VKVsgizsvWq5Xu1AYG1NdsomU9syamsQt1Dk5kS0UcwEwMrFDhzwj8BwvcQ0ninepgJrANhA==)
35. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEqYBnd6y7OJRiCNnXoYu9UD5F_xOs_aOLM5aUAjHKXb1-GPJ9NDeFcGP2kb3ndeb7gbFqxwxG2_8SK3djaX4dWdaFEfxo9YoLrDXF60Hax-CV12PCgrGVXiw==)
36. [nih.gov](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHr7PxytwoTFq_VZKYgoaKHIpVCES514Kqq5uyPlHzhTecZxa-LpgsGblXroI4zwc5HM9hcFKeXp_WPPVibkQzwY80SwcKwgrFVXEa9dj978qufyofhIPisL114xY0JNULXSXSpSA4JRA==)
37. [syncedreview.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEZQLIqAw2DaOaqnKh6dH1Vt0WnbUDRX37zLNwWc1xobtMLTkz9lP5yWioPp2iTtu8R36PdYLITEbH1o30Wupt-PPtUx7t_5cYoMme8k6J_3qEQmkIUZ8MnpUo42qrDoIiiuyH8lYqmytU1h7-xDsbVv66cmj5h8SKG0sLODKfntbx4QoUNKne_lRXJdkh83iEb4C7lTci69x9gQ0TcDZKFlJETl7wrWauOiq8ChssqW1omyHioNm1XPg==)
38. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQH9lUbjc9dS_8pSc-MjEut9w7RatiXh_1LTjMMTHPnBOpcWNjPi38ZHoFdj70wW6ggq82MLudTB4dOF2OByzS6STLVVnQSVCCBvZi7bZJUo8QGTCbu7RA==)
39. [microsoft.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF0TTKgWGnaNf3FaNvpHPkJjhuHeALGwLsQ3DyI3XX5qs-ykXsSX7mKyX7qvKL0Zqw1NTWDwMzDOB0pWDg5QAb2qCfyZ60CvTZ5hI3kf7qBlEkMBddgcYDBWDQa1ho8wqG4EkUR5MntkJn29PMVWdbrSUwwy12ufN4xbqe8bwcb3Pj9zGXpE55zVwuwnJCnHbG5CZDqUuq-60UrugnyI3GS2SMdESd-ttXq)
40. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGpTTmQaN-U6N5BwMDEULRWHeFEc01SM2XhkBBHNFLobONzf5ttZQtko3WdwdmS84zW0I6hnzKUqsl8BS8fNgoillX27eRuB7sWcNJnN4ZGu-A7NoN7it5fVg==)
41. [openreview.net](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQE8ANmWkJ07LYACqwyMijllSblRuEgubVzLAcWmZzI5KYEd6PMHtbO7ARpQpKuaIhVdz3MslAjTsL8yoqU0rn9-75y8cl0fXFYaHUcxSgwnP9Cw6cprycBwq4cCOhuaO6w=)
42. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHbfsTWn5dJStCWX7RbUbjwZE0Sgr6NBJYynzyF8clm7y9L3AeaxMIL-FfIam7HrRkVbwuF3t29N-3BNjv0HccUfxSum1KWZ-HWSNxLh4pTi-gRoO5jY9cAoOe5YjnYBRHUlyfgAf6Y2xGIr6SY50mQPvGhLS4i9Cr01slk14LkNav39lgzsN11QmGSmT00Zn7Ek9cG2ZOe1uPj_GEI-P8qJFo1oy98)
43. [emergentmind.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFmwEuTVyLaWA0s-vjTKTxsd4WXdhnR6TLeulYZO6g7RX-HgN3jAoWhwuVcinEuxjijST2tu7oREDDFYsJ_lRGR71uG-YO5JcLaGLHQNqg2-FH7MafDXISGC7A-_MthrXQUXqTgVygqekvpG7a8nPAIVFMZuubTZkJECnajwo3Jaw==)
44. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEx0FXTuBLNWhs91G1SI3KBXTQ8ZHzenpjbPiRgqwbMQxqjCFkt8Si3dcphWBJO6uL-mNuBr12-PiXiiYs8FyW0ZoNqn2T1yAjSvC26s4eMGl08e6TJe0UeVA==)
45. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHDcN765AN6SGN8IdkKdI8fXiUYEe8CVE01HwoVfBhUkvQTXik0WxGBsvd94Pc_tzhQh1Yh2J4LBH8OtVFp_gaxSU6AxwjNC31SofpcvH5p3oYuHW3H8aLtvA==)
46. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGU-Ckvnnn_xiXSagiEKE-NF6EFeZB31bDh3_dAZGJ2quOeYdriqEumehsdlxiyN2uoxHRJJsD8Z5wELQMzmeERnNqOTaDX9o0r34fxiMMPojdhi6cl8ddm_6obEMYDFb0ek7PGWjykMwrn2oCEuC5oeIXGGkbJGeYaQhnoJs5wae5ah6arJ7X3lpR3JKOD)
47. [eleuther.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGmh-od6LlRSogiL-RMHzwEa5xmYuG40Ce2xpHs61WkMaRP8JuCiGRHkvTAuhuFWnhsIASqlEpsAEgNSjCqc2u7v1SdeDO-uyQOfDhJHEQYI_wUNmo=)
48. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG2nh7b6G08UUSZj1yj__EzWoj8kbTyr5FuOPixdySwnLy_NxfetRqUVxtC1xVAaXb2_MHWkmlIjG8ZnpCS8lFPhdxq7O22_aGF1g8jlM5M_cW8pDxXiniVaK920iEfSAnPTeP4NyLVGlagawtKL-HY2OlHMUIfoEdPYncLmY3sj8oIpg==)
49. [graphcore.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFTR4gQj5pmOMgcrn--2_gTr-L7Ulu3POo8CQdwi7hz8x2EP1d4rHhuJm9PLtPmjEBd1NMmSD2SGCHOmgUu4kxssdMCqeJljbTabEcyZeoZ68kzjwNQM5Wmwe3ZGSQNMbL53E9F_zkWCc-5MwyJOtyd1030fs0oGZqyQglQTkynX7aDDm3-B3VZfL0VLSzvFO0g9Q==)
50. [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEWwmYqShFTVDCQZkqYVd929Yu-vYHepZzrsuMoPWIrM14C5YwJkC4cha48i27EcKAUvglGmvB9MGazdijMLMLwAGugJFeQZRFb4d0tab0mH4Eids773QJKwuCu)
51. [githubusercontent.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGOfspFTP2MalxiQ7EMKT5IBgLCldcwhsSYD6gTYsABiR8FaneIK7O_1zU9YXL4iQYAD_6t3zfuNY8fh5a7Me4vlvtYjM-Q-InZaOpwghIIaGGTWyxgzxyVh1-z3R8kvUyln0XxuWpnPsplaWl9L-KtYJ4EaTQYRGk6mfz2q-Rf5sWGWJcv5Ck=)
52. [iit.edu](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGqN52DmWEX7B6QNU74M5oxA2SidURWVW9iJYiXV3qvedOyeuN_7ABYpDx_cGveYz6veCcosr7l6cEXN9rZ4r7skUHblqAOyo9EsWAsbllVX9x3_XcRcIDoRfGjv-WANMpHJMvs9-w8Vxltb70yeg==)
53. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHAx_ERazu78sMx7P-LcjDEL7g0FsZdvf0JnQufqybCYmymf-NqjVZH9tzHmKglveqWxM2TSZRFACnJJNHjnPX8J4ARqL35ThcDf8vof4sdDKKdYZbI2aM8gA==)
54. [towardsai.net](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQH2-hrAAG-GmnqJQifCUjJ83CSvxyjMZrBj6XmEuyy75bS7mybB4YU-oACboXwCWObcBMEuaVLUlst3f6FMr1iUgenDkjwyluYk_T53aaRhj9EkofojbLB8W12T5wj6BFerv1E4QBZzknSOM0lXS_Wvmv8CbhPDzxl-mhCl6NFCXy9sYochtgLe)
55. [openreview.net](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGOR6FL4WyERK30YO5CSIBPShzGRXOHxxDxrVshIqZYJ6qFJK11NNp7NnLOLfEJHko9LFAq-lzfAam79UcGB1clp-gpm1YmbClHGL9sDdM9w7MK-ZD6UQx_43qnWH7F)
56. [reddit.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEOGoRwBH0lri4h1SFVrYzRdAP965-MJp8ePZQlLMpFU0ORySc2mlN_kIIR9OSueEfmb1WawYK1zGrmEHVHtCz_3aQWbooT_Qg8yJBnoApGNrVXcwRK2r6eGmRAUCad4WnM0b0kY-4YZz8xKHvREKnH391de9siJEmO0m0AWROa0NAgT7a_lMXTPfyGlEExNwLjkAcLBOSmCykqdyI20Qya)
57. [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHVh9jlOr79k_KyONtsGekcVk-su0vVBqBtDb8nHUuGAXq6sp6vJNNMU4FzxUUTS5-8o2N5pEQx_zZQPVjVMewMbVjwo4nzFc-T-Wa13Ju4yr6zHcJOcrcJicNNQd5RcE9x1K9E3td52Qb1O47XqGV8LEXjg8sWTP9u1rCtSwP62oasPv6c851Op-B4rFJXqUIx9Po8InWp)
58. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGSPwE0PjA_svNjHwO0xef15LiP9ZQKxkWNsQI9qtaPXTPUdcWTr_6wQwy_Qlg-HEPqzz6fW39E2cJw0D4M2AnsnVfo-EEoGeC0LZP22Zaoh_50JGP5iP0=)
59. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG48tVIcipV6CyG_IJWH0V_2zY8kqaaA4Y6zBR3l5rQkYITa6FfNixD8NJB0VvilGzmDylc7vuDbM4Sh_loPAaCF0VIVLVKjwRwgRrqJiBdBBaNx4MN5krpsg==)
60. [pyimagesearch.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGXPOASsSFi715tLiivLGBwXccK6krfSthv9d4WA4WlSCttX6nEyq4mt7zjxOAszNBGAw7KjzSy9MA8ElT9z0yK-Iz1p3c4e4qbrOaJ-U3SgCGRIbdyHs64RWs7zslCKe5Tk4laQAVqfEyJ3dj80ROu6tFEpuLZtdt6dpHzd82nfZVqO6QuVd2PESWNNETS56or)
61. [mlr.press](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFc_ydJVRDtiXFYUB9_qPEPNNQ4X3cGRp7VY5ViD75tNKbnblJl9jhs_CsxZPI2_yliRDs_Kl7P5GBanIph9lr96OdAiX_KRR4hhgDbad-xqkH0ax9vXGp6CO-1DTs-GVqaD2x38uW3ANqXH44fZF-A50w=)
62. [reddit.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFZIiSDKS8mbLjioUixM87GCGMGCsCYi8CYeLzaRvGNEbo2S_TbegPxwQD2i-5VZV4PxGsP67XUX9knzt49RghpwgiLyvi3zF1TkhPR4xq1mPA4j7mfmKR9Cz_z0AJ0g4I9r6xqCkaSPzoP3TwQ7dc0gH_a0xKH9UPn-jrqljClZS6GsGFnchhlzOXQQt505sZ-DCM8a4oNTFJH7w3plcI=)
63. [openreview.net](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHgKa8Imab1pie6r3Ha3cSdX-LDNzrk57Ya_8tlkMhcpE5kANxHrBDRtG7msuQCQZkVFweXS04NI2zSROfgdkTDk4Cs8Z4RM3yS1Ao04_6LhK8KzwtbHPuSXXqofgwqovk=)
64. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFMl4v6F-PSg6hgqozwULKNrSBB7G0DjIzDUxCactfctGR5lJmoGfIra5yk7XKiQYYIfFnwtzWZR6bdzhf7BvTJBt7AsnFIeuqjrDNHMxMAL8rPtDpzOA==)
65. [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGzwghumZbW7A-P3gP1j5HkTQnbbDw8O3sPBPuiC3hhp9j4XmwawIBvnac0r0mw4kGHqqlv7IQWUhXcd4wQlXtMmWclztlwK_EhE92EBTP1D6vUUXQEEcUYZqksK4HK)
66. [openreview.net](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGdm0oPX9iK5LFuNERNVWljxKjGyEobZvLlRZ2livcRkIesPywm-fuwGMKh7D7yZbr_jSvX681rIRAP7-5BeSjQcHacv-wd4iv7EsMfXeaj8K1zkyIQoB6gqL0C4xbL0f8=)
67. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHKiBEvnrSvTOHhu6586CmhwT_lDHXVQAPVLPGJmGxFSVqi5HAUyNcEl8wnc0__66vJEcicwd5MjTFT6hsDS5PNtae0i5Ga9zPyauBGl3RMydF8ji9EFEHJfXfNMJR63RHpp2Zmx7K8YMJ6ms5LqEPUSC-QP-VUNxpcTwVkPyZ92rkGDVBHzA==)
68. [maartengrootendorst.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEU2qcgPqFS2bYlcwFnMLxW70p9KzAi182YWRjXsJjMvlPdQsPm7zsSaUpaq3vOkGOsFQwtQf57gVcH4G4_cAd1nla3ZTCqH3U5ynH3pV0gVnTTSTywo7MO733a1bZjINcsh_-1N-b83bm9UPhqBpio8JbNRiks37lPzoMNPY-bwLbpquM=)