# Engineering and Science of Long-Context Language Models

The expansion of context windows in large language models has fundamentally altered the trajectory of artificial intelligence research and deployment. Early transformer architectures were computationally bound by context windows of 2,048 to 4,096 tokens, limiting their utility to processing single documents or brief conversational exchanges [cite: 1, 2, 3]. As the demand for complex agentic systems, comprehensive codebase analysis, and multi-document reasoning intensified throughout 2024 and 2025, research shifted toward achieving contexts of 128,000 to over 1,000,000 tokens [cite: 1, 4, 5]. 

This massive scale introduces severe computational and memory constraints. The fundamental mechanism of the standard transformer—self-attention—scales quadratically with sequence length in compute operations, while the intermediate state required for autoregressive generation grows linearly in memory [cite: 6, 7]. Achieving reliable 1,000,000-token context windows is not the result of a single algorithmic breakthrough, but rather a convergence of innovations across mathematical attention approximations, hardware-aware systems engineering, position encoding extrapolation, and synthetic data curricula. 

## Key-Value Cache Memory Dynamics

The primary hardware and mathematical bottleneck in serving long-context language models is the Key-Value (KV) cache. During autoregressive decoding, a transformer must attend to all previous tokens in the sequence to generate the next token [cite: 8, 9]. Recomputing the keys and values for the entire historical sequence at every step is computationally prohibitive. Instead, models cache these vectors in the high-bandwidth memory (HBM) of the accelerator. As the sequence length grows, the memory required to store this cache eventually dwarfs the memory required to store the static model weights [cite: 1, 3].

### Mathematical Scaling of the Cache

The absolute size of the KV cache is dictated by the model's structural dimensions, the sequence length, and the active batch size. The foundational formula for calculating the memory footprint of the KV cache in bytes for a single sequence is determined by multiplying the number of layers, the number of key-value attention heads, the dimension of each head, the sequence length in tokens, and the byte size of the floating-point precision [cite: 10, 11]. An initial factor of two is applied to account for the simultaneous storage of both the Key matrix and the Value matrix [cite: 10, 11].

To illustrate the severity of this constraint, one can evaluate the Llama 3.1 70B model operating at a context length of 131,072 tokens. The model features 80 layers, 8 KV heads, and a head dimension of 128. Using 16-bit precision (2 bytes), the KV cache for a single sequence requires approximately 42.9 gigabytes of memory [cite: 1]. This cache size is larger than the parameter weights of many smaller foundation models and consumes more than half the VRAM of a standard 80GB NVIDIA H100 GPU [cite: 1, 12]. 

If scaled to a 1,048,576-token context, the KV cache for that single Llama 3.1 70B sequence inflates to approximately 320 gigabytes, requiring tensor parallelism across at least four 80GB GPUs solely to hold the cache, before accounting for the 140 gigabytes needed for the FP16 model weights [cite: 3, 10].

[image delta #1, 0 bytes]

 For the larger Llama 3.1 405B model, a 128,000-token context demands approximately 66 gigabytes of KV cache per sequence, and a 1,000,000-token context demands over 500 gigabytes [cite: 3, 10, 13]. 



### The Memory Bandwidth Wall

The KV cache issue extends significantly beyond sheer capacity limits; it creates a severe memory bandwidth bottleneck that restricts inference speed. During the decoding phase, where tokens are generated sequentially, the arithmetic intensity of the workload is extremely low [cite: 9, 12]. For every single token generated, the inference engine must read the entire historical KV cache from HBM into the compute cores (SRAM) to perform the attention matrix multiplication against the new query [cite: 9, 14]. 

Because modern GPUs process arithmetic floating-point operations exponentially faster than they can transfer data across the memory bus, the decoding phase becomes strictly memory bandwidth-bound. A single user query traversing a 128,000-token context in a 70B model forces the system to transfer over 40 gigabytes of data across the GPU memory bus for every subsequent output token generated [cite: 1, 12]. At a standard memory bandwidth of approximately 3.3 terabytes per second on flagship hardware, generating merely 10 tokens per second for this sequence consumes the entire theoretical bandwidth of the chip, rendering standard attention architectures economically unviable at the 1,000,000-token scale [cite: 3, 12, 15].

| Attention Architecture | Key-Value Cache Size per Token | Total Cache at 128K Context | Compression Ratio vs MHA |
| :--- | :--- | :--- | :--- |
| **Multi-Head Attention (MHA)** | ~2.6 MB | ~332.8 GB | 1x (Baseline) [cite: 3] |
| **Grouped-Query Attention (GQA)** | ~328 KB | ~41.9 GB | 8x [cite: 3] |
| **Multi-head Latent Attention (MLA)** | ~70 KB | ~8.9 GB | 37x [cite: 3] |
| **Multi-Query Attention (MQA)** | ~41 KB | ~5.2 GB | 64x [cite: 3] |

## Structural Attention Compression Mechanisms

To circumvent the KV cache capacity and bandwidth bottlenecks, architectural research has continuously optimized how attention matrices are structured and stored in memory. The evolution from Multi-Head Attention to Multi-head Latent Attention charts the industry's progression toward supporting massive context windows.

### Grouped-Query and Multi-Query Interpolation

Standard Multi-Head Attention provisions unique Key and Value heads for every Query head, yielding high expressive quality but maximizing memory consumption [cite: 16, 17]. Multi-Query Attention (MQA) represented the first severe compression technique, collapsing all Key and Value computations into a single shared head across all Queries [cite: 16]. While MQA drastically reduced the cache footprint, it caused measurable degradation in generation quality and training stability as model capacity scaled, often struggling to route complex logic across varied semantic spaces [cite: 16, 17, 18].

Grouped-Query Attention (GQA) emerged as the dominant compromise architecture, utilized heavily by models such as Llama 3.1, Qwen 2.5, and Mistral [cite: 1, 16]. GQA interpolates between MHA and MQA by clustering Query heads into discrete groups, with each group sharing a single Key and Value head [cite: 16, 17]. A standard GQA ratio of 8:1 reduces the KV cache size by a factor of 8 compared to MHA, mitigating the memory bandwidth bottleneck while preserving the vast majority of the model's reasoning capabilities [cite: 1, 18]. However, at 1,000,000 tokens, even GQA fails to keep the cache within practical bounds. A 70B model using GQA still requires hundreds of gigabytes per sequence at the upper limits of the context window [cite: 3, 10].

### Multi-head Latent Attention

To push beyond the mathematical limits of GQA, the DeepSeek research team introduced Multi-head Latent Attention (MLA) in their V2 and V3 models [cite: 4, 16, 19]. Rather than reducing the number of stored keys and values by sharing heads, MLA fundamentally alters the data structure of what is stored. It utilizes low-rank joint compression to condense the Key and Value matrices into a unified, low-dimensional latent vector [cite: 16, 17, 20, 21].

In the MLA paradigm, the massive, full-resolution Key and Value tensors are never materialized in the KV cache. During the forward pass, the architecture projects the inputs down into a much smaller latent dimension, frequently configured to 512 dimensions [cite: 1, 16, 22]. At inference time, only these highly compressed latent representations are stored in HBM. When a specific attention computation is required during autoregressive decoding, the system reads the tiny latent vector into SRAM and applies an up-projection matrix to reconstruct the necessary full-dimensional Key and Value representations on-the-fly [cite: 20, 22, 23].

This architectural pivot transitions the bottleneck from memory bandwidth to raw arithmetic compute. Reconstructing the keys and values requires additional matrix multiplications, actively increasing the total Floating Point Operations (FLOPs) per step [cite: 22, 24]. Because modern AI accelerators possess immense computational surpluses relative to their memory bandwidth, this trade-off is highly advantageous [cite: 4, 22]. Furthermore, MLA maintains positional integrity by decoupling the Rotary Position Embeddings (RoPE). Because RoPE relies on shift-invariant rotational matrices that are incompatible with low-rank linear compression, MLA maintains a separate, minimal cache explicitly for rotational keys alongside the latent vector [cite: 20, 22].

The empirical results of MLA enable frontier performance at minimal deployment costs. DeepSeek-V3, a 671-billion parameter Mixture-of-Experts model, consumes approximately 70 kilobytes of KV cache per token—a 37x compression over equivalent MHA models [cite: 3, 17]. At a 128,000-token context, the KV cache footprint is reduced to roughly 9 gigabytes, enabling ultra-large models to process long contexts on limited accelerator hardware [cite: 3, 8].

### Sparse Attention and Indexing Constraints

Beyond compressing the cache data structure, models have introduced mechanisms to prevent computing attention over every historical token. Standard dense attention requires operations that scale quadratically; if the input sequence doubles, the computational cost quadruples [cite: 6, 15].

To break this quadratic scaling, architectures incorporate sparse attention mechanisms, effectively pruning the context space. DeepSeek-V3.2 introduced DeepSeek Sparse Attention (DSA), which utilizes a two-stage routing approach driven by a lightning indexer [cite: 15, 25, 26]. Operating in highly efficient FP8 precision, the indexer rapidly scans the entire context to compute approximate relevance scores for all historical tokens relative to the current query [cite: 15, 25]. A fine-grained selection mechanism then retrieves only the top fraction of the most relevant key-value entries [cite: 15, 25]. By restricting the dense attention calculation to a fixed number of retrieved tokens, DSA reduces the asymptotic complexity of the attention mechanism, linearizing the compute cost as the context scales toward 1,000,000 tokens without degrading performance on complex retrieval tasks [cite: 15, 25, 26].

## Linear and Recurrent Architectures

While latent and sparse attention mechanisms optimize the traditional transformer, alternative architectures seek to discard the standard attention mechanism entirely in favor of recurrent algorithms that natively scale linearly with sequence length. 

### Kimi Delta Attention and Hybrid Architectures

In late 2025, Moonshot AI published technical documentation on Kimi Linear, a hybrid architecture combining Multi-Head Latent Attention with a novel mechanism termed Kimi Delta Attention (KDA) [cite: 27, 28, 29]. Pure linear attention models historically suffer from degraded in-context learning and exact retrieval capabilities compared to full quadratic attention [cite: 29, 30]. Kimi Linear addresses this by interleaving different attention paradigms.

KDA refines the delta rule of linear attention by incorporating a highly granular, channel-wise gating mechanism [cite: 28, 30, 31]. Previous hardware-efficient linear models relied on coarse, head-wise or scalar forget gates, where an entire memory state for a specific attention head decays uniformly [cite: 30, 32]. This uniform decay often results in either excessive memory retention or catastrophic forgetting of vital context [cite: 30]. KDA resolves this by assigning an independent forgetting rate to each individual feature dimension using a specialized variant of Diagonal-Plus-Low-Rank transition matrices, enhancing the utilization of hardware tensor cores [cite: 28, 30, 32]. 

By interleaving three layers of KDA for every one layer of MLA, the Kimi Linear architecture delegates strict local context modeling to the linear layers and global retrieval to the full attention layers [cite: 29, 30, 31]. Additionally, Kimi Linear removes positional encoding from the MLA layers entirely, routing all positional awareness through the recurrent KDA mechanism [cite: 29, 30]. This hybrid structure reduces KV cache usage by an additional 75% relative to pure MLA and achieves up to a six-fold increase in decoding throughput at the 1,000,000-token boundary [cite: 27, 28, 32]. Validation testing demonstrates that deploying this architecture provides a 1.25x efficiency advantage over equivalent models, requiring 20% less training computation to achieve matching downstream performance [cite: 32, 33].

### State Space Models: Mamba and RWKV

State Space Models (SSMs) treat sequence processing as a continuous-time signal problem, compressing historical context into a fixed-size hidden state. Because the state size remains constant, SSMs require stable, constant memory during autoregressive inference and scale in linear time, inherently solving the infinite-context dilemma [cite: 7, 34].

The Mamba architecture advanced SSMs by introducing a selective scan mechanism. Unlike early state space models which applied static convolutional kernels, Mamba's parameters are data-dependent; the model dynamically decides whether to update its internal state or ignore the current input token based on its semantic relevance [cite: 7, 34]. This selective filtering allows the network to disregard noise in massive documents, retaining only critical information [cite: 34]. Mamba has proven effective as the foundation for specialized reasoning models. Researchers have successfully distilled transformer-based reasoning models into Mamba variants using reinforcement learning frameworks, matching transformer-level mathematical reasoning while delivering generation speedups of over 2.5x [cite: 35].

Similarly, the Receptance Weighted Key Value (RWKV) architecture operates as a linear-time recurrent neural network during inference but relies on parallel computation during training, mimicking a transformer [cite: 7, 34]. RWKV models utilize linear time-mixing and channel-mixing functions to avoid the quadratic attention matrix entirely [cite: 34]. Recent implementations, including visual and language variants, demonstrate that these recurrent architectures exceed standard transformers in generation throughput at massive context lengths, although they generally require hybridization to match the exact relational recall of full attention models on complex benchmarks [cite: 34, 36].

## Positional Encoding Extrapolation

Transformers possess no inherent sense of sequence order; they rely entirely on positional encodings injected into the input embeddings to understand syntax and structure. The industry standard is Rotary Position Embedding (RoPE), which encodes absolute position with a rotation matrix and naturally captures relative token distances [cite: 37, 38]. 

A critical vulnerability of RoPE emerges when a language model trained on 8,000 tokens is abruptly exposed to 100,000 tokens during inference. During pre-training, the model never observes the high-frequency and low-frequency rotations corresponding to positions beyond its initial training window [cite: 37, 39]. Consequently, the attention mechanism encounters out-of-distribution rotational values, causing the model's perplexity to collapse and generating incoherent text when the context exceeds the pre-training limit [cite: 37, 39].

### Position Interpolation and NTK-Aware Scaling

Initial attempts to extend context windows relied on Position Interpolation. Instead of forcing the model to extrapolate to unseen positions, Position Interpolation mathematically compresses the incoming long sequence into the short range the model recognizes [cite: 38, 39]. This is achieved by multiplying all position indices by a scaling factor representing the ratio of the old context length to the new context length [cite: 38, 39]. While effective for minor extensions, severe compression squeezes adjacent tokens too closely together, destroying the model's ability to differentiate fine-grained local relationships [cite: 37, 39].

To resolve this degradation, researchers applied Neural Tangent Kernel (NTK) theory, which demonstrates that neural networks struggle to learn high-frequency information in low-dimensional spaces [cite: 39]. NTK-aware scaling modifies the rotary base frequency, spreading the interpolation pressure unevenly across the hidden dimensions [cite: 38, 39]. By scaling the base according to the sequence length, NTK-aware methods preserve the high-frequency dimensions, which represent critical local token relationships, while aggressively interpolating the low-frequency dimensions that represent broad, long-range macro relationships [cite: 38, 39]. 

### The YaRN Framework

The definitive mathematical solution to RoPE extension is the YaRN (Yet another RoPE extensioN) framework [cite: 37, 39]. YaRN segments the hidden dimensions of the RoPE embeddings into three distinct groups based on their wavelength: pre-critical high-frequency dimensions, post-critical low-frequency dimensions, and an interpolation transition zone [cite: 39, 40]. 

YaRN applies a piecewise scaling function across these segments. High-frequency dimensions are left entirely untouched, ensuring local text comprehension remains flawless [cite: 37, 38]. Furthermore, YaRN addresses the attention entropy problem. In massive contexts, the attention softmax distribution becomes overly diluted across hundreds of thousands of tokens, causing the model to lose focus and hallucinate [cite: 37, 39]. YaRN introduces a temperature scaling parameter directly into the attention formulation, which re-sharpens the attention scores across vast distances [cite: 37, 39]. Using YaRN, models can be extended from short contexts to 128,000 tokens or beyond with only a few hundred fine-tuning steps, ensuring accurate retrieval without catastrophic forgetting [cite: 37, 39]. Alternative models, such as Baichuan 2, bypass RoPE entirely, achieving 192,000-token contexts using Attention with Linear Biases (ALiBi) dynamic position coding [cite: 41, 42].

## Systems Engineering and Sequence Parallelism

While architecture dictates the theoretical mathematical limits of long-context modeling, distributed systems engineering makes it physically executable. Processing 1,000,000 tokens in a single batch requires immense VRAM that exceeds the capacity of any single GPU. This necessitates Sequence Parallelism, a technique that shards a single continuous input sequence across a massive array of interconnected accelerators [cite: 43, 44]. Standard Tensor Parallelism splits the model weights across GPUs but duplicates the sequence activations across all devices, offering no memory relief for massive context lengths [cite: 9, 43]. 

### Ring Attention and DeepSpeed Ulysses

Ring Attention solves the sequence memory bottleneck by partitioning the input sequence into discrete chunks distributed across a cluster of GPUs arranged in a logical ring topology [cite: 43, 45, 46]. During the self-attention calculation, each GPU processes its local chunk of the Query sequence. The Key and Value blocks are then passed peer-to-peer around the ring. As long as the computation of a block takes longer than the network transfer of the next block, the communication latency is entirely masked by the computation [cite: 45, 46]. This zero-overhead overlapping allows the context length to scale linearly with the number of GPUs added to the ring, enabling theoretically near-infinite contexts provided the interconnect bandwidth is sufficient [cite: 46, 47].

DeepSpeed Ulysses takes a different topological approach utilizing all-to-all sequence sharding. Instead of passing sequence blocks in a sequential ring, Ulysses shards the input sequences and utilizes highly optimized collective communication protocols to distribute the attention heads themselves across GPUs [cite: 43, 48]. Each GPU calculates the full sequence length for a specific subset of the attention heads. A key advantage of this approach is that its communication cost is inversely proportional to the sequence parallelism degree, making it highly efficient within tightly coupled nodes [cite: 43, 48].

| Sequence Parallelism Strategy | Communication Topology | Architectural Strengths | Primary Limitations |
| :--- | :--- | :--- | :--- |
| **Ring Attention** | Peer-to-Peer Ring (Sequential) | Scales context length linearly with device count; hides communication latency behind compute operations. [cite: 45, 46, 49] | Highly sensitive to hardware interconnect bandwidth; a slow network link halts the entire ring process. [cite: 46] |
| **DeepSpeed Ulysses** | All-to-All (Collective) | Extremely fast for models with high head-counts; highly efficient within a single high-speed NVLink domain. [cite: 43, 48] | Less efficient across slow cross-node links due to massive all-to-all scatter/gather requirements. [cite: 43] |
| **Blockwise Parallel Transformer** | Hierarchical Block Fusion | Fuses Attention and Feed-Forward Network logic to bypass activation memory limits; dramatically reduces overhead. [cite: 50, 51] | Requires precise mathematical tuning of block sizes; complex integration with existing standard frameworks. [cite: 50] |

### Blockwise Parallel Transformer Mechanics

The Blockwise Parallel Transformer (BPT) extends memory optimization beyond the attention layer directly into the Feed-Forward Network. Standard sequence processing materializes massive activation tensors in the FFN layer, often reaching sizes of 8 times the batch size, sequence length, and hidden dimension [cite: 46, 51, 52]. BPT computes both self-attention and the subsequent FFN operations in a fused, block-by-block manner [cite: 46, 52]. 

By preventing the full materialization of the entire attention matrix and interleaving the FFN processing while the data is still resident in SRAM, BPT reduces memory demands drastically [cite: 51, 52]. Implementations utilizing these blockwise fusions report training on sequences 16 to 64 times longer than vanilla frameworks on equivalent hardware, sustaining high Model Flops Utilization (MFU) even when processing 2,000,000 tokens [cite: 50, 51, 52].

## Training Methodologies and Synthetic Data

Expanding the theoretical capacity of the context window is only half the engineering challenge; a model must be explicitly taught how to reason accurately across millions of tokens. The primary barrier to training is the scarcity of high-quality, organic long-context data. While the internet contains endless short-form content, structurally coherent documents containing complex, cross-referenced logic at the 500,000 to 1,000,000 token scale are exceedingly rare [cite: 53, 54]. 

### Synthetic Data Generation and Co-Evolution

To train frontier models, AI laboratories rely heavily on synthetic data generation pipelines [cite: 53, 55]. Frameworks such as WildLong synthesize realistic long-context instruction data by utilizing larger teacher models to extract meta-information from vast corpora and construct multi-document reasoning tasks [cite: 56]. These pipelines generate complex multi-turn simulated conversations, document-grounded task constructions, and verifiable instruction-response pairs [cite: 53, 56]. By dynamically controlling the difficulty progression and stylistic variation, synthetic data prevents models from overfitting to the limited styles of organic books or massive codebase dumps [cite: 53, 56, 57].

Furthermore, advanced synthetic curricula employ a self-instruct co-evolutionary loop. A model generates variations of a complex reasoning prompt, evaluates its own answers, and filters out low-quality context chunks using LLM-as-a-judge protocols [cite: 54, 58]. This self-improvement loop effectively bootstraps reasoning capabilities without human annotation, solving the data bottleneck required to train models on extreme inputs [cite: 54, 59]. 

### Curriculum Learning and Multi-Token Prediction

Training a model on 1,000,000 tokens from initialization is computationally wasteful and mathematically unstable. Instead, laboratories employ curriculum learning techniques. Models are first pre-trained on standard sequence lengths of 4,000 to 8,000 tokens. Once base language modeling converges, the sequence length is incrementally stepped up during a specialized continued pre-training phase, adjusting the positional encodings at each step [cite: 25, 57]. 

To maximize the efficiency of this computationally expensive training phase, architectures like DeepSeek-V3 incorporate Multi-Token Prediction. Instead of predicting a single next token, the model utilizes specialized auxiliary heads to predict multiple future tokens simultaneously [cite: 4, 20]. This objective packs denser gradient signals into every training step, significantly improving data efficiency. It forces the model to plan its reasoning further into the long-context future, aligning internal representations before the auxiliary prediction modules are discarded prior to inference deployment [cite: 20].

## Evaluation Frameworks and Benchmark Realities

As context windows expanded, the industry standard evaluation—the Needle In A Haystack test—became insufficient for measuring intelligence. This test evaluates whether a model can retrieve a specific fact randomly inserted into a massive body of text [cite: 56, 60]. While foundational, models rapidly achieved near-perfect retrieval accuracy up to 1,000,000 tokens, rendering the benchmark obsolete for distinguishing advanced reasoning [cite: 5, 56, 61]. 

### RAG Latency Overheads

When native long-context models are unavailable, systems often rely on Retrieval-Augmented Generation (RAG). RAG utilizes external embedding databases to fetch relevant text chunks to insert into a short context window [cite: 61, 62]. However, systems engineering research indicates that RAG pipelines introduce severe latency overheads, accounting for over 45% of Time-To-First-Token (TTFT) latency due to the encoding and retrieval processes [cite: 9, 62]. Native long-context models bypass this latency, though they incur massive KV cache constraints. Novel approaches like InfiniRetri attempt to bridge this gap by leveraging the LLM's internal attention distribution to execute precise retrieval directly over infinite-length tokens without relying on external embedding models [cite: 61].

### The RULER Benchmark and Frontier Performance

To rigorously assess 1,000,000-token capabilities, researchers rely on the RULER benchmark. RULER evaluates models using multi-needle aggregation, complex reasoning across disparate context blocks, and cross-document comparison [cite: 63]. Performance on RULER tasks plummets significantly compared to simple retrieval, exposing the reality that synthesizing logic from data spread across a million tokens remains a volatile challenge [cite: 60, 63].

In the current landscape, closed-source models and highly optimized open-source architectures contest the frontier. Google's Gemini 1.5 Pro reliably supports context windows ranging from 1,000,000 to 2,000,000 tokens, excelling in multi-modal retrieval and maintaining dominance in high-volume, long-context aggregation tasks [cite: 5, 60]. However, open-weight models have rapidly closed the gap. Qwen 2.5 72B and DeepSeek-V3 exhibit exceptional long-context reasoning, outperforming older models on educational and professional benchmarks at a fraction of the computational training cost [cite: 64, 65]. Similarly, Moonshot's Kimi Linear achieves Pareto-optimal speed and performance on the 128,000-token RULER boundary, operating dramatically faster than traditional attention mechanisms [cite: 32, 66].

| Long-Context AI Model | Advertised Context Window | RULER (128K) Score | Long-Context Architecture / Technique | Primary Deployment Strategy |
| :--- | :--- | :--- | :--- | :--- |
| **Gemini 1.5 Pro** | 1,000,000 to 2,000,000 | 94.4% | Sparse MoE, Proprietary Attention | Closed API, High-volume processing [cite: 63, 67] |
| **DeepSeek-V3** | 128,000 (API extends to 1M) | *Near-frontier* | MoE, MLA (Latent Attention), MTP | Open-weight, Cost-efficient inference [cite: 26, 64] |
| **Kimi Linear (48B)** | 1,000,000 | 84.3% | Hybrid Linear, KDA (Channel-wise gate) | Agentic workflows, High throughput [cite: 32, 66] |
| **Qwen 2.5 (14B)** | 1,000,000 | 92.2% | RoPE Extrapolation, Advanced GQA | Open-weight, Complex coding tasks [cite: 63] |
| **Llama 3.1 (70B)** | 128,000 | 66.6% | Grouped-Query Attention (GQA) | General purpose, Broad ecosystem [cite: 63] |

*Note: RULER scores reflect weighted averages of multi-task performance measured at the 128,000-token context boundary. Specific frontier models exhibit rapid degradation past this point depending on task complexity [cite: 63, 65].

## Conclusion

The realization of 1,000,000-token context windows in language modeling represents a triumph of cross-disciplinary engineering and applied mathematics. The fundamental quadratic limitations of the traditional transformer architecture have been systematically dismantled. Key-Value cache memory walls are being scaled through Multi-head Latent Attention and Hybrid Linear architectures, compressing the gigabytes of required memory into manageable footprints. Recurrent algorithms offer glimpses into a highly efficient post-attention future, while sequence parallelism techniques distribute massive computational burdens across interconnected hardware topologies. Coupled with synthetic data curriculum learning and mathematically scaled positional encodings, these innovations have transformed large language models from short-form text generators into encyclopedic reasoning engines.

## Sources

1. [Python Alchemist - LLM Architectures](https://www.pythonalchemist.com/llm-architectures/attention-variants)
2. [Vizuara - Decoding Multi-Head Latent Attention](https://vizuara.substack.com/p/decoding-multi-head-latent-attention)
3. [ArXiv 2503.05840v1 - Slim Attention](https://arxiv.org/html/2503.05840v1)
4. [Abhik.ai - MHA vs GQA vs MQA](https://www.abhik.ai/concepts/attention/attention-comparison)
5. [Medium - MHA vs MQA vs GQA vs MLA](https://medium.com/@zaiinn440/mha-vs-mqa-vs-gqa-vs-mla-c6cf8285bbec)
6. [Hugging Face - Infini-attention](https://huggingface.co/blog/infini-attention)
7. [Medium - Infini-attention Unlocks Limitless Context](https://kshitijkutumbe.medium.com/breaking-barriers-how-infini-attention-unlocks-limitless-context-for-transformers-48a347bbf81c)
8. [ArXiv 2404.07143v1 - Leave No Context Behind](https://arxiv.org/html/2404.07143v1)
9. [Prompting Guide - Infini-attention](https://www.promptingguide.ai/research/infini-attention)
10. [Louis Bouchard - Infini-attention](https://www.louisbouchard.ai/infini-attention/)
11. [Medium - Moonshot AI Introduces Kimi Linear](https://medium.com/@HarselGivesh/moonshot-ai-introduces-kimi-linear-linear-attention-architecture-marks-a-new-starting-point-for-792829358833)
12. [Towards AI - Kimi K2.5 Architecture](https://pub.towardsai.net/the-architecture-of-efficiency-how-kimi-k2-5-rewrites-the-economics-of-intelligence-a53afc144d23)
13. [Intuition Labs - Kimi K2 Deep Dive](https://intuitionlabs.ai/articles/kimi-k2-technical-deep-dive)
14. [Codecademy - Kimi K2.5 Guide](https://www.codecademy.com/article/kimi-k-2-5-complete-guide-to-moonshots-ai-model)
15. [Kimi Blog - Research Articles](https://www.kimi.com/blog/)
16. [Emergent Mind - YaRN RoPE Extension](https://www.emergentmind.com/topics/yarn-yet-another-rope-extension-method)
17. [ArXiv 2309.00071 - YaRN Paper](https://arxiv.org/pdf/2309.00071)
18. [Medium - YaRN: Extending RoPE](https://medium.com/@cenghanbayram35/yarn-extending-rope-without-breaking-it-ab07882b1581)
19. [ACL Anthology - YaRN Details](https://aclanthology.org/2024.findings-acl.32.pdf)
20. [EleutherAI - Extending the RoPE](https://blog.eleuther.ai/yarn/)
21. [ArXiv 2412.11854v1 - RAG Ecosystem Tradeoffs](https://arxiv.org/html/2412.11854v1)
22. [Dev.to - RAG System Architecture Trade-offs](https://dev.to/satyam_chourasiya_99ea2e4/navigating-rag-system-architecture-trade-offs-and-best-practices-for-scalable-reliable-ai-3ppm)
23. [ArXiv 2502.12962v1 - InfiniRetri](https://arxiv.org/html/2502.12962v1)
24. [USENIX - Sarathi-Serve OSDI24](https://www.usenix.org/system/files/osdi24-agrawal.pdf)
25. [Stanford CS224U - Neural IR Slides](http://web.stanford.edu/class/cs224u/slides/cs224u-neuralir-2023-handout.pdf)
26. [DeepSeek-V3 Github](https://github.com/deepseek-ai/deepseek-v3)
27. [Reddit - DeepSeek's Underrated Advantages](https://www.reddit.com/r/DeepSeek/comments/1sqw6dq/deepseeks_3_underrated_advantages_1m_context/)
28. [VitaLab - DeepSeek V3 Architecture](https://vitalab.github.io/article/2025/02/11/DeepSeekV3.html)
29. [Aarnphm - DeepSeek V3.2 Exp PDF](https://aarnphm.xyz/thoughts/papers/DeepSeek_V3_2.pdf)
30. [Medium - DeepSeek Broke the Attention Wall](https://medium.com/coding-nexus/deepseek-just-broke-the-o-l%C2%B2-attention-wall-and-it-changes-everything-for-long-context-ai-a5b34da22401)
31. [Baichuan 2 Technical Report](https://cdn.baichuan-ai.com/paper/Baichuan2-technical-report.pdf)
32. [Baichuan 2 Github README](https://github.com/baichuan-inc/Baichuan2/blob/main/README_EN.md)
33. [Medium - Baichuan 2 192K Launch](https://medium.com/@piyushlamsoge20/baichuan-intelligent-launches-large-model-baichuan2-192k-which-can-input-350-000-words-at-a-time-eed70c51d9f9)
34. [Pandaily - Baichuan 2 192K Details](https://pandaily.com/baichuan-launches-the-model-with-the-worlds-longest-context-window-baichuan2-192k)
35. [Reddit - Machine Learning News Baichuan 2](https://www.reddit.com/r/machinelearningnews/comments/16n4yfh/meet_baichuan_2_a_series_of_largescale/)
36. [ArXiv 2509.01185 - Synthetic Data Framework](https://arxiv.org/pdf/2509.01185)
37. [Confident AI - Synthetic Data Guide](https://www.confident-ai.com/blog/the-definitive-guide-to-synthetic-data-generation-using-llms)
38. [Tetrate - Synthetic Data Generation](https://tetrate.io/learn/ai/synthetic-data-generation-llms)
39. [Medium - Synthetic Data Creation at Scale](https://medium.com/@kiranvutukuri/79-synthetic-data-generation-for-llms-creating-training-data-at-scale-9e9745a19c51)
40. [IBM Research - LAB Method](https://research.ibm.com/blog/LLM-generated-data)
41. [ArXiv 2403.18276v3 - Mamba Document Ranking](https://arxiv.org/html/2403.18276v3)
42. [ArXiv 2406.19369 - RWKV-SAM Architecture](https://arxiv.org/abs/2406.19369)
43. [GOpenAI - Mamba and RWKV Models](https://blog.gopenai.com/deep-dive-into-mamba-rwkv-and-state-space-models-b45d5e6a38c9)
44. [Medium - M1 Mamba Reasoning Models](https://artgor.medium.com/paper-review-m1-towards-scalable-test-time-compute-with-mamba-reasoning-models-94955cfd4267)
45. [MLGroupJLU - RWKV Survey Github](https://github.com/MLGroupJLU/RWKV-Survey)
46. [Python Code KV Cache Calculator Output](unknown_url)
47. [Medium - Moonshot AI Kimi Linear Launch](https://medium.com/@HarselGivesh/moonshot-ai-introduces-kimi-linear-linear-attention-architecture-marks-a-new-starting-point-for-792829358833)
48. [DigitalOcean - Kimi Linear Moonshot AI](https://www.digitalocean.com/community/tutorials/kimi-linear-moonshot-ai)
49. [Hugging Face Papers - Kimi Linear](https://huggingface.co/papers/2510.26692)
50. [AlphaXiv - Kimi Linear Overview](https://www.alphaxiv.org/overview/2510.26692)
51. [Moonshot AI - Kimi Linear Github](https://github.com/MoonshotAI/Kimi-Linear)
52. [Sebastian Raschka - LLM Architecture MLA](https://sebastianraschka.com/llm-architecture-gallery/mla/)
53. [Sebastian Raschka - LLMs From Scratch MLA](https://sebastianraschka.com/llms-from-scratch/ch04/05_mla/)
54. [LLMs-from-scratch Github MLA README](https://github.com/rasbt/LLMs-from-scratch/blob/main/ch04/05_mla/README.md)
55. [Verda - DeepSeek MLA Analysis](https://verda.com/blog/deepseek-sglang-multi-head-latent-attention)
56. [Medium - MHA vs MQA vs GQA vs MLA Comparison](https://medium.com/@zaiinn440/mha-vs-mqa-vs-gqa-vs-mla-c6cf8285bbec)
57. [LLM Base - Gemini vs Qwen Compare](https://llmbase.ai/compare/gemini-1-5-pro,qwen3-5-9b/)
58. [Artificial Analysis - Kimi vs Gemini](https://artificialanalysis.ai/models/comparisons/kimi-k2-5-vs-gemini-1-5-pro)
59. [BenchLM - Gemini vs Kimi Comparison](https://benchlm.ai/compare/gemini-1-5-pro-vs-kimi-k2-5-reasoning)
60. [Medium - Gemini 1.5 Pro Architecture](https://ritvik19.medium.com/papers-explained-105-gemini-1-5-pro-029bbce3b067)
61. [ArXiv 2503.01996v2 - Long Context Evaluations](https://arxiv.org/html/2503.01996v2)
62. [Medium - Train LLMs with Synthetic Data](https://medium.com/data-science-collective/train-your-llm-with-synthetic-data-or-no-data-at-all-6939a377d101)
63. [Aussie AI - Synthetic Data Research](https://www.aussieai.com/research/synthetic-data)
64. [ArXiv 2502.16684v1 - WildLong Instruction Data](https://arxiv.org/html/2502.16684v1)
65. [Towards AI - Synthetic Data Generation Guide](https://pub.towardsai.net/synthetic-data-generation-methods-for-llms-a-comprehensive-guide-8e42ca207e1e)
66. [OpenReview - LLM Long Context Self-Generation](https://openreview.net/forum?id=EciNEiW1uq)
67. [Hugging Face Blog Issue - Llama KV Cache](https://github.com/huggingface/blog/issues/2345)
68. [Medium - Llama 3.1 Inference Memory](https://medium.com/@satojkovic/llama3-1-inference-memory-requirement-3c21d32e2f5f)
69. [Anakin AI - Llama 3.1 405B vs 70B vs 8B](http://anakin.ai/blog/llama-3-1-405b-vs-70b-vs-8bdifference/)
70. [JAX ML - Scaling Book Inference](https://jax-ml.github.io/scaling-book/applied-inference/)
71. [Vellum AI - Llama 70B vs GPT-4o vs Claude 3.5](https://www.vellum.ai/blog/llama-3-1-70b-vs-gpt-4o-vs-claude-3-5-sonnet)
72. [OpenReview - Blockwise Parallel Transformer](https://openreview.net/forum?id=KbqQMoqfLQ&noteId=qa7UDojewK)
73. [MLSys 2025 - Distributed Transformer Paper](https://proceedings.mlsys.org/paper_files/paper/2025/file/d5a655b8b373737b4f2aea8f78e5e754-Paper-Conference.pdf)
74. [Emergent Mind - TransformerFAM Architecture](https://www.emergentmind.com/papers/2404.09173)
75. [ICLR 2024 - RingAttention Transformers](https://proceedings.iclr.cc/paper_files/paper/2024/file/1119587863e78451f080da2a768c4935-Paper-Conference.pdf)
76. [Hugging Face Papers - Blockwise Computation](https://huggingface.co/papers?q=block-wise%20computation)
77. [Medium - Moonshot AI Technical Report Details](https://medium.com/@HarselGivesh/moonshot-ai-introduces-kimi-linear-linear-attention-architecture-marks-a-new-starting-point-for-792829358833)
78. [Hugging Face - Kimi Linear Instruct Model](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct)
79. [Github - Kimi Linear Hugging Face README](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/blob/cd777b7b94bc2b2b8c64d578dad5f393fbb14aaf/README.md)
80. [Github - Moonshot AI Attention Residuals Issue](https://github.com/MoonshotAI/Attention-Residuals/issues/5)
81. [Hugging Face Papers - Kimi Linear Summary](https://huggingface.co/papers/2510.26692)
82. [VitaLab - DeepSeek V3 Architecture Analysis](https://vitalab.github.io/article/2025/02/11/DeepSeekV3.html)
83. [ArXiv 2412.19437v1 - DeepSeek V3 Technical Report](https://arxiv.org/html/2412.19437v1)
84. [MLCommons - DeepSeek V3 Training v6.0](https://mlcommons.org/2026/05/deepseek-v3-training-v6-0/)
85. [TheSys - DeepSeek V3.2 Overview](https://www.thesys.dev/blogs/deepseek-v3-2)
86. [Aarnphm - DeepSeek V3.2 Experimental Setup](https://aarnphm.xyz/thoughts/papers/DeepSeek_V3_2.pdf)
87. [OpenReview - BPT Hao Liu Pieter Abbeel](https://openreview.net/forum?id=KbqQMoqfLQ&noteId=qa7UDojewK)
88. [OpenReview - BPT Hao Liu Poster](https://openreview.net/forum?id=5xc34ZSd6E)
89. [NeurIPS 2023 - BPT Virtual Poster](https://neurips.cc/virtual/2023/poster/72000)
90. [ArXiv 2305.19370 - Blockwise Parallel Transformer PDF](https://arxiv.org/pdf/2305.19370)
91. [ArXiv 2410.04731 - Transformer Modifications](https://arxiv.org/html/2410.04731)
92. [Medium - Moonshot AI Kimi Linear Analysis](https://medium.com/@HarselGivesh/moonshot-ai-introduces-kimi-linear-linear-attention-architecture-marks-a-new-starting-point-for-792829358833)
93. [AlphaXiv - Kimi Linear Tech Report Review](https://www.alphaxiv.org/overview/2510.26692)
94. [China Research Collective - Kimi Architecture Insights](https://chinaresearchcollective.substack.com/p/qbitai-kimis-new-architecture-impressed)
95. [Yzhang.site - KDA Technical Report PDF](https://yzhang.site/assets/pubs/techreport/2025/kda.pdf)
96. [APXML - Kimi Linear Transparency Report](https://apxml.com/models/kimi-linear-48b-a3b-instruct)
97. [Changlan.org - Sequence Sharding Overview](https://changlan.org/posts/sequence-sharding/)
98. [Axolotl Docs - Context Parallelism](https://docs.axolotl.ai/docs/nd_parallelism.html)
99. [vLLM Docs - Sequence Parallelism Diffusion](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism/sequence_parallel/)
100. [Hugging Face Accelerate - Sequence Parallelism Guide](https://huggingface.co/docs/accelerate/concept_guides/sequence_parallelism)
101. [NVIDIA NeMo - Megatron Bridge Parallelisms](https://docs.nvidia.com/nemo/megatron-bridge/latest/parallelisms.html)
102. [NVIDIA RULER Benchmark Github](https://github.com/NVIDIA/RULER)
103. [LLM Stats - DeepSeek V3 vs Gemini 1.5 Pro](https://llm-stats.com/models/compare/deepseek-v3-vs-gemini-1.5-pro)
104. [CodeSota - LLM Code Benchmarks](https://www.codesota.com/llm)
105. [Reddit - LocalLLaMA Reasoning Benchmarks](https://www.reddit.com/r/LocalLLaMA/comments/1k6zn5h/new_reasoning_benchmark_got_released_gemini_is/)
106. [Time To Act Group - LLM Benchmarks March 2025](https://www.timetoact-group.at/en/insights/llm-benchmarks/llm-benchmarks-march-2025)
107. [Verda - MLA Benefits in Memory and Computation](https://verda.com/blog/multi-head-latent-attention-benefits-in-memory-and-computation)
108. [Medium - How to Calculate KV Cache](https://medium.com/@mahernaija/llm-how-to-calculate-kv-cache-e29f095ac2ed)
109. [Sebastian Raschka - LLMs From Scratch MLA Details](https://sebastianraschka.com/llms-from-scratch/ch04/05_mla/)
110. [Vectors and Verbs - Attention Mechanisms Math](https://vectorsandverbs.com/posts/attention-mechanisms/)
111. [Vizuara - Decoding MLA Memory Reductions](https://vizuara.substack.com/p/decoding-multi-head-latent-attention)

**Sources:**
1. [pythonalchemist.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG1YgqWRa_s2FvQxUIFNpKJfhfuO8fnV_0mkKOojugwr3q7mGePkLPWJyXdZKFvPXNISnwDHKl2TDh6i7jl305J2guJ_VcT0PuOE4qa99RTVDPK4CIbrm3-ZAhWsnQRWUVHqc_7_j7191TnBPTkbU9gb--15oakUrKipA==)
2. [intuitionlabs.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQH7z7pzCw-8MdtLU0vQVdqdRF5BtKmWNWvraiaTHpspnskz009rbF8C0AxYDkPvaW8SRhzjxk2ytV5HPYNyVLRIB4uStYy-_sZidNlr6m4z8sNkxI5cwOg092zF16S2SfumyQ6fOi2zDKtojhiyVhx9IkLa)
3. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF2JqpcbF8PFccgZWuGqRp6RSkZGZhSq9qqmQ0UyYdJDVddlsPQTyKHLbYGr2eRfarddukSDnVyyC1lOKJffIT1rMONAvlUOd1_SV4d3wbFHdtiw9MvO222NfB0TIENBBgCMyjZaVmYz38KY8FmYCGw-04PW60M9ALlHMJryUXw)
4. [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGZMI0STXXTXXMqyEn941N1bOTz0dmj91gHHvsz2hrHC-nF-GRMn_nyCAMHxVAdXPeZu4samPYtsPAv6Dp_tnq_KNE8dyylxLVn2lZRzpDs60WDnyy8_k68ve5RZ0aXlko=)
5. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFtJNhJhHVjmdz8j7h1IirFDreo4SHQIOyJ2Ft6NSfolT2rzvZa6Zb6B21lEx9TqzCRhFqZa-gCQ0NOFSpHQMfkbOiAf9h7YGdi_fhZsLkPAuq_7XOi-pytMP5KW0OVOfkhunxNwtJnT0LzEdxybGVsUymAbxgSfK0WXwO6bjGLLfJZ)
6. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGJ7SJXY-BWK1A7t9GrZD8aLWmoQAUN6SfcL58AyZenQiNYUD6tT3OM3nadp8uPrbSpB17yI8NOP2n4XS4gMeJcGDJOKrvIgKNM0mBzQZauXetfVal-dgehFQ==)
7. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFfYfHjYnZiSLYfoJlYwYrNta0J8vc-fUuqt5_UOrNfuntU2ssaUqDMPoUZA2taPKqatjklsy_yjPnhILpA85_vlwAfJ_PjEYes9xgShA-wLD6Lqg-ji3FvLQ==)
8. [substack.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEga06UqLdHlXuIu_Z-qORPGHbmhd0w3qGjPD8xuxGvuWfweHqFxTsRsr4EyHB72Gjxc_fldQKQ70vjrzCRyhl_YQmGsueWtzOa8KUtia8WLMzuWHbkRUeqESmJnSi-Pcw0I3zv0Gy2Y7fZ-LyCJ6fQNN4CsRfx701u)
9. [usenix.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESh3Jver4Hm7LnUTW68h_EB67WmXSF4yh3RbDQMtvcdAgGPNmQuN6rd_bC4xEBjajuIOhNweKnTD0XuPsEhYJ6ZU_EvQHbFt5nOYknizoTG8fySBho1xrn8re9FhRCCXcgzLGQi_ibCGJe-Mk=)
10. [Link](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFwNQXw7mpE2TWVIuYxm6p3AV2pIcmk3uftZuSnbQ625pLbH9HaKs7VjfAqLTTknCvMKFMpEwDG606SeAMDv2lGNQ==)
11. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHTuo7CWTJzTeZr1NHN8Eabs516Amq69Cv3SnaIP03JQRigfR31OBkEb2bX9lwKu3_CAtPwNxMzO6gm5OcCr9x4b2zOHvChzWMza15LuUjd-gtSumVGcSXOooUeY0pKEsDKwGpAWJkPLwKece9IvDTIF6l4oWiZb2c78FBpSpk2Lx8iygIIe3Y=)
12. [github.io](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGKtovLeLW7TuqiX_d60s26-E8jZ9S_IwykXn3Z6qmYrF-0Ev6Xwiwhmjf495CEHv-UrVEjOHepyqzAlv3DwYqhtgIoAAGXtgQWoaasittZBtWB2JaYHZDqyk7UkIQy4CuHXicJ2_p4EZPrrfgvwQ==)
13. [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGWcLMUhOzPTGTKLxXhom7EIqjbEvgGbqSmwBx8_V0C9e0dLRetXNZOOfZMRzsOnOycxjjq_ewQYj46t2BU92vLGZm2SPf50LnGEiUcYHwhAcYtpJbtxORze40erk1Fd9LyoDL1OQ==)
14. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGI_oACnWro3FjcTjhfZ-jMjnsvsjoeye2OZE4Mo3kzltco8QUtoqxHvr5swQz7jxoE7t7fLizlOxKxEVXeQhh-3acTnGp0gw_N87XqWcY0mLCbU6po-CSnzg==)
15. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEdTU65awAL60uzrhE_Cwsro4zIZCr3tseZJ8HWZ_nJjaL7abREOP0CTaC32BWyEmgT8DWQs-McW9lj7rWGKglVhSY9iCrBEvwoCEkv0MtkF3ao7GAAN9GhAoLYk79dDCp9dZsPNX0P7-xs8vEs26S_Chijp5A3R_qeEbITAEQ-NB6r2lVGE-BCxNAxIDxfaTPMa86Md-OS6MxUWL09Jt1LQ-FBBJQkdcOUEPxYZVBrd4-ryDieJCT9vB1GkHXTg3Cpag==)
16. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEhQC2zYaIN_dyB40YZ7Qwimr07H_di7FxjKAYOp2UjbqB0tveDLv8vLEAMJRyw4owYHC_9JWkTow57qzeej2FIBvfSbLasgTsz7DIXngl48O2bcJpznD4KeK0C3aBLFfNmoByMizuKN4NdBErFKAF-isKS2f-PCYoO)
17. [vectorsandverbs.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFIwdSPCHX2NBYY6A90DpPC-HO463SfMmsqMFKqTkxDIvEl9l3z3hDzXRdAP98yGgEfXcV-uMaqatg9h2-zwWo3H58-8u--uy9aH7SVq2KgxYMs9Ek8TIBy2nijkj-D5qJPgTnqXbs4tylONgiT)
18. [abhik.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEckUtJO9m75kudp-ECcD6dcBLiFNfMQONRUL7fBmFioCctdlmeXFErniGoTvAYaA-y-Ez2zEcc0KTKYMw2ZtD7fuJflxgz4TfOCW0qghMVPRyK1TAnPjlUNekECjuKsC_xNQGvSMQiDl6JfrPIKwQmVGo=)
19. [sebastianraschka.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHLJwY9HGdrkG0w29zGrA0WdU6A86nXJojprYk1xIi7X6HOUNDNPM1tAIFHb9_MyCUDDNXzHp7IU5qy7XgXxInplE0sVvS7O3pPhPz-OCKnxPn9cWj61rVAT-8QqMWdxJGmUktq-5nrsaEWsCRvWKn6)
20. [github.io](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGkhpePigvLgtExuLG_xAq3j8RFuagKee2D_vAi4AO9-oIlcuJShWGFz56YU67hjPhNYuIuAw-lOskENH2WM8hr5G94e7Kzqnk4LcU6dpDti7WaM_qND0pONsOyYHVtK3-0XKFLNrT9p8-qRsUYug9jgHs=)
21. [mlcommons.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFIiaXP7kU-rwoloIMPTHeF8NWZ7mLkQj1idzCaTDhLRCisVckFdxrE5jRFserS_wJucEfozlqfRkR9JMxnz0JMAVxkzvNTEbg_gW0tlsYLIabHLr7AnH9ewqgqQqItXwshMCZHO-cCxI-3WJB8ZQ==)
22. [verda.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEUnJ94sj38jhqs6LxSn7x3R90Vz0EEjCw3B0B2sYnh_KLFSkG5bHvYMJ1LvOpgzbEgPHgLs7fUx4ozTkCk8KV-EzATdWtjdyO3nblH0rI72Ip0ex_bDQHuSgNQ9uQBQp7Ofg2vLtMeBHtRTn48b3agOHVLQ4jgRnfenCDbBgeTmd4O4RZXf3bmuXlD)
23. [verda.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFMjRG9bnE3kTjKDff1ghgPC2FW4_wffds1sCEAZsZ4E_K1k5lPbF67pWyCr-4DoccLiDMDDrgSLT3G2Aafi70zTNjVq_dHPuK5djMpeM_7yaJMIS9qSk4aOhiJMgWEkNKgymr9UCoh9elcTHK6DQf7y-9NgYhdaDg=)
24. [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFAhVxrm-XGJzIMaCfXneQ8-rQVf4GMebAK2lYI1DMYMW69Mh5SY-6_e4vE1-66TJAGtJMB50aPFPyeHuwykZarlP_e6oXXhQwU8o1tVfE898S4X2Nmwzot1cflD3M7cAEGBhg3N8OcYxT5n3tILWXkBRo219ze400mb1-YoDkzeg==)
25. [aarnphm.xyz](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFT2Qvyc0rXaP2HM_8DTzMZPqWjdW0V7_uu3D2ZIFMBlbY0MHoYRJ5j4VUWMGLOVmcE4Unre8Xz2wvumjn23gZ-u8hlqfR7CUJIP_7OmOTg43s1p5aPog_patoYrIaH9BL5MK_Amr1OvDZlug==)
26. [thesys.dev](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFUwg-Q8CJE6UfyieiPDK8mhNr_F66kigFPTr8EVZg-rMZAqS9uW0JY6BkJVACRYcfVq2QW5Iz6z-x-gzibAIIbT-Z0Xmh-tGnDR6rj2o_2tVf9Fszzc-m7QvfRvlKZrB0=)
27. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG1LP0H4Veh__5xLGeuTi3sQrYs-fdqGx8H4C8TPBxzk6frJU2ss6_s08faXOp_7ecN_YjBMQKZmnrQ7_LNz5G_uHI7pdUx-HcSEPKD3mIQJy7QGdQuuLldPppYbQp7ux_b-6IRdnTwXVUb0naCttYLOe_U-v9UQrztUP9cnqjwvp_4IKoVAbAfV44Ps5emFCdtBmR6nHo2gByrrhckxBBD6Bx-QakRGLpi_6-YptjHJYZRa7NcJLxyyvGhbO1ZQ1MR4AY8)
28. [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQH8B054v4MMCtfk8cveDHxfTpO-QhLjNtrTw_fsgBce8GHUCT2a5dKAwjj7gbzQQUpWJKBRjq-PWYvAIMBtaYuE1sALyE9jdokUG2AsfOtUjPezMTrSYh-1aozsWzCX)
29. [alphaxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGiDVyHgTUMSg7HGFN5SxgMVN-b07Zwy5XXvr0cF-Cmdwvr65C5J--GNGaA-FxlrfmM1uYW7aQZSiLHSdnBy3cJIbMtY7jgwjzX09V7X_CqXthBcOP3tAX1Bbm2m_GO7AnGZw==)
30. [digitalocean.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEVNen1yL6erbYu9rpB4Ns8qfnv5vTBauCLzmvTWPaeS9YnygOMK_56S8jz0Uk96ln0knjGYEYHERVfJ_ZT14JMVxG9jFAh-U8HMyhm5GGuV3mSaVYvKHeR-u3oDQywy_9-5gCcmIwuZwMDnlkLtrQabHTpOvxefT_HwBBqLbY=)
31. [apxml.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQE-yB4vtfoI8cx0hJZtU7IjFJ3_DPIbfnDJhKJxkFWO_D0r8tkfEuTux0uY7qMIznGTohw_OcwoeIfBQQHCoyfbwH3sgFWJKnC9PEZUjnZRHgmLlKiDuaj-oyfHTVXOlHb4TzJQtqSQ8MWnwA==)
32. [yzhang.site](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFuREQo-EoP90En3GfH3HAm1QTrMydWXS39g36cqocGWf05E2DngVo3_y94haeXRYeLkwOhfn_tSRdvmymwldb7ZCZ9sPA0Z7GVweRMlNfVMzKNGvI2FxxVthJ1iFTTLyn2yO6d0VrsaxE-uAfL)
33. [substack.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFC5tCXMPKmpISIQ9fYyZ5lzg_TnXxWaLuZaW7Fv4Q2OrBJWQg87qEqkIcRWj-SJuvgDO9RD8jRmAzKQQXI5-J9KCIIr2kllxyba3wMqF9K-FPh2AcjfgBf4wzdNtvGIu-TdAaQN4lrPQHgQSNRd-lbwkYaYsVmm4JwL-PDUKFHzxhPNl6Q_jXoavSVOw==)
34. [gopenai.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFiw7zLPFmpqgtGuoN1U8LXycc2wk5fh7NlMTSpMpj5Q8Yb9HnHwlFTz0BUmYzsMgYo4E9SLnaGN_qr5QgN7n61f3_ZGCGa1MSfC6QH4nVohHstvKyjC3eXECdyYmjH27PqcRuAS3l2yvq7yphqKYuf-_7QGF-FQJYN_UIpjrCv2WgsGzH8NTmIEvYX4Q==)
35. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEpi1rFCoNe87jRyyO70asyvcb1GGg7WmrgURI_-ORH9QBzvdG7z7tTr_i185jSdFAuDUrDRFpVLpRR9nla9vLS0XARmq2UGsbus-BciMoLkaFE9OkhPVwLCBE_8_s0d3c1EdV7GE3kExbfYtXuq119V_HuOtnwSHShjBzifN-NErioyyT-fiCRmEyl1CaNNZUBYuFTXsqa9lCU9r6YF_76RWzO5NWV5jlsgZs=)
36. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGoXUnZAmHbw3D7P1BpkjzsfF01OoqAe4jhaLMOpyfe_kCOjeZdmDGTuexT462ty8upc_2fUlxiN_f8MW29ADjyzJllrSjRaCkecO6thDrNZ0yBmo2a3g==)
37. [emergentmind.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGTZ0nO7VesiLWTTXQBlw4gK5BtgNVt_61AnUOp9CmKbNr-r0iFTvrMAlQ9DITnJEtCZRcbzfhbCsiHoxdphihq_ghhImdUVEeeqdi5MstB9BnvJrIvLhIo6bhGY_OeiCFMBQLPjbJH0S9Uf3sleP-dNs3rcVNzFGE446J1ymOi1A==)
38. [eleuther.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF_ZE-OVAtft7jgJ2tel8m3IBNCce4EblFjBfvH8-pVAoMLZHEm0meBjA4b3Td5KodWx0GcKAvKm-rFNbh177Lvj9GgsqQ8yR9IxC22PImwSUq_iO4=)
39. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEz6_WTNYevvxNDrTwCcisxl-kWVlS4GTjvSOUH62N1stp_QI2GWryMI2gcyr20YuYwz37aLwdnW1cLg4FwhMt7_Yx9fP09aLBcIXJlplCAhz2ldIASxMmFytFcxqm0Q1-ekxDiC1t5vLHstD0MOHWi68OkfCbG3WH2CSQWGNygapvbhUfYeRJgvCUhR1FN)
40. [aclanthology.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHHMhfzC4_V8TxIDrb7ql2UjAcQb7qhMxml-K54YwAtCX15WilSwmg7xH4OQdnyc-w45nJUhZ8L68BCYTFm-bE9GN8kdJUVyg-0EXE54ngPL-JLFWVrWW3SAo4Z9ocgS2l_5s8QE0px)
41. [baichuan-ai.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQE40lRbvydD19zJbj_FHjCgNYN1LN5sG-BX580KzvaPiSujcXNtUYVcub7PmSO3lOTLZrxBlPaCcjcMAx0HUvPS9x-_qyGQEJ3u8OyusYLXneJcMkTMDluTC2fwkf9wJcqWXlPcuUTBC1cm5hTQgHKGYBaridR2)
42. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQH9bWkjrdReA3qwrLe7mYTCP3IgMRrZBDiKo4nHcK6KRiyJzTxrdq2YPNbEnP3QPOXWsfEmafJxI-SB4x90p6WdP87bkrkP-nd-gEhv77reZbsrtsMhx8r-YUsJzEd5yoacdkMEpSz_piy8O0ctvYoxXPL22m4bhbz5NPY7uAJ3PHMa3XoqW984jD5eosgxc3sawEny4uz2ApMQBz8NORzn6JWxVmD1K8fUFkmx--GyhP1JMGBaOMRPzYrywIBYAEd2PILI_KaR_g==)
43. [changlan.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGUlEWZ_qDrzHb_jDZfO_zT6I0HuFhgdPTyX1BXlUhUmG-bpxqoOV6LBTSxaO29F_KqVOAwM5W8YaWIzgBoLAdnPe5G6hFf4DddL8zCUA5UfM8_7onSx8_hJr_7t3K70btmKaw=)
44. [axolotl.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEZgtG7-WFY0enPHFVfVHc8C6PFJrF_2sW95xDTETyKUj-Px9d_cGUmJuUY8IYElxwur8AQU_1n4-mEBanI5I93OK9h9m9ZUpeYc29ErBPnU-jKE5ZOJfDWWkkHzrmRgC-pTfzTDRU=)
45. [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHoWAyOsPc9UN2Y42sEXHF-5UlhNMyhJF3JRl5Oavhe5Ze5BlG98KsOmtFS635aaoD6wnqDvGsx7DIOZlbt66IBFcQwWWM6NfcXsxTGDSwH3FDsit2OWS0XxrSnt93_ykXudQ==)
46. [iclr.cc](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQH4HC0xJYGV4VxQHDxI62LhigRJKgabS-zPTBN4vK7Dd8G3q1qELHBUVXfjVD4ADp70SOd4d-watnTWD5zj4Rcsv2uZ1VsksLYog308sT4OiRD8C9-_U7npeTjG4BDT3GqStvgEZfxlXNKNLdJke6JpsBbNo6jjCm32hZdnEd6ifrOs2KrZGlH0ivYlvIxfqIGrGJlZH9vf2VWnSIdsxGqwDkgo)
47. [emergentmind.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFbW7rW8nwiq6IwaJsNyWb0AZdEt4D9sKcF1N7WZHa_2p9abQatXJThlV8M0Ro2vCq-eRLb1w0ubJLMtTfoJk8dFvA2z4N7qyo-SVIsOhfWXtVCzKFtnuRAF92A_0LaAtK-3qgO)
48. [huggingface.co](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGKlkOT3Cc8kT_0zcJpXV4LnVC__7G2K0ufCnYzUhzjECmwS5h1KLPhk50vDObiypzmDdQ8m1b3WdRfWf5k70EJrxK2OEY2Zouz5nPYTjvaCFTDmhhMa-3fhaePI7FfxGjhF1siLoP-Fue3G8Z6Qq7WIIf2zeZqA9ordFwqfjvIFg==)
49. [vllm.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQElP6p9Xz14Dhd8cwetGjMeFgHrW59-H7r7424fzBT0iKX1TsWby8jWArdEOpkknNlLbv7-mr4hqnzlwS8xR2A1R6YzZANZEmZ9Zbzc4Oa444cFOgb6ibTYfloh8CCO9xPz0t3z7nA0tUwjrNfqe9Vnk11x9OGuhLzG46P7_4X3Lv0UO89WpM-Rt0C3Vqm9fsgGsscVt9hSswWQRw==)
50. [mlsys.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHmKWvIUIspnr__s0QYaKF5PAWlEDW6ineBdqzM5X4FAI45NJu6F4uEP9MkT1oKsgQjOxj64eSzbvWT9CQoTCJc10uzvaz9DRNXzuAPHWBlJarsxaygkfhSawIeoHKmQDgbm8rXoMSNNfjj-mcJWmzCzu0BPnnKRpb5iJIlImdpS4XJ5EadLBWbiAO8vlyOAj0lyItt3A2xot8yyFwLXQ-JnyIYwRU=)
51. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEhvvIIBYG1fu0TpffoJiik4wKH7NP6ZVAJk6EweeogvA9_6pV_fho-x4AogBMofO__RJBbyV2torWrtPS3m6ia3t3hVEkDpM2j2IoFk_bTnk6xot3QuQ==)
52. [openreview.net](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHCHYpqPyU_plk3dOQKCA7oAts_nU3DTEqpbQtspNc317fD8OF0BQk-j6XLNc_msJOiiQf8vcVmvlEZp82xS-iI-PEutIfFm2AG4iwD3tDEs6Ov5HEOePE6WNaIEzFk6sU-RJLOOKNfIMzqJ9G-K6CTVys=)
53. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEbBmJRbDZGRRQt1QI9CA7MEgs3wvkFzgEOwB5UHFb8wz09ieOweNkWL667Q5OZUrXF19LZCerTYSf8pjlonZZwhLfqVjL-6WLroLwLDpsjG7CNZ5ea0g==)
54. [towardsai.net](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGMKVBkFw4ey3Qof3U0zDFqEMsv_695WBCsgAfAj6FevNfqBpYOs3qjLVgJlrzOzLLSEQ6b430ECGWHx6xT6WXgQYgz2-p1zMOiQ48t81OI7eYANvZ6u3j6u0-EqvoaNaFlDrIZDsCHdz5Jb5Akyx1JyQIM8KnolDPAPxRV7H7lFH4nCfOzIeTfntMRfKzHoj-epZVPt5didFkIhjvf)
55. [tetrate.io](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEp9og8rDmefaJb4KgJqMfU2R9sCLXlTfy-19mELxr2wWy6mfap_554tX-A27P0vV3meOlok6wzU3hYVE7snSw7KOgVn9bwO87h6SR36-_rfUw2wAYWjyrsryXkMhojZDymtbVG2w9gV4zX-3R25WuM)
56. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQH-KignVst3w4ClTZf3SwFBP384FRZ4wFA13FW5KawLugbPUiJIPAmG6ttOkUVY_9WN6aayNHC8rY6IWkfjt1L101KYKZ_JCczjE5ZatROozthTngiTza06yg==)
57. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFJILmsLhhAw2_7PDT7eb7wSvOwTDDcYYGQJJno3-E0zM8VfCnZHthTePel8YLvn31G1fg5ITJuqkQ6cPY_Ni7r5WQXr8-Xnuf7DIzr_qJUFIulITafc0qwbimKl5X7UmrnJZpFi1ZwFtiG7Ex3q0X865vnSnaLPFn51SERB8iSj1NnqIr7Vi6vce01RNWl7gZMiDBQP4TZs8xqW1d8woYuj9e-vmdnL32nWg==)
58. [confident-ai.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQH69uOmp3HJrQUsPozfxtsbQbnDB42FjW9Hv39P0peBWtWRRHpy25CQ2sTj3VN8hEe2cZuLySE3fsNZFiLy7Dv-6-0EKOgDml0YBAj2Uxiqkr0lu075YWkHMFVFhkokOHlQvTbg8AZtU5xta-pyIa3OwJaQNZgrKU6zYE_mG6rTnjrt7aWd5mDM16PcumMdBNc_lg0H)
59. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEmcUMnle6oaHJjYu2IRorjjQa25VXlOwVijIWIlsOETycAPi2vhv7sDph31dFsiTFE-3bhal2DEQhXQAKfeR2A7l6NpI1P6YDoSyp7-jAvuAkKJloXIBSRHa1EfCGzOsKeUfuv9voW3YEYLIetv70jccJQ9yRtwkmTgLR1Rr0XMrq8MgjQPh8JGUfY-wxWPRVrHKtjO22h17AvVkzzJQ9TVd8=)
60. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHHgfSXpM5LuWi1ETAL5V8HT8ggSip9ru9EqaM6Q1tupBMx5uoGt6ZGT9rRbt_qmkafWWJiQNwQuV71AHEh-LipEx_wwwUsdbd12DmLeSgXnif5-72Nm_aYBw==)
61. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFnW7ny1s7I3xfH5f8bNcIkhTZ3hu9srVY4zOvoXaVTUCZDe0H0pIIz4kFbGy-yoycC1QDFTUguWZvjcFasPzORtVNiVIIyK3rELu7m0Fi6XCgRya4zosq7zQ==)
62. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFdpLXSWjiOd7xzBnbm2tqVrMIhRWXMqsjzbWTOgXxmTp4LFbQ21mtVq5N-2WEuhHJk90QK-iuzpzIdqxsNQBSK_7SLL291BOOut4rhhItiOHMuTZvpdklKWw==)
63. [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHIPzbEzWIBX8c9uTULo131JPCjE7E69jHd6Hu2_ltWcc_T4ljVZ6cWo8fOUN2AnSJwEYvIt6ufMiR-XwD2Iutrz0GFOIPhROglasUksrq1v_Dn-q-P)
64. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGq3UaSFo3pOER-PBmRs-m3OM7L7Pag2WujKIvwdUM-xFfYam26xxi8hwa33xAa7LcLFDM5KTj6xDHub0b67MDHHlhxLjg1M-QN2zl_sS1eZ2PmsHb6vuqEaw==)
65. [llm-stats.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFPmKuvuDkibH1s6zo-UkT3snanjJaFbz-dk8ejHPsaz_5bk8VOIh_jqolbpA3hZ3YpbK18H41psFOyu666nQz2mtDjcjpzZWEKBGeohzA5oc7ecwH-1LyIlFEVBwWyf0sUJWNbKL3xXAcD62RFINbwIt9QfhXoG2g=)
66. [github.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEvs2Bc1D4tb4BTHAW3_4nEATXSgpNNsMsQu3Uxy9CYBKv_swz_1b4rcY5w-lDyckRUqejUFmSgFrZ4m5k84yIYUDyL5s_3SDFs70k1Wz3NKFzUk1oVxuFKqxk1PzpZNg==)
67. [llmbase.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGlniSG_TPahF71KYE_tjJ9DARstf0Kfo3NWXTnpzhw_enxDS_CmHmX0Jh-uaMreoqAYD-01TqOSRqkOJl8MMCKImt728AGy3Tw4PKaiMEzZiUbgRuW2A71hsvPjWQdFfb_pH0F9iwmfVLO5g==)