# How Quantization Shrinks LLMs to Run on Your Laptop

Model quantization mathematically compresses large language models by rounding their high-precision numerical weights into lower-resolution integers, drastically reducing the memory required to run them. This compression shrinks models by up to seventy-five percent and significantly accelerates inference speed, enabling massive artificial intelligence networks to operate entirely offline on standard consumer laptops.

For years, the language model arms race seemed to belong exclusively to massive cloud providers and developers willing to pay for API keys. If an organization wanted to run a 70-billion parameter model, it required a server rack full of enterprise-grade GPUs costing tens of thousands of dollars. Today, open-weight models have matured to the point where sophisticated, highly capable artificial intelligence can run entirely on the consumer hardware sitting under a typical desk [cite: 1]. 

The primary barrier to running a Large Language Model (LLM) locally is not raw compute power, but memory capacity and memory bandwidth. Neural network weights are simply numerical values, and by default, these values are stored in 16-bit floating-point formats like FP16 or BF16. Because each parameter requires two bytes of storage, a 70-billion parameter model occupies a staggering 140 gigabytes of Video RAM (VRAM) [cite: 2, 3]. Most consumer laptops and gaming PCs max out at 16GB to 24GB of VRAM, rendering uncompressed models impossible to load. Quantization resolves this bottleneck, serving as the critical mathematical compression technique that makes local AI possible.

## The Shift Toward Local AI Inference

Before dissecting the mechanics of quantization, it is vital to understand why millions of developers, researchers, and enterprises are actively shifting away from cloud AI platforms and moving toward local inference [cite: 4]. The transition is driven by a combination of data sovereignty, financial predictability, and system autonomy. 

When an LLM runs locally, the entire inference process happens on the user's own hardware, meaning prompts and source documents never leave the machine [cite: 1, 5]. For businesses handling sensitive legal documents, patient medical records, or proprietary software code, this localized execution completely eliminates the risk of cloud data leaks and bypasses complex regulatory compliance hurdles like the GDPR or CCPA [cite: 1, 6]. 

Furthermore, cloud APIs operate on a pay-per-token model. If a development team is processing thousands of documents or running autonomous AI reasoning agents that generate thousands of internal "thought" tokens before arriving at a final answer, monthly subscription costs and API usage bills accumulate rapidly [cite: 6, 7]. Once an organization purchases the necessary local hardware, the ongoing cost of inference drops to zero [cite: 1, 8]. Local models also function without an internet connection, making them ideal for secure facilities, remote edge environments, and offline application deployments [cite: 6, 8]. Finally, local hosting shields developers from arbitrary API rate limits, unexpected acceptable-use filters, and invisible background model updates that might suddenly alter a cloud provider's output behavior [cite: 1, 8, 9].

## The Core Concept of Quantization

Quantization is the process of mapping a large, continuous set of high-precision input values to a much smaller, discrete number of output values [cite: 10]. 

A highly effective analogy for this process is digital image compression. When a photographer captures a picture in a RAW format, the file stores every tiny gradient of light and color using 32-bit depth. The resulting image is pristine, but the file size is massive and unwieldy. Converting that RAW file into an 8-bit JPEG drastically reduces the file size. The compressed image still looks perfectly recognizable to the human eye, even though millions of subtle color gradients have been mathematically rounded to their nearest generic equivalents [cite: 10, 11]. If the image is compressed too aggressively—say, down to a 2-bit color palette—it degrades into a blurry, pixelated abstraction [cite: 11]. 

LLM quantization operates on the exact same principle, but rather than compressing pixels, it compresses the neural network's mathematical weights and activations. During the quantization process, highly precise 16-bit floating-point numbers—such as 1.45209 or -0.88712—are clustered and rounded into a limited set of discrete integer buckets, such as 1, 2, 0, or -1. Detail is inherently lost, but the overall numerical distribution and structural logic of the network remain intact [cite: 11, 12]. By reducing the precision of these numbers from 16 bits down to 8 bits, 4 bits, or even 2 bits, the model requires a fraction of the memory and data bandwidth [cite: 13, 14]. 

However, just like the heavily compressed JPEG, lowering the precision introduces an unavoidable trade-off: some mathematical information is permanently discarded, leading to slight, measurable degradations in the model's accuracy, reasoning logic, and language generation capabilities [cite: 2, 11]. 

## The Step-by-Step Mathematics of Shrinking Weights

If software developers simply took all the floating-point weights in a large language model and brutally rounded them to the nearest whole integer, the model would immediately output incoherent garbage. The delicate mathematical relationships between the network's layers would be destroyed [cite: 15]. To quantize a model intelligently without destroying its predictive capabilities, engineers utilize specialized mapping algorithms. The two foundational mathematical approaches are symmetric and asymmetric quantization [cite: 16].

### Symmetric Quantization
In symmetric quantization, the original range of floating-point values is mapped linearly to a new, smaller range that remains perfectly centered around zero [cite: 16]. 

The process begins by identifying the range of data within a specific layer. The algorithm scans the input values to find the highest absolute value, known as alpha [cite: 16]. Once this absolute maximum is determined, the system calculates a scaling factor. This scaling factor is computed by dividing the maximum possible value of the target bit-width (for example, 127 for an 8-bit signed integer) by the alpha value [cite: 16]. 

With the scaling factor established, the actual quantization occurs. Every high-precision floating-point number in that layer is multiplied by the scaling factor and then rounded to the nearest integer [cite: 16]. Later, during the inference stage when a user asks the model a question, the local computer rapidly approximates the original weight by dividing the stored integer by the scaling factor—a process known as dequantization [cite: 16]. Because this mathematical mapping is perfectly centered around zero, a float value of exactly `0.0` translates cleanly to an integer of `0` [cite: 16]. This approach is highly efficient for hardware computation but can introduce severe rounding errors if the original weight distribution was heavily skewed to one side of zero.

### Asymmetric Quantization
Asymmetric quantization, frequently referred to as zero-point quantization, is utilized when the weight distributions in a neural network layer are not perfectly balanced around zero. Instead of finding a single absolute maximum, this method maps both the exact minimum and the exact maximum of the floating-point range to the absolute minimum and maximum of the target integer range (for instance, -128 and 127 for INT8 precision) [cite: 16].

Because the entire range is shifted, the algorithm must calculate an explicit "zero-point" [cite: 12, 16]. The zero-point is a specific integer value designed to perfectly represent the original floating-point zero in the new, compressed quantized space [cite: 16, 17]. Maintaining a mathematically exact representation of zero is absolutely vital for neural network stability. If zero is poorly approximated, the network's "padding" values and inactive neurons will generate micro-errors that accumulate massively across billions of matrix multiplications, ultimately destroying the model's output [cite: 17]. 

During inference, the hardware restores the original approximate value by taking the quantized integer, subtracting the zero-point, and then dividing the result by the scaling factor [cite: 16]. 

### K-Means Clustering and Fake Quantization
More advanced approaches utilize techniques like K-means clustering. Rather than applying a strict linear scale, these algorithms cluster numbers around calculated centroids. The continuous values are mapped to a discrete set of centroids that best represent the overall distribution of the weights [cite: 12]. Furthermore, during the fine-tuning phases, developers often use "fake-quantization" operations. These operations simulate the quantization noise during the forward pass of training, allowing the model to adapt its remaining weights to mathematically compensate for the anticipated rounding errors before the final compression is applied [cite: 17].

## Memory Savings and Inference Speed

To understand the tangible, physical hardware benefits of quantization, it is necessary to establish a baseline. By default, most modern LLMs are trained in BF16 (Brain Float 16), which allocates two bytes of VRAM per parameter [cite: 2]. A general rule of thumb for quick estimation in the field is that at 8-bit precision (INT8), one billion parameters equal roughly one gigabyte of memory [cite: 18]. Therefore, an 8-billion parameter model in INT8 requires about 8GB of space. 

When quantized to 4-bit precision (INT4)—which has become the industry-standard sweet spot for local deployment—each parameter occupies just half a byte. This compression frees up 65% to 75% of the VRAM used by the original FP16 model [cite: 2, 19, 20]. 

| Model Parameter Size | FP16 / BF16 (16-bit) | INT8 (8-bit) | INT4 (4-bit) | VRAM Savings (FP16 to INT4) |
| :--- | :--- | :--- | :--- | :--- |
| **7B / 8B** (e.g., Llama 3 8B, Mistral) | ~14 - 16 GB | ~7 - 8 GB | ~3.5 - 6 GB | ~65% - 70% |
| **32B / 35B** (e.g., Qwen 3.5 35B) | ~70 GB | ~35 GB | ~18 - 20 GB | ~70% |
| **70B / 72B** (e.g., Llama 3 70B) | ~140 GB | ~70 - 75 GB | ~35 - 40 GB | ~75% |

### Overcoming the Bandwidth Bottleneck
A common misconception among local AI beginners is that pushing an LLM through a complex dequantization process during live inference would inherently slow the model down. In reality, quantized models run significantly faster than full-precision models [cite: 17, 19].

The primary bottleneck for most LLM inference is not compute power (the raw mathematical operations executed by the GPU cores); rather, it is memory bandwidth—the physical speed at which data moves from the VRAM memory chips into the processing cores [cite: 1, 19]. Because an INT4 model is roughly a quarter of the size of an FP16 model, the graphics card has to transfer 75% less physical data across the hardware buses for every single token it generates [cite: 17, 19].

As a result, INT4 quantization consistently delivers a 35% to 42% speed improvement over FP16 [cite: 19].

[image delta #1, 0 bytes]

 Shrinking the model weights also frees up massive amounts of memory for the Key-Value (KV) cache, which determines how much context the model can remember. In production settings, converting a 32B model from BF16 to INT4 can free up nearly 47GB of VRAM, translating to a 12x increase in the number of simultaneous users a single GPU can support [cite: 7].



## Evaluating Quantization Formats

The open-source AI community has developed several distinct quantization formats and algorithms to address different hardware constraints. While they all aim to reduce bit-depth, they execute the compression using entirely different methodologies [cite: 13, 15].

| Quantization Format | Optimization Target | Key Methodology & Benefits |
| :--- | :--- | :--- |
| **GPTQ** | High-end NVIDIA GPUs | Uses calibration data to measure weight sensitivity, adjusting un-quantized weights to mathematically compensate for rounding errors [cite: 14, 15]. |
| **AWQ** | Mixed hardware, Apple Silicon | Protects the 1% most "salient" weights based on activation patterns, heavily compressing the remaining 99% [cite: 14, 15]. |
| **EXL2** | Extreme VRAM maximization | Variable bit-rate. Assigns higher bits to important layers and lower bits to redundant ones (e.g., achieving exactly 4.25 bits overall) [cite: 13, 14]. |
| **GGUF** | CPUs, Edge devices, MacBooks | A unified file format utilizing "K-quants" to mix precisions. Excels at splitting workloads between a CPU and GPU simultaneously [cite: 13, 15, 21]. |

### The Mechanics of GPTQ and AWQ
Instead of bluntly rounding all weights simultaneously, GPTQ (Generative Pre-trained Transformer Quantization) operates sequentially. It uses a small dataset of sample text—known as calibration data—to measure how mathematically sensitive the neural network is to changes in specific weights via a Hessian matrix [cite: 3, 15]. GPTQ quantizes one weight at a time. After it rounds a weight, it measures the resulting mathematical error and adjusts the remaining un-quantized weights in the matrix to absorb and compensate for that error [cite: 15]. While highly effective, GPTQ is sensitive to domain shifts; if the calibration data is strictly English Wikipedia articles, the resulting quantized model may suffer severe degradation when asked to write code or generate Chinese text [cite: 22].

AWQ (Activation-aware Weight Quantization) approaches the problem from a different angle. It operates on the premise that not all neural pathways are equally important; empirical evidence shows that roughly 1% of the weights carry the vast majority of the network's critical reasoning capabilities [cite: 15]. Rather than treating all parameters equally, AWQ analyzes the model's activation patterns to identify this critical top percentile of salient weights. It protects those specific weights by keeping them at a higher precision, while aggressively quantizing the remaining 99% [cite: 15]. 

### GGUF and the llama.cpp Ecosystem
GGUF (GPT-Generated Unified Format) is not just a quantization algorithm; it is a holistic file format created by the team behind the `llama.cpp` inference engine [cite: 13, 15]. Unlike GPTQ and AWQ—which are typically spread across multiple configuration JSON files and `.safetensors` shards—GGUF packages the entire neural network, its tokenizer vocabulary, and all necessary metadata into one monolithic binary file [cite: 13, 23]. 

GGUF pioneered the widespread use of "K-quants" (e.g., `Q4_K_M`). In this naming convention, a `Q4` model utilizes 4-bit precision, the `K` indicates K-quantization (a method that mixes different bit-widths internally to preserve quality in sensitive self-attention layers while crushing feed-forward layers), and the suffix (`S` for small, `M` for medium, `L` for large) denotes the block size [cite: 10, 24, 25]. GGUF's definitive advantage is its hardware flexibility. If a 70B model is slightly too large for a local GPU, GGUF allows the inference engine to load the bulk of the layers onto the graphics card and seamlessly offload the remaining layers to the computer's CPU and standard system RAM [cite: 21, 24, 26].

## Applying Quantization to Fine-Tuning: QLoRA

The benefits of quantization are not limited to final inference; they have completely revolutionized the way local models are customized and fine-tuned. Historically, full fine-tuning of an 8B parameter model required massive data-center clusters with 80GB GPUs [cite: 23]. 

QLoRA (Quantized Low-Rank Adaptation) fundamentally changed this paradigm. QLoRA enables fine-tuning by storing the massive base model in extremely compressed 4-bit precision while training tiny, highly targeted "adapters" in 16-bit precision [cite: 20]. To achieve this without losing training fidelity, QLoRA utilizes 4-bit NormalFloat (NF4), a specialized data type designed specifically to align with the normal distribution of neural network weights [cite: 20, 26, 27]. 

Furthermore, it introduces "Double Quantization," which compresses the quantization constants themselves, cutting the metadata overhead down to a fraction of a bit per parameter [cite: 20]. Using QLoRA, a developer can fine-tune a highly customized, domain-specific version of a Llama 3 8B model using a single consumer gaming GPU with as little as 8GB to 12GB of VRAM [cite: 23]. 

## The Local Software Ecosystem

Once a quantized model is downloaded, a local inference engine is required to run it. By 2026, the software ecosystem has specialized cleanly into distinct tiers based on user technical expertise, hardware availability, and deployment goals [cite: 28]. 

| Software Engine | Primary User Base | Architecture & Hardware Focus | Performance Profile |
| :--- | :--- | :--- | :--- |
| **llama.cpp** | Power users, embedded systems | Pure C/C++ backend. Supports CUDA, Apple Metal, ROCm, Vulkan [cite: 21, 29]. | Highest raw throughput for local edge devices. Minimal overhead [cite: 9, 30]. |
| **Ollama** | App developers, prototypers | Go-based wrapper around llama.cpp. Docker-like CLI experience [cite: 4, 28]. | Introduces 30-50% overhead compared to raw llama.cpp, but excels in ease-of-use [cite: 28, 30]. |
| **LM Studio** | Non-programmers | GUI desktop application. Uses llama.cpp backend [cite: 28, 29]. | Excellent for visually browsing and testing GGUF models on Macs and PCs [cite: 28, 30]. |
| **vLLM / SGLang** | Enterprise production servers | Python-based. Optimized for data-center GPUs (NVIDIA A100/H100) [cite: 28, 31]. | Achieves 16x-20x the concurrent throughput of Ollama via PagedAttention and continuous batching [cite: 28, 31]. |

The core engine powering almost all consumer local AI is `llama.cpp` [cite: 29]. Because it is a compiled C/C++ binary with zero external dependencies, it operates exceptionally close to the "metal" of the hardware [cite: 9, 21]. 

However, many developers opt for Ollama, which wraps the complex `llama.cpp` configuration into a frictionless, one-click installation process [cite: 4]. With a single terminal command, Ollama downloads the model, manages the memory allocation, and exposes a clean local API [cite: 4]. The trade-off is efficiency; community benchmarking repeatedly demonstrates that the convenience layers added by Ollama result in 30% to 70% fewer tokens generated per second compared to running the same model through raw `llama.cpp` [cite: 28, 30]. For multi-user concurrent deployment in a business environment, neither tool is appropriate. Production servers instead rely on frameworks like vLLM, which utilize PagedAttention to eliminate memory fragmentation and handle thousands of simultaneous queries [cite: 28, 31].

## The Cost of Compression: Reasoning and Multilingual Penalties

The pervasive claim that 4-bit quantization results in "near-zero quality loss" requires critical nuance. Degradation depends entirely on the complexity of the target task. If a user is relying on an LLM for basic text summarization, creative writing, or standard dialogue, the difference between FP16 and INT4 is practically indistinguishable [cite: 2, 23, 27]. On standard benchmarks that measure "perplexity" (how predictably the model guesses the next word), the 4-bit loss is mathematically negligible [cite: 20, 32].

However, the illusion of lossless compression shatters when the model is tasked with exact logic, mathematics, and computer programming [cite: 7, 31, 33]. 

### The Reasoning and Coding Cliff
Code generation requires absolute, rigid precision. A single misplaced parenthesis or hallucinated variable name destroys the entire output. In targeted benchmarks against the 32-billion parameter Qwen model, stepping down from FP16 to FP8 resulted in zero measurable accuracy loss on coding tests (scoring an identical 39.02% on the HumanEval benchmark) [cite: 7]. But when the identical model was squeezed down to INT4, its coding score plummeted by 8 full points down to 31.10% [cite: 7]. 

Recent empirical studies focusing specifically on complex reasoning models—such as DeepSeek-R1-Distill—have revealed that aggressive quantization heavily damages the model's Chain-of-Thought (CoT) processing [cite: 22, 33]. In certain architectures, dropping to 4-bit precision causes the model to "overthink" simple problems, generating erratic, elongated output loops as it struggles to compensate for the precision lost in its internal logic pathways [cite: 22, 33]. 

### The Multilingual Penalty
Aggressive quantization also disproportionately harms non-English languages [cite: 14]. LLMs allocate their internal weight capacity proportionally based on their massive training datasets. Because English typically dominates 40% to 60% of most global training corpora, the English linguistic pathways are deeply entrenched and robust enough to survive the brutal rounding of INT4 compression [cite: 14].

Conversely, languages with lower resource representation in the training data rely on fragile, highly sensitive weights. Compressing those specific weights to 4-bit (Q4_K_M) drops non-English performance down to 90%, introducing awkward phrasing and grammatical errors. For enterprise users requiring multilingual deployments or operating heavily in CJK (Chinese, Japanese, Korean) languages, experts recommend dropping no lower than 6-bit quantization (Q6_K) to preserve 99% of the original language fidelity [cite: 14].

### Architectural Discrepancies
Not all base models tolerate quantization equally. Benchmark evaluations indicate that Meta's Llama 3.1 8B suffers steeper performance degradation below 4-bit precision than competing open-weight models [cite: 27, 32]. Conversely, Alibaba's Qwen 2.5 architecture and Google's Gemma 2 demonstrate higher resilience to aggressive 3-bit and 4-bit compression, maintaining stronger mathematical capabilities at lower bit-depths [cite: 32, 34].

## Hardware Matchmaking: RAM, VRAM, and Bandwidth

When evaluating consumer hardware for local AI inference, users must prioritize memory bandwidth over theoretical compute speed [cite: 1]. 

For modern setups, Apple Silicon (the M-Series chips) represents a massive architectural advantage. Apple utilizes a Unified Memory Architecture, meaning the CPU and GPU share the exact same pool of high-bandwidth memory. An M4 Max with 64GB or 128GB of Unified RAM can easily run massive 70B parameter models at 4-bit quantization—a workload that would normally require tens of thousands of dollars worth of dedicated NVIDIA enterprise GPUs in a PC tower [cite: 1, 28]. 

For PC users, consumer NVIDIA GPUs like the RTX 3090 and RTX 4090 remain the gold standard. Their 24GB of dedicated GDDR6X VRAM allows users to comfortably fit a 35B parameter model at INT4, or run an 8B model with an enormous context window for processing entire textbooks simultaneously [cite: 1, 19, 28]. 

Finally, for users on standard business laptops equipped with 16GB of standard system RAM and basic integrated graphics, local AI remains highly accessible. These machines are perfectly situated for the 7-billion to 9-billion parameter class (such as Llama 3.1 8B, Qwen 3.5 9B, or Mistral) [cite: 5, 19, 25]. These models, formatted as Q4_K_M GGUFs, occupy roughly 5GB to 6GB of memory, executing locally on the CPU while leaving enough overhead for the operating system to function smoothly [cite: 1, 5, 23].

## Bottom line
Model quantization is the foundational mathematical compression technique that makes self-hosted, private AI inference possible on standard consumer hardware. By stepping down from high-precision 16-bit parameters to 4-bit integer formats like GGUF, AWQ, or EXL2, users can reduce a Large Language Model's physical memory footprint by up to 75% while simultaneously increasing generation speeds by nearly 40%. While general conversational quality remains virtually untouched by this compression, users attempting complex computer programming, multi-step mathematical reasoning, or precise non-English translation should cautiously utilize 6-bit or 8-bit models, as overly aggressive quantization permanently degrades complex logical pathways.

## Sources
1. [FP16 vs INT8 vs INT4: When to Use Each for LLM Inference](https://gigagpu.com/fp16-vs-int8-vs-int4-llm-inference/)
2. [LLM Quantization Explained: INT4, INT8, FP8, AWQ, and GPTQ in 2026](https://vrlatech.com/llm-quantization-explained-int4-int8-fp8-awq-and-gptq-in-2026/)
3. [LLM Quantization Guide](https://www.hivenet.com/post/llm-quantization-guide)
4. [LLM Quantization Benchmarks](https://aimultiple.com/llm-quantization)
5. [How to Calculate GPU Memory for LLMs](https://www.youtube.com/watch?v=anhLHBi1pP4)
6. [How to Run LLMs Locally: Complete 2025 Guide](https://www.investglass.com/how-to-run-llms-locally-complete-2025-guide-to-self-hosted-ai-models/)
7. [Recommended Hardware for Running LLMs Locally](https://www.geeksforgeeks.org/deep-learning/recommended-hardware-for-running-llms-locally/)
8. [The Complete Guide to Running LLMs Locally](https://www.ikangai.com/the-complete-guide-to-running-llms-locally-hardware-software-and-performance-essentials/)
9. [The Truth About Local LLMs](https://ignesa.com/insights/the-truth-about-local-llms-when-you-actually-need-them/)
10. [How to Run Large Language Models Locally](https://ai.plainenglish.io/how-to-run-large-language-models-llms-locally-a-beginners-guide-to-offline-ai-d993631c87cc)
11. [Quantization for Local LLMs: Formats Explained](https://www.hardware-corner.net/quantization-local-llms-formats/)
12. [Understanding GGUF, AWQ, and EXL2](https://www.reddit.com/r/LocalLLaMA/comments/1ayd4xr/for_those_who_dont_know_what_different_model/)
13. [Exploring AWQ, GPTQ, EXL2, and GGUF](https://blog.gopenai.com/exploring-bits-and-bytes-awq-gptq-exl2-and-gguf-quantization-techniques-with-practical-examples-74d590063d34)
14. [Quantization Methods Compared](https://ai.rs/ai-developer/quantization-methods-compared)
15. [Understanding LLM Weight Quantization](https://medium.com/@abhi-84/understanding-llm-weight-quantization-gptq-awq-and-gguf-make-big-models-fit-in-a-small-space-518bb204cae4)
16. [Llama 3 8B Quantization Benchmarks](https://aitoolland.com/llama-ai-models-comparison-guide/)
17. [Llama 3 8B Speed: GPTQ vs AWQ vs GGUF](https://gigagpu.com/llama-3-8b-gptq-awq-gguf-speed/)
18. [Llama 3 Model Stats and RAM](https://blog.devgenius.io/llama-3-model-stats-4ce2e720161a)
19. [Empirical Study of llama.cpp Quantization](https://arxiv.org/html/2601.14277v1)
20. [Llama 3 Quantization Comparison](https://github.com/matt-c1/llama-3-quant-comparison)
21. [Quantization Explained: A Concise Guide](https://dev.to/jte0711/quantization-explained-a-concise-guide-for-llms-4b19)
22. [LLM File Names and Quantization Explained](https://www.youtube.com/shorts/cDB9SuFCqcQ)
23. [Demystifying Quantizations for LLMs](https://cast.ai/blog/demystifying-quantizations-llms/)
24. [Introduction to Quantization Math](https://www.maartengrootendorst.com/blog/quantization/)
25. [LLM Quantization Methodologies](https://joydeep31415.medium.com/llm-quantization-explained-4c7ebc7ed4ab)
26. [Quantization Hurts Reasoning: An Empirical Study](https://beckmoulton.medium.com/quantization-hurts-reasoning-an-empirical-study-on-optimizing-llms-for-complex-tasks-74be0c92ff14)
27. [Impact of Quantization on LLM Confidence](https://aclanthology.org/2024.findings-naacl.124.pdf)
28. [Benchmarking Compressed Large Reasoning Models](https://arxiv.org/html/2504.04823v1)
29. [Do Reasoning LLMs Suffer More from Quantization?](https://www.reddit.com/r/LocalLLaMA/comments/1ilyu1c/do_reasoning_llms_suffer_more_from_quantization/)
30. [Systematic Study on Quantized Reasoning Models](https://huggingface.co/papers/2504.04823)
31. [Rubra: Open-Weight Tool-Calling LLMs](https://www.reddit.com/r/LocalLLaMA/comments/1dtt32y/new_collection_of_llama_mistral_phi_qwen_and/)
32. [Qwen 3.5 vs Llama vs Mistral](https://www.aimagicx.com/blog/qwen-3-5-vs-llama-vs-mistral-china-open-source-ai-2026)
33. [Qwen Quantization with llama.cpp](https://qwen.readthedocs.io/en/latest/quantization/llama.cpp.html)
34. [The Best Open-Source LLMs for Business in 2025](https://localllm.co.za/the-best-open-source-llms-for-business-in-2025-llama-3-mistral-qwen-gemma-4-which-should-you-deploy/)
35. [Open Source vs Open Weights Debate](https://www.reddit.com/r/LocalLLaMA/comments/1sqh6kk/stop_letting_vc_bros_gaslight_us_qwen_and_llama/)
36. [Running Qwen Locally with llama.cpp](https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html)
37. [llama.cpp Contribution Guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md)
38. [Connecting Claude Code to llama.cpp and Qwen 3.6](https://www.youtube.com/watch?v=pY1i1R0rchk)
39. [Open Source AI Models Overview](https://gotechlaunch.com/open-source-ai-models/)
40. [Qwen Team Contributions to llama.cpp](https://www.reddit.com/r/LocalLLaMA/comments/1oda8mk/qwen_team_is_helping_llamacpp_again/)
41. [Ollama vs LM Studio vs vLLM vs llama.cpp vs MLX](https://codersera.com/blog/ollama-vs-lm-studio-vs-vllm-vs-llama-cpp-vs-mlx-2026/)
42. [llama.cpp vs Ollama Comparison](https://www.openxcell.com/blog/llama-cpp-vs-ollama/)
43. [Why You Should Completely Avoid Ollama in 2026](https://blog.gopenai.com/why-you-should-completely-avoid-ollama-in-2026-6135d9e8591e)
44. [Ditching LM Studio for llama.cpp](https://www.xda-developers.com/ditched-lm-studio-for-llama-cpp-and-local-llm-doesnt-feel-like-downgrade-anymore/)
45. [Why Use llama.cpp over Ollama](https://itsfoss.com/llama-cpp/)
46. [Quantized LLMs Cost and Performance Results](https://latitude.so/blog/quantized-llms-cost-performance-results)
47. [Self-Hosting Open Weight LLMs Decision Guide](https://www.digitalapplied.com/blog/self-hosting-open-weight-llms-2026-deployment-decision-guide)
48. [LLM Quantization Evaluation: Leave No Bits Behind](https://federico-ricciuti.medium.com/llm-quantization-evaluation-leave-not-bits-behind-0777e23b9f91)
49. [Evaluations of Quantized LLMs: Accuracy Recovery](https://developers.redhat.com/articles/2024/10/17/we-ran-over-half-million-evaluations-quantized-llms)
50. [Comparing Quantized Performance in Llama Models](https://www.greaterwrong.com/posts/qmPXQbyYA66DuJbht/comparing-quantized-performance-in-llama-models)
51. [Step-by-Step Math of LLM Quantization](https://www.maartengrootendorst.com/blog/quantization/)

**Sources:**
1. [ikangai.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEYjVXADoU-jQ-Yk-YuDuUomKPK2ATR-WryhRAEtbMSImIbo5pgm4FGJe7J7qeyPcA0Sz5WASm0DYfHtT1225L1flK5F9l5c9LktF4yCTphY12MgQUMfyghRBrl_2_oywxTjVqp7TWPIIxIlr7w7caRoxwFiebquIRMY4_v2J3yf-mCmyj0dSKuO1hDWz1DLeRWBDsWRiRK33K7_dsy9iL8xCUVDMAE)
2. [vrlatech.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGHX_JxtAKBWefYKQOpj7d47EO5K9munrj4z4rA59qC3iKICk-PEGZ1_3ekq3U7zJOSSIc4F0Qr4DWvqvxiHYFQvUKJOnnGGQF4_I1sAFFLPi5bqmZb7ocotOpq2IMbP-esDmK8uwszlHG3sEyqBnF1Ef_8hpH3lBn7t7NGIDVKaxVTPyDop8RclQ==)
3. [hivenet.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEkcN4daiFdAs_AxTs1fSlUCdn_Icr4DdHwOB7Cc_MK62_GkhIzvx3pZ2W_g1U90sxJAbu6ATKnJZl17SXidyWMQJ-vSfzfTt_IBlEEh1SocXVMVK6ldk1uZsyl9eeetGFIXfeK7toWfKA=)
4. [openxcell.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF13qJcfBXdnRrYDyh82AYWGVCesmFW7ZogQbNSKLPokhCsCM6cT7-InA7PqrEXgo3LV9yWoFnnECgK1rstpbUgt4UNmKkADvMzKNrwp7RVKTkG3hVVZLTdrbnGtmxUMfEkyLuFrVf9WDI=)
5. [investglass.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEBge2Hb2Cp7AZhWtUJOrDWsdKFSFC_2hjVpxIxmhGp-dkKSaHSzeE2G_mwGsCh95124ruTNlNjduS_XIgyhnjg9c5S0w0m_bERK9rrbIm6B2niuyiYyWfXxJgZkyE4kKKRn1XN9aqDuN8P4JNOPU52SKzs7gOM3qQOedPbn52cOS-3cd9SsVBxgnBL7GFw-J7OkMbzZ9g6)
6. [ignesa.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHilC0VTgXnCNFqZ4ZMMA4Gziupe0V0ZLEtkhYHFRQtIPAIGOvYsVK0jIh0YYul0Hlu_4l5u9bdDseBBiZzYyzXJq8w_BnZWkeNV_I8MkonB4stSqdD8eu-dd2h7_jxgoOhx55t-D-I0wHCKExaeuZeJlBDqvgjBr--up33i5rG3d3H8uHIaGaUxg==)
7. [aimultiple.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGs4TrppRzajWRe0zqs41AerLp6_eB5coLnM7Yaed_ekn4IjzpTUuUxMrxcjr-h0XaQz4v2NSdLK9WMdZICVQMp1rgA9ywgszabLVzCF-ED1DwS1uxFOUdf8KCrLCM=)
8. [plainenglish.io](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQE2DzhQQxA3ykh9QR1Xn7cJ7K99pAlXTT4pHRJe1bNe8_XD-AS8pr534-vUSJ2QPLD396D0JsPaPC565i_Q4sIrP_DpwBX7CQcq785QevGW7SfEuqtux8Tpl7LiykOaz53pzkBM7BzdBarRQLdJudNi_vsoEtLKi3iA2FTw894xOK7AkBnytvOVNRmqKpeiuwIPbQ2VJmRG1e6WMo3CTqL2hvXUrQ1lcv769ZQ=)
9. [itsfoss.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHTwYVXtZCzqOg5W4DwbGSPNkMYAOGRQGvsJDlZMNuWlQly7SG42_BeVGHDywZaW_oytqhOgZBrxvaWlvCIaW0yBg5YASygQToLQvnrcSDmNa8meAU=)
10. [youtube.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGkBYRGH4gxat_lY5aFFIRC7-Iy06gqLFsteN9wa_jCzvTSn1UPS9ZRjqZCa__mrTbc-z9Kl2bXvBmO1NX_EkTrcL3F6aSmIo2wYuwchZ-TzEzigs_NWaIu1Q58kd6n_vc=)
11. [dev.to](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFG4h5EQiveUvprflet96hwkpOh6evk3tYj9upUPw2TW0DJXfO7EGdrnbejKxg9f8Xbyj7p97d4M_xdWRzkFK8TpOUBNZ1rEWQVNUXoCDTfmuAQskvy71wW1X7AeTnnhIF_ctinLJEfNnvBn-C5zSSs5cCpQ8mRYFfxzM4zP6OFKcU=)
12. [cast.ai](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGjvaKFgB2Gcr3bI5txmbFRM02utTAsA--E3iPSoI7aaz-ctcGg8W4WD7HNv5sbMKUEhqB0uDhFJdMsWtDcX5llF7lg3iEWy2XXOo9Y8v7NW-KELU7PmXktxUc3_d3kNA5XUrO-aXG2uEsKrw==)
13. [hardware-corner.net](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFQot2cxCvEbihySva-WR7dmRp9uXtOXeKKuAk8LmmTUmoQvogJ_F6mdADXI6nSCijHPx0U0ZIh88Z_882yK9nqt9Lcl7ZmWeqKsnllsjjyKNsHq1RGgYGqUPQ37fgjGHOYIsbesIA9HXDRei6tHSS_j0MPBEXH)
14. [ai.rs](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFkQBhCYlwNT35FY0hBeKxZ_gFJjAULOZ2dnwOVZRARpdOpa21JPmIsc6ytuYb7eT1wmrMOajnt1nC7Zk5dSPQerE594M_kEldMRZhenYMUtQ7CXItvvPfopV6b_R39NQbJJ89xMC16kYMGtTigaA==)
15. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG4y3j-2Aj7kIlqBM9OGCboytgRePYNYfVvu5tBuSwgvgSg_OohGEqwh9AnPIdYuMJv9-EhkSo4H8mqD02lk0WoqIxnuvTB8HZ0A9X_8pWbhh0ilfuGeiF4odJrQjM0R-XVhpNu8LWeiI4WWCxx7wxat1KHK6Bnq8HgjGzx__6hb_eIoL1eq5j3HEGQlk3qXbNiHsDtMPHE0Bin5Hfr3q3fNQFqO0eA3fmtnoE3yabPOjEROE3XQCK8rRNEnA==)
16. [maartengrootendorst.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEjZLBmTwnv_g7fhDRLjQ3z3ssp0PZkYoMJzq4kJJaGr4jHyXjbNxEEL16l8MouF-4pzEk6Gk2YJeHy2p7UpWQm2noe-SfWIaXKj-jsVQdyQrCQFMVWfSV_KzOzaB79pL4hqusmADT6XXR3CRI=)
17. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHB0Dumxd5sRCw5OnHPW47mLBahn2vtx59Ot1bxSnC_LfPfXEky1dSo-0ixloKoa-l3tmEawZBvD62_-6gPMgs00jAc7hO5hkvQbFmKIq5pNP0ueI-4HRlTK5Tso32iWC8vREGI-AGhb50-DLz0-I2uMLvNlIW6NE9IEGwjyQ==)
18. [youtube.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF6ZvTErveJDPZYRZKG8qsZjnPMTKYd1FFDsllmP2zcUHa1ozhmTtRKmQkbovxqG5Cl_K_8qvWd6eufUstG6Aiz56oBY18SflSjF1zCSIOtSU_3DV90LIJIDiXm6brw_kUX)
19. [gigagpu.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFne2IAJlkzGiSfCGmFndabZrDNbPqGmJVyLO5z72DkVR_rxIUp8cal8Z_GJBfQ8VlvEjw0xtjQm1wGgNJ4SN65B35MsTpd1FWrUd5q4zgCEOTAec5R24SFRGleAB6N2S2ul8jK-SyhE6xJ14mD)
20. [latitude.so](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQG_qcJTQQee6F1VCICiTKypuSnDHs6-bYHHsZ-_8GqdGpySPKvnqEHtAzv5bauBDzPKCPBU4hNGZz2Hk6BkqoJEBG0-cPDiY6r0hhzjmRHSOuNDu6A-Dk7SnbhW6Pz_HQTlCKcw3o0B5gEEjfnVqkl90BwSOQ0c)
21. [readthedocs.io](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGZ-LCQS_l9Kd4zeJvv_l0_mG5Bc3ONrQvhaqi5f6s6aryRCI9MmUjn7DWvgKqgmpzlf8zmWSFpPDH4flk9-_3IVIMsdruYraSfLG3Bxf0XphMBZn3ch8BQq-009ePKuMEX0glA9HIHBOWYj_myVhqin5gGhQ1Z)
22. [arxiv.org](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEktnzKvhEvZU1i9GLADH_FpfrVRRYGgXGfNVKeXuFj-0JVZiaxK4dI_gHdy2zNKHuo7kTMVkUFDONV5FpfNosNOPZ0JJsZCQ9mnBr6tpAIwSfO-CcM8GublQ==)
23. [aitoolland.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGA4DQB1rW3hC46S0G5NPButWpk1L-8ELOZyXTKTMUugDZU-DXT-3-1-zWZ5MQwGU5rnYdCVFfHHHraA-ut1Xgs9yP-FtH3rKrjpSZM4JtGXpsp5Kx-0MeFX8Ngu_-_0PldL1Br1WkQb5mAaRki9w==)
24. [reddit.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGhKeOgJ5kIGzs-KZcu_Wmn7qQ1A4hDZaK2_arabZ_TbWZAKed6pWwQjEwyJcpI1eHLn7Q9K0hF30ubj15A_itXdcazziQH71o8tXivsvhnWiD3kgcGplNyP_ss8UoYk9zJtwwWLHFk3Or6_kd_xNi4LimfL0jM0aLAgfUazZho4yZZU1gtxM63DZoQWVmSNjoIp71ZWD3v3w==)
25. [gigagpu.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQH2_ETbCIfCGjWUE_lfkSL6p-sfNum8wFjm5l8o8Njd05z6PGWNCpIXOnFNZTzIodLC1rqYaTYNxxala--xNdPFHxeKEQEN-2FR8tylZ9xYuem4DEBTDx3B5xp31G04BAJL6rBKzZxk8M8=)
26. [gopenai.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGYRKipwbqhWDDAPEtBbBDobmcn0LJZFR767NzXBgOHllIxcwPoEPGO3WEFCAwYzhQAYpthRDmqcQ1-wDI9p46_BI6shbEGENWOqfmZv7uJEcl_Taf4dByBqUNQ7c1TmOKfJg8AeGuU_KUZTBBTvDOcMMbkWZ2EGV7WhxIsFsJMFFYnZNcyY3qIDxEOPZpABk6MSZ_zSo9qVWgYvr6mXeMoABwzbxK6jXMnFKUxE_xWZykD1Q3B3HHjnrW9Wg==)
27. [greaterwrong.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFuTjCzxMx9uTuThvxzrJjQ-aHcTSf6uMv8a4YrzTe4_-7Zfcg1ydp9h6w2ntU5lkTtofaw-EDhP1r-MABwnlSsAkuthlexSBr9d-IsBvwkbHwCn5lzqveiTK6zfiQqYmoUPCOE3eOEL1kD_f2p7Ojgl_r3-Xy6rfZm7qbMsEMddO6snJI8Zq8laN6dJUX-qH6_Hir_rVsotIol)
28. [codersera.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEoP9q6b4jAor_yRHBIODSDMVQFBYGlgJT2gOZ8vUTWJx3TKKtiI6z3Tcmgf4fFrBxDLTZCWNV7RJSggApt84pncpqWbjufLsccjX1pVKlaE-ivA8Isrh8_XEM3DhsCNyYvVJf_nGKkWrvBaQkzYHKl-3NP1RtZlFDGI5cxPmmvd0-34eQAUw==)
29. [xda-developers.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEP9AHShOBTvnmYFY3bVjXJ0J8hhmIUs75Lx9aHETdGdTLaaJGocm4d6Cxdhvu1lverwIsrHcn8fTUbc4gkmcLoe0qBH4A5y0FVMcbXl6d4kQJ6-JppxGV4nBpTLxWE4V5wiIgHhR2vaaTtV39nI44E5qvXP1ebyDOl6hNdLCtWtRVS33XYXckylKJ8pfJSuFYhuMNJ96hJ2gr7gyvgSLlyry9nDkjb)
30. [gopenai.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHZ1gsssQIqMtSUu4oi7n0mTNHQuyCZIb5Xzvn9r-9AfVM85HlXDSv-nuM6Er17SuL-n7j58SBhP-yBu5KV3FNYfKWCMfQD73He-Cih5Fko-zJ9Ts18W21iD0uisESvTmZfpyRqv02BsLI88KOEzNum6mTBAVbM_OyzAjyMO4l69QaUWvK7OybJe14=)
31. [digitalapplied.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHPEMKlfknUCngboQsjPFfS0B1Fw4fQJRr0FKYo9-A7c6e9gpgAo0F40GEPqa-dTO-AzPD3EPp7c34yEiGXxKJXRmh_0kKZIylV4HmbXFsW0Px9rzyc9OpM_g8e0i0i0JGsUDpeL9HtCXPvaVnz7rK31kLDa4d1lZFOgimpUFL5r6OYd0kiX1v5ahUBuVS55IihQyOwVhs=)
32. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHAF67-wat7a1iEpV6BXfevsMRtZmNC46mX6Lp5yDZZNBd-VdShluxmCLOf6sTcXi-Ys4b2apyDeF0wfIh5wHgQzpABtboJJmDI2kVEoE__7eyj5v8lPFqqxEwxujO1NWEiWqPL4qkVzoPHwYH7wips6ELARbMr4jNJ8pNPz4u3oIy4fG4BsM3sRm0Lpf4qYm5AgDeeQrVeK4c=)
33. [medium.com](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHkK1RiCLxpMDnhKH0e8YjrWaAQYNU7iFh1NPeSy1ZCpkPncEDf0wxRUlbUjvODUYWrjl1lDqAMbd3mFWyV3ZcVSRIRgOp48zswO9_HC38KkM2Sw2R0FQPm7nScyqp5o3OBg_vU8poZbNpfTjexClXTe7CWSa3nKjsXea4Dvlje-ZNxqToPKflDE5Vh3DkauwKIYGAPaBBwxihbOhhwG0umXTEWEsudDvf-TKYNjeC844__n95K8u8=)
34. [localllm.co.za](https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFGH2KLhEDIC2vqnh5ElNw8d9WmNdMI6ib4TkGc-ksqtgLUGm8bXuRrTtwQsgYF3Y9aVNRQ2EZl3ToOQjhO7y1yKrrqt4wGuWAmdyz1gEt2dnqccWDtJHXZPHyDlSMkg9shBBPANu7gdzoGxpW_DU_DILSWAbcY0Czf5bsXL5b3f6uRHC8UyQiv9WJ_9GjMrH6zkiPY4_-HbMCr4KTOCX38gg4vCJVZd2PfSjD9icJJDSA=)