LLM Model Quantization

Use when the user asks to quantize a model, GGUF, Q4_K_M, llama-quantize, GPTQ, AWQ, NF4, bitsandbytes, INT8/FP8 weights or a model too big for the GPU. Produces format choice per engine, quantization commands, memory estimate and an eval gate. Not for NVFP4/FP8 kernel-level tuning (use extreme-quantization-mastery) or QLoRA training (use llm-finetuning).

45 stars 6 forksAI, LLMs & Agents

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add j4flmao/agent-skills --skill llm-model-quantization
Manual install
git clone --depth 1 https://github.com/j4flmao/agent-skills.git /tmp/agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/agent-skills/skills/ai/llm-engineering/model-quantization ~/.claude/skills/llm-model-quantization

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Model Quantization

## Purpose
Quantization stores weights (and optionally activations) in fewer bits so a model fits in less memory and decodes faster, since decoding is usually memory-bandwidth bound. The cost is a quality loss that depends on bit width, method, calibration data, and model. Pick the format your inference engine runs fastest, quantize with a calibration-aware method, and gate the result on your own evals.

## Agent Protocol

### Trigger
"quantize", "GPTQ", "AWQ", "GGUF", "k-quants", "Q4_K_M", "INT4", "INT8", "FP8", "NF4", "bitsandbytes", "fit model in VRAM".

### Input Context
- Model and size; target hardware (NVIDIA generation, Apple Silicon, CPU).
- Inference engine (vLLM, TensorRT-LLM, llama.cpp/Ollama, transformers).
- Quality tolerance and the eval set that measures it.

### Output Artifact
Recommended format and bit width, quantization commands, memory estimate, eval comparison plan.

### Response Format
```
Target: {hardware} via {engine}
Format: {fp8|awq-w4a16|gptq-w4a16|gguf Q4_K_M|nf4}
Memory: weights ≈ params × bits/8 (+ scales) = {GB}
Steps: {commands}
Eval gate: {metric} drop ≤ {x}% vs bf16 baseline
```

### Completion Criteria
- [ ] Format supported natively by the target engine/hardware.
- [ ] Calibration data resembles production inputs.
- [ ] Baseline vs quantized evaluated on task metrics, not only perplexity.
- [ ] Throughput/latency measured, not assumed.

## How It Works

Affine (asymmetric) quantization of a group of weights:
Read the full skill on GitHub

Details

Skill name
llm-model-quantization
Path
skills/ai/llm-engineering/model-quantization/SKILL.md
License
MIT
Repo updated
17 November 2026

FAQ

What does the LLM Model Quantization skill do?

Use when the user asks to quantize a model, GGUF, Q4_K_M, llama-quantize, GPTQ, AWQ, NF4, bitsandbytes, INT8/FP8 weights or a model too big for the GPU. Produces format choice per engine, quantization commands, memory estimate and an eval gate. Not for NVFP4/FP8 kernel-level tuning (use extreme-quantization-mastery) or QLoRA training (use llm-finetuning).

How do I install the LLM Model Quantization skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add j4flmao/agent-skills --skill llm-model-quantization. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is LLM Model Quantization popular?

It lives in j4flmao/agent-skills, which has 45 GitHub stars and 6 forks.

Related skills

Agent Observability

45

j4flmao/agent-skills

Use when the user asks about agent observability, tracing agent reasoning, debugging an agent loop, OpenTelemetry GenAI spans for LLM calls, LLM cost tracking per agent, agent audit logs or hallucination/eval monitoring in production. Produces instrumentation config, trace schema, dashboards, alerts and cost attribution. Not for generic service tracing without LLMs (use devops-opentelemetry).

DevOps & Cloud

AI Agent Architectures

45

j4flmao/agent-skills

Use when the user asks to build an AI agent, design an agent that calls tools and keeps memory, write an agent system prompt or pick an agent framework (Claude Agent SDK, OpenAI Agents SDK, LangGraph). Produces an architecture brief covering loop, tools, memory, guardrails and evals, routing depth to child skills. Not for workflow patterns (use ai-agentic-workflows).

AI, LLMs & Agents

AI Agent Core Loop

45

j4flmao/agent-skills

Use when the user asks about the agent loop, ReAct vs plan-and-solve, re-planning, Reflexion, an agent stuck in a loop, max iterations or termination conditions. Produces a loop spec or patch with state, success predicate, caps and no-progress detection. Not for overall agent design (use ai-agent-architectures) or tool schemas (use tool-calling-principles).

AI, LLMs & Agents

LLM RAG Retrieval Internals

45

j4flmao/agent-skills

Use when the user asks to build a RAG pipeline over documentation, tune HNSW (M, efSearch) or IVF-PQ (nprobe), hybrid dense + BM25 search, reciprocal rank fusion, cross-encoder reranking, Self-RAG or DSPy. Produces index choice and parameters, a hybrid+rerank pipeline and recall evals. Not for query rewriting and RAG prompts (use prompt-rag-query-design).

AI, LLMs & Agents

LLM Kv Cache Paged Attention

45

j4flmao/agent-skills

Use when the user asks about KV cache size, PagedAttention, vLLM OOM, gpu_memory_utilization, max_num_seqs, prefix caching, GQA/MLA memory or long-context OOM. Produces KV-per-token math, max concurrency and vLLM flags. Not for weight quantization (use llm-model-quantization) or local llama.cpp setups (use local-slm-inference).

AI, LLMs & Agents

Multi Agent Topology

45

j4flmao/agent-skills

Use when the user asks whether to use multiple agents, supervisor-worker, agent hierarchy, pipeline of agents, handoffs or a critic/debate setup. Produces a topology diagram, role table with least-privilege tools, typed edge contracts and round caps. Not for framework runtime and checkpointing (use ai-multi-agent-systems) or R&D swarms (use swarm-architectures).

AI, LLMs & Agents

Browse the most popular Claude skills, all ai, llms & agents skills, or write your own with the skills generator.