---
name: llm-inference-and-serving
description: "Local and production LLM inference: llama.cpp GGUF, vLLM high-throughput serving, and HuggingFace Hub model discovery/quantization."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux, macos, windows]
metadata:
  hermes:
    tags: [LLM, Inference, Serving, GGUF, vLLM, PagedAttention, Quantization, GPU, CPU, HuggingFace]
    related_skills: [llm-evaluation-and-tracking]
---

# LLM Inference and Serving

Class-level skill for running LLMs locally (CPU/GPU) and serving them in production. Covers model discovery via HuggingFace Hub, local inference with llama.cpp (GGUF), and high-throughput production serving with vLLM.

> **Decision tree:** CPU-only or constrained memory → llama.cpp GGUF. Production API serving → vLLM. Model discovery/quantization → HuggingFace Hub.

---

## 1. HuggingFace Hub — Model Discovery

The canonical source for open-weight models. Used by both llama.cpp and vLLM.

### Search and Download

```bash
pip install huggingface-hub

# Search
huggingface-cli search "Llama-3" --limit 20

# Download
cd ~/models && huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct --local-dir ./llama-3-8b

# With auth (for gated models)
huggingface-cli login  # paste token from https://hf.co/settings/tokens
huggingface-cli download meta-llama/Meta-Llama-3-70B-Instruct --local-dir ./llama-3-70b

# List local models
huggingface-cli scan-cache
```

### Python API

```python
from huggingface_hub import hf_hub_download, list_models, snapshot_download

# Download single file
hf_hub_download(repo_id="meta-llama/Meta-Llama-3-8B", filename="config.json")

# Download entire repo
snapshot_download(repo_id="microsoft/Phi-3-mini-4k-instruct", local_dir="./phi-3")

# Filter by task
models = list_models(filter="text-generation", sort="downloads", limit=10)
for m in models:
    print(f"{m.id}: {m.downloads} downloads")
```

### Upload Your Own Model

```bash
huggingface-cli upload my-username/my-model ./model-files/ .
```

### GGUF Conversion (for llama.cpp)

Convert a HuggingFace model to GGUF quantized format for local CPU inference:

```bash
# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && make

# Convert to FP16
python convert_hf_to_gguf.py ~/models/llama-3-8b --outfile ~/models/llama-3-8b-fp16.gguf

# Quantize (Q4_K_M recommended for quality/size balance)
./llama-quantize ~/models/llama-3-8b-fp16.gguf ~/models/llama-3-8b-q4_k_m.gguf Q4_K_M

# Quantization levels
# Q4_K_M — balanced (4-bit, ~4GB for 7B params, good quality)
# Q5_K_M — higher quality (5-bit, ~5GB for 7B params)
# Q6_K — near-lossless (6-bit, ~6GB for 7B params)
# Q8_0 — best quality 8-bit (8GB for 7B params)
```

---

## 2. llama.cpp — Local Inference

Run quantized LLMs on CPU or GPU with minimal dependencies.

### Quick Start

```bash
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp
make -j$(nproc)  # CPU build
# or: cmake -B build && cmake --build build --config Release

# Basic inference
./llama-cli -m ~/models/llama-3-8b-q4_k_m.gguf -p "Explain quantum computing:" -n 256 --temp 0.7

# Interactive chat mode
./llama-cli -m ~/models/llama-3-8b-q4_k_m.gguf --chat-template llama3 -cnv

# Server mode (OpenAI-compatible API)
./llama-server -m ~/models/llama-3-8b-q4_k_m.gguf -c 4096 --host 0.0.0.0 --port 8080
```

### llama-server API

```bash
# Start server
./llama-server -m model.gguf -c 4096 --port 8080

# Chat completions
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "local", "messages": [{"role": "user", "content": "Hello"}], "temperature": 0.7}'

# Embeddings
curl http://localhost:8080/v1/embeddings \
  -d '{"input": "text to embed", "model": "local"}'
```

### Performance Tuning

| Parameter | Effect | Typical |
|-----------|--------|---------|
| `-ngl N` / `--gpu-layers N` | Offload N layers to GPU | `-ngl 999` (all layers) |
| `-c N` / `--ctx-size N` | Context window size | 4096 or 8192 |
| `-b N` / `--batch-size N` | Prompt processing batch | 512 or 2048 |
| `-t N` / `--threads N` | CPU threads | Match physical cores |
| `-fa` / `--flash-attn` | Flash Attention (speed + memory) | Enable if compiled |

```bash
# GPU offload example
./llama-server -m model.gguf -ngl 999 -c 8192 --port 8080
```

### GPU Build (CUDA)

```bash
cmake -B build -DLLAMA_CUDA=ON
cmake --build build --config Release -j$(nproc)
```

### Common Pitfalls

- Wrong quant format: `llama.cpp` only runs GGUF (not GPTQ/AWQ/Safetensors). Convert first.
- Insufficient context: `-c` must fit the input + output. Overshoot to be safe.
- Wrong chat template: use `--chat-template` matching the model (llama3, mistral, chatml).
- KV cache grows with sequence length; long contexts need more VRAM.
- First run builds metal/CUDA kernels — may take minutes, don't interrupt.

---

## 3. vLLM — Production Serving

High-throughput LLM serving with PagedAttention, continuous batching, and OpenAI-compatible API.

### Quick Start

```bash
pip install vllm

# Offline inference
from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-3-8B-Instruct")
sampling = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(["Explain quantum computing"], sampling)
```

### Production Server

```bash
# Basic server
vllm serve meta-llama/Llama-3-8B-Instruct

# Production deployment
vllm serve meta-llama/Llama-3-8B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85 \
  --quantization awq
```

### Client (OpenAI SDK compatible)

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="meta-llama/Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
    temperature=0.7,
    max_tokens=256,
)
```

### Optimization Flags

| Flag | Purpose |
|------|---------|
| `--tensor-parallel-size N` | Split across N GPUs |
| `--pipeline-parallel-size N` | Layer-wise pipeline across nodes |
| `--quantization awq/gptq/fp8` | Reduce memory, increase throughput |
| `--max-num-seqs 256` | Max concurrent sequences |
| `--gpu-memory-utilization 0.85` | Leave headroom for CUDA graphs |
| `--swap-space 8` | CPU swap for KV cache overflow |
| `--enforce-eager` | Disable CUDA graphs (for debugging) |

### Deployment Checklist

1. GPU drivers + CUDA toolkit installed
2. Model weights downloaded (`huggingface-cli download`)
3. VRAM sufficient: roughly 2× model size (weights + KV cache)
4. Firewall: port 8000 open
5. Health check: `curl http://localhost:8000/health`
6. Load balancer: nginx/traefik in front for multi-node
7. Monitoring: Prometheus metrics at `/metrics`

---

## Cross-Tool Comparison

| Dimension | llama.cpp | vLLM |
|-----------|-----------|------|
| Primary use | Local/edge inference | Production serving |
| Hardware | CPU, Apple Silicon, consumer GPU | NVIDIA datacenter GPU |
| Model format | GGUF (quantized) | Safetensors, AWQ, GPTQ, FP8 |
| Quantization | GGUF Q4–Q8 | AWQ, GPTQ, FP8, marlin |
| Throughput | Moderate | Up to 24× vs baseline |
| Latency | Higher per token | Lower per token |
| API | Custom / OpenAI-compatible | OpenAI-compatible |
| Context length | Up to 128K (varies by quant) | Up to 128K+ |
| Multi-GPU | Basic | Tensor + pipeline parallelism |
| Best for | Laptops, edge, privacy-first | Datacenter, high-QPS APIs |

---

## Workflow: Deploy a Model End-to-End

1. **Discover:** `huggingface-cli search` → find model
2. **Download:** `huggingface-cli download` → local weights
3. **Convert (if llama.cpp):** `convert_hf_to_gguf.py` + `llama-quantize`
4. **Serve locally:** `llama-server` (dev) or `vllm serve` (prod)
5. **Test:** `curl` the OpenAI-compatible endpoint
6. **Scale:** Add tensor parallelism, load balancer, monitoring

---

## Resources

- For detailed GGUF conversion tables, per-quant benchmarks, and GPU build guides: load archived `llama-cpp` references.
- For vLLM optimization recipes, server deployment configs, and quantization comparisons: load archived `serving-llms-vllm` references.
- For HuggingFace Hub advanced upload patterns, model cards, and repo management: load archived `huggingface-hub` references.