---
name: mlops-workbench
description: "Machine learning operations: LLM benchmarking, experiment tracking (W&B), audio generation (AudioCraft), image segmentation (SAM), and HuggingFace Hub model management."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux, macos, windows]
metadata:
  hermes:
    tags: [MLOps, LLM, Benchmarking, Evaluation, Audio, Segmentation, Weights-and-Biases, HuggingFace, Experiment-Tracking]
    related_skills: [llm-inference-and-serving]
---

# MLOps Workbench

Machine learning operations covering the full lifecycle of model evaluation, experiment tracking, and multimodal generation. Load this umbrella when the user touches: benchmarking, evaluation, W&B, audio generation, image segmentation, or HuggingFace model management.

> **Companion skill:** For inference and serving, load `llm-inference-and-serving` (covers llama.cpp, vLLM, and HF Hub downloading).

---

## 1. LLM Benchmarking with lm-evaluation-harness

Industry-standard evaluation across 60+ academic benchmarks (MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag).

### Quick Start

```bash
pip install lm-eval

# Evaluate on core benchmarks
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu,gsm8k,hellaswag \
  --device cuda:0 --batch_size 8

# List all available tasks
lm_eval --tasks list

# Evaluate via API (e.g., vLLM backend)
lm_eval --model local-completions \
  --model_args model=local,base_url=http://localhost:8000/v1 \
  --tasks mmlu --batch_size 1
```

### Core Benchmarks Reference

| Benchmark | What it tests | Good score |
|-----------|--------------|------------|
| MMLU | Multi-subject knowledge (STEM, humanities, social sciences) | >65% for 7B, >80% for 70B |
| GSM8K | Grade-school math word problems | >70% for 7B, >90% for 70B |
| HumanEval | Python code generation | >40% pass@1 for 7B |
| TruthfulQA | Truthfulness vs. common misconceptions | >60% MC1 for 7B |
| HellaSwag | Commonsense natural language inference | >80% for 7B |
| BBH | Challenging big-bench tasks | >60% for 70B |
| DROP | Discrete reasoning over paragraphs | >50% EM for 70B |

### Custom Tasks

Place a custom task YAML in `lm_eval/tasks/my_task/`:

```yaml
dataset_path: my_dataset
task: my_custom_eval
doc_to_text: "{{question}}"
doc_to_target: "{{answer}}"
metric_list:
  - metric: exact_match
    aggregation: mean
```

Run: `lm_eval --tasks my_custom_eval --model_args ...`

### Distributed Evaluation

```bash
lm_eval --model hf \
  --model_args pretrained=microsoft/Phi-3-medium,parallelize=True \
  --tasks mmlu \
  --batch_size auto:4
```

### Key Flags

| Flag | Purpose |
|------|---------|
| `--model hf` | HuggingFace Transformers backend |
| `--model vllm` | vLLM backend (faster, GPU-only) |
| `--model local-completions` | Any OpenAI-compatible API |
| `--batch_size auto` | Auto-detect from VRAM |
| `--num_fewshot N` | N-shot prompting |
| `--write_out` | Save per-document predictions |
| `--output_path DIR` | Results directory |
| `--log_samples` | Log all generated text |
| `--use_cache` | Cache dataset processing |

---

## 2. Experiment Tracking with Weights & Biases (W&B)

Track metrics, hyperparameters, artifacts, and model versions across training runs.

### Quick Start

```bash
pip install wandb
wandb login  # paste API key from https://wandb.ai/authorize

# Initialize a run
import wandb
wandb.init(project="my-project", config={"lr": 0.001, "epochs": 10})

# Log metrics
wandb.log({"loss": 0.5, "accuracy": 0.92})

# Log artifacts
artifact = wandb.Artifact("dataset", type="dataset")
artifact.add_file("train.csv")
wandb.log_artifact(artifact)

# Finish
wandb.finish()
```

### Sweeps — Hyperparameter Search

```yaml
# sweep.yaml
method: bayes
metric:
  name: val_loss
  goal: minimize
parameters:
  lr:
    distribution: log_uniform
    min: 0.0001
    max: 0.1
  batch_size:
    values: [16, 32, 64]
```

```bash
wandb sweep sweep.yaml
wandb agent PROJECT/SWEEP_ID  # launch sweep workers
```

### Model Registry

```python
# Register a model version
artifact = wandb.Artifact("model", type="model")
artifact.add_file("checkpoint.pt")
wandb.log_artifact(artifact, aliases=["best", "v1.2"])

# Promote to production
artifact = wandb.use_artifact("my-project/model:v1.2")
artifact.link(target_path="model-registry/production")
```

### Integrations

```python
# PyTorch Lightning
from pytorch_lightning.loggers import WandbLogger
trainer = Trainer(logger=WandbLogger(project="my-project"))

# HuggingFace Transformers
from transformers import Trainer
from wandb.integration.huggingface import WandbCallback
trainer = Trainer(..., callbacks=[WandbCallback()])

# Keras/TensorFlow
model.fit(X, y, callbacks=[wandb.keras.WandbCallback()])
```

### Team Collaboration

- **Runs Table:** Sort/filter/group all runs
- **Reports:** Share interactive dashboards
- **Alerts:** Slack/Email notifications on metric thresholds
- **Lineage:** Full artifact traceability

---

## 3. Audio Generation with AudioCraft

Meta's AudioCraft suite for text-to-music (MusicGen), text-to-sound (AudioGen), and neural audio codec (EnCodec).

### Quick Start

```bash
pip install audiocraft
```

### MusicGen — Text-to-Music

```python
from audiocraft.models import musicgen
from audiocraft.data.audio import audio_write

model = musicgen.MusicGen.get_pretrained('facebook/musicgen-small')
model.set_generation_params(duration=8)

wav = model.generate([
    "80s pop track with bassy drums and synth",
    "Cinematic orchestral piece, epic trailer music"
])

for idx, one_wav in enumerate(wav):
    audio_write(f'music_{idx}', one_wav.cpu(), model.sample_rate)
```

### AudioGen — Text-to-Sound Effects

```python
from audiocraft.models import audiogen

model = audiogen.AudioGen.get_pretrained('facebook/audiogen-medium')
model.set_generation_params(duration=5)

wav = model.generate(["dog barking in a park", "thunderstorm with rain"])
```

### Melody-Conditioned MusicGen

```python
import torchaudio
melody_waveform, sr = torchaudio.load("input_melody.wav")
model = musicgen.MusicGen.get_pretrained('facebook/musicgen-melody')
model.set_generation_params(duration=10)
wav = model.generate_with_chroma(["Happy pop song"], melody_waveform, sr)
```

### Key Parameters

| Parameter | Range | Effect |
|-----------|-------|--------|
| `duration` | 1–30s (small), up to 120s (large) | Generated length |
| `temperature` | 0.1–1.5 | Creativity vs coherence |
| `top_k` | 0–2500 | Nucleus sampling cutoff |
| `top_p` | 0.0–1.0 | Top-p nucleus filtering |

### Troubleshooting

- **CUDA OOM:** Reduce `max_batch_size` or use smaller model (`small` vs `large`)
- **Slow generation:** Ensure GPU is used; `torch.cuda.is_available()` must be True
- **Poor audio quality:** Increase `duration` slightly; melody conditioning helps
- **AssertionError on length:** Ensure input text isn't empty; check `torchaudio` version

---

## 4. Image Segmentation with SAM

Meta's Segment Anything Model — zero-shot segmentation via points, boxes, or automatic mask generation.

### Quick Start

```bash
pip install git+https://github.com/facebookresearch/segment-anything.git
pip install torch torchvision opencv-python matplotlib

# Download checkpoints
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
```

### Point-Prompted Segmentation

```python
from segment_anything import sam_model_registry, SamPredictor
import cv2

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam.to("cuda"))

image = cv2.imread("image.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
predictor.set_image(image)

masks, scores, logits = predictor.predict(
    point_coords=[[500, 375]],
    point_labels=[1],  # 1 = foreground, 0 = background
    multimask_output=True,
)

# masks: (N, H, W) boolean arrays
# scores: confidence per mask
# Choose highest scoring mask
best_mask = masks[scores.argmax()]
```

### Box-Prompted Segmentation

```python
masks, scores, _ = predictor.predict(
    box=[100, 100, 400, 400],  # [x1, y1, x2, y2]
    multimask_output=False,
)
```

### Automatic Mask Generation

```python
from segment_anything import SamAutomaticMaskGenerator

mask_generator = SamAutomaticMaskGenerator(sam)
masks = mask_generator.generate(image)

# masks is a list of dicts with: segmentation (bool HxW), area, bbox, predicted_iou, stability_score
for mask in sorted(masks, key=lambda x: x["area"], reverse=True)[:5]:
    print(f"Area: {mask['area']}, IoU: {mask['predicted_iou']:.3f}")
```

### Model Variants

| Model | Params | Speed | Accuracy | Best for |
|-------|--------|-------|----------|----------|
| ViT-B | 91M | Fastest | Good | Real-time, edge |
| ViT-L | 308M | Medium | Better | Balanced |
| ViT-H | 636M | Slowest | Best | High-quality offline |

### Export to ONNX (for deployment)

```python
from segment_anything.utils.onnx import SamOnnxModel

onnx_model = SamOnnxModel(sam, return_single_mask=True)
torch.onnx.export(
    onnx_model,
    (dummy_image_embedding, dummy_point_coords, dummy_point_labels),
    "sam_onnx.onnx",
    export_params=True,
    opset_version=17,
)
```

---

## 5. HuggingFace Hub — Model Management

Central model hub used by all the tools above.

### Authentication

```bash
# Login (required for gated/private models)
huggingface-cli login
# Token from https://huggingface.co/settings/tokens
```

### Download Models

```bash
# Full repo
huggingface-cli download microsoft/Phi-3-mini-4k-instruct --local-dir ./phi-3

# Single file
huggingface-cli download meta-llama/Meta-Llama-3-8B config.json

# With cache scan
huggingface-cli scan-cache
```

### Upload Models

```bash
# Upload a model
huggingface-cli upload my-username/my-model ./model-files/ .

# With README and tags
huggingface-cli upload my-username/my-model . . --repo-type model
```

### Programmatic

```python
from huggingface_hub import hf_hub_download, snapshot_download, list_models, HfApi

# Download
snapshot_download("meta-llama/Meta-Llama-3-8B", local_dir="./llama-3")

# Search
for m in list_models(filter="text-generation", sort="downloads", limit=10):
    print(f"{m.id}: {m.downloads}")

# Create repo and upload
api = HfApi()
api.create_repo("my-username/my-model")
api.upload_file(path_or_fileobj="model.bin", path_in_repo="pytorch_model.bin", repo_id="my-username/my-model")
```

---

## Workflows

### Train → Track → Evaluate → Serve

```
1. Prepare data (HuggingFace datasets or local)
2. Train with W&B logging (Section 2)
3. Evaluate with lm-eval (Section 1)
4. Register best checkpoint in W&B Model Registry
5. Convert to GGUF or deploy with vLLM (load llm-inference-and-serving)
```

### Multimodal Pipeline

```
1. Generate music with AudioCraft (Section 3)
2. Segment artwork with SAM (Section 4)
3. Package outputs as W&B artifacts (Section 2)
4. Track all in a single W&B run
```

---

## Resources

- **Benchmarking deep dive:** Archived `lm-evaluation-harness` references — `references/api-evaluation.md`, `references/custom-tasks.md`, `references/distributed-eval.md`, `references/benchmark-guide.md`
- **W&B deep dive:** Archived `weights-and-biases` references — `references/artifacts.md`, `references/integrations.md`, `references/sweeps.md`
- **AudioCraft deep dive:** Archived `audiocraft` references — `references/troubleshooting.md`, `references/advanced-usage.md`
- **SAM deep dive:** Archived `segment-anything` references — `references/troubleshooting.md`, `references/advanced-usage.md`