Distributed Training

Use when the user asks about multi-GPU training, DDP, FSDP, ZeRO, DeepSpeed, torchrun, tensor/pipeline parallel, all-reduce or NCCL slowness. Produces a parallelism plan with per-GPU memory math, launch command and config. Not for fabric tuning (use rdma-roce) or experiment logging (use ml-experiment-tracking).

45 stars 6 forksProductivity & Notes

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add j4flmao/agent-skills --skill distributed-training
Manual install
git clone --depth 1 https://github.com/j4flmao/agent-skills.git /tmp/agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/agent-skills/skills/hardware-acceleration/distributed-training ~/.claude/skills/distributed-training

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Distributed Training

## Purpose
Train models that are too slow or too large for one GPU by splitting data, optimizer state, parameters, or layers across devices. The right strategy follows from a memory budget and the interconnect topology; communication cost, not FLOPs, usually decides scaling efficiency.

## Agent Protocol

### Trigger
"distributed training", "multi-gpu", "multi-node", "DDP", "FSDP", "ZeRO", "DeepSpeed", "Megatron", "all-reduce", "tensor parallel", "pipeline parallel", "torchrun", "NCCL".

### Input Context
- Model size (parameters), sequence length, micro-batch size, precision.
- GPU type/memory, GPUs per node, node count, intra-node (NVLink/PCIe) and inter-node (InfiniBand/RoCE) links.
- Framework (PyTorch native, DeepSpeed, Megatron-Core, Accelerate, Lightning).

### Output Artifact
Parallelism plan + launch command + config. No file unless requested.

### Response Format
```
Memory/GPU: params {GB} + grads {GB} + optimizer {GB} + activations {GB}
Strategy: {DDP|FSDP/ZeRO-n|TP=x PP=y DP=z}
Launch: {torchrun/deepspeed command}
Watch: {MFU, step time, comm/compute overlap}
```

### Completion Criteria
- [ ] Per-GPU memory estimate fits with headroom.
- [ ] Global batch = micro_batch x grad_accum x data_parallel_size documented.
- [ ] Checkpointing works with the sharding scheme (sharded/distributed checkpoint).
- [ ] Scaling efficiency measured (throughput at 1 node vs N nodes).

## Memory Model
Read the full skill on GitHub

Details

Skill name
distributed-training
Path
skills/hardware-acceleration/distributed-training/SKILL.md
License
MIT
Repo updated
17 November 2026

FAQ

What does the Distributed Training skill do?

Use when the user asks about multi-GPU training, DDP, FSDP, ZeRO, DeepSpeed, torchrun, tensor/pipeline parallel, all-reduce or NCCL slowness. Produces a parallelism plan with per-GPU memory math, launch command and config. Not for fabric tuning (use rdma-roce) or experiment logging (use ml-experiment-tracking).

How do I install the Distributed Training skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add j4flmao/agent-skills --skill distributed-training. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Distributed Training popular?

It lives in j4flmao/agent-skills, which has 45 GitHub stars and 6 forks.

Related skills

Agent Observability

45

j4flmao/agent-skills

Use when the user asks about agent observability, tracing agent reasoning, debugging an agent loop, OpenTelemetry GenAI spans for LLM calls, LLM cost tracking per agent, agent audit logs or hallucination/eval monitoring in production. Produces instrumentation config, trace schema, dashboards, alerts and cost attribution. Not for generic service tracing without LLMs (use devops-opentelemetry).

DevOps & Cloud

AI Agent Architectures

45

j4flmao/agent-skills

Use when the user asks to build an AI agent, design an agent that calls tools and keeps memory, write an agent system prompt or pick an agent framework (Claude Agent SDK, OpenAI Agents SDK, LangGraph). Produces an architecture brief covering loop, tools, memory, guardrails and evals, routing depth to child skills. Not for workflow patterns (use ai-agentic-workflows).

AI, LLMs & Agents

AI Agent Core Loop

45

j4flmao/agent-skills

Use when the user asks about the agent loop, ReAct vs plan-and-solve, re-planning, Reflexion, an agent stuck in a loop, max iterations or termination conditions. Produces a loop spec or patch with state, success predicate, caps and no-progress detection. Not for overall agent design (use ai-agent-architectures) or tool schemas (use tool-calling-principles).

AI, LLMs & Agents

Ecmwf Observations

23

ecmwf/ecmwf-weather-skills

Retrieves weather station observations (SYNOP and METAR — 2 m temperature, dew point, humidity, wind, pressure, precipitation) from ECMWF's MARS archive, with the departures from ECMWF's first guess and analysis and QC flags — load it first for any request for observed or measured weather at a station or airport, past observations, a WMO station id or ICAO code, or comparing a forecast with what was observed. Use when a task mentions observations, station data, SYNOP, METAR, BUFR or ODB feedback, verifying or scoring a forecast against stations (bias, MAE, RMSE), observed heat index or humidex

Productivity & Notes

C Cpp Patterns

45

j4flmao/agent-skills

Use when the user asks about embedded C or C++ firmware, register access, volatile, memory-mapped IO, ISR data sharing, ring buffers, struct packing, MISRA or linker scripts. Produces microcontroller C/C++ with static memory and safe register access. Not for RTOS task design (use rtos) or desktop C++ (use lang-cpp).

Productivity & Notes

Enterprise Capacity Planning

45

j4flmao/agent-skills

Use when the user asks about capacity planning, forecasting infrastructure growth, headroom, peak vs sustained load, hardware procurement lead times or cost per unit of work. Produces a capacity model and multi-quarter roadmap. Not for cloud bill optimization (use devops-finops) or autoscaling config (use devops-kubernetes-autoscaling).

Productivity & Notes

Browse the most popular Claude skills, all productivity & notes skills, or write your own with the skills generator.