Distributed Training
Use when the user asks about multi-GPU training, DDP, FSDP, ZeRO, DeepSpeed, torchrun, tensor/pipeline parallel, all-reduce or NCCL slowness. Produces a parallelism plan with per-GPU memory math, launch command and config. Not for fabric tuning (use rdma-roce) or experiment logging (use ml-experiment-tracking).
Install
npx skills add j4flmao/agent-skills --skill distributed-traininggit clone --depth 1 https://github.com/j4flmao/agent-skills.git /tmp/agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/agent-skills/skills/hardware-acceleration/distributed-training ~/.claude/skills/distributed-trainingPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Distributed Training
## Purpose
Train models that are too slow or too large for one GPU by splitting data, optimizer state, parameters, or layers across devices. The right strategy follows from a memory budget and the interconnect topology; communication cost, not FLOPs, usually decides scaling efficiency.
## Agent Protocol
### Trigger
"distributed training", "multi-gpu", "multi-node", "DDP", "FSDP", "ZeRO", "DeepSpeed", "Megatron", "all-reduce", "tensor parallel", "pipeline parallel", "torchrun", "NCCL".
### Input Context
- Model size (parameters), sequence length, micro-batch size, precision.
- GPU type/memory, GPUs per node, node count, intra-node (NVLink/PCIe) and inter-node (InfiniBand/RoCE) links.
- Framework (PyTorch native, DeepSpeed, Megatron-Core, Accelerate, Lightning).
### Output Artifact
Parallelism plan + launch command + config. No file unless requested.
### Response Format
```
Memory/GPU: params {GB} + grads {GB} + optimizer {GB} + activations {GB}
Strategy: {DDP|FSDP/ZeRO-n|TP=x PP=y DP=z}
Launch: {torchrun/deepspeed command}
Watch: {MFU, step time, comm/compute overlap}
```
### Completion Criteria
- [ ] Per-GPU memory estimate fits with headroom.
- [ ] Global batch = micro_batch x grad_accum x data_parallel_size documented.
- [ ] Checkpointing works with the sharding scheme (sharded/distributed checkpoint).
- [ ] Scaling efficiency measured (throughput at 1 node vs N nodes).
## Memory ModelDetails
- Skill name
distributed-training- Repository
- j4flmao/agent-skills
- Path
skills/hardware-acceleration/distributed-training/SKILL.md- License
- MIT
- Repo updated
- 17 November 2026
FAQ
What does the Distributed Training skill do?
Use when the user asks about multi-GPU training, DDP, FSDP, ZeRO, DeepSpeed, torchrun, tensor/pipeline parallel, all-reduce or NCCL slowness. Produces a parallelism plan with per-GPU memory math, launch command and config. Not for fabric tuning (use rdma-roce) or experiment logging (use ml-experiment-tracking).
How do I install the Distributed Training skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add j4flmao/agent-skills --skill distributed-training. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Distributed Training popular?
It lives in j4flmao/agent-skills, which has 45 GitHub stars and 6 forks.
Related skills
Agent Observability
45j4flmao/agent-skills
Use when the user asks about agent observability, tracing agent reasoning, debugging an agent loop, OpenTelemetry GenAI spans for LLM calls, LLM cost tracking per agent, agent audit logs or hallucination/eval monitoring in production. Produces instrumentation config, trace schema, dashboards, alerts and cost attribution. Not for generic service tracing without LLMs (use devops-opentelemetry).
AI Agent Architectures
45j4flmao/agent-skills
Use when the user asks to build an AI agent, design an agent that calls tools and keeps memory, write an agent system prompt or pick an agent framework (Claude Agent SDK, OpenAI Agents SDK, LangGraph). Produces an architecture brief covering loop, tools, memory, guardrails and evals, routing depth to child skills. Not for workflow patterns (use ai-agentic-workflows).
AI Agent Core Loop
45j4flmao/agent-skills
Use when the user asks about the agent loop, ReAct vs plan-and-solve, re-planning, Reflexion, an agent stuck in a loop, max iterations or termination conditions. Produces a loop spec or patch with state, success predicate, caps and no-progress detection. Not for overall agent design (use ai-agent-architectures) or tool schemas (use tool-calling-principles).
Ecmwf Observations
23ecmwf/ecmwf-weather-skills
Retrieves weather station observations (SYNOP and METAR — 2 m temperature, dew point, humidity, wind, pressure, precipitation) from ECMWF's MARS archive, with the departures from ECMWF's first guess and analysis and QC flags — load it first for any request for observed or measured weather at a station or airport, past observations, a WMO station id or ICAO code, or comparing a forecast with what was observed. Use when a task mentions observations, station data, SYNOP, METAR, BUFR or ODB feedback, verifying or scoring a forecast against stations (bias, MAE, RMSE), observed heat index or humidex
C Cpp Patterns
45j4flmao/agent-skills
Use when the user asks about embedded C or C++ firmware, register access, volatile, memory-mapped IO, ISR data sharing, ring buffers, struct packing, MISRA or linker scripts. Produces microcontroller C/C++ with static memory and safe register access. Not for RTOS task design (use rtos) or desktop C++ (use lang-cpp).
Enterprise Capacity Planning
45j4flmao/agent-skills
Use when the user asks about capacity planning, forecasting infrastructure growth, headroom, peak vs sustained load, hardware procurement lead times or cost per unit of work. Produces a capacity model and multi-quarter roadmap. Not for cloud bill optimization (use devops-finops) or autoscaling config (use devops-kubernetes-autoscaling).
Browse the most popular Claude skills, all productivity & notes skills, or write your own with the skills generator.