Data Pipeline Design
Design batch and streaming data pipelines. Plan ingestion, transformation, quality checks, and failure recovery. Use when building ETL/ELT systems or data infrastructure.
Install
/plugin marketplace add sethdford/claude-skillsThe repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.
npx skills add sethdford/claude-skills --skill data-pipeline-designgit clone --depth 1 https://github.com/sethdford/claude-skills.git /tmp/claude-skills
mkdir -p ~/.claude/skills
cp -r /tmp/claude-skills/architect/data-architecture/skills/data-pipeline-design ~/.claude/skills/data-pipeline-designPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Data Pipeline Design Design robust, maintainable data pipelines that reliably move, transform, and validate data at scale. ## Context You are designing data pipelines (batch or streaming). Plan data flow, transformations, quality gates, failure recovery, and monitoring. Read source systems, target requirements, latency expectations, and volume projections. ## Domain Context Based on modern data engineering practices (Spark, Airflow, Kafka, Beam): - **Batch Pipelines**: Scheduled jobs (hourly, daily); high throughput, moderate latency - **Streaming Pipelines**: Continuous ingestion; low latency, higher operational complexity - **Micro-batching**: Spark Streaming; lower latency than batch, simpler than true streaming - **Orchestration**: DAG-based scheduling (Airflow, dbt) for complex multi-stage pipelines - **Observability**: Monitor latency, throughput, data quality, freshness ## Instructions 1. **Choose Processing Model**: Batch (daily jobs?) or streaming (realtime features?)? Hybrid (Lambda: batch + streaming for both speed and accuracy)? Consider latency SLA and cost. 2. **Design Data Stages**: Raw ingestion (as-is from source) → Bronze. Cleansing and normalization → Silver. Business logic and enrichment → Gold. This layered medallion architecture separates concerns. 3. **Implement Quality Gates**: Validation at each stage. Fail pipeline if data quality drops. Track anomalies: unexpected null rates, value distributions, cardinality changes.
Details
- Skill name
data-pipeline-design- Repository
- sethdford/claude-skills
- Path
architect/data-architecture/skills/data-pipeline-design/SKILL.md- License
- MIT
- Repo updated
- 11 March 2026
FAQ
What does the Data Pipeline Design skill do?
Design batch and streaming data pipelines. Plan ingestion, transformation, quality checks, and failure recovery. Use when building ETL/ELT systems or data infrastructure.
How do I install the Data Pipeline Design skill?
Claude Code plugin: /plugin marketplace add sethdford/claude-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Data Pipeline Design popular?
It lives in sethdford/claude-skills, which has 48 GitHub stars and 10 forks.
Related skills
A B Test Design
48sethdford/claude-skills
Design rigorous A/B tests with hypotheses, variants, metrics, and sample size calculations.
A B Test Design Pm
48sethdford/claude-skills
Analytics and experimentation skill for product managers.
Abuse Case Design
48sethdford/claude-skills
Design abuse cases (negative use cases) showing how attackers misuse system features. Use when identifying attacks that exploit intended functionality or business logic flaws.
Data Visualization
48sethdford/claude-skills
Design clear, accessible data visualizations with appropriate chart selection and styling.
Data Flow Diagram
48sethdford/claude-skills
Design data movement and transformation pipelines. Show how data flows between systems, transforms, and where it's stored. Use when architecting data integrations or ETL processes.
Experiment Design
48sethdford/claude-skills
Analytics and experimentation skill for product managers.
Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.