Data Pipeline Design

Design batch and streaming data pipelines. Plan ingestion, transformation, quality checks, and failure recovery. Use when building ETL/ELT systems or data infrastructure.

48 stars 10 forksData & Analytics Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add sethdford/claude-skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add sethdford/claude-skills --skill data-pipeline-design
Manual install
git clone --depth 1 https://github.com/sethdford/claude-skills.git /tmp/claude-skills
mkdir -p ~/.claude/skills
cp -r /tmp/claude-skills/architect/data-architecture/skills/data-pipeline-design ~/.claude/skills/data-pipeline-design

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Data Pipeline Design

Design robust, maintainable data pipelines that reliably move, transform, and validate data at scale.

## Context

You are designing data pipelines (batch or streaming). Plan data flow, transformations, quality gates, failure recovery, and monitoring. Read source systems, target requirements, latency expectations, and volume projections.

## Domain Context

Based on modern data engineering practices (Spark, Airflow, Kafka, Beam):

- **Batch Pipelines**: Scheduled jobs (hourly, daily); high throughput, moderate latency
- **Streaming Pipelines**: Continuous ingestion; low latency, higher operational complexity
- **Micro-batching**: Spark Streaming; lower latency than batch, simpler than true streaming
- **Orchestration**: DAG-based scheduling (Airflow, dbt) for complex multi-stage pipelines
- **Observability**: Monitor latency, throughput, data quality, freshness

## Instructions

1. **Choose Processing Model**: Batch (daily jobs?) or streaming (realtime features?)? Hybrid (Lambda: batch + streaming for both speed and accuracy)? Consider latency SLA and cost.

2. **Design Data Stages**: Raw ingestion (as-is from source) → Bronze. Cleansing and normalization → Silver. Business logic and enrichment → Gold. This layered medallion architecture separates concerns.

3. **Implement Quality Gates**: Validation at each stage. Fail pipeline if data quality drops. Track anomalies: unexpected null rates, value distributions, cardinality changes.
Read the full skill on GitHub

Details

Skill name
data-pipeline-design
Path
architect/data-architecture/skills/data-pipeline-design/SKILL.md
License
MIT
Repo updated
11 March 2026

FAQ

What does the Data Pipeline Design skill do?

Design batch and streaming data pipelines. Plan ingestion, transformation, quality checks, and failure recovery. Use when building ETL/ELT systems or data infrastructure.

How do I install the Data Pipeline Design skill?

Claude Code plugin: /plugin marketplace add sethdford/claude-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Data Pipeline Design popular?

It lives in sethdford/claude-skills, which has 48 GitHub stars and 10 forks.

Related skills

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.