Data Pipeline

Comprehensive guide to data pipeline design, ETL/ELT patterns, data quality, monitoring, orchestration, and cost optimization for production-grade data engineering.

492 stars 68 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add cosmicstack-labs/mercury-agent-skills --skill data-pipeline
Manual install
git clone --depth 1 https://github.com/cosmicstack-labs/mercury-agent-skills.git /tmp/mercury-agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/mercury-agent-skills/categories/data/data-pipeline ~/.claude/skills/data-pipeline

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Data Pipeline Design

## Core Principles

Data pipelines are the arteries of modern data platforms. A well-designed pipeline is **reliable**, **observable**, **idempotent**, and **cost-efficient**. The following principles guide every decision:

1. **Idempotency First** — Running a pipeline twice should produce the same result. This enables safe retries and backfills without data duplication.
2. **Observability by Default** — Every stage must emit metrics, logs, and lineage metadata. If you can't see it, you can't fix it.
3. **Fail Gracefully** — Assume failures will happen. Design dead letter queues, retry logic with exponential backoff, and alerting on anomalies.
4. **Incremental Processing** — Process only what's changed. Full refreshes are for schema migrations and backfills only.
5. **Data Contracts** — Define and enforce schemas at every boundary. Catch drift before it reaches downstream consumers.
6. **Separation of Concerns** — Extract, transform, load are distinct phases. Each should be independently testable and debuggable.
7. **Cost Awareness** — Every byte processed costs money. Partition, compress, and prune aggressively.

---

## Pipeline Maturity Model

| Level | Name | Characteristics |
|-------|------|----------------|
| **0** | Ad-hoc | Manual scripts, no scheduling, no error handling, no documentation |
| **1** | Scheduled | Cron-based scheduling, basic retries, simple logging |
Read the full skill on GitHub

Details

Skill name
data-pipeline
Path
categories/data/data-pipeline/SKILL.md
License
MIT
Repo updated
25 August 2026

FAQ

What does the Data Pipeline skill do?

Comprehensive guide to data pipeline design, ETL/ELT patterns, data quality, monitoring, orchestration, and cost optimization for production-grade data engineering.

How do I install the Data Pipeline skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add cosmicstack-labs/mercury-agent-skills --skill data-pipeline. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Data Pipeline popular?

It lives in cosmicstack-labs/mercury-agent-skills, which has 492 GitHub stars and 68 forks.

Related skills

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.