Data Pipeline
Comprehensive guide to data pipeline design, ETL/ELT patterns, data quality, monitoring, orchestration, and cost optimization for production-grade data engineering.
Install
npx skills add cosmicstack-labs/mercury-agent-skills --skill data-pipelinegit clone --depth 1 https://github.com/cosmicstack-labs/mercury-agent-skills.git /tmp/mercury-agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/mercury-agent-skills/categories/data/data-pipeline ~/.claude/skills/data-pipelinePersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Data Pipeline Design ## Core Principles Data pipelines are the arteries of modern data platforms. A well-designed pipeline is **reliable**, **observable**, **idempotent**, and **cost-efficient**. The following principles guide every decision: 1. **Idempotency First** — Running a pipeline twice should produce the same result. This enables safe retries and backfills without data duplication. 2. **Observability by Default** — Every stage must emit metrics, logs, and lineage metadata. If you can't see it, you can't fix it. 3. **Fail Gracefully** — Assume failures will happen. Design dead letter queues, retry logic with exponential backoff, and alerting on anomalies. 4. **Incremental Processing** — Process only what's changed. Full refreshes are for schema migrations and backfills only. 5. **Data Contracts** — Define and enforce schemas at every boundary. Catch drift before it reaches downstream consumers. 6. **Separation of Concerns** — Extract, transform, load are distinct phases. Each should be independently testable and debuggable. 7. **Cost Awareness** — Every byte processed costs money. Partition, compress, and prune aggressively. --- ## Pipeline Maturity Model | Level | Name | Characteristics | |-------|------|----------------| | **0** | Ad-hoc | Manual scripts, no scheduling, no error handling, no documentation | | **1** | Scheduled | Cron-based scheduling, basic retries, simple logging |
Details
- Skill name
data-pipeline- Repository
- cosmicstack-labs/mercury-agent-skills
- Path
categories/data/data-pipeline/SKILL.md- License
- MIT
- Repo updated
- 25 August 2026
FAQ
What does the Data Pipeline skill do?
Comprehensive guide to data pipeline design, ETL/ELT patterns, data quality, monitoring, orchestration, and cost optimization for production-grade data engineering.
How do I install the Data Pipeline skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add cosmicstack-labs/mercury-agent-skills --skill data-pipeline. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Data Pipeline popular?
It lives in cosmicstack-labs/mercury-agent-skills, which has 492 GitHub stars and 68 forks.
Related skills
Accessibility
492cosmicstack-labs/mercury-agent-skills
Achieve and maintain WCAG compliance through inclusive design practices, proper ARIA usage, and comprehensive testing methodologies.
Accessibility Testing
492cosmicstack-labs/mercury-agent-skills
WCAG 2.1/2.2 audit, axe, Lighthouse, manual testing, screen reader testing, and remediation
Agent Audit Logging
492cosmicstack-labs/mercury-agent-skills
Implement comprehensive audit logging and reporting for multi-agent systems. Covers event capture, structured logging, traceability, compliance reporting, forensic analysis, and real-time monitoring dashboards for agent actions and decisions.
Highlight Graph
329readwiseio/readwise-skills
Visualize your highlights and their connections in an interactive 2D graph
Data Visualization
365NTCoding/claude-skillz
Comprehensive data visualization skill covering visual execution and technical implementation. Includes perceptual foundations, chart selection, layout algorithms, and library guidance. Triggers on: charts, graphs, dashboards, 'visualize', 'plot', data presentation, D3, Recharts, Victory.
Alt Text Headings
449Community-Access/accessibility-agents
Alt text, SVGs, figures, charts, heading order, page titles and landmarks.
Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.