Data Engineering Pipelines Best Practices
Design, operate, and review reliable data pipelines with explicit contracts, idempotent processing, quality gates, lineage, privacy, observability, and safe recovery.
Install
npx skills add sandbaseai/workbuddy-skill --skill data-engineering-pipelines-best-practicesgit clone --depth 1 https://github.com/sandbaseai/workbuddy-skill.git /tmp/workbuddy-skill
mkdir -p ~/.claude/skills
cp -r /tmp/workbuddy-skill/skills/data-engineering-pipelines-best-practices ~/.claude/skills/data-engineering-pipelines-best-practicesPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Data Engineering Pipeline Best Practices Use this skill when designing, reviewing, debugging, or operating batch, stream, ELT, or event-driven data pipelines. A pipeline is a product with producers, consumers, contracts, freshness expectations, privacy obligations, and recovery behavior—not just a sequence of jobs. Do not claim data quality or freshness without evidence from the actual run and a defined baseline. ## Define the data contract Record the business purpose, source and destination owners, schema and semantic definitions, expected volume, freshness, latency, retention, lineage, allowed consumers, and sensitivity classification. For every field that matters, define type, nullability, units, validity range, identity or deduplication key, and whether it may contain personal or confidential data. Specify delivery semantics and failure behavior: at-most-once, at-least-once, or effectively exactly-once; ordering scope; watermark; late-arrival policy; replay window; checkpoint; retry and dead-letter handling; and what is authoritative after a partial run. Version contracts deliberately and define backward and forward compatibility before producers or consumers change. ## Build for correctness and recovery - Make writes idempotent with stable run, event, or record keys; bound retries and prevent duplicate side effects. - Separate raw, validated, transformed, and published layers where that makes
Details
- Skill name
data-engineering-pipelines-best-practices- Repository
- sandbaseai/workbuddy-skill
- Path
skills/data-engineering-pipelines-best-practices/SKILL.md- License
- MIT
- Repo updated
- 8 September 2026
FAQ
What does the Data Engineering Pipelines Best Practices skill do?
Design, operate, and review reliable data pipelines with explicit contracts, idempotent processing, quality gates, lineage, privacy, observability, and safe recovery.
How do I install the Data Engineering Pipelines Best Practices skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add sandbaseai/workbuddy-skill --skill data-engineering-pipelines-best-practices. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Data Engineering Pipelines Best Practices popular?
It lives in sandbaseai/workbuddy-skill, which has 9 GitHub stars and 3 forks.
Related skills
Access Control Security Best Practices
9sandbaseai/workbuddy-skill
Design, review, implement, and verify least-privilege access controls across users, services, tenants, resources, and administrative workflows.
Accessibility Review
9sandbaseai/workbuddy-skill
Use when reviewing a web interface or interaction for semantic, keyboard, focus, screen-reader, ARIA, and contrast accessibility risks.
Acquire Codebase Knowledge
9sandbaseai/workbuddy-skill
Use when the user asks to map, document, or onboard into an existing codebase and needs evidence-backed architecture, structure, conventions, integrations, testing, and concerns documentation.
Grafana Dashboards
9sandbaseai/workbuddy-skill
Use when designing Grafana dashboards for Prometheus metrics, service health, infrastructure, SLOs, or business KPIs.
Data Analysis
9sandbaseai/workbuddy-skill
Use when analyzing tabular or structured data for trends, distributions, comparisons, relationships, anomalies, or decisions; establish data quality and statistical limits before drawing conclusions.
Product Analytics
9sandbaseai/workbuddy-skill
Use when defining product metrics, event taxonomies, funnels, cohorts, retention, experiments, or dashboards; connect measurable user value to decisions without compromising privacy or statistical validity.
Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.