Data Engineering Pipelines Best Practices

Design, operate, and review reliable data pipelines with explicit contracts, idempotent processing, quality gates, lineage, privacy, observability, and safe recovery.

9 stars 3 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add sandbaseai/workbuddy-skill --skill data-engineering-pipelines-best-practices
Manual install
git clone --depth 1 https://github.com/sandbaseai/workbuddy-skill.git /tmp/workbuddy-skill
mkdir -p ~/.claude/skills
cp -r /tmp/workbuddy-skill/skills/data-engineering-pipelines-best-practices ~/.claude/skills/data-engineering-pipelines-best-practices

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Data Engineering Pipeline Best Practices

Use this skill when designing, reviewing, debugging, or operating batch, stream,
ELT, or event-driven data pipelines. A pipeline is a product with producers,
consumers, contracts, freshness expectations, privacy obligations, and recovery
behavior—not just a sequence of jobs. Do not claim data quality or freshness
without evidence from the actual run and a defined baseline.

## Define the data contract

Record the business purpose, source and destination owners, schema and semantic
definitions, expected volume, freshness, latency, retention, lineage, allowed
consumers, and sensitivity classification. For every field that matters, define
type, nullability, units, validity range, identity or deduplication key, and
whether it may contain personal or confidential data.

Specify delivery semantics and failure behavior: at-most-once, at-least-once,
or effectively exactly-once; ordering scope; watermark; late-arrival policy;
replay window; checkpoint; retry and dead-letter handling; and what is
authoritative after a partial run. Version contracts deliberately and define
backward and forward compatibility before producers or consumers change.

## Build for correctness and recovery

- Make writes idempotent with stable run, event, or record keys; bound retries
  and prevent duplicate side effects.
- Separate raw, validated, transformed, and published layers where that makes
Read the full skill on GitHub

Details

Skill name
data-engineering-pipelines-best-practices
Path
skills/data-engineering-pipelines-best-practices/SKILL.md
License
MIT
Repo updated
8 September 2026

FAQ

What does the Data Engineering Pipelines Best Practices skill do?

Design, operate, and review reliable data pipelines with explicit contracts, idempotent processing, quality gates, lineage, privacy, observability, and safe recovery.

How do I install the Data Engineering Pipelines Best Practices skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add sandbaseai/workbuddy-skill --skill data-engineering-pipelines-best-practices. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Data Engineering Pipelines Best Practices popular?

It lives in sandbaseai/workbuddy-skill, which has 9 GitHub stars and 3 forks.

Related skills

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.