Senior Data Engineer

Data engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka. Use when designing data architectures, building pipelines, adding data-quality checks, optimizing ETL/ELT, or troubleshooting pipeline failures.

1,157 stars 217 forksData & Analytics Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add borghei/Claude-Skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add borghei/Claude-Skills --skill senior-data-engineer
Manual install
git clone --depth 1 https://github.com/borghei/Claude-Skills.git /tmp/Claude-Skills
mkdir -p ~/.claude/skills
cp -r /tmp/Claude-Skills/engineering/senior-data-engineer ~/.claude/skills/senior-data-engineer

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Senior Data Engineer

Generate pipeline configurations (Airflow, Prefect, Dagster), validate data quality with profiling and anomaly detection, and optimize SQL/Spark performance with actionable recommendations.

## Core Capabilities

- **Pipeline generation** — Airflow/Prefect/Dagster DAG code for batch and incremental loads, with DAG validation.
- **Data quality** — schema validation, profiling, anomaly detection, data contracts, and Great Expectations suite generation.
- **ETL/ELT optimization** — SQL and Spark analysis, partition strategy, and query cost estimation per warehouse.
- **Architecture decisions** — batch vs streaming and warehouse vs lakehouse trade-off frameworks.
- **Reliability patterns** — incremental watermarks, dead letter queues, freshness checks, and schema-drift detection.

## When to Use

- Designing a data architecture or choosing batch vs streaming / warehouse vs lakehouse.
- Building or generating Airflow/Spark/dbt pipelines.
- Adding data-quality checks or data contracts.
- Optimizing slow ETL/ELT queries or troubleshooting pipeline failures.

## Clarify First

Before generating pipelines, settle the inputs below. When one is missing or vague and would change the result, ask about it; when it wouldn't, use a sensible default and say which one you used.

- **Orchestrator:** Airflow / Prefect / Dagster (`--type`; changes the generated DAG code).
Read the full skill on GitHub

Details

Skill name
senior-data-engineer
Path
engineering/senior-data-engineer/SKILL.md
Repo updated
8 November 2026

FAQ

What does the Senior Data Engineer skill do?

Data engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka. Use when designing data architectures, building pipelines, adding data-quality checks, optimizing ETL/ELT, or troubleshooting pipeline failures.

How do I install the Senior Data Engineer skill?

Claude Code plugin: /plugin marketplace add borghei/Claude-Skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Senior Data Engineer popular?

It lives in borghei/Claude-Skills, which has 1,157 GitHub stars and 217 forks, and is included in 1 curated skill list.

Related skills

Cto Advisor

1.2k

borghei/Claude-Skills

Technical leadership guidance for engineering teams, architecture decisions, and technology strategy. Use when assessing technical debt, scaling engineering teams, evaluating technologies, or making architecture decisions.

Software DevelopmentIn 1 list217

Customer Success Manager

1.2k

borghei/Claude-Skills

Monitors customer health, predicts churn risk, and identifies expansion opportunities using weighted scoring models for SaaS customer success

Marketing, Sales & BusinessIn 1 list217

Financial Analyst

1.2k

borghei/Claude-Skills

Performs financial ratio analysis, DCF valuation, budget variance analysis, and rolling forecast construction for strategic decision-making

Finance & CryptoIn 1 list217

Statistical Analyst

1.2k

borghei/Claude-Skills

Applied statistics for business and product questions — test selection, assumption checks, power planning, effect sizes with intervals, multiplicity correction. Use when interpreting an experiment, sizing a study, or vetting a claim.

Data & AnalyticsIn 1 list217

Product Analytics

1.2k

borghei/Claude-Skills

Product analytics for instrumenting products, defining metrics, and building retention funnels. Use when designing a metric tree, instrumenting a feature, auditing instrumentation, defining a North Star, or building an analytics roadmap.

Data & AnalyticsIn 1 list217

Wayfinder

1.2k

borghei/Claude-Skills

Chart work too big for one sprint as a map of decision tickets, resolve them in dependency order, then turn answers into tracer-bullet tickets. Use when the route is unclear, open questions block each other, or estimates swing wildly.

Data & AnalyticsIn 1 list217

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.