Senior Data Engineer

World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.

49 stars 5 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add UCSB-NLP-Chang/Skill-Usage --skill senior-data-engineer
Manual install
git clone --depth 1 https://github.com/UCSB-NLP-Chang/Skill-Usage.git /tmp/Skill-Usage
mkdir -p ~/.claude/skills
cp -r /tmp/Skill-Usage/tasks/flink-query/environment/skills/senior-data-engineer ~/.claude/skills/senior-data-engineer

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Senior Data Engineer

## Core Capabilities

- **Batch Pipeline Orchestration** - Design and implement production-ready ETL/ELT pipelines with Airflow, intelligent dependency resolution, retry logic, and comprehensive monitoring
- **Real-Time Streaming** - Build event-driven streaming pipelines with Kafka, Flink, Kinesis, and Spark Streaming with exactly-once semantics and sub-second latency
- **Data Quality Management** - Comprehensive batch and streaming data quality validation covering completeness, accuracy, consistency, timeliness, and validity
- **Streaming Quality Monitoring** - Track consumer lag, data freshness, schema drift, throughput, and dead letter queue rates for streaming pipelines
- **Performance Optimization** - Analyze and optimize pipeline performance with query optimization, Spark tuning, and cost analysis recommendations

## Key Workflows

### Workflow 1: Build ETL Pipeline

**Time:** 2-4 hours

**Steps:**
1. Design pipeline architecture using Lambda, Kappa, or Medallion pattern
2. Configure YAML pipeline definition with sources, transformations, targets
3. Generate Airflow DAG with `pipeline_orchestrator.py`
4. Define data quality validation rules
5. Deploy and configure monitoring/alerting

**Expected Output:** Production-ready ETL pipeline with 99%+ success rate, automated quality checks, and comprehensive monitoring

### Workflow 2: Build Real-Time Streaming Pipeline

**Time:** 3-5 days

**Steps:**
Read the full skill on GitHub

Details

Skill name
senior-data-engineer
Path
tasks/flink-query/environment/skills/senior-data-engineer/SKILL.md
Repo updated
8 April 2026

FAQ

What does the Senior Data Engineer skill do?

World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.

How do I install the Senior Data Engineer skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add UCSB-NLP-Chang/Skill-Usage --skill senior-data-engineer. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Senior Data Engineer popular?

It lives in UCSB-NLP-Chang/Skill-Usage, which has 49 GitHub stars and 5 forks.

Related skills

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.