Senior Data Engineer
World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.
Install
npx skills add UCSB-NLP-Chang/Skill-Usage --skill senior-data-engineergit clone --depth 1 https://github.com/UCSB-NLP-Chang/Skill-Usage.git /tmp/Skill-Usage
mkdir -p ~/.claude/skills
cp -r /tmp/Skill-Usage/tasks/flink-query/environment/skills/senior-data-engineer ~/.claude/skills/senior-data-engineerPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Senior Data Engineer ## Core Capabilities - **Batch Pipeline Orchestration** - Design and implement production-ready ETL/ELT pipelines with Airflow, intelligent dependency resolution, retry logic, and comprehensive monitoring - **Real-Time Streaming** - Build event-driven streaming pipelines with Kafka, Flink, Kinesis, and Spark Streaming with exactly-once semantics and sub-second latency - **Data Quality Management** - Comprehensive batch and streaming data quality validation covering completeness, accuracy, consistency, timeliness, and validity - **Streaming Quality Monitoring** - Track consumer lag, data freshness, schema drift, throughput, and dead letter queue rates for streaming pipelines - **Performance Optimization** - Analyze and optimize pipeline performance with query optimization, Spark tuning, and cost analysis recommendations ## Key Workflows ### Workflow 1: Build ETL Pipeline **Time:** 2-4 hours **Steps:** 1. Design pipeline architecture using Lambda, Kappa, or Medallion pattern 2. Configure YAML pipeline definition with sources, transformations, targets 3. Generate Airflow DAG with `pipeline_orchestrator.py` 4. Define data quality validation rules 5. Deploy and configure monitoring/alerting **Expected Output:** Production-ready ETL pipeline with 99%+ success rate, automated quality checks, and comprehensive monitoring ### Workflow 2: Build Real-Time Streaming Pipeline **Time:** 3-5 days **Steps:**
Details
- Skill name
senior-data-engineer- Repository
- UCSB-NLP-Chang/Skill-Usage
- Path
tasks/flink-query/environment/skills/senior-data-engineer/SKILL.md- Repo updated
- 8 April 2026
FAQ
What does the Senior Data Engineer skill do?
World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.
How do I install the Senior Data Engineer skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add UCSB-NLP-Chang/Skill-Usage --skill senior-data-engineer. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Senior Data Engineer popular?
It lives in UCSB-NLP-Chang/Skill-Usage, which has 49 GitHub stars and 5 forks.
Related skills
Pubchem Database
49UCSB-NLP-Chang/Skill-Usage
Query PubChem via PUG-REST API/PubChemPy (110M+ compounds). Search by name/CID/SMILES, retrieve properties, similarity/substructure searches, bioactivity, for cheminformatics.
Pymatgen
49UCSB-NLP-Chang/Skill-Usage
Materials science toolkit. Crystal structures (CIF, POSCAR), phase diagrams, band structure, DOS, Materials Project integration, format conversion, for computational materials science.
Rdkit
49UCSB-NLP-Chang/Skill-Usage
Cheminformatics toolkit for fine-grained molecular control. SMILES/SDF parsing, descriptors (MW, LogP, TPSA), fingerprints, substructure search, 2D/3D generation, similarity, reactions. For standard workflows with simpler interface, use datamol (wrapper around RDKit). Use rdkit for advanced control, custom sanitization, specialized algorithms.
Trend Analysis
49UCSB-NLP-Chang/Skill-Usage
Detect long-term trends in time series data using parametric and non-parametric methods. Use when determining if a variable shows statistically significant increase or decrease over time.
Search Driving Distance
49UCSB-NLP-Chang/Skill-Usage
Estimate driving/taxi duration, distance, and rough cost between two cities using the bundled distance matrix CSV. Use this skill when comparing ground travel options or validating itinerary legs.
Cohort Analysis
48sethdford/claude-skills
Analytics and experimentation skill for product managers.
Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.