Data Pipeline Engineer

Expert data engineer for ETL/ELT pipelines, streaming, data warehousing. Activate on: data pipeline, ETL, ELT, data warehouse, Spark, Kafka, Airflow, dbt, data modeling, star schema, streaming data, batch processing, data quality. NOT for: API design (use api-architect), ML training (use ML skills), dashboards (use design skills).

272 stars 46 forksData & Analytics Plugin marketplace

Install

Claude Code plugin
/plugin marketplace add curiositech/some_claude_skills

The repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add curiositech/some_claude_skills --skill data-pipeline-engineer
Manual install
git clone --depth 1 https://github.com/curiositech/some_claude_skills.git /tmp/some_claude_skills
mkdir -p ~/.claude/skills
cp -r /tmp/some_claude_skills/.claude/skills/data-pipeline-engineer ~/.claude/skills/data-pipeline-engineer

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Data Pipeline Engineer

Expert data engineer specializing in ETL/ELT pipelines, streaming architectures, data warehousing, and modern data stack implementation.

## Quick Start

1. **Identify sources** - data formats, volumes, freshness requirements
2. **Choose architecture** - Medallion (Bronze/Silver/Gold), Lambda, or Kappa
3. **Design layers** - staging → intermediate → marts (dbt pattern)
4. **Add quality gates** - Great Expectations or dbt tests at each layer
5. **Orchestrate** - Airflow DAGs with sensors and retries
6. **Monitor** - lineage, freshness, anomaly detection

## Core Capabilities

| Capability | Technologies | Key Patterns |
|------------|--------------|--------------|
| **Batch Processing** | Spark, dbt, Databricks | Incremental, partitioning, Delta/Iceberg |
| **Stream Processing** | Kafka, Flink, Spark Streaming | Watermarks, exactly-once, windowing |
| **Orchestration** | Airflow, Dagster, Prefect | DAG design, sensors, task groups |
| **Data Modeling** | dbt, SQL | Kimball, Data Vault, SCD |
| **Data Quality** | Great Expectations, dbt tests | Validation suites, freshness |

## Architecture Patterns

### Medallion Architecture (Recommended)
```
BRONZE (Raw)     → Exact source copy, schema-on-read, partitioned by ingestion
      ↓ Cleaning, Deduplication
SILVER (Cleansed) → Validated, standardized, business logic applied
      ↓ Aggregation, Enrichment
GOLD (Business)   → Dimensional models, aggregates, ready for BI/ML
```

### Lambda vs Kappa
Read the full skill on GitHub

Details

Skill name
data-pipeline-engineer
Path
.claude/skills/data-pipeline-engineer/SKILL.md
License
MIT
Repo updated
5 November 2026

FAQ

What does the Data Pipeline Engineer skill do?

Expert data engineer for ETL/ELT pipelines, streaming, data warehousing. Activate on: data pipeline, ETL, ELT, data warehouse, Spark, Kafka, Airflow, dbt, data modeling, star schema, streaming data, batch processing, data quality. NOT for: API design (use api-architect), ML training (use ML skills), dashboards (use design skills).

How do I install the Data Pipeline Engineer skill?

Claude Code plugin: /plugin marketplace add curiositech/some_claude_skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Data Pipeline Engineer popular?

It lives in curiositech/some_claude_skills, which has 272 GitHub stars and 46 forks.

Related skills

Email Composer

272

curiositech/some_claude_skills

Draft professional emails for various contexts including business, technical, and customer communication. Use when the user needs help writing emails or composing professional messages.

Productivity & Notes46

2000s Visualization Expert

272

curiositech/some_claude_skills

Expert in 2000s-era music visualization (Milkdrop, AVS, Geiss) and modern WebGL implementations. Specializes in Butterchurn integration, Web Audio API AnalyserNode FFT data, GLSL shaders for audio-reactive visuals, and psychedelic generative art. Activate on "Milkdrop", "music visualization", "WebGL visualizer", "Butterchurn", "audio reactive", "FFT visualization", "spectrum analyzer". NOT for simple bar charts/waveforms (use basic canvas), video editing, or non-audio visuals.

Data & Analytics46

2026 Legal Research Agent

272

curiositech/some_claude_skills

Expert legal research agent for finding and scraping expungement data state by state. Knows authoritative sources, URL patterns, Firecrawl configuration, and 2026 legal landscape. Activate on "find expungement data", "scrape state laws", "legal research", "court URLs", "statute sources", "Clean Slate laws", "automatic expungement research". NOT for interpreting laws (use national-expungement-expert), building UI, or legal advice.

Research & Science46

Geospatial Data Pipeline

272

curiositech/some_claude_skills

Process, analyze, and visualize geospatial data at scale. Handles drone imagery, GPS tracks, GeoJSON optimization, coordinate transformations, and tile generation. Use for mapping apps, drone data processing, location-based services. Activate on "geospatial", "GIS", "PostGIS", "GeoJSON", "map tiles", "coordinate systems". NOT for simple address validation, basic distance calculations, or static map embeds.

Data & Analytics46

Chatbot Analytics

272

curiositech/some_claude_skills

Implement AI chatbot analytics and conversation monitoring. Use when adding conversation metrics, tracking AI usage, measuring user engagement with chat, or building conversation dashboards. Activates for AI analytics, token tracking, conversation categorization, and chat performance.

Data & Analytics46

Hand Drawn Infographic Creator

272

curiositech/some_claude_skills

Generate hand-drawn style diagrams and infographics for recovery education articles. Creates anatomist's notebook aesthetic visuals - brain diagrams, timelines, social comparisons, and process flows using continuous line art, semantic color coding, and margin annotations.

Data & Analytics46

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.