Data Pipeline Engineer
Expert data engineer for ETL/ELT pipelines, streaming, data warehousing. Activate on: data pipeline, ETL, ELT, data warehouse, Spark, Kafka, Airflow, dbt, data modeling, star schema, streaming data, batch processing, data quality. NOT for: API design (use api-architect), ML training (use ML skills), dashboards (use design skills).
Install
/plugin marketplace add curiositech/some_claude_skillsThe repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.
npx skills add curiositech/some_claude_skills --skill data-pipeline-engineergit clone --depth 1 https://github.com/curiositech/some_claude_skills.git /tmp/some_claude_skills
mkdir -p ~/.claude/skills
cp -r /tmp/some_claude_skills/.claude/skills/data-pipeline-engineer ~/.claude/skills/data-pipeline-engineerPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Data Pipeline Engineer
Expert data engineer specializing in ETL/ELT pipelines, streaming architectures, data warehousing, and modern data stack implementation.
## Quick Start
1. **Identify sources** - data formats, volumes, freshness requirements
2. **Choose architecture** - Medallion (Bronze/Silver/Gold), Lambda, or Kappa
3. **Design layers** - staging → intermediate → marts (dbt pattern)
4. **Add quality gates** - Great Expectations or dbt tests at each layer
5. **Orchestrate** - Airflow DAGs with sensors and retries
6. **Monitor** - lineage, freshness, anomaly detection
## Core Capabilities
| Capability | Technologies | Key Patterns |
|------------|--------------|--------------|
| **Batch Processing** | Spark, dbt, Databricks | Incremental, partitioning, Delta/Iceberg |
| **Stream Processing** | Kafka, Flink, Spark Streaming | Watermarks, exactly-once, windowing |
| **Orchestration** | Airflow, Dagster, Prefect | DAG design, sensors, task groups |
| **Data Modeling** | dbt, SQL | Kimball, Data Vault, SCD |
| **Data Quality** | Great Expectations, dbt tests | Validation suites, freshness |
## Architecture Patterns
### Medallion Architecture (Recommended)
```
BRONZE (Raw) → Exact source copy, schema-on-read, partitioned by ingestion
↓ Cleaning, Deduplication
SILVER (Cleansed) → Validated, standardized, business logic applied
↓ Aggregation, Enrichment
GOLD (Business) → Dimensional models, aggregates, ready for BI/ML
```
### Lambda vs KappaDetails
- Skill name
data-pipeline-engineer- Repository
- curiositech/some_claude_skills
- Path
.claude/skills/data-pipeline-engineer/SKILL.md- License
- MIT
- Repo updated
- 5 November 2026
FAQ
What does the Data Pipeline Engineer skill do?
Expert data engineer for ETL/ELT pipelines, streaming, data warehousing. Activate on: data pipeline, ETL, ELT, data warehouse, Spark, Kafka, Airflow, dbt, data modeling, star schema, streaming data, batch processing, data quality. NOT for: API design (use api-architect), ML training (use ML skills), dashboards (use design skills).
How do I install the Data Pipeline Engineer skill?
Claude Code plugin: /plugin marketplace add curiositech/some_claude_skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Data Pipeline Engineer popular?
It lives in curiositech/some_claude_skills, which has 272 GitHub stars and 46 forks.
Related skills
Email Composer
272curiositech/some_claude_skills
Draft professional emails for various contexts including business, technical, and customer communication. Use when the user needs help writing emails or composing professional messages.
2000s Visualization Expert
272curiositech/some_claude_skills
Expert in 2000s-era music visualization (Milkdrop, AVS, Geiss) and modern WebGL implementations. Specializes in Butterchurn integration, Web Audio API AnalyserNode FFT data, GLSL shaders for audio-reactive visuals, and psychedelic generative art. Activate on "Milkdrop", "music visualization", "WebGL visualizer", "Butterchurn", "audio reactive", "FFT visualization", "spectrum analyzer". NOT for simple bar charts/waveforms (use basic canvas), video editing, or non-audio visuals.
2026 Legal Research Agent
272curiositech/some_claude_skills
Expert legal research agent for finding and scraping expungement data state by state. Knows authoritative sources, URL patterns, Firecrawl configuration, and 2026 legal landscape. Activate on "find expungement data", "scrape state laws", "legal research", "court URLs", "statute sources", "Clean Slate laws", "automatic expungement research". NOT for interpreting laws (use national-expungement-expert), building UI, or legal advice.
Geospatial Data Pipeline
272curiositech/some_claude_skills
Process, analyze, and visualize geospatial data at scale. Handles drone imagery, GPS tracks, GeoJSON optimization, coordinate transformations, and tile generation. Use for mapping apps, drone data processing, location-based services. Activate on "geospatial", "GIS", "PostGIS", "GeoJSON", "map tiles", "coordinate systems". NOT for simple address validation, basic distance calculations, or static map embeds.
Chatbot Analytics
272curiositech/some_claude_skills
Implement AI chatbot analytics and conversation monitoring. Use when adding conversation metrics, tracking AI usage, measuring user engagement with chat, or building conversation dashboards. Activates for AI analytics, token tracking, conversation categorization, and chat performance.
Hand Drawn Infographic Creator
272curiositech/some_claude_skills
Generate hand-drawn style diagrams and infographics for recovery education articles. Creates anatomist's notebook aesthetic visuals - brain diagrams, timelines, social comparisons, and process flows using continuous line art, semantic color coding, and margin annotations.
Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.