Data Etl Pipeline

Use when the user asks to orchestrate a daily ETL with Airflow and dbt, ETL vs ELT, data ingestion, incremental loads, Airflow 3 DAG design, retries and dead-letter handling. Produces pipeline architecture, DAG and transformation design with validation and observability. Not for real-time CDC (use data-streaming) or wiring dbt into Airflow in depth (use data-airflow-dbt-orchestration).

45 stars 6 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add j4flmao/agent-skills --skill data-etl-pipeline
Manual install
git clone --depth 1 https://github.com/j4flmao/agent-skills.git /tmp/agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/agent-skills/skills/data/etl-pipeline ~/.claude/skills/data-etl-pipeline

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Data ETL Pipeline

## Purpose
Design reliable ETL/ELT pipelines with Airflow orchestration, dbt transformations, incremental strategies, error handling, and data validation.

## Agent Protocol

### Trigger
Exact user phrases: "ETL", "ELT", "data pipeline", "Airflow", "dbt", "data transformation", "data ingestion", "batch processing", "pipeline orchestration", "incremental load", "data pipeline design", "DAG", "data workflow", "extract load transform".

### Input Context
Before activating, verify:
- Source systems (databases, APIs, files, streams)
- Target warehouse (Snowflake, BigQuery, Redshift, DuckDB)
- Volume and frequency (daily/hourly batch, CDC, real-time)
- Orchestration preference (Airflow, Dagster, Prefect)
- Transformation tool (dbt, custom SQL, Spark)
- Data volume and growth rate
- SLAs for data freshness and availability
- Existing monitoring and alerting infrastructure

### Output Artifact
ETL pipeline design with DAG structure, transformation config, error handling as YAML and SQL.

### Response Format
```python
# Airflow DAG skeleton
# Task definitions
```
```yaml
# dbt model config
# Incremental strategy
```
```sql
# Transformation query template
```

No preamble. No postamble. No explanations. No filler/hedging/transitions. Compress output — why use many token when few do trick.

### Completion Criteria
- [ ] Pipeline architecture diagram defined (sources → staging → warehouse)
- [ ] Airflow DAG structure with task dependencies and retries
Read the full skill on GitHub

Details

Skill name
data-etl-pipeline
Path
skills/data/etl-pipeline/SKILL.md
License
MIT
Repo updated
17 November 2026

FAQ

What does the Data Etl Pipeline skill do?

Use when the user asks to orchestrate a daily ETL with Airflow and dbt, ETL vs ELT, data ingestion, incremental loads, Airflow 3 DAG design, retries and dead-letter handling. Produces pipeline architecture, DAG and transformation design with validation and observability. Not for real-time CDC (use data-streaming) or wiring dbt into Airflow in depth (use data-airflow-dbt-orchestration).

How do I install the Data Etl Pipeline skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add j4flmao/agent-skills --skill data-etl-pipeline. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Data Etl Pipeline popular?

It lives in j4flmao/agent-skills, which has 45 GitHub stars and 6 forks.

Related skills

Agent Observability

45

j4flmao/agent-skills

Use when the user asks about agent observability, tracing agent reasoning, debugging an agent loop, OpenTelemetry GenAI spans for LLM calls, LLM cost tracking per agent, agent audit logs or hallucination/eval monitoring in production. Produces instrumentation config, trace schema, dashboards, alerts and cost attribution. Not for generic service tracing without LLMs (use devops-opentelemetry).

DevOps & Cloud

AI Agent Architectures

45

j4flmao/agent-skills

Use when the user asks to build an AI agent, design an agent that calls tools and keeps memory, write an agent system prompt or pick an agent framework (Claude Agent SDK, OpenAI Agents SDK, LangGraph). Produces an architecture brief covering loop, tools, memory, guardrails and evals, routing depth to child skills. Not for workflow patterns (use ai-agentic-workflows).

AI, LLMs & Agents

AI Agent Core Loop

45

j4flmao/agent-skills

Use when the user asks about the agent loop, ReAct vs plan-and-solve, re-planning, Reflexion, an agent stuck in a loop, max iterations or termination conditions. Produces a loop spec or patch with state, success predicate, caps and no-progress detection. Not for overall agent design (use ai-agent-architectures) or tool schemas (use tool-calling-principles).

AI, LLMs & Agents

Data Pipeline Cicd

45

j4flmao/agent-skills

Use when the user asks about CI/CD for data pipelines: dbt Slim CI with state:modified, SQLFluff linting, dbt Cloud jobs, environment promotion or schema change management. Produces CI workflows, lint config and promotion gates. Not for general CI platforms (use github-actions) or test design (use data-testing).

Data & Analytics

Data Bi Tools

45

j4flmao/agent-skills

Use when the user asks about BI dashboards in Metabase, Superset, Looker/LookML, Tableau or Power BI, KPI dashboards, a BI semantic layer or embedded analytics. Produces tool selection, semantic model, dashboard layout and row-level access setup. Not for dbt metric definitions (use data-science-analytics-engineering) or warehouse design (use data-data-warehouse).

Data & Analytics

Data Reverse Etl

45

j4flmao/agent-skills

Use when the user asks about reverse ETL with Hightouch or Census, syncing warehouse data to Salesforce or HubSpot, audience export, operational analytics or data activation. Produces sync design with incremental diffing, idempotent upserts and API rate-limit handling. Not for ingestion into the warehouse (use data-etl-pipeline).

Data & Analytics

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.