Data Engineering Data Pipeline

Design scalable, reliable, and cost-effective data pipelines for batch and streaming processing. Use when architecting or improving a data pipeline.

2,047 stars 295 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add rmyndharis/antigravity-skills --skill data-engineering-data-pipeline
Manual install
git clone --depth 1 https://github.com/rmyndharis/antigravity-skills.git /tmp/antigravity-skills
mkdir -p ~/.claude/skills
cp -r /tmp/antigravity-skills/skills/data-engineering-data-pipeline ~/.claude/skills/data-engineering-data-pipeline

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Data Pipeline Architecture

You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.

## Use this skill when

- Working on data pipeline architecture tasks or workflows
- Needing guidance, best practices, or checklists for data pipeline architecture

## Do not use this skill when

- The task is unrelated to data pipeline architecture
- You need a different domain or tool outside this scope

## Requirements

Take the requirements, target, and scope from the user's request.

## Core Capabilities

- Design ETL/ELT, Lambda, Kappa, and Lakehouse architectures
- Implement batch and streaming data ingestion
- Build workflow orchestration with Airflow/Prefect
- Transform data using dbt and Spark
- Manage Delta Lake/Iceberg storage with ACID transactions
- Implement data quality frameworks (Great Expectations, dbt tests)
- Monitor pipelines with CloudWatch/Prometheus/Grafana
- Optimize costs through partitioning, lifecycle policies, and compute optimization

## Instructions

### 1. Architecture Design
- Assess: sources, volume, latency requirements, targets
- Select pattern: ETL (transform before load), ELT (load then transform), Lambda (batch + speed layers), Kappa (stream-only), Lakehouse (unified)
- Design flow: sources → ingestion → processing → storage → serving
- Add observability touchpoints

### 2. Ingestion Implementation
**Batch**
- Incremental loading with watermark columns
Read the full skill on GitHub

Details

Skill name
data-engineering-data-pipeline
Path
skills/data-engineering-data-pipeline/SKILL.md
License
MIT
Repo updated
7 November 2026

FAQ

What does the Data Engineering Data Pipeline skill do?

Design scalable, reliable, and cost-effective data pipelines for batch and streaming processing. Use when architecting or improving a data pipeline.

How do I install the Data Engineering Data Pipeline skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add rmyndharis/antigravity-skills --skill data-engineering-data-pipeline. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Data Engineering Data Pipeline popular?

It lives in rmyndharis/antigravity-skills, which has 2,047 GitHub stars and 295 forks.

Related skills

Airflow Dag Patterns

2k

rmyndharis/antigravity-skills

Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.

Testing & QA295

Agent Orchestration Improve Agent

2k

rmyndharis/antigravity-skills

Systematic improvement of existing agents through performance analysis, prompt engineering, and continuous iteration.

AI, LLMs & Agents295

Agent Orchestration Multi Agent Optimize

2k

rmyndharis/antigravity-skills

Optimize multi-agent systems with coordinated profiling, workload distribution, and cost-aware orchestration. Use when improving agent performance, throughput, or reliability.

AI, LLMs & Agents295

Autopilot Charter

1.6k

mohitagw15856/pm-claude-skills

Decide which of your recurring rituals to put on autopilot — and which to keep manual. Use when asked what to automate, how to set up recurring AI runs, which reports or briefings could run on a schedule, or to design an automation charter for a team. Produces a ritual inventory with automate/assist/keep-manual calls, guardrails per ritual, and a rollout order.

Data & AnalyticsIn 1 list281

Browser Search

548

Johell1NS/browser-search

Multi-engine web search (SearXNG) + browsing/scraping (Camofox, CloakBrowser). Use whenever you need to do web research.

Data & AnalyticsIn 1 list42

Data Analysis Standard

1.6k

mohitagw15856/pm-claude-skills

Estructura un análisis de datos de producto, análisis profundo de métricas, análisis de embudo o estudio de cohortes. Úsalo cuando te pidan analizar métricas de producto, investigar una caída en conversión, explicar un cambio de datos a stakeholders o encontrar la causa raíz de un movimiento de métrica. Produce un análisis estructurado con pregunta, causa raíz, nivel de confianza y acción recomendada.

Data & AnalyticsIn 1 list281

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.