Data Engineering Data Pipeline
Design scalable, reliable, and cost-effective data pipelines for batch and streaming processing. Use when architecting or improving a data pipeline.
Install
npx skills add rmyndharis/antigravity-skills --skill data-engineering-data-pipelinegit clone --depth 1 https://github.com/rmyndharis/antigravity-skills.git /tmp/antigravity-skills
mkdir -p ~/.claude/skills
cp -r /tmp/antigravity-skills/skills/data-engineering-data-pipeline ~/.claude/skills/data-engineering-data-pipelinePersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Data Pipeline Architecture You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing. ## Use this skill when - Working on data pipeline architecture tasks or workflows - Needing guidance, best practices, or checklists for data pipeline architecture ## Do not use this skill when - The task is unrelated to data pipeline architecture - You need a different domain or tool outside this scope ## Requirements Take the requirements, target, and scope from the user's request. ## Core Capabilities - Design ETL/ELT, Lambda, Kappa, and Lakehouse architectures - Implement batch and streaming data ingestion - Build workflow orchestration with Airflow/Prefect - Transform data using dbt and Spark - Manage Delta Lake/Iceberg storage with ACID transactions - Implement data quality frameworks (Great Expectations, dbt tests) - Monitor pipelines with CloudWatch/Prometheus/Grafana - Optimize costs through partitioning, lifecycle policies, and compute optimization ## Instructions ### 1. Architecture Design - Assess: sources, volume, latency requirements, targets - Select pattern: ETL (transform before load), ELT (load then transform), Lambda (batch + speed layers), Kappa (stream-only), Lakehouse (unified) - Design flow: sources → ingestion → processing → storage → serving - Add observability touchpoints ### 2. Ingestion Implementation **Batch** - Incremental loading with watermark columns
Details
- Skill name
data-engineering-data-pipeline- Repository
- rmyndharis/antigravity-skills
- Path
skills/data-engineering-data-pipeline/SKILL.md- License
- MIT
- Repo updated
- 7 November 2026
FAQ
What does the Data Engineering Data Pipeline skill do?
Design scalable, reliable, and cost-effective data pipelines for batch and streaming processing. Use when architecting or improving a data pipeline.
How do I install the Data Engineering Data Pipeline skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add rmyndharis/antigravity-skills --skill data-engineering-data-pipeline. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Data Engineering Data Pipeline popular?
It lives in rmyndharis/antigravity-skills, which has 2,047 GitHub stars and 295 forks.
Related skills
Airflow Dag Patterns
2krmyndharis/antigravity-skills
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
Agent Orchestration Improve Agent
2krmyndharis/antigravity-skills
Systematic improvement of existing agents through performance analysis, prompt engineering, and continuous iteration.
Agent Orchestration Multi Agent Optimize
2krmyndharis/antigravity-skills
Optimize multi-agent systems with coordinated profiling, workload distribution, and cost-aware orchestration. Use when improving agent performance, throughput, or reliability.
Autopilot Charter
1.6kmohitagw15856/pm-claude-skills
Decide which of your recurring rituals to put on autopilot — and which to keep manual. Use when asked what to automate, how to set up recurring AI runs, which reports or briefings could run on a schedule, or to design an automation charter for a team. Produces a ritual inventory with automate/assist/keep-manual calls, guardrails per ritual, and a rollout order.
Browser Search
548Johell1NS/browser-search
Multi-engine web search (SearXNG) + browsing/scraping (Camofox, CloakBrowser). Use whenever you need to do web research.
Data Analysis Standard
1.6kmohitagw15856/pm-claude-skills
Estructura un análisis de datos de producto, análisis profundo de métricas, análisis de embudo o estudio de cohortes. Úsalo cuando te pidan analizar métricas de producto, investigar una caída en conversión, explicar un cambio de datos a stakeholders o encontrar la causa raíz de un movimiento de métrica. Produce un análisis estructurado con pregunta, causa raíz, nivel de confianza y acción recomendada.
Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.