Data Engineering Data Pipeline

You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.

514 stars 146 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add Dokhacgiakhoa/Agent-Skills-4-Vibe-Coding-CLI --skill data-engineering-data-pipeline
Manual install
git clone --depth 1 https://github.com/Dokhacgiakhoa/Agent-Skills-4-Vibe-Coding-CLI.git /tmp/Agent-Skills-4-Vibe-Coding-CLI
mkdir -p ~/.claude/skills
cp -r /tmp/Agent-Skills-4-Vibe-Coding-CLI/.agent/skills/data-engineering-data-pipeline ~/.claude/skills/data-engineering-data-pipeline

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Data Pipeline Architecture

You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.

## Use this skill when

- Working on data pipeline architecture tasks or workflows
- Needing guidance, best practices, or checklists for data pipeline architecture

## Do not use this skill when

- The task is unrelated to data pipeline architecture
- You need a different domain or tool outside this scope

## Requirements

$ARGUMENTS

## Core Capabilities

- Design ETL/ELT, Lambda, Kappa, and Lakehouse architectures
- Implement batch and streaming data ingestion
- Build workflow orchestration with Airflow/Prefect
- Transform data using dbt and Spark
- Manage Delta Lake/Iceberg storage with ACID transactions
- Implement data quality frameworks (Great Expectations, dbt tests)
- Monitor pipelines with CloudWatch/Prometheus/Grafana
- Optimize costs through partitioning, lifecycle policies, and compute optimization

## Instructions

## 🧠 Knowledge Modules (Fractal Skills)

### 1. [1. Architecture Design](./sub-skills/1-architecture-design.md)
### 2. [2. Ingestion Implementation](./sub-skills/2-ingestion-implementation.md)
### 3. [3. Orchestration](./sub-skills/3-orchestration.md)
### 4. [4. Transformation with dbt](./sub-skills/4-transformation-with-dbt.md)
### 5. [5. Data Quality Framework](./sub-skills/5-data-quality-framework.md)
### 6. [6. Storage Strategy](./sub-skills/6-storage-strategy.md)
Read the full skill on GitHub

Details

Skill name
data-engineering-data-pipeline
Path
.agent/skills/data-engineering-data-pipeline/SKILL.md
Repo updated
11 June 2026

FAQ

What does the Data Engineering Data Pipeline skill do?

You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.

How do I install the Data Engineering Data Pipeline skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add Dokhacgiakhoa/Agent-Skills-4-Vibe-Coding-CLI --skill data-engineering-data-pipeline. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Data Engineering Data Pipeline popular?

It lives in Dokhacgiakhoa/Agent-Skills-4-Vibe-Coding-CLI, which has 514 GitHub stars and 146 forks.

Related skills

Address Github Comments

514

Dokhacgiakhoa/Agent-Skills-4-Vibe-Coding-CLI

Use when you need to address review or issue comments on an open GitHub Pull Request using the gh CLI.

Git & Code Review146

Agent Memory MCP

514

Dokhacgiakhoa/Agent-Skills-4-Vibe-Coding-CLI

A hybrid memory system that provides persistent, searchable knowledge management for AI agents (Architecture, Patterns, Decisions).

AI, LLMs & Agents146

API Documentation Generator

514

Dokhacgiakhoa/Agent-Skills-4-Vibe-Coding-CLI

Generate comprehensive, developer-friendly API documentation from code, including endpoints, parameters, examples, and best practices

Software Development146

Firecrawl Scraper

514

Dokhacgiakhoa/Agent-Skills-4-Vibe-Coding-CLI

Deep web scraping, screenshots, PDF parsing, and website crawling using Firecrawl API

Data & Analytics146

Analytics Tracking

514

Dokhacgiakhoa/Agent-Skills-4-Vibe-Coding-CLI

Design, audit, and improve analytics tracking systems that produce reliable, decision-ready data. Use when the user wants to set up, fix, or evaluate analytics tracking (GA4, GTM, product analytics, events, conversions, UTMs). This skill focuses on measurement strategy, signal quality, and validation— not just firing events.

Data & Analytics146

Data Analysis

298

superhero-tech/superhero-skills

Guide a full data-analyst workflow from business question to recommendation. Use when the user says "analyze this data/CSV", wants EDA on a fresh dataset, asks to find patterns in this dataset (segmentation, cohorts, correlations), or wants SQL queries over files or databases via DuckDB.

Data & Analytics25

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.