Stream Pipeline Designer

Use when designing or reviewing a real-time data pipeline with Flink, Spark Structured Streaming, Kafka, or CDC. Advises on time semantics, watermarks, state management, exactly-once guarantees, throughput, backpressure, and real-time to batch alignment. 中文触发词:实时数仓、Flink、Kafka、实时管道、背压、Watermark、CDC。

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add shisuidata/ss-data-skills --skill stream-pipeline-designer
Manual install
git clone --depth 1 https://github.com/shisuidata/ss-data-skills.git /tmp/ss-data-skills
mkdir -p ~/.claude/skills
cp -r /tmp/ss-data-skills/skills/stream-pipeline-designer ~/.claude/skills/stream-pipeline-designer

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Stream Pipeline Designer

## 目标

帮助用户设计或审查基于 Flink、Spark Structured Streaming、Kafka、CDC 的实时数据管道,重点关注数据一致性、状态管理、延迟保证和实时/批次对齐。

这个 Skill 覆盖实时管道设计决策,不负责替代具体代码审查(使用 `sql-reviewer` 处理 Flink SQL 逻辑)。

## 使用场景

使用这个 Skill,当用户需要:

- 设计 Kafka + Flink 实时数仓管道
- 选择时间语义:Event Time、Processing Time 还是 Ingestion Time
- 评估 State 大小和 RocksDB 配置
- 设计 Watermark 策略和乱序容忍窗口
- 判断是否需要 Exactly-once 保证以及实现成本
- 设计 ODS 实时层 Kafka Topic 结构
- 评估实时和离线数仓数据对齐方案
- 分析背压问题的定位和缓解策略
- 选择计算引擎,设计 CDC 入湖或入仓链路

典型输入:

```text
我们要用 Flink 实时消费订单支付事件,计算渠道实时 GMV,请帮我设计管道方案。
```

## 不适用场景

不要用这个 Skill 直接完成:

- Flink SQL / Spark SQL 逻辑审查,应使用 `sql-reviewer`
- 批处理数仓表设计,应使用 `table-design-advisor`
- 实时数据质量规则设计,应使用 `data-quality-rule-generator`
- 工具接入配置,应使用 `data-tool-integration-planner`

如果用户没有描述数据量级、延迟要求和一致性需求,先列出待确认问题,不要直接给出配置参数。

## 输入信息

最少输入:

- 数据来源(Kafka Topic 或上游系统)
- 计算目标(实时指标或流处理结果)
- 使用框架

可选输入:

- 消息量级(QPS 或条/天)
- 延迟要求(端到端 P99)
- 一致性要求:At-least-once 或 Exactly-once
- 时间字段:Event Time、Kafka 时间还是系统时间
- 乱序容忍范围
- 下游系统:Kafka、ClickHouse、Redis、API
- Checkpoint 周期和状态后端
- 是否需要与离线数仓对齐

## 上下文建议

推荐使用 [实时管道上下文模板](../../context/templates/stream-context.md)。需要接入新数据源时,补充 [工具集成上下文模板](../../context/templates/tool-integration-context.md)。

最有价值的上下文是:消息量级、延迟要求、Key 基数、一致性需求和下游系统写入能力。

## 设计框架

1. 明确计算目标和输出要求。
2. 确认时间语义:Event Time 还是 Processing Time,对应业务场景是什么。
3. 设计 Watermark 策略:乱序窗口大小、迟到数据处理方式。
4. 评估 State 规模:窗口大小、Key 基数、保存时间、State 后端选择。
5. 设计 Exactly-once 方案(如需要):两阶段提交、幂等写入或 At-least-once + 下游去重。
Read the full skill on GitHub

Details

Skill name
stream-pipeline-designer
Path
skills/stream-pipeline-designer/SKILL.md
License
MIT
Repo updated
8 November 2026

FAQ

What does the Stream Pipeline Designer skill do?

Use when designing or reviewing a real-time data pipeline with Flink, Spark Structured Streaming, Kafka, or CDC. Advises on time semantics, watermarks, state management, exactly-once guarantees, throughput, backpressure, and real-time to batch alignment. 中文触发词:实时数仓、Flink、Kafka、实时管道、背压、Watermark、CDC。

How do I install the Stream Pipeline Designer skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add shisuidata/ss-data-skills --skill stream-pipeline-designer. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Stream Pipeline Designer popular?

It lives in shisuidata/ss-data-skills, which has 9 GitHub stars and 0 forks.

Related skills

Ab Test Analysis

9

shisuidata/ss-data-skills

Use when designing, checking, or interpreting A/B tests and experiments. Computes significance, confidence intervals, sample ratio mismatch, power, CUPED, and sample size with a bundled zero-dependency script, then judges validity risks, guardrails, and ship/iterate/rollback decisions and renders an HTML experiment report. 中文触发词:A/B 实验、AB 测试、实验解读、显著性、样本量计算、能不能全量。

Testing & QA

Business Root Cause Analysis

9

shisuidata/ss-data-skills

Use when diagnosing business metric changes or anomalies such as GMV decline, conversion drop, retention loss, churn increase, revenue change, lead decline, or active user fluctuation with a structured root cause analysis plan. 中文触发词:指标归因、为什么下降、异常排查、GMV 下滑、指标波动原因。

Marketing, Sales & Business

Daily Report Writer

9

shisuidata/ss-data-skills

Use when writing a concise daily report, daily worklog, standup update, shift handoff, or end-of-day summary for data analysis, data development, BI, governance, or operations work. 中文触发词:日报、日终同步、站会发言、值班交接。

Writing & Content

Public Dashboard Design

5

nekoai-lab/claude-skills

公共ダッシュボードや説明責任が求められる資料に対して、誤解防止・比較しやすさ・信頼性確保を優先した設計ルールを適用する

Data & Analytics

Exploratory Data Analysis

9

shisuidata/ss-data-skills

Use when exploring a dataset before formal analysis, modeling, reporting, or dashboarding. Runs a bundled zero-dependency data health check on CSV files that infers field types, detects encoding, pseudo-nulls, numbers stored as text, leading-zero IDs, duplicate keys, partition-vs-event-time drift, missing dates, and orphan join keys, ranks issues P0/P1/P2, and renders an HTML report; then interprets grain, usability, and next analysis directions. 中文触发词:数据探索、EDA、数据体检、数据摸底、拿到一份新数据。

Data & Analytics

Wp Analytics

9

morrealev/wordpress-manager

This skill should be used when the user asks about "analytics", "GA4", "Google Analytics", "Plausible", "Core Web Vitals", "CWV", "PageSpeed", "page speed", "LCP", "CLS", "INP", "TTFB", "FCP", "traffic report", "traffic analysis", "site performance metrics", "web vitals", "real user metrics", or mentions monitoring WordPress site analytics and performance data.

Data & Analytics

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.