Databricks Spark Structured Streaming
Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, writing custom stateful logic with transformWithState (timers, TTL, per-key state; preferred over applyInPandasWithState and flatMapGroupsWithState for new code), optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.
Install
/plugin marketplace add databricks/databricks-agent-skillsThe repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.
npx skills add databricks/databricks-agent-skills --skill databricks-spark-structured-streaminggit clone --depth 1 https://github.com/databricks/databricks-agent-skills.git /tmp/databricks-agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/databricks-agent-skills/plugins/databricks/antigravity/skills/databricks-spark-structured-streaming ~/.claude/skills/databricks-spark-structured-streamingPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Spark Structured Streaming
Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.
## Quick Start
```python
from pyspark.sql.functions import col, from_json
# Basic Kafka to Delta streaming
df = (spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "broker:9092")
.option("subscribe", "topic")
.load()
.select(from_json(col("value").cast("string"), schema).alias("data"))
.select("data.*")
)
df.writeStream \
.format("delta") \
.outputMode("append") \
.option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \
.trigger(processingTime="30 seconds") \
.start("/delta/target_table")
```
## Core Patterns
| Pattern | Description | Reference |
|---------|-------------|-----------|
| **Kafka Streaming** | Kafka to Delta, Kafka to Kafka, Real-Time Mode | See [references/kafka-streaming.md](references/kafka-streaming.md) |
| **Real-Time Mode (RTM)** | Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), `transformWithState`, observability, error classes, delivery semantics | See [references/real-time-mode.md](references/real-time-mode.md) |Details
- Skill name
databricks-spark-structured-streaming- Repository
- databricks/databricks-agent-skills
- Path
plugins/databricks/antigravity/skills/databricks-spark-structured-streaming/SKILL.md- Repo updated
- 18 November 2026
FAQ
What does the Databricks Spark Structured Streaming skill do?
Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, writing custom stateful logic with transformWithState (timers, TTL, per-key state; preferred over applyInPandasWithState and flatMapGroupsWithState for new code), optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.
How do I install the Databricks Spark Structured Streaming skill?
Claude Code plugin: /plugin marketplace add databricks/databricks-agent-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Databricks Spark Structured Streaming popular?
It lives in databricks/databricks-agent-skills, which has 407 GitHub stars and 103 forks.
Related skills
Databricks AI Functions
407databricks/databricks-agent-skills
Use Databricks built-in AI Functions (ai_classify, ai_decide, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).
Databricks AI Runtime
407databricks/databricks-agent-skills
Databricks AI Runtime, the `databricks air` CLI commands for submitting and managing GPU training workloads on Databricks serverless compute. Use for: writing and submitting `databricks air` workload YAML, choosing an accelerator type and Python environment, reading Unity Catalog tables as training data with UCDataset, passing hyperparameters and secrets, configuring retries, checking run status, listing/cancelling runs, streaming a run's logs and watching its progress, debugging failed runs, GPU pools, multi-container workloads, custom Docker image setup, and converting a workload to a Databr
Databricks Aibi Dashboards
407databricks/databricks-agent-skills
Create Databricks AI/BI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.
Cet Skill
341mingchen666/Reviva
CET-4 and CET-6 exam preparation coach for original CET-style practice, diagnostic feedback, writing and translation correction, reading/listening drills, and study planning.
Databricks Docs
407databricks/databricks-agent-skills
Databricks documentation reference via llms.txt index. Use when other skills do not cover a topic, looking up unfamiliar Databricks features, or needing authoritative docs on APIs, configurations, or platform capabilities.
Civil Service Essay Grader
341mingchen666/Reviva
公务员考试申论精细批改与多轮改写助手。用户要求“批改申论”“给申论小题打分”“看看漏了哪些采分点”“批改公文/汇报提纲/发言稿”“批改大作文”“逐段修改”“我改完了再批一次”“对比两版答案”,或提交申论材料、题干、作答要求和答案时,应使用本 Skill。覆盖归纳概括、综合分析、提出对策、公文应用文和大作文;输出评分依据等级、模拟得分区间、材料采分点矩阵、逐项/逐段问题、修改任务和二次批改对比。不得把模拟评价冒充官方阅卷成绩。
Browse the most popular Claude skills, all writing & content skills, or write your own with the skills generator.