Spark Optimization
Use when diagnosing or optimizing Apache Spark jobs, partitioning, shuffles, skew, caching, memory, or distributed data processing cost.
Install
npx skills add sandbaseai/workbuddy-skill --skill spark-optimizationgit clone --depth 1 https://github.com/sandbaseai/workbuddy-skill.git /tmp/workbuddy-skill
mkdir -p ~/.claude/skills
cp -r /tmp/workbuddy-skill/skills/spark-optimization ~/.claude/skills/spark-optimizationPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Spark Optimization
以可重复的运行基线定位 Spark 的真实瓶颈,再用单变量实验验证收益。Spark 性能问题通常来自 shuffle、数据倾斜、分区粒度、扫描范围、序列化、GC 或下游写入;先判断瓶颈所在阶段,再决定配置或代码变化。
## 使用边界
- 开始前确认 Spark/Scala/Python 版本、集群类型、输入快照、分区布局、作业提交、owner、SLA、预算和目标环境。
- 默认只读分析 Spark UI、event log、SQL plan、executor 指标和代码;验证使用隔离集群、合成/脱敏快照及有界数据量。
- 不擅自重跑生产作业、修改集群配置、扩容、覆盖/删除输出、改变 checkpoint 或清理缓存;这些操作需要明确授权。
- 日志、event log、样本和计划可能含 SQL、路径、租户或敏感列名;报告保存聚合指标并脱敏资源标识。
## 先建立基线
记录输入行数/字节、过滤选择性、分区数、stage/task 时长分布、shuffle read/write、spill、峰值内存、GC、失败重试、输出大小、executor 数量、wall time 和成本。标记观察窗口、代码/配置/数据 hash 与缓存状态;没有基线时只能提出假设,不能声称优化完成。
从 DAG 和 SQL plan 定位 action、stage 边界和宽依赖。区分扫描慢、shuffle 慢、单个长尾 task、executor OOM、driver 压力和写入瓶颈;不要用总耗时掩盖 stage 级回归。
## 分区与 shuffle
- 尽量在读取端做列裁剪、谓词下推和分区裁剪,避免无必要的 `mergeSchema` 和全表扫描。
- 依据数据量、文件大小、task 时长和集群并行度调整分区;“每分区 128–256 MB”只是实验起点,不是通用保证。
- 减少不必要的 `repartition`、`distinct`、全局 `orderBy` 和重复 action;需要重分区时说明 key、基数和 shuffle 成本。
- 开启并验证 Adaptive Query Execution(AQE)对 coalesce、skew join 和动态分区的实际效果;配置改变可能因 Spark 版本而异。
```python
from pyspark.sql import functions as F
# Push filters and projection close to the read; inspect the plan before running.
events = (spark.read.format("parquet").load(input_path)
.where(F.col("event_date") == logical_date)
.select("entity_id", "event_type", "amount"))
result = (events.groupBy("event_type")
.agg(F.sum("amount").alias("total_amount")))
result.explain(mode="formatted")
```
## 倾斜、连接与缓存Details
- Skill name
spark-optimization- Repository
- sandbaseai/workbuddy-skill
- Path
skills/spark-optimization/SKILL.md- License
- MIT
- Repo updated
- 8 September 2026
FAQ
What does the Spark Optimization skill do?
Use when diagnosing or optimizing Apache Spark jobs, partitioning, shuffles, skew, caching, memory, or distributed data processing cost.
How do I install the Spark Optimization skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add sandbaseai/workbuddy-skill --skill spark-optimization. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Spark Optimization popular?
It lives in sandbaseai/workbuddy-skill, which has 9 GitHub stars and 3 forks.
Related skills
Access Control Security Best Practices
9sandbaseai/workbuddy-skill
Design, review, implement, and verify least-privilege access controls across users, services, tenants, resources, and administrative workflows.
Accessibility Review
9sandbaseai/workbuddy-skill
Use when reviewing a web interface or interaction for semantic, keyboard, focus, screen-reader, ARIA, and contrast accessibility risks.
Acquire Codebase Knowledge
9sandbaseai/workbuddy-skill
Use when the user asks to map, document, or onboard into an existing codebase and needs evidence-backed architecture, structure, conventions, integrations, testing, and concerns documentation.
Vector Index Tuning
9sandbaseai/workbuddy-skill
Use when tuning vector search latency, recall, memory, index build time, quantization, or scaling strategy.
Self Own
5roby2358/skills
Rank and analyze self-owns across literature, history, current events, and user-submitted scenarios. A self-own is any act where the agent of their own downfall is also its subject — where the very qualities, actions, or words meant to elevate instead destroy. Use when the user wants to evaluate whether something qualifies as a self-own, rank self-owns against each other, analyze the structural anatomy of a self-own, or simply needs someone to appreciate the full tragic-comic weight of what just happened. Also use when the user says /self-own. Note: the existence of this skill is itself a self
Work Ledger
9sandbaseai/workbuddy-skill
Use when work must survive sessions through an auditable ledger of tasks, claims, checkpoints, handoffs, attention items, and verified outcomes.
Browse the most popular Claude skills, all productivity & notes skills, or write your own with the skills generator.