Spark Optimization
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
Install
npx skills add rmyndharis/antigravity-skills --skill spark-optimizationgit clone --depth 1 https://github.com/rmyndharis/antigravity-skills.git /tmp/antigravity-skills
mkdir -p ~/.claude/skills
cp -r /tmp/antigravity-skills/skills/spark-optimization ~/.claude/skills/spark-optimizationPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Apache Spark Optimization
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
## Do not use this skill when
- The task is unrelated to apache spark optimization
- You need a different domain or tool outside this scope
## Instructions
- Clarify goals, constraints, and required inputs.
- Apply relevant best practices and validate outcomes.
- Provide actionable steps and verification.
## Use this skill when
- Optimizing slow Spark jobs
- Tuning memory and executor configuration
- Implementing efficient partitioning strategies
- Debugging Spark performance issues
- Scaling Spark pipelines for large datasets
- Reducing shuffle and data skew
## Core Concepts
### 1. Spark Execution Model
```
Driver Program
↓
Job (triggered by action)
↓
Stages (separated by shuffles)
↓
Tasks (one per partition)
```
### 2. Key Performance Factors
| Factor | Impact | Solution |
|--------|--------|----------|
| **Shuffle** | Network I/O, disk I/O | Minimize wide transformations |
| **Data Skew** | Uneven task duration | Salting, broadcast joins |
| **Serialization** | CPU overhead | Use Kryo, columnar formats |
| **Memory** | GC pressure, spills | Tune executor memory |
| **Partitions** | Parallelism | Right-size partitions |
## Quick StartDetails
- Skill name
spark-optimization- Repository
- rmyndharis/antigravity-skills
- Path
skills/spark-optimization/SKILL.md- License
- MIT
- Repo updated
- 7 November 2026
FAQ
What does the Spark Optimization skill do?
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
How do I install the Spark Optimization skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add rmyndharis/antigravity-skills --skill spark-optimization. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Spark Optimization popular?
It lives in rmyndharis/antigravity-skills, which has 2,047 GitHub stars and 295 forks.
Related skills
Airflow Dag Patterns
2krmyndharis/antigravity-skills
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
Agent Orchestration Improve Agent
2krmyndharis/antigravity-skills
Systematic improvement of existing agents through performance analysis, prompt engineering, and continuous iteration.
Agent Orchestration Multi Agent Optimize
2krmyndharis/antigravity-skills
Optimize multi-agent systems with coordinated profiling, workload distribution, and cost-aware orchestration. Use when improving agent performance, throughput, or reliability.
Team Collaboration Standup Notes
2krmyndharis/antigravity-skills
Run async-first standups by generating standup notes from commit history and coordinating remote teams. Use when writing standup updates or setting up an async standup process.
Saga Orchestration
2krmyndharis/antigravity-skills
Implement saga patterns for distributed transactions and cross-aggregate workflows. Use when coordinating multi-step business processes, handling compensating transactions, or managing long-running workflows.
Team Composition Analysis
2krmyndharis/antigravity-skills
This skill should be used when the user asks to "plan team structure", "determine hiring needs", "design org chart", "calculate compensation", "plan equity allocation", or requests organizational design and headcount planning for a startup.
Browse the most popular Claude skills, all productivity & notes skills, or write your own with the skills generator.