Spark Optimization

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

2,047 stars 295 forksProductivity & Notes

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add rmyndharis/antigravity-skills --skill spark-optimization
Manual install
git clone --depth 1 https://github.com/rmyndharis/antigravity-skills.git /tmp/antigravity-skills
mkdir -p ~/.claude/skills
cp -r /tmp/antigravity-skills/skills/spark-optimization ~/.claude/skills/spark-optimization

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Apache Spark Optimization

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

## Do not use this skill when

- The task is unrelated to apache spark optimization
- You need a different domain or tool outside this scope

## Instructions

- Clarify goals, constraints, and required inputs.
- Apply relevant best practices and validate outcomes.
- Provide actionable steps and verification.

## Use this skill when

- Optimizing slow Spark jobs
- Tuning memory and executor configuration
- Implementing efficient partitioning strategies
- Debugging Spark performance issues
- Scaling Spark pipelines for large datasets
- Reducing shuffle and data skew

## Core Concepts

### 1. Spark Execution Model

```
Driver Program
    ↓
Job (triggered by action)
    ↓
Stages (separated by shuffles)
    ↓
Tasks (one per partition)
```

### 2. Key Performance Factors

| Factor | Impact | Solution |
|--------|--------|----------|
| **Shuffle** | Network I/O, disk I/O | Minimize wide transformations |
| **Data Skew** | Uneven task duration | Salting, broadcast joins |
| **Serialization** | CPU overhead | Use Kryo, columnar formats |
| **Memory** | GC pressure, spills | Tune executor memory |
| **Partitions** | Parallelism | Right-size partitions |

## Quick Start
Read the full skill on GitHub

Details

Skill name
spark-optimization
Path
skills/spark-optimization/SKILL.md
License
MIT
Repo updated
7 November 2026

FAQ

What does the Spark Optimization skill do?

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

How do I install the Spark Optimization skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add rmyndharis/antigravity-skills --skill spark-optimization. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Spark Optimization popular?

It lives in rmyndharis/antigravity-skills, which has 2,047 GitHub stars and 295 forks.

Related skills

Airflow Dag Patterns

2k

rmyndharis/antigravity-skills

Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.

Testing & QA295

Agent Orchestration Improve Agent

2k

rmyndharis/antigravity-skills

Systematic improvement of existing agents through performance analysis, prompt engineering, and continuous iteration.

AI, LLMs & Agents295

Agent Orchestration Multi Agent Optimize

2k

rmyndharis/antigravity-skills

Optimize multi-agent systems with coordinated profiling, workload distribution, and cost-aware orchestration. Use when improving agent performance, throughput, or reliability.

AI, LLMs & Agents295

Team Collaboration Standup Notes

2k

rmyndharis/antigravity-skills

Run async-first standups by generating standup notes from commit history and coordinating remote teams. Use when writing standup updates or setting up an async standup process.

Productivity & Notes295

Saga Orchestration

2k

rmyndharis/antigravity-skills

Implement saga patterns for distributed transactions and cross-aggregate workflows. Use when coordinating multi-step business processes, handling compensating transactions, or managing long-running workflows.

Productivity & Notes295

Team Composition Analysis

2k

rmyndharis/antigravity-skills

This skill should be used when the user asks to "plan team structure", "determine hiring needs", "design org chart", "calculate compensation", "plan equity allocation", or requests organizational design and headcount planning for a startup.

Productivity & Notes295

Browse the most popular Claude skills, all productivity & notes skills, or write your own with the skills generator.