Data Distributed Compute

Use when the user asks about Spark vs Dask vs Ray, Spark executor sizing, shuffle and skew, YARN or Kubernetes resource management, data locality or speculative execution. Produces engine choice, cluster topology and tuning settings. Not for Hive/Spark SQL query tuning (use data-batch-processing) or GPU training (use distributed-training).

45 stars 6 forksDatabases

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add j4flmao/agent-skills --skill data-distributed-compute
Manual install
git clone --depth 1 https://github.com/j4flmao/agent-skills.git /tmp/agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/agent-skills/skills/data/distributed-compute ~/.claude/skills/data-distributed-compute

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Data Distributed Compute

## Purpose
Design and tune distributed compute systems for large-scale data processing. Select the right framework (Spark, Dask, Ray, MapReduce), configure YARN/K8s resource management, optimize shuffle and data locality, and tune executors for throughput.

## Agent Protocol

### Trigger
Exact user phrases: "Hadoop MapReduce", "Spark", "Dask", "Ray", "YARN", "cluster computing", "resource manager", "shuffle", "data locality", "executor", "worker", "task scheduling", "distributed compute", "cluster mode", "dynamic allocation", "speculative execution".

### Input Context
Before activating, verify:
- Compute framework preference (Spark, Dask, Ray, MapReduce)
- Data size and shape (TB per run, row counts, join complexity)
- Cluster size and resource per node (cores, memory, network)
- Workload type (batch ETL, ML training, real-time inference, iterative algorithms)
- Storage backend (HDFS, S3, local SSD)
- Scheduling layer (YARN, K8s, standalone)

### Output Artifact
Distributed compute architecture with framework selection, cluster configuration, and tuning parameters.

### Response Format
```
Compute Framework: {Spark | Dask | Ray | MapReduce}
Cluster Mode: {YARN | K8s | Standalone | Slurm}
Execution Model: {driver-executor | scheduler-worker | GCS}
Resource: {N executors x M cores x G memory}
Shuffle: {sort-based | hash-based | external}
Locality: {PROCESS_LOCAL | NODE_LOCAL | RACK_LOCAL | ANY}
```
Read the full skill on GitHub

Details

Skill name
data-distributed-compute
Path
skills/data/distributed-compute/SKILL.md
License
MIT
Repo updated
17 November 2026

FAQ

What does the Data Distributed Compute skill do?

Use when the user asks about Spark vs Dask vs Ray, Spark executor sizing, shuffle and skew, YARN or Kubernetes resource management, data locality or speculative execution. Produces engine choice, cluster topology and tuning settings. Not for Hive/Spark SQL query tuning (use data-batch-processing) or GPU training (use distributed-training).

How do I install the Data Distributed Compute skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add j4flmao/agent-skills --skill data-distributed-compute. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Data Distributed Compute popular?

It lives in j4flmao/agent-skills, which has 45 GitHub stars and 6 forks.

Related skills

Agent Observability

45

j4flmao/agent-skills

Use when the user asks about agent observability, tracing agent reasoning, debugging an agent loop, OpenTelemetry GenAI spans for LLM calls, LLM cost tracking per agent, agent audit logs or hallucination/eval monitoring in production. Produces instrumentation config, trace schema, dashboards, alerts and cost attribution. Not for generic service tracing without LLMs (use devops-opentelemetry).

DevOps & Cloud

AI Agent Architectures

45

j4flmao/agent-skills

Use when the user asks to build an AI agent, design an agent that calls tools and keeps memory, write an agent system prompt or pick an agent framework (Claude Agent SDK, OpenAI Agents SDK, LangGraph). Produces an architecture brief covering loop, tools, memory, guardrails and evals, routing depth to child skills. Not for workflow patterns (use ai-agentic-workflows).

AI, LLMs & Agents

AI Agent Core Loop

45

j4flmao/agent-skills

Use when the user asks about the agent loop, ReAct vs plan-and-solve, re-planning, Reflexion, an agent stuck in a loop, max iterations or termination conditions. Produces a loop spec or patch with state, success predicate, caps and no-progress detection. Not for overall agent design (use ai-agent-architectures) or tool schemas (use tool-calling-principles).

AI, LLMs & Agents

Data Distributed Storage

45

j4flmao/agent-skills

Use when the user asks about HDFS, S3, ADLS, GCS, MinIO or Ceph for data workloads, replication and erasure coding, small-file problems, storage tiering or lifecycle policies. Produces storage backend choice, layout and durability and cost settings. Not for database storage engines (use database-internals-mechanics) or file formats (use data-formats).

Databases

Data Data Warehouse

45

j4flmao/agent-skills

Use when the user asks about a data warehouse on Snowflake, BigQuery or Redshift, clustering and partitioning, materialized views, OLAP performance or warehouse cost. Produces platform-specific table design and optimization steps. Not for Kimball grain and SCD design (use data-dimensional-modeling) or pipeline orchestration (use data-etl-pipeline).

Databases

Data Graph Database

45

j4flmao/agent-skills

Use when the user asks about Neo4j and Cypher, Amazon Neptune, Gremlin, JanusGraph, SPARQL/RDF stores, graph traversal performance or knowledge graph storage. Produces graph model, queries and indexing and scaling choices. Not for relational/graph schema theory (use data-modeling) or GraphRAG prompting (use prompt-rag-query-design).

Databases

Browse the most popular Claude skills, all databases skills, or write your own with the skills generator.