Data Batch Processing

Use when the user asks about Hive, HiveQL, Spark SQL batch jobs, Hive metastore, partitioning and bucketing, ORC vs Parquet for batch, Catalyst/AQE tuning or dynamic partition pruning. Produces batch job designs with table layout, file format and query tuning steps. Not for choosing Spark vs Dask vs Ray (use data-distributed-compute) or real-time pipelines (use data-streaming).

45 stars 6 forksDatabases

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add j4flmao/agent-skills --skill data-batch-processing
Manual install
git clone --depth 1 https://github.com/j4flmao/agent-skills.git /tmp/agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/agent-skills/skills/data/batch-processing ~/.claude/skills/data-batch-processing

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Data Batch Processing

## Purpose
Design efficient batch processing architectures using Hive, Spark SQL, and optimized file formats. Master partitioning, bucketing, Catalyst optimizer tuning, vectorized reads, file format selection, and dynamic partition pruning for large-scale analytical workloads.

## Agent Protocol

### Trigger
Exact user phrases: "batch processing", "Hive", "Spark SQL", "Pig", "HQL", "Hive partition", "Spark partition", "bucketing", "query optimization", "ORC", "Parquet", "Avro", "vectorized read", "dynamic partition pruning", "Catalyst optimizer", "Tungsten", "Hive metastore", "reduce tasks".

### Input Context
Before activating, verify:
- Query engine (Hive on Tez, Hive on MR, Spark SQL, Presto, Trino)
- File format currently used (text, Parquet, ORC, Avro, JSON)
- Table volume (row count, size in TB, partition count)
- Partition column(s) and cardinality
- Common query patterns (full scan, filtered, aggregated, joined)
- Cluster resources (cores, memory, number of nodes)

### Output Artifact
Batch processing configuration with engine selection, partition strategy, and optimization parameters.

### Response Format
```
Engine: {Hive on Tez | Spark SQL | Presto | Trino}
File Format: {Parquet | ORC | Avro}
Partition: {column: type, granularity: daily/hourly/monthly}
Bucketing: {column: cluster count}
Optimizations: {vectorized, CBO, DPP, broadcast join}
Tuning: {executor/container config, parallelism}
```
Read the full skill on GitHub

Details

Skill name
data-batch-processing
Path
skills/data/batch-processing/SKILL.md
License
MIT
Repo updated
17 November 2026

FAQ

What does the Data Batch Processing skill do?

Use when the user asks about Hive, HiveQL, Spark SQL batch jobs, Hive metastore, partitioning and bucketing, ORC vs Parquet for batch, Catalyst/AQE tuning or dynamic partition pruning. Produces batch job designs with table layout, file format and query tuning steps. Not for choosing Spark vs Dask vs Ray (use data-distributed-compute) or real-time pipelines (use data-streaming).

How do I install the Data Batch Processing skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add j4flmao/agent-skills --skill data-batch-processing. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Data Batch Processing popular?

It lives in j4flmao/agent-skills, which has 45 GitHub stars and 6 forks.

Related skills

Agent Observability

45

j4flmao/agent-skills

Use when the user asks about agent observability, tracing agent reasoning, debugging an agent loop, OpenTelemetry GenAI spans for LLM calls, LLM cost tracking per agent, agent audit logs or hallucination/eval monitoring in production. Produces instrumentation config, trace schema, dashboards, alerts and cost attribution. Not for generic service tracing without LLMs (use devops-opentelemetry).

DevOps & Cloud

AI Agent Architectures

45

j4flmao/agent-skills

Use when the user asks to build an AI agent, design an agent that calls tools and keeps memory, write an agent system prompt or pick an agent framework (Claude Agent SDK, OpenAI Agents SDK, LangGraph). Produces an architecture brief covering loop, tools, memory, guardrails and evals, routing depth to child skills. Not for workflow patterns (use ai-agentic-workflows).

AI, LLMs & Agents

AI Agent Core Loop

45

j4flmao/agent-skills

Use when the user asks about the agent loop, ReAct vs plan-and-solve, re-planning, Reflexion, an agent stuck in a loop, max iterations or termination conditions. Produces a loop spec or patch with state, success predicate, caps and no-progress detection. Not for overall agent design (use ai-agent-architectures) or tool schemas (use tool-calling-principles).

AI, LLMs & Agents

Data Clean Room

45

j4flmao/agent-skills

Use when the user asks about a data clean room, AWS Clean Rooms, Snowflake Data Clean Rooms, private set intersection (PSI), privacy-preserving joins, differential privacy budgets or secure multi-party computation for data collaboration. Produces a clean-room design with analysis rules, PET choice and privacy budget. Not for in-house masking (use backend-data-masking).

Databases

Data Airflow DBT Orchestration

45

j4flmao/agent-skills

Use when the user asks to run dbt from Airflow, Astronomer Cosmos, DbtCloudRunJobOperator, dbt in an Airflow 3 DAG, Asset-based scheduling or orchestrate a daily ETL with Airflow and dbt. Produces the integration pattern, DAG code, model-level retries and CI. Not for general ETL architecture (use data-etl-pipeline).

Databases

Data Cost Optimization

45

j4flmao/agent-skills

Use when the user asks to cut Snowflake credits, BigQuery slot or on-demand costs, Databricks DBUs, S3 storage tiering or query cost for data workloads. Produces a cost analysis with warehouse sizing, reservations, lifecycle rules and query fixes. Not for overall cloud FinOps (use devops-finops).

Databases

Browse the most popular Claude skills, all databases skills, or write your own with the skills generator.