Io Bound Data Processing

Processing, transforming, or moving datasets that may exceed RAM on a single low-compute box — covers memory discipline (streaming, generators, dtype shrinkage), I/O access patterns (sequential vs random, mmap, async), data formats (Parquet vs CSV vs JSON, predicate pushdown), chunking & batching, spill-to-disk (external merge sort, DuckDB/Polars), pipelining (bounded queues, backpressure, checkpointing), codec selection (zstd/lz4/gzip), concurrency for I/O-bound workloads (asyncio, threads, prefetch), and observability (iowait vs CPU%, rows/sec, py-spy/strace). Trigger on "process a large fil

228 stars 19 forksDevOps & Cloud

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add pproenca/dot-skills --skill io-bound-data-processing
Manual install
git clone --depth 1 https://github.com/pproenca/dot-skills.git /tmp/dot-skills
mkdir -p ~/.claude/skills
cp -r /tmp/dot-skills/skills/.experimental/io-bound-data-processing ~/.claude/skills/io-bound-data-processing

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Community I/O-bound data processing on constrained resources Best Practices

A reference for engineers processing datasets larger than RAM on a single low-compute box. Organized by execution-lifecycle impact: rules near the top of the table govern *whether the job runs at all*; rules near the bottom shave the last 10 %. Optimize from the top of the waterfall.

**Scope:** the patterns that show up in real ETL / data-engineering / batch work on a laptop, a 2-vCPU container, or a Raspberry Pi-class node — streaming, formats, chunking, spill, backpressure, codecs, and the concurrency model that actually matches an I/O-bound bottleneck. Out of scope (covered elsewhere): the algorithmic primitives themselves (see [`computer-science-algorithms`](../computer-science-algorithms/)), distributed compute beyond a single box (use Spark/Dask), and database-engine internals (see official docs).
Read the full skill on GitHub

Details

Skill name
io-bound-data-processing
Path
skills/.experimental/io-bound-data-processing/SKILL.md
License
MIT
Repo updated
15 August 2026

FAQ

What does the Io Bound Data Processing skill do?

Processing, transforming, or moving datasets that may exceed RAM on a single low-compute box — covers memory discipline (streaming, generators, dtype shrinkage), I/O access patterns (sequential vs random, mmap, async), data formats (Parquet vs CSV vs JSON, predicate pushdown), chunking & batching, spill-to-disk (external merge sort, DuckDB/Polars), pipelining (bounded queues, backpressure, checkpointing), codec selection (zstd/lz4/gzip), concurrency for I/O-bound workloads (asyncio, threads, prefetch), and observability (iowait vs CPU%, rows/sec, py-spy/strace). Trigger on "process a large fil

How do I install the Io Bound Data Processing skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add pproenca/dot-skills --skill io-bound-data-processing. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Io Bound Data Processing popular?

It lives in pproenca/dot-skills, which has 228 GitHub stars and 19 forks.

Related skills

12 Factor App

228

pproenca/dot-skills

The Twelve-Factor App methodology for building scalable, maintainable cloud-native applications. Use when designing backend services, APIs, microservices, or any software-as-a-service application. Triggers on deployment patterns, configuration management, process architecture, logging, and infrastructure decisions.

Software Development19

37signals Rails

228

pproenca/dot-skills

37signals Rails coding principles and conventions from DHH, Jorge Manrubia, and the Fizzy/Basecamp/HEY codebase. This skill should be used when writing, reviewing, or refactoring Ruby on Rails code following the 37signals philosophy — vanilla Rails, CRUD controllers, rich domain models, concerns, no service objects, Hotwire, Turbo, Stimulus, Solid Queue, Solid Cache, Solid Cable, multi-tenancy, Minitest, custom auth, or DHH conventions.

Software Development19

Acceptance Pipeline Catalog

228

pproenca/dot-skills

Use when implementing, reviewing, or debugging a Gherkin acceptance-test pipeline with mutation testing. Covers parser, JSON IR, generator, runtime, step handlers, test runner, mutator, value mutation rules, execution, result classification, reporting, project layout, conformance, and agent setup. Based on Uncle Bob's Acceptance Pipeline Specification. Trigger even when the user mentions Gherkin parsing, acceptance test generation, mutation testing for acceptance tests, or building a portable test pipeline.

Testing & QA19

Latent Space Thumbnail Director

201

swyxio/skills

Create, revise, or review Latent Space or Forward Deployed Engineering YouTube thumbnails. Use for episode thumbnail direction and artwork, not unrelated channels or general publishing.

DevOps & Cloud12

Forge

201

swyxio/skills

Operate or diagnose SmolForge repositories and Forge Deploy/Sites when the task requires Forge-specific CLI, authentication, manifest, or release behavior on forge.smol.ai or *.sites.smol.ai. Do not use for generic Git or Cloudflare work, public-page review, product strategy or analytics, or ordinary code changes that do not interact with Forge-hosted state.

DevOps & Cloud12

Marketplace Personalisation

228

pproenca/dot-skills

Personalisation and recommendation systems for a two-sided trust marketplace built on AWS Personalize — event tracking, dataset and schema design, two-sided matching, cold start, feedback loops, bias control, recipe selection, serving-time re-ranking, observability, and a diagnostic playbook for existing systems. Trigger when designing, building, debugging, reviewing, or improving such a system — and even when the user does not explicitly mention "AWS Personalize" but is working on recommendations, ranking, search, homepage personalisation, or anything that matches seekers and providers across

DevOps & Cloud19

Browse the most popular Claude skills, all devops & cloud skills, or write your own with the skills generator.