Io Bound Data Processing
Processing, transforming, or moving datasets that may exceed RAM on a single low-compute box — covers memory discipline (streaming, generators, dtype shrinkage), I/O access patterns (sequential vs random, mmap, async), data formats (Parquet vs CSV vs JSON, predicate pushdown), chunking & batching, spill-to-disk (external merge sort, DuckDB/Polars), pipelining (bounded queues, backpressure, checkpointing), codec selection (zstd/lz4/gzip), concurrency for I/O-bound workloads (asyncio, threads, prefetch), and observability (iowait vs CPU%, rows/sec, py-spy/strace). Trigger on "process a large fil
Install
npx skills add pproenca/dot-skills --skill io-bound-data-processinggit clone --depth 1 https://github.com/pproenca/dot-skills.git /tmp/dot-skills
mkdir -p ~/.claude/skills
cp -r /tmp/dot-skills/skills/.experimental/io-bound-data-processing ~/.claude/skills/io-bound-data-processingPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Community I/O-bound data processing on constrained resources Best Practices A reference for engineers processing datasets larger than RAM on a single low-compute box. Organized by execution-lifecycle impact: rules near the top of the table govern *whether the job runs at all*; rules near the bottom shave the last 10 %. Optimize from the top of the waterfall. **Scope:** the patterns that show up in real ETL / data-engineering / batch work on a laptop, a 2-vCPU container, or a Raspberry Pi-class node — streaming, formats, chunking, spill, backpressure, codecs, and the concurrency model that actually matches an I/O-bound bottleneck. Out of scope (covered elsewhere): the algorithmic primitives themselves (see [`computer-science-algorithms`](../computer-science-algorithms/)), distributed compute beyond a single box (use Spark/Dask), and database-engine internals (see official docs).
Details
- Skill name
io-bound-data-processing- Repository
- pproenca/dot-skills
- Path
skills/.experimental/io-bound-data-processing/SKILL.md- License
- MIT
- Repo updated
- 15 August 2026
FAQ
What does the Io Bound Data Processing skill do?
Processing, transforming, or moving datasets that may exceed RAM on a single low-compute box — covers memory discipline (streaming, generators, dtype shrinkage), I/O access patterns (sequential vs random, mmap, async), data formats (Parquet vs CSV vs JSON, predicate pushdown), chunking & batching, spill-to-disk (external merge sort, DuckDB/Polars), pipelining (bounded queues, backpressure, checkpointing), codec selection (zstd/lz4/gzip), concurrency for I/O-bound workloads (asyncio, threads, prefetch), and observability (iowait vs CPU%, rows/sec, py-spy/strace). Trigger on "process a large fil
How do I install the Io Bound Data Processing skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add pproenca/dot-skills --skill io-bound-data-processing. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Io Bound Data Processing popular?
It lives in pproenca/dot-skills, which has 228 GitHub stars and 19 forks.
Related skills
12 Factor App
228pproenca/dot-skills
The Twelve-Factor App methodology for building scalable, maintainable cloud-native applications. Use when designing backend services, APIs, microservices, or any software-as-a-service application. Triggers on deployment patterns, configuration management, process architecture, logging, and infrastructure decisions.
37signals Rails
228pproenca/dot-skills
37signals Rails coding principles and conventions from DHH, Jorge Manrubia, and the Fizzy/Basecamp/HEY codebase. This skill should be used when writing, reviewing, or refactoring Ruby on Rails code following the 37signals philosophy — vanilla Rails, CRUD controllers, rich domain models, concerns, no service objects, Hotwire, Turbo, Stimulus, Solid Queue, Solid Cache, Solid Cable, multi-tenancy, Minitest, custom auth, or DHH conventions.
Acceptance Pipeline Catalog
228pproenca/dot-skills
Use when implementing, reviewing, or debugging a Gherkin acceptance-test pipeline with mutation testing. Covers parser, JSON IR, generator, runtime, step handlers, test runner, mutator, value mutation rules, execution, result classification, reporting, project layout, conformance, and agent setup. Based on Uncle Bob's Acceptance Pipeline Specification. Trigger even when the user mentions Gherkin parsing, acceptance test generation, mutation testing for acceptance tests, or building a portable test pipeline.
Latent Space Thumbnail Director
201swyxio/skills
Create, revise, or review Latent Space or Forward Deployed Engineering YouTube thumbnails. Use for episode thumbnail direction and artwork, not unrelated channels or general publishing.
Forge
201swyxio/skills
Operate or diagnose SmolForge repositories and Forge Deploy/Sites when the task requires Forge-specific CLI, authentication, manifest, or release behavior on forge.smol.ai or *.sites.smol.ai. Do not use for generic Git or Cloudflare work, public-page review, product strategy or analytics, or ordinary code changes that do not interact with Forge-hosted state.
Marketplace Personalisation
228pproenca/dot-skills
Personalisation and recommendation systems for a two-sided trust marketplace built on AWS Personalize — event tracking, dataset and schema design, two-sided matching, cold start, feedback loops, bias control, recipe selection, serving-time re-ranking, observability, and a diagnostic playbook for existing systems. Trigger when designing, building, debugging, reviewing, or improving such a system — and even when the user does not explicitly mention "AWS Personalize" but is working on recommendations, ranking, search, homepage personalisation, or anything that matches seekers and providers across
Browse the most popular Claude skills, all devops & cloud skills, or write your own with the skills generator.