Cjk Token Reducer

cjk-token-reducer
Reduce Claude Code token usage by 35-50% when using CJK languages.

The Problem
CJK (Chinese, Japanese, Korean) languages consume 2-4x more tokens than English for the same semantic content.
This discrepancy leads to higher costs, faster context exhaustion, and reduced context windows for RAG/agent workflows.

*Token ratios based on BPE tokenizer analysis. Actual savings depend on text complexity and technical term density.*

Why Does This Happen?
The inefficiency stems from the mechanics of Byte-Pair Encoding (BPE) and training data distribution:
1. Vocabulary Bias: Modern tokenizers train primarily on English corpora.
   Common English words merge into single tokens.
   CJK characters, occurring less frequently in training data,
   often fail to merge into "words" and split into individual character tokens or raw bytes.
2. UTF-8 Byte Fallback: A common cause of token expansion.
   - Many LLM tokenizers process text as UTF-8 bytes.
   - An English character is 1 byte.
   - A CJK character is typically 3 bytes in UTF-8.
   - If a CJK character is absent from the tokenizer's vocabulary,
     byte-level tokenizers may expand it into multiple tokens.
     The exact expansion depends on the tokenizer's merge rules.
3. Lack of Delimiters: English uses spaces as natural word boundaries,
   aiding the tokenizer in identifying mergeable units.
   CJK languages lack these delimiters,
   forcing the tokenizer to rely purely on statistical frequency, which is lower for CJK sequences.

The consequence: API billing and model context limits measure tokens, not meaning.
Writing in CJK incurs a "tax" on both cost and memory.

The Solution
CJK Token Reducer translates your CJK input to English before sending it to Claude.
English is the "native language" of LLM…