tokenization
Preprocessing and Tokenizing Textual Datasets
Turning raw text into the tokenized, memory-mapped shards a GPT training loop can actually stream.
Free · download
Quick answer
Turning raw text into the tokenized, memory-mapped shards a GPT training loop can actually stream.
What this skill does
The preparation step that decides whether training is I/O bound before it starts: encoding, sharding, and the layout the loader reads.
Extracted by Loreto from github.com/karpathy/nanoGPT on 10 September 2026. One run, eighty six seconds, three skills — this is one of them. Nobody wrote it down first; it was read out of the source.
Free, with the whole package: SKILL.md, a README.md, and a generated test.
Get Preprocessing and Tokenizing Textual Datasets
Free to download — drops straight into .claude/skills/.