New
LLM Token Counter
Count tokens and estimate cost for GPT, Claude, Gemini and Llama prompts, with exact counts where tokenizers are public.
How it works
Markdown to AI Knowledge Base
Turn Markdown docs into clean chunks for retrieval-augmented generation. Each document is split at its headings into token-sized pieces, with the heading path and front matter kept as metadata, ready to embed or upload.
Add Markdown to build a knowledge base
Upload docs, notes or a README. Each one is split at its headings into chunks sized for embeddings, with the heading path and front matter kept as metadata.
Chunks
- 01
Upload one or more Markdown files, or paste a document. Load the sample to see how it works.
- 02
Choose the maximum chunk size and overlap. 300–800 tokens suits most embedding models.
- 03
Review the chunks: each shows its heading path and token count, and expands to the full text.
- 04
Pick an export format and download it, or copy it into your pipeline.
About this tool
Why chunking decides how good RAG answers are
Retrieval-augmented generation finds the most relevant pieces of your documents and gives them to a model as context. Those pieces are chunks, and how you cut them matters: chunks that are too big bury the answer in unrelated text, chunks that are too small lose the context that makes them meaningful, and cuts in the middle of a code block or a list produce fragments nobody can use.
This tool splits along the structure Markdown already has. Chunks never cross a heading, paragraphs, lists, tables and code blocks stay whole where they fit, and oversized ones are split at sentence boundaries. Each chunk can start with its heading path (Guide > Install > Windows) so it stands on its own, optionally overlaps the previous chunk, and carries the file name and front matter as metadata. Token counts use the GPT-4o tokenizer, which is close to most embedding models' limits.
Heading-aware
Chunks follow your document's sections and keep code blocks, lists and tables intact.
Exact token counts
Sized with the GPT-4o (o200k) tokenizer, with a configurable limit and overlap.
Metadata included
Source file, title, heading path, chunk index and front matter on every chunk.
Ready to import
Export JSONL, JSON or CSV for vector databases, or one Markdown file for custom GPTs and projects.
FAQ
Frequently asked questions
What chunk size should I use?
Start around 500 tokens with 10% overlap. Smaller chunks (200–300) suit precise Q&A over reference docs; larger ones (800–1,500) suit narrative text where context matters. Test retrieval on real questions and adjust.
Why prefix chunks with headings?
A chunk like "Run the installer and restart" is ambiguous on its own. With "Guide > Install > Windows" in front, both the embedding and the model know what it's about.
Does overlap help?
A little overlap keeps sentences that span a boundary findable from either chunk. Because this tool already splits at headings and paragraphs, 10–15% is usually enough.
Will the token counts match my embedding model?
They're exact for OpenAI's current models and close for most others, since modern tokenizers produce similar counts for English text. Leave some headroom below your model's hard limit.
Are my documents uploaded?
No. Chunking and token counting run entirely in your browser, and your files never leave your device.
Missing something?
Tell us the tool you wish existed.
Describe the job and the file types involved. The most requested tools get built first.