micky@garden:~/token-counter$cd ..

llm tokenizer / multilingual context lab

LLM token counter

A browser-side counter for checking prompt size, language density, and context pressure with the same GPT tokenizer lens used in the course slides.

GPT BPE tokenizerlocal onlymodel families vary
tokenized language samplerGPT BPE

live workbench

Compare two drafts

draft16 tokens
Please summarize the attached meeting notes into three action items, owners, and deadlines.
tokens16
characters91
bytes91
chars/token5.69
budget0.01%
comparison30 tokens
ช่วยสรุปบันทึกการประชุมที่แนบมาเป็นสามงานที่ต้องทำ ผู้รับผิดชอบ และกำหนดส่ง
tokens30
characters57
bytes221
chars/token1.9
budget0.02%
tokens46
budget used0.04%
remaining127,954
ratio1.88x
delta-14
lower token countdraft

natural language density

Same task across languages

Counts use the English task as baseline. Translation length and wording still matter, so treat this as a practical lens, not a universal ranking.

EnglishLatin

Please summarize the attached meeting notes into three action items, owners, and deadlines.

tokens16
per 100 chars17.58
vs baseline1x
Please summarize the attached meeting notes into three action items, owners, and deadlines.
ThaiThai

ช่วยสรุปบันทึกการประชุมที่แนบมาเป็นสามงานที่ต้องทำ ผู้รับผิดชอบ และกำหนดส่ง

tokens30
per 100 chars52.63
vs baseline1.88x
ช่วยสรุปบันทึกการประชุมที่แนบมาเป็นสามงานที่ต้องทำ ผู้รับผิดชอบ และกำหนดส่ง
IndonesianLatin

Ringkas catatan rapat terlampir menjadi tiga tindakan, penanggung jawab, dan tenggat waktu.

tokens22
per 100 chars24.18
vs baseline1.38x
Ringkas catatan rapat terlampir menjadi tiga tindakan, penanggung jawab, dan tenggat waktu.
VietnameseLatin

Hãy tóm tắt ghi chú cuộc họp đính kèm thành ba việc cần làm, người phụ trách và hạn chót.

tokens29
per 100 chars32.58
vs baseline1.81x
Hãy tóm tắt ghi chú cuộc họp đính kèm thành ba việc cần làm, người phụ trách và hạn chót.
JapaneseKana + Han

添付された会議メモを、担当者と期限を含む3つのアクション項目に要約してください。

tokens29
per 100 chars72.5
vs baseline1.81x
添付された会議メモを、担当者と期限を含む3つのアクション項目に要約してください。
ChineseHan

请将附件中的会议记录总结为三个行动事项，并注明负责人和截止日期。

tokens18
per 100 chars56.25
vs baseline1.13x
请将附件中的会议记录总结为三个行动事项，并注明负责人和截止日期。
KoreanHangul

첨부된 회의 메모를 담당자와 마감일이 포함된 세 가지 실행 항목으로 요약해 주세요.

tokens28
per 100 chars60.87
vs baseline1.75x
첨부된 회의 메모를 담당자와 마감일이 포함된 세 가지 실행 항목으로 요약해 주세요.
HindiDevanagari

संलग्न बैठक नोट्स को तीन कार्यों, मालिकों और अंतिम तिथियों में संक्षेप करें।

tokens24
per 100 chars54.55
vs baseline1.5x
संलग्न बैठक नोट्स को तीन कार्यों, मालिकों और अंतिम तिथियों में संक्षेप करें।
SpanishLatin

Resume las notas de la reunión adjunta en tres tareas, responsables y fechas límite.

tokens17
per 100 chars20.24
vs baseline1.06x
Resume las notas de la reunión adjunta en tres tareas, responsables y fechas límite.
FrenchLatin

Résume les notes de réunion jointes en trois actions, responsables et échéances.

tokens17
per 100 chars21.25
vs baseline1.06x
Résume les notes de réunion jointes en trois actions, responsables et échéances.
GermanLatin

Fasse die angehängten Besprechungsnotizen in drei Aufgaben, Verantwortliche und Fristen zusammen.

tokens22
per 100 chars22.68
vs baseline1.38x
Fasse die angehängten Besprechungsnotizen in drei Aufgaben, Verantwortliche und Fristen zusammen.
RussianCyrillic

Суммируй приложенные заметки встречи в три задачи, ответственных и сроки.

tokens19
per 100 chars26.03
vs baseline1.19x
Суммируй приложенные заметки встречи в три задачи, ответственных и сроки.
ArabicArabic

يرجى تلخيص ملاحظات الاجتماع المرفقة إلى ثلاثة إجراءات مع المالكين والمواعيد النهائية.

tokens25
per 100 chars29.41
vs baseline1.56x
يرجى تلخيص ملاحظات الاجتماع المرفقة إلى ثلاثة إجراءات مع المالكين والمواعيد النهائية.

file format density

JSON, TOML, YAML, and friends

The same small action-item record expressed as common interchange formats. This shows where punctuation, quoting, nesting, and repeated keys spend tokens.

JSONBraces

{ "action_items": [ { "owner": "Micky", "deadline": "Friday", "task": "Ship token counter" } ] }

tokens34
per 100 chars32.69
vs baseline1x
{ "action_items": [ { "owner": "Micky", "deadline": "Friday", "task": "Ship token counter" } ] }
YAMLIndentation

action_items: - owner: Micky deadline: Friday task: Ship token counter

tokens21
per 100 chars26.25
vs baseline0.62x
action_items: - owner: Micky deadline: Friday task: Ship token counter
TOONToken-oriented

action_items[1]{owner,deadline,task}: Micky,Friday,Ship token counter

tokens21
per 100 chars29.58
vs baseline0.62x
action_items[1]{owner,deadline,task}: Micky,Friday,Ship token counter
TOMLTables

[[action_items]] owner = "Micky" deadline = "Friday" task = "Ship token counter"

tokens22
per 100 chars27.5
vs baseline0.65x
[[action_items]] owner = "Micky" deadline = "Friday" task = "Ship token counter"
XMLTags

<actionItems> <item owner="Micky" deadline="Friday">Ship token counter</item> </actionItems>

tokens26
per 100 chars27.66
vs baseline0.76x
<actionItems> <item owner="Micky" deadline="Friday">Ship token counter</item> </actionItems>
CSVDelimited

owner,deadline,task Micky,Friday,Ship token counter

tokens14
per 100 chars27.45
vs baseline0.41x
owner,deadline,task Micky,Friday,Ship token counter
Markdown tableTable

| Owner | Deadline | Task | | --- | --- | --- | | Micky | Friday | Ship token counter |

tokens24
per 100 chars27.59
vs baseline0.71x
| Owner | Deadline | Task | | --- | --- | --- | | Micky | Friday | Ship token counter |
NDJSONLines

{"owner":"Micky","deadline":"Friday","task":"Ship token counter"}

tokens16
per 100 chars24.62
vs baseline0.47x
{"owner":"Micky","deadline":"Friday","task":"Ship token counter"}
INISections

[action_item] owner=Micky deadline=Friday task=Ship token counter

tokens16
per 100 chars24.62
vs baseline0.47x
[action_item] owner=Micky deadline=Friday task=Ship token counter