Byte-Pair Encoding (BPE) • Subword Vocabulary & UTF-8 Inspector
Neural BPE Tokenizer& Inspector
Inspect how Large Language Models decompose raw strings into subword tokens, UTF-8 byte streams, vocabulary IDs, and high-dimensional embedding vectors.
Corpus Presets:
cl100k_base • Subword StemsTotal BPE Tokens
35
Sequence Length (T = 35)
Character Count
152
19 whitespace words
Compression Ratio
4.34
Characters per token
Est. KV-Cache Footprint
4.38 MB
Llama-3 8B FP16 (128 KB/tok)
Subword Token Stream
(Click any token to inspect)Token & Byte Inspector
Position #0Surface Token
"Token"
Vocabulary ID
#82698
Token Class:subword
UTF-8 Byte Length:5 byte(s)
Raw UTF-8 Hexadecimal Stream
0x540x6F0x6B0x650x6E
Embedding Row E[82698] (First 8 dims)ℝ⁴⁰⁹⁶
+0.74
-0.75
-0.15
+0.13
-0.90
+0.74
-0.84
+0.52
Sequence Length vs. Tokenization Scheme
Character-Level (1 char/tok)152 tokens
Subword BPE (Modern LLM)35 tokens
Whitespace Word-Level (OOV Prone)19 tokens
Self-attention scales quadratically O(T²) with sequence length T. Subword BPE compresses character streams by ~3.5× while avoiding out-of-vocabulary (OOV) failures via UTF-8 byte fallback.