A Tiny Tokenizer In Zig To Try Out Labeled Switch

2026 18:40
Synopsis
I've had a lot of feedback on my video about Labeled Switch, and I figured we'd best create a tokenizer or FSM to help see if the ...

You Might Also Like

Let's write an HTML Tokenizer in Zig (Part 1/?) 3:05:03
Let's Write An Html Tokenizer In Zig (part 1/?)
2024
Zig | splitAny | Tutorial 2:26
Zig | Splitany | Tutorial
2026
Building a BPE Tokenizer from Scratch in Zig (tokenizerz) 6:31
Building A Bpe Tokenizer From Scratch In Zig (tokenizerz)
2026
Tokenizers Overview 0:56
Tokenizers Overview
2022
Character-based tokenizers 3:01
Character-based Tokenizers
2022
Let's build the GPT Tokenizer 2:13:35
Let's Build The Gpt Tokenizer
2025
Why are fast tokenizers called fast? 1:49
Why Are Fast Tokenizers Called Fast?
2022
Let's Make a Programming Language in Zig - Part 1 - Parsing Tokens 48:31
Let's Make A Programming Language In Zig - Part 1 - Parsing ...
2026
Tokens in AI Explained — Why Prompts Have Limits 1:17
Tokens In Ai Explained — Why Prompts Have Limits
2026
What is pre-tokenization? 2:50
What Is Pre-tokenization?
2022
What is an AI Token? | LLM Tokens explained in 2 minutes! 1:59
What Is An Ai Token? | Llm Tokens Explained In 2 Minutes!
2026
Download Options
Choose a download method below. All links open in new tabs.
Service Features Action
SaveFrom
MP4 & MP3 • HD Quality • Browser Extension Available
Download
Security Notice: These are third-party services. We recommend using antivirus software and being cautious of pop-up ads.