The tokenizer moves out of Kryon so any Ziran program can use it without a UI dependency. It reports token kinds and byte ranges and colors them into caller-owned SyntaxColorSpan values, which have the same fields as Kryon's TextAreaColorSpan.
The tokenizer moves out of Kryon so any Ziran program can use it without a UI dependency. It reports token kinds and byte ranges and colors them into caller-owned SyntaxColorSpan values, which have the same fields as Kryon's TextAreaColorSpan.