# A Deeper Dive Into the Fast-Lexer Changes

DevFeed: [A Deeper Dive Into the Fast-Lexer Changes](<https://devfeed.tech/articles/a-deeper-dive-into-the-fast-lexer-changes-22371.md>)

Original publisher: [Read original article](<https://www.red-lang.org/2019/10/a-deeper-dive-into-fast-lexer-changes.html>)

Author: Unknown (noreply@blogger.com)

Published: 2019-10-30T19:18:00Z

Content type: article

Language: en

Sources: [Red](<https://devfeed.tech/sources/red.md>)

Topics: [Red](<https://devfeed.tech/topics/red.md>), [Programming](<https://devfeed.tech/topics/programming.md>), [Visual Studio Code](<https://devfeed.tech/topics/visual-studio-code.md>), [benchmarking](<https://devfeed.tech/topics/benchmarking.md>)

Tags: [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [lexer](<https://devfeed.tech/tags/lexer.md>), [performance](<https://devfeed.tech/tags/performance.md>), [programming](<https://devfeed.tech/tags/programming.md>), [programming-languages](<https://devfeed.tech/tags/programming-languages.md>), [syntax](<https://devfeed.tech/tags/syntax.md>), [vscode](<https://devfeed.tech/tags/vscode.md>)

## AI overview

The article explains why Red prioritized a new fast lexer. Benchmarking found the existing lexer was about 240 times slower than Rebol's, while duplicated lexer code also affected the console and VSCode plugin. The planned solution was to implement a smaller, simpler lexer in Red/System, inspired partly by research on fast parsers.

## Source excerpt

What made the fast-lexer branch a priority? Several things. It started when @dockimbel looked into ticket #3606, which was impossible to fix currently, and we didn't want to give up on the auto-syncing between /text and /data facets. So he had to consider bigger options, including how to make the lexer instrumentable. It was not easy, because the current lexer is not re-entrant, so having the lexer emit events to a callback function could have caused serious problems. Digging through all Red's repos showed that the current lexer code was duplicated twice, beyond the basic lexing needed by load: once in the console code, once in the VSCode plugin, each time for syntax coloring purposes, and each one lagging behind the original implementation. Not good. @Dockimbel then considered changing the current lexer to make it instrumentable, but the changes were significant and would have made the parse rules much more complex. At the same time, @qtxie did some benchmarking, and the result showed Red's lexer was ~240 times slower than Rebol's. This is not due to parse, but rather because the high-level rules were optimized for readability, not performance. The lexer also caused delays in the VSCode plugin, because of its (lack of) performance. The high level code has served Red well, and was a showcase for parse, but loading larger data is also being used by community members, and data sizes will just keep growing. With some projects we have on the horizon, the lexer's performance became a higher priority. As planned since the beginning (the lexer used to be R/S-only during the pre-Unicode era), @dockimbel decided the best option was to not postpone the conversion of the lexer to pure R/S code any longer, by porting R3's C-based lexer to R/S. After studying Rebol's lexer in detail, he realized that the code was quite complex in some places (mostly the prescanner), and would lead to less than optimal R/S code that would be hard to maintain. Evaluating the state of the art in fa