# lexer Published articles for lexer. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## The simplest interpreter: Write yourself a compiler, Part I DevFeed: [The simplest interpreter: Write yourself a compiler, Part I]() Original publisher: [Read original article]() Published: 2026-07-16T22:00:00Z Content type: tutorial Language: en Sources: [Tomasz Nurkiewicz around Java and concurrency]() Topics: [Compiler](), [Parser](), [Parsing](), [Software Engineering](), [Go Language]() Tags: [code](), [compiler](), [engineering](), [go](), [how-to](), [interpreter](), [lexer](), [writing-compiler]() ### AI overview This tutorial begins a series on building a compiler by implementing a simple interpreter in Go. The interpreter lexes and evaluates an input expression containing the addition of two numbers, using a regular expression to validate the structure and extract tokens. ### Source excerpt I've always felt that writing a compiler is the most romantic software engineering task. You're writing a program that reads textual instructions describing precisely what a computer should do. Do not confuse this with prompting an LLM, where you write vague, verbose instructions that only loosely describe what a computer might do. But I digress. ## A Practical guide to building a parser in Go DevFeed: [A Practical guide to building a parser in Go]() Original publisher: [Read original article]() Author: Tom Published: 2026-01-04T00:00:00Z Content type: tutorial Language: en Sources: [Tomasz Gągor]() Topics: [Go Language](), [Parser](), [Programming](), [Programming language]() Tags: [bnf](), [compilers](), [dsl](), [go](), [guide](), [left-recursion](), [lexer](), [packrat-memoization](), [parsing](), [programming](), [recursive-descent-parser]() ### AI overview A developer describes building a recursive descent parser in Go to validate strings against a BNF-like grammar. The project began as an attempt to implement a simple BNF syntax analyzer and led to exploring parser and compiler concepts, including left recursion and Packrat memoization. ### Source excerpt A developer's journey of building a recursive descent parser in Go, from basic concepts to tackling left recursion with Packrat memoization. ## Important Change! Switching map and construction syntax. DevFeed: [Important Change! Switching map and construction syntax.]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2024-02-11T14:37:00Z Content type: opinion Language: en Sources: [Red]() Topics: [Red](), [Code](), [Script](), [Instrumentation](), [data]() Tags: [code](), [implementation](), [instrumentation](), [language](), [lexer](), [map](), [maps](), [migration](), [safety](), [switching](), [syntax]() ### AI overview The article announces that Red and Rebol3 will swap the syntactic forms used for map! values and construction syntax. It explains the evaluation behavior motivating the change, notes the resulting incompatibility with Rebol2 construction syntax, and describes tools for automatically converting scripts. ### Source excerpt Sometimes deep changes take a huge amount of code. Sometimes they take a lot of detailed explanation and consideration, leading to long discussions and people taking sides. Rarely does an important syntactic change to a language happen quickly, with universal agreement, simple implementation, and tools to help update scripts in the wild. Today is one of those rare days. Admittedly, this idea has been discussed for a long time. It would surface, people nodded their virtual heads, and it would submerge again. Today it's ready to deploy. Not only that, but Rebol3 is making the same change, so the two languages will still be compatible in this regard. What is the change? It's easy to describe. Today, map! values use this syntax: #(...) and construction syntax (sometimes called serialized form or loadable form) looks like this: #[...]. Going forward, those syntactic forms will be swapped. Why? The answer is easy. In Redbol langs, blocks do not evaluate by default, you have to do or reduce them. Parens, on the other hand, do evaluate by default. Today, maps use paren-like syntax, but they do not evaluate, while construction syntax uses block-like syntax, but does evaluate. This is a carryover from Rebol, so the major concession here is that Red and Rebol3 will no longer be compatible with Rebol2's construction syntax. If you've never heard of construction syntax, there's a nice explanation of it here. Red only supports a few values via construction syntax today, all datatype literals, true, false, none, and unset; but eventually it will support much more. If you look at the help for mold, you'll see that /all is TBD (very partially implemented for now), and that's how you create loadable, serialized, data that can safely and easily contain any value (like redbin but readable by humans). It helps avoid cases where none or true/false may load as words. This is also why construct evaluates those specific words (including also on/off/yes/no), but not others. When loading untr ## Practical parsing with Flex and Bison DevFeed: [Practical parsing with Flex and Bison]() Original publisher: [Read original article]() Published: 2021-11-28T00:00:00Z Content type: tutorial Language: en Sources: [Joe Nelson]() Topics: [Parsing](), [Parser](), [C](), [Regular expression](), [make]() Tags: [build-system](), [c](), [compiler](), [lexer](), [make](), [parsing](), [scanner]() ### AI overview A practical guide to parsing with Lex, Yacc, Flex, and Bison. It explains how lexing divides input into meaningful chunks, how parsing groups those chunks according to recursive rules, and how parser generators can be integrated into build systems and used for real data formats. ### Source excerpt 2021-11-28 Although parsing is often described from the perspective of writing a compiler, there are many common smaller tasks where it's useful. Reading file formats, talking over the network, creating shells, and analyzing source code are all easier using a robust parser. By taking time to learn general-purpose parsing tools, you can go beyond fragile homemade solutions, and inflexible third-party libraries. We'll cover Lex and Yacc in this guide because they are mature and portable. ## A New Fast and Flexible Lexer DevFeed: [A New Fast and Flexible Lexer]() Original publisher: [Read original article]() Author: Nenad Rakocevic (noreply@blogger.com) Published: 2020-08-03T12:06:00Z Content type: release Language: en Sources: [Red]() Topics: [Red](), [Programming language](), [Parsing](), [Instrumentation](), [API]() Tags: [callback](), [code](), [compiler](), [function](), [implementation](), [instrumentation](), [lexer](), [load](), [performance]() ### AI overview The Red programming language has introduced a new lexer written in Red/System. It is designed to load large quantities of Red values quickly and adds scanning and event-oriented instrumentation features. The article reports typical performance gains of 50 to 200 times over the previous lexer and describes benchmark tasks and known issues affecting older-version results. ### Source excerpt A programming language lexer is the part in charge of converting textual code representation into a structured memory representation. In Red, it is accomplished by the load function, which calls the lower-level transcode native. Until now, Red was relying on a lexer entirely written using the Parse dialect. Though, the parsing rules were constructed to be easily maintained and not for performance. Rewriting those rules to speed them up could have been possible, but rewriting the lexer entirely in Red/System would give the ultimate performance. It might not matter for most user scripts, but given that Red is also a data format, we need a solution for fast (near-instant) loading of huge quantities of Red values stored in files or transferred through the network. The new lexer main features are: High performance, typically 50 to 200 times faster than the older one. New scanning features: identify values and their datatypes without loading them. Instrumentation: customize the lexer's behavior at will using an event-oriented API. The reference documentation is available there. This new lexer is available in Red's auto-builds since June. Performance Vastly increased performance is the main driver for this new lexer. Here is a little benchmark to let you appreciate how far it gets. The benchmarking tasks are: 100 x compiler.r: loads 100 times compiler.r source file from memory (~126KB, so about ~12MB in total). 1M short integers: loads a string of 1 million `1` separated by a space. 1M long integers: loads a string of 1 million `123456789` separated by a space. 1M dates: loads a string of 1 million `26/12/2019/10:18:25` separated by a space. 1M characters: loads a string of 1 million `#"A"` separated by a space. 1M escaped characters: loads a string of 1 million `#"^(1234)"` separated by a space. 1M words: loads a string of 1 million `random "abcdefghijk"` separated by a space. 100K words: loads a string of 100 thousands `random "abcdefghijk"` separated by a space. And the ## Supporting Multiple Dialects in SQLDelight DevFeed: [Supporting Multiple Dialects in SQLDelight]() Original publisher: [Read original article]() Author: Author Published: 2020-02-13T18:15:17Z Content type: article Language: en Sources: [Posts on AlecStrong]() Topics: [SQLite](), [MySQL](), [Compiler](), [SQL](), [Parsing](), [ANTLR]() Tags: [compiler](), [compilers](), [gradle](), [intellij](), [lexer](), [mysql](), [open-source](), [parsing](), [plugin](), [python](), [sql](), [sqldelight](), [sqlite]() ### AI overview This article explains how SQLDelight supports multiple SQL dialects, including SQLite 3.18, SQLite 3.24, and MySQL. It describes the compiler, lexer, parser, AST, and the use of JetBrains Grammar Kit to build composable parsing and incrementally extend dialect support. ### Source excerpt With SQLDelight 1.3.0 and above we now support multiple dialects of SQL (at time of writing: SQLite 3.18, SQLite 3.24, and MySQL). They're not complete implementations of any of those dialects but they support necessary features and the real meat of the release is a straightforward framework for adding parts of the dialects incrementally. For example, SQLite 3.24 is just SQLite 3.18 with upsert on top (thank you Angus Holder!). ## Red announces preliminary Parse documentation and other planned projects for 2020 DevFeed: [Red announces preliminary Parse documentation and other planned projects for 2020]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2020-01-01T07:23:00Z Content type: article Language: en Sources: [Red]() Topics: [Red](), [Parsing](), [Documentation]() Tags: [compilers](), [decoding](), [documentation](), [encoding](), [languages](), [lexer](), [parsing](), [validation]() ### AI overview Red's New Year article previews projects planned for 2020, including a new product and a robust preliminary draft of documentation for the Parse dialect. It explains Parse's uses in searching, validation, extraction, modification, language processing, and encoding and decoding data formats. ### Source excerpt Hello and happy new year, friends of Red! We have some exciting projects we've been working on that will be available this year, including a new product. Let's talk a little about what the team has been working on behind the scenes. (TL;DR: A cool new product with Red in 2020...plus, a robust preliminary draft of Parse documentation can now be previewed...CLI library...fast-lexer to merge soon...GTK on the horizon...and a new native OS calendar widget!) Documentation for Parse: Red's Language Construction Tool Our esteemed forerunner, Rebol, broke new ground with its Parse dialect, which Red has expanded on. Today, in Red, Parse has become an even more powerful built-in dialect (embedded domain-specific language) that processes input series with grammar rules in a clean and simple manner. Other language building tools exist, of course, such as Lexx and Yacc. ANTLR is a modern framework built in Java, and libraries exist for other languages, but the ease of use and power that Red's Parse offers is unique. This isn't a new feature, by any means, with the first public introduction here. Parse is easy enough to use that those basic docs have been enough, for the most part. But over time, with new users joining the Parse chat room to discuss the dialect, and it was time for full reference documentation. For the uninitiated, Parse can be used to for searching, to surface various patterns; validation, in order to confirm an input's compliance to a specification; extraction, to sift through data and aggregate values; and modification--that is, changing the input stream itself (insertion of values, removing or transforming matched input). And Parse's true power lies in language processing (compilers, interpreters, and lexical analyzers), particularly for DSLs; and encoding/decoding, to "translate" data formats from one to another. What makes Red's Parse dialect a killer feature is that you can do more than parse at the character level in strings of text. That's what every ot ## Editorial: A Brief Essay on Lexical Ambiguity by G. Irwin DevFeed: [Editorial: A Brief Essay on Lexical Ambiguity by G. Irwin]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2019-11-19T23:16:00Z Content type: opinion Language: en Sources: [Red]() Topics: [Red](), [Programming]() Tags: [community](), [editorial](), [languages](), [lexer](), [syntax]() ### AI overview An editorial essay examines lexical ambiguity in Red and the trade-off between adding specialized lexical forms and preserving a language's flexibility, usability, and learnability. It argues that new lexical forms offer diminishing value and that a constrained hierarchy can help users remember rules. ### Source excerpt The original commentary was posted in Red's Gitter channel, here, by Gregg Irwin, one of our core team members, in response to various requests for the ability to create new datatypes in Red. As a writer, Red has always appealed to me because of its flexibility; but, of course, "the [lexicon] devil is in the details," as the idiom goes (okay, I edited that idiom a little, but it was too cool a link to pass up). It means the more specific we try to be, the more challenges and limitations we encounter, and we can lose some of the amazing versatility of the language. On the other hand, precision and refinement--the "exact right word at the exact right time," can powerfully enhance a language's utility. The dynamic tension between what he calls "generality and specificity, human friendliness and artifice," in the text below, can be an energetic ebb and flow that serves to strengthen our language, to make it more robust. Two quotes from community members provide some context: _____________________________ > The real problem is not number of datatypes, but the lexical syntax of the new ones. -@Oldes > ...However if something like utype! is added, nothing prevents you from (ab)using system/lexer/pre-load and reinventing whole syntax. -@Rebolek "I don't support abusing system/lexer/pre-load, and (in the long view) there will almost certainly be special cases where a new lexical form makes sense. We can't see the future, so we can't rule it out. But, and this is key, how much value does each new one add? I believe that each new lexical form adds less value, and there is a point of diminishing returns. This is not just a lexical problem for Red, but for humans. We have limited capacity to remember rules, and a constrained hierarchy helps enormously here. Think more like linguists, and less like programmers or mathematicians. In language we have words and numbers. Numbers can be represented as words, with their notation being a handy shortcut for use in the domain of mathemati ## A Deeper Dive Into the Fast-Lexer Changes DevFeed: [A Deeper Dive Into the Fast-Lexer Changes]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2019-10-30T19:18:00Z Content type: article Language: en Sources: [Red]() Topics: [Red](), [Programming](), [Visual Studio Code](), [benchmarking]() Tags: [benchmarking](), [lexer](), [performance](), [programming](), [programming-languages](), [syntax](), [vscode]() ### AI overview The article explains why Red prioritized a new fast lexer. Benchmarking found the existing lexer was about 240 times slower than Rebol's, while duplicated lexer code also affected the console and VSCode plugin. The planned solution was to implement a smaller, simpler lexer in Red/System, inspired partly by research on fast parsers. ### Source excerpt What made the fast-lexer branch a priority? Several things. It started when @dockimbel looked into ticket #3606, which was impossible to fix currently, and we didn't want to give up on the auto-syncing between /text and /data facets. So he had to consider bigger options, including how to make the lexer instrumentable. It was not easy, because the current lexer is not re-entrant, so having the lexer emit events to a callback function could have caused serious problems. Digging through all Red's repos showed that the current lexer code was duplicated twice, beyond the basic lexing needed by load: once in the console code, once in the VSCode plugin, each time for syntax coloring purposes, and each one lagging behind the original implementation. Not good. @Dockimbel then considered changing the current lexer to make it instrumentable, but the changes were significant and would have made the parse rules much more complex. At the same time, @qtxie did some benchmarking, and the result showed Red's lexer was ~240 times slower than Rebol's. This is not due to parse, but rather because the high-level rules were optimized for readability, not performance. The lexer also caused delays in the VSCode plugin, because of its (lack of) performance. The high level code has served Red well, and was a showcase for parse, but loading larger data is also being used by community members, and data sizes will just keep growing. With some projects we have on the horizon, the lexer's performance became a higher priority. As planned since the beginning (the lexer used to be R/S-only during the pre-Unicode era), @dockimbel decided the best option was to not postpone the conversion of the lexer to pure R/S code any longer, by porting R3's C-based lexer to R/S. After studying Rebol's lexer in detail, he realized that the code was quite complex in some places (mostly the prescanner), and would lead to less than optimal R/S code that would be hard to maintain. Evaluating the state of the art in fa ## Let's Build A Simple Interpreter. Part 12. DevFeed: [Let's Build A Simple Interpreter. Part 12.]() Original publisher: [Read original article]() Author: Ruslan Spivak Published: 2016-12-01T21:20:00Z Content type: tutorial Language: en Sources: [Ruslan Spivak]() Topics: [Parsing](), [syntax](), [Code](), [identifier]() Tags: [blog](), [code](), [identifier](), [lexer](), [syntax]() ### AI overview Part 12 of a tutorial series explains how to extend a simple interpreter to parse Pascal procedure declarations. It covers procedure syntax, nested declarations, grammar updates, syntax diagrams, and lexer changes for a PROCEDURE token. ### Source excerpt "Be not afraid of going slowly; be afraid only of standing still." - Chinese proverb. Hello, and welcome back! Today we are going to take a few more baby steps and learn how to parse Pascal procedure declarations. What is a procedure declaration? A procedure declaration is a language construct that ... ## Let's Build A Simple Interpreter. Part 11. DevFeed: [Let's Build A Simple Interpreter. Part 11.]() Original publisher: [Read original article]() Author: Ruslan Spivak Published: 2016-09-21T01:15:00Z Content type: tutorial Language: en Sources: [Ruslan Spivak]() Topics: [Parsing](), [Parser](), [Programming](), [syntax](), [Code]() Tags: [blog](), [compiler](), [how-to](), [lexer](), [parsing](), [recursive-descent-parser](), [tokens]() ### AI overview Part 11 recaps how to build an interpreter by performing lexical analysis, parsing, grammar mapping, recursive-descent parsing, AST construction and traversal, and interpretation of a complete Pascal program. It previews nested procedures and functions, semantic analysis, and control flow. ### Source excerpt I was sitting in my room the other day and thinking about how much we had covered, and I thought I would recap what we've learned so far and what lies ahead of us. Up until now we've learned: How to break sentences into tokens. The process is ... ## Let's Build A Simple Interpreter. Part 6. DevFeed: [Let's Build A Simple Interpreter. Part 6.]() Original publisher: [Read original article]() Author: Ruslan Spivak Published: 2015-11-02T12:00:00Z Content type: tutorial Language: en Sources: [Ruslan Spivak]() Topics: [Parsing](), [syntax](), [Code]() Tags: [blog](), [code](), [expression](), [lexer](), [recursion](), [syntax](), [tokens]() ### AI overview This tutorial extends an arithmetic-expression grammar to support parenthesized expressions with arbitrarily deep nesting. It updates the lexer and interpreter to evaluate expressions containing integers and arithmetic operators. ### Source excerpt Today is the day :) "Why?" you might ask. The reason is that today we're wrapping up our discussion of arithmetic expressions (well, almost) by adding parenthesized expressions to our grammar and implementing an interpreter that will be able to evaluate parenthesized expressions with arbitrarily deep nesting, like the expression ...