Don't create error node in lexer unless token is completely invalid
Before, any syntax error would cause the lexer to create an error leaf node. This could happen even with a valid input, if the parse stack had split and one particular version of the parse stack failed to parse. Now, an error leaf node is only created when the lexer cannot understand part of the input stream at all. When a normal syntax error occurs, the lexer just returns a token that is outside of the expected token set, and the parser handles the unexpected token.
This commit is contained in:
parent
7c859a07bb
commit
1e353381ff
13 changed files with 225 additions and 148 deletions
|
|
@ -192,7 +192,7 @@ class CCodeGenerator {
|
|||
|
||||
void add_lex_function() {
|
||||
line(
|
||||
"static TSTree *ts_lex(TSLexer *lexer, TSStateId state, bool error_mode) "
|
||||
"static void ts_lex(TSLexer *lexer, TSStateId state, bool error_mode) "
|
||||
"{");
|
||||
indent([&]() {
|
||||
line("START_LEXER();");
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue