traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Control lexer's error-mode via explicit boolean argument

Browse source 
Previously, the lexer would operate in error-mode (ignoring any garbage input
until it found a valid token) if it was invoked in the 'error' state. Now that
the error state is deduped with other lexical states, the lexer might be invoked
in that state even when error-mode is not intended. This adds a third argument
to `ts_lex` that explicitly sets the error-mode.

This bug was unlikely to occur in any real grammars, but it caused the
node-tree-sitter-compiler test suite to fail for some grammars with only one
rule.
This commit is contained in:
Max Brunsfeld 2015-12-30 09:37:40 -08:00
parent 4ad1a666be
commit 4b04afac5e
10 changed files with 442 additions and 452 deletions
Download patch file Download diff file Expand all files Collapse all files

11
src/compiler/generate_code/c_code.cc
View file

@ -188,14 +188,13 @@ class CCodeGenerator {
}
void add_lex_function() {
line("static TSTree *ts_lex(TSLexer *lexer, TSStateId lex_state) {");
line("static TSTree *ts_lex(TSLexer *lexer, TSStateId state, bool error_mode) {");
indent([&]() {
line("START_LEXER();");
_switch("lex_state", [&]() {
for (size_t i = 1; i < lex_table.states.size(); i++)
_case(to_string(i), [&]() { add_lex_state(lex_table.states[i]); });
_case("ts_lex_state_error",
[&]() { add_lex_state(lex_table.states[0]); });
_switch("state", [&]() {
size_t i = 0;
for (const LexState &state : lex_table.states)
_case(to_string(i++), [&]() { add_lex_state(state); });
_default([&]() { line("LEX_ERROR();"); });
});
});