traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Control lexer's error-mode via explicit boolean argument

Browse source 
Previously, the lexer would operate in error-mode (ignoring any garbage input
until it found a valid token) if it was invoked in the 'error' state. Now that
the error state is deduped with other lexical states, the lexer might be invoked
in that state even when error-mode is not intended. This adds a third argument
to `ts_lex` that explicitly sets the error-mode.

This bug was unlikely to occur in any real grammars, but it caused the
node-tree-sitter-compiler test suite to fail for some grammars with only one
rule.
This commit is contained in:
Max Brunsfeld 2015-12-30 09:37:40 -08:00
parent 4ad1a666be
commit 4b04afac5e
10 changed files with 442 additions and 452 deletions
Download patch file Download diff file Expand all files Collapse all files

4
src/runtime/parser.c
View file

@ -203,7 +203,7 @@ static TSTree *ts_parser__get_next_lookahead(TSParser *self, int head) {
TSStateId parse_state = ts_stack_top_state(self->stack, head);
TSStateId lex_state = self->language->lex_states[parse_state];
LOG("lex state:%d", lex_state);
return self->language->lex_fn(&self->lexer, lex_state);
return self->language->lex_fn(&self->lexer, lex_state, false);
}
static int ts_parser__split(TSParser *self, int head) {
@ -464,7 +464,7 @@ static bool ts_parser__handle_error(TSParser *self, int head, TSTree *lookahead)
LOG("skip token:%s", SYM_NAME(lookahead->symbol));
ts_parser__shift(self, head, ts_stack_top_state(self->stack, head),
lookahead);
lookahead = self->language->lex_fn(&self->lexer, ts_lex_state_error);
lookahead = self->language->lex_fn(&self->lexer, 0, true);
error_token_count++;
/*