traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Don't create error node in lexer unless token is completely invalid

Browse source 
Before, any syntax error would cause the lexer to create an error
leaf node. This could happen even with a valid input, if the parse
stack had split and one particular version of the parse stack
failed to parse.

Now, an error leaf node is only created when the lexer cannot understand
part of the input stream at all. When a normal syntax error occurs,
the lexer just returns a token that is outside of the expected token
set, and the parser handles the unexpected token.
This commit is contained in:
Max Brunsfeld 2016-05-20 20:26:03 -07:00
parent 7c859a07bb
commit 1e353381ff
13 changed files with 225 additions and 148 deletions
Download patch file Download diff file Expand all files Collapse all files

5
src/runtime/tree.h
View file

@ -13,7 +13,7 @@ extern "C" {
extern TSStateId TS_TREE_STATE_INDEPENDENT;
extern TSStateId TS_TREE_STATE_ERROR;
struct TSTree {
typedef struct TSTree {
struct {
struct TSTree *parent;
size_t index;
@ -43,7 +43,7 @@ struct TSTree {
bool fragile_left : 1;
bool fragile_right : 1;
bool has_changes : 1;
};
} TSTree;
typedef Array(TSTree *) TreeArray;
TreeArray ts_tree_array_copy(TreeArray *);
@ -65,7 +65,6 @@ size_t ts_tree_end_column(const TSTree *self);
void ts_tree_set_children(TSTree *, size_t, TSTree **);
void ts_tree_assign_parents(TSTree *);
void ts_tree_edit(TSTree *, TSInputEdit);
void ts_tree_steal_padding(TSTree *, TSTree *);
char *ts_tree_string(const TSTree *, const TSLanguage *, bool include_all);
size_t ts_tree_last_error_size(const TSTree *);