traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Allow lexer to accept tokens that ended at previous positions

Browse source 
* Track lookahead in each tree
* Add 'mark_end' API that external scanners can use
This commit is contained in:
Max Brunsfeld 2017-03-13 17:03:47 -07:00
parent 12d2a9d93f
commit d222dbb9fd
12 changed files with 96 additions and 71 deletions
Download patch file Download diff file Expand all files Collapse all files

2
src/runtime/lexer.h
View file

@ -15,11 +15,11 @@ typedef struct {
TSLexer data;
Length current_position;
Length token_start_position;
Length token_end_position;
const char *chunk;
uint32_t chunk_start;
uint32_t chunk_size;
uint32_t lookahead_size;
TSInput input;