traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1368 commits 16 branches 98 tags 37 MiB
Commit graph

299 commits

Author SHA1 Message Date
Max Brunsfeld
36608180d2 Store external token states in the parse stack 2017-01-08 22:06:05 -08:00
Max Brunsfeld
3a4daace26 Move reusable node functions to their own file 2017-01-05 10:07:27 -08:00
Max Brunsfeld
2fa7b453c8 Restore external scanner's state only after repositioning lexer 
Also, properly identify the leaf node with the external token state
 2016-12-21 13:59:56 -08:00
Max Brunsfeld
34a65f588d Tweak naming and organization of external-scanner related language fields 2016-12-21 11:24:41 -08:00
Max Brunsfeld
e6c82ead2c Start work toward maintaining external scanner's state during incremental parses 2016-12-20 17:06:20 -08:00
Max Brunsfeld
2b3da512a4 Add serialize, deserialize and reset callbacks to external scanners 
Signed-off-by: Nathan Sobo <nathan@github.com>
 2016-12-20 13:12:01 -08:00
Max Brunsfeld
7f6ec0131d Remove duplication between parser_destroy and parser_set_language 2016-12-06 10:12:49 -08:00
Max Brunsfeld
c4fe8ded95 Remove state argument to Lexer advance method 2016-12-05 16:36:34 -08:00
Max Brunsfeld
c16b6b2059 Run external scanners during error recovery 2016-12-05 11:50:24 -08:00
Max Brunsfeld
cf0d8abea1 Destroy external scanner when destroying Parser 2016-12-04 14:18:30 -08:00
Max Brunsfeld
0f8e130687 Call external scanner functions when lexing 2016-12-02 22:03:48 -08:00
Max Brunsfeld
c966af0412 Start work on external tokens 2016-12-02 16:24:19 -08:00
Max Brunsfeld
e7217f1bac Clean up some methods in parser.c 2016-11-14 17:25:55 -08:00
Max Brunsfeld
535879a2bd Represent byte, char and tree counts as 32 bit numbers 
The parser spends the majority of its time allocating and freeing trees and stack nodes.
Also, the memory footprint of the AST is a significant concern when using tree-sitter
with large files. This library is already unlikely to work very well with source files
larger than 4GB, so representing rows, columns, byte lengths and child indices as
unsigned 32 bit integers seems like the right choice.
 2016-11-14 12:19:13 -08:00
Max Brunsfeld
8edb8df530 Remove extraneous Language methods 2016-11-14 10:35:33 -08:00
Max Brunsfeld
1118a9142a Introduce Symbol::Index type alias 2016-11-14 10:25:26 -08:00
Max Brunsfeld
1fddb124b3 Remove stray LOG_STACK() call 2016-11-14 09:32:05 -08:00
Max Brunsfeld
fad7294ba4 Store shift states for non-terminals directly in the main parse table 2016-11-14 08:36:06 -08:00
Max Brunsfeld
8d9c261e3a Don't include reduce actions for nonterminal lookaheads 2016-11-10 11:33:37 -08:00
Max Brunsfeld
c9dcb29c6f Remove the TS prefix from some internal type/function names 2016-11-09 20:59:05 -08:00
Max Brunsfeld
4106ecda43 Remove logic for recovering from OOM 2016-11-04 09:18:38 -07:00
Max Brunsfeld
e53beb66c9 Avoid anonymous nested struct to silence override-init warnings 2016-10-26 11:10:56 -07:00
Max Brunsfeld
eed54d95e1 Merge branch 'master' into changed-ranges 2016-10-16 21:10:25 -07:00
Max Brunsfeld
b3140b2689 Implement ts_document_parse_and_get_changed_ranges 2016-10-15 22:31:21 -07:00
Max Brunsfeld
e149d94ff5 Remove generated parsers' dependency on runtime.h 2016-10-05 14:02:49 -07:00
Max Brunsfeld
3014101104 Fix inconsistencies in nodes sizes after edits 2016-09-19 13:35:08 -07:00
Max Brunsfeld
fcf9293d35 Use explicit stack for assigning trees' parent pointers 2016-09-19 12:40:45 -07:00
Max Brunsfeld
00528e50ce Change edit API to be byte-based 2016-09-13 13:08:52 -07:00
Max Brunsfeld
cc62fe0375 Represent Lengths in terms of Points 2016-09-09 21:11:02 -07:00
Max Brunsfeld
096ac2d4b6 Rename ts_document_set_debugger -> ts_document_set_logger 2016-09-06 17:40:26 -07:00
Max Brunsfeld
587bbe998c Don't reuse non-trivial subtrees when stack is split 2016-09-06 10:23:26 -07:00
Max Brunsfeld
b76574e01c Handle ambiguities between extra and non-extra tokens using normal GLR splitting 2016-09-06 10:22:16 -07:00
Max Brunsfeld
4f0c83ba01 Move logic for lexical error handling outside of lexer functions 
This way, less logic needs to be exposed in parser.h
 2016-09-03 23:40:57 -07:00
Max Brunsfeld
1c52c30111 Fix unexpected EOF errors getting lost 2016-09-03 22:46:14 -07:00
Max Brunsfeld
e0b0e29a2b Update parse count correctly when repairing errors & undoing reductions 2016-09-01 10:04:20 -07:00
Max Brunsfeld
e04ac7b13d Ensure the correct error stats are used when aborting versions immediately 2016-08-31 14:29:34 -07:00
Max Brunsfeld
0faae52132 Fix some inconsistencies in error cost calculation 
Signed-off-by: Nathan Sobo <nathan@github.com>
 2016-08-31 10:51:59 -07:00
Max Brunsfeld
1d0f6c3cc0 Rename error_size -> error_cost 2016-08-30 11:09:12 -07:00
Max Brunsfeld
52ccebbf80 Rename error_depth -> error_count 2016-08-30 09:44:40 -07:00
Max Brunsfeld
1d617ab5e0 Allow reductions based on error token, skipping some preceding content 2016-08-29 17:34:51 -07:00
Max Brunsfeld
e7ed92f1df Replace ts_parser -> parser 2016-08-29 12:08:58 -07:00
Max Brunsfeld
1b8843dd41 Perform all possible reductions recursively upon detecting an error 2016-08-29 11:23:35 -07:00
Max Brunsfeld
a3c943d88e Fix failure to find repairs when extra tokens precede the detection point 2016-08-29 11:23:35 -07:00
Max Brunsfeld
dcf3beb393 Abandon parse versions with > 1 error more than the best version 2016-08-29 09:31:52 -07:00
Max Brunsfeld
4e6e996a76 Simplify skipping of reduced subtrees after errors 2016-08-15 17:37:00 -07:00
Max Brunsfeld
c067b30ce8 Fix handling of errors while in error state 2016-08-06 21:25:35 -07:00
Max Brunsfeld
b561000b2a When there's no error on the stack, keep only the best version 2016-08-06 21:24:20 -07:00
Max Brunsfeld
c984876e3a Remove stack versions that are worse than the finished tree 2016-08-06 21:22:44 -07:00
Max Brunsfeld
87ca3cb099 Reuse nodes based on state matching, not sentential form validity 
I think that state matching is the only correct strategy for incremental
node reuse that is compatible with the new error recovery algorithm. It's
also simpler than the sentential-form algorithm. With the compressed parse
tables, state matching shouldn't be too conservative of a test.
 2016-07-31 21:31:19 -07:00
Max Brunsfeld
1faa7c60f9 Log stack after pruning halted versions 2016-07-17 10:44:08 -07:00