Max Brunsfeld
c966af0412
Start work on external tokens
2016-12-02 16:24:19 -08:00
Max Brunsfeld
e7217f1bac
Clean up some methods in parser.c
2016-11-14 17:25:55 -08:00
Max Brunsfeld
535879a2bd
Represent byte, char and tree counts as 32 bit numbers
...
The parser spends the majority of its time allocating and freeing trees and stack nodes.
Also, the memory footprint of the AST is a significant concern when using tree-sitter
with large files. This library is already unlikely to work very well with source files
larger than 4GB, so representing rows, columns, byte lengths and child indices as
unsigned 32 bit integers seems like the right choice.
2016-11-14 12:19:13 -08:00
Max Brunsfeld
8edb8df530
Remove extraneous Language methods
2016-11-14 10:35:33 -08:00
Max Brunsfeld
1118a9142a
Introduce Symbol::Index type alias
2016-11-14 10:25:26 -08:00
Max Brunsfeld
1fddb124b3
Remove stray LOG_STACK() call
2016-11-14 09:32:05 -08:00
Max Brunsfeld
fad7294ba4
Store shift states for non-terminals directly in the main parse table
2016-11-14 08:36:06 -08:00
Max Brunsfeld
8d9c261e3a
Don't include reduce actions for nonterminal lookaheads
2016-11-10 11:33:37 -08:00
Max Brunsfeld
c9dcb29c6f
Remove the TS prefix from some internal type/function names
2016-11-09 20:59:05 -08:00
Max Brunsfeld
4106ecda43
Remove logic for recovering from OOM
2016-11-04 09:18:38 -07:00
Max Brunsfeld
e53beb66c9
Avoid anonymous nested struct to silence override-init warnings
2016-10-26 11:10:56 -07:00
Max Brunsfeld
eed54d95e1
Merge branch 'master' into changed-ranges
2016-10-16 21:10:25 -07:00
Max Brunsfeld
b3140b2689
Implement ts_document_parse_and_get_changed_ranges
2016-10-15 22:31:21 -07:00
Max Brunsfeld
e149d94ff5
Remove generated parsers' dependency on runtime.h
2016-10-05 14:02:49 -07:00
Max Brunsfeld
3014101104
Fix inconsistencies in nodes sizes after edits
2016-09-19 13:35:08 -07:00
Max Brunsfeld
fcf9293d35
Use explicit stack for assigning trees' parent pointers
2016-09-19 12:40:45 -07:00
Max Brunsfeld
00528e50ce
Change edit API to be byte-based
2016-09-13 13:08:52 -07:00
Max Brunsfeld
cc62fe0375
Represent Lengths in terms of Points
2016-09-09 21:11:02 -07:00
Max Brunsfeld
096ac2d4b6
Rename ts_document_set_debugger -> ts_document_set_logger
2016-09-06 17:40:26 -07:00
Max Brunsfeld
587bbe998c
Don't reuse non-trivial subtrees when stack is split
2016-09-06 10:23:26 -07:00
Max Brunsfeld
b76574e01c
Handle ambiguities between extra and non-extra tokens using normal GLR splitting
2016-09-06 10:22:16 -07:00
Max Brunsfeld
4f0c83ba01
Move logic for lexical error handling outside of lexer functions
...
This way, less logic needs to be exposed in parser.h
2016-09-03 23:40:57 -07:00
Max Brunsfeld
1c52c30111
Fix unexpected EOF errors getting lost
2016-09-03 22:46:14 -07:00
Max Brunsfeld
e0b0e29a2b
Update parse count correctly when repairing errors & undoing reductions
2016-09-01 10:04:20 -07:00
Max Brunsfeld
e04ac7b13d
Ensure the correct error stats are used when aborting versions immediately
2016-08-31 14:29:34 -07:00
Max Brunsfeld
0faae52132
Fix some inconsistencies in error cost calculation
...
Signed-off-by: Nathan Sobo <nathan@github.com>
2016-08-31 10:51:59 -07:00
Max Brunsfeld
1d0f6c3cc0
Rename error_size -> error_cost
2016-08-30 11:09:12 -07:00
Max Brunsfeld
52ccebbf80
Rename error_depth -> error_count
2016-08-30 09:44:40 -07:00
Max Brunsfeld
1d617ab5e0
Allow reductions based on error token, skipping some preceding content
2016-08-29 17:34:51 -07:00
Max Brunsfeld
e7ed92f1df
Replace ts_parser -> parser
2016-08-29 12:08:58 -07:00
Max Brunsfeld
1b8843dd41
Perform all possible reductions recursively upon detecting an error
2016-08-29 11:23:35 -07:00
Max Brunsfeld
a3c943d88e
Fix failure to find repairs when extra tokens precede the detection point
2016-08-29 11:23:35 -07:00
Max Brunsfeld
dcf3beb393
Abandon parse versions with > 1 error more than the best version
2016-08-29 09:31:52 -07:00
Max Brunsfeld
4e6e996a76
Simplify skipping of reduced subtrees after errors
2016-08-15 17:37:00 -07:00
Max Brunsfeld
c067b30ce8
Fix handling of errors while in error state
2016-08-06 21:25:35 -07:00
Max Brunsfeld
b561000b2a
When there's no error on the stack, keep only the best version
2016-08-06 21:24:20 -07:00
Max Brunsfeld
c984876e3a
Remove stack versions that are worse than the finished tree
2016-08-06 21:22:44 -07:00
Max Brunsfeld
87ca3cb099
Reuse nodes based on state matching, not sentential form validity
...
I think that state matching is the only correct strategy for incremental
node reuse that is compatible with the new error recovery algorithm. It's
also simpler than the sentential-form algorithm. With the compressed parse
tables, state matching shouldn't be too conservative of a test.
2016-07-31 21:31:19 -07:00
Max Brunsfeld
1faa7c60f9
Log stack after pruning halted versions
2016-07-17 10:44:08 -07:00
Max Brunsfeld
c3a242740b
Allow lookahead to be broken down further after performing reductions
2016-07-04 12:20:23 -07:00
Max Brunsfeld
9c37c06ec7
Remove unnecessary parser debugger accessors
2016-07-04 12:20:23 -07:00
Max Brunsfeld
d554fab5b5
Remove unused tree state constant
2016-06-27 14:39:12 -07:00
Max Brunsfeld
8c26d99353
Store error recovery actions in the normal parse table
2016-06-27 14:07:47 -07:00
Max Brunsfeld
bf245f016c
Check for better stack versions before skipping a node after reducing it
2016-06-26 22:51:48 -07:00
Max Brunsfeld
6fd3edceae
Fix logic for inserting leading & trailing extras into root node on acceptance
2016-06-26 11:57:42 -07:00
Max Brunsfeld
9972709e43
Allow error recovery to skip non-terminal nodes after error detection
2016-06-24 10:28:05 -07:00
Max Brunsfeld
76c6e47d49
Fix assignment of parser split condition
2016-06-23 13:41:38 -07:00
Max Brunsfeld
df09f37a6c
Tweak parse log messages
2016-06-23 11:42:43 -07:00
Max Brunsfeld
c6e9b32d3f
Print all the same parse log messages for both debugging methods
2016-06-22 22:36:11 -07:00
Max Brunsfeld
58b1d88950
Clean up breakdown_top_of_stack function
2016-06-22 22:32:38 -07:00
