traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1852 commits 16 branches 98 tags 37 MiB
Commit graph

190 commits

Author SHA1 Message Date
Max Brunsfeld
59694e60fa Rename ts_tree_assign_parents -> ts_tree_balance 2018-05-10 09:13:46 -07:00
Max Brunsfeld
78d158899e Use atomic operations for updating syntax tree reference counts 
Co-Authored-By: Rick Winfrey <rewinfrey@github.com>
 2018-05-09 17:36:03 -07:00
Max Brunsfeld
5fa6d39578 Fix alias handling in ts_tree_string 
Co-Authored-By: Rick Winfrey <rewinfrey@github.com>
 2018-05-09 14:14:56 -07:00
Max Brunsfeld
973e4a44f0 Start work on removing parent pointers 
Co-Authored-By: Rick Winfrey <rewinfrey@github.com>
 2018-05-09 12:22:19 -07:00
Max Brunsfeld
f00d2ade46 Remove unused function 2018-04-09 19:37:53 -07:00
Max Brunsfeld
b0b8279c14 Preallocate tree arrays when popping from the stack 2018-04-09 18:14:17 -07:00
Max Brunsfeld
3672a8ad87 Remove unused tree method 2018-04-09 12:29:22 -07:00
Max Brunsfeld
33820253e8 Make stack_print_dot_graph function take a language as an argument 2018-04-09 12:29:22 -07:00
Max Brunsfeld
379a2fd121 Incrementally build a tree of skipped tokens 
Rather than pushing them to the stack individually
 2018-04-09 12:29:22 -07:00
Max Brunsfeld
94ed1b6964 Make array_splice take an array, not a pointer and length 2018-04-06 13:29:43 -07:00
Max Brunsfeld
09be0b6ef5 Store trees' children in TreeArrays, not w/ separate pointer and length 2018-04-06 13:26:18 -07:00
Max Brunsfeld
1d9d6f37ad Introduce an error cost per error instance to favor fewer errors 2018-04-06 13:26:18 -07:00
Max Brunsfeld
80f856cef5 Maintain a total node count on every tree 
This simplifies (and fixes bugs in) the parse stack's tracking of its
total node count since the last error, which is needed for error
recovery.
 2018-04-06 13:26:18 -07:00
Max Brunsfeld
16a45d4aa4 Fix hole in logic for terminating tree balancing 
It's important that the repetition nodes have a child count of 2,
because we assign to their second child. We could maybe generalize
this to allow balancing in the presence of 'extra' nodes like comments
and errors, but this might be complicated.
 2018-02-16 12:44:30 -08:00
Max Brunsfeld
134c455b80 Simplify logic for terminating tree balancing 2018-02-12 12:13:21 -08:00
Max Brunsfeld
299a146b66 Balance repetition trees after parsing 2018-02-12 11:41:56 -08:00
Max Brunsfeld
919c9d8715 Ensure root node has a null parent 2018-01-23 17:20:15 -08:00
Max Brunsfeld
d3c85f288d Start work on repairing errors by inserting missing tokens 2017-12-29 15:11:00 -08:00
Max Brunsfeld
addeb6c4c1 Allocate and free trees using an object pool 2017-12-27 10:34:29 -08:00
Max Brunsfeld
0e69da37a5 Return a character count from the lexer's get_column method 2017-12-20 16:26:38 -08:00
Max Brunsfeld
a2670a39dc Avoid empty initializer list in global constant 2017-08-07 14:46:23 -07:00
Max Brunsfeld
89e250cd63 Avoid the name 'min' for a helper function 
Windows automatically defines min or something.
 2017-08-07 12:44:33 -07:00
Max Brunsfeld
b98669c7e6 Replace general array_reverse with ts_tree_array_reverse 2017-08-07 12:44:33 -07:00
Max Brunsfeld
ed11ddbd38 Explicitly compare regions between subtrees in get_changed_ranges 2017-08-06 17:16:29 -07:00
Max Brunsfeld
09f4796f6b Get tests passing w/ new alias API 2017-08-01 14:35:34 -07:00
Max Brunsfeld
cb5fe80348 Rename RENAME rule to ALIAS, allow it to create anonymous nodes 2017-07-31 16:41:11 -07:00
Max Brunsfeld
2437287e3e Remove rename symbol when reusing a previously-renamed tree 2017-07-21 16:24:33 -07:00
Max Brunsfeld
cf445da253 Allow renaming hidden rules 2017-07-21 15:58:01 -07:00
Max Brunsfeld
f33421c53e Fix incorrect node renames in the presence of extra tokens 2017-07-18 21:24:34 -07:00
Max Brunsfeld
9a04231ab1 Remove length restriction in external scanner serialization API 2017-07-17 17:12:36 -07:00
Max Brunsfeld
4b40a1ed6c Support anonymous tokens inside of RENAME rules 2017-07-14 10:19:58 -07:00
Max Brunsfeld
b3a72954ff Introduce RENAME rule type 2017-07-13 17:17:22 -07:00
Max Brunsfeld
d8e9d04fe7 Add PREC_DYNAMIC rule for resolving runtime ambiguities 2017-07-06 15:24:45 -07:00
Max Brunsfeld
009d6d1534 Improve heuristics for pruning parse versions based on errors 
* Rewrite the error cost comparison in terms of explicit, discrete
conditions.
* Allow merging versions have different error costs.
* Store the depth of each stack version since the last error. Use this
state to prevent incorrect merging.
* Sort the stack versions in order of preference and put a hard limit on
the version count.
 2017-06-29 15:00:20 -07:00
Max Brunsfeld
66be393b78 Stack - consider empty external token state identical to NULL 2017-06-29 15:00:20 -07:00
Phil Turnbull
5ef9c4d6aa Increase size of ref_counts and add assertions 
This bumps the size of the reference counts from 16- to 32-bit counters to make
it less likely to overflow. Also assert in the retain function that the
reference count didn't overflow.

32-bits seems big enough for non-pathological examples but a more fool-proof
fix may be to bump it to 64-bits.
 2017-06-29 06:39:01 -07:00
Max Brunsfeld
0143bfdad4 Avoid use-after-free of external token states 
Previously, it was possible for references to external token states to
outlive the trees to which those states belonged.

Now, instead of storing references to external token states in the Stack
and in the Lexer, we store references to the external token trees
themselves, and we retain the trees to prevent use-after-free.
 2017-06-27 14:54:27 -07:00
Max Brunsfeld
8ee3f96960 Fix formatting of non-ascii unexpected characters 
Signed-off-by: Philip Turnbull <philipturnbull@github.com>
 2017-06-23 12:08:50 -07:00
Max Brunsfeld
704c2d5907 Fix lookahead_char type in ts_tree_make_error function 2017-04-27 14:49:04 -07:00
Max Brunsfeld
d222dbb9fd Allow lexer to accept tokens that ended at previous positions 
* Track lookahead in each tree
* Add 'mark_end' API that external scanners can use
 2017-03-13 17:06:52 -07:00
Max Brunsfeld
df520635c6 Prevent crash due to huge number of possible paths through parse stack 2017-02-20 14:34:10 -08:00
Max Brunsfeld
c14a776a3d Avoid including trailing extra tokens within error nodes unnecessarily 2017-02-19 21:21:54 -08:00
Max Brunsfeld
343887c1dd Fix miscounting of extra tokens when repairing errors 2017-02-06 17:43:07 -08:00
Max Brunsfeld
12cd2132ff Add test for retrieving last external token state in a Tree 2017-01-04 21:23:04 -08:00
Max Brunsfeld
2fa7b453c8 Restore external scanner's state only after repositioning lexer 
Also, properly identify the leaf node with the external token state
 2016-12-21 13:59:56 -08:00
Max Brunsfeld
1595a02692 Avoid referencing invalid union member in tree_set_children 2016-12-21 12:23:24 -08:00
Max Brunsfeld
e6c82ead2c Start work toward maintaining external scanner's state during incremental parses 2016-12-20 17:06:20 -08:00
Max Brunsfeld
5332fd3418 Fix build warnings 2016-11-19 20:47:43 -08:00
Max Brunsfeld
535879a2bd Represent byte, char and tree counts as 32 bit numbers 
The parser spends the majority of its time allocating and freeing trees and stack nodes.
Also, the memory footprint of the AST is a significant concern when using tree-sitter
with large files. This library is already unlikely to work very well with source files
larger than 4GB, so representing rows, columns, byte lengths and child indices as
unsigned 32 bit integers seems like the right choice.
 2016-11-14 12:19:13 -08:00
Max Brunsfeld
c9dcb29c6f Remove the TS prefix from some internal type/function names 2016-11-09 20:59:05 -08:00