traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1729 commits 16 branches 98 tags 37 MiB
Commit graph

172 commits

Author SHA1 Message Date
Max Brunsfeld
addeb6c4c1 Allocate and free trees using an object pool 2017-12-27 10:34:29 -08:00
Max Brunsfeld
0e69da37a5 Return a character count from the lexer's get_column method 2017-12-20 16:26:38 -08:00
Max Brunsfeld
a2670a39dc Avoid empty initializer list in global constant 2017-08-07 14:46:23 -07:00
Max Brunsfeld
89e250cd63 Avoid the name 'min' for a helper function 
Windows automatically defines min or something.
 2017-08-07 12:44:33 -07:00
Max Brunsfeld
b98669c7e6 Replace general array_reverse with ts_tree_array_reverse 2017-08-07 12:44:33 -07:00
Max Brunsfeld
ed11ddbd38 Explicitly compare regions between subtrees in get_changed_ranges 2017-08-06 17:16:29 -07:00
Max Brunsfeld
09f4796f6b Get tests passing w/ new alias API 2017-08-01 14:35:34 -07:00
Max Brunsfeld
cb5fe80348 Rename RENAME rule to ALIAS, allow it to create anonymous nodes 2017-07-31 16:41:11 -07:00
Max Brunsfeld
2437287e3e Remove rename symbol when reusing a previously-renamed tree 2017-07-21 16:24:33 -07:00
Max Brunsfeld
cf445da253 Allow renaming hidden rules 2017-07-21 15:58:01 -07:00
Max Brunsfeld
f33421c53e Fix incorrect node renames in the presence of extra tokens 2017-07-18 21:24:34 -07:00
Max Brunsfeld
9a04231ab1 Remove length restriction in external scanner serialization API 2017-07-17 17:12:36 -07:00
Max Brunsfeld
4b40a1ed6c Support anonymous tokens inside of RENAME rules 2017-07-14 10:19:58 -07:00
Max Brunsfeld
b3a72954ff Introduce RENAME rule type 2017-07-13 17:17:22 -07:00
Max Brunsfeld
d8e9d04fe7 Add PREC_DYNAMIC rule for resolving runtime ambiguities 2017-07-06 15:24:45 -07:00
Max Brunsfeld
009d6d1534 Improve heuristics for pruning parse versions based on errors 
* Rewrite the error cost comparison in terms of explicit, discrete
conditions.
* Allow merging versions have different error costs.
* Store the depth of each stack version since the last error. Use this
state to prevent incorrect merging.
* Sort the stack versions in order of preference and put a hard limit on
the version count.
 2017-06-29 15:00:20 -07:00
Max Brunsfeld
66be393b78 Stack - consider empty external token state identical to NULL 2017-06-29 15:00:20 -07:00
Phil Turnbull
5ef9c4d6aa Increase size of ref_counts and add assertions 
This bumps the size of the reference counts from 16- to 32-bit counters to make
it less likely to overflow. Also assert in the retain function that the
reference count didn't overflow.

32-bits seems big enough for non-pathological examples but a more fool-proof
fix may be to bump it to 64-bits.
 2017-06-29 06:39:01 -07:00
Max Brunsfeld
0143bfdad4 Avoid use-after-free of external token states 
Previously, it was possible for references to external token states to
outlive the trees to which those states belonged.

Now, instead of storing references to external token states in the Stack
and in the Lexer, we store references to the external token trees
themselves, and we retain the trees to prevent use-after-free.
 2017-06-27 14:54:27 -07:00
Max Brunsfeld
8ee3f96960 Fix formatting of non-ascii unexpected characters 
Signed-off-by: Philip Turnbull <philipturnbull@github.com>
 2017-06-23 12:08:50 -07:00
Max Brunsfeld
704c2d5907 Fix lookahead_char type in ts_tree_make_error function 2017-04-27 14:49:04 -07:00
Max Brunsfeld
d222dbb9fd Allow lexer to accept tokens that ended at previous positions 
* Track lookahead in each tree
* Add 'mark_end' API that external scanners can use
 2017-03-13 17:06:52 -07:00
Max Brunsfeld
df520635c6 Prevent crash due to huge number of possible paths through parse stack 2017-02-20 14:34:10 -08:00
Max Brunsfeld
c14a776a3d Avoid including trailing extra tokens within error nodes unnecessarily 2017-02-19 21:21:54 -08:00
Max Brunsfeld
343887c1dd Fix miscounting of extra tokens when repairing errors 2017-02-06 17:43:07 -08:00
Max Brunsfeld
12cd2132ff Add test for retrieving last external token state in a Tree 2017-01-04 21:23:04 -08:00
Max Brunsfeld
2fa7b453c8 Restore external scanner's state only after repositioning lexer 
Also, properly identify the leaf node with the external token state
 2016-12-21 13:59:56 -08:00
Max Brunsfeld
1595a02692 Avoid referencing invalid union member in tree_set_children 2016-12-21 12:23:24 -08:00
Max Brunsfeld
e6c82ead2c Start work toward maintaining external scanner's state during incremental parses 2016-12-20 17:06:20 -08:00
Max Brunsfeld
5332fd3418 Fix build warnings 2016-11-19 20:47:43 -08:00
Max Brunsfeld
535879a2bd Represent byte, char and tree counts as 32 bit numbers 
The parser spends the majority of its time allocating and freeing trees and stack nodes.
Also, the memory footprint of the AST is a significant concern when using tree-sitter
with large files. This library is already unlikely to work very well with source files
larger than 4GB, so representing rows, columns, byte lengths and child indices as
unsigned 32 bit integers seems like the right choice.
 2016-11-14 12:19:13 -08:00
Max Brunsfeld
c9dcb29c6f Remove the TS prefix from some internal type/function names 2016-11-09 20:59:05 -08:00
Max Brunsfeld
4106ecda43 Remove logic for recovering from OOM 2016-11-04 09:18:38 -07:00
Max Brunsfeld
5638fea120 Rename length_set_unknown -> length_set_unknown_chars 2016-10-16 21:21:53 -07:00
Max Brunsfeld
eed54d95e1 Merge branch 'master' into changed-ranges 2016-10-16 21:10:25 -07:00
Max Brunsfeld
b3140b2689 Implement ts_document_parse_and_get_changed_ranges 2016-10-15 22:31:21 -07:00
Max Brunsfeld
e149d94ff5 Remove generated parsers' dependency on runtime.h 2016-10-05 14:02:49 -07:00
Max Brunsfeld
ae3b912ddc Escape unexpected characters when pretty-printing error nodes 2016-09-19 13:34:24 -07:00
Max Brunsfeld
fcf9293d35 Use explicit stack for assigning trees' parent pointers 2016-09-19 12:40:45 -07:00
Max Brunsfeld
00528e50ce Change edit API to be byte-based 2016-09-13 13:08:52 -07:00
Max Brunsfeld
cc62fe0375 Represent Lengths in terms of Points 2016-09-09 21:11:02 -07:00
Max Brunsfeld
591fcc980c Update trees' offsets when editing them 2016-09-08 17:54:51 -07:00
Max Brunsfeld
820cbece20 Pretty-print unexpected EOF errors properly 2016-09-03 22:45:18 -07:00
Max Brunsfeld
0faae52132 Fix some inconsistencies in error cost calculation 
Signed-off-by: Nathan Sobo <nathan@github.com>
 2016-08-31 10:51:59 -07:00
Max Brunsfeld
1d0f6c3cc0 Rename error_size -> error_cost 2016-08-30 11:09:12 -07:00
Max Brunsfeld
31d1160e21 Base error costs on top-level trees skipped and lines of text skipped 
Rather than on the total number of tokens skipped
 2016-08-29 17:06:23 -07:00
Max Brunsfeld
f8c2e9d908 Include tree's original parse state in debug graphs 2016-07-31 21:35:53 -07:00
Max Brunsfeld
d554fab5b5 Remove unused tree state constant 2016-06-27 14:39:12 -07:00
Max Brunsfeld
df09f37a6c Tweak parse log messages 2016-06-23 11:42:43 -07:00
Max Brunsfeld
09b019c530 Fix test for invalid blank input 2016-06-23 09:24:26 -07:00