traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1565 commits 16 branches 98 tags 37 MiB
Commit graph

617 commits

Author SHA1 Message Date
Max Brunsfeld
4b40a1ed6c Support anonymous tokens inside of RENAME rules 2017-07-14 10:19:58 -07:00
Max Brunsfeld
b3a72954ff Introduce RENAME rule type 2017-07-13 17:17:22 -07:00
Max Brunsfeld
7293e6f0cc Fix compile warnings 2017-07-12 22:08:36 -07:00
Max Brunsfeld
d8e9d04fe7 Add PREC_DYNAMIC rule for resolving runtime ambiguities 2017-07-06 15:24:45 -07:00
Max Brunsfeld
8f028ebf68 Avoid deep tree comparison when both trees have errors 2017-07-05 17:33:35 -07:00
Max Brunsfeld
782bf48772 Don't do skip_preceding_subtrees recovery when there are lots of versions 2017-07-05 15:34:19 -07:00
Max Brunsfeld
17bc3dfaf7 Add a benchmark command 
This command measures the speed of parsing each grammar's examples.
It also uses each grammar to parse all of the *other* grammars' examples
in order to measure error recovery performance with fairly large files.
 2017-07-05 14:14:38 -07:00
Max Brunsfeld
d322f0b6a7 🎨 2017-07-04 21:59:54 -07:00
Max Brunsfeld
e7ccd9c17c Put back check for better existing versions during recover action 
When checking for better existing versions, only kill the current
version if there is a better version earlier in the rotation.
 2017-07-03 12:27:23 -07:00
Max Brunsfeld
f93f78ef2d Remove version-pruning criteria based on pushed node count 2017-07-02 23:42:23 -07:00
Max Brunsfeld
f722923493 Limit the search depth for skip_preceding_trees recovery 2017-06-30 17:49:09 -07:00
Max Brunsfeld
89e5037f01 Manually tail-call-optimize stack_node_release function 2017-06-30 17:49:09 -07:00
Max Brunsfeld
eccb3893eb Prune unneeded stack versions based on a depth criteria 2017-06-30 17:49:09 -07:00
Max Brunsfeld
d6579956f5 Enforce a hard version count limit during the recovery action 2017-06-30 17:49:09 -07:00
Max Brunsfeld
061fba6b92 🎨 Just call it 'inline' 2017-06-29 16:49:59 -07:00
Max Brunsfeld
a89322c5f1 Remove unneeded parameters from public interface of stack_iterate callback 2017-06-29 16:43:56 -07:00
Max Brunsfeld
009d6d1534 Improve heuristics for pruning parse versions based on errors 
* Rewrite the error cost comparison in terms of explicit, discrete
conditions.
* Allow merging versions have different error costs.
* Store the depth of each stack version since the last error. Use this
state to prevent incorrect merging.
* Sort the stack versions in order of preference and put a hard limit on
the version count.
 2017-06-29 15:00:20 -07:00
Max Brunsfeld
445be0736a Clean up ts_stack_push function 2017-06-29 15:00:20 -07:00
Max Brunsfeld
66be393b78 Stack - consider empty external token state identical to NULL 2017-06-29 15:00:20 -07:00
Phil Turnbull
5ef9c4d6aa Increase size of ref_counts and add assertions 
This bumps the size of the reference counts from 16- to 32-bit counters to make
it less likely to overflow. Also assert in the retain function that the
reference count didn't overflow.

32-bits seems big enough for non-pathological examples but a more fool-proof
fix may be to bump it to 64-bits.
 2017-06-29 06:39:01 -07:00
Max Brunsfeld
dfd7b1f5f6 Consolidate memory management logic in Stack 2017-06-27 16:17:24 -07:00
Max Brunsfeld
0143bfdad4 Avoid use-after-free of external token states 
Previously, it was possible for references to external token states to
outlive the trees to which those states belonged.

Now, instead of storing references to external token states in the Stack
and in the Lexer, we store references to the external token trees
themselves, and we retain the trees to prevent use-after-free.
 2017-06-27 14:54:27 -07:00
Max Brunsfeld
f678018d3d Avoid use-after-free when copying stack iterators 2017-06-27 14:54:27 -07:00
Max Brunsfeld
f62ee5a0f3 Fix OOB reads at ends of chunks 
Signed-off-by: Philip Turnbull <philipturnbull@github.com>
 2017-06-23 12:09:16 -07:00
Max Brunsfeld
8ee3f96960 Fix formatting of non-ascii unexpected characters 
Signed-off-by: Philip Turnbull <philipturnbull@github.com>
 2017-06-23 12:08:50 -07:00
Max Brunsfeld
513edec7c1 Merge pull request #77 from philipturnbull/scan-build-fixes 
Fix errors found by scan-build
 2017-06-20 10:15:20 -07:00
Max Brunsfeld
599367d36d Always recur into error nodes when reporting changed ranges 2017-06-15 17:06:48 -07:00
Max Brunsfeld
c66fddd3aa Add TSInput option to measure columns in bytes not characters 2017-06-15 16:35:34 -07:00
Phil Turnbull
cfca764d48 Root can never be NULL in this context 2017-06-15 07:47:16 -04:00
Max Brunsfeld
b862db766e Merge remote-tracking branch 'origin/master' into update-fixture-grammars 2017-06-14 17:11:44 -07:00
Phil Turnbull
d1b19e8196 Prevent NULL pointer dereference in parser__accept 
parser__select_tree can return true if 'left != NULL' and 'right == NULL' which
will later cause a NULL ptr deref:

src/runtime/parser.c:842:14: warning: Access to field 'ref_count' results in a dereference of a null pointer (loaded from variable 'root')
      assert(root->ref_count > 0);
             ^~~~~~~~~~~~~~~
 2017-06-14 11:12:06 -04:00
Phil Turnbull
da099d0bbe Prevent NULL pointer dereference in parser__repair_error_callback 
Because repair_reduction_count is unsigned, the default of '-1' is 0xffffffff
and will cause the loop to be entered if repair_reduction_count is NULL:

src/runtime/parser.c:691:11: warning: Dereference of null pointer
      if (repair_reductions[j].params.symbol == repair->symbol) {
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 2017-06-14 11:12:06 -04:00
Max Brunsfeld
7b401de5a6 Don't use pointer equality to compare external token states 2017-05-03 09:57:09 -07:00
Max Brunsfeld
e8a9bb7a51 🎨 Extract parser__halt_parse function 2017-05-01 14:41:55 -07:00
Max Brunsfeld
74f5ceddf7 Fix parsing of valid code with halt_on_error flag set 
Signed-off-by: Tim Clem <timothy.clem@gmail.com>
 2017-05-01 14:25:25 -07:00
Max Brunsfeld
a98d449d88 Add an option to immediately halt on syntax error 2017-05-01 13:50:49 -07:00
Max Brunsfeld
704c2d5907 Fix lookahead_char type in ts_tree_make_error function 2017-04-27 14:49:04 -07:00
Timothy Clem
91558f0a0e utf8proc_iterate can set codepoint_ref to -1 and returns negative error 2017-04-27 14:46:36 -07:00
Rob Rix
3a888b1623 Define a function providing the type of a given symbol. 2017-04-12 09:47:51 -04:00
Max Brunsfeld
1f908324dc Prevent infinite loop in skip_preceding_trees error recovery strategy 2017-03-21 12:14:44 -07:00
Max Brunsfeld
7e13eac296 Fix lookahead_char type in ts_tree_make_error function 2017-03-21 11:05:48 -07:00
Max Brunsfeld
63fb041961 Merge remote-tracking branch 'origin/check-utf8proc_iterate-return' into update-fixture-grammars 2017-03-21 09:59:35 -07:00
Timothy Clem
f394a48c0b utf8proc_iterate can set codepoint_ref to -1 and returns negative error 2017-03-20 16:54:19 -07:00
Max Brunsfeld
ed31e82ee6 Skip empty tokens when recovering from errors 2017-03-19 22:20:59 -07:00
Max Brunsfeld
d222dbb9fd Allow lexer to accept tokens that ended at previous positions 
* Track lookahead in each tree
* Add 'mark_end' API that external scanners can use
 2017-03-13 17:06:52 -07:00
Max Brunsfeld
df520635c6 Prevent crash due to huge number of possible paths through parse stack 2017-02-20 14:34:10 -08:00
Max Brunsfeld
cefc57fe86 Move error cost comparisons into their own source file 2017-02-19 21:54:06 -08:00
Max Brunsfeld
5b4e6df3ff Don't mark error nodes created in the error state as extras 2017-02-19 21:54:06 -08:00
Max Brunsfeld
c14a776a3d Avoid including trailing extra tokens within error nodes unnecessarily 2017-02-19 21:21:54 -08:00
Max Brunsfeld
135d8ef4e0 Merge pull request #58 from tree-sitter/reduce-error-recovery-branching 
Reduce the branching factor of the parse stack during error recovery
 2017-02-18 11:34:09 -08:00