traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1756 commits 16 branches 98 tags 37 MiB
Commit graph

127 commits

Author SHA1 Message Date
Max Brunsfeld
299a146b66 Balance repetition trees after parsing 2018-02-12 11:41:56 -08:00
Max Brunsfeld
8c29841adf Represent repetitions with associative structure 2018-02-12 11:41:56 -08:00
Max Brunsfeld
46dcd53090 Do not insert missing tokens if halt_on_error option is passed 2018-01-24 14:04:55 -08:00
Max Brunsfeld
b520bdd2d5
Merge pull request #126 from tree-sitter/mb-fix-epsilon-rule-loophole 
Don't allow an epsilon start rule if it is used in other rules
 2018-01-23 17:19:55 -08:00
Max Brunsfeld
2e4f76c164 Don't allow an epsilon start rule if it is used in other rules 2018-01-23 17:05:28 -08:00
Max Brunsfeld
315dff3285 Add an API for getting a node's child index 2018-01-09 14:01:36 -08:00
Max Brunsfeld
f653f2b3bb Add ts_node_first_{child,named_child}_for_byte methods 2018-01-09 13:44:59 -08:00
Max Brunsfeld
13adfe4927 Update error corpus 2017-12-29 16:21:04 -08:00
Max Brunsfeld
6304a3bcd1 Make it easier to run tests with debug graphs 2017-12-28 12:41:23 -08:00
Max Brunsfeld
addeb6c4c1 Allocate and free trees using an object pool 2017-12-27 10:34:29 -08:00
Max Brunsfeld
0e69da37a5 Return a character count from the lexer's get_column method 2017-12-20 16:26:38 -08:00
Max Brunsfeld
e5851fd9b9 Don't use non-existent \a syntax in test grammars 2017-12-13 12:21:28 -08:00
Max Brunsfeld
f426b61e7c Fix expectation around preproc directive in C error test 2017-12-13 12:21:13 -08:00
Max Brunsfeld
fbcefe25f7 Avoid creating external tokens that start after they end 2017-12-07 11:50:27 -08:00
Max Brunsfeld
90629bd45a Add some assertions to the fixture grammar tests 2017-12-07 11:50:27 -08:00
Max Brunsfeld
493db39363 Never move the start rule of a grammar into the lexical grammar 
This preserves a useful invariant that the root node of the AST is never
a token.
 2017-12-07 11:50:27 -08:00
Max Brunsfeld
36c2b685b9 Always invalidate old chunk of text when parsing after an edit 2017-10-04 15:09:46 -07:00
Max Brunsfeld
c0073c5b72 Update error corpus to reflect C grammar changes 2017-10-04 15:06:12 -07:00
Max Brunsfeld
d342b61ede Re-enable JS fuzzing example test 2017-09-14 11:39:08 -07:00
Max Brunsfeld
9d67a98510 Merge pull request #103 from tree-sitter/python-assertion-failure 
Assertion failure in parser__advance
 2017-09-14 11:38:22 -07:00
Max Brunsfeld
d291af9a31 Refactor error comparisons 
* Deal with mergeability outside of error comparison function
* Make `better_version_exists` function pure (don't halt other versions
as a side effect).
* Tweak error comparison logic

Signed-off-by: Rick Winfrey <rewinfrey@github.com>
 2017-09-13 16:38:15 -07:00
Phil Turnbull
d9a0fbc210 Add testcase for parser__advance assertion failure 
The python testcase decodes to:
```
00000000  35 63 6f 6e 88 2c 29 33  2c 2c 2c 2c 63 6f 6e 88  |5con.,)3,,,,con.|
00000010  2c 2a 2c 3a 35 63 6f 6e  2c                       |,*,:5con,|
```

which triggers:
```
Assertion failed: ((uint32_t)0 < (&reduction.slices)->size), function parser__advance, file src/runtime/parser.c, line 1202.
```
 2017-09-13 13:25:31 -04:00
Max Brunsfeld
65ed4281d4 Exclude zeros from speeds reported in benchmarks 2017-09-12 16:30:38 -07:00
Max Brunsfeld
99d048e016 Simplify error recovery; eliminate recovery states 
The previous approach to error recovery relied on special error-recovery
states in the parse table. For each token T, there was an error recovery
state in which the parser looked for *any* token that could follow T.
Unfortunately, sometimes the set of tokens that could follow T contained
conflicts. For example, in JS, the token '}' can be followed by the
open-ended 'template_chars' token, but also by ordinary tokens like
'identifier'. So with the old algorithm, when recovering from an
unexpected '}' token, the lexer had no way to distinguish identifiers
from template_chars.

This commit drops the error recovery states. Instead, when we encounter
an unexpected token T, we recover from the error by finding a previous
state S in the stack in which T would be valid, popping all of the nodes
after S, and wrapping them in an error.

This way, the lexer is always invoked in a normal parse state, in which
it is looking for a non-conflicting set of tokens. Eliminating the error
recovery states also shrinks the lex state machine significantly.

Signed-off-by: Rick Winfrey <rewinfrey@github.com>
 2017-09-11 15:22:52 -07:00
Max Brunsfeld
8b3941764f Make outstanding_allocation_indices return a vector, not a set 2017-09-07 17:48:44 -07:00
Max Brunsfeld
9d668c5004 Move incompatible token map into LexTableBuilder 2017-08-31 15:46:37 -07:00
Max Brunsfeld
4daf22ba0c Read files in binary mode in tests 2017-08-09 10:07:03 -07:00
Max Brunsfeld
90eef13aeb Fix windows directory listing function 2017-08-08 22:21:39 -07:00
Max Brunsfeld
d4b2c58dc8 Fix another hard-coded / as a path separator 2017-08-08 22:08:27 -07:00
Max Brunsfeld
f6325746aa Provide symbol metadata with dummy language in stack test 2017-08-08 17:47:24 -07:00
Max Brunsfeld
d0dc164013 Disable the default behavior of printing a logo, Microsoft. 2017-08-08 17:35:50 -07:00
Max Brunsfeld
0919f5588b Disable fuzzing example test for now 2017-08-08 17:35:16 -07:00
Max Brunsfeld
0ba6188bab Construct paths portably in test_grammars.cc 2017-08-08 17:25:43 -07:00
Max Brunsfeld
7587353ab6 Avoid unicode literals in tests 
MSVC tries to normalize them based on the current locale.
 2017-08-08 17:10:08 -07:00
Max Brunsfeld
17b58f41e1 Disable optimization when compiling grammars during tests 2017-08-08 17:09:34 -07:00
Max Brunsfeld
17310769a4 Fix usage of .so instead of .dll on windows 2017-08-08 15:20:38 -07:00
Max Brunsfeld
bb7889fac5 Don't rely on PWD env var on windows 2017-08-08 15:04:27 -07:00
Max Brunsfeld
34b5340d71 Fix paths to corpus files on windows 2017-08-08 14:06:11 -07:00
Max Brunsfeld
cc7277fd7d Avoid using IsNull bandit assertion 2017-08-08 12:52:35 -07:00
Max Brunsfeld
b43ae2826b Use C++ stdlib for random number generation 2017-08-08 12:42:49 -07:00
Max Brunsfeld
fc0f49e4ee Add windows implementations of some IO-related test helpers 2017-08-08 12:21:17 -07:00
Max Brunsfeld
947c161c2f Use a constructor rather than aggregate initialization for Production 2017-08-08 10:41:54 -07:00
Max Brunsfeld
230f89d0ff Fix build warnings in tests 2017-08-07 12:19:10 -07:00
Max Brunsfeld
94dc703bfc Require that grammars' start rules be visible 2017-08-04 17:07:37 -07:00
Max Brunsfeld
1dca3a0b58 Simplify parse version reordering 2017-08-04 14:51:14 -07:00
Max Brunsfeld
e5c3bf742d Update fixture grammars 2017-08-03 16:32:39 -07:00
Max Brunsfeld
84e4114f79 Allow conflicts involving repeat rules to be whitelisted, via their parent rule 2017-08-03 15:18:29 -07:00
Max Brunsfeld
119c67dd78 Fix conflict reporting for shift/reduce conflicts w/ multiple reductions 
We were failing to rule out shift actions with lower precedence.

Signed-off-by: Philip Turnbull <philipturnbull@github.com>
 2017-08-02 15:13:30 -07:00
Max Brunsfeld
09f4796f6b Get tests passing w/ new alias API 2017-08-01 14:35:34 -07:00
Max Brunsfeld
cb5fe80348 Rename RENAME rule to ALIAS, allow it to create anonymous nodes 2017-07-31 16:41:11 -07:00