traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1150 commits 16 branches 98 tags 37 MiB
Commit graph

634 commits

Author SHA1 Message Date
Max Brunsfeld
e686478ad2 Rename stack_merge function to stack_merge_all 2016-05-28 20:24:08 -07:00
Max Brunsfeld
e1a3a1daeb Import error corpus entries from grammar repos 
Now that error recovery requires no input for the grammar author, it shouldn't
be tested in the individual grammar repos.
 2016-05-28 20:12:02 -07:00
Max Brunsfeld
1e353381ff Don't create error node in lexer unless token is completely invalid 
Before, any syntax error would cause the lexer to create an error
leaf node. This could happen even with a valid input, if the parse
stack had split and one particular version of the parse stack
failed to parse.

Now, an error leaf node is only created when the lexer cannot understand
part of the input stream at all. When a normal syntax error occurs,
the lexer just returns a token that is outside of the expected token
set, and the parser handles the unexpected token.
 2016-05-26 14:15:10 -07:00
Max Brunsfeld
a3679fbb1f Distinguish separators from main tokens via a property on transitions 
It was incorrect to store it as a property on the lexical states themselves
 2016-05-19 16:27:25 -07:00
Max Brunsfeld
59712ec492 Clean up lex table generation 2016-05-19 13:25:46 -07:00
Max Brunsfeld
88053cf723 In tests, don’t record allocations while printing debug graphs 2016-05-16 10:44:19 -07:00
Max Brunsfeld
22c550c9d6 Discard tokens after error detection to find the best repair 
* Use GLR stack-splitting to try all numbers of tokens to
  discard until a repair is found.
* Check the validity of repairs by looking at the child trees,
  rather than the statically-computed 'in-progress symbols' list
 2016-05-11 13:49:43 -07:00
Max Brunsfeld
5b74813a5c Refine logic for which tokens to use in error recovery 2016-04-27 14:09:19 -07:00
Max Brunsfeld
fd4c33209e Select ambiguous alternatives by minimizing error size 2016-04-24 00:54:20 -07:00
Max Brunsfeld
0d19f157ed Adjust some spec assertions to reflect finer-grained error recoveries 2016-04-22 10:19:44 -07:00
Max Brunsfeld
cf19b2e58d Make repeat rules left-recursive instead of right recursive 2016-04-18 12:40:14 -07:00
Max Brunsfeld
655d374d0c Recompile test languages if parser.h changes 2016-04-18 11:17:06 -07:00
Max Brunsfeld
cad663b144 Consider multiple error repairs on the same path of the stack 
This changes the API to the stack_iterate function so that you can pop
from the stack without stopping iteration
 2016-04-15 21:28:00 -07:00
Max Brunsfeld
695be5bc79 Merge equivalent stacks in a separate stage of parsing 
* No more automatic merging every time a state is pushed to the stack
* When popping from the stack, the current version is always preserved
 2016-04-10 14:12:24 -07:00
Max Brunsfeld
5ba40f15ad Rename stack heads to versions 2016-04-04 12:25:57 -07:00
Max Brunsfeld
6bce6da1e6 Store verifying flag within parse stack 2016-03-31 12:03:21 -07:00
Max Brunsfeld
e7d3d40a59 Explicitly inform stack pop callback when the stack is exhausted 
Also, pass non-extra tree count as a single value, rather than keeping
track of the extra count and the total separately.
 2016-03-10 11:51:55 -08:00
Max Brunsfeld
240355b04c Make test for allocation failure handling fail more gracefully 2016-03-10 11:36:26 -08:00
Max Brunsfeld
4f726da881 Fix logic for whether to regenerate parsers in specs 2016-03-10 11:35:59 -08:00
Max Brunsfeld
2e35587161 Use new stack_pop_until function for repairing errors 2016-03-07 20:06:46 -08:00
Max Brunsfeld
4348eb89d4 Expose lower stack nodes via pop_until() function 
This callback-based API allows the parser to easily visit each interior node
of the stack when searching for an error repair. It also is a better abstraction
over the stack's DAG implementation than having the public functions for
accessing entries and their successor entries.
 2016-03-07 16:09:34 -08:00
Max Brunsfeld
bc8df9f5c5 Avoid recompiling test languages when possible 2016-03-03 12:05:04 -08:00
Max Brunsfeld
c0595c21c5 Halt stack pops at all error states, not just error trees 2016-03-03 11:05:37 -08:00
Max Brunsfeld
3d516aeeec Give StackPushResult enumerators shorter names 2016-03-03 10:20:05 -08:00
Max Brunsfeld
8a13b5d120 Rename StackPopResult -> StackSlice 2016-03-03 10:16:10 -08:00
Max Brunsfeld
aef7582a2a Start using the forward move to recover from errors 
Some unit tests passing. Corpus tests still failing
 2016-03-02 21:08:42 -08:00
Max Brunsfeld
76d072545d Include out-of-context states starting with non-terminals 2016-03-02 20:58:39 -08:00
Max Brunsfeld
e7abfdd373 Prevent string assertion failures from creating later memory leak errors 2016-03-02 20:58:39 -08:00
Max Brunsfeld
8c01b70ce7 Don't skip tokens that are not the start of any non-terminal 2016-03-02 20:56:05 -08:00
Max Brunsfeld
dee1f697c1 Compute the set of variables that can begin with each terminal symbol 2016-02-25 21:51:52 -08:00
Max Brunsfeld
5a34d74702 Clean up stack 2016-02-25 21:51:39 -08:00
Max Brunsfeld
da2ef7ad35 Store trees in the links between stack nodes, not in the nodes themselves 2016-02-23 17:35:50 -08:00
Max Brunsfeld
b99c257e9e Make test symbol_names array accessible in specs 2016-02-23 00:08:20 -08:00
Max Brunsfeld
9d5a260e5b 🔥 unused variable in tree spec 2016-02-22 21:29:46 -08:00
Max Brunsfeld
f444a715fd Clean up tree array assertions in stack spec 2016-02-22 09:23:25 -08:00
Max Brunsfeld
b113dc8b0f Return a TreeArray from ts_stack_pop 
Since the capacity is now included in the return value, the buffer
can be reused in the ts_parser__accept function. Also, it's just
cleaner to use Array consistently, rather than a separate buffer
and size.
 2016-02-21 22:31:13 -08:00
Max Brunsfeld
17db46eded Add test for node that has only one symbol 2016-02-20 13:38:39 -08:00
Max Brunsfeld
2b35890bbb Add ts_node_symbols() function 2016-02-19 15:41:30 -08:00
Max Brunsfeld
df1d9b2416 Move sexp-generation code to node.c 2016-02-19 15:41:09 -08:00
Max Brunsfeld
3d7df851b5 Rename Vector -> Array 2016-02-17 20:41:29 -08:00
Max Brunsfeld
6fa7eca966 Make vector struct type-safe 2016-02-17 15:30:47 -08:00
Max Brunsfeld
3f08bfb264 Fix build warnings 2016-02-12 14:11:11 -08:00
Max Brunsfeld
e90a425618 Only return one result for each revealed head from ts_stack_pop 2016-02-08 12:08:15 -08:00
Max Brunsfeld
b80a330a74 Fix assorted memory leaks in test code 2016-02-05 12:23:54 -08:00
Max Brunsfeld
b1c8b74e9c Avoid leak caused by earlier corpus spec failures 2016-02-05 11:35:40 -08:00
Max Brunsfeld
2f2ca401be Fix last few leaks, add leak checking to all randomized specs 2016-02-05 11:18:22 -08:00
Max Brunsfeld
c96c4a08e6 Use an object pool for stack nodes, to reduce allocations 
Also, fix some leaks in the case where memory allocation failed during parsing
 2016-02-04 11:19:42 -08:00
Max Brunsfeld
4a7312e514 Fix memory leaks when editing 2016-02-02 12:22:29 -08:00
Max Brunsfeld
e3d65d5cbd Change name of generated parser files 
The previous name, parser.c made it harder to set breakpoints inside of
src/runtime/parser.c in LLDB
 2016-01-29 17:33:15 -08:00
Max Brunsfeld
7c44b0e387 Fix leaked lookahead trees in normal parsing 2016-01-29 17:31:43 -08:00