Max Brunsfeld
1e353381ff
Don't create error node in lexer unless token is completely invalid
...
Before, any syntax error would cause the lexer to create an error
leaf node. This could happen even with a valid input, if the parse
stack had split and one particular version of the parse stack
failed to parse.
Now, an error leaf node is only created when the lexer cannot understand
part of the input stream at all. When a normal syntax error occurs,
the lexer just returns a token that is outside of the expected token
set, and the parser handles the unexpected token.
2016-05-26 14:15:10 -07:00
Max Brunsfeld
22c550c9d6
Discard tokens after error detection to find the best repair
...
* Use GLR stack-splitting to try all numbers of tokens to
discard until a repair is found.
* Check the validity of repairs by looking at the child trees,
rather than the statically-computed 'in-progress symbols' list
2016-05-11 13:49:43 -07:00
Max Brunsfeld
fd4c33209e
Select ambiguous alternatives by minimizing error size
2016-04-24 00:54:20 -07:00
Max Brunsfeld
0d19f157ed
Adjust some spec assertions to reflect finer-grained error recoveries
2016-04-22 10:19:44 -07:00
Max Brunsfeld
cad663b144
Consider multiple error repairs on the same path of the stack
...
This changes the API to the stack_iterate function so that you can pop
from the stack without stopping iteration
2016-04-15 21:28:00 -07:00
Max Brunsfeld
695be5bc79
Merge equivalent stacks in a separate stage of parsing
...
* No more automatic merging every time a state is pushed to the stack
* When popping from the stack, the current version is always preserved
2016-04-10 14:12:24 -07:00
Max Brunsfeld
5ba40f15ad
Rename stack heads to versions
2016-04-04 12:25:57 -07:00
Max Brunsfeld
6bce6da1e6
Store verifying flag within parse stack
2016-03-31 12:03:21 -07:00
Max Brunsfeld
e7d3d40a59
Explicitly inform stack pop callback when the stack is exhausted
...
Also, pass non-extra tree count as a single value, rather than keeping
track of the extra count and the total separately.
2016-03-10 11:51:55 -08:00
Max Brunsfeld
240355b04c
Make test for allocation failure handling fail more gracefully
2016-03-10 11:36:26 -08:00
Max Brunsfeld
2e35587161
Use new stack_pop_until function for repairing errors
2016-03-07 20:06:46 -08:00
Max Brunsfeld
4348eb89d4
Expose lower stack nodes via pop_until() function
...
This callback-based API allows the parser to easily visit each interior node
of the stack when searching for an error repair. It also is a better abstraction
over the stack's DAG implementation than having the public functions for
accessing entries and their successor entries.
2016-03-07 16:09:34 -08:00
Max Brunsfeld
bc8df9f5c5
Avoid recompiling test languages when possible
2016-03-03 12:05:04 -08:00
Max Brunsfeld
c0595c21c5
Halt stack pops at all error states, not just error trees
2016-03-03 11:05:37 -08:00
Max Brunsfeld
3d516aeeec
Give StackPushResult enumerators shorter names
2016-03-03 10:20:05 -08:00
Max Brunsfeld
8a13b5d120
Rename StackPopResult -> StackSlice
2016-03-03 10:16:10 -08:00
Max Brunsfeld
aef7582a2a
Start using the forward move to recover from errors
...
Some unit tests passing. Corpus tests still failing
2016-03-02 21:08:42 -08:00
Max Brunsfeld
e7abfdd373
Prevent string assertion failures from creating later memory leak errors
2016-03-02 20:58:39 -08:00
Max Brunsfeld
5a34d74702
Clean up stack
2016-02-25 21:51:39 -08:00
Max Brunsfeld
da2ef7ad35
Store trees in the links between stack nodes, not in the nodes themselves
2016-02-23 17:35:50 -08:00
Max Brunsfeld
9d5a260e5b
🔥 unused variable in tree spec
2016-02-22 21:29:46 -08:00
Max Brunsfeld
f444a715fd
Clean up tree array assertions in stack spec
2016-02-22 09:23:25 -08:00
Max Brunsfeld
b113dc8b0f
Return a TreeArray from ts_stack_pop
...
Since the capacity is now included in the return value, the buffer
can be reused in the ts_parser__accept function. Also, it's just
cleaner to use Array consistently, rather than a separate buffer
and size.
2016-02-21 22:31:13 -08:00
Max Brunsfeld
17db46eded
Add test for node that has only one symbol
2016-02-20 13:38:39 -08:00
Max Brunsfeld
2b35890bbb
Add ts_node_symbols() function
2016-02-19 15:41:30 -08:00
Max Brunsfeld
df1d9b2416
Move sexp-generation code to node.c
2016-02-19 15:41:09 -08:00
Max Brunsfeld
3d7df851b5
Rename Vector -> Array
2016-02-17 20:41:29 -08:00
Max Brunsfeld
6fa7eca966
Make vector struct type-safe
2016-02-17 15:30:47 -08:00
Max Brunsfeld
3f08bfb264
Fix build warnings
2016-02-12 14:11:11 -08:00
Max Brunsfeld
e90a425618
Only return one result for each revealed head from ts_stack_pop
2016-02-08 12:08:15 -08:00
Max Brunsfeld
b80a330a74
Fix assorted memory leaks in test code
2016-02-05 12:23:54 -08:00
Max Brunsfeld
c96c4a08e6
Use an object pool for stack nodes, to reduce allocations
...
Also, fix some leaks in the case where memory allocation failed during parsing
2016-02-04 11:19:42 -08:00
Max Brunsfeld
4a7312e514
Fix memory leaks when editing
2016-02-02 12:22:29 -08:00
Max Brunsfeld
7c44b0e387
Fix leaked lookahead trees in normal parsing
2016-01-29 17:31:43 -08:00
Max Brunsfeld
0cf59913ae
Fix double retain of child trees
2016-01-28 23:15:22 -08:00
Max Brunsfeld
5f27550a7a
Fix leaked nodes in stack
2016-01-28 21:18:57 -08:00
Max Brunsfeld
39417e55df
Test successful parsing after handling allocation failures
2016-01-19 19:52:07 -08:00
Max Brunsfeld
3dde0a6f39
Handle allocation failures during parsing
2016-01-19 18:08:01 -08:00
Max Brunsfeld
1543a6c7b0
Handle allocation failures when instantiating documents
2016-01-18 11:32:10 -08:00
Max Brunsfeld
0f7dbea9a3
Unify test targets, use externally defined languages as fixtures
2016-01-15 11:19:24 -08:00
Max Brunsfeld
f2e7058ad9
Support UTF16 directly
...
This makes the API easier to use from javascript
2015-12-28 13:53:22 -08:00
Max Brunsfeld
da1bc038e5
Remove nested options structs in Tree
2015-12-22 14:20:58 -08:00
Max Brunsfeld
f9e6fce62d
Remove redundant tests for the same random edit operations
2015-12-22 11:20:21 -08:00
Max Brunsfeld
0c14227396
Bound randomized edits by the character count, not the byte count
2015-12-18 13:35:02 -08:00
Max Brunsfeld
1c6ad5f7e4
Rename ubiquitous_tokens -> extra_tokens in compiler API
...
They were already called this in the runtime code.
'Extra' is just easier to say.
2015-12-17 15:50:50 -08:00
Max Brunsfeld
c495076adb
Record in parse table which actions can hide splits
...
Suppose a parse state S has multiple actions for a terminal lookahead symbol A.
Then during incremental parsing, while in state S, the parser should not
reuse a non-terminal lookahead B where FIRST(B) contains A, because reusing B
might prematurely discard one of the possible actions that a batch parser
would have attempted in state S, upon seeing A as a lookahead.
2015-12-17 13:11:56 -08:00
Max Brunsfeld
7fbb628c78
Remove TreeSelectionCallback struct
...
Just make a typedef for the function type
2015-12-17 12:09:06 -08:00
Max Brunsfeld
e6f933a21f
Add missing symbol metadata field initializers in specs
2015-12-17 12:06:43 -08:00
Max Brunsfeld
da2dfd703a
{s,}rand -> {s,}random
...
🎩 @robrix
2015-12-10 21:23:23 -08:00
Max Brunsfeld
51998ac5bb
In specs, seed random number generator with clock time by default
2015-12-10 21:01:27 -08:00
