Max Brunsfeld
1e353381ff
Don't create error node in lexer unless token is completely invalid
...
Before, any syntax error would cause the lexer to create an error
leaf node. This could happen even with a valid input, if the parse
stack had split and one particular version of the parse stack
failed to parse.
Now, an error leaf node is only created when the lexer cannot understand
part of the input stream at all. When a normal syntax error occurs,
the lexer just returns a token that is outside of the expected token
set, and the parser handles the unexpected token.
2016-05-26 14:15:10 -07:00
Max Brunsfeld
7c859a07bb
Allow null trees in the stack
2016-05-26 13:34:57 -07:00
Max Brunsfeld
a3679fbb1f
Distinguish separators from main tokens via a property on transitions
...
It was incorrect to store it as a property on the lexical states themselves
2016-05-19 16:27:25 -07:00
Max Brunsfeld
77e0e4bb16
Fix some leaks after allocation failures occur
2016-05-16 10:53:31 -07:00
Max Brunsfeld
88053cf723
In tests, don’t record allocations while printing debug graphs
2016-05-16 10:44:19 -07:00
Max Brunsfeld
d50f6a58cc
Abort parse versions w/ worse errors when repairing an error
2016-05-16 10:33:19 -07:00
Max Brunsfeld
22c550c9d6
Discard tokens after error detection to find the best repair
...
* Use GLR stack-splitting to try all numbers of tokens to
discard until a repair is found.
* Check the validity of repairs by looking at the child trees,
rather than the statically-computed 'in-progress symbols' list
2016-05-11 13:49:43 -07:00
Max Brunsfeld
9d247e45b2
Deemphasize extra trees in stack debugging graphs
2016-05-01 15:24:50 -07:00
Max Brunsfeld
31f6b2e24a
Refactor construction of out-of-context states
2016-04-25 21:59:40 -07:00
Max Brunsfeld
e99a3925e0
Merge all versions created in a given reduce operation
2016-04-24 00:55:19 -07:00
Max Brunsfeld
fe74c6fb34
Explicitly mark stack versions in debugging graphs
2016-04-24 00:54:59 -07:00
Max Brunsfeld
fd4c33209e
Select ambiguous alternatives by minimizing error size
2016-04-24 00:54:20 -07:00
Max Brunsfeld
73c99e9926
Fix returned stack version when popping halts at an error
2016-04-18 11:17:07 -07:00
Max Brunsfeld
d55df29e49
Remove unused constant
2016-04-15 21:33:57 -07:00
Max Brunsfeld
316aba76ee
Reorder stack methods
2016-04-15 21:33:31 -07:00
Max Brunsfeld
cad663b144
Consider multiple error repairs on the same path of the stack
...
This changes the API to the stack_iterate function so that you can pop
from the stack without stopping iteration
2016-04-15 21:28:00 -07:00
Max Brunsfeld
3778c63277
Clean up stack pop code
2016-04-11 23:12:50 -07:00
Max Brunsfeld
9d87bc4517
Rename stack successors to links
2016-04-11 22:41:06 -07:00
Max Brunsfeld
695be5bc79
Merge equivalent stacks in a separate stage of parsing
...
* No more automatic merging every time a state is pushed to the stack
* When popping from the stack, the current version is always preserved
2016-04-10 14:12:24 -07:00
Max Brunsfeld
5ba40f15ad
Rename stack heads to versions
2016-04-04 12:25:57 -07:00
Max Brunsfeld
b1a696085a
Clean up stack pop functions
2016-04-04 11:59:10 -07:00
Max Brunsfeld
2f3e92c9be
Add function for popping all nodes from the stack
2016-04-04 11:44:45 -07:00
Max Brunsfeld
267092940d
Collapse redundant interior error nodes
2016-04-03 23:46:43 -07:00
Max Brunsfeld
849ecf0a13
Display pending stack links as dashed lines
2016-04-02 23:02:21 -07:00
Max Brunsfeld
91e3609fbf
Write to file directly from stack debugging function
2016-04-02 22:18:44 -07:00
Max Brunsfeld
6bce6da1e6
Store verifying flag within parse stack
2016-03-31 12:03:21 -07:00
Max Brunsfeld
523923bd93
Ensure reduce only produces one slice when stopping at errors
2016-03-10 21:26:21 -08:00
Max Brunsfeld
be707970da
Fix always_inline attribute usage
2016-03-10 13:10:46 -08:00
Max Brunsfeld
e7d3d40a59
Explicitly inform stack pop callback when the stack is exhausted
...
Also, pass non-extra tree count as a single value, rather than keeping
track of the extra count and the total separately.
2016-03-10 11:51:55 -08:00
Max Brunsfeld
a8774831e8
Fix buffer overflow when limiting stack node's successor count
2016-03-10 11:43:13 -08:00
Max Brunsfeld
2e35587161
Use new stack_pop_until function for repairing errors
2016-03-07 20:06:46 -08:00
Max Brunsfeld
4348eb89d4
Expose lower stack nodes via pop_until() function
...
This callback-based API allows the parser to easily visit each interior node
of the stack when searching for an error repair. It also is a better abstraction
over the stack's DAG implementation than having the public functions for
accessing entries and their successor entries.
2016-03-07 16:09:34 -08:00
Max Brunsfeld
c0595c21c5
Halt stack pops at all error states, not just error trees
2016-03-03 11:05:37 -08:00
Max Brunsfeld
3d516aeeec
Give StackPushResult enumerators shorter names
2016-03-03 10:20:05 -08:00
Max Brunsfeld
8a13b5d120
Rename StackPopResult -> StackSlice
2016-03-03 10:16:10 -08:00
Max Brunsfeld
aef7582a2a
Start using the forward move to recover from errors
...
Some unit tests passing. Corpus tests still failing
2016-03-02 21:08:42 -08:00
Max Brunsfeld
e0c24e3be6
Remove old error recovery code
2016-03-02 20:58:39 -08:00
Max Brunsfeld
5a34d74702
Clean up stack
2016-02-25 21:51:39 -08:00
Max Brunsfeld
1850951b4f
Escape special chars in production names for debug graphs
2016-02-24 17:24:11 -08:00
Max Brunsfeld
da2ef7ad35
Store trees in the links between stack nodes, not in the nodes themselves
2016-02-23 17:35:50 -08:00
Max Brunsfeld
b36143d7a7
Add flag for logging dot graphs of the stack while parsing
2016-02-23 09:45:27 -08:00
Max Brunsfeld
6dd92c3abe
Add function for rendering the stack as a DOT graph
2016-02-23 00:08:55 -08:00
Max Brunsfeld
b113dc8b0f
Return a TreeArray from ts_stack_pop
...
Since the capacity is now included in the return value, the buffer
can be reused in the ts_parser__accept function. Also, it's just
cleaner to use Array consistently, rather than a separate buffer
and size.
2016-02-21 22:31:13 -08:00
Max Brunsfeld
3d7df851b5
Rename Vector -> Array
2016-02-17 20:41:29 -08:00
Max Brunsfeld
6fa7eca966
Make vector struct type-safe
2016-02-17 15:30:47 -08:00
Max Brunsfeld
3f08bfb264
Fix build warnings
2016-02-12 14:11:11 -08:00
Max Brunsfeld
e90a425618
Only return one result for each revealed head from ts_stack_pop
2016-02-08 12:08:15 -08:00
Max Brunsfeld
c96c4a08e6
Use an object pool for stack nodes, to reduce allocations
...
Also, fix some leaks in the case where memory allocation failed during parsing
2016-02-04 11:19:42 -08:00
Max Brunsfeld
7c44b0e387
Fix leaked lookahead trees in normal parsing
2016-01-29 17:31:43 -08:00
Max Brunsfeld
5f27550a7a
Fix leaked nodes in stack
2016-01-28 21:18:57 -08:00
