traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1184 commits 16 branches 98 tags 37 MiB
Commit graph

1184 commits

This branch
This branch
All branches
Author SHA1 Message Date
Max Brunsfeld
09b019c530 Fix test for invalid blank input 2016-06-23 09:24:26 -07:00
Max Brunsfeld
c6e9b32d3f Print all the same parse log messages for both debugging methods 2016-06-22 22:36:11 -07:00
Max Brunsfeld
58b1d88950 Clean up breakdown_top_of_stack function 2016-06-22 22:32:38 -07:00
Max Brunsfeld
f425fbad18 Break down reused node on stack whenever lookahead can't be reused 2016-06-22 22:03:27 -07:00
Max Brunsfeld
16ca038556 Don't accept input with errors on the stack 2016-06-22 21:05:05 -07:00
Max Brunsfeld
b40c0326dc Include parse tree rendering at end of debug output 2016-06-22 21:04:35 -07:00
Max Brunsfeld
57f669dfda clang-format 2016-06-22 14:11:08 -07:00
Max Brunsfeld
3ceb3a1eb2 Clean up main parse loop 2016-06-22 14:10:54 -07:00
Max Brunsfeld
e7ccebc153 Avoid duplicate counting of trees when repairing errors 2016-06-21 22:58:06 -07:00
Max Brunsfeld
43ae8235fd Remove the error action; a lack of actions implies an error. 2016-06-21 22:53:48 -07:00
Max Brunsfeld
6a7a5cfc3f Remove nesting in parse action struct 2016-06-21 21:36:33 -07:00
Max Brunsfeld
634ee29d10 Clean up main consume_lookahead loop 2016-06-21 21:27:33 -07:00
Max Brunsfeld
38c144b4a3 Refine logic for deciding when tokens need to be re-lexed 
* While generating the lex table, note which tokens can match the
  same string. A token needs to be relexed when it has possible
  homonyms in the current state.
* Also note which tokens can match substrings of each other tokens.
  A token needs to be relexed when there are viable tokens that
  could match longer strings in the current state and the next
  token has been edited.
* Remove the logic for marking tokens as fragile on creation.
* Store the reusability/non-reusability of symbols off of individual
  actions and onto the entire entry for the state & symbol.
 2016-06-21 07:28:04 -07:00
Max Brunsfeld
45f7cee0c8 Handle extra tokens properly during error recovery 2016-06-18 20:46:25 -07:00
Max Brunsfeld
773e50f26b Update error recovery specs to reflect slightly different recoveries 2016-06-18 20:46:16 -07:00
Max Brunsfeld
94721c7ec0 Rewind and re-tokenize in error mode after detecting an error 2016-06-17 21:26:03 -07:00
Max Brunsfeld
76975f56ec Display node positions in tooltips in debugging graphs 2016-06-15 11:06:00 -07:00
Max Brunsfeld
70d3cde775 Remove extra leading newline from corpus spec texts 2016-06-15 10:31:34 -07:00
Max Brunsfeld
f69d709650 Remove unused functions 2016-06-15 10:17:54 -07:00
Max Brunsfeld
ecc7399ed3 Fix stack breakdown procedure when there are trailing extra tokens 2016-06-14 20:25:33 -07:00
Max Brunsfeld
e70547cd11 Allow recoveries that skip leading children of invisible trees 
Before this, errors could only be recovered by skipping internal children.
 2016-06-14 14:48:35 -07:00
Max Brunsfeld
6dda23796b Don't count extra tokens in error cost computation 2016-06-14 14:47:16 -07:00
Max Brunsfeld
2109f0ed74 Handle allocation failures when copying tree arrays 2016-06-14 14:46:49 -07:00
Max Brunsfeld
f77c08eff5 Check for better stack versions earlier when handling errors 2016-06-12 17:41:25 -07:00
Max Brunsfeld
2b80e66188 Don't merge stack versions with different error costs 2016-06-12 17:27:08 -07:00
Max Brunsfeld
ef2c3a10e3 Fix bug when reusing formerly extra trees as non-extra trees 2016-06-12 17:26:15 -07:00
Max Brunsfeld
e1c51556f2 Check that error repair reductions have the right child count 
Also, avoid repetitively walking through the parse table to verify the
child trees in an error repair. Instead, walk through the parse table
once, and reuse the list of reductions that would be valid after that
sequence.
 2016-06-10 13:15:45 -07:00
Max Brunsfeld
6d40e317df Ensure that reductions are ordered by child count in parse table 2016-06-10 13:11:52 -07:00
Max Brunsfeld
00a0939504 Abort erroneous parse versions more eagerly 2016-06-02 14:04:48 -07:00
Max Brunsfeld
9b67b21dcd Fix an outdated error corpus entry 2016-06-02 14:04:10 -07:00
Max Brunsfeld
96e2e49ef8 Fix false positive when marking nodes unreusable due to ambiguity 2016-05-30 14:16:55 -07:00
Max Brunsfeld
1e42e68098 Handle cases where both valid and incomplete reduction paths end at the same stack node 2016-05-30 14:08:42 -07:00
Max Brunsfeld
ea47fdc0fe Rework logic for when to abandon parses with errors 2016-05-29 22:36:47 -07:00
Max Brunsfeld
6535704870 Replace stack_merge_new function with two simpler functions 
- merge(version1, version2)
- split(version)
 2016-05-28 21:22:10 -07:00
Max Brunsfeld
e686478ad2 Rename stack_merge function to stack_merge_all 2016-05-28 20:24:08 -07:00
Max Brunsfeld
e1a3a1daeb Import error corpus entries from grammar repos 
Now that error recovery requires no input for the grammar author, it shouldn't
be tested in the individual grammar repos.
 2016-05-28 20:12:02 -07:00
Max Brunsfeld
1e353381ff Don't create error node in lexer unless token is completely invalid 
Before, any syntax error would cause the lexer to create an error
leaf node. This could happen even with a valid input, if the parse
stack had split and one particular version of the parse stack
failed to parse.

Now, an error leaf node is only created when the lexer cannot understand
part of the input stream at all. When a normal syntax error occurs,
the lexer just returns a token that is outside of the expected token
set, and the parser handles the unexpected token.
 2016-05-26 14:15:10 -07:00
Max Brunsfeld
7c859a07bb Allow null trees in the stack 2016-05-26 13:34:57 -07:00
Max Brunsfeld
a3679fbb1f Distinguish separators from main tokens via a property on transitions 
It was incorrect to store it as a property on the lexical states themselves
 2016-05-19 16:27:25 -07:00
Max Brunsfeld
59712ec492 Clean up lex table generation 2016-05-19 13:25:46 -07:00
Max Brunsfeld
31cc6e6f9c Remove unused InProgressSymbolEntry typedef 2016-05-16 12:46:29 -07:00
Max Brunsfeld
77e0e4bb16 Fix some leaks after allocation failures occur 2016-05-16 10:53:31 -07:00
Max Brunsfeld
88053cf723 In tests, don’t record allocations while printing debug graphs 2016-05-16 10:44:19 -07:00
Max Brunsfeld
d50f6a58cc Abort parse versions w/ worse errors when repairing an error 2016-05-16 10:33:19 -07:00
Max Brunsfeld
507d5ad9f7 Include shift-extra actions alongside other actions in recovery states 2016-05-16 10:33:18 -07:00
Max Brunsfeld
19bd09b81d Don't include accept actions in recovery states 2016-05-11 14:02:26 -07:00
Max Brunsfeld
22c550c9d6 Discard tokens after error detection to find the best repair 
* Use GLR stack-splitting to try all numbers of tokens to
  discard until a repair is found.
* Check the validity of repairs by looking at the child trees,
  rather than the statically-computed 'in-progress symbols' list
 2016-05-11 13:49:43 -07:00
Max Brunsfeld
9d247e45b2 Deemphasize extra trees in stack debugging graphs 2016-05-01 15:24:50 -07:00
Max Brunsfeld
9ad1e36238 Rename out_of_context_states -> recovery_states 2016-04-27 14:14:56 -07:00
Max Brunsfeld
5b74813a5c Refine logic for which tokens to use in error recovery 2016-04-27 14:09:19 -07:00