traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1173 commits 16 branches 98 tags 37 MiB
Commit graph

453 commits

Author SHA1 Message Date
Max Brunsfeld
634ee29d10 Clean up main consume_lookahead loop 2016-06-21 21:27:33 -07:00
Max Brunsfeld
38c144b4a3 Refine logic for deciding when tokens need to be re-lexed 
* While generating the lex table, note which tokens can match the
  same string. A token needs to be relexed when it has possible
  homonyms in the current state.
* Also note which tokens can match substrings of each other tokens.
  A token needs to be relexed when there are viable tokens that
  could match longer strings in the current state and the next
  token has been edited.
* Remove the logic for marking tokens as fragile on creation.
* Store the reusability/non-reusability of symbols off of individual
  actions and onto the entire entry for the state & symbol.
 2016-06-21 07:28:04 -07:00
Max Brunsfeld
94721c7ec0 Rewind and re-tokenize in error mode after detecting an error 2016-06-17 21:26:03 -07:00
Max Brunsfeld
76975f56ec Display node positions in tooltips in debugging graphs 2016-06-15 11:06:00 -07:00
Max Brunsfeld
f69d709650 Remove unused functions 2016-06-15 10:17:54 -07:00
Max Brunsfeld
ecc7399ed3 Fix stack breakdown procedure when there are trailing extra tokens 2016-06-14 20:25:33 -07:00
Max Brunsfeld
e70547cd11 Allow recoveries that skip leading children of invisible trees 
Before this, errors could only be recovered by skipping internal children.
 2016-06-14 14:48:35 -07:00
Max Brunsfeld
6dda23796b Don't count extra tokens in error cost computation 2016-06-14 14:47:16 -07:00
Max Brunsfeld
2109f0ed74 Handle allocation failures when copying tree arrays 2016-06-14 14:46:49 -07:00
Max Brunsfeld
f77c08eff5 Check for better stack versions earlier when handling errors 2016-06-12 17:41:25 -07:00
Max Brunsfeld
2b80e66188 Don't merge stack versions with different error costs 2016-06-12 17:27:08 -07:00
Max Brunsfeld
ef2c3a10e3 Fix bug when reusing formerly extra trees as non-extra trees 2016-06-12 17:26:15 -07:00
Max Brunsfeld
e1c51556f2 Check that error repair reductions have the right child count 
Also, avoid repetitively walking through the parse table to verify the
child trees in an error repair. Instead, walk through the parse table
once, and reuse the list of reductions that would be valid after that
sequence.
 2016-06-10 13:15:45 -07:00
Max Brunsfeld
00a0939504 Abort erroneous parse versions more eagerly 2016-06-02 14:04:48 -07:00
Max Brunsfeld
96e2e49ef8 Fix false positive when marking nodes unreusable due to ambiguity 2016-05-30 14:16:55 -07:00
Max Brunsfeld
1e42e68098 Handle cases where both valid and incomplete reduction paths end at the same stack node 2016-05-30 14:08:42 -07:00
Max Brunsfeld
ea47fdc0fe Rework logic for when to abandon parses with errors 2016-05-29 22:36:47 -07:00
Max Brunsfeld
6535704870 Replace stack_merge_new function with two simpler functions 
- merge(version1, version2)
- split(version)
 2016-05-28 21:22:10 -07:00
Max Brunsfeld
e686478ad2 Rename stack_merge function to stack_merge_all 2016-05-28 20:24:08 -07:00
Max Brunsfeld
1e353381ff Don't create error node in lexer unless token is completely invalid 
Before, any syntax error would cause the lexer to create an error
leaf node. This could happen even with a valid input, if the parse
stack had split and one particular version of the parse stack
failed to parse.

Now, an error leaf node is only created when the lexer cannot understand
part of the input stream at all. When a normal syntax error occurs,
the lexer just returns a token that is outside of the expected token
set, and the parser handles the unexpected token.
 2016-05-26 14:15:10 -07:00
Max Brunsfeld
7c859a07bb Allow null trees in the stack 2016-05-26 13:34:57 -07:00
Max Brunsfeld
a3679fbb1f Distinguish separators from main tokens via a property on transitions 
It was incorrect to store it as a property on the lexical states themselves
 2016-05-19 16:27:25 -07:00
Max Brunsfeld
77e0e4bb16 Fix some leaks after allocation failures occur 2016-05-16 10:53:31 -07:00
Max Brunsfeld
88053cf723 In tests, don’t record allocations while printing debug graphs 2016-05-16 10:44:19 -07:00
Max Brunsfeld
d50f6a58cc Abort parse versions w/ worse errors when repairing an error 2016-05-16 10:33:19 -07:00
Max Brunsfeld
22c550c9d6 Discard tokens after error detection to find the best repair 
* Use GLR stack-splitting to try all numbers of tokens to
  discard until a repair is found.
* Check the validity of repairs by looking at the child trees,
  rather than the statically-computed 'in-progress symbols' list
 2016-05-11 13:49:43 -07:00
Max Brunsfeld
9d247e45b2 Deemphasize extra trees in stack debugging graphs 2016-05-01 15:24:50 -07:00
Max Brunsfeld
9ad1e36238 Rename out_of_context_states -> recovery_states 2016-04-27 14:14:56 -07:00
Max Brunsfeld
31f6b2e24a Refactor construction of out-of-context states 2016-04-25 21:59:40 -07:00
Max Brunsfeld
e99a3925e0 Merge all versions created in a given reduce operation 2016-04-24 00:55:19 -07:00
Max Brunsfeld
fe74c6fb34 Explicitly mark stack versions in debugging graphs 2016-04-24 00:54:59 -07:00
Max Brunsfeld
fd4c33209e Select ambiguous alternatives by minimizing error size 2016-04-24 00:54:20 -07:00
Max Brunsfeld
1fb6065f02 Move tree sexp function back to tree, for easier use in debugger 2016-04-24 00:09:32 -07:00
Max Brunsfeld
0ea2962fbe Clean up consume_lookahead function 2016-04-18 11:17:07 -07:00
Max Brunsfeld
73c99e9926 Fix returned stack version when popping halts at an error 2016-04-18 11:17:07 -07:00
Max Brunsfeld
f63fcffe95 Fix incorrect cast in ts_language_symbol_is_in_progress 2016-04-18 11:17:07 -07:00
Max Brunsfeld
2f81a6f6c7 Don't recover from errors based on extra tokens 2016-04-15 23:14:31 -07:00
Max Brunsfeld
f62e0a3d49 Ensure reference count of new tree copy is 1 2016-04-15 23:01:36 -07:00
Max Brunsfeld
d55df29e49 Remove unused constant 2016-04-15 21:33:57 -07:00
Max Brunsfeld
316aba76ee Reorder stack methods 2016-04-15 21:33:31 -07:00
Max Brunsfeld
cad663b144 Consider multiple error repairs on the same path of the stack 
This changes the API to the stack_iterate function so that you can pop
from the stack without stopping iteration
 2016-04-15 21:28:00 -07:00
Max Brunsfeld
3778c63277 Clean up stack pop code 2016-04-11 23:12:50 -07:00
Max Brunsfeld
9d87bc4517 Rename stack successors to links 2016-04-11 22:41:06 -07:00
Max Brunsfeld
695be5bc79 Merge equivalent stacks in a separate stage of parsing 
* No more automatic merging every time a state is pushed to the stack
* When popping from the stack, the current version is always preserved
 2016-04-10 14:12:24 -07:00
Max Brunsfeld
827573f1c7 Clean up reduce function 2016-04-07 10:40:33 -07:00
Max Brunsfeld
5ba40f15ad Rename stack heads to versions 2016-04-04 12:25:57 -07:00
Max Brunsfeld
b1a696085a Clean up stack pop functions 2016-04-04 11:59:10 -07:00
Max Brunsfeld
2f3e92c9be Add function for popping all nodes from the stack 2016-04-04 11:44:45 -07:00
Max Brunsfeld
267092940d Collapse redundant interior error nodes 2016-04-03 23:46:43 -07:00
Max Brunsfeld
eb47edb446 Add static function for stringifying nodes including hidden ones 2016-04-03 23:45:43 -07:00