traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1173 commits 16 branches 98 tags 37 MiB
Commit graph

70 commits

Author SHA1 Message Date
Max Brunsfeld
38c144b4a3 Refine logic for deciding when tokens need to be re-lexed 
* While generating the lex table, note which tokens can match the
  same string. A token needs to be relexed when it has possible
  homonyms in the current state.
* Also note which tokens can match substrings of each other tokens.
  A token needs to be relexed when there are viable tokens that
  could match longer strings in the current state and the next
  token has been edited.
* Remove the logic for marking tokens as fragile on creation.
* Store the reusability/non-reusability of symbols off of individual
  actions and onto the entire entry for the state & symbol.
 2016-06-21 07:28:04 -07:00
Max Brunsfeld
f69d709650 Remove unused functions 2016-06-15 10:17:54 -07:00
Max Brunsfeld
2109f0ed74 Handle allocation failures when copying tree arrays 2016-06-14 14:46:49 -07:00
Max Brunsfeld
1e353381ff Don't create error node in lexer unless token is completely invalid 
Before, any syntax error would cause the lexer to create an error
leaf node. This could happen even with a valid input, if the parse
stack had split and one particular version of the parse stack
failed to parse.

Now, an error leaf node is only created when the lexer cannot understand
part of the input stream at all. When a normal syntax error occurs,
the lexer just returns a token that is outside of the expected token
set, and the parser handles the unexpected token.
 2016-05-26 14:15:10 -07:00
Max Brunsfeld
22c550c9d6 Discard tokens after error detection to find the best repair 
* Use GLR stack-splitting to try all numbers of tokens to
  discard until a repair is found.
* Check the validity of repairs by looking at the child trees,
  rather than the statically-computed 'in-progress symbols' list
 2016-05-11 13:49:43 -07:00
Max Brunsfeld
fd4c33209e Select ambiguous alternatives by minimizing error size 2016-04-24 00:54:20 -07:00
Max Brunsfeld
1fb6065f02 Move tree sexp function back to tree, for easier use in debugger 2016-04-24 00:09:32 -07:00
Max Brunsfeld
695be5bc79 Merge equivalent stacks in a separate stage of parsing 
* No more automatic merging every time a state is pushed to the stack
* When popping from the stack, the current version is always preserved
 2016-04-10 14:12:24 -07:00
Max Brunsfeld
267092940d Collapse redundant interior error nodes 2016-04-03 23:46:43 -07:00
Max Brunsfeld
4348eb89d4 Expose lower stack nodes via pop_until() function 
This callback-based API allows the parser to easily visit each interior node
of the stack when searching for an error repair. It also is a better abstraction
over the stack's DAG implementation than having the public functions for
accessing entries and their successor entries.
 2016-03-07 16:09:34 -08:00
Max Brunsfeld
aef7582a2a Start using the forward move to recover from errors 
Some unit tests passing. Corpus tests still failing
 2016-03-02 21:08:42 -08:00
Max Brunsfeld
f444a715fd Clean up tree array assertions in stack spec 2016-02-22 09:23:25 -08:00
Max Brunsfeld
df1d9b2416 Move sexp-generation code to node.c 2016-02-19 15:41:09 -08:00
Max Brunsfeld
a8f50986e0 clang-format 2015-12-24 22:05:54 -08:00
Max Brunsfeld
7f1c5f7568 Don't use preprocesser for special tree state values 2015-12-22 14:37:29 -08:00
Max Brunsfeld
da1bc038e5 Remove nested options structs in Tree 2015-12-22 14:20:58 -08:00
Max Brunsfeld
6b11890d68 Reuse fragile parent nodes that were reduced in the current state 2015-12-22 13:59:04 -08:00
Max Brunsfeld
2bcd2e4d00 Reuse fragile tokens that came from the current lex state 2015-12-21 16:04:11 -08:00
Max Brunsfeld
66144dc28e Treat tokens that are sometimes extra as fragile 2015-12-16 20:04:45 -08:00
Max Brunsfeld
3c3d95345f Don't mutate shared lookahead nodes when setting the extra flag 2015-12-15 22:28:50 -08:00
Max Brunsfeld
d2bf88d5fe Include rows and columns in TSLength 
This way, we don't have to have separate 1D and 2D versions for so many values
 2015-12-04 20:20:29 -08:00
Max Brunsfeld
863cabc827 Don't include trailing ubiquitous tokens as children when reducing 2015-12-02 15:31:15 -08:00
Max Brunsfeld
f08554e958 Replace NodeType enum with SymbolMetadata bitfield 
This will allow storing other metadata about symbols, like if they
only appear as ubiquitous tokens
 2015-12-02 15:10:24 -08:00
joshvera
5e748fdf63 s/ts_tree_offset_point/ts_tree_total_size_point 2015-12-02 16:41:56 -05:00
joshvera
88d3432787 Merge remote-tracking branch 'joshvera/line-numbers' into line-numbers 2015-11-30 13:06:54 -05:00
joshvera
7633cbb836 indentation 2015-11-30 12:59:23 -05:00
joshvera
f5fc247c8b Merge remote-tracking branch 'origin/master' into line-numbers 2015-11-30 12:36:11 -05:00
joshvera
3d9a44d880 Calculate the column and offset separately in TSNode 2015-11-25 13:36:19 -05:00
joshvera
ad58b752e6 Rename ts_tree_total_size_point to ts_tree_offset_point 2015-11-25 11:28:28 -05:00
joshvera
2669933d06 Implement ts_find_parent_offset_point 2015-11-25 11:08:44 -05:00
Max Brunsfeld
5c95d02bd0 Move ts_tree_compare to the right file 2015-11-20 11:53:03 -08:00
joshvera
b0f6bac3ab replace start and end with padding and size 2015-11-18 16:34:50 -08:00
Max Brunsfeld
484721b0c2 Assign tree parent pointers after parse is complete 2015-11-15 12:21:16 -08:00
joshvera
8058500c5b Add source info to TSTree 2015-11-12 15:32:53 -05:00
Max Brunsfeld
36eae5d5e9 Add option to tree_string to include anonymous tokens 2015-10-28 12:09:28 -07:00
Max Brunsfeld
b3d883e128 Store edits in trees, not by splitting stack 
This allows for multiple edits per parse, though it is not exposed through
the API yet
 2015-09-18 22:02:06 -07:00
Max Brunsfeld
0467d190fe Add ts_tree_edit function 2015-09-18 22:02:06 -07:00
Max Brunsfeld
f5f24a708e Remove unnecessary named TSTreeOptions struct 2015-09-09 13:00:07 -07:00
Max Brunsfeld
245daffbc4 Add {next,prev}_concrete_sibling Node methods 2015-09-08 21:43:37 -07:00
Max Brunsfeld
c3f3f19ea8 Add concrete_child and concrete_child_count Node methods 2015-09-08 09:53:26 -07:00
Max Brunsfeld
9591c88f39 In runtime, distinguish between anonymous and hidden nodes 2015-09-06 00:12:37 -07:00
Max Brunsfeld
53926c467e Don't automatically hide singleton nodes 2015-09-02 16:36:29 -07:00
Max Brunsfeld
21258e6a9e Remove 'document' wrapper node 2015-08-22 10:48:34 -07:00
Max Brunsfeld
553ef6674f Rename TreeOptionsWrapper -> TreeOptionsSingleton 2015-08-16 10:32:12 -07:00
Max Brunsfeld
a5d73ed676 Store indices on nodes to speed up traversal methods 2015-08-15 23:35:20 -07:00
Max Brunsfeld
25af9a3adf Don't store visible children on trees 2015-08-14 16:11:37 -07:00
Max Brunsfeld
54e40b8146 Rework AST access API: reduce heap allocation 2015-07-31 15:47:48 -07:00
Max Brunsfeld
0b1d70db34 Always resolve ambiguities immediately 
No more ambiguity nodes.
Also, when merging parse stacks, merge their successors if needed.
 2015-07-15 13:15:11 -07:00
Max Brunsfeld
aff8bc3266 Split parse stack when there are multiple parse actions 2015-07-09 23:09:33 -07:00
Max Brunsfeld
381f89f8ba Create ambiguity nodes when joining stack heads 2015-06-18 17:03:16 -07:00