traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1225 commits 16 branches 98 tags 37 MiB
Commit graph

127 commits

Author SHA1 Message Date
Max Brunsfeld
31d1160e21 Base error costs on top-level trees skipped and lines of text skipped 
Rather than on the total number of tokens skipped
 2016-08-29 17:06:23 -07:00
Max Brunsfeld
f8c2e9d908 Include tree's original parse state in debug graphs 2016-07-31 21:35:53 -07:00
Max Brunsfeld
d554fab5b5 Remove unused tree state constant 2016-06-27 14:39:12 -07:00
Max Brunsfeld
df09f37a6c Tweak parse log messages 2016-06-23 11:42:43 -07:00
Max Brunsfeld
09b019c530 Fix test for invalid blank input 2016-06-23 09:24:26 -07:00
Max Brunsfeld
b40c0326dc Include parse tree rendering at end of debug output 2016-06-22 21:04:35 -07:00
Max Brunsfeld
38c144b4a3 Refine logic for deciding when tokens need to be re-lexed 
* While generating the lex table, note which tokens can match the
  same string. A token needs to be relexed when it has possible
  homonyms in the current state.
* Also note which tokens can match substrings of each other tokens.
  A token needs to be relexed when there are viable tokens that
  could match longer strings in the current state and the next
  token has been edited.
* Remove the logic for marking tokens as fragile on creation.
* Store the reusability/non-reusability of symbols off of individual
  actions and onto the entire entry for the state & symbol.
 2016-06-21 07:28:04 -07:00
Max Brunsfeld
f69d709650 Remove unused functions 2016-06-15 10:17:54 -07:00
Max Brunsfeld
6dda23796b Don't count extra tokens in error cost computation 2016-06-14 14:47:16 -07:00
Max Brunsfeld
2109f0ed74 Handle allocation failures when copying tree arrays 2016-06-14 14:46:49 -07:00
Max Brunsfeld
ea47fdc0fe Rework logic for when to abandon parses with errors 2016-05-29 22:36:47 -07:00
Max Brunsfeld
1e353381ff Don't create error node in lexer unless token is completely invalid 
Before, any syntax error would cause the lexer to create an error
leaf node. This could happen even with a valid input, if the parse
stack had split and one particular version of the parse stack
failed to parse.

Now, an error leaf node is only created when the lexer cannot understand
part of the input stream at all. When a normal syntax error occurs,
the lexer just returns a token that is outside of the expected token
set, and the parser handles the unexpected token.
 2016-05-26 14:15:10 -07:00
Max Brunsfeld
22c550c9d6 Discard tokens after error detection to find the best repair 
* Use GLR stack-splitting to try all numbers of tokens to
  discard until a repair is found.
* Check the validity of repairs by looking at the child trees,
  rather than the statically-computed 'in-progress symbols' list
 2016-05-11 13:49:43 -07:00
Max Brunsfeld
fd4c33209e Select ambiguous alternatives by minimizing error size 2016-04-24 00:54:20 -07:00
Max Brunsfeld
1fb6065f02 Move tree sexp function back to tree, for easier use in debugger 2016-04-24 00:09:32 -07:00
Max Brunsfeld
f62e0a3d49 Ensure reference count of new tree copy is 1 2016-04-15 23:01:36 -07:00
Max Brunsfeld
695be5bc79 Merge equivalent stacks in a separate stage of parsing 
* No more automatic merging every time a state is pushed to the stack
* When popping from the stack, the current version is always preserved
 2016-04-10 14:12:24 -07:00
Max Brunsfeld
5ba40f15ad Rename stack heads to versions 2016-04-04 12:25:57 -07:00
Max Brunsfeld
267092940d Collapse redundant interior error nodes 2016-04-03 23:46:43 -07:00
Max Brunsfeld
4348eb89d4 Expose lower stack nodes via pop_until() function 
This callback-based API allows the parser to easily visit each interior node
of the stack when searching for an error repair. It also is a better abstraction
over the stack's DAG implementation than having the public functions for
accessing entries and their successor entries.
 2016-03-07 16:09:34 -08:00
Max Brunsfeld
aef7582a2a Start using the forward move to recover from errors 
Some unit tests passing. Corpus tests still failing
 2016-03-02 21:08:42 -08:00
Max Brunsfeld
df1d9b2416 Move sexp-generation code to node.c 2016-02-19 15:41:09 -08:00
Max Brunsfeld
f6f02182c1 Tail-call-optimize recursive functions 
Refs https://github.com/maxbrunsfeld/node-tree-sitter-compiler/pull/7
 2016-02-04 12:59:44 -08:00
Max Brunsfeld
7c44b0e387 Fix leaked lookahead trees in normal parsing 2016-01-29 17:31:43 -08:00
Max Brunsfeld
0cf59913ae Fix double retain of child trees 2016-01-28 23:15:22 -08:00
Max Brunsfeld
3dde0a6f39 Handle allocation failures during parsing 2016-01-19 18:08:01 -08:00
Max Brunsfeld
87316f22f3 Wrap all calls to malloc and friends 2016-01-15 15:27:50 -08:00
Max Brunsfeld
a8f50986e0 clang-format 2015-12-24 22:05:54 -08:00
Max Brunsfeld
7f1c5f7568 Don't use preprocesser for special tree state values 2015-12-22 14:37:29 -08:00
Max Brunsfeld
da1bc038e5 Remove nested options structs in Tree 2015-12-22 14:20:58 -08:00
Max Brunsfeld
6b11890d68 Reuse fragile parent nodes that were reduced in the current state 2015-12-22 13:59:04 -08:00
Max Brunsfeld
2bcd2e4d00 Reuse fragile tokens that came from the current lex state 2015-12-21 16:04:11 -08:00
Max Brunsfeld
3c3d95345f Don't mutate shared lookahead nodes when setting the extra flag 2015-12-15 22:28:50 -08:00
Max Brunsfeld
4d48a2b1b5 Mark direct parents of errors as fragile 2015-12-06 20:31:10 -08:00
Max Brunsfeld
08d50c25ae clang-format 2015-12-04 20:56:33 -08:00
Max Brunsfeld
27bce56ef2 Merge pull request #12 from maxbrunsfeld/api-cleanup 
Clean up API a bit after the addition of row/column tracking
 2015-12-04 20:41:20 -08:00
Max Brunsfeld
d2bf88d5fe Include rows and columns in TSLength 
This way, we don't have to have separate 1D and 2D versions for so many values
 2015-12-04 20:20:29 -08:00
Max Brunsfeld
8939bd549d Fix named child count computation 
Signed-off-by: Josh Vera <vera@github.com>
 2015-12-04 15:51:04 -08:00
Max Brunsfeld
863cabc827 Don't include trailing ubiquitous tokens as children when reducing 2015-12-02 15:31:15 -08:00
Max Brunsfeld
f08554e958 Replace NodeType enum with SymbolMetadata bitfield 
This will allow storing other metadata about symbols, like if they
only appear as ubiquitous tokens
 2015-12-02 15:10:24 -08:00
joshvera
e52c38a68f Work correctly when node's offset point is zero 2015-12-02 17:12:32 -05:00
joshvera
5e748fdf63 s/ts_tree_offset_point/ts_tree_total_size_point 2015-12-02 16:41:56 -05:00
joshvera
e674094f64 Sum the offset_point like offset in ts_tree_assign_parents 2015-12-02 16:41:40 -05:00
joshvera
88d3432787 Merge remote-tracking branch 'joshvera/line-numbers' into line-numbers 2015-11-30 13:06:54 -05:00
joshvera
7633cbb836 indentation 2015-11-30 12:59:23 -05:00
joshvera
f5fc247c8b Merge remote-tracking branch 'origin/master' into line-numbers 2015-11-30 12:36:11 -05:00
joshvera
3d9a44d880 Calculate the column and offset separately in TSNode 2015-11-25 13:36:19 -05:00
joshvera
4663b9ce89 Add padding and size points to ts_tree_make_leaf in ts_lexer__accept 2015-11-25 11:44:13 -05:00
joshvera
ad58b752e6 Rename ts_tree_total_size_point to ts_tree_offset_point 2015-11-25 11:28:28 -05:00
joshvera
2669933d06 Implement ts_find_parent_offset_point 2015-11-25 11:08:44 -05:00