traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
684 commits 16 branches 98 tags 37 MiB
Commit graph

85 commits

Author SHA1 Message Date
Max Brunsfeld
3d0890eecf Preserve tokens within errors 2015-06-15 15:26:06 -07:00
Max Brunsfeld
d5ce3a9b5a lexer: in error mode, continue until token is found 2015-06-15 15:26:05 -07:00
Max Brunsfeld
f7e4445358 Handle goto actions after reductions in a more standard way 
Rather than letting the reduced tree become the new lookahead symbol,
and re-adding it to the stack via a subsequent shift action, just
add it to the stack as part of the reduce action. This is more in
line with the way LR is described traditionally.
 2015-05-27 10:53:02 -07:00
Max Brunsfeld
b0a89efb35 Tweak parse debugging output 2015-03-07 16:35:11 -08:00
Max Brunsfeld
2d436cf141 Identify fragile reductions at compile time 2015-02-21 15:11:03 -08:00
Max Brunsfeld
a92067702d Use a more exact test for reusability of error nodes 
Based on the concept of node fragility from wagner's incremental
parsing paper
 2015-02-21 10:39:58 -08:00
Max Brunsfeld
5dc08ccce9 Include names of in-progress rules in shift/reduce conflicts 2014-11-05 18:39:50 -08:00
Max Brunsfeld
290cbefe6d Add commas between parameters in debug statements 
This way, the debug lines can be parsed more unambiguously
 2014-10-29 18:16:34 -07:00
Max Brunsfeld
3bc4971fd5 Don't reuse any nodes that contain an error as a child node 2014-10-23 12:50:37 -07:00
Max Brunsfeld
647c3e2010 Add tests for editing all language examples 
This drove out several fixes to the incremental parsing algorithm.
There are a few examples which still don't work and which have
been explicitly excluded, for now.
 2014-10-22 20:10:08 -07:00
Max Brunsfeld
de9a48d11f Tweak debugging in parser and lexer 2014-10-22 20:10:08 -07:00
Max Brunsfeld
38ad248a53 Inline variable in parser_set_debugger 2014-10-18 18:36:39 -07:00
Max Brunsfeld
c26cee5d18 Remove debugger in parser spec 2014-10-17 23:47:20 -07:00
Max Brunsfeld
0962b21dd3 Fix parse errors with tree-reuse 
- Don't reuse error nodes
- When re-using nodes, update lexer's token_end_position
 2014-10-17 23:20:36 -07:00
Max Brunsfeld
8cf800ef5d Unify debugging API for parsing and lexing 2014-10-17 17:52:54 -07:00
Max Brunsfeld
7498725d7f Move lexer debugging logic out of public header 2014-10-17 16:20:01 -07:00
Max Brunsfeld
5c600942df Inline some helper functions for lexer 2014-10-17 15:22:01 -07:00
Max Brunsfeld
b5d022a70c Fix missing field warnings for debugger structs 2014-10-14 22:50:24 -07:00
Max Brunsfeld
b468458929 reuse_right -> reuse in debug output 2014-10-14 11:22:40 -07:00
Max Brunsfeld
8c4607c486 Add key for symbol param in debug output 2014-10-14 10:49:14 -07:00
Max Brunsfeld
87df0a24eb Add debuggging output when starting to parse 2014-10-14 10:31:18 -07:00
Max Brunsfeld
1ef744a76b Reorganize code in parser.c 2014-10-14 09:32:11 -07:00
Max Brunsfeld
ef2084d3c8 Tweak parse debugging 2014-10-13 21:20:08 -07:00
Max Brunsfeld
c594208ab8 Allow callbacks to be specified for debug output 2014-10-13 01:02:18 -07:00
Max Brunsfeld
fb38140317 Discard portion of right subtree that is within the edited region 2014-10-12 11:51:12 -07:00
Max Brunsfeld
f460b921e2 Fix off-by-one error in storing reusable right-subtree 2014-10-10 12:10:23 -07:00
Max Brunsfeld
4dcc712a8c Start work on re-using right side of parse tree 2014-10-09 19:58:15 -07:00
Max Brunsfeld
80b8a0a9fb Rename stack_right_position -> stack_total_tree_size 
I want to re-use the stack data structure for storing the
re-usable nodes from the previous parse tree during an edit.
In this case, the stack won't conceptually start at position
zero, so the name 'right_position' doesn't make sense.
 2014-10-08 17:37:21 -07:00
Max Brunsfeld
26f9e22193 Clean up parser code 2014-10-08 16:51:04 -07:00
Max Brunsfeld
af7f57a80e Fix sizing of error nodes after edits 2014-10-05 16:56:50 -07:00
Max Brunsfeld
6d37877e49 Tweak debugging output 2014-10-05 16:56:29 -07:00
Max Brunsfeld
78c5fe8e02 clang-format 2014-10-03 15:44:21 -07:00
Max Brunsfeld
5dd8778996 Clean up Parser handle_error function 2014-09-29 10:33:56 -07:00
Max Brunsfeld
10a3251fbe Remove index parameter from STACK_FROM_TOP macro 2014-09-29 10:16:24 -07:00
Max Brunsfeld
26ac5788b6 Don't use struct literal syntax for TSLength 2014-09-26 16:31:36 -07:00
Max Brunsfeld
c1565c1aae Track AST nodes' sizes in characters as well as bytes 
The `pos` and `size` functions for Nodes now return TSLength structs,
which contain lengths in both characters and bytes. This is important
for knowing the number of unicode characters in a Node.
 2014-09-26 16:15:07 -07:00
Max Brunsfeld
e23f11b7c4 Allow lexical debug mode to be enabled on documents 
- `ts_document_set_debug(doc, 1)` implies parse debug mode
- `ts_document_set_debug(doc, > 1)` implies parse and lex debug mode
 2014-09-11 13:12:06 -07:00
Max Brunsfeld
8512af712e Add debug log when re-lexing during error handling 2014-09-05 18:38:17 -07:00
Max Brunsfeld
6cf267efaf Clean up breakdown stack function 2014-09-03 22:35:52 -07:00
Max Brunsfeld
3dea1261a6 Clean up document specs for incremental parsing 2014-09-03 18:48:10 -07:00
Max Brunsfeld
c72445d808 Fix inc parsing for nodes containing ubiq tokens 2014-09-03 13:17:06 -07:00
Max Brunsfeld
ad52bdc448 Fix inc parsing when appending to end of a token 2014-09-03 07:09:15 -07:00
Max Brunsfeld
cc5f1471a8 Add debug lines for breaking down stack when re-parsing 2014-09-02 22:16:17 -07:00
Max Brunsfeld
545e575508 Revert "Remove the separator characters construct" 
This reverts commit 5cd07648fd.

The separators construct is useful as an optimization. It turns out that
constructing a node for every chunk of whitespace in a document causes a
significant performance regression.

Conflicts:
	src/compiler/build_tables/build_lex_table.cc
	src/compiler/grammar.cc
	src/runtime/parser.c
 2014-09-02 08:03:51 -07:00
Max Brunsfeld
e941f8c175 Fix error in document editing 
When breaking down the stack in parser.c, the previous code
would not account for ubiquitous tokens. This was a problem
for a long time, but wasn't noticed until ubiquitous tokens
started being used to represent separator characters
 2014-09-01 21:32:29 -07:00
Max Brunsfeld
5cd07648fd Remove the separator characters construct 
Now, grammars can handle whitespace by making it another ubiquitous
token, like comments.

For now, this has the side effect of whitespace being included in the
tree that precedes it. This was already an issue for other ubiquitous
tokens though, so it needs to be fixed anyway.
 2014-09-01 20:19:43 -07:00
Max Brunsfeld
88d07c8960 Clean up parse table lookup function 2014-08-31 21:17:32 -07:00
Max Brunsfeld
2985a98150 Build error nodes in lexer again, not in parser 2014-08-31 16:59:01 -07:00
Max Brunsfeld
16d5cf1d04 Remove expected symbols from error nodes 2014-08-31 16:39:16 -07:00
Max Brunsfeld
85d8c9df5c Handle multiple ubiquitous in a row 2014-08-31 12:11:16 -07:00