traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
996 commits 16 branches 98 tags 37 MiB
Commit graph

345 commits

Author SHA1 Message Date
Max Brunsfeld
1142e05873 Copy lookahead state to the stack before pushing it to the vector 
Valgrind caught this. If the vector needs to grow, the original pointer will
become invalid when the buffer is realloc'd
 2016-01-13 13:49:32 -08:00
Max Brunsfeld
4b04afac5e Control lexer's error-mode via explicit boolean argument 
Previously, the lexer would operate in error-mode (ignoring any garbage input
until it found a valid token) if it was invoked in the 'error' state. Now that
the error state is deduped with other lexical states, the lexer might be invoked
in that state even when error-mode is not intended. This adds a third argument
to `ts_lex` that explicitly sets the error-mode.

This bug was unlikely to occur in any real grammars, but it caused the
node-tree-sitter-compiler test suite to fail for some grammars with only one
rule.
 2015-12-30 09:43:12 -08:00
Max Brunsfeld
4ad1a666be clang-format 2015-12-29 21:17:31 -08:00
Max Brunsfeld
97a281502e Store parse table more compactly 2015-12-29 11:27:41 -08:00
Max Brunsfeld
f2e7058ad9 Support UTF16 directly 
This makes the API easier to use from javascript
 2015-12-28 13:53:22 -08:00
Max Brunsfeld
ff3a4baf42 Ensure parse stack heads are at the same positions 2015-12-27 12:53:56 -08:00
Max Brunsfeld
a8f50986e0 clang-format 2015-12-24 22:05:54 -08:00
Max Brunsfeld
13d8de3eb3 Remove stack head after it is accepted 2015-12-24 22:04:20 -08:00
Max Brunsfeld
f483178f83 Clean up main parser loop 2015-12-24 10:21:42 -08:00
Max Brunsfeld
7f1c5f7568 Don't use preprocesser for special tree state values 2015-12-22 14:37:29 -08:00
Max Brunsfeld
da1bc038e5 Remove nested options structs in Tree 2015-12-22 14:20:58 -08:00
Max Brunsfeld
30961143fe Only perform breakdown if next terminal node has changes 2015-12-22 14:00:22 -08:00
Max Brunsfeld
6b11890d68 Reuse fragile parent nodes that were reduced in the current state 2015-12-22 13:59:04 -08:00
Max Brunsfeld
2bcd2e4d00 Reuse fragile tokens that came from the current lex state 2015-12-21 16:04:11 -08:00
Max Brunsfeld
c495076adb Record in parse table which actions can hide splits 
Suppose a parse state S has multiple actions for a terminal lookahead symbol A.
Then during incremental parsing, while in state S, the parser should not
reuse a non-terminal lookahead B where FIRST(B) contains A, because reusing B
might prematurely discard one of the possible actions that a batch parser
would have attempted in state S, upon seeing A as a lookahead.
 2015-12-17 13:11:56 -08:00
Max Brunsfeld
7fbb628c78 Remove TreeSelectionCallback struct 
Just make a typedef for the function type
 2015-12-17 12:09:06 -08:00
Max Brunsfeld
66144dc28e Treat tokens that are sometimes extra as fragile 2015-12-16 20:04:45 -08:00
Max Brunsfeld
3c3d95345f Don't mutate shared lookahead nodes when setting the extra flag 2015-12-15 22:28:50 -08:00
Max Brunsfeld
71059f1d66 Don't reuse fragile tokens across different parse stack heads 2015-12-15 22:26:42 -08:00
Max Brunsfeld
4ca965019f Break down unverified reductions eagerly when the next node has changes 2015-12-10 21:06:22 -08:00
Max Brunsfeld
d713054d61 Record which tokens are fragile when lexing 2015-12-10 21:05:54 -08:00
Max Brunsfeld
6abce098c0 Track whether parse stack was split before consuming each lookahead 2015-12-10 13:00:27 -08:00
Max Brunsfeld
924ce8c1b3 Fix tracking of lookahead positions for different parse stack heads 2015-12-08 13:02:25 -08:00
Max Brunsfeld
26e34ddcbb Allow all parse stack heads to finish 2015-12-08 13:01:33 -08:00
Max Brunsfeld
a7ca28403e Log when selecting a tree in an ambiguity 2015-12-08 12:25:41 -08:00
Max Brunsfeld
10286f307f Pass reference to parser in stack's tree selection callback 2015-12-08 12:21:27 -08:00
Max Brunsfeld
4abb415113 Handle parse stack splitting while breaking down unverified reduction 2015-12-06 21:10:47 -08:00
Max Brunsfeld
4d48a2b1b5 Mark direct parents of errors as fragile 2015-12-06 20:31:10 -08:00
Max Brunsfeld
98f190d3ba When the parse stack is split, mark all reductions as fragile 2015-12-06 20:30:48 -08:00
Max Brunsfeld
738c49abc8 Log each step when breaking down the parse stack 2015-12-04 20:57:33 -08:00
Max Brunsfeld
d69fef11f9 Break down top of parse stack when needed 
Also, start randomizing the edits in the corpus specs
 2015-12-04 20:57:33 -08:00
Max Brunsfeld
08d50c25ae clang-format 2015-12-04 20:56:33 -08:00
Max Brunsfeld
27bce56ef2 Merge pull request #12 from maxbrunsfeld/api-cleanup 
Clean up API a bit after the addition of row/column tracking
 2015-12-04 20:41:20 -08:00
Max Brunsfeld
d2bf88d5fe Include rows and columns in TSLength 
This way, we don't have to have separate 1D and 2D versions for so many values
 2015-12-04 20:20:29 -08:00
Max Brunsfeld
8939bd549d Fix named child count computation 
Signed-off-by: Josh Vera <vera@github.com>
 2015-12-04 15:51:04 -08:00
Max Brunsfeld
22c76fc71b Remove TSLength from runtime header 
Refactor node functions now that character offset and byte offset are stored separately
 2015-12-04 10:45:30 -08:00
Max Brunsfeld
8e217f758c Use individual args instead of TSLength in input seek function 2015-12-03 23:06:01 -08:00
Max Brunsfeld
b3a6de6dad Replace node pos/size functions with start/end char/byte functions 2015-12-03 22:59:27 -08:00
Max Brunsfeld
fe5286f863 Merge pull request #11 from maxbrunsfeld/move-ubiquitous-tokens-outward 
Put ubiquitous tokens as high in the tree as possible
 2015-12-03 10:06:50 -08:00
Max Brunsfeld
8a146a9bef Reset lexer correctly when old input was blank 2015-12-03 10:00:39 -08:00
Max Brunsfeld
863cabc827 Don't include trailing ubiquitous tokens as children when reducing 2015-12-02 15:31:15 -08:00
Max Brunsfeld
f08554e958 Replace NodeType enum with SymbolMetadata bitfield 
This will allow storing other metadata about symbols, like if they
only appear as ubiquitous tokens
 2015-12-02 15:10:24 -08:00
joshvera
06c790e16e declare ts_point_make before using it 2015-12-02 17:51:06 -05:00
joshvera
144aab22e6 missed two 2015-12-02 17:47:46 -05:00
joshvera
debf5205fb Don't use c-style structs 2015-12-02 17:44:14 -05:00
joshvera
e52c38a68f Work correctly when node's offset point is zero 2015-12-02 17:12:32 -05:00
joshvera
4953182707 add row to node equality 2015-12-02 16:45:20 -05:00
joshvera
7ad82cf684 add const 2015-12-02 16:45:12 -05:00
joshvera
5e748fdf63 s/ts_tree_offset_point/ts_tree_total_size_point 2015-12-02 16:41:56 -05:00
joshvera
e674094f64 Sum the offset_point like offset in ts_tree_assign_parents 2015-12-02 16:41:40 -05:00