traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
3050 commits 16 branches 98 tags 37 MiB
Commit graph

3050 commits

This branch
This branch
All branches
Author SHA1 Message Date
Max Brunsfeld
96dd5c820e Add JSON schema for grammar format 2016-01-13 20:52:35 -08:00
Max Brunsfeld
1142e05873 Copy lookahead state to the stack before pushing it to the vector 
Valgrind caught this. If the vector needs to grow, the original pointer will
become invalid when the buffer is realloc'd
 2016-01-13 13:49:32 -08:00
Max Brunsfeld
d4632ab9a9 Make the compile function plain C and take a JSON grammar 2016-01-11 12:33:48 -08:00
Max Brunsfeld
b69e19c525 Add plain C API for compiling a JSON grammar 2016-01-10 13:44:22 -08:00
Max Brunsfeld
36870bfced Make Grammar a simple struct 2016-01-08 15:51:30 -08:00
Max Brunsfeld
e59f6294cb Fix bug in lexical state de-duping 2015-12-30 11:15:36 -08:00
Max Brunsfeld
4b04afac5e Control lexer's error-mode via explicit boolean argument 
Previously, the lexer would operate in error-mode (ignoring any garbage input
until it found a valid token) if it was invoked in the 'error' state. Now that
the error state is deduped with other lexical states, the lexer might be invoked
in that state even when error-mode is not intended. This adds a third argument
to `ts_lex` that explicitly sets the error-mode.

This bug was unlikely to occur in any real grammars, but it caused the
node-tree-sitter-compiler test suite to fail for some grammars with only one
rule.
 2015-12-30 09:43:12 -08:00
Max Brunsfeld
4ad1a666be clang-format 2015-12-29 21:17:31 -08:00
Max Brunsfeld
939476c947 When removing duplicate lex states, update the error state too 
Now, instead of being stored as a separate field on the parse table, the error
state is just the first state in the states vector.
 2015-12-29 21:02:24 -08:00
Max Brunsfeld
36cdb3afcd Merge pull request #21 from maxbrunsfeld/reduce-binary-sizes 
Store parse table more compactly
 2015-12-29 20:52:04 -08:00
Max Brunsfeld
97a281502e Store parse table more compactly 2015-12-29 11:27:41 -08:00
Max Brunsfeld
cead07e371 Merge pull request #20 from maxbrunsfeld/utf16 
Support UTF16 directly
 2015-12-28 14:32:49 -08:00
Max Brunsfeld
f2e7058ad9 Support UTF16 directly 
This makes the API easier to use from javascript
 2015-12-28 13:53:22 -08:00
Max Brunsfeld
3bec739202 Replace vendored copy of utf8proc w/ submodule reference 
The code is now on GitHub
 2015-12-27 13:26:56 -08:00
Max Brunsfeld
1d57390eee Merge pull request #19 from maxbrunsfeld/ensure-merging-opportunities 
Prevent alternative parses from getting out of sync
 2015-12-27 13:14:50 -08:00
Max Brunsfeld
ff3a4baf42 Ensure parse stack heads are at the same positions 2015-12-27 12:53:56 -08:00
Max Brunsfeld
a8f50986e0 clang-format 2015-12-24 22:05:54 -08:00
Max Brunsfeld
13d8de3eb3 Remove stack head after it is accepted 2015-12-24 22:04:20 -08:00
Max Brunsfeld
f483178f83 Clean up main parser loop 2015-12-24 10:21:42 -08:00
Max Brunsfeld
f0c89f36f7 Merge pull request #17 from maxbrunsfeld/optimize-fragile-token-handling 
Improve likelihood of node reuse when reparsing
 2015-12-23 09:45:53 -08:00
Max Brunsfeld
7f1c5f7568 Don't use preprocesser for special tree state values 2015-12-22 14:37:29 -08:00
Max Brunsfeld
da1bc038e5 Remove nested options structs in Tree 2015-12-22 14:20:58 -08:00
Max Brunsfeld
30961143fe Only perform breakdown if next terminal node has changes 2015-12-22 14:00:22 -08:00
Max Brunsfeld
6b11890d68 Reuse fragile parent nodes that were reduced in the current state 2015-12-22 13:59:04 -08:00
Max Brunsfeld
f9e6fce62d Remove redundant tests for the same random edit operations 2015-12-22 11:20:21 -08:00
Max Brunsfeld
2bcd2e4d00 Reuse fragile tokens that came from the current lex state 2015-12-21 16:04:11 -08:00
Max Brunsfeld
386b124866 Ensure that there are no duplicate lex states 2015-12-20 15:46:13 -08:00
Max Brunsfeld
c9db5499e9 Remove uninteresting corpus entries 2015-12-18 13:46:24 -08:00
Max Brunsfeld
66460b24fd Use more greek letters in arithmetic corpus 2015-12-18 13:46:10 -08:00
Max Brunsfeld
0c14227396 Bound randomized edits by the character count, not the byte count 2015-12-18 13:35:02 -08:00
Max Brunsfeld
1c6ad5f7e4 Rename ubiquitous_tokens -> extra_tokens in compiler API 
They were already called this in the runtime code.
'Extra' is just easier to say.
 2015-12-17 15:50:50 -08:00
Max Brunsfeld
f065eb0480 Remove unused parameter to LexConflictManager 2015-12-17 15:45:47 -08:00
Max Brunsfeld
a8d2585330 Fix resolution of shift-extra vs reduce actions 2015-12-17 15:19:58 -08:00
Max Brunsfeld
351b4f4aaa Remove unused parameters to ParseConflictManager 2015-12-17 15:19:00 -08:00
Max Brunsfeld
da115d81a4 Merge pull request #16 from maxbrunsfeld/incremental-reuse-fixes-take-3 
Fix two more types of incorrect node reuse during incremental parsing
 2015-12-17 13:54:23 -08:00
Max Brunsfeld
c495076adb Record in parse table which actions can hide splits 
Suppose a parse state S has multiple actions for a terminal lookahead symbol A.
Then during incremental parsing, while in state S, the parser should not
reuse a non-terminal lookahead B where FIRST(B) contains A, because reusing B
might prematurely discard one of the possible actions that a batch parser
would have attempted in state S, upon seeing A as a lookahead.
 2015-12-17 13:11:56 -08:00
Max Brunsfeld
7fbb628c78 Remove TreeSelectionCallback struct 
Just make a typedef for the function type
 2015-12-17 12:09:06 -08:00
Max Brunsfeld
e6f933a21f Add missing symbol metadata field initializers in specs 2015-12-17 12:06:43 -08:00
Max Brunsfeld
8e7ed275c9 Remove stray file 2015-12-17 11:55:08 -08:00
Max Brunsfeld
77a94a2929 Use ::count to check if sets and maps contain elements 2015-12-17 10:05:42 -08:00
Max Brunsfeld
66144dc28e Treat tokens that are sometimes extra as fragile 2015-12-16 20:04:45 -08:00
Max Brunsfeld
d0e107eb84 Merge pull request #15 from maxbrunsfeld/fix-es6-ambiguity-bug 
Fix ambiguity-handling bug that arises in javascript w/ ES6 support
 2015-12-15 22:36:04 -08:00
Max Brunsfeld
3c3d95345f Don't mutate shared lookahead nodes when setting the extra flag 2015-12-15 22:28:50 -08:00
Max Brunsfeld
71059f1d66 Don't reuse fragile tokens across different parse stack heads 2015-12-15 22:26:42 -08:00
Max Brunsfeld
9bff4d0b06 Add concise method syntax to javascript fixture grammar 
This exposes an ambiguity handling bug that I discovered while adding ES6 support to
tree-sitter-javascript
 2015-12-15 22:25:48 -08:00
Max Brunsfeld
8747e2d3b9 Merge pull request #14 from maxbrunsfeld/more-incremental-fixes 
Fix more errors in incremental subtree reuse
 2015-12-10 21:30:48 -08:00
Max Brunsfeld
da2dfd703a {s,}rand -> {s,}random 
🎩 @robrix
 2015-12-10 21:23:23 -08:00
Max Brunsfeld
4ca965019f Break down unverified reductions eagerly when the next node has changes 2015-12-10 21:06:22 -08:00
Max Brunsfeld
d713054d61 Record which tokens are fragile when lexing 2015-12-10 21:05:54 -08:00
Max Brunsfeld
51998ac5bb In specs, seed random number generator with clock time by default 2015-12-10 21:01:27 -08:00