traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
973 commits 16 branches 98 tags 37 MiB
Commit graph

560 commits

Author SHA1 Message Date
Max Brunsfeld
c9db5499e9 Remove uninteresting corpus entries 2015-12-18 13:46:24 -08:00
Max Brunsfeld
66460b24fd Use more greek letters in arithmetic corpus 2015-12-18 13:46:10 -08:00
Max Brunsfeld
0c14227396 Bound randomized edits by the character count, not the byte count 2015-12-18 13:35:02 -08:00
Max Brunsfeld
1c6ad5f7e4 Rename ubiquitous_tokens -> extra_tokens in compiler API 
They were already called this in the runtime code.
'Extra' is just easier to say.
 2015-12-17 15:50:50 -08:00
Max Brunsfeld
f065eb0480 Remove unused parameter to LexConflictManager 2015-12-17 15:45:47 -08:00
Max Brunsfeld
a8d2585330 Fix resolution of shift-extra vs reduce actions 2015-12-17 15:19:58 -08:00
Max Brunsfeld
351b4f4aaa Remove unused parameters to ParseConflictManager 2015-12-17 15:19:00 -08:00
Max Brunsfeld
c495076adb Record in parse table which actions can hide splits 
Suppose a parse state S has multiple actions for a terminal lookahead symbol A.
Then during incremental parsing, while in state S, the parser should not
reuse a non-terminal lookahead B where FIRST(B) contains A, because reusing B
might prematurely discard one of the possible actions that a batch parser
would have attempted in state S, upon seeing A as a lookahead.
 2015-12-17 13:11:56 -08:00
Max Brunsfeld
7fbb628c78 Remove TreeSelectionCallback struct 
Just make a typedef for the function type
 2015-12-17 12:09:06 -08:00
Max Brunsfeld
e6f933a21f Add missing symbol metadata field initializers in specs 2015-12-17 12:06:43 -08:00
Max Brunsfeld
66144dc28e Treat tokens that are sometimes extra as fragile 2015-12-16 20:04:45 -08:00
Max Brunsfeld
9bff4d0b06 Add concise method syntax to javascript fixture grammar 
This exposes an ambiguity handling bug that I discovered while adding ES6 support to
tree-sitter-javascript
 2015-12-15 22:25:48 -08:00
Max Brunsfeld
da2dfd703a {s,}rand -> {s,}random 
🎩 @robrix
 2015-12-10 21:23:23 -08:00
Max Brunsfeld
d713054d61 Record which tokens are fragile when lexing 2015-12-10 21:05:54 -08:00
Max Brunsfeld
51998ac5bb In specs, seed random number generator with clock time by default 2015-12-10 21:01:27 -08:00
Max Brunsfeld
a3fdd7f20e Improve random word generation in random corpus specs 2015-12-10 16:47:55 -08:00
Max Brunsfeld
75f31a79a3 Treat reduce actions with different production IDs as distinct 2015-12-10 13:00:26 -08:00
Max Brunsfeld
2c2c567a29 Improve randomized edits in corpus specs 2015-12-09 13:37:03 -08:00
Max Brunsfeld
7d066ff632 Change the random seed for the corpus specs 2015-12-08 17:32:16 -08:00
Max Brunsfeld
10286f307f Pass reference to parser in stack's tree selection callback 2015-12-08 12:21:27 -08:00
Max Brunsfeld
76e4599d5e For now, allow any expression as an assignment LHS 2015-12-06 14:14:17 -08:00
Max Brunsfeld
d69fef11f9 Break down top of parse stack when needed 
Also, start randomizing the edits in the corpus specs
 2015-12-04 20:57:33 -08:00
Max Brunsfeld
d2bf88d5fe Include rows and columns in TSLength 
This way, we don't have to have separate 1D and 2D versions for so many values
 2015-12-04 20:20:29 -08:00
Max Brunsfeld
8e217f758c Use individual args instead of TSLength in input seek function 2015-12-03 23:06:01 -08:00
Max Brunsfeld
b3a6de6dad Replace node pos/size functions with start/end char/byte functions 2015-12-03 22:59:27 -08:00
Max Brunsfeld
fe5286f863 Merge pull request #11 from maxbrunsfeld/move-ubiquitous-tokens-outward 
Put ubiquitous tokens as high in the tree as possible
 2015-12-03 10:06:50 -08:00
Max Brunsfeld
8a146a9bef Reset lexer correctly when old input was blank 2015-12-03 10:00:39 -08:00
Max Brunsfeld
863cabc827 Don't include trailing ubiquitous tokens as children when reducing 2015-12-02 15:31:15 -08:00
Max Brunsfeld
ad619d95f6 Add 'extra' field to symbol metadata 
This stores whether a symbol is only ever used as a ubiquitous token. This will
allow ubiquitous nodes to be reused more effectively: if they are always
ubiquitous, then they can be reused immediately, and otherwise, they must be
broken down in case they need to be used structurally.
 2015-12-02 15:10:24 -08:00
Max Brunsfeld
64e56f5acc Add assignments to C grammar 
This creates another source of ambiguity: assignments vs initializations
for declarations. This is good for testing ambiguity handling
 2015-12-02 15:10:24 -08:00
Max Brunsfeld
f08554e958 Replace NodeType enum with SymbolMetadata bitfield 
This will allow storing other metadata about symbols, like if they
only appear as ubiquitous tokens
 2015-12-02 15:10:24 -08:00
joshvera
883bb87e29 add asserts for points in language_specs 2015-12-02 17:29:10 -05:00
joshvera
2df2b58d3e Move point helpers elsewhere 2015-12-02 17:28:52 -05:00
joshvera
14bc05b8ac Fix some names and add more tests 2015-12-02 17:12:00 -05:00
joshvera
da8e48bf09 fix test names 2015-12-02 16:45:08 -05:00
joshvera
94a46d9ae1 Add more start_point and end_point tests 2015-12-02 16:40:56 -05:00
joshvera
cc4bd82e8a Fix failing tests using find 2015-12-02 16:06:55 -05:00
joshvera
9da4aeaeff columns start at 0 for sanity's sake 2015-11-30 17:22:47 -05:00
joshvera
f5fc247c8b Merge remote-tracking branch 'origin/master' into line-numbers 2015-11-30 12:36:11 -05:00
joshvera
3d9a44d880 Calculate the column and offset separately in TSNode 2015-11-25 13:36:19 -05:00
Max Brunsfeld
c88e9044d5 Make stack popping more robust 2015-11-20 00:04:21 -08:00
joshvera
b0f6bac3ab replace start and end with padding and size 2015-11-18 16:34:50 -08:00
Max Brunsfeld
ab34cfecd9 Replace TreeVector with a more generic Vector struct 2015-11-15 09:55:36 -08:00
joshvera
8058500c5b Add source info to TSTree 2015-11-12 15:32:53 -05:00
Max Brunsfeld
483577f00d Fix error when merging nodes at the base of the parse stack 2015-11-11 17:31:40 -08:00
Max Brunsfeld
40a90b551a Allow error recovery to look all the way to the bottom of the stack 
Previously, there was a bug where the first node on the stack
would never be popped
 2015-11-11 16:59:41 -08:00
Max Brunsfeld
1a5d5b3156 Make ambiguities resolve deterministically 
In the future, they should resolve according to some kind of dynamic
precedence annotations provided in the grammars. For now, this at least makes
them fully deterministic, so that tests won't fail due to ambiguities resolving
differently after undone edits.
 2015-11-11 16:54:03 -08:00
Max Brunsfeld
e11515fb74 Escape backslashes and quotes in symbol name strings 2015-11-09 09:33:24 -08:00
Max Brunsfeld
d5ce268074 Fix handling of changing precedence within lexical rules. 
A precedence annotation wrapping a sequence of characters now only affects how
tightly those characters bind to *each other*, not how tightly they bind to the
preceding character.

This bug surfaced because a generated lexer was failing to recognize a '\n' character
as a token, instead treating it as ubiquitous whitespace. It made this error
because, even though anonymous ubiquitous tokens have the lowest precedence, the
character immediately *after* the '\n' was part of a normal token, which had
*normal* precedence (0). Advancing into that following token was incorrectly
prioritized above accepting the line-break token.
 2015-11-08 13:36:15 -08:00
Max Brunsfeld
30b6530fd1 Account for parse stack merges when shifting 
Previously, when the parse stack was split into 3 or more heads, it was
possible for head 3 to be accidentally skipped if head 2 merged with head 1.
 2015-11-05 21:21:18 -08:00