traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1019 commits 16 branches 98 tags 37 MiB
Commit graph

577 commits

Author SHA1 Message Date
Max Brunsfeld
90cbd61280 Adjust static library path on linux 2016-01-15 11:40:46 -08:00
Max Brunsfeld
d6ed783c8c Improve error handling when dynamically loading languages in specs 2016-01-15 11:35:22 -08:00
Max Brunsfeld
0f7dbea9a3 Unify test targets, use externally defined languages as fixtures 2016-01-15 11:19:24 -08:00
Max Brunsfeld
569b9d4099 Allow comments within grammar JSON 2016-01-14 11:28:13 -08:00
Max Brunsfeld
ad4089a4bf Move anonymous tokens grammar into integration spec 2016-01-14 10:35:03 -08:00
Max Brunsfeld
4a5deda071 Add tests that compile a grammar and use its parser 2016-01-14 10:11:30 -08:00
Max Brunsfeld
49f393b75e Merge pull request #22 from maxbrunsfeld/c-compiler-api 
Simplify the compiler API
 2016-01-13 21:08:41 -08:00
Max Brunsfeld
d4632ab9a9 Make the compile function plain C and take a JSON grammar 2016-01-11 12:33:48 -08:00
Max Brunsfeld
36870bfced Make Grammar a simple struct 2016-01-08 15:51:30 -08:00
Max Brunsfeld
e59f6294cb Fix bug in lexical state de-duping 2015-12-30 11:15:36 -08:00
Max Brunsfeld
4b04afac5e Control lexer's error-mode via explicit boolean argument 
Previously, the lexer would operate in error-mode (ignoring any garbage input
until it found a valid token) if it was invoked in the 'error' state. Now that
the error state is deduped with other lexical states, the lexer might be invoked
in that state even when error-mode is not intended. This adds a third argument
to `ts_lex` that explicitly sets the error-mode.

This bug was unlikely to occur in any real grammars, but it caused the
node-tree-sitter-compiler test suite to fail for some grammars with only one
rule.
 2015-12-30 09:43:12 -08:00
Max Brunsfeld
939476c947 When removing duplicate lex states, update the error state too 
Now, instead of being stored as a separate field on the parse table, the error
state is just the first state in the states vector.
 2015-12-29 21:02:24 -08:00
Max Brunsfeld
97a281502e Store parse table more compactly 2015-12-29 11:27:41 -08:00
Max Brunsfeld
f2e7058ad9 Support UTF16 directly 
This makes the API easier to use from javascript
 2015-12-28 13:53:22 -08:00
Max Brunsfeld
da1bc038e5 Remove nested options structs in Tree 2015-12-22 14:20:58 -08:00
Max Brunsfeld
f9e6fce62d Remove redundant tests for the same random edit operations 2015-12-22 11:20:21 -08:00
Max Brunsfeld
386b124866 Ensure that there are no duplicate lex states 2015-12-20 15:46:13 -08:00
Max Brunsfeld
c9db5499e9 Remove uninteresting corpus entries 2015-12-18 13:46:24 -08:00
Max Brunsfeld
66460b24fd Use more greek letters in arithmetic corpus 2015-12-18 13:46:10 -08:00
Max Brunsfeld
0c14227396 Bound randomized edits by the character count, not the byte count 2015-12-18 13:35:02 -08:00
Max Brunsfeld
1c6ad5f7e4 Rename ubiquitous_tokens -> extra_tokens in compiler API 
They were already called this in the runtime code.
'Extra' is just easier to say.
 2015-12-17 15:50:50 -08:00
Max Brunsfeld
f065eb0480 Remove unused parameter to LexConflictManager 2015-12-17 15:45:47 -08:00
Max Brunsfeld
a8d2585330 Fix resolution of shift-extra vs reduce actions 2015-12-17 15:19:58 -08:00
Max Brunsfeld
351b4f4aaa Remove unused parameters to ParseConflictManager 2015-12-17 15:19:00 -08:00
Max Brunsfeld
c495076adb Record in parse table which actions can hide splits 
Suppose a parse state S has multiple actions for a terminal lookahead symbol A.
Then during incremental parsing, while in state S, the parser should not
reuse a non-terminal lookahead B where FIRST(B) contains A, because reusing B
might prematurely discard one of the possible actions that a batch parser
would have attempted in state S, upon seeing A as a lookahead.
 2015-12-17 13:11:56 -08:00
Max Brunsfeld
7fbb628c78 Remove TreeSelectionCallback struct 
Just make a typedef for the function type
 2015-12-17 12:09:06 -08:00
Max Brunsfeld
e6f933a21f Add missing symbol metadata field initializers in specs 2015-12-17 12:06:43 -08:00
Max Brunsfeld
66144dc28e Treat tokens that are sometimes extra as fragile 2015-12-16 20:04:45 -08:00
Max Brunsfeld
9bff4d0b06 Add concise method syntax to javascript fixture grammar 
This exposes an ambiguity handling bug that I discovered while adding ES6 support to
tree-sitter-javascript
 2015-12-15 22:25:48 -08:00
Max Brunsfeld
da2dfd703a {s,}rand -> {s,}random 
🎩 @robrix
 2015-12-10 21:23:23 -08:00
Max Brunsfeld
d713054d61 Record which tokens are fragile when lexing 2015-12-10 21:05:54 -08:00
Max Brunsfeld
51998ac5bb In specs, seed random number generator with clock time by default 2015-12-10 21:01:27 -08:00
Max Brunsfeld
a3fdd7f20e Improve random word generation in random corpus specs 2015-12-10 16:47:55 -08:00
Max Brunsfeld
75f31a79a3 Treat reduce actions with different production IDs as distinct 2015-12-10 13:00:26 -08:00
Max Brunsfeld
2c2c567a29 Improve randomized edits in corpus specs 2015-12-09 13:37:03 -08:00
Max Brunsfeld
7d066ff632 Change the random seed for the corpus specs 2015-12-08 17:32:16 -08:00
Max Brunsfeld
10286f307f Pass reference to parser in stack's tree selection callback 2015-12-08 12:21:27 -08:00
Max Brunsfeld
76e4599d5e For now, allow any expression as an assignment LHS 2015-12-06 14:14:17 -08:00
Max Brunsfeld
d69fef11f9 Break down top of parse stack when needed 
Also, start randomizing the edits in the corpus specs
 2015-12-04 20:57:33 -08:00
Max Brunsfeld
d2bf88d5fe Include rows and columns in TSLength 
This way, we don't have to have separate 1D and 2D versions for so many values
 2015-12-04 20:20:29 -08:00
Max Brunsfeld
8e217f758c Use individual args instead of TSLength in input seek function 2015-12-03 23:06:01 -08:00
Max Brunsfeld
b3a6de6dad Replace node pos/size functions with start/end char/byte functions 2015-12-03 22:59:27 -08:00
Max Brunsfeld
fe5286f863 Merge pull request #11 from maxbrunsfeld/move-ubiquitous-tokens-outward 
Put ubiquitous tokens as high in the tree as possible
 2015-12-03 10:06:50 -08:00
Max Brunsfeld
8a146a9bef Reset lexer correctly when old input was blank 2015-12-03 10:00:39 -08:00
Max Brunsfeld
863cabc827 Don't include trailing ubiquitous tokens as children when reducing 2015-12-02 15:31:15 -08:00
Max Brunsfeld
ad619d95f6 Add 'extra' field to symbol metadata 
This stores whether a symbol is only ever used as a ubiquitous token. This will
allow ubiquitous nodes to be reused more effectively: if they are always
ubiquitous, then they can be reused immediately, and otherwise, they must be
broken down in case they need to be used structurally.
 2015-12-02 15:10:24 -08:00
Max Brunsfeld
64e56f5acc Add assignments to C grammar 
This creates another source of ambiguity: assignments vs initializations
for declarations. This is good for testing ambiguity handling
 2015-12-02 15:10:24 -08:00
Max Brunsfeld
f08554e958 Replace NodeType enum with SymbolMetadata bitfield 
This will allow storing other metadata about symbols, like if they
only appear as ubiquitous tokens
 2015-12-02 15:10:24 -08:00
joshvera
883bb87e29 add asserts for points in language_specs 2015-12-02 17:29:10 -05:00
joshvera
2df2b58d3e Move point helpers elsewhere 2015-12-02 17:28:52 -05:00