Max Brunsfeld
d6ed783c8c
Improve error handling when dynamically loading languages in specs
2016-01-15 11:35:22 -08:00
Max Brunsfeld
eef1ae17a0
Restore USE_BOOST_REGEX build option for specs
2016-01-15 11:30:32 -08:00
Max Brunsfeld
8ec0e3f43e
Link to libld in specs
2016-01-15 11:29:17 -08:00
Max Brunsfeld
aaa77d69ec
Merge pull request #23 from maxbrunsfeld/cleaner-integration-tests
...
Make it easier to write integration tests
2016-01-15 11:23:39 -08:00
Max Brunsfeld
0f7dbea9a3
Unify test targets, use externally defined languages as fixtures
2016-01-15 11:19:24 -08:00
Max Brunsfeld
569b9d4099
Allow comments within grammar JSON
2016-01-14 11:28:13 -08:00
Max Brunsfeld
ad4089a4bf
Move anonymous tokens grammar into integration spec
2016-01-14 10:35:03 -08:00
Max Brunsfeld
bc56729b8b
Fix JSON syntax error in README grammar
2016-01-14 10:17:37 -08:00
Max Brunsfeld
4a5deda071
Add tests that compile a grammar and use its parser
2016-01-14 10:11:30 -08:00
Max Brunsfeld
49f393b75e
Merge pull request #22 from maxbrunsfeld/c-compiler-api
...
Simplify the compiler API
2016-01-13 21:08:41 -08:00
Max Brunsfeld
7121689cfe
Update README to reflect new compiler API
2016-01-13 20:52:58 -08:00
Max Brunsfeld
96dd5c820e
Add JSON schema for grammar format
2016-01-13 20:52:35 -08:00
Max Brunsfeld
1142e05873
Copy lookahead state to the stack before pushing it to the vector
...
Valgrind caught this. If the vector needs to grow, the original pointer will
become invalid when the buffer is realloc'd
2016-01-13 13:49:32 -08:00
Max Brunsfeld
d4632ab9a9
Make the compile function plain C and take a JSON grammar
2016-01-11 12:33:48 -08:00
Max Brunsfeld
b69e19c525
Add plain C API for compiling a JSON grammar
2016-01-10 13:44:22 -08:00
Max Brunsfeld
36870bfced
Make Grammar a simple struct
2016-01-08 15:51:30 -08:00
Max Brunsfeld
e59f6294cb
Fix bug in lexical state de-duping
2015-12-30 11:15:36 -08:00
Max Brunsfeld
4b04afac5e
Control lexer's error-mode via explicit boolean argument
...
Previously, the lexer would operate in error-mode (ignoring any garbage input
until it found a valid token) if it was invoked in the 'error' state. Now that
the error state is deduped with other lexical states, the lexer might be invoked
in that state even when error-mode is not intended. This adds a third argument
to `ts_lex` that explicitly sets the error-mode.
This bug was unlikely to occur in any real grammars, but it caused the
node-tree-sitter-compiler test suite to fail for some grammars with only one
rule.
2015-12-30 09:43:12 -08:00
Max Brunsfeld
4ad1a666be
clang-format
2015-12-29 21:17:31 -08:00
Max Brunsfeld
939476c947
When removing duplicate lex states, update the error state too
...
Now, instead of being stored as a separate field on the parse table, the error
state is just the first state in the states vector.
2015-12-29 21:02:24 -08:00
Max Brunsfeld
36cdb3afcd
Merge pull request #21 from maxbrunsfeld/reduce-binary-sizes
...
Store parse table more compactly
2015-12-29 20:52:04 -08:00
Max Brunsfeld
97a281502e
Store parse table more compactly
2015-12-29 11:27:41 -08:00
Max Brunsfeld
cead07e371
Merge pull request #20 from maxbrunsfeld/utf16
...
Support UTF16 directly
2015-12-28 14:32:49 -08:00
Max Brunsfeld
f2e7058ad9
Support UTF16 directly
...
This makes the API easier to use from javascript
2015-12-28 13:53:22 -08:00
Max Brunsfeld
3bec739202
Replace vendored copy of utf8proc w/ submodule reference
...
The code is now on GitHub
2015-12-27 13:26:56 -08:00
Max Brunsfeld
1d57390eee
Merge pull request #19 from maxbrunsfeld/ensure-merging-opportunities
...
Prevent alternative parses from getting out of sync
2015-12-27 13:14:50 -08:00
Max Brunsfeld
ff3a4baf42
Ensure parse stack heads are at the same positions
2015-12-27 12:53:56 -08:00
Max Brunsfeld
a8f50986e0
clang-format
2015-12-24 22:05:54 -08:00
Max Brunsfeld
13d8de3eb3
Remove stack head after it is accepted
2015-12-24 22:04:20 -08:00
Max Brunsfeld
f483178f83
Clean up main parser loop
2015-12-24 10:21:42 -08:00
Max Brunsfeld
f0c89f36f7
Merge pull request #17 from maxbrunsfeld/optimize-fragile-token-handling
...
Improve likelihood of node reuse when reparsing
2015-12-23 09:45:53 -08:00
Max Brunsfeld
7f1c5f7568
Don't use preprocesser for special tree state values
2015-12-22 14:37:29 -08:00
Max Brunsfeld
da1bc038e5
Remove nested options structs in Tree
2015-12-22 14:20:58 -08:00
Max Brunsfeld
30961143fe
Only perform breakdown if next terminal node has changes
2015-12-22 14:00:22 -08:00
Max Brunsfeld
6b11890d68
Reuse fragile parent nodes that were reduced in the current state
2015-12-22 13:59:04 -08:00
Max Brunsfeld
f9e6fce62d
Remove redundant tests for the same random edit operations
2015-12-22 11:20:21 -08:00
Max Brunsfeld
2bcd2e4d00
Reuse fragile tokens that came from the current lex state
2015-12-21 16:04:11 -08:00
Max Brunsfeld
386b124866
Ensure that there are no duplicate lex states
2015-12-20 15:46:13 -08:00
Max Brunsfeld
c9db5499e9
Remove uninteresting corpus entries
2015-12-18 13:46:24 -08:00
Max Brunsfeld
66460b24fd
Use more greek letters in arithmetic corpus
2015-12-18 13:46:10 -08:00
Max Brunsfeld
0c14227396
Bound randomized edits by the character count, not the byte count
2015-12-18 13:35:02 -08:00
Max Brunsfeld
1c6ad5f7e4
Rename ubiquitous_tokens -> extra_tokens in compiler API
...
They were already called this in the runtime code.
'Extra' is just easier to say.
2015-12-17 15:50:50 -08:00
Max Brunsfeld
f065eb0480
Remove unused parameter to LexConflictManager
2015-12-17 15:45:47 -08:00
Max Brunsfeld
a8d2585330
Fix resolution of shift-extra vs reduce actions
2015-12-17 15:19:58 -08:00
Max Brunsfeld
351b4f4aaa
Remove unused parameters to ParseConflictManager
2015-12-17 15:19:00 -08:00
Max Brunsfeld
da115d81a4
Merge pull request #16 from maxbrunsfeld/incremental-reuse-fixes-take-3
...
Fix two more types of incorrect node reuse during incremental parsing
2015-12-17 13:54:23 -08:00
Max Brunsfeld
c495076adb
Record in parse table which actions can hide splits
...
Suppose a parse state S has multiple actions for a terminal lookahead symbol A.
Then during incremental parsing, while in state S, the parser should not
reuse a non-terminal lookahead B where FIRST(B) contains A, because reusing B
might prematurely discard one of the possible actions that a batch parser
would have attempted in state S, upon seeing A as a lookahead.
2015-12-17 13:11:56 -08:00
Max Brunsfeld
7fbb628c78
Remove TreeSelectionCallback struct
...
Just make a typedef for the function type
2015-12-17 12:09:06 -08:00
Max Brunsfeld
e6f933a21f
Add missing symbol metadata field initializers in specs
2015-12-17 12:06:43 -08:00
Max Brunsfeld
8e7ed275c9
Remove stray file
2015-12-17 11:55:08 -08:00
