Max Brunsfeld
97a281502e
Store parse table more compactly
2015-12-29 11:27:41 -08:00
Max Brunsfeld
cead07e371
Merge pull request #20 from maxbrunsfeld/utf16
...
Support UTF16 directly
2015-12-28 14:32:49 -08:00
Max Brunsfeld
f2e7058ad9
Support UTF16 directly
...
This makes the API easier to use from javascript
2015-12-28 13:53:22 -08:00
Max Brunsfeld
3bec739202
Replace vendored copy of utf8proc w/ submodule reference
...
The code is now on GitHub
2015-12-27 13:26:56 -08:00
Max Brunsfeld
1d57390eee
Merge pull request #19 from maxbrunsfeld/ensure-merging-opportunities
...
Prevent alternative parses from getting out of sync
2015-12-27 13:14:50 -08:00
Max Brunsfeld
ff3a4baf42
Ensure parse stack heads are at the same positions
2015-12-27 12:53:56 -08:00
Max Brunsfeld
a8f50986e0
clang-format
2015-12-24 22:05:54 -08:00
Max Brunsfeld
13d8de3eb3
Remove stack head after it is accepted
2015-12-24 22:04:20 -08:00
Max Brunsfeld
f483178f83
Clean up main parser loop
2015-12-24 10:21:42 -08:00
Max Brunsfeld
f0c89f36f7
Merge pull request #17 from maxbrunsfeld/optimize-fragile-token-handling
...
Improve likelihood of node reuse when reparsing
2015-12-23 09:45:53 -08:00
Max Brunsfeld
7f1c5f7568
Don't use preprocesser for special tree state values
2015-12-22 14:37:29 -08:00
Max Brunsfeld
da1bc038e5
Remove nested options structs in Tree
2015-12-22 14:20:58 -08:00
Max Brunsfeld
30961143fe
Only perform breakdown if next terminal node has changes
2015-12-22 14:00:22 -08:00
Max Brunsfeld
6b11890d68
Reuse fragile parent nodes that were reduced in the current state
2015-12-22 13:59:04 -08:00
Max Brunsfeld
f9e6fce62d
Remove redundant tests for the same random edit operations
2015-12-22 11:20:21 -08:00
Max Brunsfeld
2bcd2e4d00
Reuse fragile tokens that came from the current lex state
2015-12-21 16:04:11 -08:00
Max Brunsfeld
386b124866
Ensure that there are no duplicate lex states
2015-12-20 15:46:13 -08:00
Max Brunsfeld
c9db5499e9
Remove uninteresting corpus entries
2015-12-18 13:46:24 -08:00
Max Brunsfeld
66460b24fd
Use more greek letters in arithmetic corpus
2015-12-18 13:46:10 -08:00
Max Brunsfeld
0c14227396
Bound randomized edits by the character count, not the byte count
2015-12-18 13:35:02 -08:00
Max Brunsfeld
1c6ad5f7e4
Rename ubiquitous_tokens -> extra_tokens in compiler API
...
They were already called this in the runtime code.
'Extra' is just easier to say.
2015-12-17 15:50:50 -08:00
Max Brunsfeld
f065eb0480
Remove unused parameter to LexConflictManager
2015-12-17 15:45:47 -08:00
Max Brunsfeld
a8d2585330
Fix resolution of shift-extra vs reduce actions
2015-12-17 15:19:58 -08:00
Max Brunsfeld
351b4f4aaa
Remove unused parameters to ParseConflictManager
2015-12-17 15:19:00 -08:00
Max Brunsfeld
da115d81a4
Merge pull request #16 from maxbrunsfeld/incremental-reuse-fixes-take-3
...
Fix two more types of incorrect node reuse during incremental parsing
2015-12-17 13:54:23 -08:00
Max Brunsfeld
c495076adb
Record in parse table which actions can hide splits
...
Suppose a parse state S has multiple actions for a terminal lookahead symbol A.
Then during incremental parsing, while in state S, the parser should not
reuse a non-terminal lookahead B where FIRST(B) contains A, because reusing B
might prematurely discard one of the possible actions that a batch parser
would have attempted in state S, upon seeing A as a lookahead.
2015-12-17 13:11:56 -08:00
Max Brunsfeld
7fbb628c78
Remove TreeSelectionCallback struct
...
Just make a typedef for the function type
2015-12-17 12:09:06 -08:00
Max Brunsfeld
e6f933a21f
Add missing symbol metadata field initializers in specs
2015-12-17 12:06:43 -08:00
Max Brunsfeld
8e7ed275c9
Remove stray file
2015-12-17 11:55:08 -08:00
Max Brunsfeld
77a94a2929
Use ::count to check if sets and maps contain elements
2015-12-17 10:05:42 -08:00
Max Brunsfeld
66144dc28e
Treat tokens that are sometimes extra as fragile
2015-12-16 20:04:45 -08:00
Max Brunsfeld
d0e107eb84
Merge pull request #15 from maxbrunsfeld/fix-es6-ambiguity-bug
...
Fix ambiguity-handling bug that arises in javascript w/ ES6 support
2015-12-15 22:36:04 -08:00
Max Brunsfeld
3c3d95345f
Don't mutate shared lookahead nodes when setting the extra flag
2015-12-15 22:28:50 -08:00
Max Brunsfeld
71059f1d66
Don't reuse fragile tokens across different parse stack heads
2015-12-15 22:26:42 -08:00
Max Brunsfeld
9bff4d0b06
Add concise method syntax to javascript fixture grammar
...
This exposes an ambiguity handling bug that I discovered while adding ES6 support to
tree-sitter-javascript
2015-12-15 22:25:48 -08:00
Max Brunsfeld
8747e2d3b9
Merge pull request #14 from maxbrunsfeld/more-incremental-fixes
...
Fix more errors in incremental subtree reuse
2015-12-10 21:30:48 -08:00
Max Brunsfeld
da2dfd703a
{s,}rand -> {s,}random
...
🎩 @robrix
2015-12-10 21:23:23 -08:00
Max Brunsfeld
4ca965019f
Break down unverified reductions eagerly when the next node has changes
2015-12-10 21:06:22 -08:00
Max Brunsfeld
d713054d61
Record which tokens are fragile when lexing
2015-12-10 21:05:54 -08:00
Max Brunsfeld
51998ac5bb
In specs, seed random number generator with clock time by default
2015-12-10 21:01:27 -08:00
Max Brunsfeld
a3fdd7f20e
Improve random word generation in random corpus specs
2015-12-10 16:47:55 -08:00
Max Brunsfeld
6abce098c0
Track whether parse stack was split before consuming each lookahead
2015-12-10 13:00:27 -08:00
Max Brunsfeld
75f31a79a3
Treat reduce actions with different production IDs as distinct
2015-12-10 13:00:26 -08:00
Max Brunsfeld
2c2c567a29
Improve randomized edits in corpus specs
2015-12-09 13:37:03 -08:00
Max Brunsfeld
dc28a10bf6
Merge pull request #13 from maxbrunsfeld/incremental-reuse-fixes
...
Fix some errors in incremental subtree reuse
2015-12-08 17:43:05 -08:00
Max Brunsfeld
7d066ff632
Change the random seed for the corpus specs
2015-12-08 17:32:16 -08:00
Max Brunsfeld
924ce8c1b3
Fix tracking of lookahead positions for different parse stack heads
2015-12-08 13:02:25 -08:00
Max Brunsfeld
26e34ddcbb
Allow all parse stack heads to finish
2015-12-08 13:01:33 -08:00
Max Brunsfeld
a7ca28403e
Log when selecting a tree in an ambiguity
2015-12-08 12:25:41 -08:00
Max Brunsfeld
10286f307f
Pass reference to parser in stack's tree selection callback
2015-12-08 12:21:27 -08:00
