Max Brunsfeld
135d8ef4e0
Merge pull request #58 from tree-sitter/reduce-error-recovery-branching
...
Reduce the branching factor of the parse stack during error recovery
2017-02-18 11:34:09 -08:00
Timothy Clem
93e94bf4b2
Add a happy path test for ts_document_set_input_string_with_length
2017-02-13 08:48:51 -08:00
Max Brunsfeld
93d7a75b09
Suppress one unnecessary type of error recovery variation
...
If we already have a stack version in which, for example,
a `function_call` is skipped, don't create another stack
version in which that `function_call` is reduced to an
`expression`, and then the `expression` is skipped. That
doesn't improve the error recovery at all, but adds to the
branching factor of the parse stack and makes things harder
to debug.
2017-02-07 22:07:56 -08:00
Max Brunsfeld
819b63e78d
Merge pull request #57 from tree-sitter/fix-error-recovery-bugs
...
Fix error recovery bug when error parent node contains extra tokens
2017-02-07 21:11:16 -08:00
Max Brunsfeld
5b23a8fca9
Update error corpus to reflect slightly different recoveries
2017-02-07 17:49:15 -08:00
Max Brunsfeld
343887c1dd
Fix miscounting of extra tokens when repairing errors
2017-02-06 17:43:07 -08:00
Timothy Clem
ab00f1b0da
Add support for \W and \D negated character classes too
2017-01-31 15:03:48 -08:00
Timothy Clem
902b7f9745
Allow \S for negated whitespace regex shorthand
2017-01-31 14:45:28 -08:00
Max Brunsfeld
4131e1c16e
Return an error when external token name matches non-terminal rule
2017-01-31 11:36:51 -08:00
Max Brunsfeld
60f6998485
Rename generated language functions to e.g. tree_sitter_python
...
They used to be called e.g. `ts_language_python`. Now that there
are APIs that deal with the `TSLanguage` objects themselves, such
as `ts_language_symbol_count`, the old names were a little confusing.
2017-01-31 10:29:31 -08:00
Max Brunsfeld
d853b6504d
Add version number to TSLanguage structs
2017-01-31 10:21:47 -08:00
Max Brunsfeld
5ee7cbda2c
Enable randomized incremental parsing tests for Python
2017-01-30 21:58:02 -08:00
Max Brunsfeld
896254eea5
Fix error in changed ranges calculation
...
There was an error in the way that we calculate the reference
scope sequences that are used as the basis for assertions about
changed ranges in randomized tests. The error caused some
characters' scopes to not be checked. This corrects the reference
implementation and fixes a previously uncaught bug in the
implementation of `tree_path_get_changed_ranges`.
Previously, when iterating over the old and new trees, we would
only perform comparisons of visible nodes. This resulted in a failure
to do any comparison for portions of the text in which there were
trailing invisible child nodes (e.g. trailing `_line_break` nodes
inside `statement` nodes in the JavaScript grammar).
Now, we additionally perform comparisons at invisible leaf nodes,
based on their lowest visible ancestor.
2017-01-27 23:47:34 -08:00
Max Brunsfeld
0a286d41f3
Add python error recovery tests
2017-01-08 22:06:36 -08:00
Max Brunsfeld
36608180d2
Store external token states in the parse stack
2017-01-08 22:06:05 -08:00
Max Brunsfeld
3a4daace26
Move reusable node functions to their own file
2017-01-05 10:07:27 -08:00
Max Brunsfeld
12cd2132ff
Add test for retrieving last external token state in a Tree
2017-01-04 21:23:04 -08:00
Max Brunsfeld
d57043b665
Add ability to store external token state per stack version
2017-01-04 21:22:23 -08:00
Max Brunsfeld
adae1b16a6
Remove shared setup in tree spec
2016-12-26 14:39:41 -08:00
Max Brunsfeld
0f039721dd
Compile external scanners w/ debug symbols in test suite
2016-12-26 11:08:35 -08:00
Max Brunsfeld
b833942bb8
Clean up Document spec
2016-12-21 11:42:32 -08:00
Max Brunsfeld
83c4c03a70
Remove unnecessary describes in Parser spec
2016-12-21 11:38:25 -08:00
Max Brunsfeld
75bcfed247
Rename doc variable -> document in Parser spec
2016-12-21 11:32:27 -08:00
Max Brunsfeld
4fd7b0e55d
Remove nested before_each's in Parser spec
2016-12-21 11:31:28 -08:00
Max Brunsfeld
42c41c158c
Refactor logic for handling shared internal/external tokens
2016-12-21 10:49:55 -08:00
Max Brunsfeld
006e9fe4a6
Specify c language explicitly when compiling test parsers
2016-12-20 17:28:28 -08:00
Max Brunsfeld
e6c82ead2c
Start work toward maintaining external scanner's state during incremental parses
2016-12-20 17:06:20 -08:00
Max Brunsfeld
2b3da512a4
Add serialize, deserialize and reset callbacks to external scanners
...
Signed-off-by: Nathan Sobo <nathan@github.com>
2016-12-20 13:12:01 -08:00
Max Brunsfeld
727727623a
Start work on unit test that edits python code
...
Signed-off-by: Nathan Sobo <nathan@github.com>
2016-12-20 13:10:18 -08:00
Max Brunsfeld
80b7affb74
Compile & link fixture grammars' external scanners in test suite
2016-12-20 13:07:45 -08:00
Max Brunsfeld
a1770ce844
Allow external tokens to be used as extras
2016-12-12 22:06:01 -08:00
Max Brunsfeld
0e595346be
Make lexer log output easier to read
2016-12-09 13:33:37 -08:00
Max Brunsfeld
10b51a05a1
Allow external scanners to refer to (and return) internally-defined tokens
...
Tokens that are defined in the grammar's rules may now be included in the
externals list also, so that external scanners can check if they are valid
lookaheads or not, and if so, can return them to the parser if needed.
2016-12-09 13:32:58 -08:00
Max Brunsfeld
83514293b5
Allow external tokens to be either visible or hidden
2016-12-05 17:26:11 -08:00
Max Brunsfeld
a09409900f
Silence missing intializer warnings in compiler unit tests
2016-12-05 16:37:06 -08:00
Max Brunsfeld
6073d9c0e8
Restore all languages in corpus specs
2016-12-05 16:36:50 -08:00
Max Brunsfeld
c4fe8ded95
Remove state argument to Lexer advance method
2016-12-05 16:36:34 -08:00
Max Brunsfeld
49d25bd0f8
Remove EXTERNAL_TOKEN grammar rule type
2016-12-04 15:02:32 -08:00
Max Brunsfeld
d72b49316b
Handle external tokens in apply_transitive_closure
2016-12-04 10:40:32 -08:00
Max Brunsfeld
0f8e130687
Call external scanner functions when lexing
2016-12-02 22:03:48 -08:00
Max Brunsfeld
c966af0412
Start work on external tokens
2016-12-02 16:24:19 -08:00
Max Brunsfeld
46854cc274
Compile and link test grammars in one step
2016-12-02 16:23:42 -08:00
Max Brunsfeld
be9e79db1b
Avoid incorrect application of precedence
2016-12-01 10:24:06 -08:00
Max Brunsfeld
996ca91e70
Disallow syntax rules that match the empty string (for now)
2016-11-30 23:19:54 -08:00
Max Brunsfeld
d627042fa6
Fix javascript error test
...
A single line with two function declarations now parses
successfully, so to create the desired error recovery
scenario, wrap the two functions in an assignment
2016-11-30 23:19:34 -08:00
Max Brunsfeld
06215607d1
Precompute transitive closure contributions by grammar symbol
2016-11-20 11:49:55 -08:00
Max Brunsfeld
6cf4ccb840
Represent rule metadata as a struct, not a map
2016-11-19 13:59:34 -08:00
Max Brunsfeld
cab1bd3ac5
Make conflict messages explicit about precedence combinations
2016-11-18 17:05:16 -08:00
Max Brunsfeld
42ba70a5a0
🎨
2016-11-18 16:16:51 -08:00
Max Brunsfeld
a39b5109c9
Ensure out/tmp directory exists in specs
2016-11-18 14:49:56 -08:00
