traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1373 commits 16 branches 98 tags 37 MiB
Commit graph

1373 commits

This branch
This branch
All branches
Author SHA1 Message Date
Max Brunsfeld
005308f4c9 🔥 todo.md 2017-01-31 11:39:30 -08:00
Max Brunsfeld
4131e1c16e Return an error when external token name matches non-terminal rule 2017-01-31 11:36:51 -08:00
Max Brunsfeld
60f6998485 Rename generated language functions to e.g. tree_sitter_python 
They used to be called e.g. `ts_language_python`. Now that there
are APIs that deal with the `TSLanguage` objects themselves, such
as `ts_language_symbol_count`, the old names were a little confusing.
 2017-01-31 10:29:31 -08:00
Max Brunsfeld
d853b6504d Add version number to TSLanguage structs 2017-01-31 10:21:47 -08:00
Max Brunsfeld
672d491775 Fix errors in management of external scanner's most recent state 2017-01-30 22:04:46 -08:00
Max Brunsfeld
dc6598e07e Include external token states in stack debug graphs 2017-01-30 21:58:27 -08:00
Max Brunsfeld
5ee7cbda2c Enable randomized incremental parsing tests for Python 2017-01-30 21:58:02 -08:00
Max Brunsfeld
896254eea5 Fix error in changed ranges calculation 
There was an error in the way that we calculate the reference
scope sequences that are used as the basis for assertions about
changed ranges in randomized tests. The error caused some
characters' scopes to not be checked. This corrects the reference
implementation and fixes a previously uncaught bug in the
implementation of `tree_path_get_changed_ranges`.

Previously, when iterating over the old and new trees, we would
only perform comparisons of visible nodes. This resulted in a failure
to do any comparison for portions of the text in which there were
trailing invisible child nodes (e.g. trailing `_line_break` nodes
inside `statement` nodes in the JavaScript grammar).

Now, we additionally perform comparisons at invisible leaf nodes,
based on their lowest visible ancestor.
 2017-01-27 23:47:34 -08:00
Max Brunsfeld
0a286d41f3 Add python error recovery tests 2017-01-08 22:06:36 -08:00
Max Brunsfeld
36608180d2 Store external token states in the parse stack 2017-01-08 22:06:05 -08:00
Max Brunsfeld
3a4daace26 Move reusable node functions to their own file 2017-01-05 10:07:27 -08:00
Max Brunsfeld
12cd2132ff Add test for retrieving last external token state in a Tree 2017-01-04 21:23:04 -08:00
Max Brunsfeld
d57043b665 Add ability to store external token state per stack version 2017-01-04 21:22:23 -08:00
Max Brunsfeld
adae1b16a6 Remove shared setup in tree spec 2016-12-26 14:39:41 -08:00
Max Brunsfeld
0f039721dd Compile external scanners w/ debug symbols in test suite 2016-12-26 11:08:35 -08:00
Max Brunsfeld
2fa7b453c8 Restore external scanner's state only after repositioning lexer 
Also, properly identify the leaf node with the external token state
 2016-12-21 13:59:56 -08:00
Max Brunsfeld
3706678b89 Pass const TSExternalTokenState to external scanner deserialize hook 2016-12-21 13:58:18 -08:00
Max Brunsfeld
4136dad5de Avoid referencing invalid union member in tree_path_descend 2016-12-21 13:21:21 -08:00
Max Brunsfeld
1595a02692 Avoid referencing invalid union member in tree_set_children 2016-12-21 12:23:24 -08:00
Max Brunsfeld
b833942bb8 Clean up Document spec 2016-12-21 11:42:32 -08:00
Max Brunsfeld
83c4c03a70 Remove unnecessary describes in Parser spec 2016-12-21 11:38:25 -08:00
Max Brunsfeld
75bcfed247 Rename doc variable -> document in Parser spec 2016-12-21 11:32:27 -08:00
Max Brunsfeld
4fd7b0e55d Remove nested before_each's in Parser spec 2016-12-21 11:31:28 -08:00
Max Brunsfeld
34a65f588d Tweak naming and organization of external-scanner related language fields 2016-12-21 11:24:41 -08:00
Max Brunsfeld
42c41c158c Refactor logic for handling shared internal/external tokens 2016-12-21 10:49:55 -08:00
Max Brunsfeld
006e9fe4a6 Specify c language explicitly when compiling test parsers 2016-12-20 17:28:28 -08:00
Max Brunsfeld
e6c82ead2c Start work toward maintaining external scanner's state during incremental parses 2016-12-20 17:06:20 -08:00
Max Brunsfeld
2b3da512a4 Add serialize, deserialize and reset callbacks to external scanners 
Signed-off-by: Nathan Sobo <nathan@github.com>
 2016-12-20 13:12:01 -08:00
Max Brunsfeld
727727623a Start work on unit test that edits python code 
Signed-off-by: Nathan Sobo <nathan@github.com>
 2016-12-20 13:10:18 -08:00
Max Brunsfeld
80b7affb74 Compile & link fixture grammars' external scanners in test suite 2016-12-20 13:07:45 -08:00
Max Brunsfeld
5c72c1e28b Fetch python grammar as part of CI 2016-12-20 13:06:16 -08:00
Max Brunsfeld
a1770ce844 Allow external tokens to be used as extras 2016-12-12 22:06:01 -08:00
Max Brunsfeld
0e595346be Make lexer log output easier to read 2016-12-09 13:33:37 -08:00
Max Brunsfeld
10b51a05a1 Allow external scanners to refer to (and return) internally-defined tokens 
Tokens that are defined in the grammar's rules may now be included in the
externals list also, so that external scanners can check if they are valid
lookaheads or not, and if so, can return them to the parser if needed.
 2016-12-09 13:32:58 -08:00
Max Brunsfeld
7f6ec0131d Remove duplication between parser_destroy and parser_set_language 2016-12-06 10:12:49 -08:00
Max Brunsfeld
83514293b5 Allow external tokens to be either visible or hidden 2016-12-05 17:26:11 -08:00
Max Brunsfeld
1251ff2e30 Consider externals to be named, not anonymous 2016-12-05 17:09:22 -08:00
Max Brunsfeld
505fe6d382 Add externals to grammar JSON schema 2016-12-05 17:09:09 -08:00
Max Brunsfeld
a09409900f Silence missing intializer warnings in compiler unit tests 2016-12-05 16:37:06 -08:00
Max Brunsfeld
6073d9c0e8 Restore all languages in corpus specs 2016-12-05 16:36:50 -08:00
Max Brunsfeld
c4fe8ded95 Remove state argument to Lexer advance method 2016-12-05 16:36:34 -08:00
Max Brunsfeld
c16b6b2059 Run external scanners during error recovery 2016-12-05 11:50:24 -08:00
Max Brunsfeld
49d25bd0f8 Remove EXTERNAL_TOKEN grammar rule type 2016-12-04 15:02:32 -08:00
Max Brunsfeld
cf0d8abea1 Destroy external scanner when destroying Parser 2016-12-04 14:18:30 -08:00
Max Brunsfeld
d72b49316b Handle external tokens in apply_transitive_closure 2016-12-04 10:40:32 -08:00
Max Brunsfeld
0f8e130687 Call external scanner functions when lexing 2016-12-02 22:03:48 -08:00
Max Brunsfeld
c966af0412 Start work on external tokens 2016-12-02 16:24:19 -08:00
Max Brunsfeld
46854cc274 Compile and link test grammars in one step 2016-12-02 16:23:42 -08:00
Max Brunsfeld
269ee1c120 Merge pull request #53 from tree-sitter/precedence-fix 
Avoid incorrect application of precedence
 2016-12-01 10:33:20 -08:00
Max Brunsfeld
be9e79db1b Avoid incorrect application of precedence 2016-12-01 10:24:06 -08:00