traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1423 commits 16 branches 98 tags 37 MiB
Commit graph

515 commits

Author SHA1 Message Date
Max Brunsfeld
b3edd8f749 Remove use of shared_ptr in choice, repeat, and seq factories 2017-03-17 14:28:13 -07:00
Max Brunsfeld
d9fb863bea Fix build errors w/ gcc 2017-03-17 14:03:49 -07:00
Max Brunsfeld
416cbb9def Add missing cassert includes 2017-03-17 13:54:40 -07:00
Max Brunsfeld
90d21adf3b Format make_visitor helper consistently w/ project 2017-03-17 13:37:26 -07:00
Max Brunsfeld
db4b9ebc7c Implement Rule as a union rather than an abstract base class 2017-03-17 13:29:31 -07:00
Max Brunsfeld
d222dbb9fd Allow lexer to accept tokens that ended at previous positions 
* Track lookahead in each tree
* Add 'mark_end' API that external scanners can use
 2017-03-13 17:06:52 -07:00
Max Brunsfeld
f04d7c5860 Handle unused tokens 2017-03-09 21:16:37 -08:00
Max Brunsfeld
c79fae6d21 Clean up extract_tokens function 2017-03-09 21:16:20 -08:00
Max Brunsfeld
f049d5d94c Make ParseItem a struct, not a class 2017-03-08 21:06:30 -08:00
Max Brunsfeld
64e9230071 Use LexTableBuilder to detect conflicts between tokens more correctly 2017-03-08 12:47:38 -08:00
Max Brunsfeld
abf8a4f2c2 🎨 2017-03-01 22:15:26 -08:00
Max Brunsfeld
686dc0997c Avoid introducing certain lexical conflicts during parse state merging 
The current pretty conservative approach is to avoid merging parse states which
would cause a pair tokens to co-exist for the first time in any parse state,
where the two tokens can start with the same character and at least one of the
tokens can contain a character which is part of the grammar's separators.
 2017-02-27 22:54:38 -08:00
Max Brunsfeld
3c8e6f9987 Restructure parse state merging logic 
* Remove remnants of templatized remove_duplicate_states function
* Rename recovery_tokens function to get_compatible_tokens and augment it
  also compute pairs of tokens which could potentially be incompatible
 2017-02-26 12:23:48 -08:00
Timothy Clem
ab00f1b0da Add support for \W and \D negated character classes too 2017-01-31 15:03:48 -08:00
Timothy Clem
902b7f9745 Allow \S for negated whitespace regex shorthand 2017-01-31 14:45:28 -08:00
Max Brunsfeld
0a6e5f9ee6 Fix some build warnings on gcc 2017-01-31 11:46:28 -08:00
Max Brunsfeld
4131e1c16e Return an error when external token name matches non-terminal rule 2017-01-31 11:36:51 -08:00
Max Brunsfeld
60f6998485 Rename generated language functions to e.g. tree_sitter_python 
They used to be called e.g. `ts_language_python`. Now that there
are APIs that deal with the `TSLanguage` objects themselves, such
as `ts_language_symbol_count`, the old names were a little confusing.
 2017-01-31 10:29:31 -08:00
Max Brunsfeld
d853b6504d Add version number to TSLanguage structs 2017-01-31 10:21:47 -08:00
Max Brunsfeld
3706678b89 Pass const TSExternalTokenState to external scanner deserialize hook 2016-12-21 13:58:18 -08:00
Max Brunsfeld
34a65f588d Tweak naming and organization of external-scanner related language fields 2016-12-21 11:24:41 -08:00
Max Brunsfeld
42c41c158c Refactor logic for handling shared internal/external tokens 2016-12-21 10:49:55 -08:00
Max Brunsfeld
e6c82ead2c Start work toward maintaining external scanner's state during incremental parses 2016-12-20 17:06:20 -08:00
Max Brunsfeld
2b3da512a4 Add serialize, deserialize and reset callbacks to external scanners 
Signed-off-by: Nathan Sobo <nathan@github.com>
 2016-12-20 13:12:01 -08:00
Max Brunsfeld
a1770ce844 Allow external tokens to be used as extras 2016-12-12 22:06:01 -08:00
Max Brunsfeld
10b51a05a1 Allow external scanners to refer to (and return) internally-defined tokens 
Tokens that are defined in the grammar's rules may now be included in the
externals list also, so that external scanners can check if they are valid
lookaheads or not, and if so, can return them to the parser if needed.
 2016-12-09 13:32:58 -08:00
Max Brunsfeld
83514293b5 Allow external tokens to be either visible or hidden 2016-12-05 17:26:11 -08:00
Max Brunsfeld
1251ff2e30 Consider externals to be named, not anonymous 2016-12-05 17:09:22 -08:00
Max Brunsfeld
c16b6b2059 Run external scanners during error recovery 2016-12-05 11:50:24 -08:00
Max Brunsfeld
49d25bd0f8 Remove EXTERNAL_TOKEN grammar rule type 2016-12-04 15:02:32 -08:00
Max Brunsfeld
d72b49316b Handle external tokens in apply_transitive_closure 2016-12-04 10:40:32 -08:00
Max Brunsfeld
0f8e130687 Call external scanner functions when lexing 2016-12-02 22:03:48 -08:00
Max Brunsfeld
c966af0412 Start work on external tokens 2016-12-02 16:24:19 -08:00
Max Brunsfeld
be9e79db1b Avoid incorrect application of precedence 2016-12-01 10:24:06 -08:00
Max Brunsfeld
996ca91e70 Disallow syntax rules that match the empty string (for now) 2016-11-30 23:19:54 -08:00
Max Brunsfeld
101e304a8a Avoid unnecessary lookahead set mutations in ParseItemSetBuilder 2016-11-20 21:41:36 -08:00
Max Brunsfeld
06215607d1 Precompute transitive closure contributions by grammar symbol 2016-11-20 11:49:55 -08:00
Max Brunsfeld
5332fd3418 Fix build warnings 2016-11-19 20:47:43 -08:00
Max Brunsfeld
6cf4ccb840 Represent rule metadata as a struct, not a map 2016-11-19 13:59:34 -08:00
Max Brunsfeld
cab1bd3ac5 Make conflict messages explicit about precedence combinations 2016-11-18 17:05:16 -08:00
Max Brunsfeld
5924285e69 🎨 2016-11-18 16:14:05 -08:00
Max Brunsfeld
32387400c6 Rework LR conflict resolution 
* Unify precedence/associativity-based resolution with the
  search for a whitelisted conflict
* Improve conflict error messages
 2016-11-18 13:50:55 -08:00
Max Brunsfeld
6935f1d26f Use hash_combine everywhere 2016-11-16 11:46:22 -08:00
Max Brunsfeld
6cfd009503 Compute parse state group signature based on the item set 2016-11-16 10:21:30 -08:00
Max Brunsfeld
42d37656ea Optimize remove_duplicate_parse_states method 
Signed-off-by: Nathan Sobo <nathan@github.com>
 2016-11-15 17:51:52 -08:00
Max Brunsfeld
1118a9142a Introduce Symbol::Index type alias 2016-11-14 10:25:26 -08:00
Max Brunsfeld
a89f8c086b Remove stray #include 2016-11-14 09:31:32 -08:00
Max Brunsfeld
fad7294ba4 Store shift states for non-terminals directly in the main parse table 2016-11-14 08:36:06 -08:00
Max Brunsfeld
8d9c261e3a Don't include reduce actions for nonterminal lookaheads 2016-11-10 11:33:37 -08:00
Max Brunsfeld
255bc2427c 🎨 build_parse_table 2016-11-09 20:47:47 -08:00