traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
3012 commits 16 branches 98 tags 37 MiB
Commit graph

155 commits

Author SHA1 Message Date
Riccardo Schirone
780e9cecc9 Do not use multiple unnamed structs inside of unions 2020-04-29 20:42:45 +02:00
Max Brunsfeld
a003e5f6bd generate: Avoid duplicate string tokens in unique symbol map 2020-03-20 11:35:11 -07:00
Max Brunsfeld
6cb8d24de2
Merge pull request #542 from SKalt/issue-524-document-supertypes-in-grammar-schema 
feat(cli): documented optional supertypes string[] in grammar schema
 2020-02-24 16:14:49 -08:00
Steven Kalt
d82ee739e9
Update cli/src/generate/grammar-schema.json 
Co-Authored-By: Max Brunsfeld <maxbrunsfeld@github.com>
 2020-02-24 18:13:38 -05:00
Alyssa Verkade
0e689657b7 Add a language linkage declaration to parsers 
Previously, in order to compile a `tree-sitter` grammar that contained
c++ source in the parser (ie the `scanner.cc` file), you would have to
compile the `parser.c` file separately from the c++ files. For example,
in rust this would result in a `build.rs` close to the following:
```
extern crate cc;

fn main() {
  let dir: PathBuf = ["tree-sitter-ruby", "src"].iter().collect();

  cc::Build::new()
    .include(&dir)
    .cpp(true)
    .file(dir.join("scanner.cc"))
    // NOTE: must have a name that differs from the c static lib
    .compile("tree-sitter-ruby-scanner");

  cc::Build::new()
    .include(&dir)
    .file(dir.join("parser.c"))
    // NOTE: must have a name that differs from the c++ static lib
    .compile("tree-sitter-ruby-parser");
}
```

This was necessary at the time for the following grammars: `ruby`,
`php`, `python`, `embedded-template`, `html`, `cpp`, `ocaml`,
`bash`, `agda`, and `haskell`.

To solve this, we specify an `extern "C"` language linkage declaration
to the functions that must be linked against to compile a parser with the
scanner, making parsers linkable against c++ source.
On all major compilers (gcc, clang, and msvc) this should be the only
change needed due to the combination of clang and gcc both supporting
designated initialization for years and msvc 2019 adopting designated
initializers as a part of the C++20 conformance push.

Subsequently, for rust projects, the necessary `build.rs` would become
(which also brings these parsers into sync with the current docs):
```
extern crate cc;

fn main() {
  let dir: PathBuf = ["tree-sitter-ruby", "src"].iter().collect();

  cc::Build::new()
    .include(&dir)
    .cpp(true)
    .file(dir.join("scanner.cc"))
    .file(dir.join("parser.c"))
    .compile("tree-sitter-ruby");
}
```
 2020-02-18 19:46:59 -08:00
Steven Kalt
c5ca259d09
feat(cli): documented optional supertypes Array<string> in grammar-schema.json 2020-02-15 11:23:14 -05:00
Max Brunsfeld
5922064b75 node-types: Fix ambiguity warning on rustc 1.41 2020-02-10 10:26:12 -08:00
Max Brunsfeld
8dd68c360a Fix logic for generating unique symbol map 
Previously, this didn't correctly handle the case where *multiple* 
symbols were all simply-aliased to the same *other* symbol.

Refs #500
 2020-01-27 12:06:48 -08:00
Steven Kalt
619d7cd65a
added field-rule to grammar-schema.json 2020-01-12 19:40:21 -05:00
Steven Kalt
e69430ae7d removed duplicate key (#521) 
The key "required" was duplicated on "symbol-rule". I removed the more permissive copy.
 2020-01-09 17:34:07 -08:00
Max Brunsfeld
f0e4d630d5 Reliably avoid duplicate subtypes in node-types.json 2019-12-17 14:55:51 -08:00
Max Brunsfeld
3ac047679c Fix children.required when multiple nodes are aliased as the same 2019-12-16 12:39:57 -08:00
Max Brunsfeld
7c711c5537 Move the private functions in node_types.rs to the end of the file 2019-12-16 12:38:40 -08:00
Max Brunsfeld
9a0cfa2376 Fix 'required' field when rules with diff fields are aliased to look the same 2019-12-12 11:26:58 -08:00
Max Brunsfeld
fc19312913 Fix node-types bugs involving aliases and external tokens 2019-12-12 10:06:18 -08:00
Max Brunsfeld
1b5ae380ce
Don't attempt to extract keywords that don't entirely match word token (#505) 2019-12-11 17:18:15 -08:00
Max Brunsfeld
a5a9000e29 generate: Ensure that field_map_slices array is long enough 2019-12-09 11:46:32 -08:00
Max Brunsfeld
7032dae4f6 Include alias symbols in unique symbol map 2019-12-06 12:11:09 -08:00
Max Brunsfeld
56c620c005 Store a mapping to ensure no two symbols map to the same metadata 2019-12-05 17:21:46 -08:00
Max Brunsfeld
5a979d1457 node-types: Add test for field merging with aliases 
Co-Authored-By: Timothy Clem <timothy.clem@gmail.com>
 2019-11-19 14:31:56 -08:00
Max Brunsfeld
1d63434664 Refactor node-type merging 2019-11-19 11:57:21 -08:00
Max Brunsfeld
5489bc4dc5 Fix small issues with merging node types 
* Merge the `required` field with an 'and', not an 'or'
* Merge field info in addition to children info
 2019-11-19 11:55:21 -08:00
Timothy Clem
e2325102d1 No printing 2019-11-19 09:03:35 -08:00
Timothy Clem
c346ce4a5e Try not to loose existing children 2019-11-18 14:48:24 -08:00
Timothy Clem
5c72642634 A test demonstrating the issue with named aliases 2019-11-18 14:47:48 -08:00
Max Brunsfeld
660efd8c0f Fix handling of token rules directly inside of alias, prec, or field. 
Fixes #483
 2019-11-15 10:53:17 -08:00
Max Brunsfeld
d45b98c7d9 node-types - Fix duplicate when a terminal is aliased with the same name as a non-terminal 2019-11-14 09:31:25 -08:00
Max Brunsfeld
5767bbc806 Avoid generating C char literals with control characters 
Fixes #487
 2019-11-13 10:54:34 -08:00
Max Brunsfeld
a2bbc7391b node-types: Fix handling of simple aliased inlined rules 2019-11-13 10:38:47 -08:00
Max Brunsfeld
d765332c61 Don't rely on new eof ABI in parsers unless --next-abi is passed 2019-10-31 14:32:50 -07:00
Max Brunsfeld
d3b7caa565 Add a TSLexer.eof() API, use it in generated parsers 2019-10-31 14:11:52 -07:00
Max Brunsfeld
0cceca7b4e Rename extra_tokens -> extra_symbols 2019-10-21 17:26:01 -07:00
Max Brunsfeld
fcaabea0cf Allow non-terminal extras 2019-10-21 16:08:59 -07:00
Max Brunsfeld
b79295e1a0 🔥 Everything related to property sheets 2019-10-17 11:00:31 -07:00
Max Brunsfeld
dc7997fdbb node-types: Rework the approach to computing multiple and required 2019-10-11 13:59:02 -07:00
Max Brunsfeld
fb7a5369fe Fix regression in node-types for rules with simple aliases 2019-10-02 14:11:21 -07:00
Max Brunsfeld
754cfd42bf Respect nodes' aliases when generating node types 
Fixes #455
 2019-10-02 13:59:00 -07:00
Max Brunsfeld
c9f46b8242 Fix false negative in token conflict detection 
Co-Authored-By: Timothy Clem <timothy.clem@gmail.com>
 2019-09-19 11:50:38 -07:00
Max Brunsfeld
69ab405325 In next ABI, group symbols by action in small parse state table 
This is a more compact representation because in most states, many 
symbols share the same actions.
 2019-08-30 20:29:55 -07:00
Max Brunsfeld
8037607583 Only generate the new parse table format if --next-abi flag is used 2019-08-29 17:37:33 -07:00
Max Brunsfeld
aeb2f895b4 Add --report-states flag for reporting state counts for each rule 2019-08-29 17:36:39 -07:00
Max Brunsfeld
82ff542d3b Appease MSVC by avoiding empty arrays 2019-08-29 17:31:44 -07:00
Max Brunsfeld
09a2755399 Store parse states with few lookahead symbols in a more compact way 2019-08-29 15:52:23 -07:00
Max Brunsfeld
48a883c1d4 Move external token state id computation out of render module 2019-08-29 15:48:22 -07:00
Max Brunsfeld
759c1d6e65 Reorder parse states by descending symbol count 2019-08-29 15:28:29 -07:00
Max Brunsfeld
2430733ee8 Avoid iterating hashmaps in places where order matters 2019-08-29 15:26:05 -07:00
Max Brunsfeld
f371507d39 Move TokenSet to rules module 2019-08-29 15:25:45 -07:00
Max Brunsfeld
d5b5d473ab Add missing comparisons to property item hash & equality impls 
Fixes the inconsistent failurs of property sheet unit tests
 2019-08-26 09:31:48 -07:00
Max Brunsfeld
5e04daf483 Avoid non-deterministic set iteration order when handling conflicts 2019-08-13 15:57:42 -07:00
Max Brunsfeld
56ce4e5d50 Upgrade rsass, remove hashbrown 2019-08-13 10:08:58 -07:00