Commit graph

67 commits

Author SHA1 Message Date
Amaan Qureshi
e553578696 feat: add fuzz subcommand 2024-05-30 23:00:45 -04:00
Amaan Qureshi
abc7910381 refactor(rust): misc fixes & tidying 2024-04-11 22:35:43 -04:00
Amaan Qureshi
b35efa8f33 style: format imports 2024-04-11 22:35:43 -04:00
Max Brunsfeld
09b18fad5b
Merge pull request #3181 from tree-sitter/handle-wasm-oom
When loading languages via WASM, gracefully handle memory errors and leaks in external scanners
2024-03-18 13:15:06 -07:00
Max Brunsfeld
7a9b3076ef Handle memory errors occurring in wasm scanners
* In WASM, use a custom, simple malloc implementation that lets us
  expicitly reset the heap with a new start location.
* When a WASM call traps or errors, propagate that as a parse failure.
* Reset the WASM heap after every parse.

Co-authored-by: Conrad <conrad@zed.dev>
2024-03-17 10:19:42 -07:00
Amaan Qureshi
e8dbe011d3 test: fix header writes 2024-03-17 05:36:30 -04:00
Amaan Qureshi
99b93d83a1 feat(cli)!: add a separate build command to compile parsers
This allows users to build parsers without having to run `test` or
`parse` to invoke the compilation process, and allows them to output the
object file to wherever they like. The `build-wasm` command was merged
into this by just specifying the `--wasm` flag.
2024-03-17 05:36:30 -04:00
Amaan Qureshi
f1eecf9786 feat: expose the allocator and array header files for external scanners 2024-02-26 09:43:13 -05:00
Amaan Qureshi
fd91404ab0 style: tidying 2024-02-19 16:00:50 -05:00
Amaan Qureshi
4342efd57e feat: allow specifying an external scanner's files 2024-02-16 16:31:56 -05:00
Amaan Qureshi
e996c32108 refactor!: remove the apply-all-captures flag, make last-wins precedence the default 2024-02-16 12:34:12 -05:00
Amaan Qureshi
5ea0dbf77a chore: some more clippy lints 2024-02-13 03:33:07 -05:00
Amaan Qureshi
a07f988905 refactor: extract regex check into a function and lower its precedence 2024-02-13 03:33:07 -05:00
Bedis Nbiba
7dd096c5f7
feat: implement first-line-regex 2024-02-13 01:51:41 -05:00
Amaan Qureshi
59be1edaa1
refactor: swap &Vec[T] with &[T] where appropriate 2024-02-07 02:50:31 -05:00
dundargoc
c8bd6705cf
chore: clippy 2024-02-06 23:34:14 -05:00
Amaan Qureshi
9f2dc9d6b5
fix: rework parser.h includes for test grammars and multi-grammar repos 2024-02-02 10:42:39 -05:00
Max Brunsfeld
da16cb1459 Introduce language ref-count management C APIs, remove Copy impl for Language in Rust 2023-12-27 14:59:16 -08:00
Max Brunsfeld
b59716600a Add a test that loads multiple languages in a wasm store 2023-11-24 20:03:54 -08:00
Max Brunsfeld
f4e2f68f14 Merge branch 'master' into wasm-language 2023-10-27 12:11:43 +01:00
Andrew Hlynskyi
7f7084c2cb chore(test): panic on zero pointer deallocs for alloc tracked scopes 2023-09-03 07:12:11 +03:00
Andrew Hlynskyi
67a5dbdd93 fix: dealloc calls on zero pointers 2023-09-03 06:53:18 +03:00
Andrew Hlynskyi
52f7eaff31 chore(rust): make Query extra predicates state fully immutable 2023-09-02 23:12:28 +03:00
Amaan Qureshi
dc5ec1cf53
refactor: remove &Option<T> where used 2023-09-01 17:23:39 -04:00
Amaan Qureshi
45313e0dfb
feat: add injection.self to inject a node with itself 2023-08-18 18:37:57 -04:00
Amaan Qureshi
8eb92d1c64
feat: add an --apply-all-captures argument to highlight & test 2023-08-17 22:20:57 -04:00
Andrew Hlynskyi
caaa533b8d test: add TREE_SITTER_DUMP_EDITS env var to dump corpus edits
It dumps edits to the `target/scratch` folder in a format like `<SEED>.edit`
2023-08-17 20:55:28 +03:00
Andrew Hlynskyi
0dd85c8452 Add TREE_SITTER_GRAMMAR_DEBUG env var to use debug grammars in tests 2023-08-17 18:16:41 +03:00
Amaan Qureshi
c521e9c18e
chore: improve error message in some spots loading grammar.json 2023-07-24 00:44:44 -04:00
Amaan Qureshi
e77405841e
refactor: make a subdirectory for the scratch dir depending on the machine's features 2023-07-18 20:14:21 -04:00
Amaan Qureshi
975f3f4686 refactor!: use AtomicUsize for AllocationRecorder for portability 2023-07-16 05:14:11 -04:00
Max Brunsfeld
356f68293a Fix false positive query match bug, introduced in #2085 2023-07-10 16:12:59 -04:00
Andrew Hlynskyi
ceac2380ed chore(test): reorganize test helpers 2023-05-12 19:29:51 +03:00
Andrew Hlynskyi
ddb0af9509 test: use random SEED numbers
This is needed to omit occurrences of the same seed in a sequence of
following seeds due to the reason of that two initial seed are very
close if based on unix epoch seconds.
2023-03-16 15:17:21 +02:00
Max Brunsfeld
d47713ee4a Integrate WASM compilation into the CLI's Loader 2022-11-15 17:14:33 -08:00
Max Brunsfeld
8e3dc7cd7a Add tests that randomly edit files with disjoint included ranges 2022-11-14 16:04:37 -08:00
Max Brunsfeld
8b5ff2ec8f Loosen randomized changed range assertions on newline characters
For now, don't error if the scope changes for a newline character.
2022-06-25 17:13:46 -07:00
Max Brunsfeld
7170ec7c96 Improve randomized testing setup
* Allow iterations to be specified via an env var
* Randomly decide the edit count, with a maximum
  specified via an env var.
* Instead of separate env vars for starting seed + trial, just accept a seed
* Remove some noisy output
2022-03-02 17:12:25 -08:00
Max Brunsfeld
5eb0a3090f
Merge pull request #1547 from the-mikedavis/md-test-tags
test tags queries in 'tree-sitter test'
2022-02-24 15:22:15 -08:00
Max Brunsfeld
622359b400 Simplify allocation-recording in test suite using new ts_set_allocator API 2021-12-30 16:09:07 -08:00
Michael Davis
47dc88b647
add tags config fixture test helper 2021-12-18 11:07:18 -06:00
Max Brunsfeld
f69c4861c3 Add a randomized test for query matching 2021-11-21 11:29:36 -08:00
Edgar
9feca80b08
Make HighlightConfiguration::configure use into iterator instead of a Vec<String> 2021-06-16 19:14:45 +02:00
Douglas Creager
e841fcfa1b cli: Extract CLI configuration into separate crate
This patch adds the `tree-sitter-config` crate, which manages
tree-sitter's configuration file.  This new setup allows different
components to define their own serializable configuration types, instead
of having to create a single monolithic configuration type.  But the
configuration itself is still stored in a single JSON file.

Before, the default location for the configuration file was
`~/.tree-sitter/config.json`.  This patch updates the default location
to follow the XDG Base Directory spec (or other relevant platform-
specific spec).  So on Linux, for instance, the new default location is
`~/.config/tree-sitter/config.json`.  We will look in the new location
_first_, and fall back on reading from the legacy location if we can't
find anything.
2021-06-10 10:21:22 -04:00
Douglas Creager
66c30648c2 rust: Extract runtime language detection into separate crate
This patch adds a new `tree-sitter-loader` crate, which holds the CLI's
logic for finding and building local grammar definitions at runtime.
This allows other command-line tools to use this logic too!
2021-06-09 17:42:30 -04:00
Andrew Hlynskyi
3c0152a331 chore(fmt): Apply 'cargo fmt' to the whole code base 2021-05-19 23:21:43 +03:00
Max Brunsfeld
a40045a419 When editing, properly invalidate trees that depend on get_column 2021-03-11 14:46:13 -08:00
hvithrafn
05f79f0f90 Update dependencies; fix breakage 2021-02-26 10:55:20 -07:00
Douglas Creager
a29c8d9264 Move allocation tracking into lib crate
We have several test cases defined in the `cli` crate that depend on the
`lib` crate's `allocation-tracking` feature.  The implementation of the
actual allocation tracker used to live in the `cli` crate, close to the
test cases that use it.  The `allocation-tracking` feature in the `lib`
crate was just used to tell the tree-sitter implementation to expect
that the allocation tracker exists, and to use it.

That pattern meant that we had a circular dependency: `cli` depends on
`lib`, but `lib` required some code that was implemented in `cli`.
That, in turn, caused linker errors — but only when compiling in certain
configurations! [1]

This patch moves all of the allocation tracking implementation into the
`lib` crate, gated on the existing `allocation-tracking` feature, which
fixes the circular dependency.

Note that this patch does **not** fix the fact that feature unification
causes the `lib` crate to be built with the `allocation-tracking`
feature enabled, even though it's not a default.  Fixing that depends on
the forthcoming version 2 feature resolver [2], or using the `dev_dep`
workaround [3] in the meantime.

[1] https://github.com/tree-sitter/tree-sitter/issues/919
[2] https://doc.rust-lang.org/nightly/cargo/reference/features.html#feature-resolver-version-2
[3] https://github.com/tree-sitter/tree-sitter/issues/919#issuecomment-777107086
2021-02-23 09:16:37 -05:00
Max Brunsfeld
b661050a61 Simplify setup for enabling/disabling allocation recording in the C lib 2020-12-02 15:35:13 -08:00