traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore: remove unnecessary fuzz and profile helpers

Browse source
This commit is contained in:
Amaan Qureshi 2024-10-07 23:23:54 -04:00
parent 9c08edb066
commit cad2d03101
3 changed files with 0 additions and 164 deletions
Download patch file Download diff file Expand all files Collapse all files

43
test/fuzz/README.md
View file

@ -1,43 +0,0 @@
# Fuzzing tree-sitter
The tree-sitter fuzzing support requires 1) the `libFuzzer` runtime library and 2) a recent version of clang
## libFuzzer
The main fuzzing logic is implemented by `libFuzzer` which is part of the compiler-rt project but is not shipped by distros. `libFuzzer` will need to be built from source, e.g.:
```
cd ~/src
git clone https://github.com/llvm-mirror/compiler-rt
cd compiler-rt/lib/fuzzer
./build.sh
```
## clang
Using libFuzzer requires at least version 7 of `clang` and may _not_ work with your system-installed version. If your system-installed version is too old, the easiest way to get started is to use the version provided by the Chromium team. Instructions are available at [libFuzzer.info](http://libfuzzer.info).
The fuzzers can then be built with:
```
export CLANG_DIR=$HOME/src/third_party/llvm-build/Release+Asserts/bin
CC="$CLANG_DIR/clang" CXX="$CLANG_DIR/clang++" LINK="$CLANG_DIR/clang++" \
LIB_FUZZER_PATH=$HOME/src/compiler-rt/lib/fuzzer/libFuzzer.a \
./script/build-fuzzers
```
This will generate a separate fuzzer for each grammar defined in `test/fixtures/grammars` and will be instrumented with [AddressSanitizer](https://clang.llvm.org/docs/AddressSanitizer.html) and [UndefinedBehaviorSanitizer](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html). Individual fuzzers can be built with, for example, `./script/build-fuzzers python ruby`.
The `run-fuzzer` script handles running an individual fuzzer with a sensible default set of arguments:
```
./script/run-fuzzer <grammar-name> (halt|recover) <extra libFuzzer arguments...>
```
which will log information to stdout. Failing testcases and a fuzz corpus will be saved to `fuzz-results/<grammar-name>`. The most important extra `libFuzzer` options are `-jobs` and `-workers` which allow parallel fuzzing. This is can done with, e.g.:
```
./script/run-fuzzer <grammar-name> halt -jobs=32 -workers=32
```
The testcase can be used to reproduce the crash by running:
```
./script/reproduce <grammar-name> (halt|recover) <path-to-testcase>
```

79
test/fuzz/fuzzer.cc
View file

@ -1,79 +0,0 @@
#include <cassert>
#include <fstream>
#include "tree_sitter/api.h"
extern "C" const TSLanguage *TS_LANG();
static TSQuery *lang_query;
extern "C" int LLVMFuzzerInitialize(int *argc, char ***argv) {
if(TS_LANG_QUERY_FILENAME[0]) {
// The query filename is relative to the fuzzing binary. Convert it
// to an absolute path first
auto binary_filename = std::string((*argv)[0]);
auto binary_directory = binary_filename.substr(0, binary_filename.find_last_of("\\/"));
auto lang_query_filename = binary_directory + "/" + TS_LANG_QUERY_FILENAME;
auto f = std::ifstream(lang_query_filename);
assert(f.good());
std::string lang_query_source((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
uint32_t error_offset = 0;
TSQueryError error_type = TSQueryErrorNone;
lang_query = ts_query_new(
TS_LANG(),
lang_query_source.c_str(),
lang_query_source.size(),
&error_offset,
&error_type
);
assert(lang_query);
}
return 0;
}
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
const char *str = reinterpret_cast<const char *>(data);
TSParser *parser = ts_parser_new();
// This can fail if the language version doesn't match the runtime version
bool language_ok = ts_parser_set_language(parser, TS_LANG());
assert(language_ok);
TSTree *tree = ts_parser_parse_string(parser, NULL, str, size);
TSNode root_node = ts_tree_root_node(tree);
if (lang_query != nullptr) {
{
TSQueryCursor *cursor = ts_query_cursor_new();
ts_query_cursor_exec(cursor, lang_query, root_node);
TSQueryMatch match;
while (ts_query_cursor_next_match(cursor, &match)) {
}
ts_query_cursor_delete(cursor);
}
{
TSQueryCursor *cursor = ts_query_cursor_new();
ts_query_cursor_exec(cursor, lang_query, root_node);
TSQueryMatch match;
uint32_t capture_index;
while (ts_query_cursor_next_capture(cursor, &match, &capture_index)) {
}
ts_query_cursor_delete(cursor);
}
}
ts_tree_delete(tree);
ts_parser_delete(parser);
return 0;
}

42
test/profile/heap.cc
View file

@ -1,42 +0,0 @@
#include <fstream>
#include <string>
#include <cstdlib>
#include <tree_sitter/api.h>
extern "C" {
#include "javascript/src/parser.c"
#include "javascript/src/scanner.c"
}
#define LANGUAGE tree_sitter_javascript
#define SOURCE_PATH "javascript/examples/jquery.js"
int main() {
TSParser *parser = ts_parser_new();
if (!ts_parser_set_language(parser, LANGUAGE())) {
fprintf(stderr, "Invalid language\n");
exit(1);
}
const char *source_path = GRAMMARS_DIR SOURCE_PATH;
printf("Parsing %s\n", source_path);
std::ifstream source_file(source_path);
if (!source_file.good()) {
fprintf(stderr, "Invalid source path %s\n", source_path);
exit(1);
}
std::string source_code(
(std::istreambuf_iterator<char>(source_file)),
std::istreambuf_iterator<char>()
);
TSTree *tree = ts_parser_parse_string(
parser,
NULL,
source_code.c_str(),
source_code.size()
);
}