Start work on contributing guide

This commit is contained in:
Max Brunsfeld 2019-05-30 17:52:54 -07:00
parent d274e81d0d
commit 2f06abf984
2 changed files with 143 additions and 3 deletions

View file

@ -3,7 +3,7 @@ title: Using Parsers
permalink: using-parsers
---
# Using Parsers
# Ucsing Parsers
All of Tree-sitter's parsing functionality is exposed through C APIs. Applications written in higher-level languages can use Tree-sitter via binding libraries like [node-tree-sitter](https://github.com/tree-sitter/node-tree-sitter) or [rust-tree-sitter](https://github.com/tree-sitter/tree-sitter/tree/master/lib/binding_rust), which have their own documentation.
@ -36,9 +36,9 @@ Alternatively, you can use the library in a larger project by adding one source
## The Objects
There are four main types of objects involved when using Tree-sitter: languages, parsers, syntax trees, and syntax nodes. In C, these are called `TSLanguage`, `TSParser`, `TSTree`, and `TSNode`.
* A `TSLanguage` is an opaque object that defines how to parse a particular programming language. The code for each `TSLanguage` is generated by Tree-sitter. Many languages are already available in separate git repositories within the the [Tree-sitter GitHub organization](https://github.com/tree-sitter). See [the next section](./creating-parsers) for how to create new languages.
* An `TSLanguage` is an opaque object that defines how to parse a particular programming language. The code for each `TSLanguage` is generated by Tree-sitter. Many languages are already available in separate git repositories within the the [Tree-sitter GitHub organization](https://github.com/tree-sitter). See [the next section](./creating-parsers) for how to create new languages.
* A `TSParser` is a stateful object that can be assigned a `TSLanguage` and used to produce a `TSTree` based on some source code.
* A `TSTree` represents the syntax tree of an entire source code file. Its contains `TSNode` instances that indicate the structure of the source code. It can also be edited and used to produce a new `TSTree` in the event that the source code changes.
* A `TSTree` represents the syntax tree of an entire source code file. It contains `TSNode` instances that indicate the structure of the source code. It can also be edited and used to produce a new `TSTree` in the event that the source code changes.
* A `TSNode` represents a single node in the syntax tree. It tracks its start and end positions in the source code, as well as its relation to other nodes like its parent, siblings and children.
## An Example Program

View file

@ -0,0 +1,140 @@
---
title: Contributing
permalink: contributing
---
# Contributing
## Developing Tree-sitter
### Prerequisites
To make changes to Tree-sitter, you should have:
1. A C compiler, for compiling the core library and the generated parsers.
2. A [Rust toolchain](https://rustup.rs/), for compiling the Rust bindings, the highlighting library, and the CLI.
3. Node.js and NPM, for generating parsers from `grammar.js` files.
4. Either [Docker](https://www.docker.com/) or [Emscripten](https://emscripten.org/), for compiling the library to WASM.
### Building
Clone the repository:
```sh
git clone https://github.com/tree-sitter/tree-sitter
cd tree-sitter
```
Clone the [`utf8proc`](https://juliastrings.github.io/utf8proc/) submodule:
```
git submodule update --init
```
Build the WASM library. We do this first because it gets embedded in the CLI to enable the `web-ui` command. If you have emscripten installed, this will use your `emcc` compiler. Otherwise, it will use Docker:
```sh
./script/build-wasm
```
Build the Rust libraries and the CLI:
```sh
cargo build --release
```
This will create the `tree-sitter` CLI executable in the `target/release` folder.
### Testing
Before you can run the tests, you need to clone some grammars that are used for testing:
```sh
script/fetch-fixtures
```
To test any changes you've made to the CLI, you can regenerate these parsers using your current CLI code:
```sh
script/generate-fixtures
```
Then you can run the tests:
```sh
script/test
```
### Debugging
The test script has a number of useful flags. You can list them all by running `script/test -h`. Here are some of the main flags:
If you want to run a specific unit test, pass its name (or part of its name) as an argument:
```sh
script/test test_does_something
```
You can run the tests under the debugger (either `lldb` or `gdb`) using the `-g` flag:
```sh
script/test test_does_something -g
```
Part of the Tree-sitter test suite involves parsing the *corpus* tests for several different languages and performing randomized edits to each example in the corpus. If you just want to run the tests for a particular *language*, you can pass the `-l` flag. And if you want to run a particular *example* from the corpus, you can pass the `-e` flag:
```sh
script/test -l javascript -e Arrays
```
## Published Packages
The main [`tree-sitter/tree-sitter`](https://github.com/tree-sitter/tree-sitter) repository contains the source code for several packages that are published to package registries for different languages:
* Rust crates on [crates.io](https://crates.io):
* [`tree-sitter`](https://crates.io/crates/tree-sitter) - A Rust binding to the core library
* [`tree-sitter-highlight`](https://crates.io/crates/tree-sitter-highlight) - The syntax-highlighting library
* [`tree-sitter-cli`](https://crates.io/crates/tree-sitter-cli) - The command-line tool
* JavaScript modules on [npmjs.com](https://npmjs.com):
* [`web-tree-sitter`](https://www.npmjs.com/package/web-tree-sitter) - A WASM-based JavaScript binding to the core library
* [`tree-sitter-cli`](https://www.npmjs.com/package/tree-sitter-cli) - The command-line tool
There are also several other dependent repositories that contain other published packages:
* [`tree-sitter/node-tree-sitter`](https://github.com/tree-sitter/py-tree-sitter) - Node.js bindings to the core library, published as [`tree-sitter`](https://www.npmjs.com/package/tree-sitter) on npmjs.com
* [`tree-sitter/py-tree-sitter`](https://github.com/tree-sitter/py-tree-sitter) - Python bindings to the core library, published as [`tree-sitter`](https://pypi.org/project/tree-sitter) on [PyPI.org](https://pypi.org).
## Publishing New Releases
Publishing a new release of the CLI requires these steps:
1. Commit and push all outstanding changes and verify that CI passes:
```sh
git commit -m "Fix things"
git push
```
2. Create a new tag:
```sh
script/version patch
```
This will determine the current version, increment the *patch* version number, and update the `Cargo.toml` and `package.json` files for the Rust and Node CLI packages. It will then create a commit and a tag for the new version. For more information about the arguments that are allowed, see the documentation for the [`npm version`](https://docs.npmjs.com/cli/version) command.
3. Push the commit and the tag:
```sh
git push
git push --tags
```
4. Wait for CI to pass. Because of the git tag, the CI jobs will publish artifacts to [a GitHub release](https://github.com/tree-sitter/tree-sitter/releases). The npm module of `tree-sitter-cli` works by downloading the appropriate binary from the corresponding GitHub release during installation. So it's best not to publish the npm package until the binaries are uploaded.
5. Publish the npm package:
```sh
cd cli/npm
npm publish
```