No need to restrict it to char sets used in multiple places.
This is important because the helper functions are now implemented
more efficiently than the inline comparisons (using a binary search).
We were only walking one level of depth into the `queries/` folder
during invocations of `test`, which made us attempt to open folders
rather than recurse into them.
We have to pull in the `walkdir` crate, which is required for
cross-platform walking of directories.
Fixes#938.
Right now, the strings are not used in comparisons, but they
are passed through the grammar processing pipeline, and are
available to the parse table construction algorithm.
This also cleans up a confusing aspect of the parse table
construction, in which precedences and associativities were
temporarily stored in the parse table data structure itself.
These were used to provide the `ts_record_*` functions regardless of
whether the `allocation-tracking` feature is enabled. The allocation
tracking code is now implemented entirely in the `lib` crate, and only
when the feature is enabled, and so these stubs are no longer needed.
We have several test cases defined in the `cli` crate that depend on the
`lib` crate's `allocation-tracking` feature. The implementation of the
actual allocation tracker used to live in the `cli` crate, close to the
test cases that use it. The `allocation-tracking` feature in the `lib`
crate was just used to tell the tree-sitter implementation to expect
that the allocation tracker exists, and to use it.
That pattern meant that we had a circular dependency: `cli` depends on
`lib`, but `lib` required some code that was implemented in `cli`.
That, in turn, caused linker errors — but only when compiling in certain
configurations! [1]
This patch moves all of the allocation tracking implementation into the
`lib` crate, gated on the existing `allocation-tracking` feature, which
fixes the circular dependency.
Note that this patch does **not** fix the fact that feature unification
causes the `lib` crate to be built with the `allocation-tracking`
feature enabled, even though it's not a default. Fixing that depends on
the forthcoming version 2 feature resolver [2], or using the `dev_dep`
workaround [3] in the meantime.
[1] https://github.com/tree-sitter/tree-sitter/issues/919
[2] https://doc.rust-lang.org/nightly/cargo/reference/features.html#feature-resolver-version-2
[3] https://github.com/tree-sitter/tree-sitter/issues/919#issuecomment-777107086
