We have several test cases defined in the `cli` crate that depend on the
`lib` crate's `allocation-tracking` feature. The implementation of the
actual allocation tracker used to live in the `cli` crate, close to the
test cases that use it. The `allocation-tracking` feature in the `lib`
crate was just used to tell the tree-sitter implementation to expect
that the allocation tracker exists, and to use it.
That pattern meant that we had a circular dependency: `cli` depends on
`lib`, but `lib` required some code that was implemented in `cli`.
That, in turn, caused linker errors — but only when compiling in certain
configurations! [1]
This patch moves all of the allocation tracking implementation into the
`lib` crate, gated on the existing `allocation-tracking` feature, which
fixes the circular dependency.
Note that this patch does **not** fix the fact that feature unification
causes the `lib` crate to be built with the `allocation-tracking`
feature enabled, even though it's not a default. Fixing that depends on
the forthcoming version 2 feature resolver [2], or using the `dev_dep`
workaround [3] in the meantime.
[1] https://github.com/tree-sitter/tree-sitter/issues/919
[2] https://doc.rust-lang.org/nightly/cargo/reference/features.html#feature-resolver-version-2
[3] https://github.com/tree-sitter/tree-sitter/issues/919#issuecomment-777107086
Previously, we attempted to completely separate the parse states
for item sets with non-terminal extras from the parse states
for other rules. But there was not a complete separation.
It actually isn't necessary to separate the parse states in this way.
The only special behavior for parse states with non-terminal extra rules
is what happens at the *end* of the rule: these parse states need to
perform an unconditional reduction.
Luckily, it's possible to distinguish these *non-terminal extra ending*
states from other states just based on their normal structure, with
no additional state.
In debug building modules also may happen errors and a current implementation
completely hides them, so errors like 'undefined symbol' can't be
easily identified due to wrong traceback and error message.
This is just a minimal set of changes to dependencies.
macOS aarch64 support was only introduced in `cc` version 1.0.58, so this now allows tree-sitter to build natively on M1 computers
The symbols marked as "extras" are the start symbols of secondary
languages. These should be included in the aliases map just as done
for start symbol of the main language to ensure their node type and
field information is included in the node-types.json file.
