Commit graph

112 commits

Author SHA1 Message Date
Will Lillis
0c35511aea fix(lib): improve wasm scanner serialization error handling
Co-authored-by: Amaan Qureshi <contact@amaanq.com>
2025-09-13 22:29:47 -04:00
Amaan Qureshi
821cf797f2 feat(lib)!: remove deprecated functions 2025-09-12 02:50:30 -04:00
Will Lillis
8bd923ab9e fix(lib): replace raw array accesses with array_get 2025-06-05 00:53:11 -04:00
Riley Bruins
733d7513af fix(lib): reset parser options after use
**Problem:** After `ts_parser_parse_with_options()`, the parser options
are still stored in the parser object, meaning that a successive call to
`ts_parser_parse()` will actually behave like
`ts_parser_parse_with_options()`, which is not obvious and can have
unintended consequences.

**Solution:** Reset to empty options state after
`ts_parser_parse_with_options()`.
2025-04-14 21:35:40 -04:00
Max Brunsfeld
066fd77d39
Fix cases where error recovery could infinite loop (#4257)
* Rename corpus test functions to allow easy filtering by language

* Use usize for seed argument

* Avoid retaining useless stack versions when reductions merge

We found this problem when debugging an infinite loop that happened
during error recovery when using the Zig grammar. The large number of
unnecessary paused stack versions were preventing the correct recovery
strategy from being tried.

* Fix leaked lookahead token when reduction results in a merged stack

* Enable running PHP tests in CI

* Fix possible infinite loop during error recovery at EOF

* Account for external scanner state changes when detecting changed ranges in subtrees
2025-03-04 13:50:56 -08:00
Max Brunsfeld
dedcc5255a
Ignore external tokens that are zero-length and extra (#4213)
Co-authored-by: Anthony <anthony@zed.dev>
2025-02-17 15:07:44 -08:00
Riley Bruins
9ad096ef22 fix(lib): prevent finished_tree assertion failure
**Problem:** When resetting the parser during subtree balancing, an
error is thrown:

```
parser.c:2198: ts_parser_parse: Assertion `self->finished_tree.ptr' failed.
```

**Solution:** Reset `canceled_balancing` to false in
`ts_parser_reset()`.
2025-02-01 16:19:14 -05:00
Allan Clements
cda634a1c4 feat: add error information in the progress callback
This allows users to bail parsing if an error was *definitely* detected
using the progress callback, as all possible stack versions have a
non-zero error cost.

Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>
2025-01-25 02:47:39 -05:00
Amaan Qureshi
8bb1448a6f feat: add the semantic version to TSLanguage, and expose an API for retrieving it 2025-01-25 01:14:30 -05:00
Amaan Qureshi
c8353a52af fix(lib): don't always clear the tree stack
Only do so if the parser is not resuming balancing
2025-01-21 00:31:34 -05:00
Amaan Qureshi
9365586cc3 feat: allow parser balancing to be cancellable 2025-01-20 23:52:19 -05:00
Max Brunsfeld
201b41cf11
feat: add 'reserved word' construct
Co-authored-by: Amaan Qureshi <amaanq12@gmail.com>
2024-12-23 03:06:32 -05:00
Amaan Qureshi
500f4326d5 feat: add the ability to specify a custom decode function 2024-10-31 22:51:40 -04:00
Amaan Qureshi
26b89da9bb feat(lib): add ts_parser_parse_with_options
Currently, this allows users to pass in a callback that should be
invoked to check whether or not to halt parsing
2024-10-31 21:58:35 -04:00
Amaan Qureshi
aaba7cd2f9 feat: implement a cache for get_column 2024-10-30 18:35:38 -04:00
ObserverOfTime
7715001692 build: tune compiler warnings 2024-10-11 18:33:07 -04:00
Amaan Qureshi
8943983df6 feat!: properly handle UTF-16 endianness encoding 2024-10-05 21:12:48 -04:00
Amaan Qureshi
28972f916a fix(lib): silence warnings with -Wpedantic 2024-09-27 12:58:26 -04:00
Amaan Qureshi
3f424c0121 feat: add an API to time out query executions
Currently, if a predicate is hard to match on the Rust side, a sizable
query against a very large file can take forever, and ends up hanging.
This commit adds an API function `ts_query_cursor_set_timeout_micros` to
limit how long query execution is allowed to take, thereby negating the
chance of a hang to occur.
2024-08-31 14:33:28 -04:00
Ryan Patterson
779566f588
Reset language when resetting wasm store (#3495)
* Reset language when resetting wasm store

* test behavior of language copying
2024-07-31 10:30:58 -07:00
Davis Vaughan
218a071d33 Swap sprintf() for snprintf() 2024-06-24 09:56:24 -04:00
Max Brunsfeld
78efd96bbe Crash earlier if native external scanner overflows serialization buffer 2024-04-25 13:55:47 -07:00
Torsten Schmits
cdd00defdb
fix(lib): use correct format specifier in log message 2024-04-04 14:51:15 -04:00
Max Brunsfeld
4676cd4200 Defer creation of external scanner until the beginning of parse 2024-03-17 10:20:25 -07:00
Max Brunsfeld
7a9b3076ef Handle memory errors occurring in wasm scanners
* In WASM, use a custom, simple malloc implementation that lets us
  expicitly reset the heap with a new start location.
* When a WASM call traps or errors, propagate that as a parse failure.
* Reset the WASM heap after every parse.

Co-authored-by: Conrad <conrad@zed.dev>
2024-03-17 10:19:42 -07:00
geekvest
b43e8ce902
docs: remove duplicate the's
Signed-off-by: geekvest <cuimoman@sohu.com>
2024-03-03 10:30:21 -05:00
ObserverOfTime
799833f9cf build: use c11 everywhere
And improve the makefiles
2024-02-27 15:54:38 -05:00
Max Brunsfeld
dfc6d93bae Rename wasm C sources to avoid conflicts with standard wasm headers
Co-authored-by: Marshall <marshall@zed.dev>
2024-02-26 13:08:30 -08:00
Amaan Qureshi
037c71c7bd fix: apply some scan-build suggestions (unused assignment/garbage access) 2024-02-21 15:38:13 -05:00
dundargoc
ce191e02c5 fix: use correct printf specifiers 2024-02-08 22:46:43 +01:00
dundargoc
df1fe842eb docs: various fixes
Closes https://github.com/tree-sitter/tree-sitter/issues/1317.
Closes https://github.com/tree-sitter/tree-sitter/issues/1752.
Closes https://github.com/tree-sitter/tree-sitter/issues/2439.

Co-authored-by: Simon Hengel <sol@typeful.net>
Co-authored-by: Akash Yadav <itsaky01@gmail.com>
Co-authored-by: Sebastiaan Speck <12570668+sebastiaanspeck@users.noreply.github.com>
Co-authored-by: Laytan Laats <laytanlaats@hotmail.com>
2024-02-08 00:08:59 +01:00
Max Brunsfeld
4a8e4b1963 Allow wasm languages to be deleted 2023-12-27 15:00:16 -08:00
Max Brunsfeld
034f0d0280
Merge pull request #1864 from tree-sitter/wasm-language
Add optional WASM feature to the native library, allowing it to run wasm-compiled parsers via wasmtime
2023-11-28 12:08:47 -08:00
Max Brunsfeld
5e2456c2f2 Avoid using recursion for ts_subtree_compare
This can lead to stack overflow crashes.
2023-11-27 11:35:33 -08:00
Max Brunsfeld
f4e2f68f14 Merge branch 'master' into wasm-language 2023-10-27 12:11:43 +01:00
Amaan Qureshi
05a4f9f3e3
Merge pull request #1411 from aminya/c-warnings
Fix compiling C library with Clang on Windows
2023-08-03 13:39:52 -04:00
Amaan Qureshi
13f6ec2b0c
fix: rename shadowed variables from -Wshadow warnings and apply some useful clang-tidy warnings 2023-07-19 18:12:26 -04:00
Amaan Qureshi
753fa1c3ff
fix(lib): explicitly cast numbers to the same size in potential spots for infinite loops 2023-07-19 03:49:14 -04:00
Max Brunsfeld
0b817a609f Clear the parse stack when terminating parsing early due to error cost
This fixes a bug where the parse tree would not be rebalanced if this
code path was taken.
2023-02-13 13:45:12 -08:00
Max Brunsfeld
f7c9989b48 Fix crash when parser's wasm store is taken before its language is changed 2023-01-23 15:26:54 -08:00
Max Brunsfeld
555277a102 Allow testing highlight and tag queries when testing wasm build
Replace non-mutating `ts_parser_wasm_store` function with
`ts_parser_take_wasm_store`, which removes and returns the wasm
store, in order to facilitate single ownership.
2023-01-23 11:46:59 -08:00
Max Brunsfeld
51720beeb0 Start work on a --wasm flag for the test subcommand 2022-11-15 17:14:33 -08:00
Max Brunsfeld
7dc81303f6 Get wasm store working with parser with stateless external scanners 2022-11-15 17:14:33 -08:00
Max Brunsfeld
042e6f9d57 Restucture wasm module, return a normal TSLanguage from load_language 2022-11-15 17:14:33 -08:00
Max Brunsfeld
3f1a7f9cd4 Start work on ability to load wasm languages from native lib, via wasmtime 2022-11-15 17:14:33 -08:00
Max Brunsfeld
efd22e452b Fix suppression of empty tokens during error handling at included range boundaries 2022-11-14 12:20:39 -08:00
Logan Collins
5337a678fa Resolving a potential crash freeing a null pointer when cancelling parses. 2022-10-31 11:35:52 -07:00
Matt
af6611e530
matching integer types 2022-09-21 08:50:01 -04:00
Max Brunsfeld
3ac36b0cbe Handle backslashes in token names when printing DOT debug graphs 2022-06-25 17:13:11 -07:00
Max Brunsfeld
5aa2f4dc8c Log when ignoring an empty external token after an error 2022-06-24 19:07:27 -07:00