traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1439 commits 16 branches 98 tags 37 MiB
Commit graph

577 commits

Author SHA1 Message Date
Max Brunsfeld
7e13eac296 Fix lookahead_char type in ts_tree_make_error function 2017-03-21 11:05:48 -07:00
Max Brunsfeld
63fb041961 Merge remote-tracking branch 'origin/check-utf8proc_iterate-return' into update-fixture-grammars 2017-03-21 09:59:35 -07:00
Timothy Clem
f394a48c0b utf8proc_iterate can set codepoint_ref to -1 and returns negative error 2017-03-20 16:54:19 -07:00
Max Brunsfeld
ed31e82ee6 Skip empty tokens when recovering from errors 2017-03-19 22:20:59 -07:00
Max Brunsfeld
d222dbb9fd Allow lexer to accept tokens that ended at previous positions 
* Track lookahead in each tree
* Add 'mark_end' API that external scanners can use
 2017-03-13 17:06:52 -07:00
Max Brunsfeld
df520635c6 Prevent crash due to huge number of possible paths through parse stack 2017-02-20 14:34:10 -08:00
Max Brunsfeld
cefc57fe86 Move error cost comparisons into their own source file 2017-02-19 21:54:06 -08:00
Max Brunsfeld
5b4e6df3ff Don't mark error nodes created in the error state as extras 2017-02-19 21:54:06 -08:00
Max Brunsfeld
c14a776a3d Avoid including trailing extra tokens within error nodes unnecessarily 2017-02-19 21:21:54 -08:00
Max Brunsfeld
135d8ef4e0 Merge pull request #58 from tree-sitter/reduce-error-recovery-branching 
Reduce the branching factor of the parse stack during error recovery
 2017-02-18 11:34:09 -08:00
Rob Rix
638aa87e42 Pass through to ts_string_input_make_with_length. 2017-02-10 09:27:21 -05:00
Rob Rix
eab518e5da Semicolon shame. 2017-02-10 09:20:58 -05:00
Rob Rix
c230658bae Add public API to set the input string with explicit length. 2017-02-10 09:10:31 -05:00
Rob Rix
e6927238e1 Construct TSStringInput with explicit length. 2017-02-10 09:10:06 -05:00
Max Brunsfeld
93d7a75b09 Suppress one unnecessary type of error recovery variation 
If we already have a stack version in which, for example,
a `function_call` is skipped, don't create another stack
version in which that `function_call` is reduced to an
`expression`, and then the `expression` is skipped. That
doesn't improve the error recovery at all, but adds to the
branching factor of the parse stack and makes things harder
to debug.
 2017-02-07 22:07:56 -08:00
Max Brunsfeld
b01c5404eb Ensure error_end_position variable is initialized 2017-02-07 17:48:53 -08:00
Max Brunsfeld
343887c1dd Fix miscounting of extra tokens when repairing errors 2017-02-06 17:43:07 -08:00
Max Brunsfeld
d853b6504d Add version number to TSLanguage structs 2017-01-31 10:21:47 -08:00
Max Brunsfeld
672d491775 Fix errors in management of external scanner's most recent state 2017-01-30 22:04:46 -08:00
Max Brunsfeld
dc6598e07e Include external token states in stack debug graphs 2017-01-30 21:58:27 -08:00
Max Brunsfeld
896254eea5 Fix error in changed ranges calculation 
There was an error in the way that we calculate the reference
scope sequences that are used as the basis for assertions about
changed ranges in randomized tests. The error caused some
characters' scopes to not be checked. This corrects the reference
implementation and fixes a previously uncaught bug in the
implementation of `tree_path_get_changed_ranges`.

Previously, when iterating over the old and new trees, we would
only perform comparisons of visible nodes. This resulted in a failure
to do any comparison for portions of the text in which there were
trailing invisible child nodes (e.g. trailing `_line_break` nodes
inside `statement` nodes in the JavaScript grammar).

Now, we additionally perform comparisons at invisible leaf nodes,
based on their lowest visible ancestor.
 2017-01-27 23:47:34 -08:00
Max Brunsfeld
36608180d2 Store external token states in the parse stack 2017-01-08 22:06:05 -08:00
Max Brunsfeld
3a4daace26 Move reusable node functions to their own file 2017-01-05 10:07:27 -08:00
Max Brunsfeld
12cd2132ff Add test for retrieving last external token state in a Tree 2017-01-04 21:23:04 -08:00
Max Brunsfeld
d57043b665 Add ability to store external token state per stack version 2017-01-04 21:22:23 -08:00
Max Brunsfeld
2fa7b453c8 Restore external scanner's state only after repositioning lexer 
Also, properly identify the leaf node with the external token state
 2016-12-21 13:59:56 -08:00
Max Brunsfeld
4136dad5de Avoid referencing invalid union member in tree_path_descend 2016-12-21 13:21:21 -08:00
Max Brunsfeld
1595a02692 Avoid referencing invalid union member in tree_set_children 2016-12-21 12:23:24 -08:00
Max Brunsfeld
34a65f588d Tweak naming and organization of external-scanner related language fields 2016-12-21 11:24:41 -08:00
Max Brunsfeld
e6c82ead2c Start work toward maintaining external scanner's state during incremental parses 2016-12-20 17:06:20 -08:00
Max Brunsfeld
2b3da512a4 Add serialize, deserialize and reset callbacks to external scanners 
Signed-off-by: Nathan Sobo <nathan@github.com>
 2016-12-20 13:12:01 -08:00
Max Brunsfeld
0e595346be Make lexer log output easier to read 2016-12-09 13:33:37 -08:00
Max Brunsfeld
7f6ec0131d Remove duplication between parser_destroy and parser_set_language 2016-12-06 10:12:49 -08:00
Max Brunsfeld
c4fe8ded95 Remove state argument to Lexer advance method 2016-12-05 16:36:34 -08:00
Max Brunsfeld
c16b6b2059 Run external scanners during error recovery 2016-12-05 11:50:24 -08:00
Max Brunsfeld
cf0d8abea1 Destroy external scanner when destroying Parser 2016-12-04 14:18:30 -08:00
Max Brunsfeld
0f8e130687 Call external scanner functions when lexing 2016-12-02 22:03:48 -08:00
Max Brunsfeld
c966af0412 Start work on external tokens 2016-12-02 16:24:19 -08:00
Max Brunsfeld
5332fd3418 Fix build warnings 2016-11-19 20:47:43 -08:00
Max Brunsfeld
e7217f1bac Clean up some methods in parser.c 2016-11-14 17:25:55 -08:00
Max Brunsfeld
535879a2bd Represent byte, char and tree counts as 32 bit numbers 
The parser spends the majority of its time allocating and freeing trees and stack nodes.
Also, the memory footprint of the AST is a significant concern when using tree-sitter
with large files. This library is already unlikely to work very well with source files
larger than 4GB, so representing rows, columns, byte lengths and child indices as
unsigned 32 bit integers seems like the right choice.
 2016-11-14 12:19:13 -08:00
Max Brunsfeld
8edb8df530 Remove extraneous Language methods 2016-11-14 10:35:33 -08:00
Max Brunsfeld
1118a9142a Introduce Symbol::Index type alias 2016-11-14 10:25:26 -08:00
Max Brunsfeld
1fddb124b3 Remove stray LOG_STACK() call 2016-11-14 09:32:05 -08:00
Max Brunsfeld
fad7294ba4 Store shift states for non-terminals directly in the main parse table 2016-11-14 08:36:06 -08:00
Max Brunsfeld
8d9c261e3a Don't include reduce actions for nonterminal lookaheads 2016-11-10 11:33:37 -08:00
Max Brunsfeld
c9dcb29c6f Remove the TS prefix from some internal type/function names 2016-11-09 20:59:05 -08:00
Max Brunsfeld
89c01a7b22 🎨 tree_path.h 2016-11-09 20:25:03 -08:00
Max Brunsfeld
3f0e63058b Fix handling of failure to descend when getting changed ranges 2016-11-05 21:26:01 -07:00
Max Brunsfeld
ff14cda959 Abort if malloc fails 2016-11-05 21:23:23 -07:00