traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1678 commits 16 branches 98 tags 37 MiB
Commit graph

1158 commits

Author SHA1 Message Date
Max Brunsfeld
99d048e016 Simplify error recovery; eliminate recovery states 
The previous approach to error recovery relied on special error-recovery
states in the parse table. For each token T, there was an error recovery
state in which the parser looked for *any* token that could follow T.
Unfortunately, sometimes the set of tokens that could follow T contained
conflicts. For example, in JS, the token '}' can be followed by the
open-ended 'template_chars' token, but also by ordinary tokens like
'identifier'. So with the old algorithm, when recovering from an
unexpected '}' token, the lexer had no way to distinguish identifiers
from template_chars.

This commit drops the error recovery states. Instead, when we encounter
an unexpected token T, we recover from the error by finding a previous
state S in the stack in which T would be valid, popping all of the nodes
after S, and wrapping them in an error.

This way, the lexer is always invoked in a normal parse state, in which
it is looking for a non-conflicting set of tokens. Eliminating the error
recovery states also shrinks the lex state machine significantly.

Signed-off-by: Rick Winfrey <rewinfrey@github.com>
 2017-09-11 15:22:52 -07:00
Max Brunsfeld
4c9c05806a Merge compatible starting token states before constructing lex table 2017-09-05 13:21:53 -07:00
Max Brunsfeld
9d668c5004 Move incompatible token map into LexTableBuilder 2017-08-31 15:46:37 -07:00
Max Brunsfeld
f8649824fa Remove unused function 2017-08-31 15:30:44 -07:00
Max Brunsfeld
4a0587061e Consolidate logic for deciding on a lookahead node 2017-08-31 12:19:37 -07:00
Max Brunsfeld
41074cbf2d 🎨 2017-08-30 16:48:15 -07:00
Max Brunsfeld
fdc6ee445b Remove parser__push helper function 2017-08-30 16:41:07 -07:00
Max Brunsfeld
1b1276bdbf Simplify parser__condense_stack function 2017-08-30 16:36:02 -07:00
Max Brunsfeld
96a630e5df Clean up check for leaf node reusability 2017-08-30 16:19:51 -07:00
Max Brunsfeld
8bdab7335e Remove unnecessary reusability check after breaking down lookahead 2017-08-30 16:19:11 -07:00
Max Brunsfeld
bef536a7d0 Discard fragile reusable nodes earlier 2017-08-30 16:17:10 -07:00
Max Brunsfeld
5cbd50c7d7 Remember how far ahead the lexer looked on failed calls 
This needs to be included in the 'bytes_scanned' property of the token
that is ultimately produced.
 2017-08-29 15:04:22 -07:00
Max Brunsfeld
f3977ec213 Always call deserialize on external scanner before scanning 
Remembering the last token that the external scanner produced is
not worth the complexity.
 2017-08-29 14:41:55 -07:00
Max Brunsfeld
c285fbef38 Clear LexTableBuilder's state after detecting conflicts 2017-08-25 17:11:39 -07:00
Max Brunsfeld
4d63e26e9e Clean up logic for falling back to error mode after lexing fails 2017-08-25 16:57:09 -07:00
Max Brunsfeld
86d5737fc2 Escape quotes when printing symbols to dot graphs 2017-08-25 16:26:40 -07:00
Max Brunsfeld
573b5f3671 Pass LexTableBuilder to ParseTableBuilder 2017-08-25 15:57:50 -07:00
Max Brunsfeld
eace426129 Suppress unknown pragma warnings in MSVC 2017-08-09 10:14:05 -07:00
Max Brunsfeld
9d649f3382 Remove depth-based error-recovery pruning criteria 
This code was causing ambiguities to get resolved differently depending on
whether there was an unrelated error on the stack or not.
 2017-08-09 09:53:41 -07:00
Max Brunsfeld
964dd16812 Avoid unicode escape sequences when generating conflict messages 2017-08-09 09:32:58 -07:00
Max Brunsfeld
5f40adb70c Recur to sub-rules in a deterministic order in expand_repeats 2017-08-08 17:20:04 -07:00
Max Brunsfeld
e6b43700b9 Get generated parsers compiling and loading properly on windows 2017-08-08 16:47:51 -07:00
Max Brunsfeld
9d616b3bf8 Replace size_t -> LexStateId in LexTableBuilder::remove_duplicate_states 2017-08-08 12:55:35 -07:00
Max Brunsfeld
3d351eac09 Fix some C code that MSVC doesn't like 2017-08-08 10:47:59 -07:00
Max Brunsfeld
12623deb19 Avoid struct literal syntax in point functions 2017-08-08 10:42:21 -07:00
Max Brunsfeld
947c161c2f Use a constructor rather than aggregate initialization for Production 2017-08-08 10:41:54 -07:00
Max Brunsfeld
a2670a39dc Avoid empty initializer list in global constant 2017-08-07 14:46:23 -07:00
Max Brunsfeld
e932d09908 Avoid aggregate initialization syntax in places where C++11 doesn't allow it 2017-08-07 13:07:54 -07:00
Max Brunsfeld
bf31c19d03 Avoid initializing production vectors via initializer lists 2017-08-07 12:45:37 -07:00
Max Brunsfeld
89e250cd63 Avoid the name 'min' for a helper function 
Windows automatically defines min or something.
 2017-08-07 12:44:33 -07:00
Max Brunsfeld
f0e63adc84 Use __forceinline keyword instead of always_inline attribute on windows 2017-08-07 12:44:33 -07:00
Max Brunsfeld
b98669c7e6 Replace general array_reverse with ts_tree_array_reverse 2017-08-07 12:44:33 -07:00
Max Brunsfeld
f7bf460c87 Fix scan-build errors in get_changed_ranges 2017-08-07 10:35:06 -07:00
Max Brunsfeld
ed11ddbd38 Explicitly compare regions between subtrees in get_changed_ranges 2017-08-06 17:16:29 -07:00
Max Brunsfeld
f5aa633011 Reorder iterator methods in get_changed_ranges.c 2017-08-05 20:33:38 -07:00
Max Brunsfeld
94dc703bfc Require that grammars' start rules be visible 2017-08-04 17:07:37 -07:00
Max Brunsfeld
1dca3a0b58 Simplify parse version reordering 2017-08-04 14:51:14 -07:00
Max Brunsfeld
85be0e7e75 Ensure that no backtracking occurs during tree-comparison 2017-08-04 14:50:50 -07:00
Max Brunsfeld
9260d8163c Refactor and fix bugs in tree comparison algorithm 2017-08-04 14:03:41 -07:00
Max Brunsfeld
46ec0804f8 Move tree comparison function to a separate translation unit 2017-08-04 11:03:14 -07:00
Max Brunsfeld
255f7af24b Name ParseTableBuilder fields more consistently 2017-08-04 09:47:24 -07:00
Max Brunsfeld
84e4114f79 Allow conflicts involving repeat rules to be whitelisted, via their parent rule 2017-08-03 15:18:29 -07:00
Max Brunsfeld
119c67dd78 Fix conflict reporting for shift/reduce conflicts w/ multiple reductions 
We were failing to rule out shift actions with lower precedence.

Signed-off-by: Philip Turnbull <philipturnbull@github.com>
 2017-08-02 15:13:30 -07:00
Max Brunsfeld
09f4796f6b Get tests passing w/ new alias API 2017-08-01 14:35:34 -07:00
Max Brunsfeld
cb5fe80348 Rename RENAME rule to ALIAS, allow it to create anonymous nodes 2017-07-31 16:41:11 -07:00
Max Brunsfeld
b5f421cafb Fix name collision that gcc didn't tolerate 2017-07-21 16:28:39 -07:00
Max Brunsfeld
2437287e3e Remove rename symbol when reusing a previously-renamed tree 2017-07-21 16:24:33 -07:00
Max Brunsfeld
cf445da253 Allow renaming hidden rules 2017-07-21 15:58:01 -07:00
Max Brunsfeld
1df41a9107 Avoid anonymous struct to silence gcc's override-init warning (again) 2017-07-21 10:17:54 -07:00
Max Brunsfeld
cbdfd89675 Mark reductions as fragile based on their final properties 
We previously maintained a set of individual productions that were
involved in conflicts, but that was subtly incorrect because
we don't compare productions themselves when comparing parse items;
we only compare the parse items properties that could affect the
final reduce actions.
 2017-07-21 09:54:24 -07:00