traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1221 commits 16 branches 98 tags 37 MiB
Commit graph

159 commits

Author SHA1 Message Date
Max Brunsfeld
38c144b4a3 Refine logic for deciding when tokens need to be re-lexed 
* While generating the lex table, note which tokens can match the
  same string. A token needs to be relexed when it has possible
  homonyms in the current state.
* Also note which tokens can match substrings of each other tokens.
  A token needs to be relexed when there are viable tokens that
  could match longer strings in the current state and the next
  token has been edited.
* Remove the logic for marking tokens as fragile on creation.
* Store the reusability/non-reusability of symbols off of individual
  actions and onto the entire entry for the state & symbol.
 2016-06-21 07:28:04 -07:00
Max Brunsfeld
a3679fbb1f Distinguish separators from main tokens via a property on transitions 
It was incorrect to store it as a property on the lexical states themselves
 2016-05-19 16:27:25 -07:00
Max Brunsfeld
59712ec492 Clean up lex table generation 2016-05-19 13:25:46 -07:00
Max Brunsfeld
22c550c9d6 Discard tokens after error detection to find the best repair 
* Use GLR stack-splitting to try all numbers of tokens to
  discard until a repair is found.
* Check the validity of repairs by looking at the child trees,
  rather than the statically-computed 'in-progress symbols' list
 2016-05-11 13:49:43 -07:00
Max Brunsfeld
5b74813a5c Refine logic for which tokens to use in error recovery 2016-04-27 14:09:19 -07:00
Max Brunsfeld
76d072545d Include out-of-context states starting with non-terminals 2016-03-02 20:58:39 -08:00
Max Brunsfeld
8c01b70ce7 Don't skip tokens that are not the start of any non-terminal 2016-03-02 20:56:05 -08:00
Max Brunsfeld
dee1f697c1 Compute the set of variables that can begin with each terminal symbol 2016-02-25 21:51:52 -08:00
Max Brunsfeld
6401a065ae Use different types for advance and accept-token actions 
Unlike with parse actions, lexical actions of different types never appear
in the same places in the table
 2016-01-22 22:24:11 -07:00
Max Brunsfeld
0f7dbea9a3 Unify test targets, use externally defined languages as fixtures 2016-01-15 11:19:24 -08:00
Max Brunsfeld
f065eb0480 Remove unused parameter to LexConflictManager 2015-12-17 15:45:47 -08:00
Max Brunsfeld
a8d2585330 Fix resolution of shift-extra vs reduce actions 2015-12-17 15:19:58 -08:00
Max Brunsfeld
351b4f4aaa Remove unused parameters to ParseConflictManager 2015-12-17 15:19:00 -08:00
Max Brunsfeld
d713054d61 Record which tokens are fragile when lexing 2015-12-10 21:05:54 -08:00
Max Brunsfeld
75f31a79a3 Treat reduce actions with different production IDs as distinct 2015-12-10 13:00:26 -08:00
Max Brunsfeld
d5ce268074 Fix handling of changing precedence within lexical rules. 
A precedence annotation wrapping a sequence of characters now only affects how
tightly those characters bind to *each other*, not how tightly they bind to the
preceding character.

This bug surfaced because a generated lexer was failing to recognize a '\n' character
as a token, instead treating it as ubiquitous whitespace. It made this error
because, even though anonymous ubiquitous tokens have the lowest precedence, the
character immediately *after* the '\n' was part of a normal token, which had
*normal* precedence (0). Advancing into that following token was incorrectly
prioritized above accepting the line-break token.
 2015-11-08 13:36:15 -08:00
Max Brunsfeld
d7cb48aae7 Fix handling of precedence for repeat rules 2015-11-01 21:00:44 -08:00
Max Brunsfeld
d6ee28abd0 Make precedence more useful within tokens 
Choose accept-token actions over advance actions if their rule has a higher precedence.
 2015-11-01 12:48:27 -08:00
Max Brunsfeld
998ae533da Make completion_status() a method on LexItem 2015-10-30 16:48:37 -07:00
Max Brunsfeld
c8be143f65 🔥 get_metadata function 2015-10-30 16:22:25 -07:00
Max Brunsfeld
73b3280fbb Include precedence calculation in LexItemSet::transitions 2015-10-30 16:07:29 -07:00
Max Brunsfeld
e9be0ff24e Make completion_status() a method on ParseItem 2015-10-30 14:07:33 -07:00
Max Brunsfeld
4850384b78 Include precedence calculation in ParseItemSet::transitions 2015-10-30 13:54:11 -07:00
Max Brunsfeld
58b5a10607 Fix ParseItemSet::transitions spec description 2015-10-29 12:19:44 -07:00
Max Brunsfeld
a8ead10d6f In lex error state, don't look for tokens that would match *any* line 2015-10-28 17:45:17 -07:00
Max Brunsfeld
1983bcfb60 Fix conflation of finished items w/ different precedence 2015-10-18 12:51:32 -07:00
Max Brunsfeld
8725e96a65 Fix item-set-closure bug w/ empty productions 2015-10-15 23:59:47 -07:00
Max Brunsfeld
4b817dc07c Fix linter errors 2015-10-12 19:22:05 -07:00
Max Brunsfeld
82726ad53b Define repeat rule in terms of repeat1 rule 2015-10-12 19:22:05 -07:00
Max Brunsfeld
db9966b57c Simplify lex item set transitions code 2015-10-11 22:51:37 -07:00
Max Brunsfeld
5455fb977f Use PrecedenceRange in build_lex_table 2015-10-05 18:02:59 -07:00
Max Brunsfeld
5e4bdcbaf8 Simplify handling of precedence & associativity in productions 2015-10-05 16:56:11 -07:00
Max Brunsfeld
6d748a6714 Store parse actions' precedences as ranges, not sets 2015-10-05 16:05:19 -07:00
Max Brunsfeld
ef2acf9496 Make ParseItemSet & LexItemSet classes 2015-10-05 15:13:43 -07:00
Max Brunsfeld
f01972c64e Reorganize ParseItemSet and LexItemSet 2015-10-05 15:09:06 -07:00
Max Brunsfeld
39a0934088 Remove now-unused symbol rule-transition functions 2015-10-04 22:20:34 -07:00
Max Brunsfeld
c4ef228397 Share common lookahead sets between parse item sets 2015-10-04 21:33:54 -07:00
Max Brunsfeld
a0bf3d0bd8 Compute closures of item sets lazily 2015-10-04 00:21:29 -07:00
Max Brunsfeld
ebc52f109d Merge branch 'flatten-rules-into-productions' 
This branch had diverged considerably, so merging it required changing a lot
of code.

Conflicts:
	project.gyp
	spec/compiler/build_tables/action_takes_precedence_spec.cc
	spec/compiler/build_tables/build_conflict_spec.cc
	spec/compiler/build_tables/build_parse_table_spec.cc
	spec/compiler/build_tables/first_symbols_spec.cc
	spec/compiler/build_tables/item_set_closure_spec.cc
	spec/compiler/build_tables/item_set_transitions_spec.cc
	spec/compiler/build_tables/rule_can_be_blank_spec.cc
	spec/compiler/helpers/containers.h
	spec/compiler/prepare_grammar/expand_repeats_spec.cc
	spec/compiler/prepare_grammar/extract_tokens_spec.cc
	src/compiler/build_tables/action_takes_precedence.h
	src/compiler/build_tables/build_parse_table.cc
	src/compiler/build_tables/first_symbols.cc
	src/compiler/build_tables/first_symbols.h
	src/compiler/build_tables/item_set_closure.cc
	src/compiler/build_tables/item_set_transitions.cc
	src/compiler/build_tables/parse_item.cc
	src/compiler/build_tables/parse_item.h
	src/compiler/build_tables/rule_can_be_blank.cc
	src/compiler/build_tables/rule_can_be_blank.h
	src/compiler/prepare_grammar/expand_repeats.cc
	src/compiler/prepare_grammar/extract_tokens.cc
	src/compiler/prepare_grammar/extract_tokens.h
	src/compiler/prepare_grammar/prepare_grammar.cc
	src/compiler/rules/built_in_symbols.cc
	src/compiler/rules/built_in_symbols.h
	src/compiler/syntax_grammar.cc
	src/compiler/syntax_grammar.h
 2015-10-02 23:46:39 -07:00
Max Brunsfeld
32536665f9 Remove weak build_parse_table spec 
This is best covered by integration tests
 2015-09-22 20:03:00 -07:00
Max Brunsfeld
557c8c7f28 Remove unnecessary test helpers 2015-09-06 17:07:04 -07:00
Max Brunsfeld
5982b77c97 In compiler, distinguish between anonymous tokens and hidden rules 2015-09-05 22:28:55 -07:00
Max Brunsfeld
bd77ab1ac9 Move public rule functions out of rule namespace 
This way, there's only one public namespace: tree_sitter
 2015-09-03 17:49:20 -07:00
Max Brunsfeld
97bb7a26cf Fix precedence calculations when building parse table 
* Recurse into choice rules
* Compute reduction precedence differently than shift precedence
 2015-09-02 13:05:54 -07:00
Max Brunsfeld
766e3bab2c Use 2-space continuation indent consistently in specs 2015-07-27 18:18:58 -07:00
Max Brunsfeld
31b2db12d2 Remove custom LexicalGrammar and SyntaxGrammar constructors 2015-07-19 16:12:11 -07:00
Max Brunsfeld
aabcb10cfb Respect expected_conflicts field when building parse table 2015-06-28 16:22:31 -05:00
Max Brunsfeld
c9a482bbf3 Add expected_conflicts field to grammar 2015-06-26 16:14:08 -05:00
Max Brunsfeld
a54d293317 Include 'goal' non-terminal names in conflict description 2015-04-16 17:43:30 -07:00
Max Brunsfeld
9ef52ce2fb Include precedences in unresolved conflict messages 2015-03-23 22:56:17 -07:00