traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1205 commits 16 branches 98 tags 37 MiB
Commit graph

51 commits

Author SHA1 Message Date
Max Brunsfeld
0e2bbbd7ee Compress parse table by allowing reductions w/ unexpected lookaheads 2016-07-04 12:20:23 -07:00
Max Brunsfeld
8c26d99353 Store error recovery actions in the normal parse table 2016-06-27 14:07:47 -07:00
Max Brunsfeld
38c144b4a3 Refine logic for deciding when tokens need to be re-lexed 
* While generating the lex table, note which tokens can match the
  same string. A token needs to be relexed when it has possible
  homonyms in the current state.
* Also note which tokens can match substrings of each other tokens.
  A token needs to be relexed when there are viable tokens that
  could match longer strings in the current state and the next
  token has been edited.
* Remove the logic for marking tokens as fragile on creation.
* Store the reusability/non-reusability of symbols off of individual
  actions and onto the entire entry for the state & symbol.
 2016-06-21 07:28:04 -07:00
Max Brunsfeld
22c550c9d6 Discard tokens after error detection to find the best repair 
* Use GLR stack-splitting to try all numbers of tokens to
  discard until a repair is found.
* Check the validity of repairs by looking at the child trees,
  rather than the statically-computed 'in-progress symbols' list
 2016-05-11 13:49:43 -07:00
Max Brunsfeld
31f6b2e24a Refactor construction of out-of-context states 2016-04-25 21:59:40 -07:00
Max Brunsfeld
ffcd8b5c49 Generate C code for the in-progress symbols in each parse state 2016-03-02 20:56:05 -08:00
Max Brunsfeld
00d953f507 Generate C code for out-of-context states 2016-03-02 20:56:05 -08:00
Max Brunsfeld
6401a065ae Use different types for advance and accept-token actions 
Unlike with parse actions, lexical actions of different types never appear
in the same places in the table
 2016-01-22 22:24:11 -07:00
Max Brunsfeld
386b124866 Ensure that there are no duplicate lex states 2015-12-20 15:46:13 -08:00
Max Brunsfeld
c495076adb Record in parse table which actions can hide splits 
Suppose a parse state S has multiple actions for a terminal lookahead symbol A.
Then during incremental parsing, while in state S, the parser should not
reuse a non-terminal lookahead B where FIRST(B) contains A, because reusing B
might prematurely discard one of the possible actions that a batch parser
would have attempted in state S, upon seeing A as a lookahead.
 2015-12-17 13:11:56 -08:00
Max Brunsfeld
75f31a79a3 Treat reduce actions with different production IDs as distinct 2015-12-10 13:00:26 -08:00
Max Brunsfeld
ad619d95f6 Add 'extra' field to symbol metadata 
This stores whether a symbol is only ever used as a ubiquitous token. This will
allow ubiquitous nodes to be reused more effectively: if they are always
ubiquitous, then they can be reused immediately, and otherwise, they must be
broken down in case they need to be used structurally.
 2015-12-02 15:10:24 -08:00
Max Brunsfeld
1983bcfb60 Fix conflation of finished items w/ different precedence 2015-10-18 12:51:32 -07:00
Max Brunsfeld
9959fe35b0 Allow associativity to be specified in rules w/o precedence 2015-10-13 11:25:28 -07:00
Max Brunsfeld
6d748a6714 Store parse actions' precedences as ranges, not sets 2015-10-05 16:05:19 -07:00
Max Brunsfeld
ebc52f109d Merge branch 'flatten-rules-into-productions' 
This branch had diverged considerably, so merging it required changing a lot
of code.

Conflicts:
	project.gyp
	spec/compiler/build_tables/action_takes_precedence_spec.cc
	spec/compiler/build_tables/build_conflict_spec.cc
	spec/compiler/build_tables/build_parse_table_spec.cc
	spec/compiler/build_tables/first_symbols_spec.cc
	spec/compiler/build_tables/item_set_closure_spec.cc
	spec/compiler/build_tables/item_set_transitions_spec.cc
	spec/compiler/build_tables/rule_can_be_blank_spec.cc
	spec/compiler/helpers/containers.h
	spec/compiler/prepare_grammar/expand_repeats_spec.cc
	spec/compiler/prepare_grammar/extract_tokens_spec.cc
	src/compiler/build_tables/action_takes_precedence.h
	src/compiler/build_tables/build_parse_table.cc
	src/compiler/build_tables/first_symbols.cc
	src/compiler/build_tables/first_symbols.h
	src/compiler/build_tables/item_set_closure.cc
	src/compiler/build_tables/item_set_transitions.cc
	src/compiler/build_tables/parse_item.cc
	src/compiler/build_tables/parse_item.h
	src/compiler/build_tables/rule_can_be_blank.cc
	src/compiler/build_tables/rule_can_be_blank.h
	src/compiler/prepare_grammar/expand_repeats.cc
	src/compiler/prepare_grammar/extract_tokens.cc
	src/compiler/prepare_grammar/extract_tokens.h
	src/compiler/prepare_grammar/prepare_grammar.cc
	src/compiler/rules/built_in_symbols.cc
	src/compiler/rules/built_in_symbols.h
	src/compiler/syntax_grammar.cc
	src/compiler/syntax_grammar.h
 2015-10-02 23:46:39 -07:00
Max Brunsfeld
e6f3239bef Move stream operator definitions to spec helpers 
This is one less thing for users to worry about when compiling and linking
the library itself
 2015-09-10 10:12:11 -07:00
Max Brunsfeld
bd77ab1ac9 Move public rule functions out of rule namespace 
This way, there's only one public namespace: tree_sitter
 2015-09-03 17:49:20 -07:00
Max Brunsfeld
f9b057f3a9 clang-format everything 2015-07-27 18:29:48 -07:00
Max Brunsfeld
aabcb10cfb Respect expected_conflicts field when building parse table 2015-06-28 16:22:31 -05:00
Max Brunsfeld
80ec303b10 Replace prec rule w/ left_assoc and right_assoc 
Consider shift/reduce conflicts to be compilation errors unless
they are resolved by a specified associativity.
 2015-03-16 23:12:34 -07:00
Max Brunsfeld
3458fa6e50 Fix non-deterministic order in conflict description 2015-03-07 11:02:21 -08:00
Max Brunsfeld
2d436cf141 Identify fragile reductions at compile time 2015-02-21 15:11:03 -08:00
Max Brunsfeld
98cc2f2264 Auto-format all source code with clang-format 2014-07-21 13:20:00 -07:00
Max Brunsfeld
f4287c07d0 Fix ParseStateId / size_t confusion in parse table 2014-07-07 13:21:30 -07:00
Max Brunsfeld
83a1b9439e Fix handling of ubiquitous tokens used in grammar rules 2014-07-01 20:47:35 -07:00
Max Brunsfeld
9686c57e90 Allow ubiquitous tokens to also be used in grammar rules 2014-06-26 08:52:42 -07:00
Max Brunsfeld
25eda9d889 ISymbol -> Symbol 
Interned symbols are now the main type of symbol in use
 2014-04-28 20:43:27 -07:00
Max Brunsfeld
68d44fd565 Intern symbols during grammar preparation 2014-04-22 23:38:26 -07:00
Max Brunsfeld
a437d39773 Add rule precedence construct 
Still need to add some way of expressing left and right
associativity
 2014-04-15 08:40:46 -07:00
Max Brunsfeld
3982b73ed6 Start work on recording parse action conflicts 2014-04-08 08:19:55 -07:00
Max Brunsfeld
3f770ff3c3 Remove unused consumed_symbols vector from parse items 2014-03-26 21:04:11 -07:00
Max Brunsfeld
aac0786449 Resolve token conflicts by tokens' order in grammar 2014-03-24 19:18:06 -07:00
Max Brunsfeld
9cb92a0a96 Remove code related to old error recovery function 2014-03-24 13:17:38 -07:00
Max Brunsfeld
3aaa08b948 Fix some egregiously long lines 2014-03-09 23:58:25 -07:00
Max Brunsfeld
39cb420df2 Remove uses of 'short' and 'long' 2014-03-09 23:00:14 -07:00
Max Brunsfeld
a4120f36d4 Remove all uses of 'using namespace' 2014-03-09 22:45:33 -07:00
Max Brunsfeld
31a58bc7e4 Make include guards pass cpplint 2014-03-09 22:05:24 -07:00
Max Brunsfeld
eb30429700 Make paths explicit in #includes 2014-03-09 21:43:14 -07:00
Max Brunsfeld
39aa0ccc91 Add script to trim whitespace 2014-03-09 19:49:35 -07:00
Max Brunsfeld
63718d5bad Add basic support for error recovery 2014-02-26 19:03:43 -08:00
Max Brunsfeld
e58a6d8ba7 Start work on error recovery 
- In runtime, make parse errors part of the parse tree
- Add error state to lexers in which they can accept any token
 2014-02-24 18:42:54 -08:00
Max Brunsfeld
2c56612650 Get makefile working 2014-02-18 09:07:00 -08:00
Max Brunsfeld
15c9e2d398 Make ordering of cases deterministic in generated parsers 2014-02-10 18:38:01 -08:00
Max Brunsfeld
7d297f2f9e Remove auxiliary rules from syntax tree 2014-01-28 22:09:37 -08:00
Max Brunsfeld
ca33c3942a In parse table, store symbols as Symbol objects, not strings 2014-01-27 13:40:10 -08:00
Max Brunsfeld
0877d01194 Add stream operator for parse states 2014-01-26 16:38:41 -08:00
Max Brunsfeld
5d9dc71da1 Remove default_actions from ParseTable 2014-01-25 23:40:51 -08:00
Max Brunsfeld
3ca2e126be Remove unnecessary public START and END constants 2014-01-25 21:34:46 -08:00
Max Brunsfeld
5eb5b61c14 Don’t pass rule names to code generator separately from parse tables 2014-01-25 20:45:00 -08:00