traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
790 commits 16 branches 98 tags 37 MiB
Commit graph

56 commits

Author SHA1 Message Date
Max Brunsfeld
5e4bdcbaf8 Simplify handling of precedence & associativity in productions 2015-10-05 16:56:11 -07:00
Max Brunsfeld
ebc52f109d Merge branch 'flatten-rules-into-productions' 
This branch had diverged considerably, so merging it required changing a lot
of code.

Conflicts:
	project.gyp
	spec/compiler/build_tables/action_takes_precedence_spec.cc
	spec/compiler/build_tables/build_conflict_spec.cc
	spec/compiler/build_tables/build_parse_table_spec.cc
	spec/compiler/build_tables/first_symbols_spec.cc
	spec/compiler/build_tables/item_set_closure_spec.cc
	spec/compiler/build_tables/item_set_transitions_spec.cc
	spec/compiler/build_tables/rule_can_be_blank_spec.cc
	spec/compiler/helpers/containers.h
	spec/compiler/prepare_grammar/expand_repeats_spec.cc
	spec/compiler/prepare_grammar/extract_tokens_spec.cc
	src/compiler/build_tables/action_takes_precedence.h
	src/compiler/build_tables/build_parse_table.cc
	src/compiler/build_tables/first_symbols.cc
	src/compiler/build_tables/first_symbols.h
	src/compiler/build_tables/item_set_closure.cc
	src/compiler/build_tables/item_set_transitions.cc
	src/compiler/build_tables/parse_item.cc
	src/compiler/build_tables/parse_item.h
	src/compiler/build_tables/rule_can_be_blank.cc
	src/compiler/build_tables/rule_can_be_blank.h
	src/compiler/prepare_grammar/expand_repeats.cc
	src/compiler/prepare_grammar/extract_tokens.cc
	src/compiler/prepare_grammar/extract_tokens.h
	src/compiler/prepare_grammar/prepare_grammar.cc
	src/compiler/rules/built_in_symbols.cc
	src/compiler/rules/built_in_symbols.h
	src/compiler/syntax_grammar.cc
	src/compiler/syntax_grammar.h
 2015-10-02 23:46:39 -07:00
Max Brunsfeld
67241e3052 Don't use std::set in public compiler header 
Just use vectors
 2015-09-08 23:43:45 -07:00
Max Brunsfeld
557c8c7f28 Remove unnecessary test helpers 2015-09-06 17:07:04 -07:00
Max Brunsfeld
f9316933ad Refactor logic for marking '_'-prefixed rules as hidden 2015-09-06 16:53:13 -07:00
Max Brunsfeld
5982b77c97 In compiler, distinguish between anonymous tokens and hidden rules 2015-09-05 22:28:55 -07:00
Max Brunsfeld
93259435c8 Handle tokens that appear both anonymously and as named rules 2015-07-30 17:24:08 -07:00
Max Brunsfeld
a7729c42c9 Whitespace 2015-07-29 21:47:48 -07:00
Max Brunsfeld
766e3bab2c Use 2-space continuation indent consistently in specs 2015-07-27 18:18:58 -07:00
Max Brunsfeld
31b2db12d2 Remove custom LexicalGrammar and SyntaxGrammar constructors 2015-07-19 16:12:11 -07:00
Max Brunsfeld
c9a482bbf3 Add expected_conflicts field to grammar 2015-06-26 16:14:08 -05:00
Max Brunsfeld
fd97b8a237 Dedup auxiliary repeat rules from different source rules 2015-05-02 20:42:47 -07:00
Max Brunsfeld
e8db35af6b Avoid creating redundant auxiliary repeat rules 2015-04-16 17:42:22 -07:00
Max Brunsfeld
b1f8ba6202 Replace {left,right}_assoc w/ prec, with an associativity argument 2015-03-23 21:06:31 -07:00
Max Brunsfeld
a19b0e75ac 🔥 keyword and keypattern functions 
Just make strings have higher precedence than regexps.
 2015-03-22 16:00:26 -07:00
Max Brunsfeld
8ac4b9fc17 Store productions' end rule ids in the vector 2015-02-16 22:11:03 -08:00
Max Brunsfeld
52daffb3f3 Separate syntax rules into flat lists of symbols 
This way, every ParseItem can be associated with a particular production
for its non-terminal. That lets us keep track of which productions are
involved in shift/reduce conflicts.
 2015-02-16 22:11:03 -08:00
Max Brunsfeld
34d96909d1 Move {Syntax,Lexical}Grammar into separate files 2015-01-14 21:10:41 -08:00
Max Brunsfeld
cb5ecbd491 Handle string and regex rules w/ non-ascii chars 2014-09-28 18:21:22 -07:00
Max Brunsfeld
cd8a683229 Improve error messages for invalid ubiquitous tokens 2014-09-10 13:02:16 -07:00
Max Brunsfeld
e9dad529f5 Make descriptions more consistent in compiler specs 2014-09-09 13:01:18 -07:00
Max Brunsfeld
1ff7cedf40 Unify ubiquitous tokens and lexical separators in API 2014-09-07 22:16:45 -07:00
Max Brunsfeld
a46f9d950c Handle '\s' correctly in regexps 2014-09-07 16:05:43 -07:00
Max Brunsfeld
ed11ef557a Fix expansion of repeat rules into recursive rules 
Previously, the way repeat rules were expanded, the auxiliary
rule always needed to be reduced, even if the repeating content
was empty. This caused problems in parse states where some items
contained the repeat rule and some did not. To make those cases
work, the repeat rule had to explicitly be marked as optional.
With this change, that is no longer necessary.
 2014-09-07 09:39:14 -07:00
Max Brunsfeld
d3204d3526 Include '_' in '\w' regex character class 2014-09-05 18:41:12 -07:00
Max Brunsfeld
545e575508 Revert "Remove the separator characters construct" 
This reverts commit 5cd07648fd.

The separators construct is useful as an optimization. It turns out that
constructing a node for every chunk of whitespace in a document causes a
significant performance regression.

Conflicts:
	src/compiler/build_tables/build_lex_table.cc
	src/compiler/grammar.cc
	src/runtime/parser.c
 2014-09-02 08:03:51 -07:00
Max Brunsfeld
5cd07648fd Remove the separator characters construct 
Now, grammars can handle whitespace by making it another ubiquitous
token, like comments.

For now, this has the side effect of whitespace being included in the
tree that precedes it. This was already an issue for other ubiquitous
tokens though, so it needs to be fixed anyway.
 2014-09-01 20:19:43 -07:00
Max Brunsfeld
9338249075 Remove implicit CharacterRange constructors 
Also fix misc smaller lint errors
 2014-08-23 14:52:44 -07:00
Max Brunsfeld
0bb5663f0f Refactor - represent char sets in terms of inclusions and exclusions 2014-08-23 14:25:45 -07:00
Max Brunsfeld
b155994491 Fix indentation in specs 2014-08-07 08:11:21 -07:00
Max Brunsfeld
01571da30d Handle more escaped characters in regexps 2014-08-03 21:57:21 -07:00
Max Brunsfeld
83a1b9439e Fix handling of ubiquitous tokens used in grammar rules 2014-07-01 20:47:35 -07:00
Max Brunsfeld
a9dff20658 Make grammars' separator characters configurable 2014-06-26 07:31:08 -07:00
Max Brunsfeld
7df35f9b8d Make separate types for syntax and lexical grammars 
This way, the separator characters can be added as a field to
lexical grammars only
 2014-06-25 13:27:16 -07:00
Max Brunsfeld
81880e000e Tweak header include paths in tests 2014-06-23 18:50:03 -07:00
Max Brunsfeld
e105f5cebc Remove inheritance link btwn PreparedGrammar and Grammar 2014-06-10 10:34:37 -07:00
Max Brunsfeld
54a555168d Add accessor methods on Grammar 2014-06-09 21:05:25 -07:00
Max Brunsfeld
868a09b0b0 Remove infinite loop on certain lex errors 2014-06-01 23:23:24 -07:00
Max Brunsfeld
c7266f791e Don't use std::tuples in parse regex spec 
gcc doesn't let me use initializer list syntax for them
 2014-06-01 17:34:18 -07:00
Max Brunsfeld
e93e254518 In lexer, prefer tokens to skipped separator characters 
This was causing newlines in go and javascript to be parsed as
meaningless separator characters instead of statement terminators
 2014-05-30 13:29:54 -07:00
Max Brunsfeld
c30055ba18 Fix symbol names for extracted tokens 2014-05-20 08:30:58 -07:00
Max Brunsfeld
649f200831 Expand regex/string rules as part of grammar preparation 
This makes it possible to report errors in regex parsing
 2014-05-19 20:54:59 -07:00
Max Brunsfeld
5245bc01fe Backfill tests for token extraction in auxiliary rules 2014-05-19 19:05:54 -07:00
Max Brunsfeld
4700e33746 Introduce 'ubiquitous_tokens' concept, for parsing comments and such 2014-05-06 12:54:04 -07:00
Max Brunsfeld
3a50171249 Expose all grammar compilation errors 2014-05-01 23:28:40 -07:00
Max Brunsfeld
6d40dcf881 Add token helper for building token rules 
Now you can specify the structure of tokens using
all of the rule functions, not just `str` and `pattern`
 2014-05-01 12:43:29 -07:00
Max Brunsfeld
0d763d229d cpplint 2014-04-28 21:46:43 -07:00
Max Brunsfeld
25eda9d889 ISymbol -> Symbol 
Interned symbols are now the main type of symbol in use
 2014-04-28 20:43:27 -07:00
Max Brunsfeld
faf80aadac Symbol -> NamedSymbol 2014-04-28 20:15:49 -07:00
Max Brunsfeld
93df5579b4 Trim whitespace 2014-04-25 22:17:23 -07:00