traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
960 commits 16 branches 98 tags 37 MiB
Commit graph

89 commits

Author SHA1 Message Date
Max Brunsfeld
66144dc28e Treat tokens that are sometimes extra as fragile 2015-12-16 20:04:45 -08:00
Max Brunsfeld
d713054d61 Record which tokens are fragile when lexing 2015-12-10 21:05:54 -08:00
Max Brunsfeld
75f31a79a3 Treat reduce actions with different production IDs as distinct 2015-12-10 13:00:26 -08:00
Max Brunsfeld
ad619d95f6 Add 'extra' field to symbol metadata 
This stores whether a symbol is only ever used as a ubiquitous token. This will
allow ubiquitous nodes to be reused more effectively: if they are always
ubiquitous, then they can be reused immediately, and otherwise, they must be
broken down in case they need to be used structurally.
 2015-12-02 15:10:24 -08:00
Max Brunsfeld
f08554e958 Replace NodeType enum with SymbolMetadata bitfield 
This will allow storing other metadata about symbols, like if they
only appear as ubiquitous tokens
 2015-12-02 15:10:24 -08:00
Max Brunsfeld
e11515fb74 Escape backslashes and quotes in symbol name strings 2015-11-09 09:33:24 -08:00
Max Brunsfeld
dba0726eef clang format 2015-10-28 12:10:58 -07:00
Max Brunsfeld
e543661498 Fix typo: LB -> LF 2015-10-26 16:55:59 -07:00
Max Brunsfeld
bad1bff3bb Sanitize line breaks in symbol names 2015-10-26 13:36:31 -07:00
Max Brunsfeld
1983bcfb60 Fix conflation of finished items w/ different precedence 2015-10-18 12:51:32 -07:00
Max Brunsfeld
ebc52f109d Merge branch 'flatten-rules-into-productions' 
This branch had diverged considerably, so merging it required changing a lot
of code.

Conflicts:
	project.gyp
	spec/compiler/build_tables/action_takes_precedence_spec.cc
	spec/compiler/build_tables/build_conflict_spec.cc
	spec/compiler/build_tables/build_parse_table_spec.cc
	spec/compiler/build_tables/first_symbols_spec.cc
	spec/compiler/build_tables/item_set_closure_spec.cc
	spec/compiler/build_tables/item_set_transitions_spec.cc
	spec/compiler/build_tables/rule_can_be_blank_spec.cc
	spec/compiler/helpers/containers.h
	spec/compiler/prepare_grammar/expand_repeats_spec.cc
	spec/compiler/prepare_grammar/extract_tokens_spec.cc
	src/compiler/build_tables/action_takes_precedence.h
	src/compiler/build_tables/build_parse_table.cc
	src/compiler/build_tables/first_symbols.cc
	src/compiler/build_tables/first_symbols.h
	src/compiler/build_tables/item_set_closure.cc
	src/compiler/build_tables/item_set_transitions.cc
	src/compiler/build_tables/parse_item.cc
	src/compiler/build_tables/parse_item.h
	src/compiler/build_tables/rule_can_be_blank.cc
	src/compiler/build_tables/rule_can_be_blank.h
	src/compiler/prepare_grammar/expand_repeats.cc
	src/compiler/prepare_grammar/extract_tokens.cc
	src/compiler/prepare_grammar/extract_tokens.h
	src/compiler/prepare_grammar/prepare_grammar.cc
	src/compiler/rules/built_in_symbols.cc
	src/compiler/rules/built_in_symbols.h
	src/compiler/syntax_grammar.cc
	src/compiler/syntax_grammar.h
 2015-10-02 23:46:39 -07:00
Max Brunsfeld
7ee5eaa16a Rename node accessor methods 
Instead of child() vs concrete_child(), next_sibling() vs next_concrete_sibling(), etc,
the default is switched: child() refers to the concrete syntax tree, and named_child()
refers to the AST. Because the AST is abstract through exclusion of some nodes, the
names are clearer if the qualifier goes on the AST operations
 2015-09-08 23:16:24 -07:00
Max Brunsfeld
f9316933ad Refactor logic for marking '_'-prefixed rules as hidden 2015-09-06 16:53:13 -07:00
Max Brunsfeld
9591c88f39 In runtime, distinguish between anonymous and hidden nodes 2015-09-06 00:12:37 -07:00
Max Brunsfeld
5982b77c97 In compiler, distinguish between anonymous tokens and hidden rules 2015-09-05 22:28:55 -07:00
Max Brunsfeld
21258e6a9e Remove 'document' wrapper node 2015-08-22 10:48:34 -07:00
Max Brunsfeld
c18351772a Auto-format: no single-line functions 2015-07-31 16:32:24 -07:00
Max Brunsfeld
f9b057f3a9 clang-format everything 2015-07-27 18:29:48 -07:00
Max Brunsfeld
0b1d70db34 Always resolve ambiguities immediately 
No more ambiguity nodes.
Also, when merging parse stacks, merge their successors if needed.
 2015-07-15 13:15:11 -07:00
Max Brunsfeld
f26ddf5187 Fix symbol name for ambiguity nodes 2015-07-08 17:31:21 -07:00
Max Brunsfeld
aabcb10cfb Respect expected_conflicts field when building parse table 2015-06-28 16:22:31 -05:00
Max Brunsfeld
381f89f8ba Create ambiguity nodes when joining stack heads 2015-06-18 17:03:16 -07:00
Max Brunsfeld
755894b44d Allow multiple parse actions in parse table 2015-06-18 17:03:16 -07:00
Max Brunsfeld
44774119bf Use all caps for built-in symbol names 2015-06-15 15:26:05 -07:00
Max Brunsfeld
2d436cf141 Identify fragile reductions at compile time 2015-02-21 15:11:03 -08:00
Max Brunsfeld
34d96909d1 Move {Syntax,Lexical}Grammar into separate files 2015-01-14 21:10:41 -08:00
Max Brunsfeld
5dc08ccce9 Include names of in-progress rules in shift/reduce conflicts 2014-11-05 18:39:50 -08:00
Max Brunsfeld
fc83322832 Handle more special characters in c symbol names 2014-10-31 18:37:56 -07:00
Max Brunsfeld
4fc960e4ec Give extracted string/regex tokens more descriptive names 2014-10-31 08:56:58 -07:00
Max Brunsfeld
0daada6921 Tidy up c-code generator 2014-10-12 12:33:45 -07:00
Max Brunsfeld
5b624d37f6 Use consistent iteration in c code generator 2014-10-12 11:57:52 -07:00
Max Brunsfeld
2e7ffb4d14 Tweak auto-format settings 
Prefer lines that exceed 80 characters by a small margin to
line breaks in argument lists
 2014-09-09 13:15:40 -07:00
Max Brunsfeld
1ff7cedf40 Unify ubiquitous tokens and lexical separators in API 2014-09-07 22:16:45 -07:00
Max Brunsfeld
c0a3f8d39c Remove some macros from public parser header 2014-09-05 23:47:38 -07:00
Max Brunsfeld
545e575508 Revert "Remove the separator characters construct" 
This reverts commit 5cd07648fd.

The separators construct is useful as an optimization. It turns out that
constructing a node for every chunk of whitespace in a document causes a
significant performance regression.

Conflicts:
	src/compiler/build_tables/build_lex_table.cc
	src/compiler/grammar.cc
	src/runtime/parser.c
 2014-09-02 08:03:51 -07:00
Max Brunsfeld
5cd07648fd Remove the separator characters construct 
Now, grammars can handle whitespace by making it another ubiquitous
token, like comments.

For now, this has the side effect of whitespace being included in the
tree that precedes it. This was already an issue for other ubiquitous
tokens though, so it needs to be fixed anyway.
 2014-09-01 20:19:43 -07:00
Max Brunsfeld
346cf4fe5d Remove LEX_PANIC macro 2014-08-26 13:12:12 -07:00
Max Brunsfeld
0bb5663f0f Refactor - represent char sets in terms of inclusions and exclusions 2014-08-23 14:25:45 -07:00
Max Brunsfeld
fcdcdca303 Whitespace 2014-08-09 01:07:14 -07:00
Max Brunsfeld
1e79ed794b Allow multiple top-level nodes 
Now, the root node of a document is always a document node.
It will often have only one child node which corresponds to the grammar's
start symbol, but not always. Currently, it may have more than one child
if there are ubiquitous tokens such as comments at the beginning of the
document. In the future, it will also be possible be possible to have multiple
for the document to have multiple children if the document is partially parsed.
 2014-08-09 00:00:20 -07:00
Max Brunsfeld
8da9219c3a Remove redundant functions for Documents 
There's no need for a `string` function since one already
exists for Nodes.

Now the root node is always stored on the document. This
means callers of `ts_document_root_node` don't need to release
its return value.
 2014-08-08 12:58:51 -07:00
Max Brunsfeld
9366f11dcb In generated C, only format printable characters as char literals 2014-08-07 08:12:15 -07:00
Max Brunsfeld
eecbcccee0 Remove generated parsers' dependency on the runtime library 
Generated parsers no longer export a parser constructor function.
They now export an opaque Language object which can be set on
Documents directly. This way, the logic for constructing parsers
lives entirely in the runtime. The Languages are just structs which
have no load-time dependency on the runtime
 2014-07-30 23:40:02 -07:00
Max Brunsfeld
98cc2f2264 Auto-format all source code with clang-format 2014-07-21 13:20:00 -07:00
Max Brunsfeld
18ae326459 Fix lint errors 2014-07-02 09:01:38 -07:00
Max Brunsfeld
83a1b9439e Fix handling of ubiquitous tokens used in grammar rules 2014-07-01 20:47:35 -07:00
Max Brunsfeld
9686c57e90 Allow ubiquitous tokens to also be used in grammar rules 2014-06-26 08:52:42 -07:00
Max Brunsfeld
7df35f9b8d Make separate types for syntax and lexical grammars 
This way, the separator characters can be added as a field to
lexical grammars only
 2014-06-25 13:27:16 -07:00
Max Brunsfeld
240de1ca61 Reorder methods in c code generator 2014-06-18 08:21:43 -07:00
Max Brunsfeld
c312f985c8 Refactor c code generator 
It's been rewritten in a less functional style. String copies were actually
taking significant time for large parsers.
 2014-06-16 21:29:04 -07:00