95 commits

Author	SHA1	Message	Date
Max Brunsfeld	e9dad529f5	Make descriptions more consistent in compiler specs	2014-09-09 13:01:18 -07:00
Max Brunsfeld	1ff7cedf40	Unify ubiquitous tokens and lexical separators in API	2014-09-07 22:16:45 -07:00
Max Brunsfeld	545e575508	Revert "Remove the separator characters construct" ... This reverts commit 5cd07648fd. The separators construct is useful as an optimization. It turns out that constructing a node for every chunk of whitespace in a document causes a significant performance regression. Conflicts: src/compiler/build_tables/build_lex_table.cc src/compiler/grammar.cc src/runtime/parser.c	2014-09-02 08:03:51 -07:00
Max Brunsfeld	5cd07648fd	Remove the separator characters construct ... Now, grammars can handle whitespace by making it another ubiquitous token, like comments. For now, this has the side effect of whitespace being included in the tree that precedes it. This was already an issue for other ubiquitous tokens though, so it needs to be fixed anyway.	2014-09-01 20:19:43 -07:00
Max Brunsfeld	226ffd6b5b	Fix initializer list deduction warnings in specs	2014-08-27 22:23:45 -07:00
Max Brunsfeld	9338249075	Remove implicit CharacterRange constructors ... Also fix misc smaller lint errors	2014-08-23 14:52:44 -07:00
Max Brunsfeld	0bb5663f0f	Refactor - represent char sets in terms of inclusions and exclusions	2014-08-23 14:25:45 -07:00
Max Brunsfeld	b155994491	Fix indentation in specs	2014-08-07 08:11:21 -07:00
Max Brunsfeld	6951acb13b	Fix error when grammar contains to error productions	2014-07-13 21:26:21 -07:00
Max Brunsfeld	7dfaba2954	Fix conflict manager spec	2014-07-13 18:10:32 -07:00
Max Brunsfeld	b217cd38fb	Handle built-in symbols correctly in conflict manager	2014-07-13 17:59:57 -07:00
Max Brunsfeld	77df7fe511	In lexer, always prefer the longest match ... Only use rules' precedence to decide between two tokens that match the same string	2014-07-03 08:57:35 -07:00
Max Brunsfeld	83a1b9439e	Fix handling of ubiquitous tokens used in grammar rules	2014-07-01 20:47:35 -07:00
Max Brunsfeld	3be648593e	merge_{sym,char}_transitions -> merge_{sym,char}_transition	2014-06-28 17:02:48 -07:00
Max Brunsfeld	9bad5dff3e	Avoid unnecessary std::map construction when merging transition sets	2014-06-26 13:42:42 -07:00
Max Brunsfeld	9686c57e90	Allow ubiquitous tokens to also be used in grammar rules	2014-06-26 08:52:42 -07:00
Max Brunsfeld	7df35f9b8d	Make separate types for syntax and lexical grammars ... This way, the separator characters can be added as a field to lexical grammars only	2014-06-25 13:27:16 -07:00
Max Brunsfeld	81880e000e	Tweak header include paths in tests	2014-06-23 18:50:03 -07:00
Max Brunsfeld	2c382b7363	Trim trailing whitespace	2014-06-16 21:33:35 -07:00
Max Brunsfeld	1daaf4485f	Refactor item set transition functions	2014-06-16 13:37:34 -07:00
Max Brunsfeld	39c1ab2d50	Refactor item_set_closure ... Inline unnecessary function	2014-06-16 13:20:39 -07:00
Max Brunsfeld	7a2c2c1c90	Store ParseItemSets as maps, w/ core items as keys ... ParseItem no longer has a lookahead_sym field; it now represents the 'core' of a parse item. The lookahead context is stored separately, as a set per core item. This makes iterating, copying and merging item sets more efficient, because before, the core items were repeated for each different lookahead symbol. Also, the memoization in sym_transitions(ParseItemSet) has been removed. Maybe I'll add it back later.	2014-06-16 08:35:20 -07:00
Max Brunsfeld	174f306e2a	Fix precedence of comments vs '/' operator	2014-06-11 12:27:58 -07:00
Max Brunsfeld	a42f498c59	Optimize merge_sym_transitions and merge_char_transitions	2014-06-10 14:01:32 -07:00
Max Brunsfeld	e105f5cebc	Remove inheritance link btwn PreparedGrammar and Grammar	2014-06-10 10:34:37 -07:00
Max Brunsfeld	e93e254518	In lexer, prefer tokens to skipped separator characters ... This was causing newlines in go and javascript to be parsed as meaningless separator characters instead of statement terminators	2014-05-30 13:29:54 -07:00
Max Brunsfeld	649f200831	Expand regex/string rules as part of grammar preparation ... This makes it possible to report errors in regex parsing	2014-05-19 20:54:59 -07:00
Max Brunsfeld	4700e33746	Introduce 'ubiquitous_tokens' concept, for parsing comments and such	2014-05-06 12:54:04 -07:00
Max Brunsfeld	b010e1667e	Fix parse action equality method	2014-05-06 12:51:38 -07:00
Max Brunsfeld	d91bc718a0	Add basic test for parse table builder	2014-05-04 23:28:53 -07:00
Max Brunsfeld	1d314d71c2	Separate functions for building parse and lex tables ... Now, instead of adding states to the lex table as they are needed by the parse states, we iterate over the parse states after the fact and set up their corresponding lex states. This has the nice side effect that the lex states are in a more readable order.	2014-05-04 22:07:52 -07:00
Max Brunsfeld	3a50171249	Expose all grammar compilation errors	2014-05-01 23:28:40 -07:00
Max Brunsfeld	93620b3ed1	Add keyword helper for making higher-priority string tokens	2014-05-01 13:25:20 -07:00
Max Brunsfeld	6d40dcf881	Add token helper for building token rules ... Now you can specify the structure of tokens using all of the rule functions, not just `str` and `pattern`	2014-05-01 12:43:29 -07:00
Max Brunsfeld	d685edf015	Backfill test for item set transitions	2014-04-29 08:18:41 -07:00
Max Brunsfeld	0d763d229d	cpplint	2014-04-28 21:46:43 -07:00
Max Brunsfeld	25eda9d889	ISymbol -> Symbol ... Interned symbols are now the main type of symbol in use	2014-04-28 20:43:27 -07:00
Max Brunsfeld	93df5579b4	Trim whitespace	2014-04-25 22:17:23 -07:00
Max Brunsfeld	5313bb5257	Simplify item set closure function	2014-04-23 13:35:21 -07:00
Max Brunsfeld	e8f2b788d4	Reduce allocations when computing rule transitions	2014-04-23 09:01:57 -07:00
Max Brunsfeld	68d44fd565	Intern symbols during grammar preparation	2014-04-22 23:38:26 -07:00
Max Brunsfeld	33d781f492	Refactor bookkeeping of token starts in lexical rules ... - Move lex items and parse items into their own files	2014-04-17 13:33:34 -07:00
Max Brunsfeld	a437d39773	Add rule precedence construct ... Still need to add some way of expressing left and right associativity	2014-04-15 08:40:46 -07:00
Max Brunsfeld	67243c7e2f	cpplint	2014-04-14 08:38:44 -07:00
Max Brunsfeld	e1e0cc6278	Make sure conflicts returned by compile are unique	2014-04-10 08:38:14 -07:00
Max Brunsfeld	f71d7bae16	Record parse conflicts when compiling grammars ... Need to remove duplicate conflicts	2014-04-08 18:47:42 -07:00
Max Brunsfeld	3982b73ed6	Start work on recording parse action conflicts	2014-04-08 08:19:55 -07:00
Max Brunsfeld	1da9f1fdfd	Store rule metadata as a map, not a single number ... Need to store more than just boolean values	2014-04-07 08:50:00 -07:00
Max Brunsfeld	5320cad065	Trim trailing whitespace	2014-04-04 13:10:55 -07:00
Max Brunsfeld	32e94081fa	Backfill tests	2014-04-04 08:07:46 -07:00