Remove the separator characters construct

Now, grammars can handle whitespace by making it another ubiquitous
token, like comments.

For now, this has the side effect of whitespace being included in the
tree that precedes it. This was already an issue for other ubiquitous
tokens though, so it needs to be fixed anyway.
This commit is contained in:
Max Brunsfeld 2014-09-01 20:19:32 -07:00
parent db295cebbc
commit 5cd07648fd
43 changed files with 12173 additions and 10028 deletions

View file

@ -314,8 +314,6 @@ class CCodeGenerator {
void code_for_lex_state(const LexState &lex_state) {
auto expected_inputs = lex_state.expected_inputs();
if (lex_state.is_token_start)
line("START_TOKEN();");
for (auto pair : lex_state.actions)
if (!pair.first.is_empty())
_if([&]() { condition_for_character_set(pair.first); },