4 commits

Author	SHA1	Message	Date
Max Brunsfeld	6d40dcf881	Add token helper for building token rules ... Now you can specify the structure of tokens using all of the rule functions, not just `str` and `pattern`	2014-05-01 12:43:29 -07:00
Max Brunsfeld	1da9f1fdfd	Store rule metadata as a map, not a single number ... Need to store more than just boolean values	2014-04-07 08:50:00 -07:00
Max Brunsfeld	5320cad065	Trim trailing whitespace	2014-04-04 13:10:55 -07:00
Max Brunsfeld	1cc7e32e2d	Fix handling of tokens consisting of separator characters ... The parser is no longer hard-coded to skip whitespace. Tokens such as newlines, whose characters overlap with the separator characters, can now be correctly recognized.	2014-04-03 19:10:09 -07:00