Update todo

todo.md

@@ -1,19 +1,34 @@
 TODO
 ====
 
-## batch parsing
-- more example grammars:
-  - go
-  - javascript
-  - ruby
-- add comments to generated C code describing the generated tokens (regexp pattern)
-- fix any memory leaks
+# flexibility of lexer
+- allow for tokens that contain characters in the separator character set
+  (e.g. newlines in golang and javascript)
+- make separator character set configurable, not hard-coded to whitespace
 - add special lexical behavior for indentation-aware languages
+- add helpers for constructing rules involving infix, prefix, and postfix operators
 
-## node.js wrapper
+# validation / robustness of compiler
+make the main `compile` function return errors when grammars are invalid
+- check validity of regex patterns
+- check that all rules that are referenced are also defined
+- check that all rules that are defined are also referenced (except the start rule)
+- don't require user to use the `err` helper
+  (right now, generated code doesn't compile if there are no error productions)
+
+# compiler optimization
+- implement Pager's [Practical general method](http://link.springer.com/article/10.1007%2FBF00290336)
+  for reducing the size of the LR(1) parse table
+- speed up grammar compilation
+
+# incremental parsing
+- implement the remainder of the incremental parsing algorithm: reuse parse
+  tree after the edit
+
+# node.js wrapper
+- create easy way to publish grammars as npm packages
 - add simple selector engine for trees
 
-## incremental parsing
-
-## chores
-- backfill unit-level tests for follow_sets
+# chores
+- make ordering of parse and lex tables deterministic, so that generated
+  code doesn't keep changing on every commit