Max Brunsfeld
e93e254518
In lexer, prefer tokens to skipped separator characters
...
This was causing newlines in go and javascript to be parsed as
meaningless separator characters instead of statement terminators
2014-05-30 13:29:54 -07:00
Max Brunsfeld
3e0debf814
Fix compile error on gcc
2014-05-09 15:37:30 -07:00
Max Brunsfeld
292b753914
Move lr_parser into its own header file
2014-05-09 14:43:43 -07:00
Max Brunsfeld
ccc1b41f2a
Make separate header files for stack and lexer
2014-05-09 13:32:12 -07:00
Max Brunsfeld
e4be585c43
Handle ubiquitous tokens at the beginning of programs
...
As a final step before returning the finished parse tree, check if
there are still multiple nodes on the stack. If so, make the inner
nodes children of the top node.
2014-05-09 12:46:36 -07:00
Max Brunsfeld
3f374c6547
Tidy up
2014-05-08 13:27:48 -07:00
Max Brunsfeld
34137be12d
Represent state ids as unsigned shorts
...
This fixes some signedness conversion warnings
2014-05-08 13:23:46 -07:00
Max Brunsfeld
013572671f
Use smaller integer types for parse table
2014-05-08 08:45:41 -07:00
Max Brunsfeld
4700e33746
Introduce 'ubiquitous_tokens' concept, for parsing comments and such
2014-05-06 12:54:04 -07:00
Max Brunsfeld
5708a181c2
Removed unused field on reduce parse actions
2014-05-01 23:29:01 -07:00
Max Brunsfeld
a437d39773
Add rule precedence construct
...
Still need to add some way of expressing left and right
associativity
2014-04-15 08:40:46 -07:00
Max Brunsfeld
e23604ac52
Fix debugging macros in parser.h
2014-04-14 22:31:11 -07:00
Max Brunsfeld
5145bba53d
Silence missing-initializer warnings for gcc
2014-04-12 20:16:16 -07:00
Max Brunsfeld
bd5ec68c96
Get generated parsers building under gcc
2014-04-08 22:11:20 -07:00
Max Brunsfeld
5320cad065
Trim trailing whitespace
2014-04-04 13:10:55 -07:00
Max Brunsfeld
1cc7e32e2d
Fix handling of tokens consisting of separator characters
...
The parser is no longer hard-coded to skip whitespace. Tokens
such as newlines, whose characters overlap with the separator
characters, can now be correctly recognized.
2014-04-03 19:10:09 -07:00
Max Brunsfeld
13c4e6e648
Tweak format for example grammars
2014-03-28 13:51:32 -07:00
Max Brunsfeld
4454925b5a
Clean up parser macros more
2014-03-26 13:03:12 -07:00
Max Brunsfeld
f601322956
Clean up macros in parser.h
2014-03-25 19:51:34 -07:00
Max Brunsfeld
80b19cbb83
Construct entire parse table statically
...
This removes the need for the 'init_parse_table' function,
which was not really thread safe
2014-03-25 19:34:17 -07:00
Max Brunsfeld
25861b7f03
Remove reduction-specific collapse flags in favor of globally hidden symbols
2014-03-25 09:05:55 -07:00
Max Brunsfeld
3a7c4bb5b1
Store AST nodes' non-hidden children
2014-03-24 01:03:32 -07:00
Max Brunsfeld
95188d84b6
Make tree struct private
2014-03-24 00:34:13 -07:00
Max Brunsfeld
c40411b4d1
Handle unexpected tokens properly
...
Also, add null and boolean values to json grammar
2014-03-22 15:46:58 -07:00
Karen Wang and Max Brunsfeld
f248ece3aa
Use table-driven tests for specific parsers
2014-03-22 15:19:13 -07:00
Max Brunsfeld
5869c1ea18
Clean up stack breakdown function
2014-03-21 13:02:25 -07:00
Max Brunsfeld
bdbe69258a
Fix bug in stack breakdown procedure
2014-03-21 12:46:23 -07:00
Max Brunsfeld
cbc66b359e
Cleanup
2014-03-19 23:10:28 -07:00
Max Brunsfeld
fbe8b0a905
Fix incremental parsing
...
Stop collapsing hidden symbols upon reducing them.
Sadly, this messes up the ability to re-use parse
trees. Instead, for now, hide these nodes when
stringifying parse trees
2014-03-19 19:27:31 -07:00
Max Brunsfeld
7e94a4f1b2
Start work on reading input incrementally
2014-03-18 13:23:21 -07:00
Max Brunsfeld
8e7f59db1c
Allow room in parse table for error and EOF entries
2014-03-18 08:04:17 -07:00
Max Brunsfeld
67b33a615b
Refactor generated parsers to used explicit table
...
This is slightly slower than encoding the parse table in
flow control, but allows the parser to inspect the parse
table more flexibly. This is needed for incremental parsing.
2014-03-17 18:43:17 -07:00
Max Brunsfeld
0d6435e24a
Pass edit information into parser function
2014-03-15 16:55:35 -07:00
Max Brunsfeld
464a0e8dad
Clean up parser header further
2014-03-15 15:15:59 -07:00
Max Brunsfeld
0dc3a95d0c
Refactor parser header
...
Make separate lexer, stack and parser structs.
2014-03-15 14:43:50 -07:00
Max Brunsfeld
ff0c8a98b8
Consolidate reading of input chunks in parser
2014-03-11 12:43:13 -07:00
Max Brunsfeld
42e9a264f3
In parser, read chunked input correctly
2014-03-10 13:25:31 -07:00
Max Brunsfeld
31a58bc7e4
Make include guards pass cpplint
2014-03-09 22:05:24 -07:00
Max Brunsfeld
9c6c47e02f
Make parser input reads indicate the # of bytes read
2014-03-08 16:51:08 -08:00
Max Brunsfeld
b167ee84fa
Fix integer sign warning in parser.h
2014-03-08 15:04:23 -08:00
Max Brunsfeld
0f4b6332cb
Refactor document to allow other input data structures
2014-03-01 22:43:25 -08:00
Max Brunsfeld
72a0c0e09c
Keep track of AST node sizes and positions
2014-03-01 15:44:25 -08:00
Max Brunsfeld
ded54a3a1a
Fix some memory management bugs with trees
2014-03-01 00:25:05 -08:00
Max Brunsfeld
926b4ef242
Clean up parser.h
2014-02-27 00:06:22 -08:00
Max Brunsfeld
0d4687bdad
Make start and EOF symbols into builtin symbols
2014-02-26 23:18:49 -08:00
Max Brunsfeld
63718d5bad
Add basic support for error recovery
2014-02-26 19:03:43 -08:00
Max Brunsfeld
df223d566e
Fix bug in parser error handling
2014-02-25 18:33:24 -08:00
Max Brunsfeld
4541332c2b
Put definition of EOF symbol back in parse table builder (not in parser.h)
2014-02-25 13:25:58 -08:00
Max Brunsfeld
e58a6d8ba7
Start work on error recovery
...
- In runtime, make parse errors part of the parse tree
- Add error state to lexers in which they can accept any token
2014-02-24 18:42:54 -08:00
Max Brunsfeld
a34da59889
Update naming style in parser.h
2014-02-20 18:43:35 -08:00
