Commit graph

1713 commits

Author SHA1 Message Date
Max Brunsfeld
7802bb3fcf
Merge pull request #110 from gnprice/compile
Several fixes to demo instructions in README
2017-10-31 09:43:42 -07:00
Greg Price
704b8ad810 README: Make the examples work despite variation in out/*/ structure.
Fixes #102.  While on macOS the `libcompiler.a` and `libruntime.a`
files apparently get output directly into `out/Release/` by the build,
on Linux they seem to go into a subdirectory `out/Release/obj.target/`.

This causes someone who tries to follow the instructions to get an
error like
```
/usr/bin/ld: cannot find -lcompiler
clang: error: linker command failed with exit code 1 (use -v to see invocation)
```

That looks like the setup is broken somehow, and for someone who
hasn't spent much time fixing up Unix linker errors, it doesn't give a
lot of clues as to how to fix it.

Instead, have the recommended command find the actual library file
directly, and pass that to the linker.  (Alternatively we could have
passed the containing directory to `-L`; this is a little shorter, and
of course neither version is what you'd want in an industrial-strength
build recipe.)

This isn't the world's most elegant solution, obviously -- it'd be
better (at least as far as this part is concerned) to have the layout
be the same on different platforms.  But doing that sounds likely to
require significant messing around with build scripts, so this at
least makes the instructions work reliably.
2017-10-30 22:23:40 -07:00
Greg Price
6bfe95fc95 README: Consistently use Release in examples.
Following the instructions in the README from the top, I don't
get a `Debug` build directory, only `Release`.  So, stick to that.
2017-10-28 23:41:11 -07:00
Greg Price
276ca6ab8a README: Fix the order of sources/libraries in example compilation commands.
Some linkers (among them, the GNU ld 2.28 on my laptop, which clang
relies on for linking) are picky about the order they get their
ingredients in on the command line.  When they are, they want the
leaf nodes (like your `main` function) first, and the dependencies
(like `libcompiler.a`) after.

It's kind of counterintuitive (at least to me) that this is the
preferred order, though I can understand it from the perspective
of efficient resource use in the era when some linkers were first
written.  Anyway, it's the way it is.  There's a detailed
explanation here:
  https://stackoverflow.com/a/409470/378130

So, adjust the examples in the README to an order that should
work everywhere.

This fixes part of #102.
2017-10-28 23:41:11 -07:00
Josh Vera
168d3ddb33 Merge pull request #108 from ZachTRice/fix-link
Updated links in References
2017-10-16 17:30:35 -04:00
Josh Vera
a2121cf01f Change link from paywalled version 2017-10-16 17:29:53 -04:00
ZachTRice
50e9520133 Updated links in References
The links to http://harmonia.cs.berkeley.edu got to a "forbidden, you' dont have permission to access..." error page. I have updated "Efficient and Flexible Incremental Parsing" to point to the ftp location on berkeley.edu. The only hosted reference to "Incremental Analysis of Real Programming Languages" is at ACM Digital Library, I pointed the link here.
2017-10-16 17:17:49 -04:00
Max Brunsfeld
d2ae16747c Merge pull request #106 from tree-sitter/sort-stack-versions-by-dynamic-precedence
Take total dynamic precedence into account in stack version sorting
2017-10-09 16:06:00 -07:00
Max Brunsfeld
121a6a66ec Take total dynamic precedence into account in stack version sorting
Signed-off-by: Josh Vera <vera@github.com>
2017-10-09 15:51:22 -07:00
Max Brunsfeld
36c2b685b9 Always invalidate old chunk of text when parsing after an edit 2017-10-04 15:09:46 -07:00
Max Brunsfeld
c0073c5b72 Update error corpus to reflect C grammar changes 2017-10-04 15:06:12 -07:00
Max Brunsfeld
9f24118b17 Include trees' dynamic precedence in debug graphs 2017-10-04 10:41:20 -07:00
Rob Rix
10111cc206 Update the link to the paper
The old link has gone stale.
2017-10-02 10:54:06 -04:00
Max Brunsfeld
ba607a1f84 Optimize lex state merging 2017-09-18 13:40:37 -07:00
Phil Turnbull
f82e0ec5fb Merge pull request #104 from tree-sitter/fuzzer-cflags
Allow fuzzer CFLAGS to be overridden by environment
2017-09-18 16:16:04 -04:00
Phil Turnbull
f08402014b Fuzz build should use CFLAGS when linking 2017-09-15 10:06:17 -04:00
Phil Turnbull
c4854fe595 Merge remote-tracking branch 'origin/master' into fuzzer-cflags 2017-09-15 10:05:39 -04:00
Max Brunsfeld
b0fdc33f73 Remove 'extra' and 'structural' booleans from symbol metadata 2017-09-14 12:07:46 -07:00
Max Brunsfeld
d342b61ede Re-enable JS fuzzing example test 2017-09-14 11:39:08 -07:00
Max Brunsfeld
9d67a98510 Merge pull request #103 from tree-sitter/python-assertion-failure
Assertion failure in parser__advance
2017-09-14 11:38:22 -07:00
Max Brunsfeld
037933ffc5 Bump LANGUAGE_VERSION constant due to incompatible parse table change 2017-09-14 11:09:26 -07:00
Max Brunsfeld
91456d7a17 Avoid duplicate error state entries for tokens that are both internal & external 2017-09-14 10:54:13 -07:00
Phil Turnbull
c57f8ae0a6 Allow fuzzer CFLAGS to be overridden by envrionment 2017-09-14 11:04:19 -04:00
Max Brunsfeld
2721f72c41 Represent MAX_COST_DIFFERENCE as unsigned 2017-09-13 16:49:18 -07:00
Max Brunsfeld
c1cf8e02a7 Merge pull request #101 from tree-sitter/merge-more-lex-states
Reduce the number of states in the generated lexer function
2017-09-13 16:46:58 -07:00
Max Brunsfeld
d291af9a31 Refactor error comparisons
* Deal with mergeability outside of error comparison function
* Make `better_version_exists` function pure (don't halt other versions
as a side effect).
* Tweak error comparison logic

Signed-off-by: Rick Winfrey <rewinfrey@github.com>
2017-09-13 16:38:15 -07:00
Phil Turnbull
d9a0fbc210 Add testcase for parser__advance assertion failure
The python testcase decodes to:
```
00000000  35 63 6f 6e 88 2c 29 33  2c 2c 2c 2c 63 6f 6e 88  |5con.,)3,,,,con.|
00000010  2c 2a 2c 3a 35 63 6f 6e  2c                       |,*,:5con,|
```

which triggers:
```
Assertion failed: ((uint32_t)0 < (&reduction.slices)->size), function parser__advance, file src/runtime/parser.c, line 1202.
```
2017-09-13 13:25:31 -04:00
Max Brunsfeld
71595ffde6 Only allow one stack link with a given type containing errors 2017-09-13 10:05:31 -07:00
Max Brunsfeld
07fb3ab0e6 Abort recoveries before popping if better versions already exist 2017-09-13 09:56:51 -07:00
Max Brunsfeld
65ed4281d4 Exclude zeros from speeds reported in benchmarks 2017-09-12 16:30:38 -07:00
Max Brunsfeld
47669e6015 Avoid halting the only non-halted entry in recover 2017-09-12 16:20:06 -07:00
Max Brunsfeld
ee2906ac2e Don't merge stack versions that are halted 2017-09-12 16:19:28 -07:00
Max Brunsfeld
819235bac3 Limit the number of stack nodes that are included in a summary 2017-09-12 12:00:00 -07:00
Max Brunsfeld
99d048e016 Simplify error recovery; eliminate recovery states
The previous approach to error recovery relied on special error-recovery
states in the parse table. For each token T, there was an error recovery
state in which the parser looked for *any* token that could follow T.
Unfortunately, sometimes the set of tokens that could follow T contained
conflicts. For example, in JS, the token '}' can be followed by the
open-ended 'template_chars' token, but also by ordinary tokens like
'identifier'. So with the old algorithm, when recovering from an
unexpected '}' token, the lexer had no way to distinguish identifiers
from template_chars.

This commit drops the error recovery states. Instead, when we encounter
an unexpected token T, we recover from the error by finding a previous
state S in the stack in which T would be valid, popping all of the nodes
after S, and wrapping them in an error.

This way, the lexer is always invoked in a normal parse state, in which
it is looking for a non-conflicting set of tokens. Eliminating the error
recovery states also shrinks the lex state machine significantly.

Signed-off-by: Rick Winfrey <rewinfrey@github.com>
2017-09-11 15:22:52 -07:00
Max Brunsfeld
8b3941764f Make outstanding_allocation_indices return a vector, not a set 2017-09-07 17:48:44 -07:00
Max Brunsfeld
4c9c05806a Merge compatible starting token states before constructing lex table 2017-09-05 13:21:53 -07:00
Max Brunsfeld
9d668c5004 Move incompatible token map into LexTableBuilder 2017-08-31 15:46:37 -07:00
Max Brunsfeld
f8649824fa Remove unused function 2017-08-31 15:30:44 -07:00
Max Brunsfeld
7d7e5f2c2c Merge pull request #100 from tree-sitter/fix-randomized-test-failures
Fix some bugs found using randomized tests
2017-08-31 12:51:15 -07:00
Max Brunsfeld
ac9d260734 Clean up parser fields 2017-08-31 12:50:10 -07:00
Max Brunsfeld
4a0587061e Consolidate logic for deciding on a lookahead node 2017-08-31 12:19:37 -07:00
Max Brunsfeld
41074cbf2d 🎨 2017-08-30 16:48:15 -07:00
Max Brunsfeld
fdc6ee445b Remove parser__push helper function 2017-08-30 16:41:07 -07:00
Max Brunsfeld
1b1276bdbf Simplify parser__condense_stack function 2017-08-30 16:36:02 -07:00
Max Brunsfeld
96a630e5df Clean up check for leaf node reusability 2017-08-30 16:19:51 -07:00
Max Brunsfeld
8bdab7335e Remove unnecessary reusability check after breaking down lookahead 2017-08-30 16:19:11 -07:00
Max Brunsfeld
bef536a7d0 Discard fragile reusable nodes earlier 2017-08-30 16:17:10 -07:00
Max Brunsfeld
5cbd50c7d7 Remember how far ahead the lexer looked on failed calls
This needs to be included in the 'bytes_scanned' property of the token
that is ultimately produced.
2017-08-29 15:04:22 -07:00
Max Brunsfeld
f3977ec213 Always call deserialize on external scanner before scanning
Remembering the last token that the external scanner produced is
not worth the complexity.
2017-08-29 14:41:55 -07:00
Max Brunsfeld
c285fbef38 Clear LexTableBuilder's state after detecting conflicts 2017-08-25 17:11:39 -07:00