Commit graph

1741 commits

Author SHA1 Message Date
Max Brunsfeld
ee9df753ff
Merge pull request #119 from tree-sitter/recover-by-inserting-missing-tokens
Add the ability to recover from errors by inserting missing tokens
2017-12-29 20:32:21 -08:00
Max Brunsfeld
1e04489e50 Fix error in handling of padding in get_changed_ranges 2017-12-29 18:02:06 -08:00
Max Brunsfeld
f3c3fd3c9e Make it easier to enable/disable logging in get_changed_ranges 2017-12-29 18:01:42 -08:00
Max Brunsfeld
13adfe4927 Update error corpus 2017-12-29 16:21:04 -08:00
Max Brunsfeld
21a88b1731 Don't count less-far-along versions in better_version_exists method 2017-12-29 16:10:43 -08:00
Max Brunsfeld
d3c85f288d Start work on repairing errors by inserting missing tokens 2017-12-29 15:11:00 -08:00
Max Brunsfeld
f2dc620610 Extract parser__recover_to_state method 2017-12-29 15:10:59 -08:00
Max Brunsfeld
adf47e2b57 Fix invalid usage of 'extra' field on non-shift parse action 2017-12-29 11:46:41 -08:00
Max Brunsfeld
d9094e8146 Consolidate more logic into do_potential_reductions method 2017-12-28 15:49:48 -08:00
Max Brunsfeld
777aca25c8 Fix errors in test script 2017-12-28 15:49:34 -08:00
Max Brunsfeld
6304a3bcd1 Make it easier to run tests with debug graphs 2017-12-28 12:41:23 -08:00
Max Brunsfeld
eee3db08d2 Avoid repeated calls to {start,end}_point in descendant_for_point_range 2017-12-27 11:55:52 -08:00
Max Brunsfeld
1c25aafc82
Merge pull request #118 from tree-sitter/slab-allocation
Allocate and free trees using an object pool
2017-12-27 11:42:01 -08:00
Max Brunsfeld
172cbb2d22 Fix infinite loop due to skipping empty tokens during error recovery 2017-12-27 11:18:06 -08:00
Max Brunsfeld
2625c3a96c Remove dead code in string_input.c 2017-12-27 10:34:29 -08:00
Max Brunsfeld
addeb6c4c1 Allocate and free trees using an object pool 2017-12-27 10:34:29 -08:00
Max Brunsfeld
53152658ce
Merge pull request #117 from tree-sitter/external-scanner-get-column-api
Allow external scanners to query the lexer's current column
2017-12-21 16:20:03 -08:00
Max Brunsfeld
0e69da37a5 Return a character count from the lexer's get_column method 2017-12-20 16:26:38 -08:00
Max Brunsfeld
fcff16cb86 Add get_column method to lexer 2017-12-19 17:54:15 -08:00
Max Brunsfeld
e016561887
Merge pull request #116 from tree-sitter/mb-remove-invented-character-class
Remove wrong handling of \a in a regex
2017-12-13 12:34:45 -08:00
Max Brunsfeld
e5851fd9b9 Don't use non-existent \a syntax in test grammars 2017-12-13 12:21:28 -08:00
Max Brunsfeld
f426b61e7c Fix expectation around preproc directive in C error test 2017-12-13 12:21:13 -08:00
Max Brunsfeld
532bbeca0d Remove wrong handling of \a in a regex 2017-12-12 16:50:53 -08:00
Max Brunsfeld
fbcefe25f7 Avoid creating external tokens that start after they end 2017-12-07 11:50:27 -08:00
Max Brunsfeld
90629bd45a Add some assertions to the fixture grammar tests 2017-12-07 11:50:27 -08:00
Max Brunsfeld
5d676de051 Remove unnecessary conditional in parser__accept 2017-12-07 11:50:27 -08:00
Max Brunsfeld
493db39363 Never move the start rule of a grammar into the lexical grammar
This preserves a useful invariant that the root node of the AST is never
a token.
2017-12-07 11:50:27 -08:00
Max Brunsfeld
48681c3f0e Initialize error start and end positions at their declarations
Fixes #113

Clang doesn't seem to be able to tell that these variables were guaranteed to
be initialized by the time they were read.
2017-10-31 10:06:44 -07:00
Max Brunsfeld
7802bb3fcf
Merge pull request #110 from gnprice/compile
Several fixes to demo instructions in README
2017-10-31 09:43:42 -07:00
Greg Price
704b8ad810 README: Make the examples work despite variation in out/*/ structure.
Fixes #102.  While on macOS the `libcompiler.a` and `libruntime.a`
files apparently get output directly into `out/Release/` by the build,
on Linux they seem to go into a subdirectory `out/Release/obj.target/`.

This causes someone who tries to follow the instructions to get an
error like
```
/usr/bin/ld: cannot find -lcompiler
clang: error: linker command failed with exit code 1 (use -v to see invocation)
```

That looks like the setup is broken somehow, and for someone who
hasn't spent much time fixing up Unix linker errors, it doesn't give a
lot of clues as to how to fix it.

Instead, have the recommended command find the actual library file
directly, and pass that to the linker.  (Alternatively we could have
passed the containing directory to `-L`; this is a little shorter, and
of course neither version is what you'd want in an industrial-strength
build recipe.)

This isn't the world's most elegant solution, obviously -- it'd be
better (at least as far as this part is concerned) to have the layout
be the same on different platforms.  But doing that sounds likely to
require significant messing around with build scripts, so this at
least makes the instructions work reliably.
2017-10-30 22:23:40 -07:00
Greg Price
6bfe95fc95 README: Consistently use Release in examples.
Following the instructions in the README from the top, I don't
get a `Debug` build directory, only `Release`.  So, stick to that.
2017-10-28 23:41:11 -07:00
Greg Price
276ca6ab8a README: Fix the order of sources/libraries in example compilation commands.
Some linkers (among them, the GNU ld 2.28 on my laptop, which clang
relies on for linking) are picky about the order they get their
ingredients in on the command line.  When they are, they want the
leaf nodes (like your `main` function) first, and the dependencies
(like `libcompiler.a`) after.

It's kind of counterintuitive (at least to me) that this is the
preferred order, though I can understand it from the perspective
of efficient resource use in the era when some linkers were first
written.  Anyway, it's the way it is.  There's a detailed
explanation here:
  https://stackoverflow.com/a/409470/378130

So, adjust the examples in the README to an order that should
work everywhere.

This fixes part of #102.
2017-10-28 23:41:11 -07:00
Josh Vera
168d3ddb33 Merge pull request #108 from ZachTRice/fix-link
Updated links in References
2017-10-16 17:30:35 -04:00
Josh Vera
a2121cf01f Change link from paywalled version 2017-10-16 17:29:53 -04:00
ZachTRice
50e9520133 Updated links in References
The links to http://harmonia.cs.berkeley.edu got to a "forbidden, you' dont have permission to access..." error page. I have updated "Efficient and Flexible Incremental Parsing" to point to the ftp location on berkeley.edu. The only hosted reference to "Incremental Analysis of Real Programming Languages" is at ACM Digital Library, I pointed the link here.
2017-10-16 17:17:49 -04:00
Max Brunsfeld
d2ae16747c Merge pull request #106 from tree-sitter/sort-stack-versions-by-dynamic-precedence
Take total dynamic precedence into account in stack version sorting
2017-10-09 16:06:00 -07:00
Max Brunsfeld
121a6a66ec Take total dynamic precedence into account in stack version sorting
Signed-off-by: Josh Vera <vera@github.com>
2017-10-09 15:51:22 -07:00
Max Brunsfeld
36c2b685b9 Always invalidate old chunk of text when parsing after an edit 2017-10-04 15:09:46 -07:00
Max Brunsfeld
c0073c5b72 Update error corpus to reflect C grammar changes 2017-10-04 15:06:12 -07:00
Max Brunsfeld
9f24118b17 Include trees' dynamic precedence in debug graphs 2017-10-04 10:41:20 -07:00
Rob Rix
10111cc206 Update the link to the paper
The old link has gone stale.
2017-10-02 10:54:06 -04:00
Max Brunsfeld
ba607a1f84 Optimize lex state merging 2017-09-18 13:40:37 -07:00
Phil Turnbull
f82e0ec5fb Merge pull request #104 from tree-sitter/fuzzer-cflags
Allow fuzzer CFLAGS to be overridden by environment
2017-09-18 16:16:04 -04:00
Phil Turnbull
f08402014b Fuzz build should use CFLAGS when linking 2017-09-15 10:06:17 -04:00
Phil Turnbull
c4854fe595 Merge remote-tracking branch 'origin/master' into fuzzer-cflags 2017-09-15 10:05:39 -04:00
Max Brunsfeld
b0fdc33f73 Remove 'extra' and 'structural' booleans from symbol metadata 2017-09-14 12:07:46 -07:00
Max Brunsfeld
d342b61ede Re-enable JS fuzzing example test 2017-09-14 11:39:08 -07:00
Max Brunsfeld
9d67a98510 Merge pull request #103 from tree-sitter/python-assertion-failure
Assertion failure in parser__advance
2017-09-14 11:38:22 -07:00
Max Brunsfeld
037933ffc5 Bump LANGUAGE_VERSION constant due to incompatible parse table change 2017-09-14 11:09:26 -07:00
Max Brunsfeld
91456d7a17 Avoid duplicate error state entries for tokens that are both internal & external 2017-09-14 10:54:13 -07:00