Commit graph

1456 commits

Author SHA1 Message Date
Phil Turnbull
da099d0bbe Prevent NULL pointer dereference in parser__repair_error_callback
Because repair_reduction_count is unsigned, the default of '-1' is 0xffffffff
and will cause the loop to be entered if repair_reduction_count is NULL:

src/runtime/parser.c:691:11: warning: Dereference of null pointer
      if (repair_reductions[j].params.symbol == repair->symbol) {
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2017-06-14 11:12:06 -04:00
Phil Turnbull
fdd8792ebc Correctly set is_first
From scan-build: Value stored to 'is_first' is never read
2017-06-14 11:12:06 -04:00
Max Brunsfeld
040e225a54 Merge pull request #76 from philipturnbull/remove-duplicates-oob
OOB-read in remove_duplicate_parse_states
2017-06-14 07:20:09 -07:00
Phil Turnbull
c58f6401d0 Non-terminal entries always have valid state-ids 2017-06-14 08:49:38 -04:00
Phil Turnbull
577e43f653 shift-extra actions do not have valid state_ids 2017-06-09 16:26:01 -04:00
Phil Turnbull
18ba6ebbd7 Move state_id check into each_referenced_state 2017-06-09 16:25:59 -04:00
Phil Turnbull
6897530c47 Check for invalid state indexes
Some ParseActions have a state-id of -1 which can cause an out-of-bounds read
when removing duplicate parse states. This was found by AddressSanitizer:

==90699==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6320000187f8 at pc 0x0001071220a9 bp 0x7fff595fd440 sp 0x7fff595fd438
READ of size 8 at 0x6320000187f8 thread T0
    #0 0x1071220a8 in tree_sitter::build_tables::ParseTableBuilder::remove_duplicate_parse_states()::'lambda0'(unsigned long*)::operator()(unsigned long*) const build_parse_table.cc:398
    #1 0x107121fa5 in void std::__1::__invoke_void_return_wrapper<void>::__call<tree_sitter::build_tables::ParseTableBuilder::remove_duplicate_parse_states()::'lambda0'(unsigned long*)&, unsigned long*>(tree_sitter::build_tables::ParseTableBuilder::remove_duplicate_parse_states()::'lambda0'(unsigned long*)&&&, unsigned long*&&) __functional_base:416
...
0x6320000187f8 is located 8 bytes to the left of 88264-byte region [0x632000018800,0x63200002e0c8)
allocated by thread T0 here:
    #0 0x107b1576b in wrap__Znwm (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x6076b)
    #1 0x10711da2c in std::__1::vector<unsigned long, std::__1::allocator<unsigned long> >::allocate(unsigned long) new:169
    #2 0x10711d8fb in std::__1::vector<unsigned long, std::__1::allocator<unsigned long> >::vector(unsigned long) vector:1074
    #3 0x107112f5c in std::__1::vector<unsigned long, std::__1::allocator<unsigned long> >::vector(unsigned long) vector:1068
    #4 0x1070af381 in tree_sitter::build_tables::ParseTableBuilder::remove_duplicate_parse_states() build_parse_table.cc:378
    #5 0x10709d827 in tree_sitter::build_tables::ParseTableBuilder::build() build_parse_table.cc:85
...
SUMMARY: AddressSanitizer: heap-buffer-overflow build_parse_table.cc:398 in tree_sitter::build_tables::ParseTableBuilder::remove_duplicate_parse_states()::'lambda0'(unsigned long*)::operator()(unsigned long*) const
Shadow bytes around the buggy address:
  0x1c64000030a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c64000030b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c64000030c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c64000030d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c64000030e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x1c64000030f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa[fa]
  0x1c6400003100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6400003110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6400003120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6400003130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6400003140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2017-06-07 17:23:44 -04:00
Phil Turnbull
dee86f908a Correctly check type is ParseActionTypeRecover 2017-06-07 17:05:39 -04:00
Max Brunsfeld
f829e81f30 Add MIT license 2017-05-03 10:32:12 -07:00
Max Brunsfeld
7b401de5a6 Don't use pointer equality to compare external token states 2017-05-03 09:57:09 -07:00
Max Brunsfeld
34b47536e0 Merge pull request #75 from tree-sitter/halt-on-error-option
Add an option to immediately halt on syntax error
2017-05-03 09:49:40 -07:00
Max Brunsfeld
e8a9bb7a51 🎨 Extract parser__halt_parse function 2017-05-01 14:41:55 -07:00
Max Brunsfeld
74f5ceddf7 Fix parsing of valid code with halt_on_error flag set
Signed-off-by: Tim Clem <timothy.clem@gmail.com>
2017-05-01 14:25:25 -07:00
Max Brunsfeld
a98d449d88 Add an option to immediately halt on syntax error 2017-05-01 13:50:49 -07:00
Timothy Clem
f594ed2519 Merge pull request #74 from tree-sitter/check-utf8proc_iterate-return
Check utf8proc_iterate return
2017-05-01 10:46:24 -07:00
Max Brunsfeld
82cb1c9806 Handle invalid UTF8 in encoding test helpers
Signed-off-by: Tim Clem <timothy.clem@gmail.com>
2017-04-27 14:49:44 -07:00
Max Brunsfeld
704c2d5907 Fix lookahead_char type in ts_tree_make_error function 2017-04-27 14:49:04 -07:00
Max Brunsfeld
03a555a86e Finish test for invalid UTF8 handling
Signed-off-by: Tim Clem <timothy.clem@gmail.com>
2017-04-27 14:48:16 -07:00
Timothy Clem
37f2a4745f Test demonstrating non-UT8 input failure 2017-04-27 14:46:36 -07:00
Timothy Clem
91558f0a0e utf8proc_iterate can set codepoint_ref to -1 and returns negative error 2017-04-27 14:46:36 -07:00
Rob Rix
09d3b5ef3a Merge pull request #73 from tree-sitter/symbol-types
Symbol types
2017-04-12 12:42:57 -04:00
Rob Rix
3a888b1623 Define a function providing the type of a given symbol. 2017-04-12 09:47:51 -04:00
Rob Rix
4b1f69142d Define a symbol type enum. 2017-04-12 09:46:01 -04:00
Max Brunsfeld
3298c2ce88 Merge pull request #72 from tree-sitter/unique-external-scanner-map
Make external scanner symbol map unique
2017-03-24 16:02:47 -07:00
joshvera
f76935cc7e just make it static 2017-03-24 18:38:21 -04:00
joshvera
6938b288a5 Make external scanner symbol map unique 2017-03-24 14:51:37 -04:00
Max Brunsfeld
3f0640aa8d Merge pull request #70 from tree-sitter/anonymous-external-tokens
Allow anonymous tokens to be used in grammars' external token lists
2017-03-17 17:10:13 -07:00
Max Brunsfeld
24878277e9 Use new version of python grammar in tests 2017-03-17 17:05:02 -07:00
Max Brunsfeld
6d8f9ebaba In tests, regenerate parser if its timestamp matches grammar.json
After running the fetch-fixtures script, their timestamps may be equal,
but in this situation we *do* want to re-generate the parsers.
2017-03-17 17:04:04 -07:00
Max Brunsfeld
17876b6826 Update grammar JSON schema 2017-03-17 16:41:30 -07:00
Max Brunsfeld
ed8fbff175 Allow anonymous tokens to be used in grammars' external token lists 2017-03-17 16:31:29 -07:00
Max Brunsfeld
e2baf0930b Use simple CharacterSet constructor in LexItem test 2017-03-17 14:47:11 -07:00
Max Brunsfeld
4d39f13eaf Merge pull request #69 from tree-sitter/rules-variant
Implement Rule as a union type rather than an abstract base class
2017-03-17 14:35:42 -07:00
Max Brunsfeld
b3edd8f749 Remove use of shared_ptr in choice, repeat, and seq factories 2017-03-17 14:28:13 -07:00
Max Brunsfeld
d9fb863bea Fix build errors w/ gcc 2017-03-17 14:03:49 -07:00
Max Brunsfeld
416cbb9def Add missing cassert includes 2017-03-17 13:54:40 -07:00
Max Brunsfeld
90d21adf3b Format make_visitor helper consistently w/ project 2017-03-17 13:37:26 -07:00
Max Brunsfeld
79ffc20be1 Remove RTTI flag in gyp file 2017-03-17 13:31:35 -07:00
Max Brunsfeld
db4b9ebc7c Implement Rule as a union rather than an abstract base class 2017-03-17 13:29:31 -07:00
Max Brunsfeld
272e3cc9c6 Merge pull request #68 from tree-sitter/lexer-lookahead
Allow lexer to look ahead in order to find the actual longest match
2017-03-13 17:40:52 -07:00
Max Brunsfeld
42b05b4b5e Add simple unit test for invalidating trees preceding an edit due to lookahead 2017-03-13 17:34:31 -07:00
Max Brunsfeld
d222dbb9fd Allow lexer to accept tokens that ended at previous positions
* Track lookahead in each tree
* Add 'mark_end' API that external scanners can use
2017-03-13 17:06:52 -07:00
Max Brunsfeld
12d2a9d93f Merge pull request #66 from tree-sitter/handle-unused-tokens
Handle unused tokens
2017-03-09 21:20:13 -08:00
Max Brunsfeld
f04d7c5860 Handle unused tokens 2017-03-09 21:16:37 -08:00
Max Brunsfeld
c79fae6d21 Clean up extract_tokens function 2017-03-09 21:16:20 -08:00
Max Brunsfeld
dc2035c262 Clean up gitignore 2017-03-09 20:49:11 -08:00
Max Brunsfeld
6dc0ff359d Rename spec -> test
'Test' is a lot more straightforward of a name.
2017-03-09 20:40:01 -08:00
Max Brunsfeld
7d8daf573e Remove gitattrs file
There's no longer any generated code checked in.
2017-03-09 20:31:18 -08:00
Max Brunsfeld
796fc26c74 Add a readme to another fixture grammar 2017-03-09 14:37:55 -08:00
Max Brunsfeld
53f72f0eca Fix list formatting in a fixture grammar readme 2017-03-09 12:28:25 -08:00