Commit graph

1490 commits

Author SHA1 Message Date
Max Brunsfeld
8ee3f96960 Fix formatting of non-ascii unexpected characters
Signed-off-by: Philip Turnbull <philipturnbull@github.com>
2017-06-23 12:08:50 -07:00
Phil Turnbull
8a15da90fb Update utf8proc dependency to v2.1
This includes JuliaLang/utf8proc#66 which is an out-of-bounds read when parsing
malformed utf8 characters.
2017-06-21 09:55:13 -04:00
Max Brunsfeld
513edec7c1 Merge pull request #77 from philipturnbull/scan-build-fixes
Fix errors found by scan-build
2017-06-20 10:15:20 -07:00
Phil Turnbull
7bdb0917d3 Revert "Add standalone scan-build script"
This reverts commit 9135d14b81.
2017-06-19 10:24:40 -04:00
Phil Turnbull
1b9e78add2 Fix formatting and use '-n' 2017-06-19 10:24:08 -04:00
Phil Turnbull
e9bf794cd9 Remove unneeded build step 2017-06-19 10:23:54 -04:00
Phil Turnbull
ee3caafe7b Use -j2 on the CI boxes
The travis-ci trusty container has two cores.
2017-06-16 15:16:40 -04:00
Phil Turnbull
9135d14b81 Add standalone scan-build script
For running scan-build outside of CI, e.g. `./script/scan-build -j4`
2017-06-16 15:16:40 -04:00
Phil Turnbull
7171664eec Disable DeadStores scan-build checker
This silences a true, but minor, bug in the external json-parser:

externals/json-parser/json.c:653:37: warning: Value stored to 'b' is never read
                                    b = 0;
                                    ^   ~
2017-06-16 15:16:40 -04:00
Phil Turnbull
97cdd8b738 Run scan-build during CI
This bumps the travis-ci container image to Trusty so that we have a version of
clang that includes proper support for C++14.
2017-06-16 15:16:29 -04:00
Max Brunsfeld
f29c41b7e8 Merge pull request #71 from tree-sitter/update-fixture-grammars
Run randomized tests against the latest Javascript grammar
2017-06-15 17:29:39 -07:00
Max Brunsfeld
e19393eff5 🎨 2017-06-15 17:25:21 -07:00
Max Brunsfeld
932feb2498 Fix gcc warning in test helper file 2017-06-15 17:13:01 -07:00
Max Brunsfeld
fa81a764fb Cache test grammar directory on Travis 2017-06-15 17:12:14 -07:00
Max Brunsfeld
599367d36d Always recur into error nodes when reporting changed ranges 2017-06-15 17:06:48 -07:00
Max Brunsfeld
c66fddd3aa Add TSInput option to measure columns in bytes not characters 2017-06-15 16:35:34 -07:00
Phil Turnbull
cfca764d48 Root can never be NULL in this context 2017-06-15 07:47:16 -04:00
Max Brunsfeld
b862db766e Merge remote-tracking branch 'origin/master' into update-fixture-grammars 2017-06-14 17:11:44 -07:00
Phil Turnbull
18f261ad51 Initialise all fields of TSParseOptions in tests
This should prevent any confusing failures in the unit tests:

test/runtime/document_test.cc:381:7: warning: Passed-by-value struct argument contains uninitialized data (e.g., field: 'changed_range_count')
      ts_document_parse_with_options(document, options);
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test/runtime/document_test.cc:408:7: warning: Passed-by-value struct argument contains uninitialized data (e.g., field: 'changed_range_count')
      ts_document_parse_with_options(document, options);
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2017-06-14 11:12:06 -04:00
Phil Turnbull
aa6e93820c Silence false-positive warning in ts_record_free
This is safe but I think it is technically undefined behaviour to use a pointer
after it has been freed:

test/helpers/record_alloc.cc:75:3: warning: Use of memory after it is freed
  record_deallocation(pointer);
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
2017-06-14 11:12:06 -04:00
Phil Turnbull
d1b19e8196 Prevent NULL pointer dereference in parser__accept
parser__select_tree can return true if 'left != NULL' and 'right == NULL' which
will later cause a NULL ptr deref:

src/runtime/parser.c:842:14: warning: Access to field 'ref_count' results in a dereference of a null pointer (loaded from variable 'root')
      assert(root->ref_count > 0);
             ^~~~~~~~~~~~~~~
2017-06-14 11:12:06 -04:00
Phil Turnbull
da099d0bbe Prevent NULL pointer dereference in parser__repair_error_callback
Because repair_reduction_count is unsigned, the default of '-1' is 0xffffffff
and will cause the loop to be entered if repair_reduction_count is NULL:

src/runtime/parser.c:691:11: warning: Dereference of null pointer
      if (repair_reductions[j].params.symbol == repair->symbol) {
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2017-06-14 11:12:06 -04:00
Phil Turnbull
fdd8792ebc Correctly set is_first
From scan-build: Value stored to 'is_first' is never read
2017-06-14 11:12:06 -04:00
Max Brunsfeld
040e225a54 Merge pull request #76 from philipturnbull/remove-duplicates-oob
OOB-read in remove_duplicate_parse_states
2017-06-14 07:20:09 -07:00
Phil Turnbull
c58f6401d0 Non-terminal entries always have valid state-ids 2017-06-14 08:49:38 -04:00
Phil Turnbull
577e43f653 shift-extra actions do not have valid state_ids 2017-06-09 16:26:01 -04:00
Phil Turnbull
18ba6ebbd7 Move state_id check into each_referenced_state 2017-06-09 16:25:59 -04:00
Phil Turnbull
6897530c47 Check for invalid state indexes
Some ParseActions have a state-id of -1 which can cause an out-of-bounds read
when removing duplicate parse states. This was found by AddressSanitizer:

==90699==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6320000187f8 at pc 0x0001071220a9 bp 0x7fff595fd440 sp 0x7fff595fd438
READ of size 8 at 0x6320000187f8 thread T0
    #0 0x1071220a8 in tree_sitter::build_tables::ParseTableBuilder::remove_duplicate_parse_states()::'lambda0'(unsigned long*)::operator()(unsigned long*) const build_parse_table.cc:398
    #1 0x107121fa5 in void std::__1::__invoke_void_return_wrapper<void>::__call<tree_sitter::build_tables::ParseTableBuilder::remove_duplicate_parse_states()::'lambda0'(unsigned long*)&, unsigned long*>(tree_sitter::build_tables::ParseTableBuilder::remove_duplicate_parse_states()::'lambda0'(unsigned long*)&&&, unsigned long*&&) __functional_base:416
...
0x6320000187f8 is located 8 bytes to the left of 88264-byte region [0x632000018800,0x63200002e0c8)
allocated by thread T0 here:
    #0 0x107b1576b in wrap__Znwm (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x6076b)
    #1 0x10711da2c in std::__1::vector<unsigned long, std::__1::allocator<unsigned long> >::allocate(unsigned long) new:169
    #2 0x10711d8fb in std::__1::vector<unsigned long, std::__1::allocator<unsigned long> >::vector(unsigned long) vector:1074
    #3 0x107112f5c in std::__1::vector<unsigned long, std::__1::allocator<unsigned long> >::vector(unsigned long) vector:1068
    #4 0x1070af381 in tree_sitter::build_tables::ParseTableBuilder::remove_duplicate_parse_states() build_parse_table.cc:378
    #5 0x10709d827 in tree_sitter::build_tables::ParseTableBuilder::build() build_parse_table.cc:85
...
SUMMARY: AddressSanitizer: heap-buffer-overflow build_parse_table.cc:398 in tree_sitter::build_tables::ParseTableBuilder::remove_duplicate_parse_states()::'lambda0'(unsigned long*)::operator()(unsigned long*) const
Shadow bytes around the buggy address:
  0x1c64000030a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c64000030b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c64000030c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c64000030d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c64000030e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x1c64000030f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa[fa]
  0x1c6400003100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6400003110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6400003120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6400003130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6400003140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2017-06-07 17:23:44 -04:00
Phil Turnbull
dee86f908a Correctly check type is ParseActionTypeRecover 2017-06-07 17:05:39 -04:00
Max Brunsfeld
f829e81f30 Add MIT license 2017-05-03 10:32:12 -07:00
Max Brunsfeld
7b401de5a6 Don't use pointer equality to compare external token states 2017-05-03 09:57:09 -07:00
Max Brunsfeld
34b47536e0 Merge pull request #75 from tree-sitter/halt-on-error-option
Add an option to immediately halt on syntax error
2017-05-03 09:49:40 -07:00
Max Brunsfeld
e8a9bb7a51 🎨 Extract parser__halt_parse function 2017-05-01 14:41:55 -07:00
Max Brunsfeld
74f5ceddf7 Fix parsing of valid code with halt_on_error flag set
Signed-off-by: Tim Clem <timothy.clem@gmail.com>
2017-05-01 14:25:25 -07:00
Max Brunsfeld
a98d449d88 Add an option to immediately halt on syntax error 2017-05-01 13:50:49 -07:00
Timothy Clem
f594ed2519 Merge pull request #74 from tree-sitter/check-utf8proc_iterate-return
Check utf8proc_iterate return
2017-05-01 10:46:24 -07:00
Max Brunsfeld
82cb1c9806 Handle invalid UTF8 in encoding test helpers
Signed-off-by: Tim Clem <timothy.clem@gmail.com>
2017-04-27 14:49:44 -07:00
Max Brunsfeld
704c2d5907 Fix lookahead_char type in ts_tree_make_error function 2017-04-27 14:49:04 -07:00
Max Brunsfeld
03a555a86e Finish test for invalid UTF8 handling
Signed-off-by: Tim Clem <timothy.clem@gmail.com>
2017-04-27 14:48:16 -07:00
Timothy Clem
37f2a4745f Test demonstrating non-UT8 input failure 2017-04-27 14:46:36 -07:00
Timothy Clem
91558f0a0e utf8proc_iterate can set codepoint_ref to -1 and returns negative error 2017-04-27 14:46:36 -07:00
Rob Rix
09d3b5ef3a Merge pull request #73 from tree-sitter/symbol-types
Symbol types
2017-04-12 12:42:57 -04:00
Rob Rix
3a888b1623 Define a function providing the type of a given symbol. 2017-04-12 09:47:51 -04:00
Rob Rix
4b1f69142d Define a symbol type enum. 2017-04-12 09:46:01 -04:00
Max Brunsfeld
3298c2ce88 Merge pull request #72 from tree-sitter/unique-external-scanner-map
Make external scanner symbol map unique
2017-03-24 16:02:47 -07:00
joshvera
f76935cc7e just make it static 2017-03-24 18:38:21 -04:00
joshvera
6938b288a5 Make external scanner symbol map unique 2017-03-24 14:51:37 -04:00
Max Brunsfeld
1f908324dc Prevent infinite loop in skip_preceding_trees error recovery strategy 2017-03-21 12:14:44 -07:00
Max Brunsfeld
a15e974150 Make clearer assertions about SpyInput's read strings 2017-03-21 12:14:04 -07:00
Max Brunsfeld
ca943f09a4 Update expected trees in error recovery test 2017-03-21 11:41:01 -07:00