traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1438 commits 16 branches 98 tags 37 MiB
Commit graph

1438 commits

This branch
This branch
All branches
Author SHA1 Message Date
Max Brunsfeld
f032da198e Finish test for invalid UTF8 handling 
Signed-off-by: Tim Clem <timothy.clem@gmail.com>
 2017-03-21 11:05:32 -07:00
Max Brunsfeld
63fb041961 Merge remote-tracking branch 'origin/check-utf8proc_iterate-return' into update-fixture-grammars 2017-03-21 09:59:35 -07:00
Timothy Clem
7092d4522a Test demonstrating non-UT8 input failure 2017-03-21 09:58:35 -07:00
Timothy Clem
f394a48c0b utf8proc_iterate can set codepoint_ref to -1 and returns negative error 2017-03-20 16:54:19 -07:00
Max Brunsfeld
af553420bf Update JS error corpus 2017-03-19 22:21:12 -07:00
Max Brunsfeld
ed31e82ee6 Skip empty tokens when recovering from errors 2017-03-19 22:20:59 -07:00
Max Brunsfeld
20b8983749 Handle external scanner w/ .c extension in test helper 2017-03-19 22:20:16 -07:00
Max Brunsfeld
97bcc86a3a Use master version of all fixture grammars 2017-03-19 22:19:35 -07:00
Max Brunsfeld
3f0640aa8d Merge pull request #70 from tree-sitter/anonymous-external-tokens 
Allow anonymous tokens to be used in grammars' external token lists
 2017-03-17 17:10:13 -07:00
Max Brunsfeld
24878277e9 Use new version of python grammar in tests 2017-03-17 17:05:02 -07:00
Max Brunsfeld
6d8f9ebaba In tests, regenerate parser if its timestamp matches grammar.json 
After running the fetch-fixtures script, their timestamps may be equal,
but in this situation we *do* want to re-generate the parsers.
 2017-03-17 17:04:04 -07:00
Max Brunsfeld
17876b6826 Update grammar JSON schema 2017-03-17 16:41:30 -07:00
Max Brunsfeld
ed8fbff175 Allow anonymous tokens to be used in grammars' external token lists 2017-03-17 16:31:29 -07:00
Max Brunsfeld
e2baf0930b Use simple CharacterSet constructor in LexItem test 2017-03-17 14:47:11 -07:00
Max Brunsfeld
4d39f13eaf Merge pull request #69 from tree-sitter/rules-variant 
Implement Rule as a union type rather than an abstract base class
 2017-03-17 14:35:42 -07:00
Max Brunsfeld
b3edd8f749 Remove use of shared_ptr in choice, repeat, and seq factories 2017-03-17 14:28:13 -07:00
Max Brunsfeld
d9fb863bea Fix build errors w/ gcc 2017-03-17 14:03:49 -07:00
Max Brunsfeld
416cbb9def Add missing cassert includes 2017-03-17 13:54:40 -07:00
Max Brunsfeld
90d21adf3b Format make_visitor helper consistently w/ project 2017-03-17 13:37:26 -07:00
Max Brunsfeld
79ffc20be1 Remove RTTI flag in gyp file 2017-03-17 13:31:35 -07:00
Max Brunsfeld
db4b9ebc7c Implement Rule as a union rather than an abstract base class 2017-03-17 13:29:31 -07:00
Max Brunsfeld
272e3cc9c6 Merge pull request #68 from tree-sitter/lexer-lookahead 
Allow lexer to look ahead in order to find the actual longest match
 2017-03-13 17:40:52 -07:00
Max Brunsfeld
42b05b4b5e Add simple unit test for invalidating trees preceding an edit due to lookahead 2017-03-13 17:34:31 -07:00
Max Brunsfeld
d222dbb9fd Allow lexer to accept tokens that ended at previous positions 
* Track lookahead in each tree
* Add 'mark_end' API that external scanners can use
 2017-03-13 17:06:52 -07:00
Max Brunsfeld
12d2a9d93f Merge pull request #66 from tree-sitter/handle-unused-tokens 
Handle unused tokens
 2017-03-09 21:20:13 -08:00
Max Brunsfeld
f04d7c5860 Handle unused tokens 2017-03-09 21:16:37 -08:00
Max Brunsfeld
c79fae6d21 Clean up extract_tokens function 2017-03-09 21:16:20 -08:00
Max Brunsfeld
dc2035c262 Clean up gitignore 2017-03-09 20:49:11 -08:00
Max Brunsfeld
6dc0ff359d Rename spec -> test 
'Test' is a lot more straightforward of a name.
 2017-03-09 20:40:01 -08:00
Max Brunsfeld
7d8daf573e Remove gitattrs file 
There's no longer any generated code checked in.
 2017-03-09 20:31:18 -08:00
Max Brunsfeld
796fc26c74 Add a readme to another fixture grammar 2017-03-09 14:37:55 -08:00
Max Brunsfeld
53f72f0eca Fix list formatting in a fixture grammar readme 2017-03-09 12:28:25 -08:00
Max Brunsfeld
352e678c12 Merge pull request #63 from tree-sitter/avoid-lexing-issues-when-merging-states 
Avoid introducing new lexical conflicts when merging parse states
 2017-03-09 12:19:46 -08:00
Max Brunsfeld
56ec45729c Add regression test for avoiding lexical conflicts due to state merging 2017-03-09 12:13:44 -08:00
Max Brunsfeld
ac4167fdc9 Restructure integration tests to use separate JSON and corpus files 
This makes these tests way easier to write and read.
 2017-03-09 11:49:36 -08:00
Max Brunsfeld
f049d5d94c Make ParseItem a struct, not a class 2017-03-08 21:06:30 -08:00
Max Brunsfeld
64e9230071 Use LexTableBuilder to detect conflicts between tokens more correctly 2017-03-08 12:47:38 -08:00
Max Brunsfeld
abf8a4f2c2 🎨 2017-03-01 22:15:26 -08:00
Max Brunsfeld
686dc0997c Avoid introducing certain lexical conflicts during parse state merging 
The current pretty conservative approach is to avoid merging parse states which
would cause a pair tokens to co-exist for the first time in any parse state,
where the two tokens can start with the same character and at least one of the
tokens can contain a character which is part of the grammar's separators.
 2017-02-27 22:54:38 -08:00
Max Brunsfeld
3c8e6f9987 Restructure parse state merging logic 
* Remove remnants of templatized remove_duplicate_states function
* Rename recovery_tokens function to get_compatible_tokens and augment it
  also compute pairs of tokens which could potentially be incompatible
 2017-02-26 12:23:48 -08:00
Max Brunsfeld
8d3b72e1d9 Merge pull request #62 from tree-sitter/fix-crash-when-iterating-crazy-stack 
Prevent crash due to huge number of possible paths through parse stack
 2017-02-20 14:43:17 -08:00
Max Brunsfeld
df520635c6 Prevent crash due to huge number of possible paths through parse stack 2017-02-20 14:34:10 -08:00
Max Brunsfeld
ce5b3e8284 Merge pull request #61 from tree-sitter/error-recovery-improvements 
More error recovery improvements
 2017-02-20 12:26:52 -08:00
Max Brunsfeld
cefc57fe86 Move error cost comparisons into their own source file 2017-02-19 21:54:06 -08:00
Max Brunsfeld
5b4e6df3ff Don't mark error nodes created in the error state as extras 2017-02-19 21:54:06 -08:00
Max Brunsfeld
c14a776a3d Avoid including trailing extra tokens within error nodes unnecessarily 2017-02-19 21:21:54 -08:00
Max Brunsfeld
135d8ef4e0 Merge pull request #58 from tree-sitter/reduce-error-recovery-branching 
Reduce the branching factor of the parse stack during error recovery
 2017-02-18 11:34:09 -08:00
Timothy Clem
87b8b932e5 Merge pull request #59 from tree-sitter/input-string-with-length 
Input string with length
 2017-02-13 09:58:46 -08:00
Timothy Clem
93e94bf4b2 Add a happy path test for ts_document_set_input_string_with_length 2017-02-13 08:48:51 -08:00
Rob Rix
638aa87e42 Pass through to ts_string_input_make_with_length. 2017-02-10 09:27:21 -05:00