traxys/tree-sitter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Refine logic for deciding when tokens need to be re-lexed

Browse source 
* While generating the lex table, note which tokens can match the
  same string. A token needs to be relexed when it has possible
  homonyms in the current state.
* Also note which tokens can match substrings of each other tokens.
  A token needs to be relexed when there are viable tokens that
  could match longer strings in the current state and the next
  token has been edited.
* Remove the logic for marking tokens as fragile on creation.
* Store the reusability/non-reusability of symbols off of individual
  actions and onto the entire entry for the state & symbol.
This commit is contained in:
Max Brunsfeld 2016-06-21 07:28:04 -07:00
parent 45f7cee0c8
commit 38c144b4a3
19 changed files with 337 additions and 257 deletions
Download patch file Download diff file Expand all files Collapse all files

43
src/runtime/language.h
View file

@ -8,13 +8,48 @@ extern "C" {
#include "tree_sitter/parser.h"
#include "runtime/tree.h"
typedef struct {
const TSParseAction *actions;
size_t action_count;
bool is_reusable;
bool depends_on_lookahead;
} TableEntry;
void ts_language_table_entry(const TSLanguage *, TSStateId, TSSymbol,
TableEntry *);
bool ts_language_symbol_is_in_progress(const TSLanguage *, TSStateId, TSSymbol);
const TSParseAction *ts_language_actions(const TSLanguage *, TSStateId,
TSSymbol, size_t *);
TSParseAction ts_language_last_action(const TSLanguage *, TSStateId, TSSymbol);
static inline const TSParseAction *ts_language_actions(const TSLanguage *self,
TSStateId state,
TSSymbol symbol,
size_t *count) {
TableEntry entry;
ts_language_table_entry(self, state, symbol, &entry);
*count = entry.action_count;
return entry.actions;
}
bool ts_language_has_action(const TSLanguage *, TSStateId, TSSymbol);
static inline TSParseAction ts_language_last_action(const TSLanguage *self,
TSStateId state,
TSSymbol symbol) {
TableEntry entry;
ts_language_table_entry(self, state, symbol, &entry);
return entry.actions[entry.action_count - 1];
}
static inline bool ts_language_has_action(const TSLanguage *self,
TSStateId state, TSSymbol symbol) {
TSParseAction action = ts_language_last_action(self, state, symbol);
return action.type != TSParseActionTypeError;
}
static inline bool ts_language_is_reusable(const TSLanguage *self,
TSStateId state, TSSymbol symbol) {
TableEntry entry;
ts_language_table_entry(self, state, symbol, &entry);
return entry.is_reusable;
}
TSSymbolMetadata ts_language_symbol_metadata(const TSLanguage *, TSSymbol);