Tokenizer.md 3.5 KB
Tokenizer
- Static properties
- Static methods
- firstCharOffset(source)
- isHex(code)
- isNumber(code)
- isNewline(source, offset, code)
- cmpChar(testStr, offset, referenceCode)
- cmpStr(testStr, start, end, referenceStr)
- endsWith(testStr, referenceStr)
- findLastNonSpaceLocation(scanner)
- findWhitespaceEnd(source, offset)
- findCommentEnd(source, offset)
- findStringEnd(source, offset, quote)
- findDecimalNumberEnd(source, offset)
- findNumberEnd(source, offset, allowFraction)
- findEscaseEnd(source, offset)
- findIdentifierEnd(source, offset)
- Methods
- setSource(source, startOffset, startLine, startColumn)
- lookupType(offset)
- lookupNonWSType(offset)
- lookupValue(offset, referenceStr)
- getTokenValue()
- substrToCursor(start)
- skipWS()
- skip(tokenCount)
- next()
- eat(tokenType)
- eatNonWS(tokenType)
- consume(tokenType)
- consumeNonWS(tokenType)
- expectIdentifier(name)
- getLocation(offset, filename)
- getLocationRange(start, end, filename)
- error(message, offset)
- getTypes()