b0b4699476
[lexer] Rewrite the Lexer!
...
Implement structs for each state in the lexer. The State objects are responsible
for determining how to handle a character and emit a Result indicating to the
driver (the Lexer struct) how to proceed.
2018-08-26 11:47:18 -07:00
cc43ffd135
[lexer] Lexer emits Lexes instead of Tokens
2017-06-26 21:54:57 -07:00
5f3770914f
[lexer] Basic handling of input offsets
2017-06-26 21:54:05 -07:00
2a7626c75f
Add a type def for lexer Result
2017-06-18 16:50:14 -07:00
d994316392
Character checks for identifier initial and subsequent
2017-05-13 17:21:23 -07:00
237dca4b4b
Add some character class methods to a Lexable trait for char
2017-05-13 15:37:01 -07:00
1dfdc001b3
Add an error class
2017-05-13 15:26:41 -07:00
28e5814101
Check for EOF (aka input.next() -> None) and emit what we have before finishing forever
2017-05-02 21:44:57 -07:00
5fe10fe002
Simplify the resume check -- just use an if
2017-05-02 21:44:01 -07:00
c5b769ff45
A peekable lexer
...
Use the chars() iterator on &str, with the Peekable wrapper on Iterators, to iterate the input, rather than needing to hold the whole input and do iteration by indexes.
2017-04-30 17:46:42 -07:00
a4282e7760
Blow away the lexer code =o
2017-04-30 16:32:31 -07:00
21263fb2ed
Lexer doesn't have a named_char module anymore
2017-04-30 15:57:02 -07:00
f9d9a08455
Add a failing test that the lexer finds rational numbers
2017-04-15 10:35:13 -07:00
fc947280ae
Move the lexer to its own sibillexer module
...
Lots of failing tests right now, unfortunately. :(
2017-04-15 09:37:12 -07:00
