9d40cdd995
[lexer] Move number integration tests to their own module
2018-09-13 07:55:42 -07:00
107bb394b7
[lexer] Link sibiltypes
2018-09-08 16:24:04 -07:00
9f5165f0aa
[lexer] Add expression tests; add explicit_sign to identifier_initials
2018-09-08 11:24:46 -07:00
061868d2c2
[lexer] Add Quote token
2018-09-08 11:16:53 -07:00
34d612a832
[lexer] Add state for finding Dots at the beginning of input
...
This happens to be a valid token by itself and the beginning of a decimal number.
2018-09-06 18:07:40 -07:00
15e275513d
[lexer] Pass Builders around by reference instead of implicitly Copying
2018-09-06 17:21:30 -07:00
9365e51893
[lexer] Correctly set line/offset for emitted Lexes
2018-09-05 22:18:27 -07:00
8b96eb190c
[lexer] Continue on to Digit state after lexing Prefix characters
2018-09-05 22:17:18 -07:00
c8de135d2f
[lexer] Update failing integer lexing test after fixing #12
2018-09-05 17:36:29 -07:00
e139cf0c6b
[lexer] Lex and discard whitespace
...
Closes #12 .
2018-09-04 20:09:47 -07:00
0bdfc24abd
[lexer] Simply some Option logic
2018-09-03 19:24:06 -07:00
569fe82c1a
[lexer] Identify delimiters and emit numbers
2018-09-03 17:20:27 -07:00
853312ce67
[lexer] We can lex integers!
2018-09-03 17:17:49 -07:00
d272b211ae
[lexer] WIP number::Digit state
2018-09-03 17:08:07 -07:00
176686b3a7
[lexer] Let states handle their own creation in from_char
2018-09-03 16:06:52 -07:00
eabc8f98e8
[lexer] Add number::Sign state
2018-09-03 15:49:13 -07:00
663ae3a9f1
[lexer] move char::is_hash() to chars::Lexable
2018-09-03 15:48:48 -07:00
dc8f5a7686
[lexer] Replace String error messages with Error type
2018-09-03 15:30:48 -07:00
def35966eb
[lexer] Re-kajigger the states for numbers
2018-09-03 15:19:28 -07:00
b759ee4c57
[lexer] Add Error::invalid_char constructor
2018-09-03 15:18:15 -07:00
469929fb8f
[lexer] Make a states::number module
...
There will be several number-specific states.
2018-09-03 14:05:57 -07:00
14ea07e441
[lexer] Move Bool state to its own file
2018-09-03 12:27:09 -07:00
ebee8c8646
[lexer] Add Num token type, which only takes ints for now...
2018-09-02 17:24:07 -07:00
0f18569292
[lexer] Add (failing) test for integers
2018-09-02 17:23:48 -07:00
04f2eb0937
[lexer] Remove empty Lexer impl
2018-09-02 17:23:32 -07:00
757f943fff
[lexer] Oops forgot a return type on Hash::new()
2018-09-02 14:05:56 -07:00
7a6c2b91d1
[lexer] Bit of code cleanup in Lexer
2018-09-02 14:05:35 -07:00
f35fe5fd08
[lexer] Implement Hash::new()
2018-09-02 13:50:33 -07:00
aa4de7d4bd
[lexer] Laying groundwork for lexing numbers...
2018-09-01 22:25:40 -07:00
7b6259977f
[lexer] Add fail() constructor to StateResult
2018-09-01 22:21:21 -07:00
d69c3dbc31
[lexer] Obey Rust book recommendation about order of leading lines
2018-09-01 22:21:05 -07:00
e61dc0b6a4
[lexer] Add failing test for issue #12
2018-08-27 06:56:48 -07:00
1304e04808
[lexer] Bring over the REPL from the parser
2018-08-26 13:45:52 -07:00
a23b917785
[lexer] Lex bools!
...
Extend the lexer to support Scheme bools like this: #t, #f, #true, #false.
Add some positive tests for this too.
2018-08-26 13:40:27 -07:00
b0b4699476
[lexer] Rewrite the Lexer!
...
Implement structs for each state in the lexer. The State objects are responsible
for determining how to handle a character and emit a Result indicating to the
driver (the Lexer struct) how to proceed.
2018-08-26 11:47:18 -07:00
4fa07ceec7
[lexer] Add Error::msg()
2018-08-23 20:19:16 -07:00
8475720a71
Add accessor for Lex::value
2018-08-20 15:21:20 -07:00
281c29ee26
[lexer] Expose token field via method; Clone and Copy Tokens
2018-08-19 22:30:19 -07:00
cc43ffd135
[lexer] Lexer emits Lexes instead of Tokens
2017-06-26 21:54:57 -07:00
5f3770914f
[lexer] Basic handling of input offsets
2017-06-26 21:54:05 -07:00
2a7626c75f
Add a type def for lexer Result
2017-06-18 16:50:14 -07:00
d994316392
Character checks for identifier initial and subsequent
2017-05-13 17:21:23 -07:00
237dca4b4b
Add some character class methods to a Lexable trait for char
2017-05-13 15:37:01 -07:00
1dfdc001b3
Add an error class
2017-05-13 15:26:41 -07:00
b5f76deb98
Add some single token tests for the tokens we have
2017-05-02 21:45:10 -07:00
28e5814101
Check for EOF (aka input.next() -> None) and emit what we have before finishing forever
2017-05-02 21:44:57 -07:00
5fe10fe002
Simplify the resume check -- just use an if
2017-05-02 21:44:01 -07:00
c5b769ff45
A peekable lexer
...
Use the chars() iterator on &str, with the Peekable wrapper on Iterators, to iterate the input, rather than needing to hold the whole input and do iteration by indexes.
2017-04-30 17:46:42 -07:00
a4282e7760
Blow away the lexer code =o
2017-04-30 16:32:31 -07:00
31c68034a8
Fix a good portion of the build errors in sibillexer (but not all...)
2017-04-30 16:00:51 -07:00
