feat(core): start support for hex numbers #553
Conversation
|
I'm still figuring out GitHub. I made this PR a draft because it's not ready to be merged. But I think "draft" also means "not ready for review" but I specifically want it reviewed to get feedback on what it should do with text "stuck to" the hex that is not hex. And perhaps how to change the code to handle comply to what's decided. Especially if we want to flag such "hex with appendages". If my understanding of draft status for a PR is still wrong, please let me know. |
There was a problem hiding this comment.
@hippietrail, the code looks good. Would you mind adding some test cases under harper-core/tests/test_sources?
I would add a couple Markdown files that should be considered "clean" with this lexer added. Make sure you actually import them in harper-core/tests/run_tests.rs.
Are you sure? Did you see my concerns about what to do regarding non-hex text immediately after the hex with no space etc between? After some thought I came to grok the problem more completely and refactored the code, now written in a more old-school way that I'm used to doing lexing which makes it easier to think about when it gets tricky. I added positive and negative tests each in a loop. Let me know if this is a good or bad idea. |
Clear behaviour on non-hex appended to valid hex. Added test cases. Note that the `lex_number` will match the `0` of hex with appended non-hex though. Sould be vary rare but would be counterinutive for the user.
harper-core/src/lexing/mod.rs
Outdated
| } | ||
|
|
||
| #[test] | ||
| fn lexes_bad_hex() { |
There was a problem hiding this comment.
I would rename this to does_not_lex_bad_hex. Even better would be to split these two tests into many smaller ones.
There was a problem hiding this comment.
Went with the "even better" option.
towards #543
based on the existing
lex_numberwhich cuts off when the digits finish, which is fine because a unit of measurement etc might immediately follow. But with a hex number a non-hex letter might follow and not be flagged if it's a single letter. Also if it's very wrong, the start won't be flagged if it's0xand at least one hex digit. Examples will help illustrate:So just a draft PR to solicit thoughts and mods.