Alright, so this is ugly. I'm really not happy
with how it turned out for the most part, but
that's what refactoring is for. For now, though,
i will sleep because i'm seriously tired.
I tried to parse a function called
"fn_with_callback" for the upcoming function
support and realized that the lexer mistook the
first two letters for the respective keyword.
So that is fixed now, i guess.
This adds the basic set of expressions:
assignments, binary/unary expressions, function
calls, and array accesses. Binary operator
precedence is also taken into account. What's
still missing are parenthesized expressions, but
those should be easy to do.
When skipping whitespace, the token position must
be updated accordingly. Additionally, the token
position is now updated when creating a new one.
It still doesn't work because something with the
cursor is broken.
Up to now, the cursor broke the line *before* the
\n and made that character the 0th one in the new
line. I don't know why i did this, but at least
it's fixed now.
This is probably a bad idea. Every stage in the
compiler pipeline will use the same type for
emitting errors now (good), but it's doing so
using a bunch of wrappers (bad, i think).
This is a cheap hack to just omit comments from
the token stream for now. I doubt they will ever
be used at all, so their existence would just mean
extra work for the parser.
The lexer now records every token it encounters
internally such that peeking and other useful
stuff is possible. Will come in handy for the
parser. Also, while i was at it, i also added
more primitive tokens like + - * / because i'm
probably gonna need it later anyway.
This is a little helper for the lexer so it can
read a stream forwards and backwards. It will
also keep track of line and column numbers for
better error output.
