This is probably a bad idea. Every stage in the
compiler pipeline will use the same type for
emitting errors now (good), but it's doing so
using a bunch of wrappers (bad, i think).
This is a cheap hack to just omit comments from
the token stream for now. I doubt they will ever
be used at all, so their existence would just mean
extra work for the parser.
The lexer now records every token it encounters
internally such that peeking and other useful
stuff is possible. Will come in handy for the
parser. Also, while i was at it, i also added
more primitive tokens like + - * / because i'm
probably gonna need it later anyway.
This is a little helper for the lexer so it can
read a stream forwards and backwards. It will
also keep track of line and column numbers for
better error output.
anna