The lexical rules describe how NXC breaks a source file into individual tokens.
This includes the way comments are written, the handling of whitespace, and valid characters for identifiers.