- Parsing step (scanned, split up, groped) aka. lexical (isolation from the sentence containing it) analysis. (grammar or meaning of text not taken into context. Just the meaning of the words themselves)
Lexical Analysis phase
1 Scans code
- Source text is considered as a chunk of string text.
- scanner reads the text one character at a time.
- for each char. it marks the line and position of where the character was found in source text.
2 Evaluates (lexing/tokenization)
- the lexer/tokenizer determines what type of token it has found
Example of how compiler lexes a phrase
- represented as a pair consisting of a token name and some (optional) value
- words of a program
- substring of source code.
- grouping of smallest sequence of characters.