Home › GATE CSE › computerscience › Compiler Design › The DIFFERENCE between a TOKEN and a LEXEME is
The DIFFERENCE between a TOKEN and a LEXEME is
Athey are synonyms
Blexeme = substring; token adds type
Ca token is shorter than a lexeme
Dlexemes are reserved words only
Answer & Solution
Correct answer: B. lexeme = substring; token adds type
1. LEXEME: the specific raw substring of source code that forms a token, e.g. 'while', '123', 'x', '+='.
2. TOKEN: a richer structure containing the lexeme PLUS:
- TYPE (e.g. KEYWORD_WHILE, NUMBER, IDENTIFIER, PLUS_EQUAL)
- LITERAL value (for numbers, strings — parsed into typed value)
- SOURCE LOCATION (line and column)
3. Example: '123' is the lexeme; the token is `(NUMBER, 123, line=4)`.
4. Options A, C, D mischaracterise.
_Source: Bob Nystrom, "Crafting Interpreters", Ch 4.2 (Lexemes and Tokens)._
Related questions
The FOLLOW(A) set in LL parsing isThe FIRST(α) set of a string α in LL parsing isAfter the parser produces an AST, the next compiler phase is typicallyA SHIFT/REDUCE CONFLICT in an LR parser occurs whenWhich of these is a BOTTOM-UP (LR) parser feature?An LL(1) grammar requires that, given the leftmost non-terminal and ONE token of lookaheadLL parsers (recursive descent) parse from thePANIC MODE error recovery in a parser works by