alleryStart learning →The DIFFERENCE between a TOKEN and a LEXEME is
Athey are synonyms
Blexeme = substring; token adds type
Ca token is shorter than a lexeme
Dlexemes are reserved words only
Answer & Solution
Correct answer: B. lexeme = substring; token adds type
1. LEXEME: the specific raw substring of source code that forms a token, e.g. 'while', '123', 'x', '+='.
2. TOKEN: a richer structure containing the lexeme PLUS:
- TYPE (e.g. KEYWORD_WHILE, NUMBER, IDENTIFIER, PLUS_EQUAL)
- LITERAL value (for numbers, strings — parsed into typed value)
- SOURCE LOCATION (line and column)
3. Example: '123' is the lexeme; the token is `(NUMBER, 123, line=4)`.
4. Options A, C, D mischaracterise.
_Source: Bob Nystrom, "Crafting Interpreters", Ch 4.2 (Lexemes and Tokens)._
Keep practising Compiler Design
Browse all Compiler Design questions for GATE CSE, or explore more Computer Science topics.
Related questions
The FOLLOW(A) set in LL parsing isThe FIRST(α) set of a string α in LL parsing isAfter the parser produces an AST, the next compiler phase is typicallyA SHIFT/REDUCE CONFLICT in an LR parser occurs whenLL parsers (recursive descent) parse from thePANIC MODE error recovery in a parser works byWhat is a SYNTAX ERROR?How does a parser typically encode OPERATOR PRECEDENCE in the grammar?