Description
Most modern programming languages are free format, which means that blanks and newlines appearing in their programs are ignored. A blank is the character ' ', also called a space. A newline is the character that marks the end of a line, and is written in OCaml as '\n'. Blanks, newlines, and other ignored characters are called whitespace. A language that ignores whitespace allows a program to be indented in a way that suggests its logical structure.
A lexical scanner is the part of a programming language that reads tokens tokens from a program file, ignoring whitespace. Each token is some meaningful part of a program, such as a constant, a name, a number, an operator, or a punctuation character. A lexical scanner for a subset of the programming Lisp has been discussed in recent lectures.
A comment is part of a program that provides explanatory text for a Human reader. Almost all modern programming languages ignore comments. For this laboratory assignment, you will modify the OCaml code for the Lisp lexical scanner so that it also ignores comments. The assignment is intended to give you some practice reading and modifying code that you did not write yourself.
1. Theory.
In the subset of Lisp that will be discussed in the lectures, a comment begins with a semicolon ';'. It ends with a newline, or the end of the file. For example, this Lisp program contains several comments.
; This Lisp function computes factorials.
(define factorial ; Its name is FACTORIAL.
(lambda (n) ; It has one parameter, N.
(if
(= n 0) ; If N is 0 ...
1 ; then return 1, else ...
(* n (factorial (- n 1)))))) ; return N times the FACTORIAL of N minus 1.
; That’s all, folks!
There are more comments here than any normal programmer would use, but having many comments makes a better example. The Lisp lexical scanner discussed in the lectures is able to read the tokens described in the following table.
|
TOKEN |
MEANING |
| CLOSEPARENTOKEN | A closing parenthesis ')'. |
| ENDTOKEN | The end of the file. |
| NUMBERTOKEN k | An integer constant k. |
| OPENPARENTOKEN | An opening parenthesis '('. |
| SYMBOLTOKEN n | A name n. |
If the lexical scanner was run on a file containing the Lisp program shown above, then it would read the following series of tokens, in this order.
OPENPARENTOKEN
SYMBOLTOKEN define
SYMBOLTOKEN factorial
OPENPARENTOKEN
SYMBOLTOKEN lambda
OPENPARENTOKEN
SYMBOLTOKEN n
CLOSEPARENTOKEN
OPENPARENTOKEN
SYMBOLTOKEN if
OPENPARENTOKEN
SYMBOLTOKEN =
SYMBOLTOKEN n
NUMBERTOKEN 0
CLOSEPARENTOKEN
NUMBERTOKEN 1
OPENPARENTOKEN
SYMBOLTOKEN *
SYMBOLTOKEN n
OPENPARENTOKEN
SYMBOLTOKEN factorial
OPENPARENTOKEN
SYMBOLTOKEN -
SYMBOLTOKEN n
NUMBERTOKEN 1
CLOSEPARENTOKEN
CLOSEPARENTOKEN
CLOSEPARENTOKEN
CLOSEPARENTOKEN
CLOSEPARENTOKEN
CLOSEPARENTOKEN
ENDTOKEN
Note that comments and whitespace are ignored by the lexical scanner.
2. Implementation.
The file scanner9.ml on Canvas contains source code for an OCaml module called Scanner. The module implements a lexical scanner for a subset of Lisp, as described in recent lectures. It has a type Scanner.token whose constructors make the tokens described in the previous section. It also has a higher-order function (Scanner.makeScanner path). This returns a function that acts as a lexical scanner for a file whose pathname is the string path. However, this function does not now ignore comments.
You must modify the code for Scanner.makeScanner so that it returns a lexical scanner that does ignore comments. This may involve writing one or more additional helpers for Scanner.makeScanner. It may also involve making changes to other helpers, etc. Hint: the scanner uses the zero-character '\000' to indicate the end of a file.
The file scanner9.ml also defines a function (test path) that creates a lexical scanner, runs it on a file whose pathname is path, and prints the tokens that the lexical scanner reads. It calls test on "test9.txt" , the pathname of a file that contains the Lisp program from the previous section. If your modified scanner works correctly, then test will print the same series of tokens shown in that section.
3. Deliverables.
Modify scanner9.ml as described above. When you think your code is correct, submit the modified file to Canvas. If you do not know how to submit your work, then please ask your lab TA’s. It must be submitted by 11:55 PM on Tuesday, November 23, 2021.
This laboratory assignment is worth 32 points, one point for each token correctly printed by the function test. To compute your grade, the TA’s will run the file that you submit and compare its output with the expected correct output. If your file does not run, producing no output, then you will receive 0 points for this assignment. DOUBLE–CHECK, BEFORE AND AFTER YOU SUBMIT YOUR FILE, THAT IT RUNS CORRECTLY!
