Description
1 Introduction
This assignment introduces you to Gaussian mixture modelling, recurrent neural networks with acoustic
data, and three basic tasks in speech technology: speaker identification, in which we try to determine who
is talking, speech recognition, in which we try to determine what was said, and basic sequence modelling.
The assignment is divided into two sections. In the first, you will experiment with acoustic-based
classification for 1) speaker identification, and 2) deception detection. In the second section, you will
evaluate two speech recognition engines.
The data come from the CSC Deceptive Speech corpus, which was developed by Columbia University, SRI International, and University of Colorado Boulder. It consists of 32 hours of audio interviews
from 32 native speakers of Standard American English (16 male, 16 female) recruited from the Columbia
University student population and the community. The purpose of the study was to distinguish deceptive
speech from non-deceptive speech using machine learning techniques on extracted features from the corpus.
Data are in /u/cs401/A3/data/; each sub-folder represents speech from one speaker and contains raw
audio, pre-computed MFCCs, and orthographic transcripts. Further file descriptions are in Appendix A.
2 Sequence classification: Speakers and lies [35 marks]
Speaker identification is the task of correctly identifying speaker sc from among S possible speakers si=1..S
given an input speech sequence X, consisting of a succession of d-dimensional real vectors. In the interests
of efficiency, d = 13 in this assignment. Each vector represents a small 25 ms unit of speech called a frame.
Speakers are identified by training data that are ascribed to them. This is a discrete classification task
(choosing among several speakers) that uses continuous-valued data (the vectors of real numbers) as input.
Gaussian Mixture Models
Gaussian mixture models are often used to generalize models from sparse data. They can tightly constrain
large-dimensional data by using a small number of components but can, with many more components,
model arbitrary density distributions. Sometimes, they are simply used because the domain being modelled
appears to have multiple modes.
Given M components, GMMs are modelled by a collection of parameters, θ = {ωm=1..M, µm=1..M, Σm=1..M},
where ωm is the probability that an observation is generated by the mth component. These are subject
to the constraint that P
m ωm = 1 and 0 ≤ ωm ≤ 1. Each component is a multivariate Gaussian distribution, which is characterized by that component’s mean, µm, and covariance matrix, Σm. For reasons
0Copyright © 2022, University of Toronto. All rights reserved.
1
of computational efficiency, we will reintroduce some independence assumptions by assuming that every
component’s covariance matrix is diagonal, i.e.:
Σm =
Σm[1] 0 · · · 0
0 Σm[2] · · · 0
.
.
.
0 0 · · · Σm[d]
for some vector Σ⃗ m. Therefore, only d parameters are necessary to characterize a component’s (co)variance.
2.1 Utility functions [10 marks]
We start with implementing three utility functions in /u/cs401/A3/code/a3 gmm.py. For your convenience, there is also another starter code, a3 gmm structured.py, with slightly different structures. You
can start with either one.
First, implement log b m x, which implements the log observation probability of xt for the mth mixture
component, i.e., the log of:
bm (⃗xt) =
exp ”
−
1
2
X
d
n=1
(xt
[n] − µm[n])2
Σm[n]
#
(2π)
d/2
qQd
n=1 Σm[n]
(1)
Next, implement log p m x, which is the log probability of m given xt using model θ, i.e., the log of:
p (m|⃗xt
; θ) = ωmbm (⃗xt)
PM
k=1 ωkbk (⃗xt)
(2)
Finally, implement logLik, which is the log likelihood of a set of data X, i.e.:
log P
X˜; θs
=
X
T
t=1
log p (⃗xt
; θs) (3)
where
p (⃗xt
; θ) = X
M
m=1
ωmbm(⃗xt) (4)
and bm is defined in Equation 1. For efficiency, we just pass θ and precomputed bm(⃗xt) to this function.
2.2 Training Gaussian mixture models [5 marks]
Now we train an M-component GMM for each of the speakers in the data set. Specifically, for each speaker
s, train the parameters θs = {ωm=1..M, µm=1..M, Σm=1..M} according to the method described in Appendix B. In all cases, assume that covariance matrices Σm are diagonal. Start with M = 8. You’ll be asked
to experiment with that in Section 2.4. Complete the function train in /u/cs401/A3/code/a3 gmm.py.
2
2.3 Classification with Gaussian mixture models [5 marks]
Now we test each of the test sequences we’ve already set aside for you in the main function. I.e., we check
if the actual speaker is also the most likely speaker, ˆs:
sˆ = argmax
s=1,…,S
log P
X˜; θs
(5)
Complete the function test in /u/cs401/A3/code/a3 gmm.py. Run through a train-test cycle, and save
the output that this function writes to stdout, using the k = 5 top alternatives, to the file gmmLiks.txt.
2.4 Experiments and discussion [10 marks]
Experiment with the settings of M and maxIter (or ϵ if you wish). For example, what happens to
classification accuracy as the number of components decreases? What about when the number of possible
speakers, S, decreases? You will be marked on the detail with which you empirically answer these questions
and whether you can devise one or more additional valid experiments of this type.
Additionally, your report should include short hypothetical answers to the following questions:
• How might you improve the classification accuracy of the Gaussian mixtures, without adding more
training data?
• When would your classifier decide that a given test utterance comes from none of the trained speaker
models, and how would your classifier come to this decision?
• Can you think of some alternative methods for doing speaker identification that don’t use Gaussian
mixtures?
Put your experimental analysis and answers to these questions in the file gmmDiscussion.txt.
2.5 End-to-end truth-and-lie detection with GRUs [5 marks]
Each of the utterances has been labelled as either truthful or deceitful (see Appendix A). Your task is to
train and test models to tell these utterances apart using the provided data.
• Using the acoustic sequences from the previous GMM questions as input, use torch.nn.GRU to create
a simple unidirectional GRU with one hidden layer. This GRU takes in MFCC vectors as inputs,
and output a single output (truth or false). The starter code labels lies as 1 and truth as 0.
• For this part, modify the __init__ method of the LieDetector in model.py to create a GRU as
specified above. In addition, fill in the code to create a linear layer to project the GRU’s outputs to
logits.
• Experiment by training the model using different hidden sizes of 5, 10, and 50.
• To train, run:
python train.py –source
If you wish, you can experiment with the hyperparameters by specifying –batch_size, –lr,
–epochs or –optimizer (either ‘adam’ or ‘sgd’; by default, it is ‘adam’).
Is there a trend in performance with different hidden sizes? Explain this trend or lack thereof in 1-2
sentences. Write your model configurations, the detection performance (accuracy) and answers to the
discussion questions in the file detectionDiscussion.txt.
3
3 Automatic Speech Recognition [15 marks]
Automatic speech recognition (ASR) is the task of correctly identifying a word sequence given an input
speech sequence X. To simplify your lives, we have ran two popular ASR engines on our data: the opensource and highly customizable Kaldi (specifically, a bi-directional LSTM model trained on the Fisher
corpus), and the neither-open-source-nor-particularly-customizable Google Speech API.
3.1 Evaluation with WER [10 marks]
We want to see which of Kaldi and Google are the most accurate on our data. For each speaker in
our data, we have three transcript files: transcripts.txt (the gold-standard transcripts, from humans),
transcripts.Kaldi.txt (the ASR output of Kaldi), and transcripts.Google.txt (the ASR output of
Google); see Appendix A.
Complete the file at /u/cs401/A3/code/a3 levenshtein.py. Specifically, in the Levenshtein function, accept lists of words r (reference) and h (hypothesis), and return a 4-item list containing the floatingpoint WER, the number of substitution errors, the number of insertion errors, and the number of deletion
errors.
W ER =
numSubstitutions + numInsertions + numDeletions
numReferenceWords
Assume that the cost of a substitution is 0 if the words are identical and 1 otherwise. The costs of insertion
and deletion are both 1.
Note: During implementation, if multiple possibilities are possible, you can “break the tie” by arbitrary
orders. The autograder accepts different answers. An example is r=“recognize speech”. The hypothesis
h=“not recognize” can have either of {numSubstitutions=2} or {numInsertions=1, numDeletion=1}.
In the main function, iterate through each of the speakers, and iterate through each line i of their
transcripts. For each line, preprocess these transcripts by removing all punctuation, setting the text to
lowercase, and any tags such as [laughter], [noise],
etc. Output the following to stdout:
[SPEAKER] [SYSTEM] [i] [WER] S:[numSubstitutions], I:[numInsertions], D:[numDeletions]
where [SYSTEM] is either ‘Kaldi’ or ‘Google’.
Save this output and put it into asrDiscussion.txt.
On the second-to-last line of asrDiscussion.txt, in free text, summarize your findings by reporting
the average and standard deviation of WER for each of Kaldi and Google, separately, over all of these
lines. If you want to be fancy, you can compute a statistical test of significance to see if one is better than
the other, but you don’t need to.
On the last line of asrDiscussion.txt, add a sentence or two describing anything you observe about
the types of errors being made by each system, by manually examining the transcript files.
3.2 Hands-on ASR pipeline [5 marks]
The goal of this question is to get familiar with the ASR pipeline end-to-end (from voice to text). There
are several steps to building this pipeline.
Step 1. Prepare 3 transcripts of different lengths. Read out the transcripts, record the sound files yourself.
Here are some possible examples:
• One word (around 1 second)
• One sentence (around 5 seconds)
• A sentence, but read very slowly (around 10 seconds)
Step 2. Find an ASR system of your choice. There are many systems that come with pre-trained models:
4
• Kaldi – the most popular system for ASR. There are many small (i.e., computationally-efficient) pretrained models with good WERs. Kaldi also provides many bash scripts (“recipes”) and tutorials to
train and use their pre-trained models.
• PyKaldi – a Python wrapper for Kaldi.
• SpeechBrain – a Python package as a Kaldi replacement. You don’t need to play around with the
bash scripts (except for the pip install speechbrain). Note that many SpeechBrain models are
large. Transcribing with their models can take multiple minutes.
• Online ASR transcription services like AWS, Google, or Microsoft. Some of them only have limitations
regarding the free acoustic data that you can transcribe. After that, users will need to pay for the
transcription. Note that if you choose to use online transcription services, we can not support any
associated cost.
Step 3. Transcribe the three sound files using the ASR system of your choice. Remember to delete the
sound files after you finish this problem.
In asrPipeline.txt, briefly summarize the transcripts, your procedure to collect the sounds, and the
ASR system you use. Comment on the performance of the speech recognition.
5
4 Bonus [up to 5 marks]
We will give up to 5 bonus marks for innovative work going substantially beyond the minimal requirements.
These marks can make up for marks lost in other sections of the assignment, but your overall mark for
this assignment cannot exceed 100%. You may decide to pursue any number of tasks of your own design
related to this assignment, although you should consult with the instructor or the TA before embarking
on such exploration. Certainly, the rest of the assignment takes higher priority. Some ideas:
4.1 Improving the truth-lie detection algorithm [5 marks]
Make the models better than the GMM or GRU would be a nice improvement. Some ideas include:
• Balance the training samples in different classes by, e.g., subsampling or giving different weights to
some data samples.
• In addition to using only the MFCC features, you can also extract engineered features, such as those
extracted in Assignment 1, from the text transcripts and classify using discriminative models in
scikit-learn. Are words more discriminative than the audio?
• Consider how errors in transcripts affect those extracted features and therefore overall system accuracy. See Zhou L, Fraser KC, Rudzicz F. (2016) Speech recognition in Alzheimer’s disease and in its
assessment. In Proceedings of the Annual Conference of the International Speech Communication
Association, INTERSPEECH.
Write the discussions to the file bonusDetectionDiscussion.txt.
4.2 Dimensionality reduction [5 marks]
Principal components analysis (PCA) is a method that converts some multivariate representation of data
into a lower-dimensional representation by transforming the original data according to mutually orthogonal
principal components.
Implement an algorithm that discovers a d×d
′ matrix W that transforms a d-dimensional vector, ⃗x into
a d
′
-dimensional vector ⃗y through a linear transformation based on PCA, where d
′ < d. Repeat speaker
identification using data that has been transformed by PCA and report on what you observe, e.g., for
different values of d
′
.
Write the findings in the file bonusPCA.txt. Submit all code and materials necessary to repeat your
experiments.
4.3 ASR with sequence-to-sequence models [5 marks]
Try to do better than Kaldi or Google by implementing, for example:
Chiu C-C, Sainath TN, Wu Y, et al. (2017) State-of-the-art Speech Recognition With Sequence-toSequence Models. http://arxiv.org/abs/1712.01769.
Consider using open-source end-to-end ASR using PyTorch or TensorFlow, e.g., deepSpeech.
Write your procedure and the results in the file bonusBetterASR.txt. Submit all code and materials
necessary to repeat your experiments.
6
5 General specification
We may test your code on different training and testing data in addition to those specified above. Where
possible, do not hardwire directory names into your code. As part of grading your assignment, the grader
may run your programs using test harness Python scripts. It is therefore important that your code precisely
meets the specifications and formatting requirements, including arguments and file names.
If your code uses a file or helper script, it must read it either from the directory in which that code is
being executed (i.e., probably ‘pwd’), or it must read it from a subdirectory of /u/cs401 whose absolute
path is completely specified in the code. Do not hardwire the absolute address of files in your home
directory; the grader does not have access to that directory.
All your programs must contain adequate internal documentation to be clear to the graders. External
documentation is not required.
This assignment is in Python 3.
5.1 Submission requirements
This assignment is submitted electronically. Submit your assignment on MarkUs. You should submit:
1. All your code for a3 gmm.py, a3 gmm structured.py, model.py (from Sec 2.5), and a3 levenshtein.py
(including helper files, if any).
2. The output file gmmLiks.txt
3. Your discussion files gmmDiscussion.txt, asrDiscussion.txt, asrPipeline.txt, and
detectionDiscussion.txt
4. The ID file available from the course web-site.
Do not tar or compress your files, and do not place your files in subdirectories.
5.2 Academic integrity
This is your last assignment of this semester. In the past years, some students posted their implementations
to their own GitHub repositories after the semesters. We have requested that they change the visibility to
private. Please do NOT publicize your solutions. Copying the solutions of other students violates
the academic integrity – your codes should be the results of your honest efforts. The questions change
from year to year anyways.
6 Using your own computer
If you want to do some or all of this assignment on your laptop or other computer, you will have to do
the extra work of downloading and installing the requisite software and data. You take on the risk that
your computer might not be adequate for the task. You are strongly advised to upload regular backups
of your work to teach.cs, so that if your machine fails or proves to be inadequate, you can immediately
continue working on the assignment at teach.cs. When you have completed the assignment, you should
try your programs out on teach.cs to make sure that they run correctly there. A submission that does
not work on teach.cs will get zero marks. Question 3.2 is an exception – it is more convenient that
you use a system with admin access.
7
A Appendix: Details on CSC data set
Each utterance is represented by the following file types:
*.wav The original speech waveform sampled at 16kHz.
*.mfcc.py The Mel-frequency cepstral coefficients obtained from an analysis of the waveform,
in numPy format. Each row represents a 25ms frame of speech and consists of 13
floating point values.
*.txt Label and orthographic transcription of each utterance, for each of Kaldi and Google
ASR, and human gold-standard.
Participants were told that they were participating in a communication experiment which sought to
identify people who fit the profile of top entrepreneurs in America. To this end, participants performed
tasks and answered questions in six areas; they were later told that they had received low scores in some
of those areas and did not fit the profile. The subjects then participated in an interview where they were
told to convince the interviewer that they had actually achieved high scores in all areas and that they did
indeed fit the profile. The interviewer’s task was to determine how he thought the subjects had actually
performed, and he was allowed to ask them any questions other than those that were part of the subjects’
tasks. For each question from the interviewer, subjects were asked to indicate whether the reply was true
or contained any false information by pressing one of two pedals hidden from the interviewer under a table.
Interviews were conducted in a double-walled sound booth and recorded to digital audio tape on
two channels using Crown CM311A Differoid headworn close-talking microphones, then downsampled
to 16 kHz. Interviews were orthographically transcribed by hand using the NIST EARS transcription
guidelines. Labels for local lies were obtained automatically from the pedal-press data and hand-corrected
for alignment, and labels for global lies were annotated during transcription based on the known scores of
the subjects versus their reported scores.
MFCCs were obtained using the python speech features module using default parameters, i.e., 25
ms windows, 13 cepstral coefficients, and 512 fast Fourier transform coefficients
Each transcript file has the same format, where the i
th line is:
[i] [LABEL] [TRANSCRIPT]
where i corresponds to i.wav and i.mfcc.npy, [LABEL] is the Global Lie label, and [TRANSCRIPT]
is the actual transcript orthography. Global Lie valence and the version of the pre-interview task for
the utterance appears before the colon (e.g., T/H) and the section name appears after the colon (e.g.,
INTERACTIVE).
Global Lie valence is indicated as: T == Truth; LU == Lie Up (subject claims better performance
than was actually achieved); and LD == Lie Down (subject claims worse performance). The task version
is indicated as: H == Hard; and E == Easy. So, for example, T/H:INTERACTIVE indicates that the
subject is telling the truth based on having performed the hard version of the Interactive task.
8
B Appendix: Training Gaussian mixture models
Input: MFCC data X, number of components M, threshold ϵ, and maxIter
begin
Initialize θ ;
i := 0 ;
prev L := −∞ ; improvement = ∞;
while i =< maxIter and improvement >= ϵ do
ComputeIntermediateResults ;
L := ComputeLikelihood (X, θ) ;
θ := UpdateParameters (θ, X, L) ;
improvement := L − prev L ;
prev L := L ;
i := i + 1 ;
end
end
Algorithm 1: GMM training algorithm.
For ComputeIntermediateResults, it is strongly recommended that you create two M × T numPy
arrays – one to store each value from Equation 1 and the other to store each value from Equation 2. In
fact, we’ve set up the function logLik to encourage you to do this, to avoid redundant computations. You
will use these values in both ComputeLikelihood and updateParameters, where the latter is accomplished
thus:
ωˆm =
PT
t=1 p (m|⃗xt
; θ)
T
ˆ⃗µm =
PT
t=1 p (m|⃗xt
; θ) ⃗xt
PT
t=1 p (m|⃗xt
; θ)
Σˆm =
PT
t=1 p (m|⃗xt
; θ) ⃗x2
t
PT
t=1 p (m|⃗xt
; θ)
− ˆ⃗µ
2
m
(6)
In the third equation, the square of a vector on the right-hand side is defined as the component-wise square
of each dimension in the vector. Note that you don’t need to break up Algorithm 1 into separate functions
as implied – it is only written that way above to emphasize the sequence of steps
9
