Assignment 4: Word Embeddings¶

Welcome to the fourth (and last) programming assignment of Course 2!

In this assignment, you will practice how to compute word embeddings and use them for sentiment analysis.

• To implement sentiment analysis, you can go beyond counting the number of positive words and negative words.
• You can find a way to represent each word numerically, by a vector.
• The vector could then represent syntactic (i.e. parts of speech) and semantic (i.e. meaning) structures.

In this assignment, you will explore a classic way of generating word embeddings or representations.

• You will implement a famous model called the continuous bag of words (CBOW) model.

By completing this assignment you will:

• Train word vectors from scratch.
• Learn how to create batches of data.
• Understand how backpropagation works.
• Plot and visualize your learned word vectors.

Knowing how to train these models will give you a better understanding of word vectors, which are building blocks to many applications in natural language processing.

Important Note on Submission to the AutoGrader¶

Before submitting your assignment to the AutoGrader, please make sure you are not doing the following:

1. You have not added any extra print statement(s) in the assignment.
2. You have not added any extra code cell(s) in the assignment.
3. You have not changed any of the function parameters.
4. You are not using any global variables inside your graded exercises. Unless specifically instructed to do so, please refrain from it and use the local variables instead.
5. You are not changing the assignment code where it is not required, like creating extra variables.

If you do any of the following, you will get something like, Grader Error: Grader feedback not found (or similarly unexpected) error upon submitting your assignment. Before asking for help/debugging the errors in your assignment, check for these first. If this is the case, and you don't remember the changes you have made, you can get a fresh copy of the assignment by following these instructions.

Table of Contents¶

• 1 - The Continuous Bag of Words Model
• 2 - Training the Model
  • 2.1 - Initializing the Model
    • Exercise 1 - initialize_model (UNQ_C1)
  • 2.2 - Softmax
    • Exercise 2 - softmax (UNQ_C2)
  • 2.3 - Forward Propagation
    • Exercise 3 - forward_prop (UNQ_C3)
  • 2.4 - Cost Function
  • 2.5 - Training the Model - Backpropagation
    • Exercise 4 - back_prop (UNQ_C4)
  • 2.6 - Gradient Descent
    • Exercise 5 - gradient_descent (UNQ_C5)
• 3 - Visualizing the Word Vectors

1 - The Continuous Bag of Words Model¶

Let's take a look at the following sentence:

'I am happy because I am learning'.

• In continuous bag of words (CBOW) modeling, we try to predict the center word given a few context words (the words around the center word).
• For example, if you were to choose a context half-size of say $C = 2$, then you would try to predict the word happy given the context that includes 2 words before and 2 words after the center word:

$C$ words before: [I, am]

$C$ words after: [because, I]

• In other words:

$$context = [I,am, because, I]$$ $$target = happy$$

The structure of your model will look like this:

Figure 1

Where $\bar x$ is the average of all the one hot vectors of the context words.

Figure 2

Once you have encoded all the context words, you can use $\bar x$ as the input to your model.

The architecture you will be implementing is as follows:

\begin{align} h &= W_1 \ X + b_1 \tag{1} \\ a &= ReLU(h) \tag{2} \\ z &= W_2 \ a + b_2 \tag{3} \\ \hat y &= softmax(z) \tag{4} \\ \end{align}

Mapping words to indices and indices to words¶

We provide a helper function to create a dictionary that maps words to indices and indices to words.

2 - Training the Model¶

2.1 - Initializing the Model¶

You will now initialize two matrices and two vectors.

• The first matrix ($W_1$) is of dimension $N \times V$, where $V$ is the number of words in your vocabulary and $N$ is the dimension of your word vector.
• The second matrix ($W_2$) is of dimension $V \times N$.
• Vector $b_1$ has dimensions $N\times 1$
• Vector $b_2$ has dimensions $V\times 1$.
• $b_1$ and $b_2$ are the bias vectors of the linear layers from matrices $W_1$ and $W_2$.

The overall structure of the model will look as in Figure 1, but at this stage we are just initializing the parameters.

Exercise 1 - initialize_model¶

Please use numpy.random.rand to generate matrices that are initialized with random values from a uniform distribution, ranging between 0 and 1.

Note: In the next cell you will encounter a random seed. Please DO NOT modify this seed so your solution can be tested correctly.

Expected Output¶

tmp_W1.shape: (4, 10)
tmp_W2.shape: (10, 4)
tmp_b1.shape: (4, 1)
tmp_b2.shape: (10, 1)

2.2 - Softmax¶

Before we can start training the model, we need to implement the softmax function as defined in equation 5:

• Array indexing in code starts at 0.
• $V$ is the number of words in the vocabulary (which is also the number of rows of $z$).
• $i$ goes from 0 to |V| - 1.

Exercise 2 - softmax¶

Instructions: Implement the softmax function below.

• Assume that the input $z$ to softmax is a 2D array
• Each training example is represented by a vector of shape (V, 1) in this 2D array.
• There may be more than one column, in the 2D array, because you can put in a batch of examples to increase efficiency. Let's call the batch size lowercase $m$, so the $z$ array has shape (V, m)
• When taking the sum from $i=1 \cdots V-1$, take the sum for each column (each example) separately.

Please use

• numpy.exp
• numpy.sum (set the axis so that you take the sum of each column in z)

Expected Ouput¶

array([[0.5       , 0.73105858, 0.88079708],
       [0.5       , 0.26894142, 0.11920292]])

2.3 - Forward Propagation¶

Exercise 3 - forward_prop¶

Implement the forward propagation $z$ according to equations (1) to (3).

\begin{align} h &= W_1 \ X + b_1 \tag{1} \\ h &= ReLU(h) \tag{2} \\ z &= W_2 \ h + b_2 \tag{3} \\ \end{align}

For that, you will use as activation the Rectified Linear Unit (ReLU) given by:

$$f(h)=\max (0,h) \tag{6}$$

Expected output¶

x has shape (3, 1)
N is 2 and vocabulary size V is 3
call forward_prop

z has shape (3, 1)
z has values:
[[0.55379268]
 [1.58960774]
 [1.50722933]]

h has shape (2, 1)
h has values:
[[0.92477674]
 [1.02487333]]

2.4 - Cost Function¶

• We have implemented the cross-entropy cost function for you.

Expected output¶

tmp_x.shape (5778, 4)
tmp_y.shape (5778, 4)
tmp_W1.shape (50, 5778)
tmp_W2.shape (5778, 50)
tmp_b1.shape (50, 1)
tmp_b2.shape (5778, 1)
tmp_z.shape: (5778, 4)
tmp_h.shape: (50, 4)
tmp_yhat.shape: (5778, 4)
call compute_cost
tmp_cost 8.9542

2.5 - Training the Model - Backpropagation¶

Exercise 4 - back_prop¶

Now that you have understood how the CBOW model works, you will train it.
You created a function for the forward propagation. Now you will implement a function that computes the gradients to backpropagate the errors.

Note: z1 is calculated as W1⋅x + b1 in this function. In practice, you would save it already when making forward propagation and just re-use here, but for simplicity, it is calculated again in back_prop.

As reference, below are the equations of backpropagation as taught in the lecture:

Expected output¶

get a batch of data
tmp_x.shape (5778, 4)
tmp_y.shape (5778, 4)

Initialize weights and biases
tmp_W1.shape (50, 5778)
tmp_W2.shape (5778, 50)
tmp_b1.shape (50, 1)
tmp_b2.shape (5778, 1)

Forwad prop to get z and h
tmp_z.shape: (5778, 4)
tmp_h.shape: (50, 4)

Get yhat by calling softmax
tmp_yhat.shape: (5778, 4)

call back_prop
tmp_grad_W1.shape (50, 5778)
tmp_grad_W2.shape (5778, 50)
tmp_grad_b1.shape (50, 1)
tmp_grad_b2.shape (5778, 1)

2.6 - Gradient Descent¶

Exercise 5 - gradient_descent¶

Now that you have implemented a function to compute the gradients, you will implement batch gradient descent over your training set.

Hint: For that, you will use initialize_model and the back_prop functions which you just created (and the compute_cost function). You can also use the provided get_batches helper function:

for x, y in get_batches(data, word2Ind, V, C, batch_size):

...

Also: print the cost after each batch is processed (use batch size = 128)

Expected Output¶

iters: 10 cost: 9.686791
iters: 20 cost: 10.297529
iters: 30 cost: 10.051127
iters: 40 cost: 9.685962
iters: 50 cost: 9.369307
iters: 60 cost: 9.400293
iters: 70 cost: 9.060542
iters: 80 cost: 9.054266
iters: 90 cost: 8.765818
iters: 100 cost: 8.516531
iters: 110 cost: 8.708745
iters: 120 cost: 8.660616
iters: 130 cost: 8.544338
iters: 140 cost: 8.454268
iters: 150 cost: 8.475693

Your numbers may differ a bit depending on which version of Python you're using.

3 - Visualizing the Word Vectors¶

In this part you will visualize the word vectors trained using the function you just coded above.

You can see that man and king are next to each other. However, we have to be careful with the interpretation of this projected word vectors, since the PCA depends on the projection -- as shown in the following illustration.