Assignment 1: Deep N-grams¶

Welcome to the first graded assignment of course 3. In this assignment you will explore Recurrent Neural Networks RNN.

In this notebook you will apply the following steps:

• Convert a line of text into a tensor
• Create a tensorflow dataset
• Define a GRU model using TensorFlow
• Train the model using TensorFlow
• Compute the accuracy of your model using the perplexity
• Generate text using your own model

Before getting started take some time to read the following tips:

TIPS FOR SUCCESSFUL GRADING OF YOUR ASSIGNMENT:¶

• All cells are frozen except for the ones where you need to submit your solutions.

• You can add new cells to experiment but these will be omitted by the grader, so don't rely on newly created cells to host your solution code, use the provided places for this.

• You can add the comment # grade-up-to-here in any graded cell to signal the grader that it must only evaluate up to that point. This is helpful if you want to check if you are on the right track even if you are not done with the whole assignment. Be sure to remember to delete the comment afterwards!

• To submit your notebook, save it and then click on the blue submit button at the beginning of the page.

Table of Contents¶

• Overview
• 1 - Data Preprocessing Overview
  • 1.1 - Loading in the Data
  • 1.2 - Create the vocabulary
  • 1.3 - Convert a Line to Tensor
    • Exercise 1 - line_to_tensor
  • 1.4 - Prepare your data for training and testing
  • 1.5 - Tensorflow dataset
  • 1.6 - Create the input and the output for your model
    • Exercise 2 - create_batch_dataset
  • 1.7 - Create the training dataset
• 2 - Defining the GRU Language Model (GRULM)
  • Exercise 3 - GRULM
• 3 - Training
  • Exercise 4 - compile_model
• 4 - Evaluation
  • 4.1 - Evaluating using the Deep Nets
  • Exercise 5 - log_perplexity
• 5 - Generating Language with your Own Model
  • Optional Exercise 6 - GenerativeModel (Not graded)
• On statistical methods

Overview¶

In this lab, you'll delve into the world of text generation using Recurrent Neural Networks (RNNs). Your primary objective is to predict the next set of characters based on the preceding ones. This seemingly straightforward task holds immense practicality in applications like predictive text and creative writing.

The journey unfolds as follows:

• Data Preprocessing: You'll start by converting lines of text into numerical tensors, making them machine-readable.

• Dataset Creation: Next, you'll create a TensorFlow dataset, which will serve as the backbone for supplying data to your model.

• Neural Network Training: Your model will be trained to predict the next set of characters, specifying the desired output length.

• Character Embeddings: Character embeddings will be employed to represent each character as a vector, a fundamental technique in natural language processing.

• GRU Model: Your model utilizes a Gated Recurrent Unit (GRU) to process character embeddings and make sequential predictions. The following figure gives you a summary of what you are about to implement.

• Prediction Process: The model's predictions are achieved through a linear layer and log-softmax computation.

This overview sets the stage for your exploration of text generation. Get ready to unravel the secrets of language and embark on a journey into the realm of creative writing and predictive text generation.

And as usual let's start by importing all the required libraries.

1 - Data Preprocessing Overview¶

In this section, you will prepare the data for training your model. The data preparation involves the following steps:

• Dataset Import: Begin by importing the dataset. Each sentence is structured as one line in the dataset. To ensure consistency, remove any extra spaces from these lines using the strip function.

• Data Storage: Store each cleaned line in a list. This list will serve as the foundational dataset for your text generation task.

• Character-Level Processing: Since the goal is character generation, it's essential to process the text at the character level, not the word level. This involves converting each individual character into a numerical representation. To achieve this:

  • Use the tf.strings.unicode_split function to split each sentence into its constituent characters.
  • Utilize tf.keras.layers.StringLookup to map these characters to integer values. This transformation lays the foundation for character-based modeling.

• TensorFlow Dataset Creation: Create a TensorFlow dataset capable of producing data in batches. Each batch will consist of batch_size sentences, with each sentence containing a maximum of max_length characters. This organized dataset is essential for training your character generation model.

These preprocessing steps ensure that your dataset is meticulously prepared for the character-based text generation task, allowing you to work seamlessly with the Shakespearean corpus data.

1.1 - Loading in the Data¶

Let's examine a few lines from the corpus. Pay close attention to the structure and style employed by Shakespeare in this excerpt. Observe that character names are written in uppercase, and each line commences with a capital letter. Your task in this exercise is to construct a generative model capable of emulating this particular structural style.

1.2 - Create the vocabulary¶

In the following code cell, you will create the vocabulary for text processing. The vocabulary is a crucial component for understanding and processing text data. Here's what the code does:

• Concatenate all the lines in our dataset into a single continuous text, separated by line breaks.

• Identify and collect the unique characters that make up the text. This forms the basis of our vocabulary.

• To enhance the vocabulary, introduce two special characters:

  • [UNK]: This character represents any unknown or unrecognized characters in the text.
  • "" (empty character): This character is used for padding sequences when necessary.

• The code concludes with the display of statistics, showing the total count of unique characters in the vocabulary and providing a visual representation of the complete character set.

1.3 - Convert a Line to Tensor¶

Now that you have your list of lines, you will convert each character in that list to a number using the order given by your vocabulary. You can use tf.strings.unicode_split to split the text into characters.

Using your vocabulary, you can convert the characters given by unicode_split into numbers. The number will be the index of the character in the given vocabulary.

Tensorflow has a function tf.keras.layers.StringLookup that does this efficiently for list of characters. Note that the output object is of type tf.Tensor. Here is the result of applying the StringLookup function to the characters of "Hello world"

Exercise 1 - line_to_tensor¶

Instructions: Write a function that takes in a single line and transforms each character into its unicode integer. This returns a list of integers, which we'll refer to as a tensor.

Expected output

Result: [55 56 57  4 78 79 80]
Output type: <class 'tensorflow.python.framework.ops.EagerTensor'>

You will also need a function that produces text given a numeric tensor. This function will be useful for inspection when you use your model to generate new text, because you will be able to see words rather than lists of numbers. The function will use the inverse Lookup function tf.keras.layers.StringLookup with invert=True in its parameters.

Use the function for decoding the tensor produced by "Hello world!"

1.4 - Prepare your data for training and testing¶

As usual, you will need some data for training your model, and some data for testing its performance. So, we will use 124097 lines for training and 1000 lines for testing.

1.5 - TensorFlow dataset¶

Most of the time in Natural Language Processing, and AI in general you use batches when training your models. Here, you will build a dataset that takes in some text and returns a batch of text fragments (Not necesarly full sentences) that you will use for training.

• The generator will produce text fragments encoded as numeric tensors of a desired length

Once you create the dataset, you can iterate on it like this:

data_generator.take(1)

This generator returns the data in a format that you could directly use in your model when computing the feed-forward of your algorithm. This batch dataset generator returns batches of data in an endless way.

So, let's check how the different parts work with a corpus composed of 2 lines. Then, you will use these parts to create the first graded function of this notebook.

In order to get a dataset generator that produces batches of fragments from the corpus, you first need to convert the whole text into a single line, and then transform it into a single big tensor. This is only possible if your data fits completely into memory, but that is the case here.

Create a dataset out of a tensor like input. This initial dataset will dispatch numbers in packages of a specified length. For example, you can use it for getting the 10 first encoded characters of your dataset. To make it easier to read, we can use the text_from_ids function.

But we can configure this dataset to produce batches of the same size each time. We could use this functionality to produce text fragments of a desired size (seq_length + 1). We will explain later why you need an extra character into the sequence.

You can verify that the data generator produces encoded fragments of text of the desired length. For example, let's ask the generator to produce 2 batches of data using the function data_generator.take(2)

But as usual, it is easier to understand if you print it in human readable characters using the 'text_from_ids' function.

1.6 - Create the input and the output for your model¶

In this task you have to predict the next character in a sequence. The following function creates 2 tensors, each with a length of seq_length out of the input sequence of lenght seq_length + 1. The first one contains the first seq_length elements and the second one contains the last seq_length elements. For example, if you split the sequence ['H', 'e', 'l', 'l', 'o'], you will obtain the sequences ['H', 'e', 'l', 'l'] and ['e', 'l', 'l', 'o'].

Look the result using the following sequence of characters

The first sequence will be the input and the second sequence will be the expected output

Now, put all this together into a function to create your batch dataset generator

Exercise 2 - create_batch_dataset¶

Instructions: Create a batch dataset from the input text. Here are some things you will need.

• Join all the input lines into a single string. When you have a big dataset, you would better use a flow from directory or any other kind of generator.
• Transform your input text into numeric tensors
• Create a TensorFlow DataSet from your numeric tensors: Just feed the numeric tensors into the function tf.data.Dataset.from_tensor_slices
• Make the dataset produce batches of data that will form a single sample each time. This is, make the dataset produce a sequence of seq_length + 1, rather than single numbers at each time. You can do it using the batch function of the already created dataset. You must specify the length of the produced sequences (seq_length + 1). So, the sequence length produced by the dataset will seq_length + 1. It must have that extra element since you will get the input and the output sequences out of the same element. drop_remainder=True will drop the sequences that do not have the required length. This could happen each time that the dataset reaches the end of the input sequence.
• Use the split_input_target to split each element produced by the dataset into the mentioned input and output sequences.The input will have the first seq_length elements, and the output will have the last seq_length. So, after this step, the dataset generator will produce batches of pairs (input, output) sequences.
• Create the final dataset, using dataset_xy as the starting point. You will configure this dataset to shuffle the data during the generation of the data with the specified BUFFER_SIZE. For performance reasons, you would like that tensorflow pre-process the data in parallel with training. That is called prefetching, and it will be configured for you.

Expected output

Prints the elements into a single batch. The batch contains 2 elements: 

Input0	: b'and sight distra'

Target0	: b'nd sight distrac'


Input1	: b'when in his fair'

Target1	: b'hen in his fair '

1.7 - Create the training dataset¶

Now, you can generate your training dataset using the functions defined above. This will produce pairs of input/output tensors each time the batch generator creates an entry.

2 - Defining the GRU Language Model (GRULM)¶

Now that you have the input and output tensors, you will go ahead and initialize your model. You will be implementing the GRULM, gated recurrent unit model. To implement this model, you will be using TensorFlow. Instead of implementing the GRU from scratch (you saw this already in a lab), you will use the necessary methods from a built-in package. You can use the following packages when constructing the model:

• tf.keras.layers.Embedding: Initializes the embedding. In this case it is the size of the vocabulary by the dimension of the model. docs
  • Embedding(vocab_size, embedding_dim).
  • vocab_size is the number of unique words in the given vocabulary.
  • embedding_dim is the number of elements in the word embedding (some choices for a word embedding size range from 150 to 300, for example).

• tf.keras.layers.GRU: TensorFlow GRU layer. docs) Builds a traditional GRU of rnn_units with dense internal transformations. You can read the paper here: https://arxiv.org/abs/1412.3555
  • units: Number of recurrent units in the layer. It must be set to rnn_units
  • return_sequences: It specifies if the model returns a sequence of predictions. Set it to True
  • return_state: It specifies if the model must return the last internal state along with the prediction. Set it to True

• tf.keras.layers.Dense: A dense layer. docs. You must set the following parameters:
  • units: Number of units in the layer. It must be set to vocab_size
  • activation: It must be set to log_softmax function as described in the next line.

• tf.nn.log_softmax: Log of the output probabilities. docs
  • You don't need to set any parameters, just set the activation parameter as activation=tf.nn.log_softmax.

Exercise 3 - GRULM¶

Instructions: Implement the GRULM class below. You should be using all the methods explained above.

Now, you can define a new GRULM model. You must set the vocab_size to 82; the size of the embedding embedding_dim to 256, and the number of units that will have you recurrent neural network rnn_units to 512

Expected output¶

Model: "grulm"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 embedding (Embedding)    (None, 100, 256)          20992     
                                                                 
 gru (GRU)                [(None, 100, 512),        1182720   
                              (None, 512)]                       
                                                                 
 dense (Dense)            (None, 100, 82)           42066     
                                                                 
=================================================================
Total params: 1245778 (4.75 MB)
Trainable params: 1245778 (4.75 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

Now, let's use the model for predicting the next character using the untrained model. At the begining the model will generate only gibberish.

The output size is (1, 100, 82). We predicted only on the first sequence generated by the batch generator. 100 is the number of predicted characters. It has exactly the same length as the input. And there are 82 values for each predicted character. Each of these 82 real values are related to the logarithm likelihood of each character to be the next one in the sequence. The bigger the value, the higher the likelihood. As the network is not trained yet, all those values must be very similar and random. Just check the values for the last prediction on the sequence.

And the simplest way to choose the next character is by getting the index of the element with the highest likelihood. So, for instance, the prediction for the last characeter would be:

And the prediction for the whole sequence would be:

Those 100 numbers represent 100 predicted characters. However, humans cannot read this. So, let's print the input and output sequences using our text_from_ids function, to check what is going on.

As expected, the untrained model just produces random text as response of the given input. It is also important to note that getting the index of the maximum score is not always the best choice. In the last part of the notebook you will see another way to do it.

3 - Training¶

Now you are going to train your model. As usual, you have to define the cost function and the optimizer. You will use the following built-in functions provided by TensorFlow:

• tf.losses.SparseCategoricalCrossentropy(): The Sparce Categorical Cross Entropy loss. It is the loss function used for multiclass classification.
  • from_logits=True: This parameter informs the loss function that the output values generated by the model are not normalized like a probability distribution. This is our case, since our GRULM model uses a log_softmax activation rather than the softmax.
• tf.keras.optimizers.Adam: Use Adaptive Moment Estimation, a stochastic gradient descent method optimizer that works well in most of the cases. Set the learning_rate to 0.00125.

Exercise 4 - compile_model¶

Instructions: Compile the GRULM model using a SparseCategoricalCrossentropy loss and the Adam optimizer

Now, train your model for 10 epochs. With GPU this should take about one minute. With CPU this could take several minutes.

You can uncomment the following cell to save the weigthts of your model. This allows you to use the model later.

The model was only trained for 10 epochs. We pretrained a model for 30 epochs, which can take about 5 minutes in a GPU.

4 - Evaluation¶

4.1 - Evaluating using the Deep Nets¶

Now that you have learned how to train a model, you will learn how to evaluate it. To evaluate language models, we usually use perplexity which is a measure of how well a probability model predicts a sample. Note that perplexity is defined as:

$$P(W) = \sqrt[N]{\prod_{i=1}^{N} \frac{1}{P(w_i| w_1,...,w_{n-1})}}$$

As an implementation hack, you would usually take the log of that formula (to enable us to use the log probabilities we get as output of our RNN, convert exponents to products, and products into sums which makes computations less complicated and computationally more efficient).

$$\log P(W) = {\log\left(\sqrt[N]{\prod_{i=1}^{N} \frac{1}{P(w_i| w_1,...,w_{n-1})}}\right)}$$$$ = \log\left(\left(\prod_{i=1}^{N} \frac{1}{P(w_i| w_1,...,w_{n-1})}\right)^{\frac{1}{N}}\right)$$ $$ = \log\left(\left({\prod_{i=1}^{N}{P(w_i| w_1,...,w_{n-1})}}\right)^{-\frac{1}{N}}\right)$$$$ = -\frac{1}{N}{\log\left({\prod_{i=1}^{N}{P(w_i| w_1,...,w_{n-1})}}\right)} $$$$ = -\frac{1}{N}{{\sum_{i=1}^{N}{\log P(w_i| w_1,...,w_{n-1})}}} $$

Exercise 5 - log_perplexity¶

Instructions: Write a program that will help evaluate your model. Implementation hack: your program takes in preds and target. preds is a tensor of log probabilities. You can use tf.one_hot to transform the target into the same dimension. You then multiply them and sum them. For the sake of simplicity, we suggest you use the NumPy functions sum, mean and equal, Good luck!

Now load the provided pretrained model just to ensure that results are consistent for the upcoming parts of the notebook. You need to instantiate the GRULM model and then load the saved weights.

Now, you will use the 1000 lines of the corpus that were reserved at the begining of this notebook as test data. You will apply the same preprocessing as you did for the train dataset: get the numeric tensor from the input lines, and use the split_input_target to generate the inputs and the expected outputs.

Second, you will predict the next characters for the whole dataset, and you will compute the perplexity for the expected outputs and the given predictions.

Expected Output: The log perplexity and perplexity of your model are around 1.22 and 3.40 respectively.

So, the log perplexity of the model is 1.22. It is not an easy to interpret metric, but it can be used to compare among models. The smaller the value the better the model.

5 - Generating Language with your Own Model¶

Your GRULM model demonstrates an impressive ability to predict the most likely characters in a sequence, based on log scores. However, it's important to acknowledge that this model, in its default form, is deterministic and can result in repetitive and monotonous outputs. For instance, it tends to provide the same answer to a question consistently.

To make your language model more dynamic and versatile, you can introduce an element of randomness into its predictions. This ensures that even if you feed the model in the same way each time, it will generate different sequences of text.

To achieve this desired behavior, you can employ a technique known as random sampling. When presented with an array of log scores for the N characters in your dictionary, you add an array of random numbers to this data. The extent of randomness introduced into the predictions is regulated by a parameter called "temperature". By comparing the random numbers to the original input scores, the model adapts its choices, offering diversity in its outputs.

This doesn't imply that the model produces entirely random results on each iteration. Rather, with each prediction, there is a probability associated with choosing a character other than the one with the highest score. This concept becomes more tangible when you explore the accompanying Python code.

Now, it's time to bring all the elements together for the exciting task of generating new text. The GenerativeModel class plays a pivotal role in this process, offering two essential functions:

1. generate_one_step: This function is your go-to method for generating a single character at a time. It accepts two key inputs: an initial input sequence and a state that can be thought of as the ongoing context or memory of the model. The function delivers a single character prediction and an updated state, which can be used as the context for future predictions.

2. generate_n_chars: This function takes text generation to the next level. It orchestrates the iterative generation of a sequence of characters. At each iteration, generate_one_step is called with the last generated character and the most recent state. This dynamic approach ensures that the generated text evolves organically, building upon the context and characters produced in previous steps. Each character generated in this process is collected and stored in the result list, forming the final output text.

Optional Exercise 6 - GenerativeModel (Not graded)¶

Instructions: Implementing the One-Step Generator

In this task, you will create a function to generate a single character based on the input text, using the provided vocabulary and the trained model. Follow these steps to complete the generate_one_step function:

1. Start by transforming your input text into a tensor using the given vocab. This will convert the text into a format that the model can understand.

2. Utilize the trained model with the input_ids and the provided states to predict the next characters. Make sure to retrieve the updated states from this prediction because they are essential for the final output.

3. Since we are only interested in the next character prediction, keep only the result for the last character in the sequence.

4. Employ the temperature random sampling technique to convert the vector of scores into a single character prediction. For this step, you will use the predicted_logits obtained in the previous step and the temperature parameter of the model.

5. To transform the numeric prediction into a human-readable character, use the text_from_ids function. Be mindful that text_from_ids expects a list as its input, so you need to wrap the output of the temperature_random_sampling function in square brackets [...]. Don't forget to use self.vocab as the second parameter for character mapping.

6. Finally, return the predicted_chars, which will be a single character, and the states tensor obtained from step 2. These components are essential for maintaining the sequence and generating subsequent characters.

Expected output

hear he has a soldier.
Here is a 

________________________________________________________________________________
Dear gold, if thou wilt endure the e 

________________________________________________________________________________
KING OF THE SHREW
IV	I beseech you,  

________________________________________________________________________________

Now, generate a longer text. Let's check if it looks like Shakespeare fragment

In the generated text above, you can see that the model generates text that makes sense capturing dependencies between words and without any input. A simple n-gram model would have not been able to capture all of that in one sentence.

On statistical methods¶

Using a statistical method like the one you implemented in course 2 will not give you results that are as good as you saw here. Your model will not be able to encode information seen previously in the data set and as a result, the perplexity will increase. Remember from course 2 that the higher the perplexity, the worse your model is. Furthermore, statistical ngram models take up too much space and memory. As a result, they will be inefficient and too slow. Conversely, with deepnets, you can get a better perplexity. Note, learning about n-gram language models is still important and allows you to better understand deepnets.