To implement Word2Vec, there are two flavors which are — Continuous Bag-Of-Words (CBOW) and continuous Skip-gram (SG).
In this post I will explain only Continuous Bag of Word (CBOW) model with a one-word window to understand continuous bag of word (CBOW) clearly. If you can understand CBOW with single word model then multiword CBOW model will be so easy to you.
While explaining, I will present a few small examples with a text containing a few words. However, keep in mind that word2vec is typically trained with billions of words.
Continuous Bag of Words (CBOW):
It attempts to guess the output (target word) from its neighboring words (context words). You can think of it like fill in the blank task, where you need to guess word in place of blank by observing nearby words.
Continuous Bag of Words (CBOW) single-word model:
In this section we will be implementing the CBOW for single-word architecture of Word2Vec. The content is broken down into the following steps:
Data Preparation: Defining corpus by tokenizing text.
Generate Training Data: Build vocabulary of words, one-hot encoding for words, word index.
Train Model: Pass one hot encoded words through forward pass, calculate error rate by computing loss, and adjust weights using back propagation.
Output: By using trained model calculate word vector and find similar words.
I will explain CBOW steps without code but if you want full working code of CBOW with numpy from scratch, I have separate post for that, you can always jump into that.
To make it simple I have chosen a sentence without capitalization and punctuation. Also I will not remove any stop words (“and”, “the” etc.) but for real world implementation you should do lots of cleaning task like stop word removal, replacing digits, remove punctuation etc.
After pre-processing we will convert the text to list of tokenized word.
Now to prepare training data for single word CBOW model, we define “target word” as the word which follows a given word in the text (which will be our “context word”). That means we will be predicting next word for a given word.
Now let’s construct our training examples, scanning through the text with a window will prepare a context word and a target word, like so:
For example, for context word “i” the target word will be “like”. For our example text full training data will looks like:
One-hot encoding: We need to convert text to one-hot encoding as algorithm can only understand numeric values.
For example encoded value of the word “i”, which appears first in the vocabulary, will be as the vector [1, 0, 0, 0, 0]. The word “like”, which appears second in the vocabulary, will be encoded as the vector [0, 1, 0, 0, 0]
So let’s see overall set of context-target words in one hot encoded form:
So as you can see above table is our final training data, where encoded target word is Y variablefor our model and encoded context word is X variable for our model.
Now we will move on to train our model.
3. Training Model:
So far, so good right? Now we need to pass this data into the basic neural network with one hidden layer and train it. Only one thing to note is that the desire vector dimension of any word will be the number of hidden nodes.
For this tutorial and demo purpose my desired vector dimension is 3. For example:
“i” => [0.001, 0.896, 0.763] so number of hidden layer node will be 3.
Dimension (n): It is dimension of word embedding you can treat embedding as number of features or entity like organization, name, gender etc. It can be 10, 20, 100 etc. Increasing number of embedding layer will explain a word or token more deeply. Just for an example Google pre-trained word2vec have dimension of 300.
Let’s look back to our example text “I like natural language processing”. Suppose we are training the model against the first training data point where the context word is “i” and the target word is “like”.
Ideally, the values of the weights should be such that when the input x=(1,0,0,0) – which is for the word “i” is given to the model –it will produce output which should close to y=(0,1,0,0) – which is the word “like”.
So now let’s create weight matrix for input to hidden layer (w).
Above picture shows how I have created weighted matrix for first two input nodes. So in the similar manner if we are creating weight matrix for all input nodes it should looks like below, whose dimension will be [3 X 5], in different way it’s
So now let’s calculate error / loss for first iteration only. For the first iteration “like” will be the target word and its position is 2.
So now if we want to generalize above equation we can write like this:
Where j* is the index of the actual word (target word) in the output layer. For first iteration index of target word is 2. So for first iteration j* will be 2 as position of word “like” is 2.
3.5 Back Propagation:
As we have done with forward propagation and error calculation now we need to tune weight matrices through back propagation.
Back propagation is required to update weight matrices (w and w/). To update weight we need to calculate derivative of loss with respect to each weight and subtract those derivatives with their corresponding weight. It’s called gradient descent technique to tune weight.
Let’s take a look for only one example to update second weight (w/).
In this back propagation phase we will update weight of all neurons from hidden layer to output (w/11 , w/12, w/13 …. w/15)
Step1: Gradient of E with respect to w/11:
Now as we know
Derivative of log function with chain rule.
We can generalize above derivative as:
Taking derivative of E with respect to uj:
Note: tj=1 if tj = tj* else tj = 0
That means tj is the actual output and yj is the predicted output. So ej is the error.
So for first iteration,
Now finally coming back to main derivative which we were trying to solve:
So generalized form will looks like below:
Step2: Updating the weight of w/11:
Note: To keep it simple I am not considering any learning rate here.
Similarly we can update weight of w/12, w/13 …. w/35.
Now let’s update 1stweight which (w).
Step1: Gradient of E with respect to w11:
In this back propagation phase we will update weight of all neurons from input layer to hidden layer (w11, w12, …..w53)
Again I will take you through only first weigh (w11)
As to change E with respect to h1, u1to u5 all are involved.
As for u1 and h1,
Step2: Updating the weight of w11:
Note: Again to keep it simple I am not considering any learning rate here.
Similarly we can update weight of w12, w13 … W53.
Final CBOW Word2Vec:
After doing above training process for many iteration final trained model comes with tuned and proper weights. At the end of entire training process we will consider first weight matrix (w) as our vector for our given sentence.
Our sentence or document was“i like natural language processing”
And let’s say first weight of our trained model is:
So word2vce of word “i” will be [w11, w12, w13]
Similarly word2vec for all word will be as follows: