Differential D1672 Diff 5784 swh/web/tests/resources/contents/other/extensions/word2vec.ipynb

Changeset View

Standalone View

swh/web/tests/resources/contents/other/extensions/word2vec.ipynb

This file was added.

# Learning word embeddings - word2vec

\- [Saurabh Mathur](https://saurabhmathur96.github.io/)

The aim of this experiment is to use the algorithm developed by [*Tomas Mikolov et al.*](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) to learn high quality vector representations of text.

## The skip-gram model

Given,

a sequence of words $ w_1, w_2, .., w_T $, predict the next word.

The objective is to maximize average log probability.

$$ AverageLogProbability = \frac{1}{T} \sum_{t=1}^{T} \sum_{-c \leqslant j\leqslant c, j \neq 0} log\ p (w_{t+j} | w_t) $$

where $ c $ is the length of context.

### Basic skip-gram model

The basic skip-gram formulation defines $ p (w_{t+j} | w_t) $ in terms of softmax as -

$$ p (wo | wi) = \frac{ exp(v'^{T} _{wo} \cdot v_{wi}) }{ \sum^{W}_{w=1} exp(v'^{T} _{w} \cdot v_{wi} ) } $$

where $vi$ and $vo$ are input and output word vectors.

This is extremely costly and this impractical as, W is huge ( ~ $10^5-10^7$ terms ).

There are three proposed methods to get around this limitation. - Heirarchial softmax - Negative sampling - Subsample frequent words

I'm using Google's Tensorflow library for the implementation

import tensorflow as tf

For the data, I'm using the [text8 dataset](http://mattmahoney.net/dc/textdata) which is a 100MB sample of cleaned English Wikipedia dump on Mar. 3, 2006

import os, urllib

def fetch_data(url):

filename = url.split("/")[-1]

datadir = os.path.join(os.getcwd(), "data")

filepath = os.path.join(datadir, filename)

if not os.path.exists(datadir):

os.makedirs(datadir)

if not os.path.exists(filepath):

urllib.urlretrieve(url, filepath)

return filepath

url = "http://mattmahoney.net/dc/text8.zip"

filepath = fetch_data(url)

print ("Data at {0}.".format(filepath))

import os, zipfile

def read_data(filename):

with zipfile.ZipFile(filename) as f:

data = tf.compat.as_str(f.read(f.namelist()[0])).split()

return data

words = read_data(filepath)

Take only the top $c$ words, mark rest as UNK (unknown).

def build_dataset(words, vocabulary_size):

count = [[ "UNK", -1 ]].extend(

collections.Counter(words).most_common(vocabulary_size))

word_to_index = { word: i for i, (word, _) in enumerate(count) }

data = [word_to_index.get(word, 0) for word in words] # map unknown words to 0

unk_count = data.count(0) # Number of unknown words

count[0][1] = unk_count

index_to_word= dict(zip(word_to_index.values(), word_to_index.keys()))

return data, count, word_to_index, index_to_word

vocabulary_size = 50000

data, count, word_to_index, index_to_word = build_dataset(words, vocabulary_size)

print ("data: {0}".format(data[:5]))

print ("count: {0}".format(count[:5]))

print ("word_to_index: {0}".format(word_to_index.items()[:5]))

print ("index_to_word: {0}".format(index_to_word.items()[:5]))