Word Embeddings, NLP and I18N in H2O

Word embeddings can be thought of as a dimension-reduction tool, needing a sequence of tokens to learn from. They are really that generic, but I’ve only ever heard of them used for languages; i.e. the sequences are sentences, the tokens are words (or compound words, or n-grams, or morphemes).
This blog post is for code I presented recently on how to use the H2O implementation of word embeddings, aka word2vec. The main thing being demonstrated is that they apply equally well for any language, but you may need some language-specific tokenization, and other data engineering, first.
Here is the preparation code, in R; bring in H2O, and define a couple of helper functions.

library(h2o)

h2o.init(nthreads = -1)


show <- function(w, v){
  x = as.data.frame(h2o.cbind(w, v))
  x = unique(x)
  plot(x[,2:3], pch=16, type="n")
  text(x[,2:3], x[,1], cex = 2.2)
}


reduceShow <- function(w,v){
m <- h2o.prcomp(v, 1:ncol(v), k = 2, impute_missing=T)
p <- h2o.predict(m, v)
show(w, p)
}

Then, I define an artificial corpus, and try with word embedding dimensions of 2, 4 and 9. For dimensions above 2, reduceShow() is using PCA to just show the first two dimensions.
eg1 = c(

  "I like to drive a blue car",
  "I like to drive a red car",
  "I like to drive a green car",
  "I like to drive a blue lorry",
  "I like to drive a yellow lorry",
  "I like to drive a brown lorry",
  "I like to drive a green lorry",
  "I like to drive a red Ferrari",
  "I like to drive a blue Mercedes"
)

eg1.words <- h2o.tokenize(
  as.character(as.h2o(eg1)), "\\\\W+")

head(eg1.words,12)

eg1.wordsNoNA <- eg1.words[!is.na(eg1.words),]

eg1.wv <- h2o.word2vec(eg1.words,
                       min_word_freq = 1,
                       vec_size = 2)

eg1.vectors = h2o.transform(eg1.wv,
                            eg1.wordsNoNA,
                            "NONE")
show(eg1.wordsNoNA, eg1.vectors)


eg1.wv4 <- h2o.word2vec(eg1.words,
                        min_word_freq = 1,
                        vec_size = 4)

eg1.vectors4 = h2o.transform(eg1.wv4,
                             eg1.wordsNoNA,
                             "NONE")
reduceShow(eg1.wordsNoNA, eg1.vectors4)

eg1.wv9 <- h2o.word2vec(eg1.words,
                        min_word_freq = 1,
                        vec_size = 9,
                        epochs = 50 * 9)

eg1.vectors9 = h2o.transform(eg1.wv9,
                             eg1.wordsNoNA,
                             "NONE")
reduceShow(eg1.wordsNoNA, eg1.vectors9)

Those results are fairly poor, as we only have 9 sentences; we can do better by sampling those 9 sentences 1000 times. I.e. more data, even if it is exactly the same data, is better!

eg2 = sample(eg1, size = 1000, replace = T)

#(rest of code exactly the same, just changing eg1 to eg2)

What about Japanese? Here are the same 9 sentences (well, almost “This is a …” each time, rather than “I drive a …”), but hand-tokenized in a realistic way (in particular の is a separate token). I’ve gone straight to having 1000 sentences, as we know that helps:

  # これは青い車です。
  # これは赤い車です。
  # これは緑の車です。
  # これは青いトラックです。
  # これは黄色いトラックです。
  # これは茶色のトラックです。
  # これは緑のトラックです。
  # これは赤いフェラーリです。
  # これは青いメルセデスです。

ja1 = c(  #Pre-tokenized
  "これ","は","青い","車","です","。",NA,
  "これ","は","赤い","車","です","。",NA,
  "これ","は","緑","の","車","です","。",NA,  # ***
  "これ","は","青い","トラック","です","。",NA,
  "これ","は","黄色い","トラック","です","。",NA,
  "これ","は","茶色","の","トラック","です","。",NA,  # ***
  "これ","は","緑","の","トラック","です","。",NA,  # ***
  "これ","は","赤い","フェラーリ","です","。",NA,
  "これ","は","青い","メルセデス","です","。",NA
  )
ja2 = rep(ja1, times = 120)
# nrow(ja2) is 7920 tokens; representing 1080 sentences.

The code to try it is exactly as with the English, except we just import ja2 and don’t run it through h2o.tokenize().
ja2.words = as.character(as.h2o(ja2))

head(ja2.words, 12)

ja2.wordsNoNA <- ja2.words[!is.na(ja2.words),]

ja2.wv2 <- h2o.word2vec(ja2.words,
                        min_word_freq = 1,
                        vec_size = 2,
                        epochs = 20)
ja2.vectors2 = h2o.transform(ja2.wv2,
                             ja2.wordsNoNA,
                             "NONE")
show(ja2.wordsNoNA, ja2.vectors2)

ja2.wv4 <- h2o.word2vec(ja2.words,
                        min_word_freq = 1,
                        vec_size = 4,
                        epochs = 20)
ja2.vectors4 = h2o.transform(ja2.wv4,
                             ja2.wordsNoNA,
                             "NONE")
reduceShow(ja2.wordsNoNA, ja2.vectors4)


ja2.wv9 <- h2o.word2vec(ja2.words,
                        min_word_freq = 1,
                        vec_size = 9,
                        epochs = 50)
ja2.vectors9 = h2o.transform(ja2.wv9,
                             ja2.wordsNoNA,
                             "NONE")
reduceShow(ja2.wordsNoNA, ja2.vectors9)

In Python? I’ll just quickly show the key changes (there are full word embedding examples in Python, for H2O, floating around, e.g. https://github.com/h2oai/h2o-meetups/blob/master/2017_11_14_NLP_H2O/Amazon%20Reviews.ipynb )
To bring the data in and tokenize it:

eg1 = [...]
sentences = h2o.H2OFrame(eg1).ascharacter()
eg1_words = sentences.tokenize("\\W+")
eg1_words.head()

Then to make the embeddings:

from h2o.estimators.word2vec import H2OWord2vecEstimator
eg1_wv = H2OWord2vecEstimator(vec_size = 2, min_word_freq = 1)
eg1_wv.train(eg1_words)

And to get the vectors for visualization:

eg1_vectors = eg1_wv.transform(eg1_words_no_NA, "NONE")

(The Python code is untested as I type this - if I have a typo, let me know in the comments or at darren at dcook dot org, and I will fix it.)

Umlauts, pound signs and more!

Until I learnt this tip, whenever I needed some fancy character (£,°,á,ß, etc.) I went hunting for somewhere to copy and paste it from. Not any more!

I assigned my previously useless windows key to be my Compose key. To type the above pound sign I pressed the windows key, then pressed the L key, then pressed the minus key. Scharfes-S (ß) is simply windows key then press the S key twice. To type á (a with an accent) you press ' (shift-7) and a after the window key.

(Note: until I wrote this post I thought you had to hold the Compose key down while pressing the other keys. That works too, but it is easier to treat it as a modal switch. In other words, you press the Compose key to enter Compose Mode, then the next two characters you type are interpreted together. And if the two characters you type have no special meaning then nothing happens. Either way you exit Compose Mode after your two keypresses.)

See here for the full list of all the compose options

Naturally the above is for linux. This page answers how to do it on Windows and here is the same concept on a Mac



Bonus Tip. Ever wanted to write a Japanese post office sign ( 〒 ) ? Enter your Japanese IME and type yuubin (ゆうびん) (then press space). The ～ symbol can be done by typing kara (から).


˙unɟ ɹoɟ ʇsnɾ ˙uʍop ǝpısdn ʇxǝʇ ɹnoʎ suɹnʇ ʇɐɥʇ ǝʇıs ɐ puıɟ :dıʇ snuoq ɹǝɥʇouɐ
(Yes, this is part of Unicode too.)

Google Messing Up I18n

People often seem to think Google can do no wrong; an image helped no doubt by being surrounded by companies such as Microsoft and Apple. But, even with all those computer science PhD's it snapped up, it sometimes just does not get it.

Once a month or so, Google resets back to showing me the search results in Japanese, and only searching Japanese pages. My browser is set to say I prefer English pages, but it ignores that and uses my IP address. Japanese people use a Japanese browser that will send a header saying they want to see Japanese. Unless they actually want to see English. Why is Google ignoring this?

Perhaps Google needs fewer PhD's and more people who know what this header means:
Accept-Language en-gb,en;q=0.5

Google (!) tells me Google need to look at section 14.4 of RFC2616.

But, the user-unfriendliness goes deeper. Here is how I get out of Japanese mode.
Click: 検索オプション
Find: 言語 検索対象にする言語
and scroll down to find and select 英語

No! No, no, no! It just goes to show, even though I read Japanese I still messed up. Here is the correct way:

In the top-right of the search results click "設定 ▼" then 検索設定
In the line that says 表示言語の設定 scroll down to select 英語.
Then find the button in top-right that says 保存。When it pops up something that looks like an error message (but is actually a success message), press OK.

Hopefully I'm good for another month.

But the point of my rant is the only piece of English on any of those pages was "google". Not a single helpful line such as "switch this page to English". Perhaps they should use the Accept-Language header to decide what language the user would most like to see a "switch this page to XXX" link in?

Well, rant over. Google have been doing this for 10 years, so I don't expect they're going to change any time soon.

Japanese NLP mailing list

A new mailing list for discussing Japanese NLP (natural language processing) in English has been set up, by Jim Breen (of JMDict fame):
http://groups.google.com/group/nlp-japanese?hl=en

There is a lot of software for processing Japanese text which is only documented in Japanese, and even then only minimally documented. So the new list is an ideal place (for those of us more comfortable in English than Japanese) for asking about how to use chasen, cabocha, namazu, etc. Or to describe what you are trying to do and get program and data suggestions. Hopefully people will also post about new software and data releases, related conferences, new academic papers, and so on.

Also, if the above interests you then you will also want to know about this Ubuntu repository for all kinds of NLP software:
http://cl.aist-nara.ac.jp/~eric-n/ubuntu-nlp/dists/hardy/japanese/

Much of the Japanese stuff is UTF-8 ready (as opposed to the EUC-JP that academic Japanese software still likes to default to).