I Thought He Came With You (ITHCWY) is Robert Ellison's blog.

Upgrading from word2vec to OpenAI

[1536]

In 2018 I upgraded the related posts functionality on this blog to use word2vec. This was hacked together by averaging the vectors for interesting words in each post together and then looking for the closest vectors. It worked quite well, but the state of the art has moved on just a little bit since then.

OpenAI has an embeddings API and recently released a cheaper model called text-embedding-ada-002. The vectors have 1,536 dimensions, a pretty significant increase from the 300 I was using with word2vec. Creating vectors for all my posts took a few minutes and cost $0.11 which is pretty affordable. As you'd expect those related posts are now significantly more related and useful. Thanks OpenAI!

I shared some code previously for the word2vec hack. This is a lot more straightforward - call the API with the post text and then compare the vectors with cosine distance to find the most related. It works well for search too.

Add your comment...

(All Code Posts)

(Published to the Fediverse as: Upgrading from word2vec to OpenAI #code #ml #openai #ithcwy #word2vec Using the Open AI embeddings API to find better related posts for a blog. )

code, ml, openai, ithcwy, word2vec

Better related posts with word2vec (C#)

Updated on Tuesday, February 13, 2024

I have been experimenting with word2vec recently. Word2vec trains a neural network to guess which word is likely to appear given the context of the surrounding words. The result is a vector representation of each word in the trained vocabulary with some amazing properties (the canonical example is king - man + woman = queen). You can also find similar words by looking at cosine distance - words that are close in meaning have vectors that are close in orientation.

This sounds like it should work well for finding related posts. Spoiler alert: it does!

My old system listed posts with similar tags. This worked reasonably well, but it depended on me remembering to add enough tags to each post and a lot of the time it really just listed a few recent posts that were loosely related. The new system (live now) does a much better job which should be helpful to visitors and is likely to help with SEO as well.

I don't have a full implementation to share as it's reasonably tightly coupled to my custom CMS but here is a code snippet which should be enough to get this up and running anywhere:

// load word2vec model _vocabulary = new Word2VecBinaryReader().Read(HostingEnvironment.MapPath("~/App_Data/GoogleNews-vectors-negative300-SLIM.bin")); // ... collect all words in post (words is List<string>, not included as implementation specific) ... double[] vector = new double[_vocabulary.VectorDimensionsCount]; // add all words that exist in the vocabulary int inVocabularyCount = 0; foreach(string word in words) { try { float[] wordVector = _vocabulary.GetRepresentationFor(word).NumericVector; for (int i = 0; i < _vocabulary.VectorDimensionsCount; i++) { vector[i] += (double)wordVector[i]; } inVocabularyCount++; } catch { // ignore anything not in vocabulary } } // return the average of the vectors for (int i = 0; i < _vocabulary.VectorDimensionsCount; i++) { vector[i] /= inVocabularyCount; } return vector; // ... once all posts have a vector we just find the closest ones, compare this post to others and sort the list by distance ... double d = Distance.Cosine(vectorThisPost, vectorOtherPost);

The first step is getting a vector representation of a post. Word2vec just gives you a vector for a word (or short phrase depending on how the model is trained). A related technology, doc2vec, adds the document to the vector. This could be useful but isn't really what I needed here (i.e. I could solve my forgetfulness around adding tags by training a model to suggest them for me - might be a good project for another day). I ended up using a pre-trained model and then averaging together the vectors for each word. This paper (PDF) suggests that this isn't too crazy.

For the model I used word2vec-slim which condenses the Google News model down from 3 million words to 300k. This is because my blog runs on a very modest EC2 instance and a multi-gigabyte model might kill it. I load the model into Word2vec.Tools (available via NuGet) and then just get the word vectors (GetRepresentationFor(...).NumericVector) and average them together.

I haven't included code to build the word list but I just took every word from the post, title, meta description and tag list, removed stop words (the, and, etc) and converted to lower case.

Now that each post has a vector representation it's easy to compute the most related posts. For a given post compute the cosine distance between the post vector and every other post. Sort the list in ascending order and pick however many you want from the top (the distance between the post and itself would be 1, a totally unrelated post would be 0). The last line in the code sample shows this comparison for one post pair using Accord.Math, also on Nuget.

I'm really happy with the results. This was a fast implementation and a huge improvement over tag based related posts.

Updated 2023-01-29 22:00:

I have recently moved this functionality to use an OpenAI embeddings API integration.

Add your comment...

(All Code Posts)

(Published to the Fediverse as: Better related posts with word2vec (C#) #code #software #word2vec #ithcwy #ml How to use word2vec to create a vector representation of a blog post and then use the cosine distance between posts to select improved related posts. )

code, software, word2vec, ithcwy, ml

Upgrading from word2vec to OpenAI

Related Posts

Better related posts with word2vec (C#)

Related Posts

Newsletter

Popular Posts

Recent Posts