Word From Our Editor: How Spotify, Netflix and Amazon Know What You Love

Playlists and recommendations are the gateway to great discoveries, but just how do they work? We take a look at the science behind the magic.

It’s fair to say we take a lot of pretty amazing technology for granted these days. Self-parking cars? Boring. 100mbps WiFi? Too slow man. Autonomous drones? Meh. And the science that goes on behind the scenes to allow the likes of Spotify, Netflix, Amazon and many more, to know exactly the right kind of music, movie, or product to recommend or curate, is another such marvel that we routinely take for granted.

Spotify

Spotify’s Weekly Discovery recommendations are a must for millions of users.

In a recent interview with Wired, software engineer Edward Newett revealed how the Spotify algorithm can determine what tens of millions of people want to listen to every week.

He explained: “There are two parts to how the algorithm works: on one side, every week we’re modelling the relationship of everything we know about Spotify through our users’ playlist data.

“On the other, we’re trying to model the behaviour of every single user on Spotify – their tastes, based primarily on their listening habits, what features they use on Spotify and also what artists they follow. So we take these two things and every Monday we recommend what we think you would like, but might not have heard about.”

So how does the algorithm decide what to offer? “By trying to mimic the behaviour of all of our users when trying to put together their perfect mix, we can leverage Spotify’s two billion playlists, target individual tastes and come up with playlists that will be interesting.”

As Quartz (qz.com) points out, automated music recommendations are hardly new, but somehow Spotify seems to have identified the ingredients of a personalised playlist that make it feel fresh and familiar at the same time. Which of course is potentially a big advantage over competitors like Google, and Apple, who tend to feature the same bottomless catalogue of music, but take very different approaches to picking the best songs for each user.

Neflix

Netflix constantly looks to hone its personlisation. Flixtape is one of the latest options available to its users.

With more than 75 million subscribers in over 190 countries, knowing how to serve up what viewers want is something streaming giant Netflix takes very seriously.

The company employs around 1,000 people based in Silicon Valley who work solely on its personalisation algorithm. It’s so complex that it resets every 24 hours to ensure users discover the exact content they want to watch from an estimated 13,000 titles at any given time.

Speaking to Business Insider, Netflix vice president of product innovation, Chris Jaffe, explained: “We work on constantly making that experience better and better. It’s a unique approach. In some companies [that are] evolving the product, the product team might be the driver: the team comes up with the idea, design, builds, launches, and sees what happens. My team can’t make that decision. We come up with the ideas, but what drives product decisions is our customers and what customers do and how they use the product.”

“Even search is personalized on Netflix, with the results influenced by the kinds of things a user has watched and the popularity of a particular title.”

Amazon

Amazon’s recommendations game has been strong since day one.

It feels like a long time ago that Amazon simply sold stuff (let alone just books). But pretty much since day one, Amazon’s recommendation game has been really strong: amazing levels of content personalisation all served up at lightning speed, contributes billions of dollars in profit through enhanced click through and conversion rates.

A paper written by Greg Linden, Brent Smith, and Jeremy York, all from Amazon, explains the science behind it. They state the most common approaches to algorithm-based recommendation are collaborative filtering, cluster models, and search-based methods.

• Trusty old collaborative filtering works by looking at the user’s purchase and/or ratings history and generates recommendations based on a selection of similar customers, disregarding any items in the set that they may have already been purchased.
• Cluster models divide the customer base into a number of segments and then classify users into segments that contain the most similar customers. Purchase and ratings history of users in the segment are then used to generate recommendations.
• Search-based models build keyword, category, and author indexes, however they struggle with large volumes of customer preference data.

Build it and they will come

Now, these are all well and good, but none of the traditional methods were up to the demands and ambitions of Amazon – so they invented their own: item-to-item collaborative filtering.

The paper explains, “Our algorithm, item-to-item collaborative filtering, scales to massive data sets and produces high-quality recommendations in real time… Rather than matching the user to similar customers, item-to-item collaborative filtering matches each of the user’s purchased and rated items to similar items, then combines those similar items into a recommendation list.”

The key to item-to-item collaborative filtering’s scalability and performance is that it creates the expensive similar-items table offline. The algorithm’s online component – looking up similar items for the user’s purchases and ratings – scales independently of the catalogue size or the total number of customers; so it’s dependent only on how many titles the user has purchased or rated. As a result, the algorithm is lightning fast, even for extremely large data sets – meaning of course we end up buying waaay to much stuff! Especially if our shopping spree happens to be after a trip to the pub…

Here at FileHippo we can recommend the best range of great, free software, including Spotify. For our full range, click here.