Build a Better Recommendation Engine

Recommendation engines aren't just for media and retail - they can be used across industries to provide value either to end customers or to employees of the organization itself.

Recommendation engines drive business value in a host of industries and organizations. They are versatile and adaptive, in addition to providing that personalized touch that drives user dependency. With user loyalties rapidly evolving, the ability to quickly understand what a user might want and provide it is critical to continued success.

That said, recommendation engines are not only useful in the retail landscape. They are an important part of everything from disease diagnosis to loan risk mitigation. Recommendation engines are a sort of jack-of-all-trades and are well suited to many use cases.

Individualism, Collectivism, and Training Machines to Humanize Dating Watch Video

Recommendation Engines: Main Techniques

Advanced recommendation engines depend on a host of variables to generate the best value for users. Most recommendation engines are based on some combination of collaborative filtering and content-based analysis.

  • Content-based analysis occurs when an engine recommends items based on fundamental information about the thing itself. This could be as simple as where the name falls in the alphabet, or as complicated as an amalgamation of safety test scores. Since content-based analysis relies on core attributes, it is less mercurial than user behavior, but can still change based on the availability of new information and model adjustment.
  • Collaborative filtering, on the other hand, relies on similar users’ consumption and purchasing habits to drive what to recommend. This way, engines can depend on the human ability to pick up on the nuance of experience. Collaborative filtering is especially well suited to creative content like movies and music, which statistics and quantifiable traits cannot sufficiently address (imagine if movies were recommended based on similar runtimes).

 

Tackling the Cold Start Problem

While collaborative filtering is powerful, it is limited in its ability to include new options. Engines suffer from the cold start problem, where new content that hasn’t been engaged with by users cannot adequately be placed in the matrix of recommendations. Since it has no engagement or reviews, users are less likely to try it out, relegating the content to an inescapable limbo.

The cold start problem is difficult to resolve in part because it depends on so many external factors and opinions. However, AI can use predictive modeling to help integrate new items into collaboratively filtered recommendation engines, thus enabling the correct users to engage. Evolving based on past behaviors is a key strength of machine learning and one reason why recommendation engines are taking off.

 

The Cold-start Problem

 

Asking the Right Questions

Recommendation engines are good for more than just suggesting trendy coffee shops; the business value of customization and personalization is clear. Thus, when getting started it’s key to ask core questions.

  • What is the end goal of the project? Are recommendations really necessary? Are there other solutions (such as anomaly detection) that may be better suited to the problem at hand?
  • What data will they be based on? How will the data be stored and processed quickly?
  • When will recommendations occur? WIll the necessary data be available at this point in the user experience? Will it be fast enough to influence engagement?
  • Are product changes needed for successful recommendations? Is there adequate data at the model’s disposal, or how can more data be made accessible?
  • Should all content be treated equally? Or should new content be artificially promoted in an attempt to break the cold start problem?
  • How should users be segmented? Geography, age, and gender are common divides, but as models evolve these interest groups can become more sophisticated.
The Story of Recommendations at Kickstarter Watch Video

Dataiku is for Recommendation Engines

Dataiku is the platform democratizing access to data and enabling enterprises to build their own path to AI. Hundreds of companies use Dataiku daily to build, deploy, and monitor predictive data flows, including to build custom recommendation engines.

 

The all-in-one platform provides:

  • Features for technical, marketing, and business teams so that everyone can bring expertise and contribute to the production of relevant recommendation engines.
  • Support for the recommendation engine process, from start to finish – because a recommendation engine that’s not in production offers no value – including robust operationalization and model monitoring features in production.

Bringing AI to Marketing

The evolution of AI, machine learning, and data science have been an increasingly integral part of the transformation of many industries, and marketing is no exception.

Read more

Go Further

Behind Tinder's Recommendation Engine

How exactly do dating apps figure you (and your preferences) out? We did a deep dive into the facts.

learn more

A Beer Recommendation Engine

Data scientist Pierre Gutierrez has built a beer recommendation engine, and it’s the perfect project for your Super Bowl Sunday (or anytime).

learn more

Why Teams Need AI Tools

Enterprises and teams need the proper tools with which to work with data, share results, and collaborate on projects. But how to choose?

get the white paper

A Recommendation Engine in 1 Day

A team at Dataiku decided to challenge themselves to build a recommendation system and put it in production in one business day.

learn more