Exploring AI, Machine Learning, and Sitecore
27 novembre 2018 • 7 Minute Read • Elizabeth Spranzani, Chef de la direction technologique
Last year at Symposium 2017, Sitecore introduced new and curious buzzwords to the Sitecore community: Cortex, Machine Learning, and AI. Over the course of 2018, through various webinars, Sitecore user groups, whitepapers, and blog posts, Sitecore revealed more and more information around what these words meant in the Sitecore world, and how they would manifest when Sitecore 9.1 was released. As a programmer at heart, I was looking to understand how the pieces all connected. This was my focus going into the 2018 Sitecore Symposium – I registered for 3 or 4 different sessions on the topic and learned more at the MVP summit.
I also did some research of my own before attending, since AI and Machine Learning (ML) are key areas in Verndale's NECXT campaign I wanted to come away from Symposium 2018 with the ability to tell our clients exactly what they need to start down the road of using Cortex and Machine Learning, how it works, and some real world examples of it in use with Sitecore.
There were three sessions that influenced this post and I want to take a moment to give credit to these:
- Real-world machine learning - Thirty scenarios where ML makes sense right now: Michael Greenberg - Head of Strategy, Hedgehog
- Where machine learning meets social #ThinkYouKnowMe: Una Verhoeven - Senior solution developer
- Automated personalization that "just works": Marcos Perreau Guimaraes - VP, Chief Data Scientist, Sitecore
Basic Premise
AI Queries and Machine Learning algorithms take the manual work out of analyzing collected data, and decision-making around how to personalize user experiences based on that data.
Machine Learning vs Artificial Intelligence
These terms are not mutually exclusive. It’s best to think of Machine Learning as a type of Artificial Intelligence: “General” AI, which aims to improve results over time as it processes and iterates over data. This is opposed to “Applied” AI (robots, autonomous cars, conversational apis, etc). Machine Learning is allowing the computer to crunch huge amounts of data to see patterns that a human being simply couldn't put together on their own (clustering).
Prerequisites
Before jumping right in, plan to have an instance of Sitecore 9.1 running. Additionally, you’ll need:
- A Sitecore XP license (not XM), which gives you the Analytics (xDB) database.
- A separately licensed Machine Learning service, available from Microsoft or other vendors like IBM.
- You may also need a Cognitive service such as Microsoft's Cognitive Toolkit (CNTK) or IBM's Personality Insights.
- A nice-to-have would be a data scientist who understands data modelling.
- A must have is lots and lots of relevant and recent data (large web traffic volumes from which to collect data). In an effort to collect more data, don't sacrifice the need for it to be relevant.
Let's not forget the reason to go through this effort: You’ll need defined personalization needs, and methods for doing A/B tests to validate what is being learned.
Data and Data Scientists
The key to all of this is data. You must gather as much data as possible including: actions, context, feedback and outcomes. It must be associated in such a way that allows for predictions and probability (classification). Those predictions must then be assessed for correctness and fed back into the next set of predictions (feedback). All visitor data must be leveraged. How much is a lot of data? You realistically need a few hundred thousand users in a short period of time. Old data is not relevant.
Machine Learning is first configured by a person and then run for a period of time. Periodically the data models need to be analyzed again by someone who understands what they should be looking for to determine if predictions have been accurate or not. This is where a data scientist would play a critical role. There are many different methodologies for Machine Learning and deciding which is best for a particular goal or situation would best be suited for a data scientist who has studied the topic. Options include collaborative filtering, matrix factorization, topic modeling, informative theory, and deep learning. That said, if you don't happen to have a data scientist at your disposal, you can still work with some of the Machine Learning services, since many have default/recommended settings during initial configuration.
You can visualize the Machine Learning process in the following way:
Source Data -> Data Cleaning -> Feature Building (structure the data into the predefined model) -> Clustering (find patterns in the data) -> Regression -> Recommendation for user
Specifically involving Sitecore:
xDB -> Model Factory -> Daily Compute -> Online ML Services <-> Sitecore Visitor and Content
Machine Learning Applied in Sitecore
At this point you may be wondering, “what can I realistically be doing with Machine Learning that would make all of this worthwhile?” One of your basic needs must involve a desire for personalization. You may have thousands of visitors to your website, but only care about those that are likely to subscribe. Machine Learning will help you to identify who is likely to be that important customer, who truly matters. You may have a lot of content or options you don't want your end user to have to scroll through (think thousands of options). Machine Learning will be able to make recommendations and advance target the content that will mean the most for your users and keep them coming back. Don't want to keep personal information about your users? With Machine Learning, there are ways to still infer deep knowledge about them without saving that personal information. You can even predict the actions and desires of a first-time user before you’ve captured any direct behavioral interactions.
There was one session that really hit it home and I want to give big props to Una Verhoeven for one of the best sessions I have ever attended at a Sitecore Symposium. She was clear, well-spoken, knew her material very well (clearly a subject matter expert in it) and she broke it down into simple steps that resulted in a pretty cool use case!
As an example, for her presentation, Una built a bookstore website. It allowed users to log in via their twitter account social credentials. Once provided access to their twitter feed, Una was able to analyze the twitter feed via IBM's Personality Insights. (It uses natural language processing to understand and rate various emotional ranges from text including openness, agreeableness, introversion/extroversion, etc.) The IBM service returns JSON that includes a percentage rating for each emotion for that twitter feed. Note that this data is NOT saved to the user's xConnect profile, it will only become relevant if/when they make a purchase and even at that time, it won't be connected to them, keeping them more anonymous.
The magic happens when the user makes a purchase. The purchased Book ID is then related to the personality insights. This is what is saved into the xDB via xConnect, nothing personally identifiable. The purpose is to answer the question: “what type of person is likely to buy this book?” The hypothesis here is that a twitter feed and the emotion taken from it is a better indicator for personal book preferences rather than more explicit data like age, gender, or geographic location. For every purchase of a book, there will be an entry with the corresponding personality insights.
Sitecore Cortex (Projection Framework) transforms this structured data into non-structured data. This is what the Machine Learning service will 'learn' from.
The structured Data is then fed to the Machine Learning service. There are several to choose from, weighing both requirements and the service costs is important to decide which service is best. IBM Watson was the option used in the above example, and it is free for a certain number of API requests.
Una explained the following steps to take within Watson:
Create a model. This can be done (to some degree) without knowledge of how Machine Learning works because Watson provides guidance. Define a Name, Description and type of Model.
Provide the previously mentioned unstructured CSV data created by Cortex.
Specify a few key values: The “Feature Columns” are the personality insights, the “Value to Predict” is the book, and Watson will recommend a “Technique”.
Finally, choose one of several different “Estimators”. Now Watson is ready to make predictions when it is requested (code snippets on how to do this are provided).
Using the system is simple. When a new website customer has connected via Twitter, Una's POC retrieved personality insights data from the IBM service again. Then it made a call to the Watson service, passing in that data and Watson simply returns a single recommended Book Id. It has predicted what Book this user will like, based on what other users of similar personalities have already purchased. Armed with this Book, Sitecore can now output some content in a rendering upselling this Book to the end user, even though they may not have done anything on the website.
Although this is a really simple proof of concept, it drilled home both the process and the potential uses for me. Combining that with some of the other sessions on the topic of Machine Learning, I have some ideas for where to go next in our own explorations. Not only did Sitecore select some great sessions and speakers, they did a great job of making sure that when strung together, many of the gaps would be filled in. I thank specifically Una, Michael, and Marcos for all the knowledge that they shared with us. I'm looking forward to the opportunities this year to get this in place for some of our clients.