Predict User Intent with Machine Learning (AI, Recommender Systems)

Intent Marketing

Intent data is the behavioral and transactional information collected from individual B2B transactions. Intent marketing processes the data to predict user intent to buy, process, or to adopt a specific service or product. The user can be an anonymous visitor, a lead, a prospect, or an existing customer.

There are several benefits in predicting user intent, especially in a B2B environment:

• It increases your ROI by removing ‘the absence of awareness' stage in promoting a service or product.
• As it is the cornerstone of predictive lead scoring and account based marketing, it increases prospect conversion to customers and assures better customer engagement and after sales service.

But the question is, how this is achieved.

Data sources

There are several sources of data you can be use for retrieving user intent data. These may include:

Internal data

Internal data, or 1st party data, comes from your own data sources, such as your company’s web site, past transactions, or any financial information coming from an integrated CRM or ERP platforms. They may be structured or unstructured, such as behavioral information coming from social media.

You need a powerful computer to effectively handle your 1st party data but, depending on the model, you can still find insights in them. According to some recent studies, 48% of marketers surveyed admitted that their 1^st party data was important to their operations, while the remaining 51% considers both 1^st and 3^rd party important.

External data

You acquire your external, or 3^rd party, data from data providers. While they only provide less-valuable “generalized” information many marketers find this data useful as well.

Regardless of the type of data, you can use user intent data to predict the outcomes of your campaigns, processes, and operations.

Machine learning

Machine learning is a subset of artificial intelligence and computer science, where systems “learn” from data to make decisions and predictions. It enables computers to improve performance and make data-driven decisions without being explicitly programmed.

Machine learning, along with statistics, is a part of data science. Its algorithms and predictions are only as good as the dataset used to build and evaluate its predictive model and parameters. Furthermore, within data analytics and computational statistics machine learning is a method for designing and building predictive models and algorithms. These analytical models, known as predictive analysis, allow researchers, data analysts, and scientists to produce reliable and repeatable decisions and results.

In a programmed prediction method, you need to implement a decision path based on your business rules resulting in a series of complex if-then-else clauses. In machine learning however, someone may start with a series of data for example outcomes, and then apply a pattern-driven model to the data to predict the output of a another dataset based on the model/pattern.

Machine learning methods can be categorized as supervised or unsupervised.

Supervised machine learning

Although not explicitly implied above, in supervised machine learning, you might have a series of example or historical data for a given outcome that you want use to predict the outcome of another dataset. You can do this by separating the data you use to serve as your model, known as training data, from the data you want evaluated (evaluation data).

Since there always is an outcome, this will result in a comparison between the predicted and the actual.

An example of supervised machine learning is testing prices against man days to predict the price for a new service. This is achieved by applying a model that best fits with existing data and then re-evaluating it when new data arrives. Supervised machine learning models may include regressions (linear, polynomial, logistics) and classifications.

Unsupervised machine learning

With unsupervised machine learning, you still have a training set of example or historical data, but there is no specific desired output. Unsupervised machine learning is the most common category and is mainly used in pattern detection and descriptive modelling.

A typical example of unsupervised machine learning is having several customer product ratings that you want to use to predict what a new customer may buy. Unsupervised machine training models may include clustering, association discovery, and anomaly detections.

Reinforcement machine learning

Reinforcement learning allows machines and software to automatically identify the ideal behavior within a specific context, an objective of which is to maximize performance. In reinforcement machine learning, a machine / software (the agent) takes an action that has the bigger probability to return an award from the environment. After several irritations, machines “learn” to avoid actions that didn't result in award and to reach the optimum performance:

Reinforcement machine learning does not require historical data. Common algorithms are neural networks.

Machine learning process

Regardless of the model type (for supervised / unsupervised) used, there are several steps you must take, when implementing a machine learning model.

Data gathering and preparation:

• Collecting intent data (either 1^st or 3^rd party data).
• Noise separation and keeping only valuable and exploitable ones.
• Data consolidation by removing duplicates coming from different systems. In the case of supervised learning, for assuring better reliability after data consolidation, data will be distinguished to training and evaluation data:
• Those that will be used for building the model (training data).
• Those that will be used for evaluating the model (evaluation data).

After choosing and building a model, you should evaluate the model and only deploy it if successful.

As described above, the model definition depends on the used dataset size. Bigger datasets are more reliable. This is a basic problem of the machine learning method, the so called “cold start”.

Choosing a model

Choosing the model type and applying model parameters are the most critical points in any machine learning method. Intent data sets are often large with numerous exploitable attributes, and depending on the case, they may form a multifactorial system of equations that cannot be accessed using if-then-else clauses. Therefore you must define your machine learning models to minimize the possible errors for a specific dataset.

How it works? A (very) simplified example

In order to illustrate the above, consider the following example: a customer asks a delivery company to estimate the time required to deliver something from a specific store to a specified delivery location.

Conventionally, time required should be proportional to location distance from the store. Proportional means that it should follow an algorithm of type:

y=ax+b

Here, y is the time required and x is the distance. Values of a, b may be defined according to speed limits, working hours, etc.

Often in machine learning, and this is the case of supervised machine learning, you may apply a linear regression model. For a specific store, historical deliveries may produce a distribution like:

Linear regression is also described by y=ax+b. But in this case:

• Values of a, b are defined under the condition of what are the values that produce the minimum of the sum of the square of the vertical distance between delivery times (the red points) and the line.
• Values of a, b are continuously adjusted, according to successive deliveries delivery time.

User intent and machine learning

Predicting user intent may not be as simple as the above example:

• There are several customer features that should be taken into account: What are the geographical data? Is there any similarity with other customers? What is the customer group?
• There are several product features that may be taken into account. What other products are related? What are the competitive ones? How a product group is ranked to a customer segment profile?
• There are several outcomes that may be of interest: is the customer financially reliable? What is the ROI? What is the lead scoring etc.?

All these result in multifactorial equation systems that can be solved only computationally. Big data analysis, computer hardware evolution and cloud architecture have resulted in making user intent predictive analysis feasible and efficient. Today cloud architecture with API integrated platforms give the opportunity to even small sized organizations to exploit such features.

Recommender engines

A most common example used in predicting user intent is recommender systems. Recommender systems or engines is a subclass of information filtering system that aims to predict the preference of a user into an item. It is seen as an intelligent and sophisticated salesman who knows the customer behavior and can make intelligent decisions about what recommendations should benefit the customer most. Reliable recommendations can result to more effective personalized content and advertising, thus increasing lead conversion rate.

There are several types of recommendation engines depending on the algorithm applied for filtering. These include collaborative filtering, content-based filtering, demographic filtering, etc., with the first two (and the combination of them) being the most widely used. From a machine learning point of view and regardless of type, recommender systems are based on unsupervised machine learning models.

In collaborative filtering, recommendations are based on product ratings, derived either from explicit data such as user ratings or from implicit data such as website user activity or social data. In content based filtering, the system makes suggestions based on the user profile related to the product features vector. User profile may be based on customer segmentations or groupings

Recommender systems are currently used in several industries :

• Customer who bought this also bought. (www.amazon.com)
• Up next (Recommended videos). (www.youtube.com )
• Other movies you may enjoy. (www.netflix.com )
• People you may know. (www.linkedIn.com )

Furthermore someone can reveal recommender engines through all of the customer journey:

• Personalized content
• Personalized ads
• Voice recognition (NLP), chatbots

According to studies 35% of Amazon total sales and 78% of Netflix total watches are based on recommendations and these values are continuously increasing.

Conclusion

Thanks to latest technology achievements, machine learning techniques have become a building block in automated marketing based on user intent. It can derive more accurately and efficiently the users intent throughout the customer journey leading to better customer engagement.