Predicting Sales Closures using Supervised Machine Learning Techniques

Predicting Sales Closures using Supervised Learning Techniques

In the world of B2B selling, predicting sales closures in a time window is one of the most difficult asks for a Business Leader. Most of the times it is a hit and miss and depends solely on an individual, adhoc techniques or just plain luck. Measuring Opportunity Pipeline Quality, Velocity and Predictability (QVP) of an opportunity poses a very difficult challenge for Orders, Revenue and most importantly ,Cash. A lot of techniques have been explored in literature, but it is rare to find a “one-size-fits-all” strategy because the QVP index of every business is uniquely different.

This paper talks about a data science approach that was used to predict sales closures, with an improved certainty, of a business that predominantly sells and delivers Engineered Software into the Oil & Gas, Chemicals and Mining market.

This business uses CRM systems to capture opportunity updates as each opportunity moves through the sales pipelines. The data is entered by Direct Sales reps who are geographically distributed. This business enforces a strong Operating system to ensure all Sales reps update each active opportunity in their customer accounts, at least once a week. The minimum expectation from the business is for them to update the sales stage (Early, Pursuing, Bidding) or if the opportunity is either Lost to a competitor or Abandoned by the customer.

Also, the expectation is to update an estimated Win Probability (Target Variable) for a particular date, Probability that the opportunity will be funded by the customer (Go%), A flag to Commit the Opportunity for a certain date, and a Free text field that has notes capturing next steps. Yet every opportunity has its own story, twists and turns but always leaves behind a trail to help predict outcomes.

The objective of the paper was to predict Win Probability based on the above changes by tracking Quality of the inputs and the Velocity of the updates. It was recognized by the business leadership that with an increased predictability of each opportunity, the biggest impact will be on working capital. However, the challenges were many due to the size of the database, nature of the variables and the additional complexity of the sales pipeline being elastic (i.e. opportunity leads came in and moved out randomly as weekly status changed). The resolution of data was weekly and not any shorter because the updates were mandated on weekly basis as per the Sales Operating System.

In view of the above challenges it was decided that multiple techniques will be used to put some structure in the data. Using un-supervised Learning techniques (like Principal Component Analysis and Correlation plots), the main predictor variables were identified. Since there were only a few variables that were being updated weekly, no further dimension reduction techniques were used.

The next step was to organize the data. Some key activities performed on the data was as follows

1.     Organizing the weekly updates chronologically by opportunity

2.     Identifying and removing outliers (fat-fingered data, wrong customer names, opportunities that came in and went out in less than 2 weeks, very small value opportunities are some of those factors)

3.     Value of the opportunities didn’t change as it moved through the pipelines. So that variable had little significance

4.     Word clouds were used on Notes field to identify any pattern of sentiments for opportunities that closed vs those that did not. Opportunities which had Action Words for a week were flagged

5.     Univariate Analysis of the Target/Dependent Variable (Win Percent) was done to identify if there was any pattern. This variable behaved randomly for every opportunity over weekly time intervals. 

6.     The target variable certainly had a Time Series component but since the Mean was changing it could not be used.

7.     Velocity of data updates were measured based on last update date and how frequently each update were being made by the Sales Reps

Considering the challenges above and keeping in mind the objective of coming up with the most valid, parsimonious and interpretable predictive model, it was decided to use a Mixed Model approach for each opportunity. This approach uses a Linear Regression technique with a Repeated Covariance Structure of 1st order Autocorrelated errors to bring in the Time Series element into the model. The target variable was log transformed since it was highly skewed.

JMPPro from SAS® was used to model 1300 unique opportunities over 72 weeks of data. The first 60 weeks were used to train the model and the last 12 weeks were used to validate. The model development utility which has the significant variables is shown below

After developing hundreds of candidate models with different Fixed, Random and Repeated Structures, the best model selected was based on using Semi Variogram to identify how closely the model Variance deviated from the Empirical Variogram. The selection criterion was the lowest Bayesian Information Criteria (BIC) metric.

The training sample was used for 60 weeks and the hold out for the balance 12 weeks to get the best window of prediction

The validation model and the diagnostics are shown below

The Marginal Model Profilors for each of the significant variables are shown below

Simple interpretation of the model output gave us the following information

1% increase in ProjectGo% increases WinProb% by 0.6%

Sales Operating Process requires flagging opportunities as

-       Gap Closers: first level of commitment

-       Committed Upside: second level of commitment

-       Committed Forecast: third and highest level of commitment

Every 3 days after Committed Forecast flag increase WinProb% by 0.04%

Every 3 days after Committed Upside flag increases WinProb% by 0.03%

Every 3 days after Gap Closure flag increases WinProb% by 0.03%

Sales Opportunity Pipeline State has a big impact on WinProb%

-       Finalizing highest

-       Bidding medium

-       Pursuing average

-       Every 1% increase in weekly change in WinProb% increases Prediction by 1%

The next step of the analysis was to use the Predicted Win Probabilities for certain time slices over the hold out data of 12 weeks and use that to Predict a WIN over the next 12 weeks (Test data not used so far in the modeling). Three Machine Learning Techniques available in JMPPro from SAS® were used

-       Logistic Normal

-       Random Bootstrap Forest

-       Neural Network

As seen here, the Random Bootstrap Forest technique gave the best Sensitivity and Specificity over the Decision Window which was 4 weeks before the end of the hold out period.

The Bootstrap Forest platform predicts a response value by averaging the predicted response values across many decision trees. Each tree is grown on a bootstrap sample of the training data. A bootstrap sample is a random sample of observations, drawn with replacement. In addition, the predictors are sampled at each split in the decision tree. The decision tree is fit using the recursive partitioning methodology. For an individual tree, the bootstrap sample of observations that is used to fit the tree is drawn with replacement. One can specify the proportion of observations to be sampled. The observations used in fitting the tree are called in-bag observations. In this case a Categorical Response of WON or NOT WON was used. For a categorical response, the predicted probability for an observation is the average of its predicted probabilities over the collection of individual trees. The observation is classified into the level for which its predicted probability is the highest.

Business Implications

Total Revenue Acceleration Possibility by avg. 4 weeks

              - Translates to Improved Cash Flow and improves Cash Turns by 10% (in this case)

Improved False Positive Rates throughout Decision Window

              - Translates to Reducing Wasteful Expenditure

Earlier Prediction of 16 out of 40 orders by avg. 4 weeks

              - Translates to Improved Customer Satisfaction. Each 1 point increase in NPS

              increase customer retention by 4% for large corporations

(ref: Managing Customer Value in Business-to- Business Markets by Vittal, Frennea & Westbrook)

Acknowledgements:

I would like to Acknowledge the guidance and insights by my Project Coaches Dr. Simon Sheather and Mr. Jamey Johnston. I also acknowledge all my professors of Texas A&M, Department of Statistics and Mays Business School. The learnings from Marketing Analytics, SAS(R) Programming, Multivariate Regression and Machine Learning Lectures helped me tremendously to deconstruct the problem for better understanding and attempting to solve it effectively. Lessons in Spatial Statistics helped interpret the Variogram Plots.