In recent years, marketers have been getting increasingly comfortable with complex analytical frameworks. Many are moving away from simple linear and logistic models to more complex ensemble models. An ensemble model can be thought of as a combination of simpler models. As a result, they are typically much more complex than traditional linear models. The reason is that they often outperform the simpler traditional models. Simply put, the benefit of an ensemble model is this: by combining several models the right way, you get a model that is better than the best of the individual models. Think of the individual models as capturing one specific effect that influences the predicted outcome; for example, a consumer responding to a solicitation. The ensemble then combines all these models into one that captures all the main drivers of the predicted outcome. This was famously demonstrated by the winning model of the Netflix Prize.
But what are the drawbacks of these newer, more complex models?
For starters, you’d need a team of analysts that have experience with these kind of complex models.
Secondly, and this may also be obvious, complex models are harder to understand than simple models. This makes it harder to understand whether all factors in the model contribute to the predictions in an intuitive fashion. In a linear model it’s easy to understand how each factor in the model influences the model predictions. But in a more complex model it can be challenging to understand how each variable influences the predictions. This creates the risk that the model predictions are based on unintuitive effects in the model and therefore create unintuitive and unintended outcomes. To illustrate this point, let’s consider the story of a marketing campaign that evaluated the customer base of a company on a regular basis and identified customers eligible for promotions. Initially, the marketing team developed a fairly simple framework based on a couple of statistical models and a handful of business rules. Over time, additional components were added based on new learnings and insights and changing business conditions. As a result, the complexity of the campaign grew considerably. At some point this resulted in a breakdown of the campaign as different factors started to conflict with each other. As a consequence, some customers that were eligible for a promotion were not selected to receive it. Others, that were not eligible, did receive it. Due to the complexity of the program, identifying the root cause was not a simple task. The marketing team had to take the program offline for a few months before launching it again. This also illustrated that it can be very challenging to diagnose a change in performance of an overly complex model or campaign.
But there are other drawbacks that may not be obvious when kicking off the development of such a model. Consider the case of a marketing team that developed a new model based on an ensemble technique. After several months they had developed a model that clearly outperformed the existing linear model. However, when they tried to onboard the model into their scoring platform, they found that they were not able to run the model. It was simply too complex for the scoring platform to handle. This created the need to make significant upgrades to the scoring platform, requiring a lengthy and expensive IT project. This in turn delayed the roll-out of the new model by several months.
So what is the lesson learned from these examples?
Marketers should balance the complexity of their marketing models and campaigns with the resulting performance. Extra complexity of a model or campaign should only be permitted if it results in meaningful superior performance. In other words, a marketing model/campaign should be as complex as needed to achieve the desired performance, but at the same time as simple as possible — you want extra bang for your buck. In reality, the hard part is often figuring out where to draw that line. How much revenue or profit is required to warrant the extra complexity? If you can increase your profit by 1-2%, but need a model that is 10 times as complex to achieve that gain, is that a recommended trade-off? In most cases the answer is probably no. But I’ve seen plenty of cases where the complexity/benefit trade-off what much less clear cut. That is why it’s important for marketers to consider the following aspects of the complexity/benefit trade-off before developing a complex model or marketing campaign:- Do you have analysts skilled enough to develop and maintain a more complex model? Does your situation allow for the extra time that may be needed to develop such a model?
- What are the operational limits of your scoring platform? Will you be able to run a more complex model in the required time without impacting other processes?
- How do you plan to monitor and diagnose the performance of your model? Do you have the tools and expertise in your team to diagnose a complex ensemble model?
- And finally, is your client or responsible executive comfortable with the increased complexity?
