How to simulate impact of different channel spends using an estimated MMM

[nheusch-se]

Hi all,

using an estimated MMM, I'm conducting "what-if" analyses and got a little stuck on how to do it in pymc-marketing. I would really appreciate your help and thinking on this.

Let me outline what I'm after:

I have an MMM (estimated using data up to Nov 2024) and would like to know "what does the model think the difference in sales in December would have been if instead we had spent 0 on channel 1 in the 4 weeks of December".

If doing this with pen and paper I'd do the following:

1. Sample alpha, beta, lambda (for that channel) from the posterior
2. given these parameters calculate: contribution of channel 1 in every week of December using the actual spends on channel 1 in December
3. given these parameters also calculate: contribution of channel 1 in every week of December, when I've set the spends to 0 for the 4 weeks of December (this might still be positive due to adstock at the start of the December)
4. calculate the difference and save it

Repeat this 1000 times.

This would give me 1000 estimates of the impact of setting the spend to 0. It is a distribution (i.e. 1000 estimates), because there is uncertainty about the parameters alpha, beta, lambda.

Is there any way to do this in pymc-marketing?

Here is what I'm doing so far, but it feels unfair to the model and gives me an excessive amount of uncertainty.

So far I'm using sample_posterior_predictive twice to estimate channel contributions for December, once using my actual channel spends and once using a copy of the data, where I set the channel spends for channel 1 to zero. I then extract the contributions for the channel and calculate their difference. However, this feels weird because I'm not comparing apples-to-apples.

# create two versions of channel spends for December: actual and hypothetical
# X_test are my actual channel spends in December
X_test_counterfactual = X_test.copy()
X_test_counterfactual['channel1'] = 0

# predict outcomes and estimate channel contributions for December using actual spends:
y_dec = mmm.sample_posterior_predictive(
    X_pred=X_test, extend_idata=True, include_last_observations=True, 
    var_names=["channel_contributions", "control_contributions", "mu", "y"],
    test_data = True, 
    progressbar=False
)

# predict outcomes and estimate channel contributions for December if spends on channel 1 had been 0:
y_dec_counterfactual = mmm.sample_posterior_predictive(
    X_pred=X_test_counterfactual, extend_idata=True, include_last_observations=True, 
    var_names=["channel_contributions", "control_contributions", "mu", "y"],
    test_data = True, 
    progressbar=False
)

channel_contribution_dec = y_dec['channel_contributions'] \
    .sel({'channel':'channel1') \
    .sum(dim='date') # sum the estimated channel contributions over the 4 weeks in December (gives me an array of length 1000, because I sampled 1000 times from the distribution of alpha, beta, lambda)

channel_contribution_dec_counterfactual = y_dec_counterfactual['channel_contributions'] \
    .sel({'channel':'channel1') \
    .sum(dim='date') # array of length 1000, as above

diff_due_to_channel1 = channel_contribution_dec - channel_contribution_dec_counterfactual 

This is a weird comparison, because for example channel_contribution_dec[0] and channel_contribution_dec_counterfactual[0] were estimated different sampled values of alpha, beta, lambda (as was probably every other pair channel_contribution_dec[k], channel_contribution_dec_counterfactual[k]). Hence, the difference between them hence does not only result from different channel spends, but also different parameter values for alpha, beta, lambda.

Thus, diff_due_to_channel1 gives me an excessively wide distribution. A quick fix might be to just sort channel_contribution_dec and channel_contribution_dec_counterfactual and then take their difference, but ideally I'd like to compare apples to apples, i.e. estimated channel contributions when alpha, beta, lambda are identical. (And do this for different values of alpha, beta, lambda).

Does this make sense to you and is there a way to do this? I'd really appreciate your help!!