Across the globe, CMOs and performance marketing managers are attempting to take their marketing budgets and distribute them across various performance marketing platforms to drive increased growth.

One of the core challenges faced when making growth an objective, whether short or long term, lies in understanding the incremental value of a channel. In-platform reporting – where a given media buying platform, such as Google Ads or Meta, only sees performance within its walled garden – usually doesn’t reflect reality.

Running controlled experiments, like GeoLift tests, can help to understand real performance. The results from these experiments can be used to calibrate internal attribution models, leading to improved performance insights and an understanding of how to most efficiently distribute marketing budgets, all of which ultimately drives growth.

Having run hundreds of experiments for companies of all sizes, in this article we explore the process of setting up a GeoLift test on Meta, an open souce tool available to advertisers.

Read on to learn the basics, how to prepare for a test, setting up a test, analysing and interpreting results and understanding best practices.

When deciding the optimal test and control markets to use for your analysis, you will need to:

1. Use a clean and representative dataset to conduct your initial analysis

Use recent data and ensure you have ample volume in order to be able to draw statistically significant conclusions from it. A good starting point is taking daily data from the last 6 months (you may want to exclude any anomalous periods from the analysis, such as flash sales and Christmas).

2. Understand local marketing efforts

When analysing performance, ensure you take into account any local media efforts that could skew the data e.g. TV, regional offers and offline marketing campaigns.

3. Classify the locations appropriately

Map the performance data to Commuting Zones/DMAs/GMAs (Generalised Marketing Areas). All of these concepts are geographic classifications that enable us to account for commuting distances and thus reduce the risk of spill/contamination of users between groups during our test. In this example, we use the idea of GMAs (which divide the UK into 82 unique marketing areas) for us to determine the optimal test regions & design.

Note: due to its size and unique characteristics, London (and more specifically here, GMA 1) is often excluded from targeting within GeoLift tests, as there are often no areas that can act as a fair comparison for the other arm of the experiment.

For more information on commuting zones please refer to this documentation.

4. Output interpretation

Once the optimal KPI has been selected, and the data has been cleaned (step 1), it is ready to be passed into the GeoLift tool to identify the appropriate test region(s) and interpret the associated power analysis. As part of this process, multiple different conditions needs to be defined with key inputs detailed below:

• N: list of the number of locations that could be tested – if unsure provide a large range.
• Treatment_periods: dictates different lengths of test duration that you deem appropriate for your business.
• Effect_size: the range of effect sizes you expect to observe. If you only expect a small uplift, you can define this here.
• Include_markets & exclude_markets: these are two parameters where you define regions you do or don’t want to be considered in the test. As an example, here you might want to define that GMA 1 (London) should not be considered.
• Holdout: a range of the relative size you are happy for the holdout/control region to be e.g. if you want a low risk test, you might want the holdout to be large.
• CPIC: cost per incremental conversion i.e. how much we typically need to spend on average in order to achieve one incremental conversion.
• Budget: what is your maximum total budget for a test of this nature?

Once the above is fed into the model, a set of different simulated test designs are returned with the key information summarised as follows:

• abs_lift_in_zero (Abs. Lift): this parameter evaluates the most recent treatment_period as if it were the test. It then measures the difference between the synthetically generated control from the holdout regions vs. the achieved results from the test. Ideally you want this value to be 0%, as there should be a perfect correlation.
• Investment: the minimum budget required to achieve the results desired. This value is heavily influenced by the inputted CPIC, so if this figure is somewhat unknown it is better to overinflate the CPIC than under-estimate it.
• EffectSize: represents the minimum lift needed to have a well-powered test. In most instances, we want this to be as small as possible so we only need to observe a small change to have statistically sound outcomes.

Once the optimal test design is selected the test regions to be targeted can be visualised, helping to ensure a fair geographic distribution. For example, we may realise we do not want to have all our target geographies in the north of England, as it might not be an accurate reflection of the market. Two examples of potential outputs are shown below:

After the geo-locations are validated, you can visualise the absolute lift for the most recent treatment cycle (shown below for 30 days) to see how well the synthetic control represents the actuals. Ideally here you want the synthetic control (the dotted purple line) to perfectly mirror the actuals (solid green line).

Collating the results into a single unified view allows you to paint a picture of the testing landscape and determine whether or not the suggested test design is appropriate for the business case in hand. As an example, if you are looking to test the incremental impact of Meta using a £5,000 budget but spend £200,000 across all other paid media it is highly unlikely you’d observe an effect size of 5% based on the business landscape.

Power analyses are often used in marketing experiments to determine the optimal sample size for a test, and from there the optimal test length, given a variety of minimum desirable effect scenarios. GeoLift operates in a somewhat similar way; enabling the user to state the optimal testing window from which simulations will then suggest the likelihood of a test being successful in the given timeline.

It’s worth noting that the ideal test length heavily depends on product and vertical, as these factors will influence factors like conversion lag. A good rule of thumb when deciding duration however is to ensure that the test period can contain at least one full purchase cycle. If the purchase cycle is not fully known, a conversion lag/maturity analysis can be run to understand and confirm the data.

The key to maximising the effectiveness of your GeoLift test can often lie in maintaining a simple and robust campaign structure. By focusing on a singular objective and minimising other variables, you create a more controlled environment for accurate measurement. This allows you to easily isolate the effects of your advertising efforts and leads to more reliable results.

Setting clear KPIs and regularly monitoring performance metrics is crucial when conducting your test.

Adopting an iterative testing approach can also bolster validity. This would involve starting with a pilot test, analysing the data from it, and then using that to inform the approach taken on follow up tests. This helps refine your strategy and mitigates the risk of drawing incorrect conclusions.

Capitalising on Meta’s built-in experimentation tools to run parallel or A/B tests can further help to verify results, offering an additional layer of validation and ensuring that the insights gleaned are reliable.

Analysing test vs. control performance

Using GeoLift, the pre-test approach is very similar to the post-test approach, except with new input parameters. Instead of trying to determine the optimal test regions, here we pass in the: treatment locations, treatment start & end times, and a dataset that covers all of the pre period, learning phase and test periods.

From this the package builds a synthetic control from your untreated GMAs (Generalised Marketing Areas) and then compares the test regions to this control to determine the average effect of the treatment with associated confidence:

One of the additional benefits of GeoLift is its ability to improve the model fit even further and reduce bias. There are several options for augmentation of the standard GeoLift model such as regularisation and an application of a Generalised Synthetic Control Model (GSC).

While each of these approaches provides its own set of advantages – for instance Ridge regularisation usually performs well when the number of units and time-periods aren’t large, while GSC helps improve fit for situations with many pre-treatment periods – GeoLift offers the option to let the model decide which is the best approach by setting the model parameter to “best”.