Marketing programs rely heavily on A/B testing to understand and validate what layout, changes, or additions work best for their users. It’s one of the growth levers a business uses to increase it’s revenues, AOV, or conversion rate of a specific metric. It is also utilised to run sanity tests that verify nothing is broken when releasing a new version, tool, or script on the site.
In short - it’s essential to growth.
A/B tests show a specific treatment - and change you have made and want to validate - to random users while measuring how the change affects the metrics you are testing for. To make sure any specific user sees the same version of the experiment for the whole duration of the test a cookie is set that stores all sorts of information and behaviours related to the experiment.
A big question for businesses in established (located or targeting EU based users)in the EU is if one can run an A/B test considering that it requires you to drop cookies.
The ePrivacy Directive (this is not really a GDPR issue) says that you need to get consent for all not strictly necessary cookies. Yes, that is even for cookies that do not hold any personal data.
To make it all more fun the ePrivacy Directive is interpreted slightly different by all individual EU member states but most agree that strictly necessary means anything that is required to make the site function - nothing else.
Load-balancing is ok but analytics is not.
This would imply that you can not drop a cookie for you A/B test unless you have consent. And since you can not drop a cookie pre-consent so you have to wait to get consent before loading a test.
This eliminates running an experiment on your home page or landing page as you don't want to re-load a page and show the experiment once a user has given consent. It also limits your sample size to the users who do consent.
A Solution (or rather, my way of thinking on this)
(This is not legal advice and always check with your DPO or legal team first)
In short - run the experiment.
Or, consider a soft-opt in for A/B test cookies.
Yes, I’m essentially saying that even thought you are dropping a cookie and you did not ask you users consent you should run the experiment. Especially if you are eCommerce or SaaS.
Why?
Because countries are not on the same page regarding if A/B testing is an exemption to the "strictly necessary" cookie rule.
The ePrivacy directive is applied by each country as it is not a regulation that has to be enforced in the same manner across the EU.
For example the CNIL - that is the supervisory authority in France - has an exemption for cookies used for A/B testing. Whereas the ICO, the Brits, say clearly that you can’t use an exemption for A/B testing.
Some other countries have not been tested or have not voiced their opinion.
What do I do if I'm not in France?
You evaluate your risk.
It’s essential, when making any privacy vs growth decisions, to evaluate the risks involved.
In most scenarios I would say the risk is quite low for A/B testing considering:
Tests is only run for a limited time
One usually only collects aggregated data with no intention to understand users individual behaviours
Testing is expected by the user as it is a technique utilised to improve their user experience.
So....How?
(Again, make sure you run this by your legal team or DPO - I am not a lawyer and this is not legal advice.)
When trying evaluate if it is an experiment that could be run, keep it simple.
Ask your self the following:
What is my consent rate?
If your consent rate is above 50-60% and that is enough traffic to reach a decent sample size only run the experiment when you have consent.
Risk level: almost none
Is soft-opt in an option?
Can you use soft opt in to gather consent for testing?
Check you local laws (or the laws of the country you are targeting)
Is it possible to be clear and transparent so that the user knows what we are tracking and not?
Clearly communicate what it is you are doing when asking for soft-opt in.
What are the risks to your users data when running the experiment?
Mitigate any risks to your users data and only collect as little as you can get a way with.
When using soft opt-in make sure to:
Be clear and transparent about what you are doing
State why you are doing it (the purpose)
Add experimentation to your Privacy Notice
Make it easy to opt out
Risk level: medium-low
If all of the above fail
You don't have enough traffic with consent only and you don't think soft opt-in is an option?
Now what?
It all comes down to the amount of risk you are comfortable with. If the test is high value and the risk is relatively low (placing cookies with fast expiration, no personal data, etc.) then it might be worth it to run the test anyway.
