Contents
A/B testing in mobile apps shows whether a change actually works – instead of letting the loudest opinion in the meeting decide. A randomly selected share of users sees variant A, the rest see variant B, and a metric defined in advance determines the winner. This article explains what is worth testing in apps, how the technical setup works and which mistakes render results worthless.
When A/B tests pay off – and when they do not
An A/B test needs enough users to tell a real difference from chance. How many depends on the baseline rate and the expected improvement: small effects on rare events require very large samples, clear effects on frequent events far fewer. Online sample size calculators give you a realistic estimate before you start.
A/B tests are particularly worthwhile when:
- a meaningful number of new or active users reaches the tested area every day,
- the decision matters commercially, such as paywall, price or onboarding,
- the team disagrees and there are several plausible variants.
They make less sense with very small user numbers, for obvious bug fixes or for changes that are legally or technically required anyway.
What you can test in apps
| Area | Example hypothesis | Primary metric |
|---|---|---|
| Onboarding | Without mandatory sign-up, more users reach the core action | Activation rate |
| Paywall | Pre-selecting the annual plan increases revenue per user | Revenue per user after 30 days |
| Trial | A longer free trial increases paid conversions | Conversion after the trial ends |
| Push notifications | Personal wording brings back more inactive users | Return within 3 days |
| Features | A new search function increases completed bookings | Bookings per active user |
| Store listing | Different screenshots increase installs | Store page conversion |
Prioritise tests with high leverage. A different button colour rarely moves anything measurable; a different onboarding flow often does. Our article on app onboarding optimization offers practical starting points.
Technical setup: remote config, feature flags and store experiments
Unlike on the web, you cannot ship an app update instantly; every release goes through store review and reaches users with a delay. That is why variants in apps are almost always built in upfront and switched on remotely:
- Remote config and feature flags: on launch, the app asks which variant a user should see. Services such as Firebase Remote Config with A/B Testing or dedicated feature flag platforms handle assignment and analysis; alternatively, the logic can live in your own backend.
- Server-side tests: when content, prices or recommendations come from the server, the variant can be decided there. This is flexible and independent of app versions.
- Store experiments: Google Play offers store listing experiments and Apple offers product page optimization. Both let you test icons, screenshots and copy without changing the app.
Make sure assignment is stable (a user stays in their variant), there is a default variant in case the configuration fails to load, and old flags are removed once a test ends. Otherwise the number of code branches quickly turns into technical debt.
Step by step to a clean experiment
- Write a hypothesis: “If we change X, Y will increase because of Z.” Without a reason, you learn little even from a clear result.
- Set one primary metric: exactly one success measure that decides win or loss.
- Define guardrails: metrics that must not get worse, such as crash rate, cancellations or refunds.
- Plan sample and duration: calculate the required number of users and run for full weeks to even out weekday effects.
- Check assignment: are users split across variants as planned? A clear deviation points to a technical bug.
- Analyse at the end: do not check significance daily and stop at a favourable interim result.
- Decide and document: roll out the winner, record the result and the learning, remove the flag.
Common mistakes that distort results
- Peeking: checking repeatedly and stopping at the first “significant” value produces many false positives.
- Too many metrics: look at twenty metrics and one will almost always stand out by pure chance.
- Novelty effect: existing users click on anything new at first. New users often give the more honest picture.
- Wrong level: optimising paywall taps instead of revenue after 30 days can lead to more trials and fewer paying customers.
- Inconsistent tracking: if events are captured differently on iOS and Android, you are comparing platforms rather than variants. A clean tracking plan, as described in our article on app analytics KPIs, is a prerequisite.
Privacy and pricing experiments
Experiments rely on usage data. Where this involves accessing information on the device, the consent rules of the GDPR and – in Germany – the TDDDG apply; our article on tracking consent in apps explains the details. A pseudonymous ID is sufficient for assignment; personal data does not belong in the experiment configuration.
Pricing experiments call for extra care: prices for in-app purchases and subscriptions are managed as products in the stores, so variants have to be set up as separate products. You should also review consumer-law questions and the impact on customer trust before users see different prices for the same service. This is not legal advice. How to approach subscription pricing in general is covered in our article on subscription app pricing.
Agentic coding workshop for your team? From €799 net – request a quote by e-mailConclusion
A/B tests in apps pay off as soon as enough users reach the tested area and the decision matters commercially. Define hypothesis, metric, guardrails and duration upfront, implement variants via remote config or server-side, and only analyse at the end. If you want experimentation infrastructure built into a new app from the start, we are happy to include it in our fixed-price offer.
Frequently Asked Questions
How many users do I need for an A/B test?
That depends on the baseline rate of the metric and the expected effect. A sample size calculator shows in advance how many users each variant needs; for small effects, that can easily be several thousand.
How long should an A/B test run?
At least until the planned sample size is reached, and ideally for full weeks so that weekday fluctuations even out. Stopping early at a favourable interim result distorts the outcome.
Do I need a new app release for A/B tests?
For the initial setup, yes, because the variants must exist in the code. After that, tests can be controlled via remote config or the backend without shipping a release for every change.
What if my app has too few users for A/B tests?
Rely on user tests, interviews and larger, clearly distinguishable changes whose effect you observe cautiously in a before-and-after comparison. Store experiments can be an option if your store page gets more visitors than the app has active users.