app-entwickeln-lassen.de LogoApp entwickeln lassen

Growth & Monetization

A/B Testing in Mobile Apps: Data-Driven Optimisation

app-entwickeln-lassen.de editorial team 6 min read
Contents
  1. When A/B tests pay off – and when they do not
  2. What you can test in apps
  3. Technical setup: remote config, feature flags and store experiments
  4. Step by step to a clean experiment
  5. Common mistakes that distort results
  6. Privacy and pricing experiments
  7. Conclusion
  8. Frequently Asked Questions

A/B testing in mobile apps shows whether a change actually works – instead of letting the loudest opinion in the meeting decide. A randomly selected share of users sees variant A, the rest see variant B, and a metric defined in advance determines the winner. This article explains what is worth testing in apps, how the technical setup works and which mistakes render results worthless.

When A/B tests pay off – and when they do not

An A/B test needs enough users to tell a real difference from chance. How many depends on the baseline rate and the expected improvement: small effects on rare events require very large samples, clear effects on frequent events far fewer. Online sample size calculators give you a realistic estimate before you start.

A/B tests are particularly worthwhile when:

  • a meaningful number of new or active users reaches the tested area every day,
  • the decision matters commercially, such as paywall, price or onboarding,
  • the team disagrees and there are several plausible variants.

They make less sense with very small user numbers, for obvious bug fixes or for changes that are legally or technically required anyway.

What you can test in apps

AreaExample hypothesisPrimary metric
OnboardingWithout mandatory sign-up, more users reach the core actionActivation rate
PaywallPre-selecting the annual plan increases revenue per userRevenue per user after 30 days
TrialA longer free trial increases paid conversionsConversion after the trial ends
Push notificationsPersonal wording brings back more inactive usersReturn within 3 days
FeaturesA new search function increases completed bookingsBookings per active user
Store listingDifferent screenshots increase installsStore page conversion

Prioritise tests with high leverage. A different button colour rarely moves anything measurable; a different onboarding flow often does. Our article on app onboarding optimization offers practical starting points.

Technical setup: remote config, feature flags and store experiments

Unlike on the web, you cannot ship an app update instantly; every release goes through store review and reaches users with a delay. That is why variants in apps are almost always built in upfront and switched on remotely:

  • Remote config and feature flags: on launch, the app asks which variant a user should see. Services such as Firebase Remote Config with A/B Testing or dedicated feature flag platforms handle assignment and analysis; alternatively, the logic can live in your own backend.
  • Server-side tests: when content, prices or recommendations come from the server, the variant can be decided there. This is flexible and independent of app versions.
  • Store experiments: Google Play offers store listing experiments and Apple offers product page optimization. Both let you test icons, screenshots and copy without changing the app.

Make sure assignment is stable (a user stays in their variant), there is a default variant in case the configuration fails to load, and old flags are removed once a test ends. Otherwise the number of code branches quickly turns into technical debt.

Step by step to a clean experiment

  1. Write a hypothesis: “If we change X, Y will increase because of Z.” Without a reason, you learn little even from a clear result.
  2. Set one primary metric: exactly one success measure that decides win or loss.
  3. Define guardrails: metrics that must not get worse, such as crash rate, cancellations or refunds.
  4. Plan sample and duration: calculate the required number of users and run for full weeks to even out weekday effects.
  5. Check assignment: are users split across variants as planned? A clear deviation points to a technical bug.
  6. Analyse at the end: do not check significance daily and stop at a favourable interim result.
  7. Decide and document: roll out the winner, record the result and the learning, remove the flag.

Common mistakes that distort results

  • Peeking: checking repeatedly and stopping at the first “significant” value produces many false positives.
  • Too many metrics: look at twenty metrics and one will almost always stand out by pure chance.
  • Novelty effect: existing users click on anything new at first. New users often give the more honest picture.
  • Wrong level: optimising paywall taps instead of revenue after 30 days can lead to more trials and fewer paying customers.
  • Inconsistent tracking: if events are captured differently on iOS and Android, you are comparing platforms rather than variants. A clean tracking plan, as described in our article on app analytics KPIs, is a prerequisite.

Privacy and pricing experiments

Experiments rely on usage data. Where this involves accessing information on the device, the consent rules of the GDPR and – in Germany – the TDDDG apply; our article on tracking consent in apps explains the details. A pseudonymous ID is sufficient for assignment; personal data does not belong in the experiment configuration.

Pricing experiments call for extra care: prices for in-app purchases and subscriptions are managed as products in the stores, so variants have to be set up as separate products. You should also review consumer-law questions and the impact on customer trust before users see different prices for the same service. This is not legal advice. How to approach subscription pricing in general is covered in our article on subscription app pricing.

Agentic coding workshop for your team? From €799 net – request a quote by e-mail

Conclusion

A/B tests in apps pay off as soon as enough users reach the tested area and the decision matters commercially. Define hypothesis, metric, guardrails and duration upfront, implement variants via remote config or server-side, and only analyse at the end. If you want experimentation infrastructure built into a new app from the start, we are happy to include it in our fixed-price offer.

Frequently Asked Questions

How many users do I need for an A/B test?

That depends on the baseline rate of the metric and the expected effect. A sample size calculator shows in advance how many users each variant needs; for small effects, that can easily be several thousand.

How long should an A/B test run?

At least until the planned sample size is reached, and ideally for full weeks so that weekday fluctuations even out. Stopping early at a favourable interim result distorts the outcome.

Do I need a new app release for A/B tests?

For the initial setup, yes, because the variants must exist in the code. After that, tests can be controlled via remote config or the backend without shipping a release for every change.

What if my app has too few users for A/B tests?

Rely on user tests, interviews and larger, clearly distinguishable changes whose effect you observe cautiously in a before-and-after comparison. Store experiments can be an option if your store page gets more visitors than the app has active users.

This article was written by the editorial team of app-entwickeln-lassen.de – the app and AI development service of jocapps GmbH, based in Freiburg, Germany, and building apps since 2014. The content is for general information and does not replace individual (legal) advice.

You might also like

Growth & MonetizationHow to Improve App Ratings on the App Store and Google Play

Improve app ratings: when to ask for reviews, what the App Store and Google Play prohibit and how to handle criticism – for more stars and more downloads.

6 min read
Growth & MonetizationApp vs. Website: Which Generates More Revenue?

App vs. website revenue: which channel earns more depends on purchase frequency and business model – with comparison table, calculation and decision guide.

6 min read
Growth & MonetizationDeep Links and Web-to-App: Guiding Users Seamlessly Into Your App

Deep links and web-to-app explained: how to set up and test Universal Links, Android App Links and deferred deep links – for fewer drop-offs and more app usage.

7 min read
Growth & MonetizationApp User Retention Strategies: How to Keep Users Coming Back

App user retention strategies: how to measure retention in cohorts, spot churn early and use onboarding and push to keep more users active for longer.

7 min read
Growth & MonetizationSEO for Web Apps: Ranking on Google Despite JavaScript

SEO for web apps: how server-side rendering, clean URLs and good Core Web Vitals make JavaScript applications visible on Google and in AI search.

6 min read
Growth & MonetizationApp Marketing Strategy: The Launch Plan for Your First 90 Days

App marketing strategy for launch: what to prepare before going live and what to do in the first 90 days to win users, keep them and make growth measurable.

6 min read