You have an idea for a B2C product, for a mobile app even. But you're worried about bugs & bad reviews on app store. So, you only release it for a very limited audience. That's probably wrong...
As I described previously I came up with an idea, a value hypothesis, KPIs. Now it was time to confirm or discard that hypothesis. There are numerous ways of doing that, but I opted out for a mobile app as an MVP. That would be something tangible users could play with, while I would be gathering data on their usage.
However, mobile app development has certain barriers to entry. Unlike a website which you can just launch and make available to the whole world, a mobile app has to go through an approval process before it appears on an app store. All that sounded daunting when I quickly wanted to validate my idea. I was also worried about negative reviews because of bugs or less than ideal user experience. After all, mobile users are a few taps away from giving your app a 1 star rating. And that could really kill an early app idea, if people immediately see poor reviews. Or... so I thought at that time.
Luckily, I had a dozen of friends who were were also into food & drinks (my target group). So I had an audience I could test my app with.
As a result, I embarked on scoping features, content, platform and building an actual MVP (I’ll talk about it in a upcoming blog post). I also figured out a way to have a private distribution for both iOS and Android. (Unfortunately, my friends didn’t use a single platform, so I had to target both of them). Anyway, after some time, I had an MVP for my limited audience.
And I even organised a field testing session among a group of foodies. We spent a night wandering around town in search of the best wine and craft beer thanks to Best in Town.
And their phones... and beer
The idea and the app got a positive reception and it was great to observe people using your app.
I even got a few e-mails after from a few friends of friends. They gave me detailed feedback on the app and some new ideas.
But as the time went on my analytics system (Google Analytics at the time) registered very little activity. At that time I managed to sign up around 20+ users to the app. And if we look at the KPI we have set for validating the hypothesis it was a Monthly Retention. The hypothesis would be validated, if we achieved a 20% monthly retention rate.
Okay, so we had around 20 people and 20% of them would be what 4 persons? That hardly looks good from the point of view of statistical significance.
Your test audience should be large enough to make your KPIs statistically significant.
So this leads to the following points when it comes to releasing an B2C product to really small audiences. By the way, you may still have a limited launch - think of when Gmail was by invitation only. But probably, they released it to more than 20 people.
Some people may say ‘But what about the benefits of qualitative analysis’? It is useful indeed, but releasing an app for a general audience, does not exclude qualitative analysis.
I launched the app among the people I knew. We also did field testing together. Of course, it gave me lots of insightful information. I concur it was great to see how people were using the app. That really helped to spot various usability issues.
However, releasing an app to everyone does not exclude qualitative analysis. You can still have sessions in person with users to see how they use an app, while at the same time you gather quantitative data from all other users you acquired through app stores.
Jumping a bit forward the app is now on the market - both on Apple App Store and Google Play Store. There are close to 1,000 downloads in total. And there haven’t been any rating below 4 stars. There is the data that suggests that a ratio of rating per download can vary from 1 to 75 to 1 to 600 (1 rating per 600 downloads.
So, clearly there was nothing to fear.
It’s not enough to have 20 installs if you are working on a mass market B2C app. So, I decided to release the app on app stores and spent money on advertising to acquire users. I’ll talk about that in the future posts.
The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way. My personal thoughts tend to change, hence the articles in this blog might not provide an accurate reflection of my present standpoint.
© Mike Borozdin