Measure the effectiveness of new features with myTracker

News feed

Product analytics helps teams re-evaluate their vision for an app and further tailor it to audience expectations.

It's even better if you can draw connections between product and marketing metrics, because that’ll quickly show what attracts and retains users.

myTracker combines product and marketing metrics in a single interface and allows you to break them down by channel. There are tables and charts at your disposal that’ll tell you how the newly released feature is affecting user interest (for both existing and new audiences). You can see how well the update is doing with users, if it has made them more engaged, and if there are problems that urgently need fixing.

myTracker offers several ways to see how an update is performing, but for best results it is important to first understand what exactly you’re after. Is it more subscriptions, a lower abandonment rate, or longer sessions? Your concrete goal will dictate the metrics to track.

There are multiple ways to get insights into the effectiveness of a new feature using myTracker. Ideally, you’d want to mix and match to find the perfect combination for your app.

1. Custom events

Before a feature is shipped, mark it out with custom events:

  • A custom event will be triggered upon using the feature.
  • The event status will let you know that the new function is being used.
  • Its state will tell you that the function is no longer being used.

Custom events also allow you to configure additional parameters to see the user journey in more detail.

Let’s say, a person wants to subscribe, and it takes four screens to complete the process. There are multiple ways to get to each screen (different buttons, windows, drop-down lists). In myTracker, you can register a viewing of a screen without tracking how a person got there. This way you will see the total number of visits to a screen and how many people made it to the next one. If one screen proves especially “popular”, but not the next one, it may signal that people often go back a screen to perform a different action, or that the button for jumping to the next screen is not convenient enough.

By only tracking screen visits, you won’t see how successful the actions are, but will be able to quickly home in on any potential problems. This functionality is extremely useful for app features that span multiple screens with lots of ways to switch between them.

Another way of setting custom events is based on completed actions, meaning the event will be triggered when a user presses a button or chooses from a list. The tracking system will register a successful action as opposed to mere idling on a screen, and that in turn will reveal specific weak points such as button design or text not doing its job to entice users.

2. Rollout to 10–15% of users

It often makes sense to do a staged rollout of a new release. Why? Because of the issues and potentially critical errors that may be present in an update. Deploying your release in stages while carefully monitoring its performance with myTracker will make it easy to iron out any potential bugs with minimum inconvenience to users and little risk to the company.

Thanks to myTracker’s flexible Report Constructor, you can build reports for different versions of an app. This way you can compare key metrics as they were before and after the update, which will show the effectiveness of the new functionality.

Custom events can help you discover the user journey and see if people can make it to the end. If at some point they encounter a bug that prevents them from accessing some functions, you will know where that roadblock is.

When deploying a new (test) release, pay attention to DAU, retention rate, and the usage frequency for the new feature. The latter will tell you if the addition is useful, and any decrease in DAU or retention will indicate that users don’t like the update.

Moreover, myTracker supports segmentation of users based on their device specs, allowing you to compare their behaviour and localise problems that have to do with a specific manufacturer, screen resolution, or OS version.

3. Measuring the number and engagement of users

myTracker’s “Unique events” metric gives you the exact number of users or devices that make the target journey. You can see the number of unique views of a screen or presses of a button you're tracking, which can then be compared to the total number of users.

To check the usage frequency for a feature (button, function, screen), you might want to calculate the ratio of the total number of custom events to the total number of unique events. This will exclude the users who repeat actions, and produce an average indicator.

A high frequency is not necessarily good. It may signal confusion on the part of the user, leading them to return to a screen or repeatedly press the same button.

Both abnormally high and abnormally low engagement is a red flag that requires investigation. As part of a detailed analysis, try to look at the data at a different angle. It’s possible there are no errors in the app, and the higher frequency of an event is due to the update’s release and the associated rise in installs.

4. Additional methods to track effectiveness

There are times, especially in the case of apps with small user bases, when developer actions muddy the key indicators. It’s particularly noticeable during updates and deployment of new features, as developers or testers potentially skew the metrics by running the app to identify faults.

If you are serious about your analytics, roll out updates often and want your data to be unaffected by testing, we recommend using myTracker with two different apps (two different SDK keys, one for the test pre-release build, the other – for the main, stable version).

Having a separate version of the app plugged into the tracking system offers you much greater freedom. First, it’ll allow for a clear and easy distinction between the testing data and the actual user statistics. Second, it’ll simplify the export of raw event data for external analysis (fewer events = less data to export).

See here to find out about creating requests to export raw data. 

5. Audience segments

Audience segments in myTracker let you create any user samples to identify patterns that are not evident in standard reports. Set aside the users who’ve used a specific function and filter data and metrics based on this sample to build reports.

You’ll only work with the data on the group you’ve sampled. For example, you can filter by users who’ve used your app in the last 30 days, then look separately at those who triggered custom events related to the new function. Now, you can compare the users who’ve used the new feature to those who haven’t.

Have those using the new feature started making a lot less purchases in the app? Is there an increase or decrease in subscriptions? You can also compare the churn rates and see changes in LTV and retention, number of transactions, and days active.

Life Time (LT) metrics would be a useful tool here. They allow you to view statistics on the audience attracted over a certain period and to assess the effectiveness of related initiatives (ad campaigns, promotions and, of course, updates). In addition, LT metrics can be used in the same way as cohorts – to monitor how groups of users interact with the new feature.

For details on the metric types (Flow, LT, AT, CA), see here.

To create a new user segment for analysis, go to the Reports tab in the Segments section. Here, you’ll just need to configure the filtering options. It can be a custom event for interacting with the new feature, device model, app version, in-app purchase amount, subscriptions, age, gender, etc.

If you then want to build a report using the data on a specific user segment, go to Report Constructor and pick this option under Other filters → Segment.