AI Mobile Testing Guide

Ensuring consistent mobile app performance can be daunting, given the thousands of device-OS combinations, unpredictable network conditions, and frequent app updates. Traditional testing methods struggle to keep up with the modern pace of work. Test automation—in which testing is triggered by code updates and/or set schedules—has emerged as one key step in solving traditional challenges. Automated testing is an excellent addition to an engineering team’s toolkit, but it is not a silver bullet by itself.

AI-driven testing is the next evolution of testing solutions. Traditional code testing techniques can require the (severely repetitive) effort of updating old test code every time the app code is updated. AI mobile testing scales far more efficiently across devices and environments by leveraging intelligent automation, self-healing tests, and real-time adaptation.

This article explores AI-powered automation and its ability to streamline testing efforts. We demonstrate AI mobile testing and show future applications in mobile environments.

Mobile testing challenges

The table below summarizes the challenges and solutions for traditional mobile testing.

Mobile testing environments are also quite complex, as many applications interact with the native device OS simultaneously. Advanced mobile environment considerations include:

Device shutdown handling

Devices are prone to sudden shutdowns, which can happen for many reasons. Many teams test their most critical workflows to ensure that ill-timed (or malicious) shutdowns can’t break their systems.

Certain app features can often be disastrous if a device shuts down improperly. For example, a mobile finance app developer might test how their app responds to shutdown when a user is processing a transaction.

Low-battery device states

Many things change when a device enters low-battery mode. Resources are allocated differently, background processes may be de-prioritized, and apps themselves may behave differently. This may even be a key product feature.

Resource-intensive device capabilities are generally used this way. In low-battery conditions, apps typically restrict built-in mobile device features like the flashlight, GPS, and compass functionality.

Network throttling

Globally, more users have limited connections (3G or 4G) than high-speed internet users. This is but one of many reasons to simulate slow network conditions. Unless a team’s user base is limited to a specific region, high-tech domain, or user type, it is best to assume that some or most users will have limited internet connections.

Scaling test suites

The manual testing of applications is always recommended, especially during major releases. For newer codebases, it is common to start with one or a few tests (usually a login test or a happy-path test of the application) and continue to write tests along with new code features. However, this technique doesn’t scale well on its own. Maintaining testing requires increasing effort, as new code changes require updating dozens of tests simultaneously. This is where AI automation can supercharge testing efforts. 

{{banner-large-dark-2="/banners"}}

AI mobile testing explained

AI mobile testing uses AI tools to automate the mobile testing process and mitigate traditional testing challenges. 

Testing begins with an application, sometimes called the System Under Test (SUT). The AI automatically generates test cases based on the app structure when activated. This is useful in itself, but the real power of AI automation is that it can automatically add and update tests as code updates. Continuous integration (CI) triggers enable the AI to adapt to UI changes as code commits are introduced. AI test automation tools can also handle cross-device and cross-browser inconsistencies.

The application lives within one of potentially many environments, known as the Environment Under Test (EUT). In automated testing, the engineer supplies the SUT and EUT details, then tests are run against them. You can test the same application across multiple environments so long as you have an automated way to retrieve the credentials for each.

Automated AI mobile testing analyzes the output of test suite runs, along with SUT and EUT details, and makes suggestions based on anomalies in these results. It doesn’t limit manual testing - instead, it standardizes and enhances it. 

Setting up AI mobile testing

This guide walks you through the mobile testing automation process using Qualiti, an AI software testing platform. Qualiti can automatically generate test coverage for an entire test suite. Its dashboard allows you to develop, organize, and manage tests.

The following sections provide step-by-step instructions for setting up a test suite and scheduling automated testing in an example application.

Collect test information

Begin by gathering the following information about the SUT.

In addition to these, consider any expectations around performance or SLA requirements. As a best practice, document this information in one place for historical reference. These questions will all come up in some form throughout any substantive testing effort.

Configure testing

Log in to the Qualiti dashboard and walk through test creation. On the “Set Up” page, enter the previously collected information.

Set up CI triggers

Github CI/CD is often used to trigger automated tests for continuous code evaluation. As code is committed to a codebase, the Qualiti integration triggers AI automation. Make sure to differentiate testing environments, especially if planning to test against all of them. This is common in scenarios like regular app performance testing.

In the dashboard, the “Integrations” tab lets you set up CI triggers and use the Qualiti API to trigger test plan runs on each commit. Qualiti provides a Bash script you can add to your pipelines. Use the "Create Integration" action on the “Set Up” page to generate an API key and client ID to add to your script. Once done, you will see this screen where you can delete the integration as needed.

See the Integrations Setup Guide for further reference.

Create user profiles

This step is optional but helpful for saving test user information outside your code repository. 

Some examples of user profiles include a default user, an admin user, a user with an expired subscription, and an unauthorized user (pictured below). Choose the credentials for the user profile and select your EUT for the environment.

Enter SUT details

Click on the “Components” tab to enter details about your SUT. This will require much of the same information you collected earlier. As needed, customize concurrency and timeout numbers to performance expectations.

For basic authentication, use the “User & Password” credential type. You can enter credentials as a simple username and password or as an advanced API authentication request.

To allow requests with cookies and local and session storage, you can enter API request credentials using the API authentication credential type.

Lastly, set up the steps for your app login. You can skip this step if you don’t have an authentication layer.

You can enter your login steps using code or plain English descriptions if you do.

Schedule test runs

Qualiti’s AI automatically generates tests as code is introduced. Each test consists of a series of steps. These tests are then grouped into different folders for clarity and quick lookup.

You can schedule testing for automatic runs under Test Plans (the “Plans” tab on the menu). These could be scheduled triggers—for instance, daily test suite runs—or triggered by new code commits using CI triggers. Let’s look at a test plan called the Core Feature Plan. The plan creation steps are given in a previous chapter on mobile testing tools.

A hybrid mobile app benefits from test coverage in both mobile and web browsers. In one experience, users might sign up through a web page and then be directed to install an app. Test plans allow you to schedule testing across multiple browsers.

Let’s select our Core Feature Plan. Currently, it only runs on Chrome. 

Let’s add the mobile Chrome browser environment.

Once this is done, your scheduled tests will run in mobile view, and any test failure reports will come with mobile Chrome screenshots. If errors occur, Qualiti’s platform automatically determines what has changed in the SUT and makes suggestions for you to review and confirm.

The image below shows what the dashboard looks like after running a test suite.

Self-healing tests and anomaly detection power AI mobile testing functionality in Qualiti.

{{banner-small-3="/banners"}}

Self-healing tests

Flaky tests intermittently fail with inconsistent error states (also known as “non-deterministic behavior,” in which we can’t always guess the outcome of running the code). This can lead to developer confusion, test suite unreliability, and delayed development releases.

Engineers learn to ignore errors in that test suite if one test gives frequent errors for no apparent reason. The problem is that when real issues emerge in the component covered by that test, engineers tend to ignore it, causing problems to escalate.

Such unstable tests are usually caused by subtle test code and environment issues. For example, tests dependent on timezone calculations may run successfully in local testing 100% of the time but fail on average once every 10 runs in CI/CD environments. The CI/CD environment uses the UTC zone, which sometimes causes the calculation results to break from expectations and sometimes doesn’t.

Qualiti automatically interprets test results to determine whether a potential defect exists within the app, test code, or environment. Anomaly detection automatically scans logs and test results for in-app bugs and defects. It identifies common errors instantly and alerts engineers before the code is moved to production.

This dashboard shows an example of anomaly detection in action.

Self-healing capabilities also have other uses beyond reducing flaky tests. For example:

• Adaptive handling of dynamic UI updates, such as changes to button placements, color schemes, and user segments. Traditionally, these changes would require manual updates of test code components.
• In regression testing, where engineers seek to prevent new code updates from breaking existing code.

What about functionality that AI can’t generate? 

Specific code, scenarios, or bugs may go outside the norms of computer programming. This is when a human should review and update the tests. For this purpose, Qualiti uses Playwright under the hood and runs Playwright tests directly from your codebase. 

Enhancing mobile tests with Playwright

Microsoft’s Playwright is an automated testing tool that allows you to set up an end-to-end test environment in minutes. It uses simple syntax for complex tasks like device emulation and automatic screenshot captures. 

Qualiti auto-generates Playwright tests, and it can manage existing Playwright tests in a codebase. You can add pre-written Playwright tests to your code repository when requirements go beyond standard testing techniques, like tracking performance with custom metrics.

The following code example tracks the First Contentful Paint on a web page. This metric measures how long it takes for content to appear on the page.

const { test, expect, chromium } = require('@playwright/test');
const { devices } = require('playwright');

test('Measure critical performance metrics', async () => {
  // Launch browser with iPhone 14 emulation
  const browser = await chromium.launch();
  const context = await browser.newContext({
    ...devices['iPhone 14 Pro'],
    viewport: { width: 393, height: 852 },
    deviceScaleFactor: 3 // Retina display simulation
  });

  const page = await context.newPage();
  
  // Navigate to test URL (use environment variable in CI/CD)
  await page.goto(testURL);

  // Capture First Contentful Paint (FCP)
  const FCP = await page.evaluate(() =>
  performance.getEntriesByName('first-contentful-paint')[0]?.startTime || 0
  
);

  expect(FCP).toBeLessThan(targetFCP);
  await browser.close();
});

‍

On the Qualiti dashboard, Playwright tests can be generated and edited in plain English. Writing and committing test code is optional. 

In the dashboard, tests are edited in the Test Library. Enter a description of what the test must cover, and Qualiti’s AI creates the Playwright script.

Auto-sorting tests

Qualiti’s AI can auto-sort tests into their Feature Folders. The AI can then suggest new tests and features for your application. 

To generate test suggestions, you will need to create a User Profile. Let’s create a new user profile called “Default Admin.” Your profile details will depend on the information you collected during the test setup.

Auto-generating tests

Navigate to the Test Library, where you can view all folders. 

Generate new tests based on your user profile in the Test Library.

You can then review any new tests and edit them further if necessary.

{{banner-small-4="/banners"}}

Conclusion

AI mobile testing increases testing efficiency and reduces engineering overhead. Nonetheless, it’s essential to do the proper research upfront and document all your requirements for your SUTs and EUTs. Do your best to narrow your scope of testing and set clear objectives.

Configure testing environments that closely match your production environment. Set up scheduled runs or CI triggers to activate continuous code coverage. AI testing's full power lies in its ability to review the large quantities of logs, errors, and data generated by each test suite run. Use AI to automate regression testing and eliminate the time spent updating old tests to match new code changes.

Selecting the right testing tool is one of the most pivotal decisions. Qualiti’s AI-driven automation provides a low-code, scalable solution for teams struggling with testing complexity, delivering robust test coverage with minimal engineering effort.