3 ways to setup parallel tests for API testing

Can you run parallel tests using Mocha test framework? Absolutely. In the world of Continuous Integration (CI) and Continuous Delivery (CD), where every engineering minute counts - it is very important to execute tests very fast, as many times as needed, and to get fast feedback on test results, without compromising test coverage or the quality of the test suite itself.

In order to decrease test execution time and to run tests in parallel, tests must not be dependent on each other, and running them in parallel must not interfere with test results.

how to run mocha tests in parallel?

This blog post will show examples of running parallel tests using the Mocha JavaScript test framework with HTTP request library – Supertest, and the different approaches to achieving that.

We will go through 3 different ways to achieve this in Mocha:

1. Using Mocha's new parallel mode, available since Mocha 8.0.0 version
   If using lower versions of Mocha:
2. Using npm package Mocha-Parallel,
3. Using npm package Mocha-Parallel-test.

creating a test environment

First, let’s set up our test environment step by step, and then we will try them all out and compare the execution times!

Setup:

We will need Node version >10, Supertest version 6.0.0, and for the first example, we'll use Mocha 8.0.0.

To simulate real application scenarios, we will use Reqres test API's - https://reqres.in/

1. The first step is to create a test project (I called mine Mocha-Parallel) and to install Node. I am using Node v14.15.0. Here you can follow the steps on how to install Node on your machine.
2. Once we have Node & NPM installed, we can install our Supertest and Mocha packages.
3. To install dependencies with NPM, we'll need to create a package.json file. To do that, enter the root folder of your project and Run $ npm init. You will be asked to fill out some fields, and then it will create a package.json file in the current directory.
4. All we need to do now to install our packages is to run the following commands in the terminal in the project root:

$ npm install supertest --save-dev

$ npm install mocha --save-dev

5. Now that we have everything installed, let's create a tests folder, where we will keep the tests and create several test files with extension .test.js

We will be using users' login, registration, and creation sample scenarios, so go ahead and name them users-login.test.js, users-registration.test.js, users-creation.test.js.

We won’t go too much into the details about Mocha and Supertest tests themselves in this blog, but if you have used similar JavaScript test frameworks, you might be familiar with Describe and It, where Describe is a test suite containing It blocks – actual test cases and assertions.

getting started with mocha api test framework

As we stated above, we will use Reqres test API’s to emulate real-world scenarios.

In the first test, we will be checking user login, user creation in the second, and user registration in the third.

In your editor you can add the following code in:

users-login.test.js

users-registration.test.js

users-creation.test.js.

These are very basic examples, but it should be sufficient for what we are trying to achieve here.

You'll notice that we added "sleep" in each test step, which is only to help us see the difference with parallel test execution. In reality, your test suite would probably be much larger, or you will need to wait for some action, like file upload, or your app would be communicating with other services over queue, REST, etc. This will just help us imitate those actions. The sleep function we are using is this:

ready to go! let's run the first test with mocha api test framework! 🙂

First, we’ll try to run the test in serial.

Open the terminal and run: 

npx mocha tests/*.test.js --no-timeout

Note: Mocha’s default timeout is 2 seconds so if the test is taking longer than that it will be marked as failed. Because we don’t want that right now, we will use --no-timeout

You should see something like this in your terminal:

Pay attention to the time in brackets - it took 19 seconds to execute these test suites in series.

running mocha tests in parallel mode

1. Mocha parallel mode

• Add --parallel to the test command: npx mocha tests/*.test.js --parallel --no-timeout, simple as that! 🙂

As you can see, execution time is reduced by more than a half! From 19 seconds to 7 seconds!

In this case, it might not look significant but imagine having a test suite that runs for 30 minutes, and that you are able to get it down to 15 minutes, or 10 or even 5 minutes! This would save a lot of time with multiple daily runs!

Mocha parallel mode was only introduced in Mocha version 8. If, for any reason, you are using an older version <8, you can achieve similar results with the npm package mocha-parallel-tests.

2. Mocha-parallel-tests

mocha-parallel-tests executes test files in a separate process.

Depending on the number of logical CPU cores that you have on the machine ( you can check it with os.cpus().length), you can limit the number of tests executing at the same time by using

--max-parallel. It is set to 0 by default, and that would start all tests at the same time. That can be pretty difficult for your machine if you are executing large test suites with a small number of CPU's, so it is not recommended by Mocha to leave at 0.

How to install mocha parallel tests?

You should simply follow the instructions below:

1. We first need to switch to a lower version of Mocha (we’ll use version ~7). Update version to ^7 in package.json, open the terminal in the root of your test project, and run $ npm install 
2. Run $ npm install mocha -parallel -tests --save-dev 
3. Run $ npm mocha-parallel-tests tests/*.test.js --no-timeout  and let's see the execution time! 

Similar to mocha parallel mode, it took 7 seconds to execute all tests in parallel. Not bad compared to 19!

One issue that I stumbled upon when trying to run all my tests in parallel was a very obvious one - they couldn't all be run in parallel! Even though we should aim to have independent tests – unfortunately, that's not always the case. Sometimes we lack test data, or running one test can interfere with the result of the other, if ran at the same time.

However, that doesn't mean that we can't still try to reduce test execution time!

Reducing test execution time when tests cannot all be run in parallel

One way to do it is to:

• group tests that cannot be run in parallel and have to be run in series, in a separate "container" test suites and then run those suites in parallel!

Let's see how that would look in our example!

Let say that we cannot run login tests and the user creation test at the same time, because users cannot log in if they are not yet created! Nevertheless, we can still test a user's registration at the same time as login and creation.

That means that we have to run user creation and users login in serial, but nothing stops us from running registration tests at the same time!

To do that, we will add a new folder "parallel-suites" and two different files - test1.parallel.js and test2.parallel.js , that would act as containers for our tests.

in test1.parallel.js add the following code:

and in test2.parallel.js add:

Let’s change the path in test command to *.parallel.js and run these 2 .parallel test suites:

$ npx mocha-parallel-tests tests/parallel-suites/*.parallel.js --no-timeout

You can see that it's not as efficient as running all tests in parallel, but it still saves us some time when running all test suites in parallel is not doable!

Finally, the last option I explored is mocha.parallel

3. Mocha.parallel

Mocha.parallel allows tests to be run in parallel, and it's compatible with Mocha 2.3.5 - 5.2.x.

Unlike mocha-parallel-tests and mocha parallel mode, tests are written a little bit differently, and it requires changes to the tests.

 The difference between mocha.parallel and the other two options is that mocha.parallel actually runs test cases - "it" blocks inside one test suite in parallel, while running different test suites in series.

This seems a little more complex as it would require that all steps in one test suite can be run in isolation.

In our case, it would first run "Create Users", then "Register Users" and then "Login with Users", but steps (it blocks) inside the suite, for example, "Register users" - "it('Verify status code 200 is returned when registering new user'" and "it('Verify status code 400 is returned when a new user is missing password'" would be run in parallel.

Let's set it up and see how it looks:

1. As stated above, it is supported only in lower Mocha versions, so we need to switch to Mocha version ~5. Update version to ^5 in package.json and open the terminal in the root of your test project and run $ npm install
2. Install mocha.parallel:$ npm install mocha.parallel --save-dev 
3. Update each test to use parallel – instead of Describe, like this:

1. run tests: $ npx mocha tests/*.test.js --timeout 5s

If you look at the test execution, you'll see that Create Users was executed first, followed by Register Users and Login With Users, but test steps in each suite were executed in parallel, resulting in a total time of 10 seconds. Depending on how you design your tests and the use case, this might be the choice for you.

Note from the creators: use of the word "parallel" is in the same spirit as other nodejs async control flow libraries. This library does not offer true parallelism using multiple threads/workers/fibers, or by spawning multiple processes.

Take care of your test strategy and choose the right scenarios to be tested

Let’s draw some conclusions together!

If your tests take too long to execute, there is also the possibility that test strategy is the one that needs improving first!

Before jumping into setting parallel tests, we should look at selecting the right scenarios to be tested, choosing the right candidates for automation, choosing the right tool, and adapting the approach of writing independent tests where possible - and we'll already be one step closer to running tests in parallel.

Each of the options that we explored has its pros and cons, limitations and advantages (Mocha versions limitation, report limitations, use of exclusive tests etc., explained in great detail on Mocha, git and npm websites). However, if you are looking for an option to speed up your test execution and hence speed up the development process and CI/CD pipeline while also saving that precious time - this could be something to look at (if you are using Mocha or looking for a new JavaScript framework for test automation! 🙂)

The goal is not only to save the dev time and increase team productivity, but to get faster feedback and get test results in a shorter time, which will allow the team to react faster if changes or code fixes are required. Or, in the best case scenario, to relax when you see those green builds sooner! 🙂