Use Docker Container for Test Studio Test Execution

Test Studio test lists can be executed in Chrome Headless mode within a Docker container. Read below how to prepare the environment and how to start testing in a Docker container.

Please visit our website for more information on this topic.

The prerequisites from Test Studio and Docker side are listed below. Make sure that you comply with them, before proceeding with the next steps.

• Docker container can run on a physical or virtual machine with Windows installed.
• Installed Docker on the machine, where tests will be executed.
• Active Test Studio Run-Time license and .msi installer for it. The minimum version of Test Studio is 2021 R3.

Executing tests in a Docker container has its specifics and limitations. It is better to know about them before you proceed.

• Tests in a Docker container can be executed only in Headless mode, because the container does not have UI. This is the Chrome Headless browser type in Test Studio.
• Element's image find logic is not supported in Docker container test runs.
• Image verification step is not supported in Docker container test runs.
• Text from image verification step (OCR) is not supported in Docker container test runs.

Download Docker from the official website. Then, follow the instructions and install it on the machine that will be used for executing tests.

You can refer to the Docker documentation for commands and questions related with that software.

Below are listed the steps to setup a Docker container for testing.

1.  Switch Docker to use Windows containers, instead of the default Linux containers. Once you switch it successfully, the option will show as "Switch to Linux containers...", meaning that the current state is Windows containers.

2.  Pull the official Microsoft Windows Docker image from the Docker Hub. Use the command that is provided in the Docker Hub and make sure to specify the exact build to be pulled.

//For Windows build 1909, use the following command
docker pull mcr.microsoft.com/windows:1909

Note

The build version of the Windows image to install in the Docker container must match the Windows build on the current host machine.

The Microsoft Windows Docker image does not support "latest" tag. For further information check the documentation on the DockerHub's page for this image.

3.  Create a container from the official Microsoft Windows image. To do that, go to Images in the Docker desktop application and click on the Run button.

Then, expand the Optional Settings and specify Host Path and Container Path. The Host Path is a folder on the machine, which is available to the Docker container and the Container Path is how you will reference this resources from the Host Path within the container.

4.  Copy the .msi installers of Test Studio Run-Time and Chrome Enterprise in the Host Path folder. They become available in the Container Path in the Docker container and are ready to be installed. Open the Command Line Interface (CLI) of the container and navigate to the Container Path folder.

Install both Test Studio Run-Time and Chrome browser enterprise edition in passive mode, because there is no UI in the container and active installation can not complete. You can use the following command, but make sure that each installer is finished without errors before you proceed with the next one.

msiexec.exe /i msiInstallerFileName.msi /passive /le c:\tools\errorLogForThisInstaller.txt

Note

Additional information about the msi installers can be found on the Microsoft documentation.

Note

Tests and test lists execution in the Docker container is only supported for Chrome Headless mode.

After completing the above steps in this article, you need to copy the project, from which you want to execute tests and test lists, to be available in the Docker container. Place the project folder in the Host Path folder on your local machine and they also show up in the Container Path folder.

Open the container's Command Line Interface (CLI) and use the ArtOfTest.Runner.exe to start the execution. This test runner provide multiple options for execution and output, which are all explained in the linked documentation. The below example command triggers a test list execution.

//navigate to the Test Studio installation bin sub-folder
ArtOfTest.Runner.exe /list="full path to the test list file with extension .aiilist"

You can apply specific settings for Docker container test list execution, like the browser type. Since the test list execution runs only in Headless mode, you can choose different settings file to be used from the ArtOfTest.Runner.exe. Using such file allows you to not change the Test List Settings on your original test list and still execute it in Headless mode in the container.

Software evolves over time and automated testing is an essential prerequisite for Continuous Integration and Continuous Delivery. Developers write various types of tests, such as unit tests, integration tests, performance tests, and E2E tests for measuring different aspects of the software.

Usually, unit testing is done to verify only business logic. And depending on the part of the system that is tested, external dependencies tend to be mocked or stubbed.

But the unit tests alone don’t give much confidence because the actual end-to-end functionality depends on various external service integrations. So, integration tests are used to verify the overall behavior of the system by using real dependencies.

Traditionally integration testing is a complex process that can involve:

• Installing and configuring the required dependent services such as databases, message brokers, etc.
• Setting up the web or application server
• Building and deploying the artifact (jar, war, native executable, etc) on the server
• Running integration tests

With Testcontainers, you can have the lightweight experience and simplicity of unit tests, combined with the reliability of integration tests running against real dependencies.

Why is testing with real dependencies important?

Tests should enable the developers to verify application behavior with quick feedback cycles during the actual development activity.

Testing with mocks or in-memory services not only gives the wrong impression that the system is working fine but can also to significantly delay the feedback cycle. Tests using real dependencies exercise the actual code and give more confidence.

With competitive price and timely delivery, Cell Instruments sincerely hope to be your supplier and partner.

Explore more:
Ultrasonic Thickness Gauges
The True Cost of Seismic Surveys

Consider a common scenario of using in-memory databases like H2 for testing while using Postgres or SQL Server in production. There are a couple reasons why this is a bad practice.

1. Compatibility Issues

Any non-trivial application will leverage some of the database-specific features that might not be supported by in-memory databases. For example, a common way to apply pagination is using LIMIT and OFFSET.

SELECT id, name FROM employee ORDER BY name LIMIT 25 OFFSET 50

Imagine using the H2 database for testing and MS SQL Server for production. When you test with H2, the tests will pass, giving a wrong impression that your code is working fine. But it will fail in production because MS SQL Server doesn’t support LIMIT … OFFSET syntax.

2. In-memory databases may not support all the features of your production database

Sometimes applications use database vendor-specific advanced features which may not be fully supported by in-memory databases. Examples can include XML/JSON transformation functions, WINDOW Functions, and Common Table Expressions (CTE). In these cases, it’s impossible to test using in-memory databases.

These frequently grow into even larger problems when you’re mocking services in your own code. While mocks can help test scenarios where you can successfully extract the mock definition to use as a contract for services, this verification of compatibility oftentimes only adds complexity to the test setup.

The typical use of mocks won’t allow you to reliably verify that the your system behavior will work in the production environment. It also won’t give you confidence in the test suite’s ability to catch issues caused by code incompatibilities and third-party integrations. 

So, it’s strongly recommended to write tests using real dependencies as much as possible and use mocks only when needed.

Testing with real dependencies using Testcontainers

Testcontainers is a testing library that enables you to write tests using real dependencies with disposable Docker containers. It provides a programmable API to spin up required dependent services as Docker containers. This way, you can write tests using real services instead of mocks. So, regardless of whether you’re writing unit, API, or end-to-end tests, you can write tests using real dependencies with the same programming model.

Testcontainers libraries are available for the following languages and integrate well with most of the frameworks and testing libraries:

• Java
• Go
• Node.js 
• .NET
• Python
• Rust

Case study

Let’s see how Testcontainers can be used to test various slices of an application and how all of them look like “Unit tests with real dependencies”.

We’ll use example code from a SpringBoot application implementing a typical API service that’s consumed via a web app and uses Postgres for storing data. But since Testcontainers provides you with an idiomatic API for your favorite language, a similar setup can be achieved in all of them.

Treat these examples as illustrations to get a feel of what’s possible. And if you’re in the Java ecosystem, then you’ll recognize the tests you’ve written in the past or take inspiration on how you can do it.

Testing data repositories

Let’s say we have the following Spring Data JPA repository with one custom method.

public interface TodoRepository extends PagingAndSortingRepository<Todo, String> {
   @Query("select t from Todo t where t.completed is false")
   Iterable<Todo> getPendingTodos();
}

As we mentioned above, using an in-memory database for testing while using a different type of database for production isn’t recommended and can cause issues. A feature or query syntax supported by your production database type might not be supported by an in-memory database.

For example, the following query (which you might have in your data migration scripts) would work fine in Postgresql but will break in the case of H2.

INSERT INTO todos (id, title)
VALUES ('1', 'Learn Modern Integration Testing with Testcontainers')
ON CONFLICT do nothing;

So, it’s always recommended to test with the same type of database that’s used for production.

We can write unit tests for TodoRepository using SpringBoot’s slice test annotation @DataJpaTest. We’ll do this by provisioning a Postgres container using Testcontainers as follows:

@DataJpaTest
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
@Testcontainers
class TodoRepositoryTest {
    @Container
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:14-alpine");

    @DynamicPropertySource
    static void configureProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.datasource.url", postgres::getJdbcUrl);
        registry.add("spring.datasource.username", postgres::getUsername);
        registry.add("spring.datasource.password", postgres::getPassword);
    }

    @Autowired
    TodoRepository repository;

    @BeforeEach
    void setUp() {
        repository.deleteAll();
        repository.save(new Todo(null, "Todo Item 1", true, 1));
        repository.save(new Todo(null, "Todo Item 2", false, 2));
        repository.save(new Todo(null, "Todo Item 3", false, 3));
    }

    @Test
    void shouldGetPendingTodos() {
        assertThat(repository.getPendingTodos()).hasSize(2);
    }
}

The Postgres database dependency is provisioned by using Testcontainers JUnit5 Extension, and the test talks to the real Postgres database. For more information on using container lifecycle management see Testcontainers and JUnit integration.

By testing with the same type of database that’s used for production, instead of using an in-memory database, the chance of database compatibility issues is avoided altogether and increases the confidence in our tests.

For database testing, Testcontainers provides special JDBC URL support which makes it easier to work with SQL databases.

Testing REST API endpoints

We can test API endpoints by bootstrapping the application along with the required dependencies such as the database provisioned via Testcontainers. The programming model for testing REST API endpoints is the same as the Repository unit test.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@Testcontainers
public class TodoControllerTests {
    @LocalServerPort
    private Integer port;
    
    @Container
    static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:14-alpine");

    @DynamicPropertySource
    static void configureProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.datasource.url", postgres::getJdbcUrl);
        registry.add("spring.datasource.username", postgres::getUsername);
        registry.add("spring.datasource.password", postgres::getPassword);
    }

    @Autowired
    TodoRepository todoRepository;

    @BeforeEach
    void setUp() {
        todoRepository.deleteAll();
        RestAssured.baseURI = "http://localhost:" + port;
    }

    @Test
    void shouldGetAllTodos() {
        List<Todo> todos = List.of(
                new Todo(null, "Todo Item 1", false, 1),
                new Todo(null, "Todo Item 2", false, 2)
        );
        todoRepository.saveAll(todos);

        given()
                .contentType(ContentType.JSON)
                .when()
                .get("/todos")
                .then()
                .statusCode(200)
                .body(".", hasSize(2));
    }
}

We’ve bootstrapped the application using the @SpringBootTest annotation and used RestAssured for making API calls and verifying the response. This will give us more confidence in our tests as there are no mocks involved, and it enables developers to do any kind of internal code refactoring without breaking API contact.

End-to-end testing using Selenium and Testcontainers

Selenium is a popular browser automation tool for performing end-to-end testing. Testcontainers provides a Selenium module that simplifies the execution of selenium-based tests in a Docker container.

@Testcontainers
public class SeleniumE2ETests {
   @Container
   static BrowserWebDriverContainer<?> chrome = new BrowserWebDriverContainer<>().withCapabilities(new ChromeOptions());
 
   static RemoteWebDriver driver;
   
   @BeforeAll
   static void beforeAll() {
       driver = new RemoteWebDriver(chrome.getSeleniumAddress(), new ChromeOptions());
   }
 
   @AfterAll
   static void afterAll() {
       driver.quit();
   }
 
   @Test
   void testViewHomePage() {
      String baseUrl = "https://myapp.com";
      driver.get(baseUrl);
      assertThat(driver.getTitle()).isEqualTo("App Title");
   }
}

We’re able to run Selenium tests using the same programming model with the WebDriver provided by Testcontainers. Testcontainers even makes it easy to record videos of the test execution without having to go through a complex configuration setup.

You can take a look at the Testcontainers Java SpringBoot QuickStart project for reference.

Conclusion

We looked at various types of tests that developers use for their applications: data access layer, API tests, and even end-to-end tests. We also discovered how using Testcontainers libraries simplifies the setup to run these with the real dependencies like the actual version of the database you’ll use in production. 

Testcontainers is available in multiple popular programming languages for example Java, Go, .NET, and Python. It also offers an idiomatic approach to transforming your tests with real dependencies into unit tests that developers know and love.

Testcontainers-based tests run the same way in your CI pipeline and locally, whether you choose to run an individual test via your IDE, a class of tests, or even the whole suite from the command line. This gives you unparalleled reproducibility of issues and developer experience.

Finally, Testcontainers enables writing tests using real dependencies without having to use mocks which brings more confidence to your test suite. So, if you’re a fan of a practical approach, check out the Testcontainers Java SpringBoot QuickStart, which has all the test types we looked at in this article available to run from the get-go.

Want more information on Container Tester? Feel free to contact us.

Learn more