Novalys is a distributor of AscentialTest, a Zeenyx product

Integration with Azure DevOps for Continuous Integration

Although it has always been possible to integrate AscentialTest with Azure manually, the new AscentialTest Azure pipeline task extension makes that integration much easier to accomplish.

The extension configures and runs the AscentialTest command lines to execute tests and generate reports. It also outputs a set of results files to summarize ‘passed’, ‘failed’, ‘blocked’ and ‘not run’ tests to make it easy to integrate with other extensions for generating custom reports.

The AscentialTest Azure extension can be installed from Microsoft Azure Marketplace.

How to Use the AscentialTest Azure Extension?

  • From the [Pipelines Tasks] panel, select ‘AscentialTest’ from the list of available tasks.
    Pipelines Task Panel
  • The task panel captures all of the parameters necessary to configure the ‘atrun’ and ‘atreports’ command lines. The task extension provides support for running either a ‘Test Set’ or an individual ‘Test’.
    All Parameters for Task Panel

The Task panel indicates required fields with a Mandatory Field Symbol symbol and provides ‘help’, using standard the Azure tool tip.
If the user clicks on More Information Symbol the icon, a help panel will display to describe the associated parameter.
When the user completes the Task Panel and clicks on the ‘Add’ button, the configured task is displayed in the pipeline as pictured in the image below:

Configured Task Pipeline Display

When the pipeline is triggered, ‘atrun’ and ‘atreports’ will be executed accordingly.

Once the run is complete, the reports selected in the new plugin panel will be available in the ‘Report Path’ folder as set in the task panel.

In addition to running the command lines to execute tests and generate AscentialTest Test Set reports, the extension also generates a set of result data files that can be graphed using third-party plotting tools:

Test Set Reports

Each of these files is updated for every run, recording the number of tests with the respective outcomes:

Test Number Records

Publishing AscentialTest Results in JUnit Format for Integration with Azure

To facilitate integration of AscentialTest results with Azure Dev Ops dashboards, the following project components are available from Zeenyx Support.

  1. CreateXMLResultsFileInJUnitFormat
    This test creates a results file in XML-JUnit format. It should be included as the first test at the top of each Test Plan:
    XML Results JUnit Format
    Create XML Results File in JUnit Format
  2. WriteTestResultInJUnitFormat
    This function updates the results file with the result of the current test:
    Current Test Result Update
  3. Appstate Requirements
    In order to implement the provided project components, the following lines need to be added to the OnFinish() function of your appstate:
    Appstate Requirements - Add lines
    Test appstate needs to be further modified to add a new String member called sErrorMsg. The OnException() in the base ‘Test’ class needs to be overridden to set the value of sErrorMsg as displayed below:
    Override Test Class
  4. Project Data Requirement
    In order to implement the provided project components, ResultsFileName and ResultsFilePath string parameters must be added to the Project Data:
    Implement Project Components
    Below is an image depicting the results as displayed in Azure DevOps. This example was generated using the ‘Publish Test Results’ Task.
    Azure DevOps Result
    Publish Test Results

Azure Agent Configuration

To successfully run the AscentialTest Azure extension, you will need to target a self-hosted agent in interactive mode. Running a self-hosted agent as a service, although more convenient, will most likely result in permissions problems when attempting to run tests. We found in our trials that self-hosted agents run as a service are not allowed to interact with the desktop even if the related windows service setting is configured. This appears to be a limitation of Azure self-hosted agents.

If you have any questions about the AscentialTest Azure Pipeline Task Extension or would like a copy of the Junit Results generation, please contact Novalys Support at We are here to support you and answer your questions.