Integration with Azure DevOps for Continuous Integration

Although it has always been possible to integrate AscentialTest with Azure manually, the new AscentialTest Azure pipeline task extension makes that integration much easier to accomplish.

The extension configures and runs the AscentialTest command lines to execute tests and generate reports. It also outputs a set of results files to summarize ‘passed’, ‘failed’, ‘blocked’ and ‘not run’ tests to make it easy to integrate with other extensions for generating custom reports.

The AscentialTest Azure extension can be installed from Microsoft Azure Marketplace.

How to Use the AscentialTest Azure Extension?

• From the [Pipelines Tasks] panel, select ‘AscentialTest’ from the list of available tasks.
  
• The task panel captures all of the parameters necessary to configure the ‘atrun’ and ‘atreports’ command lines. The task extension provides support for running either a ‘Test Set’ or an individual ‘Test’.
  

The Task panel indicates required fields with a symbol and provides ‘help’, using standard the Azure tool tip.
If the user clicks on the icon, a help panel will display to describe the associated parameter.
When the user completes the Task Panel and clicks on the ‘Add’ button, the configured task is displayed in the pipeline as pictured in the image below:

When the pipeline is triggered, ‘atrun’ and ‘atreports’ will be executed accordingly.

Once the run is complete, the reports selected in the new plugin panel will be available in the ‘Report Path’ folder as set in the task panel.

In addition to running the command lines to execute tests and generate AscentialTest Test Set reports, the extension also generates a set of result data files that can be graphed using third-party plotting tools:

Each of these files is updated for every run, recording the number of tests with the respective outcomes:

Publishing AscentialTest Results in JUnit Format for Integration with Azure

To facilitate integration of AscentialTest results with Azure Dev Ops dashboards, the following project components are available from Zeenyx Support.

1. CreateXMLResultsFileInJUnitFormat
   This test creates a results file in XML-JUnit format. It should be included as the first test at the top of each Test Plan:
   
   
2. WriteTestResultInJUnitFormat
   This function updates the results file with the result of the current test:
   
3. Appstate Requirements
   In order to implement the provided project components, the following lines need to be added to the OnFinish() function of your appstate:
   
   Test appstate needs to be further modified to add a new String member called sErrorMsg. The OnException() in the base ‘Test’ class needs to be overridden to set the value of sErrorMsg as displayed below:
   
4. Project Data Requirement
   In order to implement the provided project components, ResultsFileName and ResultsFilePath string parameters must be added to the Project Data:
   
   Below is an image depicting the results as displayed in Azure DevOps. This example was generated using the ‘Publish Test Results’ Task.
   
   

Azure Agent Configuration

To successfully run the AscentialTest Azure extension, you will need to target a self-hosted agent in interactive mode. Running a self-hosted agent as a service, although more convenient, will most likely result in permissions problems when attempting to run tests. We found in our trials that self-hosted agents run as a service are not allowed to interact with the desktop even if the related windows service setting is configured. This appears to be a limitation of Azure self-hosted agents.

If you have any questions about the AscentialTest Azure Pipeline Task Extension or would like a copy of the Junit Results generation, please contact Novalys Support at ascentialtest@novalys.net. We are here to support you and answer your questions.