All software projects that produce GUI applications for users face the same problems:
1) How to maintain documentation, that is always up-to-date?
Containing current screenshots showing realistic data, telling a user story, explaining step-by-step how to use the application, also explaining the failure scenarios.
Even worse when the GUIs support multiple languages, in which case we need the screenshots and documentation in different languages as well…
2) How to test the GUIs automatically?
We want continuous, fast, automatic feedback of tests written to test the GUI, the validation, checking if errors and events are handled and displayed correctly …
The solution to these problems is never easy or cheap to achieve and depends on the GUI technology used in the software. We made our dream come true for our applications written in Adobe Flex and here you can read how our solution looks like.
The idea and the tools
The basic idea was, that the UI-integration-tests should walk through the application, telling a story, making the screenshots, doing assertions, and – as the result of a successful run – produce the documentation. To make the tests maintainable, their definition must be brief and simple, avoiding glue-code and low-level APIs to write a test, so that writing the tests is as much fun as implementing a feature.
We could not use Selenium here (we use Selenium to test web applications) for different reasons, instead, we implemented our own solution using
- FlexUnit
Used to execute the use-cases as tests, providing the features for assertions and asynchronous tests. - FxMarker
Used for template processing to format output, replace variables in documentation text. Unfortunately this framework is an early alpha version, so we had to do our branch to implement some fixes. - Adobe Air-Framework
Used to package our SWF-Flash application and starting them as a standalone application container to execute the tests and write the screenshots and documentation to files.
The test
The tests themselves are written in XML. We declare in a single document, how the test runner should remote control the GUI, click on buttons, enter values in fields, simulating events (that come from input devices like scanners or the hardware of the delivery machine when a locker is being opened or closed, when a receipt is being printed or when the credit card payment terminal is doing some transactions). The XML contains the text of the documentation manual for each step and the assertions to check on the way.
To test different languages, the test runner starts the test for each available language and provides some variables to the XML template that clicks on the adequate button for language selection as the first step in the test.
Here is the source of one of our tests:
<?xml version="1.0" encoding="UTF-8"?>
<spec title="Resending a lost TAN">
<intro>
A customer arrives at the machine to pick up his parcel. When asked to enter the TAN, he
discovers that he accidently deleted the message containing the TAN needed
to pick up his parcel. In this case, re-sending a new TAN to the customer is possible.
</intro>
<step title="Start">
<enter element="startFlow.localeChooser.selectedLocale" value="${vars.language}"/>
<intro>When the customer arrives at the machine, it shows the start page.</intro>
<screenshot>Start screen</screenshot>
<p>To start the pickup process, the user touches the "${app.startFlow.tanButton.label}" button.</p>
<screenshot element="startFlow.tanButton">Receive Shipment button.</screenshot>
<click element="startFlow.tanButton"/>
<wait/>
</step>
<step title="TAN input">
<intro>At this point, the customer is asked to enter the TAN. He discovers that he deleted the message.</intro>
<screenshot>The screen asking for a TAN.</screenshot>
<p>To get a new TAN, the user must click the "${app.tanFlow.tanLogin.resendTanButton.label}" button.</p>
<screenshot element="tanFlow.tanLogin.resendTanButton">Re-send pickup code button.</screenshot>
<click element="tanFlow.tanLogin.resendTanButton"/>
</step>
<step title="Entering the mobile number">
<intro>A Screen asking for the mobile number of the customer is shown.</intro>
<screenshot>Screen asking for the mobile number.</screenshot>
<p>At this point, the user enters his mobile number.</p>
<p>The entered mobile number must match the mobile number the original TAN was sent to to prevent fraud.</p>
<assert element="tanFlow.phoneNumberScreen.phoneNumberInput">
<exists/>
</assert>
<enter element="tanFlow.phoneNumberScreen.phoneNumberInput.value" value="00491773453453453"/>
<assert element="tanFlow.phoneNumberScreen.phoneNumberInput.value">
<equals/>
<value>00491773453453453</value>
</assert>
<screenshot>Screen after the mobile number has been entered</screenshot>
<p>When finished, the customer touches the "${app.tanFlow.phoneNumberScreen.forwardButton.label}" button.</p>
<click element="tanFlow.phoneNumberScreen.forwardButton"/>
</step>
<step title="Message screen">
<intro>The user is informed that the request has been successfully submitted.</intro>
<screenshot>Screen explaining the next steps.</screenshot>
<p>The user will receive a mesage with a new TAN within a short time.</p>
<p>The user touches the "Back" button.</p>
<screenshot element="tanFlow.resendTanRemote.cancelButton">Back button</screenshot>
<click element="tanFlow.resendTanRemote.cancelButton"/>
</step>
<step title="Entering the new TAN">
<intro>Now, the user can enter the freshly received TAN.</intro>
<p>After receiving a new TAN, the old one will not work any more.</p>
<screenshot>Screen asking for the TAN</screenshot>
<p>The user can now pick up his parcel using the new TAN.</p>
<click element="tanFlow.tanLogin.cancelButton"/>
<screenshot/>
<wait ms="5000"/>
</step>
</spec>
Reads quite simple, isn’t it? The FlexUnit test that starts the XML file (named „resendTan.xml“) contains nothing more than this:
public class TestMyApp extends FlexUITestBase {
[Embed(source="/tests/myApp/resendTan.xml",mimeType="application/octet-stream")]
public var resendTanSpec:Class;
public function TestMyApp() {
languages = [ "en_US", "ar_AE" ];
}
[Test(async,ui)]
public function runResendTanSpec():void {
runScenarioWithLanguages(XML(new resendTanSpec()), "resendTan");
}
}
… and another [Test] for each further use-case of our application to be tested. The output of a run is a folder of screenshots and an HTML file containing the documentation with links to the screenshots.
Here it is: The English documentation of the „resendTan“-use-case with Arabic screenshots.
As you can see, the XML contains ${} expressions to include the actual labels/values of GUI elements (e.g. button labels, etc.) into the documentation. This is done using FxMarker. The element can be used to implement assertions that could make the test fail.
Putting the things together
Applications consist of many use-cases. Each test case tests a single use-case, but the user manual should contain them all in a specific sequence as a single document or bundled into documents for different groups of readers (administrators, customers, etc.).
We use the xml-maven-plugin to concatenate the body parts of the HTML files and adjusting the screenshot links to create a documentation file (as HTML, PDF, or whatever format) using XSLT transformation.
So whenever we commit some changes to our application, our continuous integration server builds the application and generates current documentation automatically!
If you are interested in more details of the solution (the flex-UI-runner, the scenario parser, the XSLT transformation, …) just contact us.