Testing my product’s features with CutwormBDD and Swift
I’ve been using Gherkin feature files to define the functionality of Garner, my career focused reflective journal app. These features and scenarios have been used to build the prototypes for the app and to define the analytics approach I’ve used with PostHog.
In the original blog in which I wrote about my decision to go down the Gherkin route I mentioned that I wanted to use the features and scenarios to create an automated test suite to form an executable specification and build a living document of the application.
In the past I’ve used a framework like Cucumber to achieve this but I don’t really like the complexity that Cucumber adds as you have to juggle the same step action being performed in different contexts.
After a bit of searching in the Swift Package Index for a Gherkin based testing framework I came across CutwormBDD which allows you to define your tests in the usual XCTestCase manner and annotate the test with the steps being performed.
This meant that if there was a different context for a step across two tests I didn’t need to manage this with a context object passed across tests, I could just write the test and add the annotation. A much nicer approach for me.
Setting up Cutworm to test a Gherkin Feature & Scenario
The first step is to install Cutworm via the Github URL in Xcode’s package management UI — https://github.com/KaiaHealth/cutworm-bdd.git
Once you’ve added the Cutworm package then you need to ensure that your test target (and if you don’t have a test target, add one) has the Cutworm library added under “Frameworks and Libraries” in the target’s “General” screen.
This will get Cutworm added to the test target but you still need to ensure that it has access to the Feature files it’s going to execute against.
To add the Feature files you need to head to the “Build Phases” section of the target’s configuration and under “Copy Bundle Resources” add the feature folder, it’s important that when you do this you select “Copy items if needed” and “Create folder references”. I also had to use the “Add other” option in the initial dialog to be able to select the folder.
Once you’ve got Cutworm added to the test target and set up the copying of your Feature files the next step is to write a test!
Cutworm comes with a handy GenerateScenario_EXPERIMENTAL() function which when passed the scenario name will add the appropriate step annotation to the test. I found this to be quite buggy however as if you have multiple scenarios in your test suite then it will overwrite them all with the last scenario.
You don’t have to use the GenerateScenario though, as long as you have the Feature and Scenario annotations added it will correct you if it cannot find a step and I found it easy enough to add the step annotations manually.
With the step annotations added you can then add the UI Test interactions and assertions as you would normally, using the Cutworm step annotations as a means of arranging the code to show which parts of the scenario they call under.
Tips for working with Cutworm
Cutworm is pretty easy to work with but as with any BDD framework there are potential downsides to be mitigated.
One Test Suite per Feature
In the setUp method of BDDTestCase you define the Feature being executed. To keep things simple I created on test suite per feature, this kept the file length short and focused on that feature.
If you have very long features with complex scenarios then you could also look to break the Feature across a number of test suites but I like the cohesion of one feature per test suite.
Cutworm doesn’t support the full Gherkin specification
I’m a big fan of But steps, but (no pun intended) Cutworm doesn’t support them, it only supports And so I found myself having to rewrite my scenarios to handle this.
When I have some time I feel like this would be an easy thing for me to add to the library so I'm going to attempt to do this, although my Swift knowledge is still pretty basic.
Cutworm’s README has a set of limitations listed https://swiftpackageindex.com/KaiaHealth/cutworm-bdd#user-content-limitations
Use Page Object Models to reduce duplicate interactions
When I started learning how to write XCUITests I found the framework really easy to use and it felt like my tests could just be written querying the UI elements and strings in the app.
As I started writing more and more tests it became clear that I needed some form of structure to reduce complex series of actions into re-usable steps across multiple tests.
To achieve this I used a pattern common in test automation — the Page Object Model. This pattern creates a model of a “page” (screen would be more apt) which has methods for interacting with the UI of that page/screen.
Within the test you can then call these methods to perform the actions and the test code reads well as you can see which action was being performed on which page/screen.
I created POMs for each screen in my app as well as a POM that represented the tab bar, with the POMs for each tab under it. In my test suite I then created an instance of the app POM with the XCUIApplication and this allowed me to write clean automation code.
Next Steps for my Cutworm usage
Now I’ve got Cutworm executing tests and giving me feedback of the features of my app behaving as I expect them to I have a few more things I want to build into those checks.
- I want to verify that the logging I’ve added to my app is being fired off correctly as the different flows through the app are triggered, this would involve expanding my Page Object Models to check the log output
- I want to verify that the analytics events that I’m using to measure how users are using features are being fired at the correct time with the correct information. This again would involve expanding my Page Object Models to check that PostHog is being called with this data
- I want to add screenshot generation at interaction points in the tests so that I can use these in visual regression tests and also in support documentation
- I want to build a means to extract the test results from running Cutworm in CI so that I can then upload these results into a testing management tool like X-Ray so that I can have a test execution history
