During a recent conversation with another developer I had Notion suggested to me as a really good alternative to Atlassian’s Jira and after having a play around with it and seeing the templates it offers I was completely sold on it being the one tool I needed.
Notion for those who aren’t familiar with it is essentially a wiki that has loads of different blocks and templates for displaying different information and these blocks and templates can be powered by databases (essentially just lists of different blocks).
Compared to Jira and confluence this makes it really flexible to work with as within the same space you can have a page for art assets, a page for specifications, a page for development notes and a page for test reports.
One of the things that really interests me about testing is test reporting and how it’s unfortunately way too common for test reports to look like a big (hopefully green) pie chart to show everything is working but never really explaining what that actually means in terms of the feature set of the product.
At the time of me playing with Notion there wasn’t a test reporting template available publicly so I set myself the goal of building one.
Building a test reporting template in Notion
Building templates in Notion is pretty simple.
You create a new page, add a table to the page and edit the properties of the table’s entries.
The entries in the table can then be turned into a template
You can then use the template to add more instances of the template to the table.
The tricky part of building something like a test reporter is figuring out how to visualise the data and as Notion doesn’t currently support group by arguments in its roll up properties how to show a count of each status.
I decided on showing the test reports in a board view as this supports grouping the cards by properties and the cards show the test case names so it gives a much better view on what’s passing and what’s not passing without the need for writing complex formulas.
A filter can be applied to the board to ensure that only items associated with the currently selected item are shown.
Building a test reporter to integrate with Notion
Notion at the time of writing does not have an official API but is a number of unofficial libraries for working with the notion API.
As I’ve previously built a PyTest reporter for my Phillips Hue lights and the
notion Python library is one of the better libraries available for the Notion API I decided to build another PyTest reporter.
When the test runner starts up the reporter tells Notion to create a new instance of the test cycle template which will hold the test executions run.
Once the test has been run the reporter will create the test execution instance and link the test execution to the test cycle, and if the user has used the
@pytest.mark.notion_test() decorator on their test case, the test execution will linked to the test case.
If the test fails then the stack trace of the test is added to the test execution instance.
Finally at the end of the test run the status of the test cycle is updated and a Linked Database block is added to the page with a filter so only the test executions ran during the test run are displayed.
As the test cycle is linked to the test executions it’s pretty easy to see which tests were run and which tests failed as the unexecuted and failed columns of the board view show this.
Clicking into the test execution instance will then show any information such as console logs or attachments.
As the test execution is also linked to the test case, clicking into the test case object shows all the test executions associated with the test case and the status of the test execution, so the health of functionality under test can be determined.
Additionally a calendar view of the same test execution table shows how often the test case is being executed.
The test case can then be linked to other entities such as specifications which then allows full bi-directional traceability between the tests and specifications.
Improvements and wants
Here’s a few things that could be improved that would make Notion a truly killer app for test reporting.
An official API library with extensive documentation would be the biggest improvement for my existing approach as this would not only make things easier to work with (some of the functionality in Notion is hidden away in the
notion library) but also Notion supporting an API would mean there’s no risk of user’s of an API library suddenly finding it no longer works.
Group by on database in roll ups would be a really good win for those looking to manually manage their tests within Notion as this would allow for a summary of the different test execution statuses in a linked database.
Grab it and have a play
One of the great things about Notion is that you can share your pages and having others use these as templates.
Have a play with my Test Reporting template on Notion and be sure to share any improvements you make on it.