Finding the right user research platform when bootstrapping your app idea

I’ve recently picked up an old app idea from 2020 (JiffyCV) with the aim of rebuilding it in SwiftUI and expanding on it with the ideas that I had before I burned out and ultimately stopped wanting to work on it.

When I built the first iteration of the JiffyCV I had conducted some “user research” to understand the relationships that people had with their CVs. I created a basic survey in Google Forms and asked people on my various social networks to fill it out.

I got some pretty promising results from the survey which then gave me enough confidence to execute the rest of the idea, however once I had the app out in people’s hands I started to realise the flaw in the approach.

I hadn’t asked enough qualifying questions in the survey to understand if those answering the survey were actually my target audience and because of the social networks used, there was a high likelihood that those who answered were not those I looked to serve with the app.

A year or so later after I burned out from building JiffyCV I found myself with another idea and decided to try a new approach. This was to use Maze, a user research tool to help me find the right messaging and design for the website for the idea I had by recruiting people from their pool of testers.

This was my first experience with using a tool that allowed me to recruit users to give me feedback and I mostly decided to go down this route as I was slowly running out of people I knew who had the time for me and my crazy ideas.

It turns out that when you’re starting out on your entrepreneurial journey and pivoting around multiple ideas you burn through a lot of social capital. The initial act of you striking out will garner interest, but once that doesn’t succeed and you change course it gets harder and harder to get people interested in providing feedback.

With a very small pool of people I could ask for feedback from I decided to look for tools that had pools of people I could recruit to get feedback from. A task that back in 2022 seemed quite hard, a lot of user research tools focused on asking existing users or website visitors and didn’t provide a means to recruit respondents.

At the time Maze had a free tier that allowed for three tests to be run and you could pay to recruit participants to provide the feedback. This allowed me to iterate a couple of times over the way I was trying to sell the idea and after those iterations I felt like I had a pretty solid website for the tool.

These iterations cost around £75 a go so it was a relatively big expense for an idea I hadn’t built yet but I saw this as a learning experience, and also it had a nice side effect of helping me define my idea better.

Since this experiment in 2022 I’ve changed jobs to a product company where I’ve had some exposure to Ballpark, a competitor of Maze. So when I decided to revisit the JiffyCV I wanted to use this as an excuse to determine which tool would be best suited to help me with my future ventures.

Building a user research survey

After failing to get the original JiffyCV off the ground and reflecting on what went wrong I read a book called “The Mom Test” by Rob Fitzpatrick, which looks to teach you how to validate your idea in such a way that even if you were to ask your mum for feedback that she would, regardless of her biasses be unable to give you feedback on your idea that would mislead you.

I really enjoyed reading that book as I could see exactly where I had gone wrong previously so when it came to writing my user research survey this time around I did my best to ensure that the questions were focused on uncovering the facts and not opinions and that they were not leading.

The questions I asked where:

• How often the respondent reviewed the information on their CV
• How long the respondent typically spent working in a role
• If the respondent recorded the achievements and challenges overtime in their current job
• What tooling they use to record their achievements and challenges at work if they do this

I then asked them to rank

• The features they saw as most important for an app that recorded their achievements and challenges at work
• Their biggest annoyances with apps

I felt that the questions would allow me to gauge if my idea resonated with what people were already doing and uncover what they used for this. The rankings would then allow me to see which of the features I was looking at would be most important to people.

I then signed up for both Ballpark and Maze and set up the user research survey so I could compare the two tools.

Ballpark

Initially I was a little worried about Ballpark’s offering, Maze offers the creation of three tests as part of it’s free tier compared to Ballpark’s one and while Ballpark includes 10 recruitment credits this limits your target audience as you can’t purchase more. Also the trial is limited to 14 days before your data is locked down.

When you use Ballpark’s free trial though you are given access to all the functionality of Ballpark’s starter tier, that is to say that you have all the functionality to ask filtering questions and to filter the audience that you recruit participants from.

This means that those 10 recruitment minutes (realistically 5 respondents) can be put to good use. For example I was able to target the audience I asked towards those currently in employment. This reduced the likelihood of those minutes being wasted on getting responses from those who weren’t actively in work and thus didn’t need the idea I was looking to validate.

Ballpark’s free tier also allows for conditional paths in the user survey so I was able to build into the survey some follow up questions to my question to if the respondent currently recorded their achievements and challenges in their current job. These follow up questions then allowed me to understand why those who didn’t record this information didn’t do so.

I was really impressed with Ballpark after using the free trial but after those 14 days your data is locked until you pay for their starter tier which at it’s lowest cost is £913 for a year. This year does come with 100 minutes however and then allows you to purchase more recruitment minutes should you need them but it feels like a steep price for someone who has the skills to develop an app themselves for next to nothing.

Maze

On paper Maze’s free tier seems like a better deal compared to Ballpark. You have three tests and your data isn’t locked down after a trial period and you’re able to purchase recruitment minutes on this tier so you can ask as many people that you can afford to ask without paying for the ‘platform’.

You don’t get any free minutes though so to get the 5 responses you’d get with Ballpark’s free trial you’d have to shell out $25 to do so. These respondents won’t be as effective as Ballpark’s though as Maze’s free tier does not let you filter the audience or let you build in any conditional flows.

I also had to add an additional question in my survey on Maze to capture if the respondent was currently in employment so I could filter out these responses (3/20 of the respondents I ended up getting).

While working on the survey in Maze I was kinda shocked at how much functionality was locked away behind the higher tier paywalls. You can only have 7 “blocks” (questions, context setting slides etc) in your survey, you can’t filter the audience, you can’t have conditional logic. Compared to Ballpark it felt quite restrictive, well aside from the data lock-in.

Decision time

Regardless of which platform I decided to use I was going to be paying around the same price for the lowest paid plan. For Ballpark this is $100 a month if paid annually or $128 a month if paid monthly, for Maze this is $99 a month with no discount for an annual purchase. I could also choose to stick with Maze’s free tier and recruit 20 users at $100 a month.

What you get for that price though is very different.

For Maze you’re getting the means to run 1 study a month, you can expand to more than 7 “blocks” and you can add conditional logic to your survey. You still need to purchase recruitment credits at $5 a respondent.

For Ballpark you’re getting the means to run 5 projects at a time (but can archive projects), you get conditional logic in your survey, you get the audience targeting and you get 100 recruitment minutes for the year. You can purchase more minutes if you run out at $1 a minute (so $2 a respondent for a small survey).

So while Ballpark is technically a little more expensive ($12 on the annual purchase) it actually provides more for the price because the cost to recruit is cheaper and the outcomes of that recruitment with the audience targeting will be better as you’re not wasting money on respondences that won’t have the experiences you need to know more about your idea.

Summary

I’m still in the early days of building up my user research skills but if someone were to ask me which platform to use to validate their idea I would put them in the direction of Ballpark over Maze.

I ended up purchasing a year subscription to Ballpark as I want to get a lot of feedback from potential users as I build the next iteration of JiffyCV and I think I can justify the expense to learn these skills.

I’d be interested in hearing from those who have used either Ballpark, Maze or both and how they found using these tools.