Critical View on AI Tools for Product Management

In our work during the discovery phase, we aim to reduce product risks regarding usability, value, and feasibility, which is critical for both our clients and the end-users. However, nowadays the IT market is cooling down and many companies decide to reduce their fundings and cut the scope of product design. Unfortunately, many times it leads to limiting, or even skipping the scope of the discovery process, inherently increasing a product’s risks. 

Therefore, the obvious question was how we could be more productive, work faster, and deliver quality results despite lower budgets. The hypothesis was to leverage the capabilities of AI tools used during interview preparation, conducting interviews, analyzing results, and forging them into backlog items 

The next question was how well such tools can do that. This article explores the existing capabilities of some of the AI solutions. We will try to compare them and assess their usefulness in the process of user research. 

QoQo.ai 

QoQo is a Figma plugin that supports researchers in the discovery process. It allows you to quickly summarize client briefs, generate your own brief, and prepare templates of interview questions using the AI engine. Such template can be used by researchers in the course of user interviews, and the results can be summarized by QoQo. Based on prompt input, it generates cards to build a persona with user goals, needs, motivations, frustrations, opportunities, and tasks. It can summarize the transcript of the interview and cluster highlights. We can even automatically build journey maps and information architecture of solution to be designed.  

The “how could we?” feature is especially interesting. The tool can generate ideas to overcome obstacles or pain points that users encounter while interacting with a product or service. 

One thing that was not working for me was the discovery questionnaire feature that was just fixed text, given by QoQo without any domain context, and no AI was used for that.  

UX experts are familiar with Figma and having such a tool is convenient. The pricing is an affordable $7/month. It's worth trying. 

Dovetail 

It is a well-known transcription solution allowing to manage interview results, get valuable insights, find patterns, and quantify them.  

I used this tool for a couple projects and I’m familiar with its interface. The ability to add highlights and group them on a separate bord is especially useful for interview analysis. I highly appreciate the feature of adding a custom vocabulary to be used for transcript creation. 

It can generate summary of the interview and assign tags for sentiment to statements using AI. However, I noticed that sentiment is based on keywords, and the platform does not understand general context.  

Please note that Dovetail is constantly working on AI features, and I have noticed that quality of summary improved in the last few weeks.  

Notably AI 

Yet another transcript tool. Similarly to Dovetail, it allows us to analyze the text and add AI-generated summary. It also cleans up the transcript removing unnecessary “filler words.”  

It also includes automatic tagging, but I was not getting any productive results, therefore I had to rely on manual tagging.  

The summary can be automatically translated, but the number of languages is still limited.  

Notably comes with a “templates” feature, which is, in fact, a Chat GPT overlay. You can use it to ask specific questions about your transcripts. Why they are called “templates” you may ask. That’s because Notably comes with pre-set prompts to be used with your transcripts. And some of such templates are available only on higher tiers.  

Honestly, that’s just like copy-pasting the text of your transcript to generative AI and providing instructions to get results.  

Entropik  

The company developed two products marketed as Decode (consumer research platform) and Qatalyst (user research platform). 

We tested Qatalyst, a tool that allows to conduct user interviews. The information provided on the website is limited, there is no trial, therefore a demo session with the company salesman was necessary.  

The potential seemed to be huge: the ability to build moderated and non-moderated test sessions, including Figma prototypes could be tested, and the results could be quantified with a description of the main and alternative paths of achieving an expected goal. The number of successful tests and dropouts, as well as the conversation transcription and sentiment are scored based of facial expressions. With eye-tracking hotmaps, we can check how much attention certain parts of the design are gaining.  

Sadly, we have run into some issues while rendering this platform: we were losing a session in the middle of the work, the preview of the test was not showing the whole content, and enabling camera didn’t work. After conducting eight interviews we have only received the results of two. 

The platform seems to have lots of potential but it’s in beta stage. I would recommend checking it out in a couple of months when it is more stable.  

Synthetic users 

This solution prepares full-fledged user research without any actual users. 

Now, that’s a very interesting idea: let’s create AI bots to act like real users. You can change their demographics, country of origin, and their characteristics (for example a 45-year-old man from Oregon with military experience, tech savvy, father of two). It can be used to identify problems and evaluate proposed solutions. The result of AI work looks just like well-led interview. 

My impression is that if I handed you such report, you would think that was a transcript of a real interview.  

The question is: would you rely on such data? As a discovery phase of your project needs to reduce the risk of introducing a product that does not fit users' demands, would you invest money in a project, that was not based on information gathered by interviewing real people? I would not, and having in mind that Synthetic Users exist, I would check the results of my researches’ work to ensure it was not AI-generated.  

On top of that: I had an unpleasant surprise when I tried to cancel my subscription – signing off simply does not work, and my emails to the company support remained unanswered. As a result, I had to restrict my card and apply for a new one. 

Userdoc 

Once you’ve conducted interviews and gathered requirements you can move forward and plan your development work. Userdoc is a tool for user stories, user journeys, and acceptance criteria creation with an AI assistance.  

First, you start with your project name and description. It’s good to be specific at this point. AI will suggest user personas and explain what they can do with the system. For me, the number of personas was insufficient, and I had to add a couple more.  

Then, at the title level, Userdoc creates drafts of user stories. The personas are enhanced by AI, adding their description, goals, and frustrations. Userdoc can then create user journeys, including multiple actors. Somehow the amount of space for user journey description was limited to only 512 characters. If you are working on a bigger, multi-faceted system, that’s certainly not enough.  

After user stories are created, AI generates acceptance criteria from User Story description, taking into consideration other stories and criteria. Of course, it won’t do everything you need in your project, but it can save time by adding draft versions of User stories for you to validate as well as other features you hadn’t thought of.  

In my opinion, this system is a valuable canvas to get the product creation process organized, eliminating the need for writing generic user stories, like CRUD or login/logout. It speeded up my process, allowing me to get a draft version of the backlog in an hour or so. On the other hand, more complex systems will be hard to map with Userdoc, but if you are building something relatively simple, like food delivery app, it might become handy. Unfortunately, the pricing starts at 99$/month 

Final thoughts 

I have not included in this article all of the tools I tested, because they were simple Chat GPT overlays, and their only value is that they eliminate the need of launching the next tab on your browser to write a prompt. Actually, similar the outcomes from these tools could be achieved by copying and pasting content into generative AI and asking for a summary. 

Overall, AI implementation in product discovery tools is in its early stage, but there are sparks of creativity and innovation resulting in the reduction of time spent on our daily work. Certainly, AI usage in product discovery could be a thing in the future, but we are not here yet, their capabilities are limited, and the promised value is hardly ever delivered.  

Personally, I would recommend trying QoQo for user interviews and ideas generation, Dovetail or Notably for transcript and tagging, and Userdoc for quick backlog creation.