Continuous Product Discovery. Using Kanban to Run Experimentation Flows

Before you kick off Continuous Product Discovery

If you have not read our article on why we believe Continuous Product Discovery is the best way to kick off growth initiatives in your company, I recommend checking it out right now.

To summarise, if you ever performed a successful Design Sprint workshop, you know that a week spent deep-diving into a specific product dilemma can yield breakthrough results. Results usually are attainable in a typical product lifecycle in weeks or even months rather than five days. The purpose of Continuous Product Discovery is to deliver those breakthroughs in a weekly (or bi-weekly) routine.

To help your team turn design workshop methods into a productive and constant product development flow, you will need a framework to automate and delegate the experimentation processes and thus drive your team's growth productivity. Here a Kanban management board will come in handy.

New Idea (experiments) is introduced in a way a development task is introduced into the product development lifecycle. Those new ideas will flow through the experimentation pipeline, where they will be evaluated, researched, tested, and validated. Each concept will have a chance to turn into a breakthrough feature in a controlled and defined timeframe before you begin any significant investment.

Continuous Product Experimentation as a Product Growth Flow

If you're considering experimentation as part of your product growth strategy, you are likely to have few theories worth validating or testing. The question now is, how to start managing these? Here is a breakdown of steps laid out and organised on a Kanban board - from a simple idea through a constructive validation and conclusion.

1. Ideas

Ideas are submitted into this column directly by your growth team or any team members involved in the project. It's not unlikely for new ideas to stem from different departments, often departments with direct contact with the product's userbase. It is vital to freely allow all ideas to fall into this "idea bucket freely" and only afterwards perform qualification.

2. Chosen For Research

A crucial step in your experimentation flow, as the tasks picked for this column, should follow a strict categorisation procedure. Each experiment your team is about to perform should have a predetermined purpose and a measured metric. 

We recommend using the ICE prioritisation framework (impact, confidence, ease) to help decide on which should chosen for research. It's a method scoring three key areas of the idea, from 1 to 10 and ten multiplying the scores of each variable.

Impact x Confidence x Ease = ICE Score    

• Expected impact – how much the result will affect a specific metric
• Confidence of success – how certain is the prediction of impact
• Ease of implementation – effort to complete the project (from 10 - the easiest to 1 - most difficult)

The only way to ensure your experiment is valid is by pinpointing an impactful growth metric and defining unbiased goals. Remaining faithful to your product goals is critically important. Experiments performed only for the sake of experimentation or to confirm the acknowledged status quo have little to do with growth or change in that matter.

3. Scheduled experiments

Experiments in this column have received a green light and will now involve the Design and Research teams. The experiment at this stage should be assigned to a lead researcher or designer. All exercises associated with the experiment (interviews, whiteboard sessions, etc.) should be broken down into scheduled and delegated tasks.

4. In Progress

A column for tracking experiments currently in progress. The most crucial role of this column is to give a clear overview of what tests are being performed – giving you also a clear sign of what your team is currently involved in. Important to note is that the outcome for specific tasks in this column might affect other experiments in your pipeline.

5. Analysis

Each completed experiment must be analysed against its original goal and hypothesis before it can move forward. We record the result, any observations, and whether the experiment underperformed, met, or overperformed the metrics proposed in the original hypothesis. There are scenarios where the experiment will fail the actual scenario or will have no impact on the desired metric but might improve products performance in the long run? For example, in a subscription-based application, adding a clear pricing guideline on start might not convert into more paid customers but is more likely to eliminate users who will avoid payment or conversion anyway.

Results interpretation is one of the most significant stages in the entire process, as humans are biased towards interpreting data, so it works with their original hypothesis. Therefore, you should try your best to challenge your analysis and theories and have your evaluations peer-reviewed by other team members.

6. Conclusion

After your results have been analysed, there are generally three ways to go forward with the experiment.

• For implementation – when experimentation results match the hypothesis or achieve the expected result, and there are no critical obstacles in execution, this should be actioned. From here, the task is likely to be transferred to a different board, where the implementation and development process is managed.
• Further experimentation – You can further investigate the experiment if the results are inconclusive. There is a likelihood of receiving better results in further investigation, for example, by targeting a different user group.
• Declined – this is for experiments that did not achieve the expected result. It doesn't have to mean that the experiment data gets deleted; rather, think of the Declined column as a separate area on your board where the experiment gets archived.

Before you start your first experiments