Challenges Product Managers Face as Products Evolve & A/B Testing 101

[1] Let’s start by identifying 2–3 challenges that product managers need to consider when directing the evolution of their product, and then I will explain some techniques suggested to address these.

There are several challenges that product managers need to consider when directing the evolution of their product, here are two key ones:

1) Defining the Product Roadmap: short-term vs. long-term goals

A product roadmap is a useful tool that allows product managers to plan for both the short-term and long-term growth and evolution of their product. This plan is essential to gain stakeholder buy in and cross-functional support, and also helps PMs develop a budget for their product. The main challenge that product managers face while developing a product roadmap is striking a balance between short-run and long-run goals. Effective product managers are able to attain successful results with their products by being both visionary, bold, and forward-looking, and also having a narrow and realistic view of what are the most important issues and priorities that need to be addressed in the present and near future.

Oftentimes, what happens is that short-term goals and urgent concerns that need to be addressed or met as soon as possible, tend to overshadow long-term goals, and the vision of a product which is extremely important for product managers to spend time thinking and reflecting about. Thus, there are important techniques that can help product managers find a balance between these two.

The first technique is prioritization — which essentially allows PMs to focus on creating a timeline about what is most urgent and important to work on now vs. what can be done tomorrow.

Second, time management is extremely important for PMs to ensure that they are working on what adds the most value today and contributes to the longer vision for tomorrow.

Lastly, conflict resolution skills are necessary for product managers to address competing deadlines, working and collaborating with teams, and when prioritizing product features, goals, and objectives.

2) The Market Transformation: uncertainty and changing dynamics

Product Managers have to be ready for change. As a product evolves, the market dynamics will also evolve and change. This can lead to new players entering the market, changing consumer preferences, higher demand for new or less features in your product, and even new research and development insights and discoveries that can drive users away from your product or platform. Thus, dealing with change, and making the “right calls,” as to the direction you want to take your product, are some of the biggest challenges that PMs face while having very limited amount of knowledge or data to make these decisions. However, there are several techniques that can help PMs deal with market transformations.

First, is crucial conversations and strong communication skills. As a product manager, you need to be able communicate what is the best strategy for your product to all of the relevant stakeholders. Being able to talk out your ideas and plan can help you gather feedback, gain advice, and support from your teammates who can have industry knowledge and expertise to complement your plan.

Second, feedback. Gathering direct customer feedback can and will allow you to move in the right direction. Like our Professor Ken mentioned in the first lecture, a great Product Manager listens to its users, and even after you have launched a product successfully into the market, it is extremely important to continue gathering user feedback to drive your product decisions and vision.

Lastly, the product roadmap and long-term vision will be extremely useful when thinking about what is the best course to take, when market dynamics are changing. A great example is Instagram Stories. Last year Facebook launched stories on Instagram and later on facebook and messenger, since Snapchat was gaining a lot of traction, specifically in regards to engagement metrics, and was their biggest competitor. Thus, Facebook decided to introduce that new feature, change the way their users created content, gathered a lot of feedback, and decided to develop a camera within the app. The results have shown that Instagram stories have surpassed and doubled engagement over Snapchat, and the moral of this story is that listening to users always wins.

Sources:

[2] A/B or Split Testing

What is it?

A/B Testing (aka Split Testing) is comparing two versions of a product against a specific product metric (ex. conversion rate) to see which one performs better and to help product managers drive product prioritisation decisions. It is important to note that anything that your user interacts with in your product can be tested, here is a short list:

  • UI/UX
  • Content
  • Headlines
  • Links
  • Images
  • Buttons
  • Pricing/Promotions (free vs. paid)
  • Free trials length

Here is a really informative video on A/B testing, check it out:

How it works?

In summary, the way A/B testing works is by first creating two different versions of a product, A and B. Then you invite users to test the two versions at the same time, and track which one yields the best results compared to the metric you are testing. Refer to the visual diagram above as an example of an A/B test for a website testing for conversion rate of visitors.

Many suggest that A/B testing is like a scientific process:

1) Analyze your product and user data

2) Observe user behavior and how they interact with your product

3) Develop a hypothesis, and choose your goal(s)

4) Test your hypothesis (the A and B versions of your product)

5) Gather results and develop conclusions

6) Communicate conclusions and insights to stakeholders (Product Implementations)

Pros:

  • Refine the product/user experience
  • Discover bugs and errors
  • Test new ideas/features
  • Customer validation and user feedback
  • Fast and easy to setup and modify
  • Real-time/live testing with real users/visitors
  • Can quantify results and insights

Cons:

  • Takes time to build the tests/versions of product
  • Testers may not become end users (sample)
  • Short-term insights and user data can be useless after experiment
  • Only good to test a specific goal/hypothesis
  • Results might be inconclusive (wasting resources)
  • Problems/bugs with tests
  • Most problems/challenges are not solved with A/B testing

Sources:

--

--