How does the Scientific Method apply to building products and companies? It applies to everything! Building a company, product, or feature requires testing the hypothesis ‘Have I picked a meaningful problem to solve and have I solved it in a valuable way?”

Moving into an experimentation frame of mind can have a profound effect on the psyche. We’ve long been told to admit what we don’t know and to ask questions, but we still get in our own way all too often. By running an experiment we’re telling ourselves that we’re going to learn something new and become informed. Instead of saying we know the answer and potentially being wrong, we can lower the cost of failure because there is no real failure. We’re going out with a guess and we’re going to test to see if the results are what we expected. If not, that’s not a failure, that’s additional information we now have to better inform our next guess. The idea is to increase the probability of success by never investing too much in our assumptions.

It can often seem daunting to read about how successful companies are continually running user tests and collecting hordes of data. Where do you start? Is this even possible at a typical company? The first step is the hardest. If you start experimenting, even on a small problem, you can build this into a habit. Create change from within, and with science to back you who’s not going to get onboard? Next time you’re in a meeting and there’s a discussion, debate, or all out argument – bring up the question “How can we test this?” You’ll likely stop a few people in their tracks because we’ve become so used to right and wrong and having to arrive at a decision in the moment. Chances are someone in that conversation measures their self worth on whether they have all the answers at a moment’s notice.

Change the momentum from problem solving to problem identification and understanding, and then move it to experimentation to find the right solution.

The scientific method starts with a question. Pump the brakes on your discussion of potential solutions and ask:

1) What problem are we solving?
2) What outcome are we expecting?

Keep it simple, get everyone to agree on the problem and how the solutions will be measured. Now take the potential solutions and conduct an experiment.

The experiment may be simple – go and ask 5 or 10 people that aren’t in the room that are closer to the problem at hand. Ask your support team to ask customers 2 questions at the end of every call (when they’ve solved the issue and the customer is in a good mood). If you have a few dollars to spend create two different ads on Facebook to a targeted group and track which is clicked more often (this can still be done cheaply).

Here’s how we’ve used science in our own path to bootstrapping a startup.

credit xkcd

We started out with a high level hypothesis for vspr.

“We can make organizations more successful by aligning goals and strategy with product roadmap and enhanced communication.”

On target if a bit broad. Our target customer is a Product Manager so the center of their universe is the product roadmap. We think they could use help aligning items in the roadmap to goals and company strategy. We also think the entire organization needs to know what the plan is and why it was chosen – hence the communication. However, we found that it wasn’t helping us prioritize our ideas and choose what to build for our first version of vspr. We needed to talk to more product managers to refine our questions in order to kickoff the scientific method.

When you’re stuck or something isn’t quite clicking, get out and talk to people. Anyone. Call a customer, a friend, post a question on LinkedIn. Go out to lunch and sit at the bar and strike up a conversation.

Our next pass came directly from a potential customer.

“Can we break the cycle of continuous disappointment through collaboration and clarity on strategy?”

The cycle of disappointment makes us laugh and cry every time we read it, but it’s all too true. Product Managers are constantly the bearers of bad news and find themselves having to disappoint the executives, customers, sales, marketing, and implementation teams. This customer empathy and centricity led to a flood of additional questions. Exactly what we needed.

  • What’s at the root of the “cycle of disappointment” for product managers?
  • Who’s a part of that cycle?
  • Why do we think collaboration on strategy would help break the cycle?
  • Why do we think clarity on strategy would help break the cycle?
  • How can we get more people collaborating with the product manager on strategy?
  • How would we help the rest of the organization gain clarity on strategy?

Out of this refinement exercise we came up with two experiments to test our new hypothesis. Can we get teams to collaborate on the strategy and to understand “why” an item is being prioritized where it is? Will this help address the cycle of disappointment?

#1 – Idea/Feature Evaluation Wizard

  • Goal – alleviate the “sell/defend” position when a new idea is proposed to a product team.
  • Customer Problem Addressed – “Am I investing in the right things at the right time?”
  • Potential Solution – Instead of going into gut reaction mode and intuitively evaluating whether the idea is good or trying to push back because it doesn’t seem to fit, the product manager pulls up They are greeted with a “how can I help you?” interface. Instead of going directly to the roadmap (this should allow them to better independently evaluate an idea), vspr walks them through the evaluation process and makes “next step” recommendations at each decision point.
  • Success Metrics – Product Managers report that they’ve reconsidered an idea after the initial evaluation process. More than 50% of ideas move forward to the “gathering estimates” phase.

#2 – Engage Internal Teams with Establishing Idea/Feature Value

  • Goal – engage team members to better estimate the value of an idea
  • Customer Problem Addressed – “Am I investing in the right things at the right time?”
  • Potential Solution – The product manager will select from groups of internal contacts to validate their initial assumptions on the impact of an idea to the business and customers. These contacts will receive a request to participate in an estimation exercise and the scores will be rolled up and presented back to the product manager to be used in the next step of the evaluation and prioritization process.
  • Success Metrics – 50% of added contacts submit their value estimate. Product Managers report that these estimates helped them understand the value of new ideas.

These experiments will provide observable data to tell us if we’re on the right track to solving our customer problems and proving our hypothesis. These may become the primary features of our initial launch or they may end up in the scrap heap of good ideas that don’t have the impact we expect. Either way this scientific process leaves us less attached to their success and more attached to observing how they perform and reflecting on what that data tell us is our next best move.

If we do this right we’ll be healthier in our work relationships, our organizations will be more successful, and we’ll build more trust in our product managers as decision scientists.

Cobb & DG