the scientific method of building products

July 20, 2017 in Blog, Process

How does the Scientific Method apply to building products and companies? It applies to everything! Building a company, product, or feature requires testing the hypothesis ‘Have I picked a meaningful problem to solve and have I solved it in a valuable way?”

Moving into an experimentation frame of mind can have a profound effect on the psyche. We’ve long been told to admit what we don’t know and to ask questions, but we still get in our own way all too often. By running an experiment we’re telling ourselves that we’re going to learn something new and become informed. Instead of saying we know the answer and potentially being wrong, we can lower the cost of failure because there is no real failure. We’re going out with a guess and we’re going to test to see if the results are what we expected. If not, that’s not a failure, that’s additional information we now have to better inform our next guess. The idea is to increase the probability of success by never investing too much in our assumptions.

It can often seem daunting to read about how successful companies are continually running user tests and collecting hordes of data. Where do you start? Is this even possible at a typical company? The first step is the hardest. If you start experimenting, even on a small problem, you can build this into a habit. Create change from within, and with science to back you who’s not going to get onboard? Next time you’re in a meeting and there’s a discussion, debate, or all out argument – bring up the question “How can we test this?” You’ll likely stop a few people in their tracks because we’ve become so used to right and wrong and having to arrive at a decision in the moment. Chances are someone in that conversation measures their self worth on whether they have all the answers at a moment’s notice.

Change the momentum from problem solving to problem identification and understanding, and then move it to experimentation to find the right solution.

The scientific method starts with a question. Pump the brakes on your discussion of potential solutions and ask:

1) What problem are we solving?
2) What outcome are we expecting?

Keep it simple, get everyone to agree on the problem and how the solutions will be measured. Now take the potential solutions and conduct an experiment.

The experiment may be simple – go and ask 5 or 10 people that aren’t in the room that are closer to the problem at hand. Ask your support team to ask customers 2 questions at the end of every call (when they’ve solved the issue and the customer is in a good mood). If you have a few dollars to spend create two different ads on Facebook to a targeted group and track which is clicked more often (this can still be done cheaply).

Here’s how we’ve used science in our own path to bootstrapping a startup.

credit xkcd

We started out with a high level hypothesis for vspr.

“We can make organizations more successful by aligning goals and strategy with product roadmap and enhanced communication.”

On target if a bit broad. Our target customer is a Product Manager so the center of their universe is the product roadmap. We think they could use help aligning items in the roadmap to goals and company strategy. We also think the entire organization needs to know what the plan is and why it was chosen – hence the communication. However, we found that it wasn’t helping us prioritize our ideas and choose what to build for our first version of vspr. We needed to talk to more product managers to refine our questions in order to kickoff the scientific method.

When you’re stuck or something isn’t quite clicking, get out and talk to people. Anyone. Call a customer, a friend, post a question on LinkedIn. Go out to lunch and sit at the bar and strike up a conversation.

Our next pass came directly from a potential customer.

“Can we break the cycle of continuous disappointment through collaboration and clarity on strategy?”

The cycle of disappointment makes us laugh and cry every time we read it, but it’s all too true. Product Managers are constantly the bearers of bad news and find themselves having to disappoint the executives, customers, sales, marketing, and implementation teams. This customer empathy and centricity led to a flood of additional questions. Exactly what we needed.

  • What’s at the root of the “cycle of disappointment” for product managers?
  • Who’s a part of that cycle?
  • Why do we think collaboration on strategy would help break the cycle?
  • Why do we think clarity on strategy would help break the cycle?
  • How can we get more people collaborating with the product manager on strategy?
  • How would we help the rest of the organization gain clarity on strategy?

Out of this refinement exercise we came up with two experiments to test our new hypothesis. Can we get teams to collaborate on the strategy and to understand “why” an item is being prioritized where it is? Will this help address the cycle of disappointment?

#1 – Idea/Feature Evaluation Wizard

  • Goal – alleviate the “sell/defend” position when a new idea is proposed to a product team.
  • Customer Problem Addressed – “Am I investing in the right things at the right time?”
  • Potential Solution – Instead of going into gut reaction mode and intuitively evaluating whether the idea is good or trying to push back because it doesn’t seem to fit, the product manager pulls up They are greeted with a “how can I help you?” interface. Instead of going directly to the roadmap (this should allow them to better independently evaluate an idea), vspr walks them through the evaluation process and makes “next step” recommendations at each decision point.
  • Success Metrics – Product Managers report that they’ve reconsidered an idea after the initial evaluation process. More than 50% of ideas move forward to the “gathering estimates” phase.

#2 – Engage Internal Teams with Establishing Idea/Feature Value

  • Goal – engage team members to better estimate the value of an idea
  • Customer Problem Addressed – “Am I investing in the right things at the right time?”
  • Potential Solution – The product manager will select from groups of internal contacts to validate their initial assumptions on the impact of an idea to the business and customers. These contacts will receive a request to participate in an estimation exercise and the scores will be rolled up and presented back to the product manager to be used in the next step of the evaluation and prioritization process.
  • Success Metrics – 50% of added contacts submit their value estimate. Product Managers report that these estimates helped them understand the value of new ideas.

These experiments will provide observable data to tell us if we’re on the right track to solving our customer problems and proving our hypothesis. These may become the primary features of our initial launch or they may end up in the scrap heap of good ideas that don’t have the impact we expect. Either way this scientific process leaves us less attached to their success and more attached to observing how they perform and reflecting on what that data tell us is our next best move.

If we do this right we’ll be healthier in our work relationships, our organizations will be more successful, and we’ll build more trust in our product managers as decision scientists.

Cobb & DG


feedback – early and often

June 23, 2017 in Blog, Process

The feedback cycle was something that we’ve struggled with in previous organizations and for this round we made it a top priority to engage with potential customers as early as we possibly could. We’re happy to report that we were able to do this within the first 2 weeks of starting You really can’t start this process too early – startups live and die by how early and how often they get feedback.

Brain tricks and gamification can help turn goals into problems to be understood and solved. In these early stages we decided we needed feedback at least weekly and came up with a process to create a continuous feedback cycle. We adopted the DEFCON warning system. In the modern world the imminence of nuclear war is no joking matter (never promised we’d stay out of politics), but we feel the sense of urgency in getting feedback is very real when heads down building a new product. It’s entirely too easy to fall into the echo-chamber of self-reinforced “great” ideas. We could easily spend weeks burning through our limited capital reserves building features that no one will pay for.

To keep us focused on this pitfall we decided to post a DEFCON countdown on the whiteboard in our office. True to the DEFCON inspiration this countdown starts at 5 and every day during our morning scrum if we haven’t spoken to a potential user in the last day we decrement the DEFCON number. We chose this aggressive pace because we believe it’s possible to get weekly feedback in these early stages and it’s also necessary. We’ve picked a juicy problem to solve but the ideal solution is going to take many iterations. The only way to know if you’re on the right path is to continuously engage with people by getting out of the office or enticing users to come to you (we recommend cooking breakfast or having craft cocktails at arm’s reach).

Implementing systems like this, as simple and silly as it might be, enforces what gets measured gets done. We can spend all day saying we want to do things differently than we have in the past, but how will we prevent ourselves from falling back into comfortable routines? How easy will it be to explain away a few weeks spent perfecting an idea that no one really wants? This forces us to track and measure how often we’re getting feedback, which presents the feedback issue to our brains as a puzzle that needs to be solved every week.

This has manifested itself in a number of ways and caused us to do a few things that I don’t think we would have otherwise. In order to prevent the last minute scramble to call someone and get poor quality feedback (cheating the system), we combed through our contacts to build a list of people we thought would provide good feedback and would dedicate the time. Never hurts to ask, right? We divided this list a few different ways – prioritizing people local to us here in Denver so we could meet in person, people that would be potential end users, and people that have experienced the problem we’re trying to solve personally. We narrowed our list down to 20 people and divided this list into “waves”. This “Waves of Feedback” process will hopefully help us prevent the destruction of our world going past Defcon 1. We reached out to the “First Wave” of 5 people and scheduled them over a 2 week period. Our idea is that there will be at least a month between reaching back out to the same wave for follow up feedback. This keeps us from calling in too many favors and wasting time without showing significant progress and new ideas for feedback. It also gives us time to implement feedback from the other waves before coming back around to the same person. Our hope is that by the time we roll back around we’ll wow them with progress and continue the snowball effect. We’ll report back and see how well this process is holding up.

On that note – we need to expand our feedback circle! If you or someone you know is working in product management or a role that intersects with product roadmaps and strategic initiatives in an organization, we’d love to talk to you. If you happen to be in the Denver area we’ll even cook you breakfast. Shoot us a note at

Cobb & DG


let’s do this

June 4, 2017 in Announcements, Blog, Launch

We’re gathering early stage feedback. To find out more about our process and help us out, check out the latest blog post.

If you could fit your entire company in a single room this would be easy. Remember when communication came for free? When everyone knew what was important and why? As organizations and teams grow there’s still a way – align your product roadmap clearly to your business goals and metrics. Communicate every change, align every task with a goal and provide insightful views to business stakeholders and implementation teams. Measure, adjust, realign, repeat. Man, this is starting to sound like work.

Product managers spend a lot of time as intermediaries – communicating changes and decisions to executives, employees, and customers. When there’s a breakdown at this intersection between strategy, marketing, and execution it can be game over. Having seen this too many times first hand, we decided we should do something about it. is a business alignment and roadmap prioritization tool. We believe all companies struggle to answer the question “Am I investing in the right functionality at the right time?” vspr’s origins are from a time when the planet Venus was thought to be two different stars – the day star “phosphorus“ and the evening star “vesper.” Our goal is to prevent this type of misalignment, miscommunication and duplication of work by ensuring an organization’s goals and strategy are aligned around their product roadmaps and are clear to all members of the organization.

We’re going to learn from our mistakes and build the company we’ve always wanted to work for. Who else will? We know we’ll need help along the way and that’s where you come in. By documenting our progress and engaging the community with problems we face, we hope to build a continuous improvement feedback loop not just for our product, but for the entire company and ourselves.

Follow along on our journey. We’ll be podcasting every two weeks and openly discussing every aspect of starting a business and building a product from scratch. Our first episode is live and you can hear us discuss how we took this leap. We’ll be talking about things technical, business, philosophical…we’ll try to stay out of political but who knows.

This is going to to be fun.

Cobb & DG