How Biz Ops Can Help Drive Product Analytics At A High-Growth Start-Up

I’ve worked in BizOps environments of many sizes, from large Fortune 500 companies to high-growth startups like LinkedIn and NerdWallet. And the key question I’ve addressed has been fundamentally the same regardless of company size: How can we help our partners become more effective and efficient at building products our users love?

Each BizOps team answers that question differently. The size of the organization, scope of products being managed, and scale of resources at a BizOps professional’s disposal may vary significantly – and those factors have significant implications for what problems should be tackled and how they should be approached.

In this article, I share five ways BizOps can drive product analytics at a high-growth startup.

1. Clarify and align business and product goals

Startups are likely to have more competing priorities and less strategic clarity than established companies. But if individual teams optimize for their goals instead of the company’s goals as a whole — say, if a product manager continuously pushes to optimize a product, even though the right thing for the company is to sunset it — there’s a higher risk for sub-optimal outcomes.

It’s very easy to get caught up investigating questions that are important to particular products or teams, but actually aren’t as relevant to the company’s strategic objectives. In part, this happens  at fast-paced hyper-growth organizations because there are many possible directions to move in; it’s hard to remove yourself from the details to ask whether your assumptions or the company’s hypotheses have changed.

The implication here is that you need to help your partners understand what they should be optimizing for and measuring. As a BizOps team member, you can help clarify and frame your core product hypotheses — i.e., what do you most need to know? — and push back where the team is exploring in adjacent or orthogonal areas.

2. Prioritize impact over precision

Working slowly to reach clear, actionable conclusions at a large company can mean lost money and missed opportunity. But lost time at a startup can mean failure and bankruptcy.

The implication for BizOps is that alpha matters a lot less than effect size. When we interpret a typical A/B test, for example, we usually think in terms of a p = 0.95 world: Unless something is statistically significant at 95% confidence, it’s not a valid result. But at a startup, if you’re 80% sure that something is real, that’s pretty good!

An example from NerdWallet: We recently found ourselves two days into a weeklong A/B test for a product feature. We were surprised to see a multiple-percentage-point improvement in conversion rates, but we were on a low-traffic page and needed more data to get to 95% significance. Ultimately, we decided to cancel the test; the purpose was to determine if this feature was better than the previous version, not to get to mathematical precision. It was already clear that the new feature was better.  

The bottom line is that It’s important not to conflate statistical significance with material significance; when you need to bend the curve on a company’s growth trajectory, you’re trying to find big effects. If you don’t see a big effect size within a test — one that you can detect with smaller samples — it may not be a priority in a resource-constrained environment.  

3. Make the most of your (limited) data

When I was at LinkedIn, I was deep in the heart of “big data.” We ingested dozens of terabytes daily; to manage the sheer volume of data, the company even needed to develop new technologies. NerdWallet is far more sophisticated in its approach to data than most companies its size, but it’s not necessarily dealing with the same amount of information.

What that means is that every data point we do have is that much more critical. Every piece of information we have on how users are interacting with our products and what they value helps fundamentally shape what we’re building.

It’s crucial for our team to quickly establish clear instrumentation and universal access to data, so it can be shared across multiple use cases and teams. One thing we’re focused on right now is making sure that every product ships with a basic set of analytics — such as event tracking and funnel measurement — in an easily automated package.

BizOps plays a critical role in highlighting data requirements, helping provide more visibility for data, and applying that data effectively to generate key product insights.

4. Be scrappy in gathering direct user feedback

At a startup, it’s much tougher to engage with a representative sample of your user base. After all, you may not even have an existing install base of users, let alone a large one. So how do you ensure you’re getting a good cross-section of your target audience, and that your feedback is consistent?

While a lot of this work is the typical purview of marketing or user research, BizOps can get creative to address these issues. Aside from helping product managers build feedback loops into products, you can use scalable third-party tools like Mechanical Turk or SurveyMonkey, or  stitch together disparate data sources — like consumer support feedback and user behavior data — to build a picture of your users over time.

5. Focus on adding leverage, not the details of your job description

A product manager tends to be a jack of all trades. PMs aren’t averse to getting their hands dirty in data, or user research, or project management. But they often face technological barriers at a larger companies. For example, the expertise required to fully understand a complex tech stack, or build up a working knowledge of Hive, Pig or Python, can be prohibitive. A layer of data scientists, analysts and BizOps professionals may be necessary just to get basic data sets.

At startups, practical concerns dominate. While a PM is probably fine knocking out some SQL or R, there are many other demands on their time — and managing data probably isn’t what their team needs most from them. Here is where BizOps comes in: We efficiently generate and help to investigate hypotheses, get deep into the data with whatever tools are available, and do whatever it takes to accelerate results.

Members of my team at NerdWallet build complex SQL queries, orchestrate A/B tests, and even write the occasional spec or mock to free up our partners to focus on higher-value activities instead. In my experience, that’s the kind of value that partners appreciate most.

Hopefully you’ve found some of these thoughts valuable — I’d love to hear your feedback in the comments below!

And if you’re in the Bay Area at the end of August, I’d like to invite you to BizOpsCon 2.0 to discuss the intersection of analytics and BizOps more deeply. At the first version of BizOpsCon in 2015, we hosted more than 400 BizOps professionals for networking and professional development. We’re doing the same thing this year on August 17, and you can sign up to attend here!