Check out my prior entries in this series: Research ROI: Floors & Ceilings & Research ROI: Problem, Scope, Impact

I recently read the Harvard Business School Case Study on Wiz. One aside caught my eye:

In the early decision to pivot to a cloud security product, the Wiz team also effectively decided to enter an existing market, rather than try to create a market around a new product. The move “to a red ocean strategy was counterintuitive,” Herzberg said.

A Red Ocean strategy involves:

• Competing in existing market space
• Beating the competition
• Exploiting existing demand
• Making the value-cost trade-off
• Aligning the whole system of a firm’s activities with its strategic choice of differentiation or low cost

It struck me that Red Oceans abound in security research. Just in recent history, I’ve seen froth around:

• vulnerabilities in openclaw
• prompt injection
• vulnerable and malicious Agent Skills
• malicious npm packages
• public exposure of MCP servers
• malicious models on Hugging Face
• the evergreen idea of “a ton of secrets leaked in location Y” (h/t Truffle)

These Red Oceans tend to snowball. One researcher inspires another to look at the same problem, or one just barely adjacent. Companies who focus on marketing-driven-research¹ jump in. Vendors publish derivative works - with or without citation. The regular characters push FUD: to boost their profile, sell you something, or just because they don’t know any better.

That being said, these research oceans are red for a reason! There is chum in the water.

So, how can you responsibly and successfully navigate researching in a red ocean?

There are two tricks.

The first is to identify if you have something to add by diving in. This can fall into a few categories:

1. Ability to scale beyond state of the art. For example, expanding piecemeal research like malicious skills to an entire ecosystem.
2. A unique perspective, often the intersection of your expertise with details. For example, applying a strong malware detection engine to a new class of tool.
3. A differentiated right to win, often based on unique data or unique capabilities. For example, reviewing prevalence of a supply chain attack against representative data. Or using long-term data collection to perform unique retrospective analysis.
4. A compelling narrative, often focused on real world impact versus abstract risk. For example, I worked on a project where we investigated the venn diagram of secrets and AI through the lens of the AI Top 50. The mechanics of the research were simple, but the target scope drove meaningful analysis and industry influence. We identified the patterns of secrets leakage in AI, and were able to identify major gaps in secrets tool coverage.

The second trick is executing with integrity once you’ve decided to wade in:

1. Cite generously, both your inspirations and high quality adjacent research. Do good enough work that citation doesn’t fuel insecurity.
2. Find opportunities to collaborate, instead of duplicating work.
3. Focus on actionable guidance, and progressing the industry. Make sure you answer the implicit question of “so what”

The ROI of Red Oceans

Red oceans offer an opportunity for differentiation. They also offer a chance to work in conversation with competitors and the industry. The built-in audience and proven market demand mean your research can have immediate impact, but only if you bring unique value.

The trick is to avoid low leverage follow-on research, and instead respond to the existing hype and demand with differentiated work. Know what you’re adding, execute with integrity, and focus on moving the industry forward.

Sometimes the best research isn’t about finding a novel lane, but about swimming smarter in a crowded one.

Check out my prior entry in this series: Research ROI: Floors & Ceilings

This time, I want to share the framework I lean on when trying to coach researchers in maximizing ROI. The framework exists to cut through the vagaries of “this had insufficient impact” feedback. Chase ROI responsibly by decomposing the skills necessary: Problem Selection, Project Scoping, & Operationalizing Impact.

Problem Selection

In many ways, problem selection is the core “art” of Research. Picking high leverage, solvable problems is a matter of taste. However, there are still techniques for improving problem selection. The start is a deep understanding of the domain and landscape. Pay attention to the firehose of information: industry publications, academic research, competitors, adjacent domains, new technologies and developments. In a team, the benefit is your overlapping points of view, lightening this load on any individual.

Porting research is a powerful technique. Consider the possible variants of any research you encountered. This can be application across domain, technology, or simply in a new context or against new data. There are whole genres of repeatable research, like scanning marketplaces for secrets.

Fundamentally, picking a research problem is a balance. The idea project involves:

1. Your interests and curiosity
2. Your skills and comparative advantages, the work only you can do
3. The risk landscape and gaps in existing research / literature
4. The goals of your team and organization

“What is the most important problem in your field, and why aren’t you working on it?”

• Nicholas Carlini

Cultivate a garden of interesting problems. Always keep a list of research ideas, and inspirational research. This can start as a simple bibliography or text document. If you think of a topic, or just a blog title, scribble it down. This compounds shockingly well.

Project Scoping

The next core element of successful research impact is to scope your work correctly. The ROI equation means that increased investment requires outsized return. As such, structuring your level of investment is essential. An amazing research outcome when delivered in a week may be a failure if it takes six months. This is without going into the timeliness component of research, where outcomes can be perishable.

Floors and ceilings is one framework that can help in rightsizing research. Long-term research balance is also essential, mixing big swings on big problems with quick wins. Set exit criteria for research to lock in ROI at the right moment.

Managing your projects for efficiency is another key element. As one key approach, stolen from Kanban: limit work-in-progress to increase throughput.

Operationalizing Impact

Finally, having selected an interesting problem and adequately scoping your approach, you also need to be proactive in setting up for success.

The foundation is identifying the beneficiaries of your research. Consider stakeholders, audience, and artifacts that will ensure your research generates impact. If you are struggling to define audience, there are safe defaults: your customer, your internal partners, or just your 6-months-ago self.

Ideally, research output should balance quality and quantity. Avoid sacrificing quality for quantity. Focus on hitting your floors; don’t lower them. Get work out the door, where you can analyze its impact.

Pair that with an understanding of possible vectors for impact, including:

• influencing product direction - by proving the possible
• enabling new product capabilities
• upleveling the industry - by advancing the state of the art
• generating marketing opportunities
• resolving core industry risk - and responsibly disclosing
• identifying vulnerabilities and getting them resolved
• creating new frameworks for managing security

Take the time after each effort to review and retro your impact. Research can be a hamster wheel, get off long enough to learn and continuously improve.

Putting It All Together

This framework has served me well, although I’m continuously refining it. Regardless, it gives you a means to proactively and systematically plan for and assess research ROI.

How do you think about maximizing ROI? Do you have a framework you like? I’m always looking to diversify my own methods!

Interested in more internals of Industry Research? Check out my coworker Amitai Cohen’s brilliant collection of essays at Rhythms of Research.

How do you lead a Research team? Since joining Wiz, I’ve moved into a role leading a small part of our Research Organization.

Wiz has obviously been a success, and Research has always held a core role. Leadership has valued it accordingly. There is no micro-focus on measuring and managing for Return on Investment. In my team, Risk Research, we have diverse stakeholders and impact, spanning Product, Engineering, customer-facing, and (yes) marketing.

However, despite the latitude we have for research that might be high-risk or long-horizon, it’s useful to have rules of thumb to guide us. In talking to other industry research teams, candidates, and colleagues, one shorthand I keep falling back on is “Floors & Ceilings”.

The basic idea is simply: in planning research, we should set a Floor and Ceiling goal.

Floors

The floor will often be an internal presentation, documentation, a blog post, or a proof-of-concept tool. It’s the baseline value we get from our work. This ensures that even if we aren’t fully successful with our research, the work isn’t entirely wasted.

This isn’t just a way for leadership to squeeze out value. Setting a floor ensures freedom to experiment and psychological safety for researchers. You can go down a rabbit hole safely, knowing how to de-risk the creative work. Researchers often get trapped in a search for absolute truth, and anything less than success feels like failure. But research is a risky endeavor. The right balance should have you “failing” often. The floor compensates for that.

It sets a foundation that future research can build on. It ensures that “failures” compound.

Ceilings

The ceiling might be a Tier 1 conference talk, a new Product SKU, or protecting X% of the industry. Setting a ceiling is an opportunity for strategic alignment. It ensures moonshots get aligned to company, team, and individual goals.

For the researcher, it empowers and encourages them to dream big and take big swings. Research can have legs, not just get cut off as soon as it hits a value threshold. Without a ceiling, good research gets shipped too early.

Together

Setting the right floor and ceiling sets you up for success. Hitting the floor needs to be achievable with high-confidence. The ceiling should be plausible, but sufficiently audacious.

If a project’s floor is “learn something” and the ceiling is “twitter thread,” that’s more hobby than professional Research. Setting floor and ceiling to “blog post” means you’re probably snacking on low-risk, low-reward work.

The floor also formalizes the stages of research. It offers an opportunity to stop-loss a project and recoup salvage value. If we’re able to meet the floor, we should have a good understanding of the viability of reaching the ceiling. It’s a chance for the researcher to advocate for a go/no-go on further investment.

Interested in more internals of Industry Research? Check out my coworker Amitai Cohen’s brilliant collection of essays at Rhythms of Research.

AI Coding Agents are running rampant, and it’s thrilling.

• Ramp’s Inspect background agent authored 30% of merged PRs.
• Stripe’s Minions merge >1000 PRs a week.
• Spotify’s “best developers” are using “Honk” and “have not written a single line of code since December.”

Agents have vindicated Cloud Development Environments, and made them essential for embracing the next wave of engineering patterns.

Cloud Development Environments (CDEs) are nothing new.

• Stripe’s public marketing of its Developer Environments [2022] was one of the watershed moments for this pattern.
• Gergely Orosz, of The Pragmatic Engineer, covered Cloud Development Environments in 2023.
• Gartner added Cloud Development Environments to their 2023 Hype Cycle.

However, they never seemed to break containment. They remained restricted to elite SaaS “startups.” Vendors for CDEs never hit escape velocity.

It’s a shame. CDEs solve essential challenges in scaling development: standardization, ergonomics, speed, and security isolation.

Background agents have changed the calculus. CDEs give you sandboxed execution for untrusted agent code. Reproducible environments mean predictable behavior. Spin-up is trivial, enabling parallelism. Secrets stay centralized, not scattered across developer laptops. The companies that invested in CDEs are seeing them port cleanly and safely over to their army of agents.

If you’re not in the anointed few, you’re trying to tack Agents onto clunky CI/CD and the terrifying security landscape of your developer machines (agents and ~/.aws don’t mix). You won’t see the same pace or the same outcomes.

These agent success stories aren’t lucky. It’s companies earning the dividends of CDE investment. If you’re trying to adopt the latest and greatest in AI Coding patterns, skip the hard lessons. Don’t reinvent the wheel. Start from a CDE.