How to AI UXR is brought to you by Strella, an AI-powered customer research platform that partners with you to build, moderate, and synthesise interviews, allowing you to go from question to actionable insights in just a few hours.

In This Conversation

In this conversation, Michelle shares how deadline pressure, combined with the possibilities of AI, led to one of the most compelling examples I’ve seen of AI being used not just to speed up research but also to uplift the craft. Michelle built a transcript-based coaching system that evaluates research calls, provides “tough but fair” feedback, and helps ninety non-researchers improve their craft within minutes of a research session. In this episode, Michelle will walk you through the tool, so make sure your screen is on.

In this episode, we cover:

1. How Michelle built the UXR Interview Coach in a matter of hours

2. How the system scores research interviews against a custom-built rubric and delivers private, actionable feedback in Slack

3. Why “tough but fair” feedback lands differently when it is timely, specific, and evidence-based

4. What the team learned by analysing research quality across roles, call types, and quarters

5. How the rubric evolved as the system encountered more nuanced customer conversations

6. Why AI’s real value for research may be in enabling new systems, not simply automating old workflows or augmenting analysis

7. Michelle’s advice for researchers experimenting with AI: start with one painful or underperforming area and make it better one step at a time

Partway through the episode, Priya Krishnan, the cofounder and COO of Strella, shares her take on the conversation.

Connect with the Guests

• Michelle Bejian Lotia, Staff UX Researcher at Ramp

• Priya Krishnan, cofounder and COO of Strella

How to AI UXR is brought to you by Strella, an AI-powered customer research platform that partners with you to build, moderate, and synthesise interviews, allowing you to go from question to actionable insights in just a few hours.

Subscribe now

The ResearchOps Review is supported by User Interviews, now part of UserTesting. User Interviews makes it fast, easy, and affordable to recruit participants so you can scale research without sacrificing quality.

At the first event in Berlin, I was definitely the one breaking a sweat. It was March 2025, and I had my printed checklist in hand as I ran around the office making sure that everything was in place: the product users, my colleagues, the tables, the timer, and, of course, the pizza. Some of the users we had invited hadn’t shown up, so I sprinted into the common areas of our building and tried to recruit anyone relaxing in the corridors to take part. Most people politely declined, and I didn’t blame them. If a sweaty stranger offered me pizza in exchange for chatting to a company I’d never heard of—Miro who?—I wouldn’t have known how to respond either.

In the end, five minutes before the event started, an office manager from the same building invited me to ask around their offices: Would you like to take part in a one-hour continuous discovery session with Miro’s product builders? Right on time, I came back with two extra participants; I’ll never forget the look of relief my colleague Anthony and I shared.

In this article, I’ll share how to build the operations that make continuous discovery sessions run like clockwork, and the value these sessions can deliver. I’ll walk you through who to invite, what you need to do to create a repeatable (and sustainable) setup across locations and in remote contexts, and the small decisions that can make or break the event dynamic.

The Continuous Discovery Method

Product discovery coach Teresa Torres coined the term continuous discovery to describe the practice of building regular, structured user contact into product development so that, rather than arriving in waves through ad hoc studies done by someone else, insights flow continuously—and continuously build empathy and inform product decision-making.

Just over a year ago, when I first announced on LinkedIn that we were kickstarting a continuous discovery program in Miro’s Berlin office, someone left a comment declaring it the ultimate solution to all research problems. My reply then reflects my enduring perspective today: continuous discovery is one methodology for collecting evidence from users, but it’s not a replacement for other research approaches, such as user interviews or usability testing. Still, when done well, these in-person, pizza-fueled programs can help move the needle on customer-centricity in ways that other methods often can’t.

At a high level, continuous discovery sessions run as follows: once a month, you’ll invite five product users to your offices for one hour to take part in short, ten-minute interviews with five different product teams, each with an interviewer and a notetaker. Each team can use their ten minutes to ask questions about their product area or test designs. Every ten minutes, users rotate clockwise to the next table for a new set of questions asked by a different team. By the end of the hour, every team has spoken with every user, and every user has weighed in on five product ideas. Rather than a library or research lab atmosphere, the dynamic is social, fun (and loud at times), and it always ends with pizza.

Continuous discovery sessions are often thought of as a type of casual or informal research—less valid than, say, one-on-one interviews. Yes, by design these sessions are informal, but continuous discovery earns its place. It’s neither generative (an open exploration of hunches in the problem space) nor purely evaluative (rigorous evidence-gathering in the solution space). Instead, it sits in the middle: a bit more towards validation, where most product decisions actually live, and where uncertainty needs to be reduced quickly rather than completely. It’s not generally a good methodological choice when a deep diagnosis is needed, or when you’re dealing with sensitive topics or high-stakes validation that requires rigor. It is a good choice for early product ideas, rough prototypes, and assumption checks.

Time Constraint Is a Feature, Not a Bug

The feedback we most often receive from colleagues is that there just isn’t enough time with each user to dig in as deep as they’d like—ten minutes can feel short when you have a lot to cover. But the time limit doesn’t only facilitate getting as many users as possible in front of as many team members as possible.

An in-person continuous discovery session gets closer to ecological validity—the degree to which a study’s conditions resemble the real-world context in which the behavior normally occurs—than a controlled one-on-one study. I’ve come to believe that nobody approaches a new feature with the patience of a usability test participant. People approach an app or task the way they approach everything else: while distracted, half-committed, perhaps in a hurry. A compressed time period exploring a product mirrors how users tend to meet your product in the wild. They scan, process, and make decisions under cognitive load—in situations not unlike busy rooms with limited time to talk—and don’t necessarily give your app or their task undivided attention. How would they, with hundreds of other tabs, apps, tasks, and people claiming their attention?

The ten-minute time limit also exposes the real hierarchy of attention. Teams see firsthand which interface elements actually guide behavior versus their assumptions; that gap between the observed and the assumed is where much of continuous discovery’s value lies: the most honest findings show up when users don’t have time to be polite.

Going Deep, Not Wide

For people who do research (PWDR),1 choosing one or two questions to ask a participant and going deep rather than wide in scope can be a challenge. That’s why it’s essential to align on the study design and scope with colleagues before the session begins—it’s also reasonable to reiterate alignment goals during the event.

Even with the challenge of choosing only one or two questions, teams rarely arrive at a continuous discovery session with nothing to explore: there’s always something that they need to validate, whether a sketch, a prototype, a vibe-coded idea, or an assumption they want to test before pushing it to production or pausing to do more research. They rarely arrive needing statistical confidence; they’re seeking enough signal to make the next call—directional guidance as I like to call it. Honestly, sometimes they just need validation for their own peace of mind, or they simply want to keep improving their interviewing skills. Regardless, the format is calibrated for the specific moment when teams need to be unblocked.

While PWDRs might feel stifled by the time limit, for users it’s a feature rather than a bug; they get to spend an hour exploring a handful of new product ideas and meeting new people who want to learn from them. It’s social and fun.

Recruitment, Roles, and Responsibilities

Perhaps you’re now convinced that these sessions would be a valuable addition to your research toolkit; now you need to know how to run them. To run a continuous discovery session, you’ll need eighteen people in total: ten colleagues who will form five “teams,” each with an interviewer and a notetaker, five product users, and a support crew comprising the following:

1. A lead to moderate the session and present.

2. A support person to handle reception duties, welcome users, and back up the moderator if something goes off script.

3. An additional support person to run the parallel workshop when there are surplus users or to step in should anything unexpected happen.

If you can find only one other coconspirator, you can split these duties between you. It’s a little less comfortable, but it’s still doable.

Recruiting Colleagues

Product designers, product managers, engineers, and data analysts are the most natural fit for these sessions as their work tends to benefit most from short, frequent check-ins with users. But over time, colleagues with all sorts of job titles may join the sessions: customer success managers, marketing, and community professionals. Basically, everyone whose job involves thinking about users.

Key to attracting your colleagues is crafting a clear message in the most relevant communication channel that explains when the continuous discovery session will take place, what it involves, the benefits of participating, and how to sign up. For first-timers, the entire experience can feel daunting, so we also include a short set of dos and don’ts, interviewing tips, and note-taking templates on the signup page. Plus, we offer a quick preparatory call to walk them through roles and responsibilities, ask what they want to test, and explain what to do if they have questions. This information tends to settle the nerves of anyone who’s not practiced in talking to users. We also keep a waiting list, because the five spots tend to fill up fast, and occasionally folks change their minds before the event starts.

Recruiting Product Users

This is one of the rare cases where the profile of the people you recruit doesn’t really matter. You want to recruit “active users,” full stop. Here’s why: your users will be exposed to five radically different ideas in one hour, so there’s no point hunting for specific profiles. If one of the teams insists they really need a particular audience, perhaps a continuous discovery session isn’t the right approach. In which case, help them find the right method for their question.

Finally, to ensure that you don’t run short on users on the day, learn from my sweaty Berlin office sprint: over-recruit. Aim for ten confirmed users instead of five. While more than five users is never a problem, if fewer than five turn up, teams are left waiting with no one to talk to, breaking the dynamic of the session for everyone. We often confirm well past ten participants and run a parallel cocreation workshop, which one of our researchers moderates, turning the surplus into a live collaboration opportunity and inviting colleagues to get involved.

To find users, follow your standard participant recruitment process if you have one. You can also use these tactics:

• When you’re recruiting people based in the city where you’re running the event, a LinkedIn post can be an effective addition.

• Draft a simple message that your colleagues can share across their networks; this travels surprisingly well.

• Tap into forums and user communities related to your product or service.

• Consider printing and sticking QR codes up at friends’ offices.

• Ask office managers to invite their staff in relevant roles.

• Finally, as a last resort, don’t forget the always invaluable help of family and friends.

The most efficient way to consolidate your recruitment operations is to create a single panel for product users to sign up. Don’t overcomplicate it; a simple form requesting a name and contact details is enough. If you have access to a specialized event tool like Luma (see Figure 1), use it to do the heavy lifting on waiting lists, reminders, and cancellations. Whatever you end up using, make sure to explain in detail how to get to your offices and what to expect.

Setting Up for (Repeatable) Success

Once you’ve successfully recruited all your participants, there are a few essential things you’ll need to coordinate and source. Most can be reused from session to session and require one-time investments of time and minimal budget. Here’s a list of must-haves:

• A physical room. Don’t get too picky. Make sure the room comfortably fits five tables, each with enough space to seat three people—two team members and a user. If the tables are conference-room-long, divide the groups at each end of the table. We run sessions in my office’s lunch area because there’s plenty of space and the tables move easily. Reserve the space ahead of time, and make sure a screen is available to share a slide deck.

• A presentation. Create a brief slide deck that covers who you and your team are, what the agenda is, and how the dynamic works (most of which you can borrow straight from this article).

• Nondisclosure agreement (NDA) forms. Print plenty of them, make sure users sign them, then forget about them. You can also send the NDAs ahead of time in recruitment, confirmation, and reminder emails, but people typically aren’t great at signing them in advance, so always keep paper copies in the office for anyone who hasn’t signed.

• A Sharpie and name tags. Raid the stationery closet; having more than you need is always better.

• A timer. You can use any timer, but we’re self-professed geeks, so we love Jake Knapp’s Design Sprint clock. Whatever you use, make sure it’s visible to teams so they can keep an eye on their time.

• Pizza. The jewel of the crown, and the only recurring fiscal investment in the effort. After many iterations, we’ve concluded that the right amount is six large or eight small pizzas. Always include one gluten-free pizza. Toppings like pepperoni, truffle, and ricotta are extra impressive. Order in advance and schedule delivery to happen halfway through the session: you won’t have much to do by then because the rounds are running, and the pizzas will still be hot and crispy by the time the session wraps.

• Feedback forms. You’ll need one feedback form for participants (include a QR code at the end of the presentation); leave it visible while everyone enjoys pizza and resend it in a thank-you message. You should also send a feedback form to your colleagues. Keeping it to a few basic questions is most effective: what to keep, what to improve, and what to add.

We don’t offer monetary incentives for taking part in a continuous discovery session. We get enough signups by letting users see what we’re working on, cocreate future solutions with us, take home some swag, get sent the event photos afterwards, and of course, network over drinks and food.

Timing Is Everything

We’ve found that Thursday at 5:00 p.m. works best: it’s a good preamble to the weekend and a strong competitor to the standard after-work get-together. We also make it look and feel like a social gathering.

Once the date is locked in, create two events in the calendar:

1. One for your colleagues, starting thirty minutes prior to the event start in case there are last-minute questions or help is needed with setting up the room. Also, having everyone on your team gathered in the room before users arrive is crucial: you’ll start on time, and it shows users they’ve walked into something that respects them and their time.

2. The second invite is for users. In the calendar invite, set the event to start fifteen minutes before the kickoff. That buffer works wonders for anyone commuting, and it protects your start time.

One hour before the start time, confirm all participants are still joining. Finalize the room setup: five tables of three chairs each, or five numbered spots distributed along longer tables (numbering the tables or spots in advance makes group assignments much easier). Place water and snacks (an optional add-on) on the tables. Test the tech and make sure your presentation and timer are visible. Order the pizza and schedule the delivery.

Thirty minutes before the start time, meet with your core event team for the day and walk through the checklist together. Once everything is in order, the support crew can head to the reception area to welcome the product users.

Designing the Doorknob Moment

At the end of the hour, when the final session is done, serving pizza is the perfect bridge for teams and users to naturally pick up conversations they didn’t have time to follow up on. It’s the equivalent of stopping the recording in a moderated interview, only to hear the user suddenly open up about something that wasn’t evident before—a trick I learned from Steve Portigal’s classic book, Interviewing Users,2 in which he wrote, “Physicians and therapists are familiar with the ‘doorknob phenomenon,’ where crucial information is revealed just as the patient is about to depart.”

So design the doorknob moment into your event intentionally. We aim to move to a narrower space so that colleagues and users have to walk through the crowd to reach the pizza, and conversations are struck up more naturally. It’s true that some folks prefer to use this time to quickly debrief and even make decisions on the spot, but we always make sure we’re the first to start conversations.

Finally, we also created a simple ritual to serve as our record of the event: a group selfie over pizza (see Figure 2). The photo is also incredibly useful for recruiting new participants through social media to join in future events.

Two Things That Should Work, but Don’t

Back to the event in Berlin, which I mentioned at the start of this article. Aside from starting late, we also decided to open with an icebreaker so everyone could introduce themselves: share your name and a superpower you’d choose, or something along those lines. We did this fifteen times (once for each colleague and product user), wasted a lot of time, and overwhelmed everyone before the event had even started. In short, cut the icebreakers and let the name tags do their job.

Also, since the product users were our guests of honor, my instinct had been to ask them to stay seated at the same table while colleagues moved to a new user. After all, I like to be accommodating. But that decision came with an unexpected cost. Moving five teams of colleagues is more disruptive than moving one user, and it only gets worse when the team has a “just one more question” look in their eyes. In contrast, when you ask a user to move, they tend not to object, and the team doesn’t complain about the move either. It works like a charm.

Turning Signal into Action

I keep telling everyone around me that if you don’t put enough effort into activating findings, it’s as if the research never happened—and the same goes for continuous discovery sessions. We have four mechanisms in place to help ensure that teams follow up on what they learned:

• Light documenting. The day after the event, an automated message pops up in various chat channels, reminding colleagues to put together a top-three-learnings slide using a template we’ve prepared (see Figure 3). We include this slide in our monthly UX research newsletter.

• Cross-pollination. We review the findings, distribute them across key channels, DM individuals, and often set up meetings between teams whose findings overlap. We believe that findings can act as a glue between teams and help a company focus on the user rather than its internal organizational chart.

• Continuity. Sometimes teams realize that what they learned isn’t enough, or that the problem is bigger than they anticipated. When that happens, we either support them with a dedicated follow-up study or fold the question into our own research roadmap.

• Praise, praise, praise. The first and second time you run a continuous discovery session, colleagues will likely join out of novelty. At some point, they’ll realize that the sessions create additional work on top of their day-to-day responsibilities, so participation might stall. To combat this natural decline, we drop colleagues’ names in relevant conversations, reinforcing that they took the time to talk to users and acknowledging the positive impact of doing so. This also sets them up as role models within the team. Finally, with their consent, we tag them in our external company social media posts, because who doesn’t like a little profile boost?

That’s as far as we go. We don’t chase people to act on findings or to explain the product decisions they made. For us, it’s more than enough if colleagues are exposed to users and have their assumptions challenged, even for an hour.

Scaling Across Locations

Kate Towsey’s book, Research That Scales,3 opens with a chapter titled “Research Does Not Scale—Systems Do,” and I couldn’t agree more. What we, the Miro UXR team, have built is exactly that: a continuous discovery research system that lets us run quick, cost-effective rounds of moderated, in-person qualitative interviews with users. Our system—the system I’ve just shared—is now so effective that once you’ve built reusable assets, you can run continuous discovery sessions across many locations. If you or your core team can’t travel to a particular location, you can easily train others to replicate the format. In our case, we started in Berlin, brought continuous discovery to Amsterdam, then we equipped people from the design team to run their own version in our Tokyo office.

Running Remote Sessions

While continuous discovery works best in person, we’ve also successfully run a remote version, though the format is obviously different. The remote version runs weekly rather than monthly, and aims to get five users in front of a single team in one day. We run it on Wednesdays, our no-meeting day at Miro, so it doesn’t conflict with anyone’s calendar. If you can find a similar opportunity in your organization’s calendar, leverage it.

In the case of remote continuous discovery, participant recruitment can be more specific. Users aren’t rotating among five teams with radically different ideas; they’re talking to just one product team. So if a team wants to meet with a particular profile, this can also be the place to do so. The rest works pretty much the same as the in-person version. You meet with the team beforehand to help shape the research plan and the discussion guide. They run the sessions using whatever moderated research tools you already have in place, and you keep an eye on your messaging app during the day in case something goes sideways.

If you can, run both in-person and remote sessions. They don’t compete. In fact, they complement each other in really nice ways. While the in-person version creates rhythm (the monthly cadence becomes a known organizational beat) and social proof (fueled by pizza), the remote version provides the opportunity to explore targeted questions. And if you want to run in-person events but don’t have a brick-and-mortar office, coworking spaces also work, as do partner organizations willing to lend you a room.

Breaking Silos, Building Empathy

Continuous discovery moves the needle in two ways that other research methodologies can’t: it breaks silos and builds empathy. Seating a customer success manager, engineer, and a product designer at adjacent tables and giving them the opportunity to listen to the same product users encourages curiosity about each other’s questions in a way that a messaging channel never could. I’ve watched more cross-team collaboration efforts start over a slice of pizza than in any alignment meeting.

Colleagues also leave continuous discovery sessions with a sharper sense of users’ needs, and with a better understanding of what the UX research team does and why it matters. That second form of empathy is the benefit that nobody plans for, and it’s the one that shifts how the organization approaches research. Teams stop seeing research as a bottleneck and become advocates because they finally have a low-stakes way to practice “customer-centricity.” Once you’ve created ease around something someone already wanted to be good at, they don’t need convincing to come back next month.

Credits

Thanks to my continuous discovery partner-in-crime, Anthony Li. Thanks, also, to Deniz Kartepe; Michael Schade; Lene Bayerlein; Gabriel Lovato; Zuza Jablonska; Bo Liu; Roman Mazur; and every PWDR and product user who has ever shown up.

The ResearchOps Review is made possible by User Interviews, now part of UserTesting. With a vast participant network, precise matching, and fraud prevention, User Interviews can reliably fill any research study. Source, screen, track and pay participants, then move seamlessly from data collection to deep analysis, all in one place. → Learn more about User Interviews for ResearchOps.

Subscribe now

The ResearchOps Review is supported by User Interviews, now part of UserTesting. User Interviews makes it fast, easy, and affordable to recruit participants so you can scale research without sacrificing quality.

Have you ever seen a screener response that looked flawlessly written, one in which every answer had perfect grammar and structure? Or perhaps you’ve been in a research interview and wondered whether the participant was simply an unusually focused person or an ardent fan of the product.

About two years into my role as a ResearchOps administrator at the software company, Red Hat, I noticed that skepticism of research participants among researchers had gradually become the default during recruitment and virtual interviews. Researchers started flagging up to 20 percent of participants in their studies as “fraudulent;” suspecting that they were using AI tools to answer questions. Encountering occasional fraudulent participants wasn’t new; we offered monetary incentives and had optimized our recruitment screeners to filter out both bots—built and managed by professional fraudsters—and less sophisticated individuals who feigned their expertise in order to qualify. Instead of being skeptical of incomplete, slow, or unpolished answers as they were previously, researchers were now also skeptical of people who seemed too polished. Too well informed.

In a User Interviews blog post titled “The Next Decade of Fraud in User Research: A Guide to Staying Ahead,”1 Michele Ronsen, founder of the user research consultancy, Curiosity Tank, framed this new wave of fraud as “faster, smarter, and oftentimes indistinguishable from legitimate participation.” Also from User Interviews, the “State of Research Operations 2025”2 report flagged junk data from AI bots and “participants using AI to circumvent open response screeners and even using AI during live sessions” as top concerns for ResearchOps teams heading into 2025. Because an increasing number of participants were being disqualified, I wondered how accurately research and ResearchOps colleagues could discern AI from humans. If we incorrectly flag a person as a “cheater,” we could damage relationships with customers and company partners who genuinely want to help improve the product. If we miss too many cheaters, we risk polluting our data with synthetic responses that we believed came from humans.

To better understand the challenge, I read up on what talent acquisition leaders were doing to combat cheating in job interviews—a similar context to research interviews. What detection tools, cheating “tells,” and methods did they use to screen out or scare away cheaters? While we can learn a lot from their frameworks, I wanted to investigate this threat specifically within a user research context. I designed and executed a pilot study in which both participants and researchers were anonymized to test our own team’s defenses against real-time AI tools in interviews. The results uncovered flaws in our own detection instincts and identified concrete operational shifts we could make to protect our data.

The Cheating Tools

In addition to the most popular AI tools, like ChatGPT and Claude, which can generate on-the-spot answers, a growing, well-funded industry of products has emerged in the past two years to help people answer questions during video calls. These tools listen to both the interviewer and interviewee, watch the screen, and generate context-specific answers within seconds of a question being asked. I didn’t find any cheating tools specifically designed with research participants in mind, but it’s clear that these meeting tools would work for remote research sessions, too.

Cluely is openly marketed as a tool to “Cheat on everything.” They claim a 95 percent transcription accuracy, a 300 millisecond response time (as long as it takes you to blink) and an ability to follow the eyes of the user to be undetectable. Immediately after discovering this tool, I asked my manager to do a quick test run with me. She downloaded the tool and quickly answered my questions about rock climbing (a topic she knew nothing about) using Cluely-generated responses. After witnessing the natural responses and real-time speed of the tool, the next step was to explore detectability. Researchers were confident that they could spot cheaters, but how accurate were they? If participants were able to fake answers so easily, cheaply, and covertly, even about topics they knew nothing about, could we continue to rely on intuition to disqualify participants? To find out, I ran a small pilot study to see if we could spot the cheaters and use it as a jumping off point for future investigations and to build solutions into our operations.

The Pilot Study

The study design involved four members of my team acting as interview participants. Each participant completed two short interviews on fun, non-work-related topics that they were either very familiar or completely unfamiliar with. Topics spanned TV shows like Friends, movies like Pirates of the Caribbean, hobbies like Brazilian jiu-jitsu, and interests like Anime. In the “AI-assisted” condition, participants were allowed to use Cluely. In the “Human” condition, they answered from genuine knowledge. A single moderator conducted all the interviews without knowing which condition any participant was in, in many ways mirroring the party game “Mafia,” in which unwitting players try to sniff out the “killers” based on body language. Then eleven observers from across the user experience design team, including researchers, managers, and people who do research (PWDRs), analyzed the recordings, identified which condition they thought each interview belonged to, and rated their confidence on a five-point scale. I also asked them to note why they made their judgments.

The results at a glance were as follows:

• Overall accuracy. Observers correctly classified responses as accurate in sixty-four out of eighty-two judgments, indicating that researchers were right more than they were wrong

• Genuine human responses. When evaluating authentic human interactions (research in which no AI assistance was utilized), observers successfully verified authenticity in four out of every five assessments. Again, that’s a good result.

• AI-assisted responses. When evaluating sessions in which participants used real-time covert AI software, detection accuracy dropped to three out of four assessments.

• The misidentification risk (false positives). The biggest data risk emerged when observers incorrectly flagged an authentic human participant as an AI user. Despite being entirely wrong, observers reported a confidence metric of 4.5 out of 5—they felt “very confident” to “extremely confident” in their incorrect judgment.

It was encouraging to see that researchers, overall, could identify when AI was being used in these interviews. However, the findings revealed a weakness when it came to using intuition: When observers were wrong about AI usage, they weren’t tentative; they were most confident. The moments when observers were sure they had caught a cheater were, in this study, the moments they were most likely to be wrong.

The Confidence Paradox

Researchers are pattern-matchers, and consistent interviewing experience can help them develop a sense or intuition of what authentic responses do—and don’t—look and sound like. But if their intuition hasn’t kept pace with the capabilities of the latest technology, and if more participants are using AI to cheat and researchers are on high alert, that same intuition can be misleading.

The business risk isn’t only false negatives, though that’s a risk worth addressing. A researcher operating on gut feel might disqualify a genuine participant, losing the data the team paid for, and damaging the participant’s relationship with the panel in the process. And because research operations rarely include a way to confirm the authenticity—or falsity—of the participant and the insights they shared afterwards, the researcher may walk away believing they made the right judgment and apply the same flawed pattern to additional sessions.

Luckily, intuition isn’t our only tool. If we can identify the true tells of AI assistance and build that know-how into how research operates, we can confidently flag the cheaters and protect our data. So how do you tell if someone is using AI to cheat?

The Tells that Told On Cheaters

The qualitative results of the study were especially interesting because they surfaced which cues the observers were using and which were predictive in remote interview contexts. The observers who took part in the pilot study said things like, “He repeated the questions aloud a lot, which felt like stalling for time,” and “she seemed to get flustered when the software was lagging.” And they were right. These markers showed up consistently in correct AI identifications:

• A fixed off-screen gaze. Especially one that returned to the same spot for every answer. Unlike casual upward glances while someone thinks, these were precise, repeated glances to the same location that almost always indicated the participant was reading a generated response.

• Filler language used as a stalling pattern. Participants inserted phrases like “I guess” or “that’s a great question” at the start of every answer, with a consistent pause length afterwards. This is often correlated with someone buying processing time to review the AI-generated answers. As another stalling tactic, participants using AI also often echo questions back to an interviewer.

• Hyper-specificity. AI-generated responses tended to be encyclopedic. They used complete sentences, contained unusual detail, and lacked the natural mess of how people actually talk about their jobs. Real experts ramble, contradict themselves, and sometimes even forget the name of the system they use every day, while synthetic experts do not.

• Typing after each question. Many models have done away with the need for participants to type. However, if participants use older models or their software doesn’t work as planned (a few glitches did occur when we were using Cluely), then clear typing sounds, especially during the silence after a question, are a strong indicator.

On the other hand, the markers that correlated most reliably with authentic responses were:

• Personal analysis. Participants frequently anchored their answers in subjective framing with phrases like “I feel like…” or “in my experience…,” and included mentions of messy realities or opinions they don’t believe are held by the majority. For example, a participant may admit that while the industry standard is to use a specific automated tool, they find it cumbersome and rely on simple spreadsheets instead.

• Natural forgetting or slipups. In authentic responses, it was common to hear mid-sentence contradictions, rambling, or participants admitting to forgetting a fact.

• Eye movement that drifts. When people think, their eyes often wander. An authentic participant’s gaze will tend to drift to retrieve a memory or formulate a complex thought. Crucially, these movements never lock into a rigid, repetitive loop. Unlike the fixed movements of someone scanning an AI-generated paragraph, authentic eye movement is less structured.

• Hand and body gestures that synchronize naturally with speech. Authentic participants used their hands, leaned into the camera, or nodded to emphasize a point at the exact time the corresponding word left their mouths. Because they weren’t dividing their cognitive energy between listening, reading a hidden screen, and speaking, their body language flowed with their voice.

The Cues That Fooled Us

It could be argued that pilot participants were aware that they were looking for a cheater and were, therefore, especially alert to suspicious behaviors. It’s true. But, as mentioned earlier, the prevalence of AI cheating means that researchers are now often in that state of vigilance or skepticism by default. Comments like “he responded too quickly to be using AI in real time,” or “she seemed to lean in to check her screen for responses” are tells that observers thought were dead giveaways, but they turned out to be unhelpful—or worse, misleading.

The two most prominent mistaken tells involved the speed of responses:

• Long pauses fooled almost everyone. People take a long time to think, especially when asked about something they haven’t articulated before. A real expert who pauses to find the right word is indistinguishable, on the surface, from a fraudulent participant waiting for a language model. Pauses aren’t an accurate indication of cheating.

• Quick, confident responses aren’t evidence of authenticity. AI tools can feed answers to participants in close to real time—again, in the blink of an eye—and a coached participant can read them fluently. With these tools advancing so quickly, you can no longer rely on quick delivery as a sign of authenticity.

Ultimately, these false indicators prove that the technology has evolved to the point that we must now re-evaluate our detection methods.

What This Means for ResearchOps

Most ResearchOps teams I know have built participant governance around a familiar set of risks, such as fraudulent identities, repeat participants, response bias, and the gaming of screeners to ensure acceptance into a study. AI-assisted participation overlaps with all of those, but it’s not a perfect fit for existing controls. Besides, the pace of AI tool improvement means that ResearchOps professionals, or anyone recruiting participants or interviewing users, is constantly playing catch-up.

To safeguard against the current generation of AI tools, here are four shifts to build into your operations:

• Replace gut-feel disqualifications with multi-cue thresholds. The confidence paradox means that an observer, however experienced or confident, should not disqualify a participant based on intuition alone. To support my team, I put together a moderator authenticity checklist of AI-cheating indicators (the fixed gaze, audible typing, and encyclopedic answers), authenticity markers (the personal analysis and natural forgetting), and an explicit reminder that long pauses and fast responses are not reliable tells.

• Encourage storytelling. Prompt researchers to include questions in their discussion guides that encourage participants to share personal stories, feelings, and opinions rather than just facts. Questions about failure, frustration, and abandonment can be especially telling. Real experts likely have a long list of things that didn’t work, workarounds they’re embarrassed about, and features they’ve given up on. AI tends to generate diplomatic, balanced responses that praise concepts and identify areas for improvement.

• Include AI provisions in your participant agreement. State plainly in the participant agreement that the use of AI assistance disqualifies the response and forfeits the incentive. Encourage participants to use their own voice and emphasize that perfection isn’t required. While this won’t stop the most determined cheaters, it signals to participants that you’re on alert for this type of deception.

• Consider deploying specialized interview fraud detection software. Some options include platforms like Sherlock AI and Polygraf AI that use multimodal machine learning and real-time speech transcription to flag unnatural response latency, mechanical eye-movement patterns, or perfect, AI-scripted linguistic structures. For qualitative AI-moderated interviews, Conveo or Strella add an extra layer of defense. These tools use adaptive AI moderators to push back on vague or diplomatic answers with follow-up questions intended to throw off participants using real-time AI cheating assistants.

Future Studies Are Essential

While this pilot study relied on a localized pool of eleven observers from the same team and focused on the tech industry, I see an opportunity for future research in this area—research that could help everyone better manage AI-enabled participant fraud. Expanding research to include a larger, more diverse pool of observers across the broader research community would reveal how detection accuracy varies across professional backgrounds and industries, and scaling the research to a larger dataset would provide the statistical significance needed to validate these qualitative patterns.

Additionally, shifting the research design from lighthearted, non-work topics to highly technical subjects would help determine if AI detection rates change when the cognitive load and complexity of the discussion increase. Investigating a broader variety of covert assistants beyond Cluely would also allow researchers to distinguish universal AI “tells” from product-specific cues.

Finally, introducing screenshare tasks, even with tools designed to remain undetectable, would add an extra layer of operational complexity that might discourage participants from attempting fraud. And if you do implement fraud-detection software to prevent this kind of cheating, test those tools to ensure automated systems don’t inherit the same biases found in human observers.

AI Cheating Versus Intentional Synthetic Data

When I first set out to explore AI cheating in user research, I found that conversations tended to cover two related problems and that people had strong opinions on how AI can and should be incorporated in user research. The first problem was participant AI fraud, the second was synthetic users: AI-generated profiles deliberately used in place of real participants.

In their article, “Synthetic Users: If, When, and How to Use AI-Generated “Research,”3 Kate Moran and Maria Rosala of the Nielsen Norman Group concluded that synthetic participants “cannot replace the depth and empathy gained from studying and speaking with real people. They often provide shallow or overly favorable feedback.” It’s worth noting that their article was published in 2024. Still, many researchers would come to that conclusion. However, just as general-use AI tools have greatly improved, so has the quality of synthetic data. Now, synthetic participants can be a quick, cost effective way to simulate a wide range of demographic profiles that are traditionally harder to reach.

In my opinion, synthetic data will be increasingly used as an intentional tool in user research, particularly for rapid ideation and hypothesis generation. But if we use it to supplement data from human participants, it’s even more critical that we can trust the humans we recruit to provide genuine opinions—long pauses, rambling, and musings included—rather than the generic AI responses that tools like Cluely generate.

The polished, perfectly phrased responses that once made researchers marvel at finding the “ideal” participant have become the very baseline of our suspicion. This constantly shifting landscape forces us to re-evaluate what authentic human participation actually looks like in a remote research environment. As ResearchOps professionals, our challenge is to stay ahead of the tools designed to deceive us, be aware of our occasionally flawed intuitions, and continue to seek out genuine responses. Ultimately, the very technology creating this trust crisis will likely be the tool we use to solve it.

The ResearchOps Review is made possible by User Interviews, now part of UserTesting. With a vast participant network, precise matching, and fraud prevention, User Interviews can reliably fill any research study. Source, screen, track and pay participants, then move seamlessly from data collection to deep analysis, all in one place. → Learn more about User Interviews for ResearchOps.

Subscribe now

The ResearchOps Review is supported by User Interviews, now part of UserTesting. User Interviews makes it fast, easy, and affordable to recruit participants so you can scale research without sacrificing quality.

In a March 2026 Harvard Business Review article,1 which is paywalled, Julie Bedard and colleagues introduced the memorable term “AI brain fry” to describe the cognitive fatigue that can result from using AI in particular ways. They surveyed 1,488 full-time U.S.-based workers (48 percent male versus 51 percent female; 58 percent independent contributors versus 41 percent leaders) at large companies across industries and found the most taxing piece of AI usage was “oversight:” the mental burden of monitoring, editing, fact-checking, and validating AI-generated outputs. Respondents who reported high oversight demands used 14 percent more mental effort and experienced 12 percent more mental fatigue than those who reported lower AI oversight demands. They also found that the prevalence of brain fry varied by function, with marketing, human resources, and operations most affected (see Figure 1).

Shifts in the economy, layoffs, a focus on operational efficiency, and the emergence of AI mean that many researchers are now formally, or informally, delivering research operations, and they’re enthusiastically building AI systems to meet the demand. Over the past two years, I’ve been one of those researchers. Apart from the cognitive cost of overseeing those systems and their outputs—the oversight that Bedard et al described—self-built and managed AI tools produce another type of cognitive load: a “second layer” of orchestration, governance, and maintenance work; operational work that I now largely handle—work that’s mediated through prompting.

Every AI system I’ve created has helped me solve real research problems—and created new ones. I call this the second layer. Some of these challenges look like the same old research operations problems, such as figuring out who maintains the tools. As a solo researcher who now also builds AI systems, I feel the burden of the second layer more than ever. The cognitive load of building (and maintaining) AI capabilities, such as research repositories or data organizers, is different from the load of writing up a research report or producing a recruitment screener using an LLM, but they both produce brain-frying mental loads. So, what’s the solution?

With new technology and the power to build new things, I’ve come to realize that the only way forward is to answer these new questions: Who’s going to manage these newly created tools that I’m building? How should I adapt to better manage brain fry? If the list of tools I’ve built grows, and my list of maintenance to-dos grows with them, how will I ever have time to do research?

Putting a Pause on the Builder Mindset

Two years ago, I joined the insurance company ERGO NEXT Insurance as a solo UX researcher responsible for end-to-end research across the entire company, from product to marketing. Since then, I’ve used AI to create simple things like an app that mass-converts file types, and more complex systems such as a video clipping tool, a transcript-processing pipeline, and a Claude Code “living brain” that cross-references findings across more than 100 interview transcripts and other data sources. For the first time, a lack of tooling isn’t a blocker to doing research exactly as I want. If I need a survey tool, I can build my own. If I need a knowledge management system, I can make it happen. This builder mindset2 is shifting the field for research professionals, as it is for most professions. But the fact that you can build an AI tool doesn’t mean that you should, or that it will be easy to maintain, or that it will scale endlessly.

An AI Pandora’s Box

Not long ago, I used Google’s Retrieval-Augmented Generation (RAG) tool, NotebookLM, to build a research repository; since then, I’ve used Claude Code to build an even better repository. When I first presented the NotebookLM at our product offsite, stakeholders from across the company wanted immediate access, for their segment, their project, and their use case. To provide relevant content to everyone, I needed to manually review and add metadata tags to over 100 additional transcripts and add them to the NotebookLM. Normally, this process would take hours, but I quickly built an automation with the help of the AI coding tool, Cursor, and Gemini. Now, I could drag a raw transcript into a folder, where a Python script removed timestamps, standardized speaker names, removed filler words and Personally Identifiable Information (PII), and scanned the interview to add structured metadata tags: date, user segment, interviewee role, and topic keywords. (The metadata tags help the LLM find the right resources without needing to scan the entire text, which also happens to save on tokens.) Finally, the script renamed the file in a standardized format and moved it to a cleaned folder. The only thing that wasn’t automated was uploading the transcripts to the NotebookLM.

This story is important because it’s rare for one tool to handle a complete workflow on its own. Building one tool often means building other tools to make the system work, and as one technology evolves, you may want to retire one instance and all its associated tools to spin up something that’s even more powerful. The lesson is this: though AI offers the ability to build tools from a photo of a whiteboard, it’s still useful to pause and consider the primary principles of system design3—scalability, availability, reliability, latency, throughput, consistency, and fault tolerance—before you start building.

AI Multiplies Throughput (and Operational Overhead)

AI has shifted not only what research is possible but also how much research is possible, and the scale of the data I collect. For instance, research interviews—such as post-purchase and churn interviews—which were previously out of scope are now done regularly with the help of AI moderation tools. As a result, there are more interview transcripts, each of which needs to be cleaned, de-identified, tagged, and tracked before it can be added to the research repository. Of course, I built the tool that handles this cleaning process, but maintaining that system is its own job, too, and the repository can’t function fully without it.

When you build a tool, it’s essential to ask: What scale will this system need to handle: how many transcripts, queries, or recruits, for instance? What other systems will it rely on or create reliability with? How reliable does it need to be? But systems aren’t purely mechanical, so you should also consider, as always, the larger ecosystem of people and culture in which it will launch and live.

Beware of Cognitive Surrender

Once I’d rolled out access to the Google NotebookLM, stakeholders started saying things like, “We made this decision because Jordan’s NotebookLM said to.” But NotebookLM’s function is to surface research, not to make product decisions—an excellent example of Wharton researchers, Steven Shaw and Gideon Nave’s “cognitive surrender.”4 Admittedly, the outputs are good enough that it’s easy to treat them as authoritative, but as the architect of these sorts of “instant knowledge” systems, is it now my job to govern and educate people on how to use AI systems for research? It seems it is.

Building any tool should include a consideration of the cultural implications: Who will design and provide training, governance, and guardrails? How will you onboard people so they’re equipped to use the tool responsibly? How will you monitor correct usage? What is correct usage? And, importantly, how will you evaluate output?

As any AI-fluent researcher will tell you, regularly evaluating AI outputs to ensure reliability is essential (I believe that, too), but, as a solo researcher and now an AI tool builder and maintainer, I don’t have the time to design and run a full-scale evaluation (or “eval”) process every time I use a tool. Instead, I usually run ten evals on a single transcript, and if everything looks good, I’m good, too. But the lesson is: when you build an AI system, plan to design a permanent quality control loop, and plan in time to manage it. When I have the capacity, I plan to incorporate a more systematic review process using Lindsey DeWitt Prat’s five-step evaluation blueprint, which she shared in this Review article, “Winning the Game of Broken Telephone: A Blueprint for Evaluating AI Across the Research Pipeline.”5

Escalating Orchestration

Not long ago, one of my colleagues shared a tweet by Andrej Karpathy, a prominent AI researcher and founding member of OpenAI, about a concept he called an “LLM wiki.” I gave Claude Code the tweet with some explanation about what I wanted to build: a compounding knowledge system in which each new piece of content is cleaned, tagged, analyzed, summarized, and cross-referenced against everything that’s already stored in the system. I called this system the research brain. Using the research brain, I can control the methodology for how transcripts are analyzed, how research findings are cross-referenced, and how themes are surfaced. I can give the brain an outline of the business context and ask it to write up articles—and all of this information compounds over time to make the brain even more relevant.

Two years ago, building these capabilities would have required a dedicated engineering team. That I can achieve this on my own is remarkable, but building and maintaining a system of this complexity requires a lot of orchestration and detailed workflow mapping. I even drew the entire workflow on a whiteboard so Claude Code could use it to help improve it (see Figure 2). The builder is building the app, but the app is also building itself.

To return to the notion of AI brain fry, when you’re building or using a relatively simple tool like NotebookLM—you ask a question and get an answer with citations—the oversight expense is fairly low. If you’re building agentic systems like the “research brain,” the orchestration escalates. You’re mapping multi-step workflows, writing metadata schemas, creating Claude Skills (a Skill is a simple set of instructions that Claude can use to handle repeatable workflows) for each stage of the pipeline, testing outputs, and evaluating reliability. Managing this even for one user, never mind dozens or hundreds of people across the organization, can get so complicated that, soon enough, you’ll regularly need to document the entire workflow just to explain it to your team, and to Claude itself. In an effort to mitigate some of the brain fry, I recommend creating a Claude Skill for each step. Once you have a set of Skills, make sure each of them works independently, then string the Skills together to create a multi-step workflow. A piece-by-piece understanding of your system makes for better management than building it all at once.

Building AI Sustainably

In February 2026, Aruna Ranganathan and Xingqi Maggie Ye at UC Berkeley shared findings from an eight-month study of how employees at a 200-person tech company worked with AI. They found that AI expanded rather than reduced employees’ work because people took on more, stopped taking breaks, and kept multiple threads with AI. Read more about this work in their Harvard Business Review article, “AI Doesn’t Reduce Work—It Intensifies It.”6 To combat issues of AI brain fry and overwhelm, they recommend building in intentional pauses before major decisions, sequencing work into phases rather than reacting to every AI output, and protecting time for human connection. Why not meet with your colleagues to share what you’re working on and get feedback? I’ve also found human connection to help me slow down what I’m working on—to pause, even if for a moment.

I built all of these tools because I had problems that needed solving, but managing brain fry and the second layer is increasingly essential to maintaining operations—how I operate as part of that system, too. If you’re a research professional, lean into the builder mindset, play and experiment, stay current with AI tools, and don’t get discouraged by blockers. These tools will give you the ability to create capabilities that would have been unthinkable even two years ago. But as you adopt this new mindset, find ways to proactively manage the burnout, which sometimes means building less, slowing down to consider system design, or retiring inventions that aren’t useful.

So before you vibe a tool into existence, ask these age-old operational questions:

• Will this system scale? Will it stay reliable? And what will it cost, both financially and cognitively, to maintain? In systems terms: scalability, availability, reliability, latency, throughput, consistency, and fault tolerance.

• How will it jibe with the people and the culture in your organization?

• What will you need to do to manage how the tool is used?

Also, plan and document. If you want to build an extensive system, create a project plan with goals, a timeline, and an approach for how you will evaluate success so that you know what you’re working towards. Document or journal your daily progress so you can reflect on and share your building process. I often build something, then discover that I’ve got no idea as to how I got there, and that’s how the second layer creeps in.

In order to address the second layer, you need to name it, and I find that regularly documenting your process helps. In addition, by creating documentation, you’ll need to manually write out the workflow you want the AI to replicate (you’ll pause and think), which will also help you explain it to the LLM more accurately. You can do this by creating visuals or a written document. If you’re working in Claude Code, you might ask it to keep a running decision and work log, recorded by date, during a building session. By having Claude maintain the decision log, you’re offloading some of that second-layer orchestration overhead onto the system itself, rather than owning it yourself. And if you’re ever unsure how to solve a problem, Claude can probably help!

Avoiding the Iteration Trap

The 2025 User Interviews State of User Research7 report found that 80 percent of UX researchers use AI to support their work, up from 56 percent in 2024. Researchers appear to be using AI more, but the same report found that sentiment is mixed. Forty-one percent of UX researchers reported feeling negatively toward AI’s impact on user research, and only 32 percent reported feeling positive. That’s a pretty big gap. These numbers confirm my sense that AI is a double-edged sword: it’s empowering, but it’s equally easy to feel forced to use it to survive—to move faster and faster still. From the same report, this quote stood out:

“AI is becoming a focus over bigger customer problems in a way I don’t love, and although it helps with lower lift tasks, I feel pressured to use it or else I’ll be left behind. It speeds up my work but at the cost of my retention and sanity and quality of output. I fear I’m losing my voice the more I use AI, but I have to use it out of necessity to get things done at the pace I’m expected to.”

When it comes to building AI systems, I keep coming back to these questions: When is something that I’ve built truly finished? When is any of it good enough? Is it ever “finished?” Like a painter in front of a canvas, it’s hard to know when to stop—when to put the brush down. Claude gets better with every iteration, and I can keep adding steps, refinements, and complexity without a clear benchmark for “done.” I’m also building as quickly as I can, which is fast. Nowadays, fast doesn’t feel fast enough. Part of the solution is to use system design thinking to more carefully consider the technical and cognitive implications of what you build—just because you can build something doesn’t mean that you should.

The ResearchOps Review is made possible by User Interviews, now part of UserTesting. With a vast participant network, precise matching, and fraud prevention, User Interviews can reliably fill any research study. Source, screen, track and pay participants, then move seamlessly from data collection to deep analysis, all in one place. → Learn more about User Interviews for ResearchOps.

Subscribe now

Worldwide, research professionals are finding practical ways to use AI to operationalise research, and they’re working out the guardrails needed to ensure that gains don’t come at the expense of quality, governance, and trust—particularly as research is democratised.

To explore these themes, The ResearchOps Review and Rally are co-hosting Rally Around ReOps, a one-day, in-person ResearchOps conference. We’re bringing together seven speakers and eighty research professionals for a day of talks, a closing panel, and plenty of time to connect.

A Snapshot of the Agenda

Speakers will share six practical talks covering two core themes:

The machines of research. You’ll hear from speakers who are delivering significant AI-augmentation gains, particularly in research democratisation, as well as the operational changes and philosophies that made those results possible.

The humans of research. Despite the potential benefits of augmentation and automation, human curiosity and connection remain essential not only in research but also in operations. But which aspects of humanness matter most?

Find Out More & RSVP

We’ll close the day with a panel, “Building for Speed without Losing the Human Touch,” hosted by me, Kate Towsey, and Oren Friedman, ​Rally’s cofounder and CEO, and featuring speakers from the day.

There will be coffee and lunch breaks, as well as time to connect with speakers and attendees at this exclusive event.

Research Privacy and Ethics in the Age of AI

Join me the following day for an in-person, two-hour working session focused on research privacy and ethics in the age of AI. It sounds incredibly serious—and it is! It’s also an excellent opportunity to contribute to the field and connect with others who are thinking about this important topic.

I hope to see you there.

Kate Towsey
Founder & Editor in Chief of The ResearchOps Review

Brought to You By

Scale research operations with ​Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack. Join the future of research operations.

Subscribe now

How to AI UXR is supported by Strella, a customer research platform that uses AI to run in-depth interviews and generate actionable insights in just a few hours.

Successfully integrating AI into the user experience research (UXR) workflow is the top objective on almost every research and ResearchOps professional’s mind. How to AI UXR is a map for building AI-augmented research operations. It was produced to give you an overview of the key trends, help you identify the maturity level at which you’re implementing AI—crawl, walk, or run—and provide you with a list of implementations being used by research professionals across the globe, which you can experiment with too.

Download the Map

As we discovered while making this map, the topic is vast and progressing so quickly, both within and beyond the field, that one can hardly keep up. In fact, the Run level detailed on the map is still evolving—it’s a primordial soup of innovation. As such, the How to AI UXR map is a snapshot in time (who knows how long it will remain accurate), but we hope it will set the tone for the kind of systemic, operational work you can (and should) do with AI rather than the “I-Me-Mine-AI”1 approach that has dominated the field.

How to AI UXR: The Map

113KB ∙ PDF file

Download

Download the five-page map, including an introduction, the Crawl, Walk, and Run levels of AI augmentation, and a glossary.

Download

The How to AI UXR map includes information on how it was produced, how to “read” it, a list of contributors, and a glossary—AI is introducing so many new terms, it can be hard to keep up. As you will no doubt agree, the map is also detailed. The production unearthed significantly more information than could be shared in a single infographic. Apart from introducing the map, this article covers what didn’t fit: key trends, the specifics of AI-augmented research operations, and the downsides that must be managed alongside the significant upsides that AI is offering many intrepid research teams.

The Five Shifts Reshaping UXR

It will come as no surprise that AI is fundamentally reshaping how UX research is done and consumed. But what’s more interesting is how quickly and intrinsically AI is changing the roles of research professionals,2 and how it’s being used to enrich and amplify both human-produced research and research operations.

If there’s one insight the How to AI UXR map aims lands, it’s this: AI’s most meaningful impact on research won’t be a collection of time-saving tactics; it will be the shift from individual work to systems work; from “How do I use AI?” to “How does our organisation build research that is faster, safer, and more reusable with the help of AI?” This shift is particularly well illustrated in the Run level of the map (see page 4). These are the major themes that emerged from the production:

1. Proactive rather than reactive. “The insights are great, but too late,” is fast becoming a complaint of the past. Well-designed AI systems allow research professionals to operate proactively. The focus is now on insight quality management and on predicting what stakeholders will need before they can articulate it. To support this shift, research teams are doing gap analyses across roadmaps, repositories, and recruitment panels, then delivering operations updates, insights, and weekly summaries before stakeholders have thought to submit a request.

2. Conversing with data. Analysis and synthesis are shifting from linear coding and sorting to a reflexive dialogue. As one contributor put it, “The amount of time I spend ‘conversing with my data.’ I think I’ve actually developed a deeper understanding of our users.” Rather than use AI to automate analysis and synthesis, savvy researchers are using AI as a whetstone to sharpen their thinking, query their data for contradictions and edge cases, and surface their own cognitive biases. For more on this topic, read “Calibration Matters More Than Automation: What AI’s History Suggests About Building Agentic Research Systems”3 by ResearchOps consultant George Jensen.

3. The “make” phase. The democratising power of AI means researchers can now more easily bridge the gap between insights and implementation. Researchers are vibe-coding functional prototypes, co-creating designs with participants in real time, fixing low-risk user interface issues in the product, producing on-brand copy, and, most interestingly of all, sharing insights not via decks or reports but in the “language” of designers and product managers: as high-fidelity prototypes. No longer is the democratisation of specialised crafts only a concern for researchers; designers and engineers are now concerned, too.

4. Qualitative research at quantitative scales. AI-moderated sessions are enabling researchers to run hundreds of interviews per week, and AI analysis is allowing them to parse huge qualitative datasets in ways that were previously impossible. For some researchers, that sentence may be enough to keep them up at night, but, rather than create black box research—invisible processes that seem impossible to interrogate—researchers are using these technologies to size issues before they commit to human research, prepare for high-risk human research, or to enable low-risk studies that would previously have been deprioritised. AI moderation is proving to be a valuable lever for research teams, but as with unmoderated research, you must define and implement the right operational guardrails and repeatable evaluation, or “eval,” practices to ensure speed doesn’t deteriorate quality.

5. Insights delivered in multimedia formats. Static PDF reports (another traditional bugbear for research professionals: “Why did no one read my report?”) are being replaced by multimedia content like podcasts, interactive vibe-coded websites, and Retrieval-Augmented Generation (RAG) chatbots that allow research consumers to use natural language to discover what they need to know, when they want to know it. If you’re unfamiliar with RAG systems, McKinsey offers a useful explainer: “What Is Retrieval-Augmented Generation (RAG)?”.4 Which of these multimedia content experiments will stand the test of time—novelty is finite—and the increasing issue of information overwhelm is anyone’s guess. This writer thinks that RAG-based chatbots are the best bet.

ResearchOps Is Becoming Agentic System Design

AI is ultimately a knowledge technology that has made most, if not all, activities within the research workflow systematisable. For this reason, the How to AI UXR map doesn’t include a dedicated section for ResearchOps or, on that note, knowledge management—it’s all systems, so it’s all research operations, and it’s all about managing knowledge. This same evolution is also making research and ResearchOps roles increasingly indistinct. Many researchers are transitioning from “traditional” research roles to almost full-time system design roles. Simultaneously, ResearchOps professionals can no longer focus solely on research system delivery; to deliver AI-augmented research systems, they must now also have a deep understanding of research craft.

That said, six themes emerged that are specifically related to the ResearchOps role, and they’re worth noting here:

1. From support to system design. This transition was already underway well before AI,5 but AI has accelerated it. Rather than offer administrative support, the primary role of ResearchOps and the increasing number of researchers delivering operations, is to build the infrastructure that allows AI-assisted research to happen efficiently and safely across entire organisations. “Run” research professionals (see page 4) are now designing complex agentic systems and repeatable eval processes to support ongoing reliability.

2. Intake, triage, support, and routing. At the Walk level (see page 3), operational systems now regularly include “front door” support bots and agentic intake processes. These systems use custom parameters to offer advice and decide whether a request should be handled via a self-serve template or routed to a human. As one contributor noted, “I’m working on an AI skill this week to act as the intake process for people who do research (PWDR). With certain parameters, such as low or high risk, the agent suggests whether to self-serve or work with UX management on researcher resourcing.”

3. Knowledge management and semantic search. Research knowledge management has been an ongoing challenge for research teams for over a decade, and AI is fundamentally changing the landscape. Primarily, there’s a move from keyword-based search to semantic and vector-based search layers. Repositories trained on research methodology are becoming increasingly hierarchical, and RAG-grounded environments where stakeholders can “converse” with research rather than reading static reports are becoming standard. I built a RAG repository to support the synthesis of the How to AI UXR data.

4. Data-informed learning and development. This is one of the most unexpected and powerful areas for leveraging AI. It’s shifting professional development from generic training to real-time coaching and automated critique systems that analyse research activities against organisational best practices. These tools—often implemented as specialised agents—provide immediate feedback on interviewing skills, identify leading questions in discussion guides, and help non-researchers navigate complex customer archetypes. Interestingly, in the case studies I’ve seen, people seem far more open to constructive feedback from a well-trained AI than from a fellow human.

5. Automation and pipeline integration. Research professionals operating at the Run level are building end-to-end AI pipelines, often Python or n8n-based, to automatically clean transcripts, remove personally identifiable information (PII), enrich metadata, and stage files for analysis the moment a research session ends. Files are also often prepared for intake into the research repository, with human validation. As you’ll learn in the following section, this sort of end-to-end automation requires significant, operationalised mediation.

6. Quality assurance (QA) as a new discipline. AI evals are emerging as a core operational discipline. This includes building systems that enable regular human-in-the-loop (HITL) monitoring of agentic outputs, such as building multi-agent auditing systems in which one agent extracts findings while another independently checks for alternative interpretations or missing evidence. As one contributor noted, “I created an agentic system with Claude Code: one agent extracts findings from interviews, another generates insights, another checks interviews for missing evidence, and another checks for alternative interpretations.” For more on evals, read “Winning the Game of Broken Telephone: A Blueprint for Evaluating AI Across the Research Pipeline” by Lindsey DeWitt Prat.

Seven Risks to Manage as You Scale AI

This map and the contents of this article will no doubt agitate many researchers: “Are you trying to say that AI can do my job?” Many researchers are now embracing AI, and it’s delivering promising upgrades to research, as the How to AI UXR map illustrates. But researchers working at the cutting edge of AI augmentation are equally aware of the critical downsides—not all are technological downsides—and the hard requirement that skilled humans remain in the loop. As a result, they’re building essential “AI antidotes” into their systems. These are the primary concerns:

1. The disintermediation risk. There’s a growing concern that AI-powered “answer engines” will reduce researchers to data aggregators, and that partners will shop for insights that align with their direction of travel rather than commission original research that may contradict it—never mind that it may take substantially more time to produce. An equally important risk is that AI makes research seem so easy that knowledge seekers bypass the essential interpretive scaffolding that makes research findings trustworthy—and that rushed or inexperienced researchers do this, too. None of these problems is new, but AI (the ultimate amplifier) has made these issues more obvious and pressing.

2. Critical familiarisation. Even though AI can process data and produce polished-sounding results, often in moments, experts emphasise that “steeping” in the data remains essential. A well-architected model council (a multi-model research architecture enabling you to simultaneously query several AI models to provide a unified, cross-verified response) may be able to do some evaluation for you, but a direct understanding of the raw data is the only way to guarantee that insights aren’t only accurate but also retain their richness—and truth.

3. Synthetic data loops. When AI study preparation, AI moderation, and even AI participants produce all of the insights (in other words, when human insight is completely removed from research), the risk of unreliable insights is, unsurprisingly, significant. But that doesn’t mean that these tools aren’t handy. To counter synthetic loops, savvy AI users are building systems that provide clear visibility into every stage of the automated pipeline, such as utilising model councils to audit findings and identify alternative interpretations.

4. Fraud and quality. Increasingly realistic fake participants and AI-enabled responses mean that research professionals need to implement both manual and automated checks. One contributor said, “I’ve come across sessions where it appears that participants are reading out AI responses to questions, potentially creating unwanted synthetic data,” while another shared that they’ve seen “increasingly realistic fake participants—even in video and audio.” Ironically, AI is also being used to counter fraud: research professionals use heuristics (rules of thumb) to flag low-quality participants, and agents to watch for inconsistencies, overly generic or scripted responses, and suspect applications.

5. The verification paradox. This is a critical strategic concern: as the volume of AI-generated research increases, the human ability to verify every citation or summary shrinks, leading to a reliance on potentially hallucinated patterns that can’t be unpicked. To counter the verification paradox, researchers are implementing the multi-agent auditing systems already mentioned, utilising source-linked RAG repositories for instant validation, and refusing to skip the human-led data familiarisation stage for high-stakes work.

6. Organisational and contextual blindness. Current AI models don’t understand office politics, organisational dynamics, or the stakes involved in a research study, or which insights will land with specific stakeholders. At the Run level (see page 4), researchers are designing agentic systems grounded in internal strategy and organisational topology while retaining human-led interpretive scaffolding to ensure insights resonate with the specific stakes and social dynamics of their audience.

7. Shallow insights, false speed, and bias amplification. As is now well known, AI outputs look polished but often lack substance, context, and nuance, increasing the risk that teams build the wrong solution, just more efficiently. Savvy researchers are no longer accepting the first summary a model provides. Instead, they command the model to search for the “needle in the haystack” and ask for contradictory evidence, outliers, and edge cases to sense-check dominant patterns. They’re also working to augment human-led research rather than hand over research generation entirely.

A Map to Help You Navigate, Negotiate, and Slow Down

In producing this map, the feedback has been contradictory: the map presents too much information; it’s overwhelming. The map doesn’t present enough information: we want a list of tools, citations, pace layers,6 and more. As a research professional, you’re likely to know this conundrum well.

We chose to offer a detailed-as-possible map, but one that stripped out as much noise as possible so you can read it as a sort of menu of the ways other research professionals are using AI to augment their workflows. It’s an attempt at documenting enormous complexity in a space that’s moving at breakneck speed, and brings to mind these lyrics from “Headlong” by Queen:

And you’re rushing headlong
You’ve got a new goal
And you’re rushing headlong
Out of control

This map is an attempt to offer some control, a launch pad for negotiating with leadership, and a way to set an AI strategy (while you play and experiment with this new technology, which is an important theme, too). But this map would be even more powerful were you to interpret it and evolve it. The map is copyrighted by The ResearchOps Review, but should you annotate it or be inspired to create something better, please let us know. We would be excited to see what you create, and inspired to keep the conversation going. Please mention The ResearchOps Review on LinkedIn.

Contributors

Thanks to the following AI innovators and makers for their contributions to this production: Adam Valerio, Allison Robins⁠, Angelica Eling⁠, Anette Petersen⁠, Anshuk Chhibber⁠, Arev Pivazyan⁠, Austen Lazarus, Aya Abdelgawad⁠, Brian Greene⁠, Brooke Sykes⁠, Carina Cook⁠, Caroline Cox-Orrell⁠, Casey Gollan, Christen Penny, Corina Kesler⁠, Diana Sapanaro⁠, Dr Asma Qureshi⁠, Emily DiLeo, Farah Faisel, Filip Uzarevic⁠, Graham Gardner, Hannah Mattil⁠, Heidi Austin⁠, Jordan Brinkman⁠, Kaleb Loosbrock, Kalee Dankner, Kathy Shi⁠, Kerttu Sobak, Katie Roehrick⁠, Lydia Iana, Madeline Winer⁠, Marshall Baker, Michel Vogel⁠, Naki Ossom⁠, Nathan Pena⁠, Nicole Hack⁠, Rachel Wigen-Toccalino⁠, Rebecca Klee⁠, Rita Casillas, Shane Melton⁠, Sohvi Silius⁠, Stephanie Kingston, Stephanie M. Pratt⁠, Tamia Sheldon⁠, Tarah Srethwatanakul⁠, Theresa Flood⁠, and Uyhun Ung⁠.

How to AI UXR is supported by Strella, a customer research platform that uses AI to run in-depth interviews and generate actionable insights in just a few hours.

Subscribe now

The ResearchOps Review is supported by User Interviews, now part of UserTesting. User Interviews makes it fast, easy, and affordable to recruit participants so you can scale research without sacrificing quality.

On many occasions, I’ve watched product executives sit around a table with their peers and make the case for major initiatives based on research data they happened to remember in the moment. Even if their memory was correct (and that’s a big “if”), and customer research was being applied to great effect, that particular insight couldn’t be verified because it was impossible to find the data that sat behind the insight: there was no traceability. If the author of the research happened to be in the room, they might have been able to point out the insight’s origin and context. But research authors can’t be guaranteed to be in the room every time their research is quoted, or be expected to remember every source. While it’s possible to systematize this capability, most organizations—at least those that aren’t in the scientific or medical fields—haven’t made that investment. As a result, they’re not able to easily check an insight’s source—more formally, a citation—to gain an on-the-spot understanding of its context.

“Context” can mean different things in different studies, initial research uses, and applications of insights over time. For instance, robust ethnographic methods can build thick contextual descriptions of customers’ worlds, while narrow-scoped surveys can maintain the contextual fingerprint of a particular sample and its timing in product development. Even in sparse, agile documentation, carefully chosen wordings can tell a larger story of uncovered needs and possible actions. During a research study, at least when collaboration is ideal, researchers and their stakeholders tend to share stories during meetings, observation sessions, show-and-tells, and via messaging apps. Over time, they’ll build up a shared context that colors every interaction they have about the study. These stories are knowable because they’re grounded in collaboration. But after a study is complete and the researcher and their stakeholders move on, layers of context are easily lost. Maintaining contextual provenance can become a major knowledge management challenge, and an opportunity to build storage, structures, and connections that operationally squeeze value from existing research.

This article outlines how leveraging citations in research repositories can enable more context-rich usage of research insights, how links from plans into repository contents add value for both product decision makers and insight generators (and provide more research value to the organization), and how to effectively operationalize those linkages via citation patterns. I’ll share good, better, and best examples.

Understanding Citations—and Why They Matter

If you’ve read (or written) an academic research paper or book, you’re likely to have come across a formal academic citation. A citation is simply a reference or credit to the original source or sources. In the context of user experience (UX) research, citations are links that a researcher, team, or ResearchOps professional can follow to understand whether and where a particular insight has been applied in product decision-making. Every researcher loves it when decision makers cite the insight—their insight—that informed a particular choice.

Even though generative AI tools are normalizing reference links—deep-research tools like Google’s NotebookLM and Perplexity include citations as standard—the word “citation” can seem just a bit too academic for the product development context. I suggest avoiding the formal citation standards, such as APA, MLA, Chicago, or IEEE, for this type of usage.1 The goal of using citations is to create context between research and product documents; the focus is on making context traceable, not nitpicking or achieving academic perfection. Still, the primary premise of a citation is invaluable in the product context, and for the following reasons.

Enabling a Deeper Dive into the Evidence

While working in ResearchOps knowledge management, I often had to follow up with the authors of roadmap themes, product requirements documents, designs, and other deliverables to learn where a cited customer quote or statistic came from. For some of these queries, the answers were less than convincing, balancing on a cherry-picked snippet of information used to justify an idea already in-flight. Other times, because there was a citation, I could clearly see the insight’s source research. I knew I could drill down into a report or reach out to the researcher to learn more. The ability to gather more context helped me build trust in using traceable evidence to support a choice. But it’s not just about what worked for me personally; citations can help scale trust across a product organization, too.

Improving Research Impact Tracking

Researchers working in technology organizations are often asked to demonstrate the impact of their work. To effectively meet this expectation, many research practitioners find ways to proactively record their influence and results. In a Learners conference talk called “Impact and UX Research: What Is It and How Do We Know We’ve Achieved It?” Victoria Sosik, a senior director of experience research, shared some basic trackable categories of impact, such as prompting future collaborations and influencing product strategy, as well as logging individual wins (or citing them) so they can be compiled and shared in status reporting.

After researchers wrap up their studies, tracking connections between research and planning can get cloudy. Researchers often hope that decision makers will continue to find value in and make use of their research repository beyond an initial project. But even when “old” insights make a difference in current plans (backlogs, specifications, user stories, roadmaps, goals, and designs, etc.), those impacts can be untraceable unless there’s some sort of citation.

Building Context in Research Repositories

I began focusing on research repository design in 2013, while I was a principal researcher at Amazon (eventually transitioning to a ResearchOps role) and when research repository work was at the frontier. The UX research team at the time had generated extensive, in-depth insights that had informed measurable impacts for customers and the business. Even with these immediate wins, researchers’ collected body of past learning still had a lot to offer future initiatives—there was still juice to squeeze from the research fruit. To harvest that research wealth, the team adapted a commonly used, off-the-shelf issue-tracking tool to serve as an insight-level research repository. Over time, adoption of the repository grew across a range of product teams in three divisions.

As adoption increased, I noticed something surprising: decision makers started adding research citations into their own work artifacts, such as product requirements, roadmaps, and strategy documents. References to specific insights appeared in all sorts of deliverables, sometimes created by colleagues whom none of the researchers had met. Notably, citations appeared in product teams’ launch announcements. Some product owners, as part of taking credit for a lift in a key business metric, were citing links to the research insights that motivated and shaped their launch. As a result, researchers were able to connect their earlier work to concrete wins that had taken some time—and many actors—to realize. This was not just a general, conceptual influence. These launch numbers were specific, traceable improvements to a north star key performance indicator (KPI). Because research was cited, the research team was able to tally these metrics across launches to demonstrate collective impact.

When it comes to maintaining (and even building) this kind of citation practice, a research repository is an important lever. While definitions vary widely, I tend to think of research repositories as intentional “places” for research storage, access, and use. A repository doesn’t have to be a purpose-built tool—people use standard-issue workflow tools to build repositories all the time—but it can be. Though the technology is important, it’s the collaborative intention and operations built around it that make a repository effective. Increasingly, those operations can extend into building Retrieval-Augmented Generation (RAG) systems and seeing AI agents (along with humans) as regular repository users, with key requirements around providing structured and machine-legible content via Model Context Protocol (MCP).

To return to the Amazon story, before you rush out and try to replicate it, remember that the experience was just one idea for a research repository, built for a specific time and context. Research repositories are systems, and systems can take on many forms depending on the problem the system is designed to solve. For example, some research organizations need a repository that supports key aspects of the research workflow, such as transcribing video interviews and tagging emergent themes. Other research organizations are focused on building evidence for important user insights across a variety of studies. There are many other possible outcomes, and not every solution will meet every requirement, but whatever your requirements or technology, being thoughtful about how you capture context is key.

Establishing Patterns for Citing Research

It takes effort to locate evidence and reference it—to cite it—and in the midst of the busyness of doing their primary job, decision makers may not prioritize that effort. Even when AI has done the work of creating links to supporting research content, decision makers may not think to preserve those connections. When they do cite research sources, you’ll likely see a lot of variation in approach, for no real reason, and some approaches will be better than others.

Patterns for research citations can tackle some of these issues by clarifying what’s valued and expected and providing standards that people and AI can use to embed clear, consistent, (human- and machine-) legible references. In Chapter 11 of Stop Wasting Research, I wrote about how research teams and internal communities can come together to design citation patterns. I wrote, “Early adopters of your knowledge management tools will develop their own varying approaches for incorporating research-based rationale into their backlogs, roadmaps, goals, designs, specifications, and other planning deliverables. You don’t have to wait for those patterns to emerge—you can develop standards for traceable research citations and encourage their adoption” (page 237). Those “standards for traceable research citations” are what I now call citation patterns. In that chapter, called “Link Research Rationale to Plans,” I identified example patterns with different quantities of cited content, from basic references that include only an informative link to comprehensive excerpts of research that bring the deep dive into the citation itself.

More recently, I’ve realized that the type of information that gets linked to in a citation is a factor that’s worthy of some patterns as well. If you do an audit to review how people are citing research in planning documents in your organization—assuming they exist—you may find that decision makers cite very different types of research content, from a single point of evidence to a whole research report. These existing examples can be a great place to start in building more formalized patterns.

The following three patterns represent a point of view on what good, better, and best research citations might look like in a product decision maker’s—or anyone’s—deliverables. Please keep in mind that this pattern set is not exhaustive. Also, these patterns aren’t foolproof. Decision makers may still cite the wrong insights in their plans. However, these patterns can make the application of research-based rationale more discoverable and assessable, enabling anyone in your organization to more easily evaluate how data either supports or contradicts product decisions.

Good (Pattern 1): Linking to Research Evidence

The first pattern is about citing snippets of research evidence as a rationale for a decision, and providing a link to where the evidence is stored. For example, I once reviewed a designer’s PowerPoint slides that linked to a research highlight reel stored in the project’s folders in a shared drive, which was acting as a basic centralized research repository. It was the kind of video evidence that makes a strong point and could be used to justify a design direction. But to understand the video’s context, and the research of which it was a part, I had to navigate around a folder structure, which initially felt like an uninformative list of file names, to find the study name and plan (see Figure 1). It took some poking around to learn anything more about the video, and the context from which it came.

Alternatively, this citation pattern may deep-link to a specialized analysis tool that’s serving a dual role as a research repository. If the viewer has permission to access the application and is invested in exploring, they may be able to see related themes, analyses, and data in the project.

Pattern 1 pros:

• The link destination may provide contextual clues about who created the evidence and the nature of the study (or it may be fairly opaque).

• Following the link may enable some contextual investigation to learn more.

Pattern 1 cons:

• The decision maker needs to know where to look for evidence and browse content until they find relevant support for their rationale.

• Limited context may make it difficult to tell whether cited evidence has been cherry-picked to support a choice or is the result of a well-designed research process.

• Depending on the repository tool used to store evidence, the reviewer may have few options to navigate and find more information.

• Additional evidence for the same learning across studies isn’t included in the link destination, reducing the clarity and persuasiveness of the research.

• Research governance policies may lead to the deletion of evidence, leading to dead links in citations.

This pattern is not ideal, but it’s better than nothing. Using consistent descriptive metadata can increase the legibility of individual pieces of evidence. A colleague once called descriptive metadata the “nutrition facts” of a research asset. This descriptive information can also help AI agents choose which information to pull into their context window when executing a task.

There’s value in coming together as a community of research repository contributors to define a standardized set of metadata. A published standard like the ResearchOps Community’s "Minimum Viable Taxonomy Level 1“2 by Ian Hamilton, Emily DiLeo, India Anderson, Mark McElhaw and Annette Boyer (on behalf of the entire Research Repositories Program Team) can act as a starting point for discussion.

Better (Pattern 2): Linking to Insights in a Research Report

This second pattern is about a headline that cites rationale from an individual research report and links to its destination in a research repository. That linked research report can provide rich context for the specific finding in the citation. Linked reports can also encourage additional learning from other areas of the same study. For example, I’ve been excited to see citations linked to research reports in product leaders’ requirements documents. The citation text alone provided strong contextual cues, and the linked document provided many more (see Figure 2).

Pattern 2 pros:

• A named study link gives contextual clues as to who created the cited evidence and the nature of the study.

• Research language is used in the citation itself, showcasing the researcher author’s interpretative labor and intentional framing.

• Linked reports can provide rich contextual information, depending on the depth of the documentation.

• Linked reports may themselves include links to specific evidence, similar to Pattern 1.

• Researchers typically create reports, so the effort of contributing those outputs to a research library is not extensive (and can potentially be supported with some automation).

Pattern 2 cons:

• A decision maker using this pattern needs to be aware of the research repository, have access to the tool, and have knowledge of desired citation standards.

• Depending on the repository tool being used, you may not be able to use anchor links (links that link straight to a particular location in a document). If a link doesn’t take a reader to the right place on the page, understanding and assessing cited rationale may require extra reading and cognitive overhead, which might be rewarded with additional understanding but risks a loss of attention or patience—or both.

• Additional evidence for the same learning across studies isn’t included in the link destination. If the same insight was found ten other times, an individual report typically won’t incorporate this extended evidence.

Best (Pattern 3): Link to an Insight Summary

This third pattern is about a defined insight headline that links to a particular pre-prepared insight summary containing evidence excerpted from across reports. These insight summaries act as a one-stop shop for research about a single insight. Given that insight summaries take dedicated effort to prepare (a task that may be supported by generative AI), researchers may only prepare them for top, high-priority, underutilized insights that they want to see more action towards.

I’ve seen CEO-level initiative planning documents, prepared by product managers, that contain links to insight summaries to justify particular backlog items. While reviewing the planning documents, the titles of these insight summaries could be taken as standalone statements about user needs (see Figure 3). Following a citation link leads to a rich inventory of related evidence from studies to date. The specificity of these citations enables the tracking of progress toward individual customer problems in complex topic areas.

Insight summaries can be implemented as simple documents in a common productivity tool, or as something more elaborate in specialized repository software. Similar to documenting and updating “nutrition facts” (or descriptive metadata), repository contributors can come together to create shared standards for how they document and update insight summaries.

Pattern 3 pros:

• A clear identifier provides a branded clue that this research comes from a particular community and is stored in a specific, trusted research repository.

• A research-authored insight title provides a core user observation and business implication, showcasing researchers’ collective interpretative labor and precisely framing next steps.

• Insight seekers can dive into and be persuaded by a single “place” for a specific insight, reviewing all supporting (and even contradicting) evidence. Insight summaries preserve context from excerpted studies while also building new meaning across studies.

• Similar to Pattern 2, linked reports within an insight summary provide rich contextual information, depending on the depth of the documentation. Insight summaries may allow for deeper inspection of evidence, containing links similar to Pattern 1.

• Every time an insight summary has an impact, all of the researchers whose work is represented in the summary can claim credit. (This can be a strong motivator for researchers to take part in authoring summaries.)

Pattern 3 cons:

• As with Pattern 2, a decision maker using this pattern needs to be aware of the research repository, have access to the tool (or have received a list of “their” relevant insights from the collected catalog), and have knowledge of desired citation standards.

• Researchers will need to put in extra effort beyond typical study processes to create insight summaries for top insights. Generative AI may be useful in this process, answering questions about standards, identifying potential evidence from across studies to inform an insight summary, and helping to format content within a summary.

Preservation Through Positive Feedback Loops

Similar to the value of surfacing references in a generative AI chat summary, displaying links to relevant research in planning deliverables is the “double click” that expands into a clearer, more trustworthy picture. As some teams explore how entire UX research repositories could become contextual inputs to AI agents, it’s tempting to think that these kinds of citations are old news—like knowing where research is being applied is somehow becoming an antiquated idea. It’s not. Important product decisions will always benefit from clear rationale, and that basic need is something that research operations can look for new ways to enable. If anything, new technology may provide novel ways of identifying and supporting citations.3

As with many aspects of product development, no single role or individual owns successful research citation in product decision-making. With this in mind, insight-driven leaders and repository contributors can spotlight the behaviors they want to see become more prevalent. I’ve found that visibly celebrating winning product launches and experiments that cite research can propagate the idea of citation as a best practice worthy of adoption.

Even if your organization has a richly populated research repository, you’ll never see complete adoption of research citations in key product decisions. But turning up the frequency of citation can infuse researchers’ existing learning into more strategic planning decisions—the type that executive leaders tune into. Increasing citation frequency, in turn, can motivate researchers to be more invested in contributing to their shared research repository, building the depth of connected learning. As positive feedback loops continue to cycle, research impact broadens, and outcomes improve for customers and the business. All from keeping research learning alive, preserving its context, connecting it into new, meaningful choices—from headline to deep dive.

The ResearchOps Review is made possible by User Interviews, now part of UserTesting. With a vast participant network, precise matching, and fraud prevention, User Interviews can reliably fill any research study. Source, screen, track and pay participants, then move seamlessly from data collection to deep analysis, all in one place. → Learn more about User Interviews for ResearchOps.

AI is transforming how research is done, but the ethics and privacy frameworks that should govern these changes are largely absent from the conversation. How do we take care of participants, their data, and ourselves as we build AI into our research systems? Few people are talking about this important topic. We want to change that.

In this free, two-hour working session, facilitated by the founder of The ResearchOps Review and author of Research That Scales, Kate Towsey, you’ll gather with research and ResearchOps professionals to map the biggest privacy and ethics challenges we should be considering.

Register to Join

The information shared will be used to produce an infographic illustrating what we should all, as research professionals, be thinking about. If you take part, you’ll be offered the opportunity to be credited as a contributor. The session will operate under the Chatham House Rule, and anything you share will be anonymised.

Rally Around ReOps 2026 Conference

If you’re in San Francisco, you may also want to join the Rally Around ReOps one-day conference on June 10. The conference is hosted by Rally and curated by The ResearchOps Review.

You’ll hear from six speakers exploring two themes: the machines of research and the humans of research. The talks bring new ideas to the surface and stress-test what’s next for research operations, whether you do the work as a researcher or in a dedicated ResearchOps role. Register here.

​Brought to You By

Scale research operations with ​Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack. Join the future of research operations.

Subscribe now

The ResearchOps Review is supported by User Interviews, now part of UserTesting. User Interviews makes it fast, easy, and affordable to recruit participants so you can scale research without sacrificing quality.

When I first lived in Tokyo, getting lost was a daily routine. It was the mid-2000s, pre-iPhone, and pre-Google Maps, so to find my way, I carried a paper subway map folded into quarters and a pocket notebook in which I’d written train times and transfer stations by hand. Back then, Tokyo’s rail network was operated by two companies and ran twelve subway lines across nearly 300 stations, plus an overground rail and private commuter lines feeding in from the suburbs (see Figure 1).1 Each network had its own fare system, signage conventions, and maps—they were beautiful and overwhelming. I could read Japanese pretty well by then, so navigational problems weren’t a result of language; it was orientation. I could find my station on the map, but I couldn’t locate myself in the system—the map on its own didn’t tell me that part.

If you’re in charge of delivering research operations, this feeling—that you know your surroundings, but it’s still hard to navigate—may be familiar, particularly if you’re also navigating the novelty of AI. One researcher working alone can afford to get lost in the tooling, networks, and workflows of doing research; a wrong turn simply becomes another learning. But when a research pipeline is being designed to repeatedly carry dozens or hundreds of people towards similar goals, not unlike a rail system, the operations equivalent of pouring over every map the night before (or in research parlance, every vendor comparison, prompt library, quality assurance checklist, or workaround doc), isn’t an option.

The Orientation Gap in AI Research

It’s a truism to say it’s important to know where you are and how to get where you want to go. That’s why we have maps for so many things these days, not only to help navigate cities and subway systems but also disease outbreaks, supply chains, and the internet itself. Research operations has maps, too. Kate Towsey’s foundational piece, “A framework for #WhatisResearchOps,” and her book, Research That Scales, lay out the primary elements of the discipline.2 For the research tooling landscape, the newly published User Interviews 2026 UX Research Tools Map catalogues nearly 800 tools. The map makes clear that AI tools in 2026 are their own kind of spiderweb-like network: more than 100 targeting qualitative analysis tools, seventy-nine “AI Research Companions,”3 forty-seven transcription services, and eighteen claiming to outright replace researchers or participants, or both. These maps are useful, but, like my paper subway maps, none of them situates you within the system. Many research professionals are navigating a complex system without knowing whether those tools are getting them where they need to go, and how to measure and monitor failures.

Why We Miss Failures

In his book Thinking, Fast and Slow, psychologist Daniel Kahneman outlined his famous two-system model of cognition: System 1 (fast, intuitive thinking) and System 2 (slow, deliberative thinking and risk weighing). A new preprint from the Wharton School of the University of Pennsylvania, “Thinking—Fast, Slow, and Artificial: How AI is Reshaping Human Reasoning and the Rise of Cognitive Surrender,”4 by Steven Shaw and Gideon Nave, adds a third model: artificial cognition operating outside the brain. The pressure to produce, the pace, and the volume all push toward Kahneman’s System 1: fast, intuitive thinking. Everyone is moving fast, and being pushed to move faster still. System 2, the slower, careful process that says “wait, let me check this,” is being squeezed. System 3 makes the squeeze worse, not better, imparting false confidence even as the underlying accuracy of the incoming information falls.

Shaw and Nave’s research showed 80 percent compliance with AI outputs even when the AI is systematically wrong, which suggests we “cognitively surrender” to fluent, formatted, confident output. It’s literally never been harder, on a brain level, to notice and catch when things break.

Earlier this year, I published an essay called “The Research Risk Cascade: Why Even ‘90% Accurate’ AI Tools Break Pipelines,” an attempt to bring some of Kahneman’s System 2 into the mix, map out what breaks and where in the typical research workflow, particularly when we rely only on System 3, and give you a map for making better decisions about which AI tools to use, how to use them, and where to insert human checkpoints.

In this article, I’ll share why and how AI research pipelines fail through compounding, obscure errors; why we miss those errors without thoughtful interventions; and how my five-step evaluation blueprint can help you locate where in the pipeline research accuracy is breaking.

How Meaning Degrades Across a Pipeline

Remember that game called “broken telephone”? You and your friends sit in a circle; the first person thinks of a phrase, such as “In Japanese, ‘maybe’ usually means ‘no’,” and whispers it into the ear of the person sitting next to them, and around the circle the whisper goes. Invariably, by person five, the phrase has morphed into something completely different, like “Japanese babies know.” The same morphing of meaning happens in research pipelines. Each stage of the research pipeline, from research question to insight (see Figure 2), inherits everything that came before it, and reshapes it: transcription narrows what’s available to synthesize; synthesis can flatten variance, suppress outliers, smooth contradictions; and analysis can take place on outputs that appear fluent and tidy, even when the structure underneath has thinned.

Whether you’re using AI or not, every stage is like a “telephone game” whisper. But AI does introduce additional distance and confidence: distance, because you’re further from the source than if you were transcribing and coding by hand, and confidence, because AI outputs often read as fluent and certain, even if it’s not the case. As a result, the losses in data integrity are harder to notice. That’s System 3 overriding System 2. What looks like one tool on a research platform is often five or six different AI systems chained together—transcription, generative, and agentic aren’t all the same technology—and each has its own way of failing.5

Errors at each stage of the qualitative pipeline don’t add up so much as they multiply, or cascade, and language is at the heart of the error cascade because it’s the material that the whole pipeline is made of. Every step in the pipeline is a translation of some kind, from speech to text to theme to decision, and every translation loses something. For instance, an accent might be miscaptured at transcription; a hedge—an uncertainty or ambiguity—may disappear during synthesis; or a stance may be flipped during analysis. If you multiply those error rates together per stage, you’ll have a cascade of compounding errors. I call this a research risk cascade.

The Research Risk Cascade

As part of research for the essay I mentioned earlier, I modeled the risk cascade for various language configurations. The results are sobering. In a pipeline with Standard American English, the language most AI tools are optimized for, about 70 percent of the original signal likely survives from data capture to insight. That’s the best case, and it means that roughly a third of what was actually said, meant, or conveyed by the participant is gone before anyone reads the output. Add a well-resourced translation pair, such as French to English, and the fidelity of the insights drops to 46 percent. Working from Hindi, a language with over 600 million speakers worldwide, into English one could expect a fidelity loss of 35 percent. These aren’t cherry-picked numbers meant to shock you. In my view, they’re actually quite generous figures, built from public benchmark scores under clean, controlled test conditions—the same benchmarks model builders themselves use to publicly report accuracy.6

The messy, overlapping, ambient-noise audio recordings of actual research interviews is far harder than any benchmark. The prime question is: If your pipeline is losing 30 percent of its signal under ideal conditions, what’s happening under real ones?

Because many research tools are moving towards an agentic state: they’re not just “AI-powered” in the way that a transcription service is, instead, they chain multiple AI agents together to orchestrate entire workflows with minimal human intervention: upload your recordings; get your insights; your agents handle the rest. Because parts or all of the research pipeline are handled by the system itself, often in ways the interface keeps hidden, the error cascade is harder to see, and therefore the losses harder to catch.

The people building and deploying agentic AI systems—engineers, product teams, and the growing wave of vibe coders without traditional engineering backgrounds—are arriving at the same structural insight, and they’re saying so openly. Baptiste Jamin, CEO of Crisp, stood on stage at Paris AI Day 2026 and presented a slide titled “The Real Problem: Compounded Error.” While a single AI step might have 85 to 90 percent accuracy, if you chain it five times, that percentage will drop to roughly 44 percent overall reliability. “Agents don’t fail loudly,” Jamin told the packed room, “They fail statistically.”

The Evals² Experiment: What Happens When You Compare Outputs

To bring the error cascade to life, I took three five-minute clips from episodes of Lenny’s Podcast about AI evaluation practices, or “evals” for short, which featured three English accents (Standard American, Vietnamese-accented, and Indian-accented). I ran each through several transcription tools and several summarization and analysis models, and compared the results.7

Note that this experiment isn’t a recommendation to use off-the-shelf large language models (LLMs) for qualitative analysis, and it doesn’t claim the methodological rigor of a controlled study. It’s a diagnostic exercise: an eval. A way to pull the pipeline apart, examine each stage on its own, and see where a signal degrades, which is particularly important if you’re building research systems that will enable dozens or hundreds of research studies. Anyone can do it, with any source, such as a research interview or podcast episode, and you don’t need a technical background, custom tooling, or an engineering team. A side-by-side test on one source, done carefully, will tell you more about your research pipeline in an afternoon than any vendor accuracy claim.

I’m calling the places where outputs differ “divergences:” where two or more tools or LLM models produced different readings of what was said, what it meant, or what mattered. Some are outright errors, while others are subtle shifts in tone or emphasis. More importantly, all of them are easy to miss if you only look at one output.

Across all three clips, every tool introduced errors at some stage of the pipeline (see Figure 3); none matched the ground truth I produced manually, including the human transcriber. The following is an example of where the compounding showed clearly.

In one five-minute snippet of “AI Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix),” I logged forty-eight divergences at transcription, eighty-five more at synthesis, and sixty at analysis. In total that’s 193 divergences from just five minutes of clear accented English audio. Here’s how they unfold across the pipeline:

At transcription. When Lenny asks Chip whether she’s bearish on data labeling companies (specialized service providers that prepare, clean, and tag raw data), she thinks out loud: “I’m not sure if I’m bearish.” Gemini captured “I’m much less bearish;” Reduct returned “I’m actually a bit bearish;” and Rev’s human transcription preserved Chip’s hedge. A second moment showed a different failure mode. When Chip explains what makes evaluation design interesting, she says you need someone who understands “creative writing.” Gemini transcribed “creative writing,” correctly. Reduct returned “curve writing,” a term that doesn’t exist. Rev’s human transcript rendered it “code writing,” a real term from a different domain. Each transcript reads fine on its own; the divergence only shows up when you compare across transcripts and against the source.

At synthesis. The pattern compounds. Chip’s explicit disclaimer—“this is not the philosophy I follow”—was dropped from eight of twelve AI-generated summaries. By the analysis stage, four of twelve AI-generated analyses reversed her distancing entirely. For example, Claude Opus 4.6’s analyses of multiple transcript paths produced the line, “Chip argues that not every feature needs rigorous evals.” She didn’t argue that at all; she explicitly said it was not her philosophy. The creative-writing example mutated differently. GPT-4o passed “curve writing” through as a legitimate evaluation domain. GPT-5.4 produced “story writing,” an invented fourth term. Gemini, summarizing its own correct transcript, changed “creative writing” to “how well it writes code.” The input was right; the synthesis broke it anyway.

At analysis. By the analysis phase, the creative-writing cascade had split into two research directions. The path that preserved “creative writing” recommended investing in people who deeply understand the domain. The path that inherited “code writing” recommended embedding evaluation checkpoints into the feature release cycle. One says hire for taste; the other says build for process.

Divergences Are Decision Points

Chip’s clip is just one example of the error cascades you can expose when you leverage close (human) attention and pull a pipeline apart stage by stage. If you were to run the same exercise on any real source, whether a recorded interview, a focus group, or a podcast clip, you’ll find similar divergences of your own. These divergences matter because each one marks a decision point—translational, interpretive, and often both—where one reading of reality was selected on your behalf, whether by a machine or a fellow human. Errors at these points are inevitable, but they shouldn’t go unlogged. Catching them requires two things:

1. Transparency into the pipeline;

2. The ability to stay close to the source.

The potential impact of an error cascade widens, too, with the complexity and stakes of the work: a usability test on button placement can absorb slippage that would wreck a cross-cultural diary study or a dyadic interview. The good news is that most cascades can be prevented, and, using the following blueprint, you can start mitigating them right now.

A Five-Step Blueprint for AI Evals

The following blueprint draws from evaluation work across product disciplines, including from some of the speakers whose podcasts I used in the “evals²” experiment. There are five key steps in the blueprint: define, check, compare, maintain, and ask.

1. Define What Accuracy Means in Your Context

First, you and the researchers using the pipeline must define what it needs to preserve, whether for a single study or for a system that will run many studies through the same steps. The pipeline can hold the criteria but it cannot generate them. It’s likely that your screeners, discussion guides, and research questions already outline what matters, so make sure to review them, especially if they’ve been generated by AI. This so-called “grunt work,” which AI tools promise to spirit away, is nonnegotiable. So, instead of just hopping on the next fastest AI or “train” that comes by, study the maps and the “train schedule” of your research system.

Also, be wary of borrowed metrics. Andrew Bean and colleagues reviewed 445 benchmarks for evaluating language models and found that most lack basic construct validity, meaning they don’t reliably measure what they claim to measure. So when a vendor says that their platform is “88 percent accurate,” that number is only as meaningful as the construct behind it, and may not correspond to anything your research needs to preserve. In short, your own definitions matter.

Hamel Husain and Shreya Shankar, whose work on LLM evals has trained thousands of engineers, warn against “vibes-based” assessment and generic metrics like “helpfulness” that obscure specific failure modes. Follow their lead and examine the step-by-step record of what a model actually did, or its “traces,” to catch where the accuracy of outputs drifts. Start with open coding of real outputs, identify failure patterns inductively, and continue until new examples stop producing new error types. Researchers will recognize this as theoretical saturation applied to AI evaluation. Husain and Shankar recommend spending 60 to 80 percent of development time on this definitional work, and they’re clear about why: “error analysis is the most important activity in evals.” Quality definitions drift just like codebooks, which means the work of defining is an ongoing interpretive practice rather than a one-time setup. Whoever owns the system, whether the researcher or the operations lead building it, has to stay close to the definitions and the judgment behind them.

2. Check Outputs Against the Definition

Next, take a sampling of outputs from your pipeline (ideally spanning multiple phases like transcription, synthesis, and analysis) and compare it against the definitions you’re working with. Did the transcript preserve hedging? Did synthesis surface contradictory perspectives? When you do this, use binary pass/fail, not scales, to indicate success; binary forces clearer thinking and eliminates the ambiguity of middle-ground ratings.

Most of the practical discourse around AI evals is being developed in relatively simple contexts like customer support tickets and code review. Those examples are valuable, but qualitative research happens across an enormous range of interpretive complexity and consequential stakes, and you cannot port a validation framework from one context to the other and expect it to catch what matters. And that is why minimum viable rigor has a twin I call “minimum viable context,” which is about how much the system needs to know about your research for any amount of evaluation to be meaningful. A pipeline with context (i.e., what “good” looks like in the study, who the participants are, and what’s at stake) can do more with less checking. A pipeline without that context cannot be rescued by more checking, because the checks have nothing to measure against.

3. Compare Tools to Surface Divergences

The Chip Huyen experiment is an example of this step. I did it manually, but the principle is scalable. Open-source tools like Promptfoo, which was recently acquired by OpenAI, let you define source texts as test cases, specify multiple models to evaluate, set custom evaluation criteria, and run batch comparisons across all of them. Another set of tools, Monica.im and Poe, will allow you to compare LLM outputs. What doesn’t exist yet, as far as I know, is a tool that’s purpose-built for researchers to prevent error cascades. Imagine if you could run the same source material through multiple AI systems, systematically log where outputs diverge, and track how those divergences compound across pipeline stages. Until that tool exists, the workaround is the method: run at least one source from your actual research through your pipeline, at least once, and the shape of your cascade will come into view. Where tools agree, your confidence is well placed. Where tools disagree, you’ve found the place where human judgment is needed to maintain accuracy.

4. Maintain the Evals Infrastructure

This is easy to say, and harder to do in practice because tools update, models change, and the benchmarks vendors use to make accuracy claims often go stale. Deborah Raji and colleagues showed in 2021 that common evaluation benchmarks get treated as general measures of capability they were never designed to capture. And the EvalEval Coalition, hosted by Hugging Face, the University of Edinburgh, and EleutherAI, exists now because AI evaluation as a whole is unstable and needs ongoing scrutiny. If major published benchmarks aren’t stable, your own evals probably aren’t either. Each project you or a researcher runs using your systems teaches you something about your pipeline, what held up, what broke, and what you missed, learnings that should inform your next round of definitions and checks.

5. Ask Questions That Reveal Hidden Risks

If your technical stack is employer-imposed, or you’re working with a bundled platform rather than individual tools, you may not be able to run comparisons across alternatives. But you can still ask questions that close the gap. For instance:

• What’s the model mix: how many and which models are chained, and what does each one do?

• Can you check intermediate outputs, or only the final result?

• Can you export at each stage?

• What benchmarks are the accuracy claims based on, and on what kind of data were they validated?

• Can you trial the tool on your own data before committing?

These are information-gathering questions, not “gotcha” traps. If you’re working with a vendor, you need them to understand where your signal might be degrading in the pipeline. If a vendor can’t or won’t answer these questions, that tells you something about both the vendor and the weaknesses you stand to inherit from them.

Evaluation Is a Process, Not a Destination

The promise of AI tools, again, was that the grunt work of doing research would disappear. The reality is that grunt work doesn’t ever really disappear so much as it shapeshifts. Instead of doing transcription and synthesis manually, or managing the people or systems who did it, you must now do the harder and less visible work of defining quality in your context, building that into your configurations (and operations), checking whether outputs preserve what matters, and updating your criteria as you learn. That’s more skilled labor, not less, and it’s labor that doesn’t register as traditional “research,” whether you’re the researcher doing a single study or the operations lead protecting the work of fifty or 500 researchers.

Remember what Chip Huyen was actually talking about in the experiment around “creative writing.” Evaluation design requires someone who deeply understands the domain and can think about what makes something good in that specific context. That exercise of understanding is also where craft lives, whether in a researcher’s interpretive judgment and proximity to participants or in a ResearchOps professional’s grasp of how a pipeline behaves across many studies. And it’s a key source of competitive differentiation, because no one else has exactly that combination. The job of anyone building a research system around craft is to make sure the key ingredients of judgment and understanding show up reliably even when they aren’t the one doing the work.

I remember the exact backstreet I was on in 2008—a tiny, dead-end lane in the heart of Tokyo, and a prime place for getting lost. A friend who was an early adopter of tech pulled out his iPhone and showed me Google Maps’ blue dot for the first time. It conveyed “you’re somewhere in this area” rather than “you’re exactly here.” Over time, the blue dot improved, becoming more precise, more responsive, and more directional. Someone had to iteratively create the dot: first, it was a rough circle; then a directional arrow; then it offered turn-by-turn guidance that could reroute if you missed your stop. Each version built on the last, and none of them had to be perfect to be useful.

The AI evaluation blueprint works the same way: your first set of definitions will be incomplete; your first checks will miss things; your first comparison will raise more questions than it answers. But see that as the process working, not failing. Understanding your research pipeline, your definition of good enough, and which tools and models can help you get there is not something you can solve all at once. Instead, you must compare, learn, and iterate, so that the next time you know more.

The ResearchOps Review is made possible by User Interviews, now part of UserTesting. With a vast participant network, precise matching, and fraud prevention, User Interviews can reliably fill any research study. Source, screen, track and pay participants, then move seamlessly from data collection to deep analysis, all in one place. → Learn more about User Interviews for ResearchOps.

Subscribe now

The ResearchOps Review is supported by User Interviews, now part of UserTesting. User Interviews makes it fast, easy, and affordable to recruit participants so you can scale research without sacrificing quality.

Whenever I hear someone use the term “AI,” I stop and ask them which type of AI they mean. “AI” could mean a surprisingly large range of things. It could mean large language model (LLM) prompting, agentic multi-step reasoning, image generation, classical machine learning, recommendation systems, predictive analytics, computer vision, or natural language processing in its older, pre-LLM forms. But they’re not one and the same thing. Unless you know precisely how the system is configured, or how to configure the system, AI can have vastly different capabilities, confidence calibrations, error profiles, readiness levels, and governance requirements.1

In distributed organizations, where knowledge and terminology are naturally fragmented across silos, establishing a shared glossary of AI classifications is one way to ensure aligned discussions and informed decision-making about AI. If my organization already has a glossary of terms and acronyms, I usually request that these classifications be added to it. If no glossary exists yet, I’m usually the first to make one. But making the right decisions about AI often requires more than a glossary to disambiguate terms.

To build successful AI-enabled research systems, you’ll need to understand the difference between probabilistic and deterministic AI systems, why shoehorning the former into the latter won’t work, and why aiming to “de-weird” AI isn’t the right strategy. Ironically, to better understand these themes, the best place to start is in the 1950s.

A Story of Two AI Models: Deterministic and Probabilistic

The term “artificial intelligence” was first coined by the American computer and cognitive scientist John McCarthy in the late 1950s, primarily to brand a funding proposal for the Dartmouth Conference, widely considered the founding event of artificial intelligence as a field. McCarthy’s work produced what’s called symbolic AI: an approach that uses symbols, rules, and logic to enable reasoning and is entirely dependent on explicit human programming: a deterministic model, unlike today’s deep-thinking AI, which relies on a probabilistic model.

Though McCarthy named the field, the AI model he developed largely sits outside the deep-thinking AI that’s currently reshaping the world as we know it. So, who does the current AI evolution belong to? For me, this history sits close to home, and it’s a key story to understanding the two models that drive AI and how to leverage AI to build research systems.

Circa 1959, my uncle Thomas Osborn (see Figure 1), a student at Cornell University, was hired by an American psychologist called Frank Rosenblatt, the father of machine learning. His job was to assemble, test, and run experiments on a large device called the Mark I Perceptron; recognized as the first artificial neural network device. The Perceptron was equipped with a grid of four hundred electronic “eyes” and processed crude snapshots of the letters of the alphabet. Instead of relying on a human to program the explicit definition of the alphabet, it physically adjusted its own internal paths until it taught itself how to recognize the letters. Where McCarthy’s “Dartmouth” system relied on deterministic rules written by humans, Rosenblatt’s Perceptron learned patterns from random examples in the machine, making it difficult to pinpoint how it worked based on rigid rules. (If you’re currently experimenting with AI, that may sound frustratingly familiar.)

This reliance on probabilistic randomness—given the same input, the system doesn’t always produce the same output—rather than deterministic rules—given the same input, the system always produces the same output—laid the groundwork for today’s deep-thinking AI. It’s precisely why modern agentic tools are good at detecting patterns in data, including research data. Today, if you work in research and know what you’re doing, it’s possible to apply similar probabilistic approaches to qualitative transcript analysis using custom agentic workflows to detect semantic patterns across hundreds of unstructured, in-depth interview transcripts.

Understanding the distinction between deterministic rules and probabilistic patterns is what matters most for understanding today’s AI landscape, and, as researchers and architects of research systems, it’s important information for how you should, and should not, use AI to automate and augment the research process—because trying to generate certainty from probabilistic deep-thinking AI, or trying to “de-weird” it, is literally a case of fighting the machine.

A Profound Strategic Mistake: Treating AI Like Ordinary Software

In a recent The Economist article (paywalled), Wharton professor, AI expert, and author Ethan Mollick wrote: “...the dominant instinct across the corporate world is to treat artificial intelligence as if it were just another piece of enterprise software....This is a profound strategic mistake. Companies are racing to de-weird AI, and in doing so they are squandering what makes it transformative, turning it into just the latest wave of office automation.” This specific failure is heavily reflected in current market data. Despite Gartner forecasting a massive 40 percent surge in agentic applications by 2026, they also predict that 40 percent of those projects will be cancelled by the end of 2027. Concurrently, in “Rise of agentic AI: How trust is the key to human-AI collaboration,” the Capgemini Research Institute reports that a mere 2 percent of enterprises have successfully scaled agentic workflows.

Particularly as a research systems architect, as you seek to extract efficiencies from AI, it helps to avoid de-weirding it. AI might make parts of the research workflow easier and faster—even instant—but the greater transformation lies in using AI to assess, provoke, and augment human ways of thinking: to turn it into a sparring partner rather than an oracle.

In a very human way, American academic and podcaster Brené Brown and popular science author Adam Grant exemplify this kind of friction. They’ve spent years acting as public sparring partners. By deliberately exposing their different methodologies and pitting themselves against one another, they prove the value of intentional conflict—or antithesis in synthesis. As Brown wrote in her recent book, Strong Ground: “The difficult and disciplined commitment of rethinking and questioning what we know is where Adam’s love of quantitative cognitive science and organizational psychology crashes into my love of the deeply human, and often qualitatively understood, issues of emotions, courage, and vulnerability.”2 For researchers and research systems designers, this is both the conundrum and the opportunity: How should AI be used to enable efficiency without missing the primary purpose of qualitative research to discover “issues of emotions, courage, and vulnerability”?

This is exactly why qualitative analysis is the perfect domain for using probabilistic AI.3 Where quantitative research aims to measure predictable outcomes by flattening human thinking and emotion into simple, easy-to-digest dashboards, qualitative research forces active listening and requires a level of empathy that rigid logic can’t replace. Research professionals need a tool—AI and otherwise—that can actively engage with this messy, weird, ambiguous (probabilistic) qualitative data; a good example of which is research transcript data.

Calibration as a Standard of Practice

Understanding how to actually spar with a machine requires understanding how it’s built. Because my sons are developers working in agentics, I was exposed to this technology early and quickly got up to speed with the technical stack sitting under the hood. I wanted to see if the machine could act as a calibrator rather than an automator. In one experiment, I configured individual agents to analyze a set of research interview transcripts, then presented the output to a senior researcher who had already manually analyzed the same data. After comparing the machine’s work alongside their own, they noticed several artifacts they had missed, as well as several the agent had missed. For example, the researcher had logged irrelevant “psychological scene setting” instead of the actual decision triggers, while the agent had failed to flag when an interviewer asked a leading question, thereby disqualifying the participant’s response. The researcher commented, “It was definitely useful because I can now see how humans and agents can get it wrong. Both need some tweaking.”

Automation bias, or the tendency to over-trust machine outputs because they appear formally structured, has become one of the critical risks in using generative AI today. LLMs are masters at mirroring. A well-formatted LLM summary typically mimics the vocabulary of analysis and looks grammatically clean, but as most researchers will agree, mimicking isn’t the definition of well-executed analysis. That’s not to say that deep thinking AI isn’t a useful partner in the analysis and synthesis process. If a researcher flips the automation bias dynamic on its head, refuses to trust the AI prompt, and instead uses it to test or calibrate their own research standards, the model will stop acting like an oracle and start acting like a research partner. Rather than an error-prone automator, AI becomes a vital structural critic, positioned to point out researchers’ natural human biases. In this case, per Mollick’s article, you’ll leverage rather than squander what makes AI so transformative.

In a recent LinkedIn post, researcher and AI trailblazer, Caitlin Sullivan, summarised this exact shift: “You don’t just build workflows. You systematically test them. You run the same analysis multiple times on the same data to check if findings are stable. You test across models. You know which skills produce consistent results and which ones drift when the data changes, and how to fix them.” In short, researchers move from conducting the analysis to both conducting the analysis and actively governing the analysis and synthesis workflow—or perhaps they’re lucky enough to have a ResearchOps professional in situ to design and maintain parts of that governance for them. Leveraging agentics for mutual calibration changes the role of research analysis entirely. Rather than dedicated analysts and synthesisers, researchers become rigorous testers of the machine’s contextual analysis, elevating their own critical thinking to manage vastly more data touchpoints.

As an organization scales its agentic practices, the core responsibility of ResearchOps must fundamentally shift from procuring isolated tools to architecting governed, systemic workflows. For a next-generation ResearchOps lead, this introduces two vital strategies. First, building agentic networks that orchestrate the broader research lifecycle, and second, training practitioners to use these models for deep analysis and to rigorously audit their outputs and report deviations. These frameworks must be designed to treat the AI for what it actually is: a “mirror” that continually recalibrates and elevates the organization toward greater human-centered rigor. Through this lens, the technology stops functioning as a basic summarization tool and instead operates as a systemic participant in the research pipeline.

Turning Qualitative Standards into Agentic Workflows

One of the weirdnesses of AI is that it’s not a tool that you evaluate against a set of requirements and then procure, configure, and onboard people into the way you would have a SaaS tool a year or two ago. Once it had been configured, the SaaS system was designed to behave consistently, but probabilistic AI isn’t. It changes how the tool and performance quality are assessed, how researchers and ResearchOps professionals work together, and the level of craft a ResearchOps professional must have to do their job well.

As a ResearchOps practitioner, I’ve recently deepened my understanding of qualitative analysis by completing two of Indi Young’s Data Science that Listens (DStL) courses, which rigorously explore the mechanics of concept summaries and emergent patterns. Mastering these specific qualitative research tools is critical for building AI-augmented research operations because they provide the exact cognitive frameworks required to translate chaotic human thought into structured, actionable data—data that’s aligned with the requirements of today’s probabilistic AI.

I specifically use Young’s standard as the baseline for evaluating the quality of the analysis and synthesis results produced by custom-built AI agents. This synthesis acts as the foundation for everything else I do, such as configuring new analytical skills for agentic pipelines, upskilling researchers to actively manage their own agentic models, managing and curating “agent libraries,” and troubleshooting agent handoffs and outputs to ensure they’re following compliance policies and that their tasks aren’t overlapping—all foundational line items that should be on every ResearchOps professional’s to-do list.

The goal isn’t only to enable individual researchers to use AI to deliver rigorous research—or the self-oriented activity of “I-Me-Mine AI,”4 as ResearchOps thought leader Kate Towsey recently called it. Scaling Young’s level of qualitative rigour across dozens or hundreds of practitioners requires shifting focus from delivering isolated tools or training to systemic governance of AI usage to support or augment countless research studies.

On a practical note, agentic ResearchOps revolves around using natural language to write specific analytical skills (step-by-step cognitive descriptions and rules that tell the model exactly how to parse subjective data), which are stored in centralized markdown (.md) files. These files act as strict, governed boundaries that configure the enterprise’s multi-agent workflows. Knowing how to tune these agents or skills might not be overtly technical, but it does require a deep methodological grounding, such as encoding Young’s DStL criteria for “emergent patterns.”

By standardizing these rules, ResearchOps leads can deploy specialized multi-agent networks to support the entire organization (see Figure 2). Instead of deploying a single, error-prone agent for researchers, you could architect specialized clusters or teams of agents, each with its own skillset, each working together to achieve a goal. For instance, you might configure one agent to scan transcripts strictly for empathic markers, a second for cognitive logic, and a third to observe and resolve friction among the agents before presenting the summary to the researcher.

For people who do ResearchOps—the emergence of AI means that, for many researchers, delivering research operations is now a primary part of their role, too—this is the ultimate operational frontier. ResearchOps professionals are literally codifying the organization’s methodological standards into the agentic architecture itself, guaranteeing systemic rigour across every project.

Why Repositories Are a Practical Starting Point

Beyond deep data analysis, research repositories represent the most immediately tractable “integration model” for AI from an operational standpoint. The foundational problem with most traditional repository platforms isn’t technical deployment; it’s that they completely lack the infrastructure required to actively manage a controlled vocabulary. Organizations inevitably suffer from a severe taxonomy problem when traditional metadata remains static while corporate vocabulary continuously evolves. As a result, because historic research is rarely findable or connectable, teams treat repositories like passive filing cabinets. Agentic workflows can solve this exact taxonomy problem by tracking researchers’ semantic searches. Say a product manager searches the repository for “chat interface,” but all the relevant historical research is hidden because it was tagged three years ago under the older term “conversational UI.” An agent can monitor internal keyword searches, identify conceptual gaps, and automatically update metadata across past studies to match the vocabulary the organization is naturally using today. By automating metadata generation based on real retrieval behaviour, the repository becomes a living system that continually refines its search taxonomy. This can be semi-automated, with a librarian reviewing it before the new tags are deployed.

As ontologist Jessica Talisman wrote in “The Ontology Pipeline™, Refresh,” a controlled vocabulary is the absolute foundation of architecture. The taxonomy in your file structures is literally the first layer of governance your agent encounters. If researchers use inconsistent tags, the agent simply inherits and compounds that ambiguity. By forcing a consistent, machine-readable map—deploying strict frontmatter fields for domains, themes, and specific thinking styles—the agent can securely route and filter at the metadata level, protecting the LLM usage budget and behaving strictly according to our defined rules.

This type of hybrid workflow illustrates Ethan Mollick’s earlier point: if we treat AI merely as an automation efficiency tool, we miss its true value. Here, the machine isn’t replacing the librarian; it is acting as an active sparring partner, introducing helpful friction by challenging the organization’s outdated vocabulary and forcing the human to actively mature the taxonomy. This also has important implications for operational budgets.

When I cloned copies of my son’s agentic setup and ran them locally on Opencode using a Gemini integration, I racked up a large bill for LLM tokens in just one month. Translated to scaled-up enterprise costs, I quickly realised that making agentic-driven research viable for large organizations requires more than just enabling effective LLM usage; it must also be cost-efficient. Based on my experience spanning both the tech and research sectors, scaling agentic AI is rarely a technical deployment challenge; it’s fundamentally a governance and contextual integration challenge.

Without a robust taxonomy intentionally built into file names and frontmatter (metadata, or the elements that introduce the reader to the body of a document), an agent crawling the repository has no explicit map; it is forced to open every single file and read its contents just to determine relevance. Every token the model is forced to read burns operational budget and rapidly consumes precious context window limits. Cost control isn’t just an IT concern; if ResearchOps professionals aren’t actively protecting operational budgets through disciplined taxonomy work, enterprise leadership will simply pull the plug. A mature system will know how to cache and reuse LLM outputs without burning expensive tokens on repeated prompts. And agents running on self-hosted open-source LLMs won’t solve this problem, because you simply inherit the same token ceiling on hardware you now have to maintain yourself.

From Adoption to Integration

Reading the history of agentic AI, particularly the long period between Rosenblatt’s Perceptron and the deep learning revival, dominated by OpenAI and Anthropic, is useful for research systems builders because it shows how long genuine foundational shifts take to stabilize. The lesson isn’t simply that things take time; it’s that without that historical framing of the journey, the cycle repeats: a capability arrives, expectations inflate, integration is rushed, the tools underdeliver against expectations, and the field either dismisses them or defaults to the next trend. The massive project cancellations predicted by Gartner and Capgemini are a direct symptom of this cycle. My opinion is that most of the AI capabilities being marketed to research organizations are in the adoption rather than the integration phase. That doesn’t mean that these capabilities should be ignored; it means they should be trialed with appropriate methodological controls rather than deployed as finished infrastructure.

Making agentics-driven research systems work for the large enterprise isn’t predominantly a technical challenge; it is a governance challenge. First, you must govern quality: the strict analytical and taxonomical boundaries and review loops that keep machine-assisted synthesis methodologically sound. Second, you must govern the financial implications: the information architecture that prevents token burn and keeps agentic workflows affordable at enterprise scale. Finally, you must govern the risk: the systemic accountability ensures that AI remains an introspective sparring partner, rather than an oracle permitted to shape decisions unchecked.

If we stop treating generative AI as an automated oracle, we can begin architecting the custom, well-governed workflows our research pipelines actually need. By using the technology as a mirror rather than a replacement, ResearchOps leads can continually recalibrate and elevate our systems toward a richer outcome of human-centred rigour. Research relies on our ability to listen, adapt, and evolve together. Let’s pave the way for a research practice as diverse as the world it seeks to understand.

The ResearchOps Review is made possible by User Interviews, now part of UserTesting. With a vast participant network, precise matching, and fraud prevention, User Interviews can reliably fill any research study. Source, screen, track and pay participants, then move seamlessly from data collection to deep analysis, all in one place. → Learn more about User Interviews for ResearchOps.

Subscribe now

The ResearchOps Review is brought to you by Rally—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack.

In 2011, venture capitalist Marc Andreessen published an essay in The Wall Street Journal (and later on his blog)1 that opened with a prophetic line: “Software is eating the world.” Andreessen was right: software did “eat the world.” Amazon became the largest global bookseller (and marketplace); Spotify took a bite out of record labels; and Netflix devoured video stores and cinemas. As Andreessen wrote at the time, software programming tools and internet services made it “easy to launch new global software-powered start-ups in many industries—without the need to invest in new infrastructure and train new employees.” Replace “software” or “internet” with “AI,” and his statement is almost just as true now. Over the past fifteen years, software companies have transformed entire industries, the global economy, and your life and mine—as too will AI.

A few weeks ago, I sat down with Oren Friedman, the cofounder and CEO of Rally, to talk about how these economic and technological shifts are impacting research and ResearchOps, and how Rally is responding. We spoke about AI, of course, and the vision of a platform- or UI-agnostic future. But we also discussed how, just as technological advancements are making physical resources like rare earth minerals, oil, and gas more valuable2—worth fighting for, even—the ability to build human systems, and the soft skills required to build them, are becoming more valuable, too.

Soft Skills as Hidden Assets

“I don’t even know what else to talk about because it seems that the only thing that’s important right now is figuring this stuff out: stakeholder management, change management, relationship building, and the ability to sell the value of research,” said Oren during our conversation. “Every discipline is dealing with it, but I think it’s just extra potent for research professionals because of all the years spent trying to build people’s understanding of the value of research. And I do think it still comes down to soft skills. We’ve been talking about stakeholder management for years, but it’s never been more important than now to know how to talk to design, product, and other leaders and how to get in front of them.”

If you’ve worked in research for any length of time, you’ll know the ongoing challenge of encouraging colleagues from other disciplines to appreciate the value of research—or more pointedly, the value of good research…and, more recently, the value of researchers doing research at all. Most organisations are sold on the idea of including research in the product development lifecycle, and they’re investing in the operations required to make that happen, albeit through democratisation efforts.3 If you’re a research leader, the key challenge now is to upsell the importance of research quality to the same audience, which should, in turn, resolve the assumption that all research can and should be done by non-specialists and that researchers and their operational counterparts are pedantic bottlenecks.

During our conversation, Oren stressed that this stakeholder dynamic isn’t unimportant to research technology companies, like Rally: “Top of mind for us is how we can help equip research professionals to better manage these relationships so they can appeal to product, design, and other teams,” he said. “Research professionals, including ResearchOps, are coming from a position of knowledge, strength, and credibility, so how do we help reframe this subservient, hierarchical structure that’s happened within so many companies?”

Part of the solution is empathy and meeting stakeholders where they are; a core product principle at Rally. Oren spends a lot of time talking to product managers (PMs) and shared this observation: “A lot of the work people are trying to do, that PMs are trying to do, they may not even call it ‘research.’ They may not think of it in the context of a study the way a researcher would. And so, even with the basic primitives that researchers use, it can be easy to gloss over PMs who just aren’t thinking about research that way. They’re using different terminology, methods, or frameworks to reach their goal of making good decisions. They might call research ‘continuous discovery’ or ‘talking to a customer,’ for instance.”

The concept of meeting people where they are came up several times in my conversation with Oren, not only because it’s a Rally product principle, but because shifts in the product landscape mean that research professionals are being obliged (forced seemed too aggressive a word, but it’s not inaccurate), to build how research operates around stakeholders’ language and workflows—a user-centric approach, ironically.

Democratisation Is Now a Directive

Four years ago, research democratisation was a strategic choice for research leaders. In most companies, now focused on efficiency and profit, it’s a directive. Every week or two, I hear a story about a company that has laid off all or most of its research team while retaining its ResearchOps capability, tasked with democratising research across the entire company. Oren has seen this dynamic first-hand: “Just a few weeks ago, a company we work with laid off the entire research team but kept the ResearchOps team. It’s as if research and ResearchOps now have to redefine and re-articulate their value to the product team, even though a light democratization program has already been in place. Now it’s like their whole job rides on the democratization program working—and working well.”

Rather than resist the change, many savvy research leaders are leveraging the value organisations now place on well-designed research operations (used to build scaled-up research democratisation programmes) to gain the buy-in required to build better relationships with stakeholders and, eventually, to rebuild their research team. As part of this tactic, these leaders are becoming operations specialists in their own right, in line with an interesting trend in the research professional landscape. One in which researchers are doing research operations; ResearchOps specialists are being asked to lead and sometimes do research; and both are being tasked with enabling everyone else to do research. No doubt, the world is topsy-turvy. But when these savvy leaders do start hiring, they hire specialised researchers to deliver high-quality, high-priority insights to the most important parts of the organisation, hinting at an optimistic future for their operations-forward research teams.

Optimism aside, for many, the transition isn’t easy. Oren shared, “I’m noticing it’s a really hard bridge to cross for research and ResearchOps teams who, for the first time, are supporting people who are less in tune with a healthy research process.”

A Healthy Research Process

That healthy research process is key, particularly when there are fewer or no researchers involved to model good research practice. ResearchOps can do some of this work, but researchers are needed for research strategy and craft advocacy. This is truer still because AI is making knowledge or insight generation seem effortless, speedy, and requiring little skill. “The tempting thing for product teams is to try to find the fastest route to whatever insight they’re trying to capture.” Oren’s deep in the weeds of this democratization shift. “But that’s a fundamentally flawed way of approaching it because if there’s no rigour or guardrails, the quality of the insights deteriorates, then the quality of decision making deteriorates, and what does that point back to? It always points back to research, which degrades the value of research and the understanding of it within the company.”

The research profession has always been aware of the importance of pacing—slowing people down in an increasingly fast-paced world to observe, learn, question, and absorb knowledge is the ongoing operational challenge—but demonstrating the value of pausing to a sceptical product leader in the middle of a cost-cutting cycle, with access to AI, is another matter altogether.

The knee-jerk reaction is often to insert more operational guardrails and standards, but when people feel forced to work in ways that seem onerous, they tend to rebel. As Oren put it, “Research or ResearchOps might be worried that PMs are just going to log into Gong, for example, search through all the insights, reach out to customers, not obtain consent, not log anything, not send incentives, not put the data in any central repository, and so on. But for each PM, designer, or engineer, that’s the easiest path. They aren’t incentivised to think in systems; they’re incentivised to think about the fastest way to get their own job done.”

Finding the Right Balance

This push-and-pull of seemingly opposing forces—control versus convenience—has been a key challenge for research professionals for years, but solving it is more critical than ever. In a world where anyone can chat their way to an insights summary, search a call library, or vibe-code a workaround, a heavy-handed (if correct) research process is easier than ever to ignore.

“It’s this yin-yang concept of convenience: PMs and other folks want the ultimate convenience of being able to query something and go (that’s the yin).” Oren invokes the traditional Chinese symbol representing opposite yet interdependent forces. “And the other side (the yang) is control: to support good practice, ResearchOps wants to set controls in a way that enables both ultimate convenience and safe, responsible research.”

Oren says it as it is: “For too long, ResearchOps has indexed on control at the cost of convenience, and that’s what I think could lose them their jobs. Because if it’s overly controlling, then it’s like, ‘Move out of the way! I’m just going to go figure it out on my own.’ So, we’re trying to design a system that helps ResearchOps teams create these convenient experiences while retaining control. Yes, still simplifying the UI because UIs aren’t going anywhere quite yet for the non-researcher, but also starting to work towards a more agentic future where you don’t need to open a specific application to do something.”

That agentic future won’t only change how people interact with research platforms, but also how they engage with research operations: “I think gone are the days in the not-too-distant future of training people on how to use a new SaaS application that your company is adopting. Because no one has the patience or time to learn new workflows or tools, and they no longer need to. They want agents to do it all for them; to be the orchestrators of their work. So that’s the kind of vision we’re trying to work towards at Rally.”

“I-Me-Mine AI” Isn’t Enough

PMs, designers, and engineers aren’t the only people who aren’t thinking in systems. In a recent article for The ResearchOps Review,4 I wrote about the notion of “I-Me-Mine AI” and the urgent importance of research and ResearchOps professionals taking a step back from their individual use of AI to consider and design its place within their wider research system, before anyone else does.

Oren also sees the chasm between those two modes of working—individual AI use and systemic AI design—as a once-in-a-career opportunity and ticking time bomb: “The beauty is that AI now allows for quality, speed, and cost, all things that were trade-offs before. Of course, there are still trade-offs, but it’s never been easier than it is today to sacrifice less of those things, as long as things are done right. And product leaders do care about doing things right—just not if it bottlenecks progress.”

The need for mutual empathy across disciplines (again, those soft skills) is also key for Oren: “I recently spoke to a product leader who explained that AI is moving so fast that he doesn’t want to fall behind, and he doesn’t want his organisation to fall behind. So he’s doing a lot of research and investigation on the right infrastructure needed for the product team to stay ahead of the curve because they’re also worried about their jobs, and about engineers moving into their realm, and design—you know, the blending between research, design, product, and engineering…it’s very intense right now. It’s also a really exciting opportunity, but how do we get research and ResearchOps folk to the point where they can have that systems-level conversation with them?”

Research and ResearchOps professionals are well-positioned to have these conversations, but many are understandably overwhelmed by the state of the profession, the world in general (a fair reaction), the speed at which AI is developing, and the implications for their careers.

“We’re trying to encourage ResearchOps teams to lean in and become more literate with APIs5 and MCPs,”6 Oren noted. “ResearchOps professionals can help set up these systems, such as setting up the right consent forms and cool-down periods, hooking up the right integrations, and inserting the right legally approved communications throughout the entire experience. ResearchOps needs to play the role of the chief architect of scaled-up, AI-augmented research systems.”

Oren’s right, and the moment is now.

When the Interface Disappears

Just as the boundaries between roles are becoming less distinct and the shape of organisations is evolving at a dizzying pace (read Carolyn Morgan’s article, “A Practical Guide to Structuring ResearchOps Through Organizational Change”),7 the format of research platforms is changing at breakneck speed. Everywhere, CEOs and founders like Oren are working hard to keep up with the pace of change—not just to secure their own survival, but to ensure their customers can deliver the experiences their stakeholders feel they need to keep pace. “Stakeholders are already demanding a NotebookLM-like storage as an output of all research materials,” a ResearchOps professional shared in a recent exchange. For many people, their personal professional success depends on their ability to work faster, and they’re looking to colleagues in operational roles to provide the tools they need—now. As a result, ResearchOps professionals (including researchers delivering operations) are seeing their roadmaps, standard operating procedures, and purpose shift overnight from support, training, and administration to technical systems design. But they don’t have to make this shift alone; research platforms like Rally are also responding to the change.

“It’s a new experience we have to design for,” is how Oren sees it. “Basically, using MCPs, we’re trying to figure out how, through Claude or through Slack, you might run these actions without necessarily knowing what’s under the hood. Because I think that’s another one of these leaps that research and ResearchOps teams have to make. We’re thinking about AI a lot in the context of meeting teams where they are, and how they’re trying to design the future of their work, which is orchestrating different agents that interact with your organisational tools and giving ResearchOps the Ferrari on the back end—the engine on the back end—and they are the engine mechanics, making sure that things run smoothly when you get in the car and you don’t all of a sudden have a flat tire or no gas.”

The New Competitive Advantage Is Interoperability

Until recently, most research platforms were developed in isolation—it was rare for them to collaborate—but that’s starting to change. I asked Oren to discuss Rally’s perspective on building open research systems, that is, systems that can integrate with other systems, and their approach to partnerships.

“Well, I’m happy to talk about it because partnerships are a very important part of our strategy as a company. We’re building a best-in-class solution, not an all-in-one solution. Rally launched as a user research customer relationship management tool (CRM), but we’re evolving into a best-in-class research infrastructure. No matter what research you’re trying to do as an organization, you need the infrastructure underlying it to make sure that it’s efficient, safe, governed, and fast—all the value props that the different people in the company care about, whether it’s from product to legal.”

The suite of tools now available for research, both specialist and non-specialist, has exploded over the past ten years. I asked Oren how they choose the tools they integrate with. “As infrastructure, we’ve tried to be agnostic about the tools we connect to because one of our product principles is to meet our customers where they are. So, whether you’re using Qualtrics for surveys, Listen Labs for AI moderation, Zoom, Microsoft Teams, Copilot, Claude or whatever, to make research work, your tools need to talk to each other. If they don’t integrate nicely and play nicely, then you’ll spend your time reconciling two spreadsheets, which no one’s got the time for, especially these days. So, integrations have always been a really important part of our strategy. We’ve been trying to create partnerships for a while, and we’ve found that some companies are really leaning in like Marvin; they’re aligned with us philosophically as we’ve been able to collaborate on certain customer pain points that we wouldn’t be able to do if we didn’t collaborate.”

Being Human Matters

If there’s a through-line to all of this (AI, democratisation, MCPs, partnerships, and the shifting shape of product and UX teams), it’s that research and ResearchOps professionals are being drawn into a more architectural role. Research operations are now less about enforcement and more about building: creating the conditions for speed without sacrificing quality; designing “friendly guardrails” that demand perfection only where essential; and meeting teams where they are, whether that’s in Slack, Claude, Rally, or an interface that’s yet to be launched.

On the future of human-led research: “I don’t think we’re going to be moving away from solving human problems,” Oren said, “because I think there’s always going to be problems in the world that people will pay money to solve. And that’s where every product fills a certain space. So, if you believe that we’re not going to stop solving problems for humans, especially when things are moving fast and changing so much, there’s an even greater need for understanding people. If you’re going to win in this highly competitive world, you need to find those gaps in understanding and build a solution for them. And that’s what research is, whether you want to call it research or seeking understanding or learning or whatever, I think that’s not going to go away. I don’t see a world in which AI moderators are talking to AI participants, and everything is fully abstracted. Because then, what (and who) are we designing for? If there are no humans in the conversation at all, then it’s like we don’t exist. I don’t think we’re going to move into that type of world. I might be wrong. I do think there will always be a need for a better understanding of humans and solving new problems. Because I don’t think we’re going to ever solve every problem. And if we do, then it doesn’t matter anyway, because we’ll have unlocked unlimited money and resources, and no one will need to work ever again!”

In other words, even as the tooling and teams shift, and even as “research” is renamed, remixed, and increasingly mediated by machines, the crux of research remains stubbornly (and gloriously) human: to reduce the risk of building the wrong thing, for the wrong people, for the wrong reasons—and to build beautiful things, ideally.

A final word from Oren: “I would like to see the industry and the profession championing that you don’t have to sacrifice quality for speed, that it’s possible to meet the timelines and needs of the organisation, to be user-centric, and to do so with a high bar, just like all these other disciplines are being challenged to do. No one is putting rubbish designs out there because it’s faster. The bar is still really high for good design work. The same goes for marketing and other disciplines. So, if other disciplines can do it, research can do it, too. And don’t give up on that because you’re the only ones championing it. No one else is going to fly that flag. And it’s important. It matters. It may not seem like it matters today because people are getting laid off left and right, but it almost has less to do with you as individuals—it’s a market correction, I think, in the context of an overinflated tech sector. As a research or ResearchOps leader, you’re getting the short end of the stick, but don’t lose faith in your skills and your inherent value.”

Sponsor and Credits

Rally sponsored the first year of The ResearchOps Review, when it was just an idea, making the dream of a dedicated professional publication focused on ResearchOps a reality. Thank you, Oren and the Rally team.

Rally UXR—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack. Join the future of Research Operations. Your peers are already there.

Edited by and .

Subscribe now

The ResearchOps Review is brought to you by Rally—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack.

Everything about reorganizations and layoffs is messy. They aren’t just “challenging” or “complex,” they’re the kind of messy where you’re in a meeting or workshop, trying to figure out reporting structures and teams while people are worried about losing their relevance or their jobs, or what their future role will be. It’s the type of messy where you have a handful of options on the table, and no obvious choice. It’s the type of messy where every decision will leave someone disappointed.

In the last four years, I’ve navigated two major restructurings and three rounds of layoffs at the same company. I’ve worked as a ResearchOps specialist in a distributed team in a decentralized model; led a ResearchOps team in a centralized research organization leaning heavily on hybrid practices; and I’m currently leading research operations as part of a decentralized organization with centralized operations. If you’re having trouble keeping track of all those organizational structures, you’re not alone.

In my experience, there’s no perfect organizational structure. There are as many permutations of how research and ResearchOps teams can be organized as there are research and ResearchOps teams. Reorganizations happen at multiple levels—business-wide, within organizations, or within specific teams—and they all affect each other in ways that aren’t immediately obvious.

In this article, I’ll share the patterns I’ve seen, the traps to avoid, and the questions to ask when you face your next reorg. Because one thing is true: change is the only constant.

Shifting Tides in Organizational Models

There’s something almost contagious about organizational design trends. One year, everyone’s decentralizing; the next year, centralization is the gold standard; then hybrid models promise to solve everyone’s woes. Organizations lurch from one structure to another, and researchers, designers, and ResearchOps professionals must hold on tight and go along for the ride. It feels chaotic because it is.

But here’s the thing: when companies reorganize, someone has to figure out how each separate team, and all the teams together, will function. That’s organizational design, and when structures change, your research, design, or ResearchOps model must change, too.

When I’ve needed to reconsider the structure of my team in the past, I’ve asked myself several questions:

• Should I centralize or decentralize operations, or opt for a hybrid of the two?

• What role does my team play in these scenarios?

• What makes the most sense for the business?

• How do I best align my teams’ work to the business’s needs?

These decisions are never made in a vacuum; they’re usually accompanied by endless possibilities of team structures, changes in business direction, changes in personnel count, and many other variables. The first time I faced a reorg, I searched everywhere for guidance: I read design blogs and had countless anxiety-filled conversations with fellow managers across the industry about the what ifs and why nots of every permutation.

What became clear in my search for guidance is this: there are three predominant organizational structures (centralized, decentralized, and hybrid), plus four common operating models (solitary, specialized, distributed, and elevated) that define how a ResearchOps team might operate.

Decentralized: Embedded, but High-Context

A decentralized model is fragmented yet effective; one in which researchers are embedded in product and design teams (see Figure 1). In this setup, researchers can build deep product expertise and strong stakeholder relationships, enabling them to become true subject matter experts for that specific product and to understand its nuances, strategy, and users. And because they work side by side with designers and product managers (PMs), they’re regularly invited to meetings where they can influence decisions, resulting in more impactful research.

These are significant benefits, but there are downsides. Being embedded within product and design teams means that researchers aren’t working alongside other researchers day-to-day and risk becoming islands. Without some form of connection or communication with others in your craft, the upskilling and learning that tends to happen naturally when working with other researchers is often lost. In short, researchers might become experts in their product areas, but professional isolation and stagnation can set in.

One antidote is to create an informal network of researchers that replicates a research federation, such as via monthly meetups, “lunch and learn,” cross-team review sessions, or a common Slack or Microsoft Teams channel. Whatever mechanism you choose to use, the goal is to foster a sense of shared identity and connection that the organization’s structure doesn’t automatically provide. These sorts of efforts aren’t a panacea, but they can help combat the most common downsides of decentralization.

You might ask, “What is the best ResearchOps setup for a decentralized research organization?” The frustrating answer is: it depends.

In some cases, there are no ResearchOps specialists (see Figure 2, Teams 1 and 2). In that situation, designers and researchers must take over the operational tasks. Team 3 illustrates a scenario in which a single ResearchOps specialist supports all operational tasks. In Team 4, the small ResearchOps team is made up of specialists who focus on specific areas, such as participant recruitment, budget and tooling, research knowledge management, or other operations-related areas necessary for your team.

If there’s more than one ResearchOps specialist on a team, it’s worth evaluating whether to further specialize, leveraging the economies of scale that come with specialization. When people become experts at one thing, both wasted time and the effort of being a “jack-of-all-trades” are reduced, making the operations cheaper to deliver as you scale up.

Finally, the ResearchOps functions in Teams 5 and 6 are more complex because they support multiple teams in a decentralized model.

Throughout the article, I’ll dig into the questions and points to consider in each scenario.

Centralized: Consistent, but Distant

In a centralized mode, researchers are organized in a unified team that operates like an internal agency (see Figure 3). Researchers can be quickly deployed to tackle high-priority projects anywhere in the organization, while maintaining consistent standards and facilitating cross-team learning.

Because researchers work side by side and regularly talk to each other, they tend to maintain consistent methods, leading to more reliable and comparable insights. Researchers also typically find themselves in the same meetings, so knowledge sharing becomes natural.

Finally, from a leadership perspective, leaders can easily see what’s happening across the organization and allocate resources strategically, rather than reactively.

Those are all the upsides, but on the downside, a centralized model can often distance researchers from where product and development work is happening, disengaging them from day-to-day product conversations. And because they’re not involved in the daily work, they don’t tend to build the depth of expertise or the strength of stakeholder relationships that come from being embedded. Through lack of exposure, product teams rarely fully understand what the research team can offer, leading to further dissonance.

Finally, the research team may end up responding to requests on an ad hoc basis rather than shaping product strategy from the outset. Worse still, rather than supporting product teams, research might look like an impediment to good decision-making, or even be used as a reason for poor decision-making.

Here’s the reality: a centralized research model can create bottlenecks, giving product and design teams good reason to claim that researchers are slow and a blocker. Sometimes they’re right, but sometimes they’re wrong.

Fighting or trying to enlighten research detractors won’t help. What does help is transparent prioritization and decision-making about which study is supported, self-service tools that can be used by non-researchers to conduct simple studies, and strong ResearchOps handling of enablement-oriented processes and tasks, such as participant recruitment, so researchers can focus on research.

Even with all this, the “slow research” perception can prevail because a strong ResearchOps function tends to introduce process, and process can feel like bureaucracy. It’s in times like these that I remind myself that “slow is smooth and smooth is fast,” meaning that deliberate, controlled processes are faster and more effective than rushing, making mistakes, or building one-off solutions for every problem.

So what’s the best ResearchOps model for a centralized research team, you might ask? Frustratingly, the answer is: it depends. It depends on the team, the research team’s business objectives, and the personalities, skills, and specialties of the individuals on the research team.

But here’s what tends to be true: in a centralized model, the ResearchOps team’s model becomes more straightforward, especially if it reports directly to the research team. If you have a smaller centralized research team (usually fewer than ten people) and a ResearchOps team of one, your only option is for your ResearchOps team of one to be a generalist: someone who handles all of the elements of ResearchOps.

If you’re a ResearchOps manager and the research team you support transitions from a decentralized to a centralized model, you might choose to leverage economies of scale and centralize the ResearchOps practice so that each of your team members is able to develop an area of expertise rather than focus on everything.

If ResearchOps is embedded in decentralized research teams, rather than having five participant recruitment efforts, you might maintain one standardized recruitment process across all of the decentralized teams. This standardization and specialization opens up bandwidth for ResearchOps to focus on other research-supporting initiatives, the list of which can be endless.

Hybrid: Best of Both Worlds, but Complicated

A third operating model research teams might adopt is the hybrid model (see Figure 4). The hybrid approach is the corporate equivalent of having your cake and eating it, too. Researchers are embedded in product teams, gaining deep expertise and strong relationships, but remain part of a centralized research function, gaining craft development and cross-product learning.

It’s the best of both worlds, in theory. In practice, it can get complicated.

In a hybrid model, all researchers belong to a centralized research organization, but while some researchers are aligned to specific product teams, others work on cross-cutting problems. In other words, they focus on insights or problems that transcend specific product areas.

In this arrangement, all research teams report to a centralized “super-team,” so to speak, but each sub-team is aligned to its specific problem or product area. This promotes a strong research-centric culture (the feeling that everyone is working together in one large team) while supporting the specialization needed to focus on product areas and the ability to support cross-cutting initiatives.

This structure also creates flexibility: researchers can be temporarily pulled into cross-cutting initiatives that span multiple product areas, or the organization can shift the balance between embedded and centralized work as the business’s needs evolve, as shown in Figure 4.

There’s a lot of upside, but an unintended complication of the hybrid model can be dual reporting. You might end up with two bosses with potentially different priorities: a manager who manages you for the research stuff, and a manager who manages you for the product-centric stuff.

In this case, whose deadlines take priority? Who’s looking out for your professional and career development opportunities? If your leaders aren’t aligned on your work, managing your work becomes the work, and you’re frequently left stuck in the middle.

Overall, without strong coordination, the hybrid approach can result in processes and research standards becoming fragmented, leading to inconsistent quality and duplication. Leadership alignment can also be a struggle, with product and research leaders sometimes disagreeing on strategy or resource allocation.

While the hybrid model aims to capture the best of both centralized and embedded approaches, to avoid these pitfalls, you must encourage clear processes, strong communication practices, and well-coordinated leadership.

Patterns to Follow and Traps to Avoid in ResearchOps Operating Models

You’ve now got an overview of the models that research teams use to operate, but how do these impact how the ResearchOps team operates?

In her article, “DesignOps: 5 Common Team Structures” NN/G’s Kate Kaplan outlined four operating models for ResearchOps teams: solitary, specialized, distributed, and elevated. But which model is best, when?

Well, it depends on your size, structure, and what your team and research partners need most, and how they’re operating. But there are helpful patterns and traps to avoid (see Table 1). Keeping some quick decision-making criteria handy can help:

• If you have one ResearchOps person supporting a centralized research team, default to a solitary generalist model and protect time for the highest-friction operational work.

• If you have many embedded researchers across product areas with inconsistent practices, prioritize an elevated or specialized layer to standardize the essentials (recruitment, tooling, knowledge management), even if some support remains distributed.

• If the organization is hybrid and priorities shift week to week, plan for a mixed model and document decision rights (who owns what) to avoid double-reporting chaos becoming “the work.”

• If leaders are misaligned, treat alignment as a prerequisite: no operating model will hold if leadership incentives pull in opposite directions.

Solitary: Nimble, but Constrained

In smaller research organizations, say fewer than ten researchers, distributed or solitary models work best. The solitary model (a ResearchOps team of one) is the base model, defined by a single ResearchOps specialist covering the most pressing tasks.

A solitary model doesn’t typically suit larger organizations. In that case, a specialized or elevated model, in which a centralized operations team supports the whole organization, will give you the scale to actually make an impact.

In hybrid research organizations, you might need a mix of the four.

Specialized: Focused, but Conditional

In a specialized operating model, each operations team member is set up to focus on one activity, such as recruitment, tooling, knowledge management, or event planning (see Figure 5). This focus means that quality goes up across the board and there’s an economy of scale, but there’s a catch: you need enough team members to specialize to make this work, and if one of them leaves, you’re left with a major gap.

A simple solution is to institute a buddy system: pair a specialist with someone who knows the ropes (but isn’t an expert) who can fill in if the specialist is unavailable.

Distributed (In Hybrid Teams): Agile, but Siloed

In a distributed model, operations specialists are embedded within specific research teams as generalists (they may handle everything from knowledge management to participant recruitment), enabling them to build deep relationships and gain an intimate understanding of their team’s context (see Figure 6).

On the plus side, these teams are agile, responsive, and can handle whatever comes up. But this model can also create silos because what one team learns doesn’t naturally spread to other teams. Processes can also be unintentionally duplicated, and as a generalist, you might not have the depth to tackle complex operational challenges on your own.

Connection and community among operations specialists can promote cross-collaboration and learning, addressing duplication and knowledge gaps, so keep this in mind if you take this route.

Elevated (In Hybrid Teams): Broad Impact, but Unintegrated

An elevated model enables the ResearchOps function to sit at the organization level, supporting all teams equally (see Figure 7). This model can help build programs with broad impact, maintain standards, and allocate resources strategically, but it’s also harder for team members to know what’s happening on the ground because they aren’t involved in daily conversations with the research team.

As a result, the ResearchOps team can be slower to respond, and sometimes solutions won’t fit every research team’s needs.

Seven Principles for Navigating a Reorganization

This article may have proven that organizational design is mind-bending and complicated. Even so, the frameworks set out above should help you navigate your next reorganization with more clarity and, hopefully, grace.

I’ve learned a lot from living through several reorganizations involving the models I referenced, and I’ve compiled a list of principles for navigating reorgs—principles I wish I’d known earlier:

• Your leaders must be aligned. If your research leaders and ResearchOps leaders have fundamentally different philosophies, in particular about the role of research, you’re in trouble. That misalignment causes friction, mistrust, and makes effective operations impossible. Make it your top priority to mend, or at the very least to acknowledge, the misalignment before you do anything else.

• ResearchOps is service design. You should constantly and continuously analyze needs, build frameworks, and recognize that today’s solution becomes tomorrow’s bottleneck, and nothing is permanent. Make sure you and your team are well-trained in the craft.

• Centralized operations provides stability. Even when research teams decentralize, a centralized ResearchOps function can provide much-needed stability and consistency in managing tools, maintaining compliance, and coordinating cross-team initiatives.

• A proactive posture wins out over a reactive posture. If you’re stuck in reactive mode (constantly “hamster wheeling” to put out fires), make a concerted effort to shift into a proactive, strategic building mode. It’s essential for long-term success—and peace of mind.

• Community and connection matter. Fostering a community of practice is vital for knowledge sharing, mutual support, and maintaining cohesion. In short, find ways to bring people together.

• Standardization enables scale; customization enables quality. Standardization provides the bedrock of scalable operations, enabling consistency and automation. However, customization addresses specific needs at specific times. Generally, low-volume activities can sustain high variety, while high-volume activities require low variety to remain manageable and repeatable.

• “Slow is smooth. Smooth is fast.” Aim to create a predictable cadence of work that relies on accuracy, consistency, and steadiness. This is key to success.

Too Important Not to Share

This article was hard to write—harder than writing my dissertation on immigrant minorities, xenophobia, and hate crimes. I think that’s because writing about reorganizations and layoffs means that I have to write about decisions that affected real people, including people who are my friends and people with whom I still work. It also means being honest about what didn’t work without burning bridges and sharing, in the open, what would usually be communicated only via an industry-wide whisper network. But we need to talk about this openly.

An industry colleague once told me that they loved reorgs because it gave them the chance to see what they could get away with. They are truly stressful, but each one forces you to reevaluate what’s working, what’s not working, and how to build more resilient systems. What doesn’t work is not talking about what we are all experiencing during these changeable times. To learn what does and doesn’t work, we need to talk openly and courageously about what does and doesn’t work. I hope this article encourages and continues the conversation.

Disclaimer: The opinions expressed are the author’s own and don’t represent those of their employer.

Sponsor and Credits

The ResearchOps Review is made possible thanks to Rally UXR—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack. Join the future of Research Operations. Your peers are already there.

Edited by and .

Subscribe now

The ResearchOps Review is brought to you by Rally—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack.

In 2023, Jakob Nielsen published a series of articles urging UX professionals to embrace AI urgently or risk professional obsolescence: “either you’re the windshield, or you’re the bug,” he wrote.1 That’s to say, if AI were a high-speed car, a windshield would master it; a bug would be squashed by it. “It’s not AI that will snatch your job,” he wrote, “but the individual leveraging AI to outpace your performance.” Three years later, Nielsen’s articles are remarkably prescient.

AI is fundamentally changing how research operates (to be accurate, how everything operates), and with it, the research capabilities and responsibilities of everyone in its orbit. Researchers intent on being the windshield are leveraging AI to do more than simply alleviate logistics or speed up the doing of research (the past purpose of research operations) they’re also using AI to expand, or augment, their purpose: in the midst of doing research, they’re drafting product briefs, building lightweight prototypes, writing on-brand copy, and even making incremental product changes.2

In an article published in March 2025, the NN/Group offered that “AI is broadening the scope of what any individual can accomplish, regardless of their specific expertise.”3 (See Figure 1.0.) The notion of “experience designers” who, with the help of AI, can cover everything from content strategy to service design and research to frontend development is on the rise. AI is prompting entire specialisms to become democratised. (At this point, the idea that research leaders might choose whether research is democratised or not feels almost quaint!)

From Individual Expertise to Shared Systems

It might be an easy assumption that this “expanded-T” trajectory will further isolate disciplines or lead to an extinction event, wiping out entire specialisms.4 If everyone can do everything on their own—or rather, with an LLM or team of skills (the Claude kind), or agents in tow—then why have all these specialisms, and why work together?

In reality, the opposite is true. AI is currently only capable of delivering value (more on this later) when humans use AI to augment their own specialist intelligence and skills. To achieve these augmentative “wins,”5 teams working across the product development lifecycle (PDLC) must collaborate to build and train common-use AI agents, establish templates, and design workflows that enable non-specialists to independently deliver parts of their job with the help of AI. But systems are as much about control as they are about enablement, and it’s the system owner who tends to set the controls. So teams must also collaborate to establish guardrails, set standards, and formulate risk profiles for the parts of their jobs that should not be democratised or handled by, or via, AI.

The word “must” features twice in the above paragraph, which begs the question: Why must we do any of this work? Because democratisation is no longer a choice.

The research profession has been grappling with democratisation for more than half a decade, the implementation of which has usually meant providing resources and support to whoever wanted to do research; an egalitarian approach that rarely delivers return on investment to the organisation. But the accelerative, magnifying effect of AI has made the design of priority-and-risk-based research systems urgent. Never mind poorly facilitated user interviews; it’s no longer science fiction that a program manager or an engineer might deliver an advanced quantitative research study, replace customers with synthetic personas, or launch dozens of AI-moderated interviews overnight. Research teams must set standards and create workflows for when and how AI should, and should not, be used.

It would be compelling to assume that ResearchOps professionals—practitioners in a profession that emerged precisely to handle these systems and governance tasks—are currently consumed with designing well-articulated AI-augmented research operations. Plot twist: that’s not the case.

Every week, I engage with dozens of ResearchOps professionals as the editor-in-chief of this publication and as an advisor and educator, giving me an elevated perspective on what ResearchOps professionals are thinking about—and sometimes what’s on their roadmaps. Earlier this year, I ran a workshop with thirty ResearchOps professionals from across industries to explore what was on their 2026 roadmaps. Attendees collectively shared 170 tasks; remarkably, only twenty-nine were AI-related. That’s just 17 percent.

ResearchOps professionals aren’t innovating or augmenting operating models at speed behind closed doors, bereft of time to share their learnings. Instead, for a variety of systemic reasons, they’re late to the party—the most important party yet—and researchers are taking the lead. That researchers are taking the lead isn’t the issue. The greater concern is that neither researchers nor ResearchOps professionals are paying attention to the systems- or organisation-level design of AI-augmented research, which is problematic (extinction-level problematic) for several reasons.

The Hidden Cost of “I-Me-Mine AI”

There are four clear camps in the AI research space. Those who don’t want to use AI at all are in camp one, but public discourse indicates this group is shrinking. In camp two are those who are interested in AI but are too time-poor, energy-poor, or overwhelmed to take the leap. In camp three are those using AI to speed up or lighten the load of their own work (I call this “I-Me-Mine AI” to play off the title and lyrics of a Beatles song).6 Finally, in camp four is an alarmingly small cohort setting collective standards or leveraging AI at the operating system level.

To go back to Nielsen’s quote from 2023: “It’s not AI that will snatch your job, but the individual leveraging AI to outpace your performance.” Three years later, in an agentic age, I would argue that it’s not AI that will snatch your job, but the team, the discipline, or, more pointedly, the operating system that’s leveraging AI to outpace your performance.

Because UX roles are becoming less specialised, your individual value (the value that might win you a promotion) depends on whether you can do that “expanded-T” set of jobs or “jobs to be done”7 (JTBD) well. However, the past few years have taught too many people that individual value isn’t enough to protect your role. For your job to be secure, your discipline’s organisational value (the executive-level value that keeps your discipline employed) depends on how effectively you and your team can enable others to do parts of your job, all of which hinges on the design and delivery of scalable operating systems. This phenomenon is far from brand-new—it’s been defining research teams over the past half-decade via democratisation—but as mentioned earlier, AI is amplifying and magnifying the dynamics, and the opportunity.

An Operational Imperative

Unlike the generalisation of UX roles, which has largely been prompted by artificial intelligence (pun intended), the recent shift in how corporations operate is perpetuating an evolution in research operations: the JTBD and the role, too.8 “The great fiscal reset” that has characterised the lives of tech workers since 2022 has led to the destruction of many research teams in favour of skeleton-crew research operations functions tasked with democratising research.9 To go back to NN/G’s “expanded-T:” just as UX roles are conflating, in many organisations, the line between research and research operations is blurring, too. Every week, a researcher messages me to let me know they’re now doing research operations, or a ResearchOps professional lets me know they’re working solo—that’s to say, without the partnership of a research leader.

The growth trajectory of research teams is in the process of being flipped, a point that is highly relevant in an AI-augmented, profit-oriented world: where research leaders once built a team of user researchers, then constructed an operations function beneath them, ResearchOps professionals (or research professionals who have absorbed operational responsibility) are now building operations programs focused on democratisation, then making the case for research headcount on the strength of the scaled-up value they’re delivering. In many cases, democratisation is no longer a research leader’s choice—if they’re there to make the choice at all—but an operational imperative. The message from corporations is this: we want you to deliver systems that empower a spartan, generalist workforce, then we’ll invest in specialists where they’re most valuably employed.

This is precisely where ResearchOps should be stepping in—or where researchers now working operationally should focus their efforts. ResearchOps exists explicitly to build and maintain the infrastructure that makes accessing research insights (not just doing research) possible at large, and even super-large scales. In an AI-accelerated research environment, that work hasn’t become less important; it’s become urgent, not only because organised is nicer than disorganised, or orchestrated is more efficient than not, but because this is an unprecedented opportunity for research and ResearchOps professionals to secure a high-power seat at a table that’s desperate for trustworthy knowledge, and desperate to make the most of AI.

From Individual Efficiency to Organisational Impact

A 2025 PwC survey found that 56 percent of CEOs reported their companies were not yet seeing financial returns from AI investments.10 Writing in the Harvard Business Review in February 2026, Prabhakant Sinha, Arun Shastri, and Sally Lorimer concluded: “In most organisations, digital underperformance isn’t a failure of change management, but a mismatch between new ways of working and old organisational designs.”11 Though few ResearchOps professionals are delivering AI-augmented workflows, whether new or based on existing ones, there are pockets (a vague designation, I’ll admit; the number is currently unquantifiable) of researchers delivering AI-augmented research operations. An excellent, publicised example comes from the DoorDash research team. Their articles “Researchers Are Rewriting the Playbook with AI” and “How Cursor & Claude Code Are Changing Research at DoorDash and Deliveroo” are essential reading, in which they share how the company is systematically building AI-augmented research operations from the ground up. Indeed, the articles give the impression that the entire organisation is on the same trajectory, which tends to help with innovation.

DoorDash’s forward-thinking approach is inspiring because AI-in-research content is predominantly self-focused: I-Me-Mine AI (researchers using AI to augment their personal efficiency) is the dominant “operating model,” with little, if any, conversation dedicated to continuity across individuals or research teams, let alone the wider organisation. This is problematic because corporations don’t invest in or maintain teams simply because the individuals working in them are more efficient or happy; an unfortunate fact. If the research profession’s I-Me-Mine approach to AI doesn’t change, it will be a case of history repeating itself.

When ResearchOps first became widely recognised as a profession, circa 2018, research leaders were making a strategic misstep: typically, people were hired into ResearchOps roles to help make research (in effect, researchers) more effective and efficient. Rather than being tasked with delivering value to the organisation, their work tended to focus on, for instance, unburdening researchers of the task of participant recruitment by assuming the duty altogether, building a library to enable secondary research, or handling the undesirable work of procurement and vendor management. It’s not to say that no value was delivered, but did the corporation earn more money because their researchers were better prepared, enjoyed their work more, and were more focused on delivering research? In most cases, the answer is no.

The Clock is Ticking

To return to the Harvard Business Review article, consider this observation related to commercial AI efforts: “Repeatedly, executives tell us that digital initiatives fall short of expectations: improving execution but not strategy, increasing efficiency but without freeing capacity for higher-value work, showing strong adoption but limited business impact, and creating disconnected solutions that don’t fit with existing workflows.”

From the corporate perspective, the value of AI doesn’t lie in what individuals can do to augment their own workflows, for themselves—useful for building initial ideas and confidence, but a narcissistic and limiting approach if it becomes the status quo. I-Me-Mine AI might enable an individual researcher to save hours on analysis and synthesis, or reclaim time for more strategic thinking. But unless AI experimentation and adoption deliver tangible, cross-functional efficiencies to the organisation (an operations-level boon that’s obvious to a CEO), the research profession—and especially the ResearchOps profession—will have missed the moment. And what a moment it is.

AI is, without doubt, going to revolutionise how research is conducted and consumed across organisations, with or without researchers or ResearchOps in charge. One needn’t be Nostradamus to make that prediction! AI is powerful and empowering enough that, if research professionals don’t get ahead of building scaled-up research systems and purposefully setting standards for how they and others in their organisation do, create, or consume AI-enabled research, someone else will—perhaps with less concern for methodology, rigour, and ethics. This is the immediate opportunity for research and ResearchOps professionals, and now is the time because the canvas won’t remain blank forever, habits and preferences will become embedded (and oh-so-much harder to change), and the opportunity to naturally lead the design of organisation-wide systems is finite. The clock is ticking. Some might say it’s a time bomb. (At this point, I must reference another musical classic, Rancid’s “Time Bomb.”)

You Already Have the Required Skills

Researchers paired with ResearchOps are better placed than most to deliver AI-augmented systems that are impactful organisation-wide and prized at the executive level. To do this, you’ll need to develop AI literacy, for sure (see Anthropic Academy), you’ll also need to reimagine the purpose of your role—it’s being reshaped around you anyway—and make new use of your specialist skills.

The following list will sound so familiar it might even seem trite, but that’s the point. To seize the moment, you must hone your inner:

1. Service or system designer. Take a step back and study the full ecosystem; don’t dive in with a limited view.

2. Researcher. Learn from individual researchers or people who are doing or consuming research to identify good and bad practices. How are people approaching AI? What’s worth repeating? What’s worth designing out?

3. Strategist. Pick the most value-adding and low-risk places to implement AI. Where does AI augment human intelligence and skill? Where does speed threaten rigour? Where is rigour most important?

4. Change manager. Gather the makers and innovators in your organisation into a working group or council to encourage collaboration and the development of shared systems and rules. Governance is always better when it’s developed by a group.

5. Communicator. Start talking about what’s working and what isn’t working. Find ways of sharing your learnings to bring others on the journey, establish the value of your work, and help others build on your learnings.

Above all, you must switch the conversation from “Where should I use AI in my workflow?” and “Which AI tools should I use?” to “Where should we use AI in our workflow?” and “Which AI tools should we use?” Through action, language, and politics, you must design the culture in which your AI-augmented research operations will live—and the future of research in your organisation.

The Conversation Is Yours to Lead

The breakdown of siloes and expansion of roles across UX signifies that shared technologies, standards, and cross-functional collaboration are more essential (and powerful) than ever, but someone has to lead the conversation. It might have been assumed that ResearchOps would lead it, but researchers are centre stage right now.

It’s excellent news that the conversation is happening, no matter the altitude and who’s leading it. But to summon Nielsen’s windshield-bug analogy one last time, when it comes to AI-augmented research, without a significant shift, the ResearchOps professions are collectively in danger of becoming the bug, no matter how good you are at prompt engineering. More importantly, the research profession as a whole is in danger of being passed by the power of the moment: a real chance to get ahead of everyone else in setting system- and operating-level standards, and the future agenda for research.

“I Me Mine” was the final new song recorded by The Beatles before their 1970 breakup. Written by George Harrison, it’s a commentary on the human ego and the tendency for humans, and perhaps AI (time will tell), to look after oneself. But change often requires just one person to think of the whole; to bring people together and design a mutually beneficial system.

Sponsor and Credits

The ResearchOps Review is made possible thanks to Rally UXR—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack. Join the future of Research Operations. Your peers are already there.

Edited by and .

Subscribe now

The ResearchOps Review is brought to you by Rally—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack.

I started my career selling surgical cameras for keyhole surgery, then sold grand-piano-sized blood analysers that could process hundreds of samples a day for every blood parameter you could think of. For the last twenty years, I’ve worked in cardiac operating theatres in a part of the heart surgery world called electrophysiology, both as technical support during procedures and as a salesperson. Essentially, a day in the “office” for me involves collecting three-dimensional data of a patient’s heart mapped in real time (see Figure 1.0). During the procedure, we analyse the patient’s heart anatomy, tissue voltage (viability), and, most importantly, the direction of travel of the heart’s electrical current. Using this data, the physician performs small ablations, or burns, to intricately adjust the flow of electricity in the heart during fast heart arrhythmias. In short, if your heart occasionally beats faster than you would like, the people I work with can fix it.

At this point, you might be wondering why a sales guy who helps fix hearts is writing an article for The ResearchOps Review, and it’s a good question. Fifteen years ago, when I was selling one of these systems to the cardiac department of a hospital, I learned a primary lesson that helped me become an award-winning salesperson, not just a good one. When I started working with user experience professionals in 2024—a story for another time—I realised that the sales tactics that had worked so well for me in medical sales were equally valuable in helping research professionals show their impact and value, too.

In this article, you’ll learn why sales is not just for salespeople. I’ll share a simple approach for how you can articulate (and sell) the value you’ve delivered, or want to deliver, how to communicate your achievements so they land with your key audience, and how these tactics will help you turn stakeholders into champions.

Selling Isn’t Just About Transaction, It’s About Transformation

I have an excellent track record in medical sales: I sell US$200K hardware, large magnetic-based sensors, energy generators, and steerable catheters—all of which provide hundreds of thousands of dollars in ongoing sales revenue to my employer. To sell this system to the hospital, I need to convince the specialists involved that it’s the best choice; the purchase may also need to be approved by a committee of experts, and, finally, the annual operating costs must be signed off by the hospital’s finance department. This process can take six months or more (a timeline that will sound familiar to ResearchOps professionals) and must be completed with every hospital that plans to use our system. It’s no easy task!

Fifteen years ago, I’d successfully juggled all of these requirements and was ready to close a deal, but there was a perplexing problem. My own team didn’t share the same sense of priority for delivering quotes, bundling agreements, obtaining internal sign-off, and meeting with key stakeholders. This was a great deal for the company, so why weren’t my colleagues jumping on board? In this instance, timing was everything, so the deal was at risk. That’s when I realised the importance of in-house sales. I was taking my customers on a journey of increased success—I’d successfully sold the hospital staff the picture of increased profits due to shorter procedure times and a better reputation due to fewer complications—but I had completely ignored the job of selling the importance of the agreement to my own colleagues.

This is key for research and ResearchOps professionals because, as knowledge workers, your value is measured (frequently subjectively) by your stakeholders and colleagues. In a corporate environment, great ideas and excellent work without enthusiasm from a team or backing from stakeholders are as good as if they never happened at all. So if you don’t sell the value of your insights or your research systems, or effectively prove a successful track record, you’ll be fighting to achieve your full potential—and sometimes to keep your job.

It’s often assumed that sales is primarily about signing deals or convincing someone to buy something, and it is. But it goes beyond the transactional—it’s also about getting someone onboard, or, put differently, helping them see how the exchange will transform their impact, approach, success, and outcomes.

That moment when the penny dropped fifteen years ago, the hospital needed a custom agreement, but the typical turnaround for creating this type of agreement would have far exceeded the window available to close the deal. By writing an elevator pitch outlining the needs, features, and benefits tailored to our internal finance, legal, and management teams, I was able to start the process. As a result, I got the agreement approved in just one morning. Not only that, but everyone now understood why we required a non standard agreement, and because they played a special part in putting it together, became invested in the outcome of the sale. This sale made my work visible to many levels of management and gave me easier access the next time I needed something, all because I asked my colleagues for a favour in a way that brought them on the journey.

Research operations is an emerging role; few stakeholders fully understand its scope and purpose, and much of the work is infrastructural and can take time to show results. Plus, you’ll often require cross-functional collaboration to achieve your goal. The sales tactics I’ll share in this article are crucial to addressing these challenges and going beyond transaction. But if you’re a research, product, design, or marketing professional, read on, because you’ll be able to leverage these insights to get buy-in and boost your impact, too.

Sell Your Work In Thirty Seconds, or the Art of the Elevator Pitch

Imagine you’re at the office and you step into the elevator only to walk straight into your boss’s boss, someone who makes decisions about your role but doesn’t really understand what you do. Or perhaps you work remotely, and you find yourself on a call with an executive for thirty seconds before anyone else arrives, and they ask what you’re working on. After a five-storey elevator ride, or a remote scenario equivalent, would they walk away thinking that your team offered good value? Would they ask for a follow-up meeting to discuss allocating additional funding to support your work? Would you be able to succinctly articulate the crux (and value) of what you do?

I’ve run several workshops1 for research and ResearchOps professionals, and most people struggle to define their goals in a way that shows benefits not only from their perspective but also for the business’s leaders. Workshop participants are often caught off guard when they have to explain their work to a team that doesn’t understand the value of research or research operations—they even struggle to explain it to each other. This is a huge problem: if you, the person who is most up to date on the initiatives you’re working on, can’t articulate what you’re doing in a way that’s easily understood, how will your work ever be valued or understood by the busiest people in the organisation? Let alone getting your colleagues onboard.

Objections I’ve heard at this point range from: “but it’s my manager’s job to promote my work, not mine,” or “I send senior leaders a monthly report, which they should read,” to “they’re running the organisation, so it’s on them to keep up to date with what we do,” or “this initiative is important for the user experience or for ethical reasons, but they just don’t get what we do.” And maybe some (or most) of that is true, but the reality is, if you want your work to get noticed and be valued, it’s up to you to do the selling by communicating it in such a way that your voice is heard. For many people, it can feel daunting, but there’s a straightforward framework from the sales world that makes it much easier to do.

Crafting Your Elevator Pitch: Needs, Features, and Benefits

Imagine you’ve identified a need to simplify the consent process for research participants. The current consent form isn’t written in plain English (it’s ten pages of legalese) and isn’t available in multiple languages, which increases the dropout rate among carefully screened, hard-to-reach participants from enterprise companies, costing the organisation time and money. Perhaps the consent form also doesn’t meet accessibility requirements or comply with new regulations and standards, which, if not amended, could cost the company millions in data privacy fines or lead to the loss of government contracts. So, you make it a priority to update the consent process, which takes several months and a lot of your and the legal team’s time.

To get buy-in for this initiative, you set up a meeting with your manager and prepare a pitch, covering three critical components: needs, features, and benefits. Now, most people can articulate why something is needed (a problem to be fixed or an opportunity to benefit from) but few can articulate the features (how you’ll fix the problem or leverage the opportunity, and what the solution will do) and benefits (how your solution will generate revenue, save money, or manage risk for the company) in a way that appeals to the person they need to get buy-in from.

So, let’s take the need as previously outlined (updating the consent process)—that’s the obvious bit—and add the features and benefits.

• Needs. The current consent form is overly complicated, leading to a higher-than-average dropout rate amongst hard-to-reach enterprise participants. It also doesn’t cater to participants living with disabilities, slowing down accessibility research and putting government contracts at risk. We’ve set this quarter’s goal to rework it. This will require an allocation of 𝖷 hours of internal time across operations and legal, and 𝖷 dollars to achieve.

• Features. We’ll create a simplified, one-page consent form in easy-to-understand language suitable for 90 percent of enterprise participants, so they won’t feel the need to run it by their legal department before taking part in a study. This will halve the number of dropouts for every fifty participants recruited.

• Benefits. Most people forget to share the benefit of doing something, but it’s the most important part! Within six months of deployment, we’ll have recouped the investment cost by reducing dropouts by 𝖷, saving 𝖷 recruitment hours currently wasted on dropouts, and by speeding up research delivery by 𝖷 weeks per research study.

This is your elevator pitch: a succinct statement that gets to the point and outlines the needs, features, and benefits in a memorable format. It might not contain every detail or metric, but it should communicate the basic idea clearly and boldly.

You can use this format to sell the benefits of a past achievement, or to propose a new project to secure buy-in, headcount, or funding—or all of the above. You can also adapt your elevator pitch to share in a one-on-one, send in a monthly report, share via a DM, or to fill the thirty seconds in which you found yourself face-to-face with your boss’s boss in the elevator (or onscreen). It’s happened to me before!

Without practice, speaking a pitch out loud is awkward and even hard; it’s not something that comes naturally to most people. But once you see how effective it is, you won’t want to communicate your idea any other way. You can bring your own style to it, too; just make sure to keep it succinct and stick to the basic format. It’s a great idea to practice in private with a friend, or announce to your manager that you’d like to get better at this and ask if they’re willing to coach you. They’re sure to learn something from you, too.

This elevator pitch, by itself, is a strong message and a great way to frame your work in a simple, memorable way. But you can take your elevator pitch one (important) step further by tailoring it to the person from whom you need to get a “yes.”

Tailoring Your Pitch to Your Audience

Consider the personality of whom you’re pitching your idea to. Do they care about data? Are they always looking for wins that can be advertised to the wider organisation, perhaps even helping them gain a promotion? Are they often more interested in the social credit—the story or the opportunity for collaboration—behind an idea? Or are they typically interested in a concise summary so they can move on to the next thing?

I don’t like leaning too heavily on generic personality frameworks—the Myers-Briggs Type Indicator (MBTI), Gallup’s StrengthsFinder, and the Enneagram Test are among the best known—and here’s why:

1. Personality frameworks are often overcomplicated. For instance, the Myers-Briggs has four preference pairs and sixteen different personality types, such as ISTJ, ISFJ, INFJ, and INTJ. If you have to spend too much time trying to decode the personality type of the person in front of you, you might miss your moment.

2. There’s a tendency to lump people into one category. The reality is that people’s personalities tend to change depending on the situation. For example, when I present technical data on product safety or patient outcomes, the doctors I work with are data-oriented, so I lean into the details. But that same doctor will become a decisive “driver” if an emergency arises during a procedure, which makes sense!

Besides, almost all of these frameworks hinge on the same four primary professional personality types: the results-driven person, the data or analytical type, the storyteller or people person, and the “expressives” or visionary future thinkers.

While they will never capture how extensively different we all are, these primary personality types (see Figure 2.0) can help you understand how your key stakeholders lead, collaborate, and make decisions in the workplace—and, of course, how to sell them your ideas.

• A driver or results-driven person is typically time-efficient and a logical planner.

• A detail-driven person is focused on the data and finer details; they want to know everything.

• A storyteller or people person is typically a supportive, relationship-driven team player.

• An expressive, future thinker is typically a visionary and creative influencer.

If you’re pitching to a results-driven person (a time-efficient, logical planner), you want to be brief and get to the point. For a detail person, you might share your elevator pitch, then add, “I’ve got a lot more detail to share. Could I book a time to show the highlights from the data, so we can dive deeper where you’re most interested?” For a storyteller or people person who’s, say, looking for wins that can be advertised far and wide, you might offer to pass on a one-page win-focused summary, which they can share at their next team or management meeting, or better still, offer to present it yourself. Finally, for the visionary, don’t limit the benefits of what you have achieved to the immediate future; draw a picture that shows further progress down the line—these people love to hear about your five-year vision.

A concise elevator pitch that focuses on needs, features, and benefits, combined with a tailored approach based on personality type, will take you far. But even with all this preparation, it’s worth remembering that what you wanted to say, what you said, and what your audience heard don’t always align.

What You Wanted to Say, What You Said, and What They Heard (and Didn’t Hear)

Several years ago, I worked as a skydiving instructor, taking people on 15,000-foot skydives. They received a full day of training, then jumped out of a plane for a fifty-second freefall before opening their own parachutes. As scary as this sounds, there would be an experienced instructor, like me, on each side of the student, holding them upright, so if the student curled up in a ball of fear (which sometimes happened), we’d hold them in the correct position, and they’d generally acclimate in time to open their own parachute (see Figure 3.0). The student would then float to the ground and have adrenaline-filled stories to tell.

Before the jump, I would train the student on the exit procedure—how to jump out of the plane—and ask them whether they understood. The answer was almost always “yes.” I then asked the would-be skydiver to explain the exit procedure back to me. On more than one occasion, they would ask, “Actually, can we go through it one more time?” They were literally going to leap out of a plane for the first time, and hadn’t listened closely enough to understand what to do!

The lesson is this: What you want to say, how you actually say it, and how what you say is interpreted are all entirely different things. This is a highly applicable lesson, whether you’re training adrenaline junkies, selling medical equipment, or getting buy-in for your work as a research operations professional. So what should you do about this?

The hardest part is noticing the disconnect. At the end of your pitch, make it a habit to ask, “Which parts about that made the most sense or appealed to you—or not?” or “How does this fit into the company goals?” If pressed for time, you can even simply ask, “How does that land for you?” Occasionally, you won’t get a satisfactory response, but more often than not, these kinds of post-pitch questions will start a conversation in which you can learn more about your manager’s needs and wants in relation to your ideas and deliverables. This is also the moment that you can share a bit more (personality-oriented) detail about your proposition. Listen and watch closely, because you’ll gather invaluable insights to hone your direction and pitch, and better meet their needs (or speak their language to sell them your needs) in the future.

Selling Your Work in Uncertain Times

These tactics—the elevator pitch, tailoring to personality types, and checking for understanding—might seem like extra work, but they’re more important now than ever. And it’ll feel like a much lighter lift, the more you practice. If you’ve lived for long enough, you’ll know that economic challenges are a historical constant. In just the last thirty years, the dot-com bubble has burst, there was the global financial crisis (GFC), the COVID pandemic, and now layoffs and the unfolding impact of AI. For better or worse, change is constant. Humans will continue to invest in (and invent) new technologies, pushing companies to find new efficiencies and cost savings, adapt to market trends and forces, future-proof their operations, and keep investors investing.

In a changeable world, it’s often not enough to do a good job; you need others to know that you’re doing a good job, and you need to make it easy and beneficial for them to champion you. Ideally, by communicating your value and successes, they’ll feel invested in sharing and promoting your value and success, too. To achieve this, first, you must make smart choices about how you deliver value to the organisation. I recommend reading Kate Towsey’s recent article, “Why the Distributed Growth Model Is Failing Research Teams—and What to Build Instead,” and then applying the sales concepts I’ve shared in this article. Using these simple sales techniques takes a bit of courage and practice, but it doesn’t require extra time or resources; in fact, it takes less. If your initiatives and achievements are supported by those around you, you’re more likely to be put at the front of the line, just as I was after I landed that deal fifteen years ago—and have been ever since.

Sponsor and Credits

The ResearchOps Review is made possible thanks to Rally UXR—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack. Join the future of Research Operations. Your peers are already there.

Edited by and .

Subscribe now

The ResearchOps Review is brought to you by Rally—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack.

A few weeks ago, I saw a graph in The Economist that was deeply enlightening. Unfortunately, I can’t find the article again, but it illustrated the surge in investment that defined the tech wave between 2010 and 2022. More than just an illustration of the economy, this graph helped me understand that my career had been buoyed not only by hard work, ingenuity, and (I like to think) smarts, but also by impeccable timing. I joined the tech workforce at just the right time to enjoy an era of heedless spending on salaries, equity, and snacks—and sometimes rampant hiring. With this new framing in mind, plus years of studying the economy, I’ve come to understand that companies in emerging fields, like technology and now AI, operate according to a growth model, not a profit model, with significant implications for how teams within the company should operate to succeed. This context is important, vital even, because it helps explain what’s happening in the fields of research and ResearchOps, and how to build teams that thrive even when the economy shifts, or a novel field becomes mundane—tech1 is no longer sexy, instead AI, defence, and the space sectors are.

How Growth and Profit Models Are Reshaping Research

The growth of the ResearchOps profession in the past decade has been remarkable. What started as a niche role in the most progressive technology companies of our time is now a role that’s hired by all sorts of companies, from startups to the BFSI2 sector, and legacy media giants. As part of that evolution, ResearchOps job descriptions and, by extension, ResearchOps professionals themselves, are becoming more specialised.3 Laundry-list job descriptions—the type that list an impossible scale of work, never mind the scope—are becoming less prevalent and are being replaced with well-defined requests for strategists and systems designers, and knowledge and data specialists tasked with building the capability, both human and technical, for entire organizations to generate and consume insights at high speed and at large scales. And sometimes they’re being asked to do this without researchers.

If you work in ResearchOps and you’re looking for a more expansive career path, this may sound like great news, and it is. If you’re a researcher, this may sound horrifying, and it should. The expectation that ResearchOps can run the show solo isn’t just a problem for researchers; it’s also a problem for research operations and, in the long term, for the companies that take this tack. In this article, I’ll explain why this trend emerged (hint: growth and profit models are key), why it’s unsustainable, and why—and importantly how—research and ResearchOps leaders must partner in new ways to build the future of research.

An Unsustainable Trend: ResearchOps Without Researchers

Over the past two years, I’ve heard an increasing number of stories about entire research teams being laid off, and yet the ResearchOps function has remained intact, tasked with the job of enabling the rest of the organization to do what the research team used to do, and more. I’ve also heard stories of companies in which ResearchOps is their first research hire: they’ve skipped the researchers and gone straight to ops. These teams aren’t only being asked to democratize the doing of research, they’re also being asked to integrate AI wherever it makes sense (literally) and build insights traffic systems, a term I’m introducing here, defined as systems that enable research insights to flow through the organisation in the right cadence, format, and grammar so that the audience, whether product, design, marketing, or executives, can easily access and digest it. Shivanjali Mishra’s recent article, “The Systems Linguist: How Mapping Data, AI, and Language Builds Smarter ResearchOps,” captures this beautifully and is essential reading.

The vision of democratized research, combined with AI integration and insights traffic systems, is exciting—it is the future of research—but a world in which researchers are either not involved or have been reduced to a tiny team of usability testers (as if it’s 2010 again) is not sustainable. ResearchOps professionals are highly capable strategists, systems architects, and business analysts, but unless they come from a research background, they’re just not as equipped to make decisions about research strategy, methodology, or quality management: all key to successful research operations.

If you’re a research leader, this message is for you: Whether companies know it or not, they need you. And they need you to respond to the change in how they’re operating by building and operating research teams, or research capabilities, in entirely new ways, too.

So, what does that look like?

To understand how you should operate now, it’s useful to understand how research scaled during the 2010–22 tech wave and synthesize the lessons learned for rebuilding a more robust research capability in your organization today.

How We Got Here: The Distributed Growth Model

According to “The Crunchbase Tech Layoffs Tracker,” since 2022, a total of 509,000 tech jobs have been impacted in the US. The layoffs weren’t a blunt response to economic pressures—the NASDAQ index has never been higher—or the promise of AI. Instead, it’s symptomatic of the reshaping of how technology companies operate, which is reshaping how every person and team within them operates, from product to design, and research to ResearchOps. But what pushed tech companies to make such a significant employment correction—one that’s affected countless professions—and what does it mean for research?

Between 2010 and 2022, UX research teams ballooned off the back of well-funded growth in tech—and even in the odd government. Companies were focused on growing the size of their customer base, number of monthly active users (MAU), and even the number of employees. Interest rates were low, growth was paramount, and profit was second fiddle—and the talent market was highly competitive.

To support growth in customers and MAU, companies invested in user researchers: in simple terms, happier MAU equals more MAU, which equals growth, which equals happy investors. (You can say the same thing for AI companies today.) This dynamic isn’t likely new to you, but here’s the important bit: often, this hiring didn’t happen as a centralized effort, one in which the “company” or, more accurately, the company’s executive said, “Let’s build a user research team that helps us make decisions about critical business areas.” Instead, unknowingly riding the tech wave, the research team grew via distributed investment.

Here’s how it played out: a product or design manager realised that the amount of research they needed, often on pre-launch usability testing—let’s make sure we’re not launching a flop!—exceeded the number of hours they had available. So, they secured the funds to hire a researcher, either as a contractor or a full-time employee. The researcher focused their efforts on usability testing and, without the complexities of a scaled-up research department or too many rules, could often deliver insights fairly quickly without anyone else needing to lift a finger. So, the manager, keen to maintain this new superpower, hired the researcher full time and, soon enough, hired more researchers, putting the first researcher in charge. And just like that, the first researcher on the scene became a research manager.

Soon, other product and design managers, envious of their colleagues’ research capabilities, secured headcount to hire their own researcher. So they “flipped a headcount” to the research manager, on the condition that the researcher they hired would be dedicated to their specific team. Over time, the research manager became a manager of ten researchers, then twenty, then thirty, and, in some cases, a hundred or more researchers, most, if not all, acquired through flipped headcount. As the team grew (and as the notion of ProductOps, DesignOps, and ResearchOps became more popular), the research manager secured headcount for a ResearchOps professional tasked with making researchers’ work easier. In truth, these folk often acted more as research assistants than research system designers, making the research team even more expensive to operate with little measurable value delivered beyond the research team…unless they were put in charge of democratizing research. In this case, they were given a platform to showcase their skills as highly efficient, business-aligned enablers for hundreds of people—an important point in this narrative arc.

There are lots of ways this story can play out, but the central theme remains the same: instead of executive teams allocating a centralized budget to build a research capability aligned with its goals, and therefore geared to deliver executive-level value, product and design managers flip headcount one at a time, and, in doing so, fund the growth of a research capability without anyone necessarily being aware of the collective organizational investment. But the collective investment isn’t invisible, or small. It’s accurately recorded, down to the cent, in the company’s accounts against a line item labelled “research.”

The Hidden Cost of Distributed Growth

You might be surprised by how many research managers don’t know the total cost of their team to the business, or the average cost of each research study, which is a major managerial mistake. As a back-of-the-napkin calculation (corroborated in detail by Claude), a ten-person research team based in San Francisco costs between $2.2 and $3.2 million per year, depending on benefits. That’s peanuts when a company’s annual revenue is $53.439 billion (that’s Intel’s revenue in 2025; Intel also had the most layoffs in 2025). But in a cost-cutting, profit-focused context, it all adds up—and there’s one more bit of interesting maths that you should do. On average, a researcher can deliver two or three qualitative studies per quarter, which means that every research study costs between $20,000 and $30,000 to deliver.

The composite of these numbers, along with distributed investment, is where the rubber hits the road. If a research study or insight only delivers value within its immediate context and then disappears into thin air, from the executive’s elevated point of view, research is simply expensive vaporware—$3.2 million-per-year vaporware, to be exact.

When investment in research is distributed, and research outcomes aren’t constantly repurposed and redistributed across the company, fail to mimic the product development beat, or don’t hit the nail on the head for the highest-priority audiences, each stakeholder who flipped a headcount might know the value their researcher delivered, but no one else will.

This kind of spending may pass muster while the company is focused on growth rather than profit, but when the focus shifts to maximising profits, as it did in 2022, the executive will comb through the financial reports and find ways to cut costs. If they’re unable to communicate or point to the value a team delivers, or are convinced there are cheaper ways to achieve the same goal (say, by democratizing the effort or leveraging AI), that team will find itself on the chopping block. But if there’s a small team of business- and tech-savvy operators who are already enabling hundreds of people to do and consume research…well, we’ll keep them, thanks.

The suggestion here is not that researchers find a way to tie their value (or the insights they deliver) to the bottom line, as sales or manufacturing might. That’s a wild goose chase, and something I cover in detail in my book Research That Scales (see Chapter 2, “Lost and Won on Strategy”). The suggestion isn’t even that you must compromise and democratize research to make it seem like you’re delivering value—and save your job. Research is a cost center (a team that’s not expected to generate revenue directly), and there’s no need to pretend otherwise. But cost center or not, every team in an organization must operate in a way that makes its value, or the perception of its value, obvious to the executive, which requires being highly strategic about how you operate. In practical terms, this means you must have a research strategy, research operations strategy, and operating model that conscientiously balance distributed and vertical value.

If the terms “research strategy,” “research operations strategy,” and “operating model” have caught your attention but you’re not sure what they are or how to create them, I’ve literally written the book for you. The first four chapters of Research That Scales are dedicated to these concepts. Over the past eight years, I’ve run masterclasses with hundreds of research managers, and I can count on one hand how many of them had a research strategy and a resulting operations strategy (and operating model) that wasn’t happenstance. That’s a huge problem. If you’ve been able to deliver executive-level value without purposefully defining what you’ll do—and not do—(your strategies), and designed a model for how your organization should operate to achieve those goals (your operating model), you’ve been lucky, not smart.

So what does smart look like?

An Intentional Paradigm Shift: Building Vertical Value Into Your Operating Model

Sometimes the only way to grow a research team is through distributed investment, and this type of growth isn’t something to avoid. But if you’re a research leader, rather than continuing to grow a bigger and bigger research team that only delivers distributed value, you must, from day one, find ways to secure buy-in or find inventive ways to build vertical value into how you operate.

Vertical value delivers value vertically, as the name suggests—upwards through the hierarchical layers of the organization—ideally all the way up to the executive. While horizontal value often requires scaled-up operations and a lot of busyness to deliver value (lots of studies, logistics, stakeholders, and communications), in the world of vertical value, if you make the right choices, you need only deliver one to three perfectly aligned and articulated research studies or systems to be perceived as worth your weight in gold. Here, your operations aren’t focused on enabling quantity or pace. Instead, the goal is to become indispensable to senior leadership by consistently empowering them to hit the bullseye on million- or billion-dollar decisions, and, if possible, all the day-to-day decisions, too.

When it comes to approach, your inventiveness is the limit. But you’ll only succeed if you respond to the unique context you find yourself in. A paint-by-numbers, checklist approach simply won’t cut it—you’ll rarely find that kind of advice in my writing. And unless you work for an AI startup, this is not the era for grand visions that require significant startup investment. Instead, take a “LeanOps” approach and make the most of what you have. That said, let’s look at four key pointers for going vertical.

1. Align with Executive Priorities

As is likely clear by now, you must find ways to deliver research value that’s directly and unequivocally aligned with the executive’s priorities. Delivering “research value” needn’t mean delivering more research—it might, but that shouldn’t be the assumption. Instead, or in complement, you might provide a voice-of-customer report or access to a beautifully curated research library focused entirely on the executive’s priority: those new AI features for government customers, say. What you do is dictated by the needs and personalities of the people you’re trying to empower with knowledge. Again, the options are limited only by your ingenuity and budget.

I regularly hear a number of excuses for why this kind of alignment isn’t possible. To be blunt, most of it is procrastination. The most common things I hear: “I don’t have a seat at the table,” or “the executive hasn’t published a strategy, so I don’t know what their priorities are,” or “we can’t do this kind of work without additional funding, and getting the funding is hard.” Here’s how to handle these scenarios:

• I don’t have a seat at the table. If you regularly and artfully communicate your achievements in the language of the executive, doing this kind of work will likely eventually get you a seat at the table because you’re working in alignment with executive priorities—by definition, this is what they care about the most.

• The executive doesn’t have a strategy. You can find out the executive’s priorities by asking someone in finance where the executive is spending the most money. It’s a simple but highly effective hack.

• We don’t have the funding. If you align with a chief business priority and can offer a compelling story of how you can help, I guarantee you that ample funding will be made available to support the right efforts. You may not secure millions right off the bat, but you will certainly secure enough to deliver a minimum-viable example from which you can grow.

In the past and on multiple occasions, I’ve used these tactics to secure headcount, build specialist teams, deliver global research systems, and secure access to innovative research tools.

2. Repackage and Redistribute Insights

If you’re a research manager, I’ve got news for you: you’re not a research manager, you’re a research services manager—managing researchers is just one part of delivering a knowledge or insights service; it’s not the core purpose of the role. Core to your role is finding ways to create, repackage, and redistribute insights—the same insights generated through distributed investment—so they’re relevant to more senior levels of the organisation. If you’re not able to do this, perhaps because the initial insights are too shallow to repurpose for senior management, reconsider how research is done and whether you can build this kind of knowledge capture into your workflows without slowing delivery.

Research knowledge management is a huge topic, and thankfully, there are now excellent resources that you should devour. Here’s a short list of must reads:

• Research That Scales, Chapter 5, “Long Live Research Knowledge”

• Stop Wasting Research by Jake Burghard

• “The Systems Linguist: How Mapping Data, AI, and Language Builds Smarter ResearchOps” by Shivanjali Mishra is worth mentioning again

• “Pragmatic Knowledge Management: From Scattered Insights to Serendipitous Intelligence” by Lilyth Ester Grove

3. Apply the Prioritized-Access Principle to Democratization

It’s common for research leaders to attempt to deliver vertical value, or scale up access to insights, by democratizing research. In other words, by enabling designers, product managers, engineers, and others to do research themselves. In principle, this is usually a good move, but democratization efforts are typically based on a significant strategic mistake, making them far less effective than they could be. That mistake? For-profit organizations are not egalitarian: not everyone’s need for access is equal.

Your operating strategy, including your strategy for democratising research, must acknowledge that there are high-risk or high-priority stakeholders or topics, less influential people or topics, and those whose input or research topic needs are inconsequential. Though all of these groups might want access to the capabilities for doing research (or getting research done), in a profit-oriented world, enabling equal access to everyone is unwise and inefficient. It uses precious resources to deliver horizontally rather than vertically oriented value.

To use a concert-going analogy, too many research democratization efforts either aim to give everyone the all-access pass experience or to give everyone, including those working on high-priority projects, general-admission tickets. Key to a democratization strategy is ranking the access needs of various people or teams, the risk or importance of their work, and the research capabilities that are already available within those teams. Based on that information, you can design a system that provides all access to some, select viewing to others, and to everyone else, access to general admission or even an invitation to observe from the sidelines—or online. This advice isn’t about being elitist; it’s about being strategic with limited resources to demonstrate measurable value where it matters most to the business.

4. Regard ResearchOps as Strategic Partners, Not Administrators

If you approach building research as building a research capability (not just a research team), you’ll likely build a vastly more diverse team than you might have done in the past. As a research services manager, your team will likely include highly specialised researchers—the type who can do the high-stakes, strategic research that no one else in the organisation can do—as well as systems designers, librarians (AI means that librarians are more crucial than ever), communications and data specialists, and, yes, administrators to keep everything moving.

To design, build, and maintain these systems, you’ll need a senior ResearchOps specialist, even as a consultant, in the senior ranks (use The Universal ResearchOps Career Ladder as a guide) to partner with you and help design the operating systems that will make your research strategy an operable reality.

A small, well-designed ResearchOps team can enable hundreds of people to learn about customers while ensuring that insights flow to where they’re most valuable. That’s the value proposition that keeps many ResearchOps functions intact even when research teams are cut. But this only works if ResearchOps is working in the service of a deliberate research strategy, not simply in reaction to distributed, egalitarian demands.

Nostalgia Is Not a Strategy

At the 2026 World Economic Forum’s Davos event, Canadian Prime Minister Mark Carney said in his seminal speech about the changing world order in politics and finance that “nostalgia is not a strategy.” His words rang true on so many levels, and they ring true in the corporate world, too.

The companies hiring ResearchOps without researchers aren’t making a mistake about the value of operations. They’re making a calculated bet that they can get customer insights more cheaply and at greater scale without a dedicated research team. In some cases, they might be right. But they’re also making a bet that they can democratize research quality, maintain research rigour, and build insights traffic systems without the craft expertise, strategic thinking, and quality control that skilled researchers bring.

They can’t.

The real question is whether research leaders will recognize this moment for what it is: not a crisis to weather, but an opportunity to redesign how research operates from the ground up. To build research capabilities, not just teams of researchers.

The old model of distributed growth and horizontal value is gone. If you’ve been given headcount and you think you’re reorganizing things back to what they used to be, I encourage you to rethink your position. The current hire-and-fire habits show that companies value operational capability and small teams that deliver outsized value. Research leaders need to design operating models that deliver vertical value from day one, and partner with ResearchOps professionals to do so. ResearchOps professionals need research leaders who understand this shift so they can build sustainable systems that scale the value of research—they must also leave logistics management behind and become research systems designers. When both roles recognize their interdependence and operate accordingly, as a partnership, that’s when research (not the team, but the capability that enables curiosity and knowing) becomes indispensable. That’s a future worth building.

Sponsor and Credits

The ResearchOps Review is made possible thanks to Rally UXR—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack. Join the future of Research Operations. Your peers are already there.

Edited by and .

So I scanned LinkedIn to understand the key worries, such as “Will User Interviews become inaccessible for teams with a small budget?” and asked members of the Cha Cha Club, a members-only club I founded for ResearchOps professionals, about their key concerns. Then I sat down with Basel Fakhoury, the CEO and cofounder of User Interviews, and Baran Erkel, the chief strategy officer at UserTesting, as a guest host on the Insights Unlocked podcast, to ask them your questions.

This wasn’t a polished public relations moment, and I wasn’t paid for my time. Instead, it was a candid look at what this acquisition means, now and in the long term, for the people who use these tools.

If you’ve got more questions for Basel and Baran, please post them in the comments.

Things Mentioned in This Episode

• Research That Scales: The Research Operations Handbook by Kate Towsey

• ResearchOps 2.0, an audio documentary about the past, present, and future of ResearchOps

• Cha Cha Club, a members’ club for ResearchOps professionals

In November 2025, we published The Universal ResearchOps Career Ladder, an industry-defining asset that’s already become a go-to reference for ResearchOps professionals, hirers, and managers worldwide.

But what was it like for ResearchOps practitioners before this resource existed? Caitlin Faughnan, the Senior UX ResearchOps Specialist at GitLab, caught up with to discuss exactly that. She shared how misunderstandings about the role and scope of ResearchOps have impacted her career growth, and how she overcame them by advocating for a custom career ladder for ResearchOps in her organisation.

When Caitlin did this work, she didn't have an industry benchmark to refer to. Now, of course, we have The Universal ResearchOps Career Ladder to lead the way.

The Universal ResearchOps Career Ladder

The Universal Researchops Career Ladder

16.7MB ∙ PDF file

Download

Download it. Explore it. Print it (if you like).

Download

Credits

The Universal ResearchOps Career Ladder was produced by , with contributions from the following Cha Cha Club members: Wyatt Hayman, Saskia Liebenberg, Rodrigo Dalcin, Jared Forney, Lauren Galanter, Caitlin Faughan, Stephanie Marsh, Stephanie Kingston, Kalee Dankner, Leah Kandel, Jamie Williams, Jenna Lombardo, Christen Penny, Luana Cruz, Alma Krezla, Carolyn Morgan, Lydia Iana, and Rebecca Dennigan.

Brought to You By

User Interviews—the only solution you need to recruit high-quality participants for any kind of research.

Subscribe now

The ResearchOps Review is brought to you by Rally—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack.

About six years ago, I transitioned from academia to UX research. While in academia, I spent seven years studying linguistics, including a master’s in clinical linguistics and psycholinguistics. During that time, I learned about how languages are wired in our brains and how to assess and inform targeted interventions to treat language disorders.

Given my background, I think a lot about research systems and knowledge management—and working on a research repository in some capacity has been a consistent item on my to-do list for more than four years (almost as long as I’ve been involved in user research). In that time, I’ve set up research repositories in four very different tools (in terms of features and popularity), then reevaluated and offboarded those tools, and finally settled on one that seems to be sticking. And it’s sticking, not because of its features, but because of one key insight that changed everything.

That insight? Easy access to relevant and accurate user insights isn’t about finding the perfect software. It’s about embracing something more fundamental: A research repository isn’t just a tool; it’s a linguistic system.

If those two words—linguistic system—just muddled your mind, don’t worry. By the end of this article, you’ll know exactly what they mean and why they’re important. I posit that the modern ResearchOps professional must become a systems linguist: someone who can map an organization’s hidden data architecture, translate between different functional “languages,” and build truly integrated research systems. Given my formal training as a linguist, this article isn’t just a theoretical musing. It presents a practical framework that has transformed the way I approach every aspect of research operations, from tool selection and stakeholder engagement to preparing for an LLM-enabled future. And I believe this approach can transform how you manage research knowledge, too.

The Source of Meaning

First, let’s look at what makes a research repository unsticky. Even though a repository might have promising features, getting people to regularly use it—and the information stored in it—as a functional tool, can be challenging. On the surface, the challenge may seem like an onboarding problem, but it usually turns out to be deeper than that. I’ve learned that research knowledge can’t be treated as distinct from organisational knowledge and context; it must be seen as one and the same. Additionally, most companies aren’t leveraging the many sources of user voice they have available—social media mentions, support tickets, survey answers, interview transcriptions, and more—to support decision making. (More on this in a moment). These realizations fundamentally changed what I was building, and, therefore, the questions I asked. Instead of asking, “Which repository tool has the best features?” I asked:

• Who needs to find answers, and how do they naturally look for them?

• What questions are they asking, and in what language?

• Where does relevant data already live in the organization?

• How can we make these disparate sources speak to each other?

These aren’t tool-selection questions; they’re systems questions. And more than that, they’re linguistic questions. They’re about understanding how meaning flows through an organization, how different groups encode and decode information, and how we can build translation layers between them.

This mirrors a fundamental principle in linguistics: meaning emerges from systems, not isolated units. In other words, meaning is relational, not inherent. For example, in a medical context, the word “sick” means unwell (I am sick), but “sick” also could be used to communicate awesomeness (That painting is sick!), or to express frustration (I’m sick of this traffic!). A word doesn’t mean anything by itself; it means something in relation to other words, in a particular context, used by particular people for particular purposes. The same is true for organizational knowledge.

Diagnosing the Language Problem

The role of a clinical linguist involves observing how language breaks down. You use the tools of linguistics to dissect the unique pattern of communication, move from surface-level symptoms to a theoretical understanding of the underlying breakdown, and, in turn, drive precise and effective interventions.1

When I first became a ResearchOps professional, I couldn’t help but see similar patterns. I saw communication break down between teams due to incoherent processes and disconnected goals. Often, different teams were simultaneously and separately trying to solve the same problems, and research insights weren’t reaching the teams that needed them. Not just because of workflow issues, but because of siloed communication.

The real breakthrough came when I realized that teams weren’t just using different terms to describe similar meanings, they were operating with entirely different grammars to communicate about data, processes, and even value.2 And by grammars, I mean unwritten, foundational rules, structures, and logic. For instance:

• Product teams speak in the grammar of metrics, outcomes, and user flows. They need research insights that are tied to key performance indicators (KPIs) and roadmap decisions.

• Design teams speak in the grammar of experience, pain points, and journey maps. They need insights rich in context and emotional resonance.

• Engineering teams speak in the grammar of constraints, requirements, and edge cases. They need insights that are specific, actionable, and technically grounded.

• Leadership teams speak in the grammar of strategy, market positioning, and business value. They need insights that connect customers’ needs to a competitive advantage.

These grammars aren’t wrong per se; they just vary depending on the role of the insights consumer. But when a research finding is documented in only one grammar—say the grammar of a product manager—it becomes invisible to everyone else. This isn’t a training problem or a process problem, or even a tooling problem; it’s a translation problem.

This is where an understanding of linguistics comes in handy. Linguistics gives you frameworks for understanding how language works across three dimensions:

• Syntax (Structure). How elements are organized and related to each other.

• Semantics (Meaning). What those elements signify.

• Pragmatics (Use). How meaning changes based on context and who’s communicating.

You needn’t have a master’s degree in linguistics to leverage this understanding. You can start applying a linguist-informed approach by exploring the following questions across dimensions:

• Syntax. Consider all the sources of customer feedback your organisation collects, such as customer support tickets, feedback channels, and social media, and identify the patterns among them. Questions to ask:

  • How is data currently organized in your systems?

  • How are insights tagged, categorized, and linked to each other?

• Semantics. “Engagement is down,” can mean different things for different teams. To a product manager, it could mean a drop in activation, average session time or retention rate. For designers, it could signal a usability issue. For customer relationship management (CRM) and marketing teams, it could mean lower email open rates or fewer social media interactions. All of which leads to the next point. Questions to ask:

  • What does one common word or term mean to one group versus another?

  • What does ”customer satisfaction” or “usability issue,” for instance, mean to your colleagues in product versus design versus support?

• Pragmatics. If you’re presenting findings from discovery interviews to a product team, you might include concrete recommendations for features to add and pain points to address. If you were to present the same findings to leadership, you might want to focus on business value, market share, and long-term product strategy. Questions to ask:

  • How does the same finding need to be presented differently depending on who’s receiving it and the decision they’re making?

By exploring these questions, you’ll stop treating the “language problem” as a “tooling problem,” or as something that only happens between internal teams, and start seeing knowledge exchange (and research impact) as a systemic property of how knowledge moves through an organisation. And once you have a linguist’s eye for syntax, semantics, and pragmatics, you can apply it not just to how insights are communicated, but to how they are collected, stored, and surfaced in the first place. In other words, you’ll be able to map the hidden architecture of your organization’s language—the often invisible structure that shapes how customer realities either get amplified or lost in translation.

Mapping the Hidden Architecture: What Linguists See That Others Miss

As research professionals, customer language is our data. It flows in from every channel: support tickets, in-product feedback, survey verbatims, app store reviews, and social media mentions, to name a few. The goal of every ResearchOps professional should be to architect a knowledge system that doesn’t just count keywords but understands the underlying messages, concerns, and emotions of customers and end users within the constant stream of words—written and spoken.

Research (and the operations that make it tick) is about transforming fragmented, raw text and dialogue into structured, contextual insights that teams can act on, ensuring the customer’s true voice is never lost. But here’s what most organizations miss: organizational data is inherently variable. Just as sociolinguists understand that language varies systematically (not randomly) across demographics, regions, and contexts, a ResearchOps professional needs to understand that customer data varies systematically across sources, channels, and collection methods. For example:

• Customer support tickets typically capture problem-focused language. Support requestors are often stressed or frustrated, so the register is often transactional, the urgency is high, and the context is reactive.

• Survey responses capture reflective language and are collected in a structured environment. So the register is more formal, the urgency is lower, and the context is evaluative.

• Social media mentions use conversational language, often performative or community-oriented. The register is informal, the authenticity varies, and the context is public.

• User interviews use narrative language that’s cocreated with the researcher. The register adapts to the interviewer’s style, the depth is greater, and the context is exploratory.

Each of these sources provides a different “dialect” of customer truth. And just as a linguist would never claim that one dialect is “correct” while another is “incorrect,” a systems linguist (that’s you!) must recognize that each source provides legitimate but only partial insight. This variety of dialects needn’t be read as a hindrance, but as an opportunity to build knowledge systems that enable the pairing or triangulating of data sources to provide a more complete picture of the customer experience. More data isn’t always better—that’s not the lesson here—but linguistic diversity in your data ensures that you’re capturing the full range of how customers express their needs, frustrations, and desires.

Data diversity is especially important if you’re integrating AI into your knowledge stack. And who isn’t? Studies on language models3 make one thing clear: the variety baked into the training data―its words, grammar, and topics―sets a ceiling on how varied the model’s own language can be. Any model essentially mirrors the specific dialects and registers present in its corpus (a large, structured dataset), so voices that never make it into the data remain invisible in the model’s responses.

The AI Imperative: Why This Perspective Matters Now

It’s near cliché to say it, but the industry is consumed by talk of AI. To prepare your knowledge systems for the future—the not-so-far-away future—it’s important to see through the AI hype by understanding the mechanisms underlying LLMs. In the simplest terms, current AI offerings are Large Language Models (LLMs), and the solutions they provide are based on the data on which they’re trained.

So, what happens when we apply this understanding to AI-enabled research knowledge systems that span the entire organization? LLMs are fundamentally linguistic technologies: they learn patterns in language use and generate text based on those patterns. When you deploy an AI tool to help stakeholders find insights in your research repository, in a way, that tool is learning the language of your organization, or more accurately, the language of the context you’ve provided it with for that search. But here’s the challenge: if your organizational data is siloed, inconsistently structured, or dominated by only one type of source (say qualitative interview transcripts), the AI will learn a partial, biased language. It will excel at answering certain types of questions while remaining oblivious to others. It will perpetuate existing silos rather than break them down.

Research on LLMs makes this clear. A sociolinguistically informed approach to curating training data can improve the social impact of language models. Linguistic insight can inform the broader development and application of modern LLMs, including reinforcement learning—when LLMs learn by interacting with users through trial and error—and prompt engineering, all of which are ultimately grounded in patterns of language use, or data.

Translated to ResearchOps: your research repository is training data. Every time someone searches for insights, tags a finding, uploads a study, or links studies together, they’re teaching the system (and any AI tools built on top of it) what matters and how things connect. Without understanding the sociolinguistics of your organization—who speaks what language, what gets prioritized in which contexts, what voices are systematically excluded—AI tools will:

• Miss nuances that matter

• Perpetuate existing silos

• Reflect biases in dominant data sources

• Fail to serve teams whose dialect wasn’t well-represented in the training

This is why your role as a systems linguist isn’t just about organizing knowledge; it’s about curating the linguistic diversity of your organization’s learning system to ensure that when AI tools are implemented, they serve not just the loudest voices, but everyone who needs to understand user needs and accurately represent the user.

Building the System: Practical Applications

So what does all of this actually look like in practice? There are three key steps to take a systems linguistics approach to ResearchOps. First, you must partner with the product organization, then you must work with data teams, and, finally, you must close the feedback loop—a common callout when it comes to research operations.

1. Partner with the Product Organization

First, you’ll need to work closely with the product organization—even better if you partner up with ProductOps. Partner with these teams to create a shared vocabulary for what insights are needed, when they’re needed, and in what format. Remember that the goal of research isn’t only to produce research, it’s to build a common grammar for how research integrates into product decisions. ResearchOps is a team sport, and identifying knowledge gaps doesn’t have to (and shouldn’t) be a one-person job.

ProductOps professionals explicitly understand the operational cadence of product development—the rhythms of planning cycles, launch timelines, and metrics reviews—and they’re fluent in the language of the product organization. Do everything you can to leverage this, by:

• Cocreating taxonomies that make sense to both researchers and product teams

• Aligning research tagging systems with how product teams actually organize their work

• Building bridges between research findings and product briefs, OKRs, and roadmaps

You’ll want to work with data teams, too, to help you understand the existing data-related ecosystem driving your organization, and identify the right dots to connect.

2. Work with Data Teams

I don’t often hear ResearchOps folks talking about data teams as allies. Data (platform, analysis, and engineering) teams have a wealth of knowledge about what data is being collected, how it is collected, and how to access it. Data teams understand user behavior patterns, engagement signals, and the technical structure of how information flows through systems. There are so many opportunities for ResearchOps professionals to partner with data teams, I can’t list them all! So, I’ll stick to highlighting those most relevant to this article. Data teams can help you understand:

• Where data quality issues might introduce linguistic noise: elements in open-text data that create a hindrance in data analysis, such as typos, misspellings, filler words, or cynical and sarcastic remarks.

• How different data sources can be integrated without losing context.

• What metadata needs to be preserved for insights to remain meaningful.

• How to ensure that your data sources are diverse and representative of your customer base.

As mentioned earlier, the data you collect from different sources must be paired with metadata to triangulate it. When research and data analysis teams have access to enriched data, they can form more meaningful insights and models and support continuous learning.

Once you’ve defined a common language and understand the flow of information, it’s essential to close the loop to maintain both data quality and focus.

3. Create Feedback Loops

As AI systems become more integrated into research workflows, it will become essential for you to build validation mechanisms. A validation mechanism is a feedback loop that encourages researchers (or people who do research)—your linguistic experts—to check AI-generated summaries, suggested connections, and correct automated categorizations. The system needs to learn your organization’s language over time, which only happens if there’s a continuous cycle of:

1. AI suggesting patterns or connections

2. Humans validating or correcting those patterns and suggestions

3. The system learning from validation and correction

4. And so, accuracy improving over time

This is exactly how computational linguistics, an interdisciplinary field concerned with the computational modelling of natural language approaches and model refinement, can make your research repository continuously (and almost automatically) smarter and more effective.

The Interoperable Future: From Fragmented to Fluent

There’s something deeply compelling to me about the idea that research—in some form—can be done by anyone with a serious commitment to intellectual inquiry.

—Celine Nyungen, Designer and Writer

For research to be impactful, it must be understandable and relatable, which means it must be contextual, complete, diverse, and accessible. It also needs to feel approachable, like something any team member with a genuine curiosity and a serious commitment to intellectual inquiry can engage in. This is the promise of a systems linguistics approach: to build research systems that don’t just store information—but instead actively translate information across the different languages, or grammars, practised in your organization. When it works, the payoff is transformative across disciplines:

• Researchers find the information they need without the requirement of knowing exactly where it came from or what it was originally called. The system understands synonyms, related concepts, and contextual meaning.

• Product teams understand insights in their context, connected to metrics they care about, framed in terms of product decisions, and linked to relevant roadmap items.

• Engineers can trace decisions back to the data that informed them, understanding not just what users want but why, with enough specificity to inform technical implementation.

• Leadership sees the through-line from customer voice to strategy, understanding how insights connect to business outcomes and competitive positioning.

This represents a fundamental shift from reactive support to strategic architecture, from tool management to systems design, and from research gatekeeping to researchers functioning as organizational translators. The research repository stops being a place where research goes to live (or die), and, instead, becomes a living system—one requiring continuous care, maintenance, and translation across the linguistic communities in your organization.

Zooming out and viewing my ResearchOps role as a systems linguist’s role helped me translate research across researchers, data engineers, and product teams to build a more integrated, meaningful, and impactful research practice. You don’t need a degree in linguistics to adopt this perspective; you only need to develop linguistic awareness and uncover opportunities to integrate it.

The challenge facing ResearchOps isn’t whether we’ll evolve, it’s whether we’ll learn to speak the language of the systems we’re building. As AI becomes more and more embedded in organizations’ customer learning practices, the quality of that learning will depend entirely on the intentionality and sophistication of the underlying linguistic systems. Systems we’ll build.

Sponsor and Credits

The ResearchOps Review is made possible thanks to Rally UXR—scale research operations with Rally’s robust user research CRM, automated recruitment, and deep integrations into your existing research tech stack. Join the future of Research Operations. Your peers are already there.

Edited by and .

In November 2025, we published The Universal ResearchOps Career Ladder, an industry-defining asset that’s already become a go-to reference for ResearchOps professionals, hirers, and managers worldwide.

Jared Forney, the ResearchOps Principal at Okta, caught up with to talk about The Ladder, why he's envious of ResearchOps newbies who now have this industry-defining asset to guide their work, and why it's important. The punchline? It makes obvious the growing role of ResearchOps as strategists, dot connectors, and system designers—not just as administrators.

On the topic of ResearchOps as strategists and system designers, if you haven’t already, make sure to listen to ResearchOps 2.0, “EP #3: Taking a Platform Approach to ResearchOps.” It’s a deep dive into the fast-moving evolution of ResearchOps, which Jared refers to in this minipod.

The Universal ResearchOps Career Ladder

The Universal Researchops Career Ladder

16.7MB ∙ PDF file

Download

Download it. Explore it. Print it (if you like).

Download

Credits

The Universal ResearchOps Career Ladder was produced by , with contributions from the following Cha Cha Club members: Wyatt Hayman, Saskia Liebenberg, Rodrigo Dalcin, Jared Forney, Lauren Galanter, Caitlin Faughan, Stephanie Marsh, Stephanie Kingston, Kalee Dankner, Leah Kandel, Jamie Williams, Jenna Lombardo, Christen Penny, Luana Cruz, Alma Krezla, Carolyn Morgan, Lydia Iana, and Rebecca Dennigan.

Brought to You By

User Interviews—the only solution you need to recruit high-quality participants for any kind of research.

In November 2025, we published The Universal ResearchOps Career Ladder, an industry-defining asset that’s already become a go-to reference for ResearchOps professionals, hirers, and managers worldwide.

One might assume that a career ladder, or a career progression framework, is purely useful for writing accurate job descriptions, hiring the right people, providing a clear pathway for professional growth, advocating for promotions, and managing performance, which includes, dare I say, firing underperformers. While that’s a very good list, there’s one more way The Universal ResearchOps Career Ladder, in particular, can help you.

In this seven-minute podcast, Rodrigo Dalcin, Staff User Experience ResearchOps at Wealthsimple, shares why the Ladder is also an invaluable tool for assessing the maturity of research and ResearchOps in your organisation and pinpointing where growth opportunities lie.

The Universal ResearchOps Career Ladder

The Universal ResearchOps Career Ladder

16.7MB ∙ PDF file

Download

Download it. Explore it. Print it (if you like).

Download

Credits

The Universal ResearchOps Career Ladder was produced by , with contributions from the following Cha Cha Club members: Wyatt Hayman, Saskia Liebenberg, Rodrigo Dalcin, Jared Forney, Lauren Galanter, Caitlin Faughan, Stephanie Marsh, Stephanie Kingston, Kalee Dankner, Leah Kandel, Jamie Williams, Jenna Lombardo, Christen Penny, Luana Cruz, Alma Krezla, Carolyn Morgan, Lydia Iana, and Rebecca Dennigan.

Brought to You By

User Interviews—the only solution you need to recruit high-quality participants for any kind of research.