What to Look For When Hiring a CRO Agency (10 Questions to Ask)

Every CRO agency pitch sounds similar. Data-driven, results-focused, structured experimentation, case studies with strong percentages.

What separates a rigorous partner from an agency that ships design changes and calls it optimisation isn't in the pitch. It's in how they answer specific questions about methodology.

Here are ten questions to ask before signing, what a good answer sounds like, and what should give you pause. Written for D2C brands on Shopify, with the Indian market's specific realities in mind.

1. Do you audit our tracking before testing?

Why it matters: CRO measures success against your conversion data. If your GA4 purchase event fires twice for a share of orders, your baseline conversion rate is inflated which corrupts every sample size calculation and produces tests that appear significant before they are. If begin_checkout isn't firing on mobile, your funnel shows a collapse that isn't real, and the entire test roadmap points at the wrong step.

Most agencies don't check. They assume the analytics are fine and start forming hypotheses from whatever the funnel report shows.

Good answer: They describe a validation step before the first hypothesis, reconciling GA4 against Shopify order data, checking for duplicate events, confirming funnel events fire across devices and payment paths.

Worrying answer: "We'll work with your existing setup." It means every result they produce rests on a foundation nobody has verified.

2. Walk me through your research process before test one launches.

Why it matters: This is the single most revealing question in an evaluation, because weak agencies answer it with tools and strong agencies answer it with a process.

Good answer: Named stages with specific deliverables, quantitative funnel analysis to locate where revenue leaks, qualitative research to understand why, then hypotheses developed from those findings.

Worrying answer: A tool list. "We use Hotjar, Clarity, and VWO." Any competent team can operate those. The tool isn't the capability.

3. How do you decide which test runs first?

Why it matters: Your traffic caps how many tests you can run. Sequencing determines how much value the programme produces in its first two quarters.

Good answer: A structured framework; ICE (Impact, Confidence, Ease) or PIE (Potential, Importance, Ease). The specific framework matters less than having one.

Worrying answer: Intuition, or "whatever looks most promising." That's preference, not prioritisation.

4. What confidence threshold do you use, and how do you calculate sample size?

Why it matters: Tests called early because one variant looks ahead on day six produce results that don't replicate in production. This is where most programmes quietly fail.

Good answer: They calculate required sample size before launch from your baseline conversion rate and minimum detectable effect, commit to a 95% threshold, and don't act until it's reached.

Worrying answer: Fixed calendar durations with no sample size calculation, or checking daily and stopping when significance appears.

5. What's your typical win rate?

Why it matters: Counterintuitively, a high win rate is a warning sign rather than a selling point.

Good answer: 20–30%. Two or three tests in ten produce significant positive results. That's what rigorous experimentation looks like.

Worrying answer: Above 50%. It usually means testing only safe changes that can't lose, peeking at results, or measuring against an inflated baseline. Ask to see a test log that includes the losers, case studies show winners, a full log shows methodology.

6. Based on our traffic, how many tests can we realistically run per quarter?

Why it matters: Test velocity is capped by conversion volume, not by ambition. A store with 20,000 monthly sessions at 1.5% conversion supports roughly three to four tests per quarter not eight per month.

Good answer: A calculation using your actual numbers, delivered honestly even when the answer is lower than you'd like.

Worrying answer: A number from a package tier. If the answer doesn't reference your conversion volume, they haven't done the arithmetic, and you'll be paying for testing capacity your traffic can't use.

7. How will you handle our COD and prepaid funnels differently?

Why it matters: This is the question that separates agencies who work in the Indian market from agencies who apply a global playbook here.

Cash on delivery still accounts for a substantial share of Indian D2C orders, and COD buyers behave differently at every funnel step different intent at checkout, different abandonment triggers, and materially different post-purchase outcomes. A checkout optimisation that increases COD conversion can raise reported conversion rate while increasing RTO, which makes the business worse.

Good answer: They discuss segmenting funnels by payment method, tracking payment_method as a custom dimension, and measuring against delivered orders rather than placed orders.

Worrying answer: Blank looks, or treating COD as a payment detail rather than a distinct buyer journey.

8. What metrics will you report, and how do they connect to revenue?

Why it matters: Click-through rate, scroll depth, and time on page have diagnostic value. They don't pay salaries.

Good answer: Conversion to purchase, revenue per visitor, average order value, and for Indian D2C, delivered order rate. Percentage lifts are always accompanied by absolute revenue figures.

Worrying answer: Conversion rate improvements reported without reference to absolute revenue, or revenue metrics treated as out of scope.

9. Who will actually be doing the work day to day?

Why it matters: There's a meaningful difference between an account manager relaying updates and a practitioner with hands-on testing experience running your programme.

Good answer: A named person with relevant experience, and a clear answer on how many other accounts they run. Ask to meet them during the sales process.

Worrying answer: Senior people in the pitch who don't appear again after signing.

10. What happens if you find tracking problems mid-engagement?

Why it matters: This reveals whether they'd pause and fix, or work around it and keep testing on data they know is compromised.

Good answer: Testing pauses. The tracking gets fixed. Prior results affected by the issue get flagged as unreliable rather than quietly retained in the reporting.

Worrying answer: "That's not really our scope." It tells you what happens when the programme stops producing results and nobody can explain why.

Quick Reference

One Check to Run Before You Evaluate Anyone

Pull your GA4 purchase event count and your Shopify order count for the same 30-day window. If they differ by more than 10%, your conversion data has a problem — and an agency that doesn't fix it will spend your first quarter optimising against a number that isn't real. Our guide to why GA4 and Shopify numbers don't match explains what each gap size means.

Disclosure: FunnelFreaks is a CRO agency, so score us against this list too. Question one sits first because it's the phase we built the agency around, we validate and rebuild analytics infrastructure before any hypothesis is written, since every decision after that point inherits whatever errors exist in the data. Ask us all ten and we'll answer without a deck.

If you're comparing options, our breakdown of CRO agencies for D2C brands in India covers who specialises in what, and our post on what a CRO agency actually does explains the work itself in more detail.

Want to run this checklist on us? Talk to FunnelFreaks.