How to tell if a fix actually reduced restaurant complaints

A done task is not a fixed problem

There are two different questions, and most teams collapse them into one:

• Did the work happen? A manager retrained the drive-thru team, recalibrated the fryer timer, added a bagging check. That is proof of work. In Replio it is verified by an owner or leader, never by the person who did it.
• Did guests stop writing about it? That is proof of improvement, and only later reviews can answer it.

A checklist that says handled answers the first question. It says nothing about the second. Keep both open until the reviews come in.

The measurement rules

These are the inclusion rules and the test Replio runs on every completed and verified fix. They come from our measurement specification, not from a marketing page, and you can apply them by hand.

1. One category, one store. Measure the complaint rate in the category the fix targeted (speed of service, order accuracy, food quality, hospitality and their peers) at the store where the work happened. Never the star average, never the whole group.
2. Equal windows. Before = the days leading up to when the fix was issued. After = the days starting when it was completed. Windows are 30, 60 or 90 days, and while the after window is still filling, both sides are shortened to the same length so the comparison stays like for like.
3. Wait at least 7 days after completion. Earlier than that, the after window is noise.
4. At least 5 reviews in each window. A rate over fewer than 5 reviews is not a rate.
5. At least 3 complaints in the before window. You cannot measure the reduction of a problem that was barely there.
6. Date reviews by when the guest wrote them, not when they were imported. A backfill of old reviews otherwise lands as a one day spike.
7. Run a two-sided Fisher exact test on complaints versus other reviews, before versus after, at p below 0.05. Report the rates with Wilson score intervals rather than the normal interval, which claims false certainty at zero complaints.

Replio also skips windows that start before its review history does, and fixes issued on the day an account went live. If any of rules 3 to 5 fails, the answer is insufficient evidence, and no p-value is computed. Inventing one for a comparison the data cannot support is the exact failure this method exists to prevent.

A worked example where the easy math is wrong

Suppose a store has 5 hospitality complaints in 18 reviews before a fix and 1 in 22 after. These are illustrative numbers chosen to show the math, not a customer result.

The common two-proportion z-test agrees: z = 2.05, which clears the usual bar. But at counts this small the normal approximation behind that test does not hold. The Fisher exact test on the same table gives p = 0.073. Not significant.

So the honest label is no meaningful change: measured, and too close to call. That is not a failed fix. It means you need more reviews before you know, and it is the reason Replio uses the exact test. The z-test would have called this a win.

Label what the numbers did

Every result gets one of these labels. Each describes the numbers, not the cause.

• Awaiting evidence: the work is done and the after window is still filling.
• Observed improvement: complaints in this category dropped by more than chance explains.
• Observed decline: complaints went up after the fix. Worth another look.
• No meaningful change: measured, and too close to call either way. Not the same as a failed fix.
• Insufficient evidence: not enough reviews came in to measure this fairly.

Keep no meaningful change and insufficient evidence apart. Collapsing them is how a tool talks a Business Owner out of a fix that is working.

What this cannot tell you

A before and after comparison at one store cannot separate your fix from everything else that changed in the same weeks: the season, a new shift lead, a menu change, a rush on a delivery app. Complaint rates across a whole group can swing for months with no fix at all.

Replio does not adjust for that today. We looked at control groups, matched stores and difference in differences, and none of them can be built honestly from the data Replio has today, so we do not show a number that pretends otherwise. An observed improvement is operational evidence, not a controlled experiment. The full reasoning is on our methodology page.

Questions to ask any tool that claims to measure fixes

Whether you use Replio, another platform or a spreadsheet, these questions separate measurement from a chart that moves:

1. Does it measure one complaint category at one store, or only an overall rating or sentiment score?
2. Are the before and after windows the same length?
3. What is the minimum number of reviews before it gives a verdict, and what does it say below that?
4. Which statistical test does it use, and is it valid at small counts?
5. Does it keep "too close to call" separate from "not enough data"?
6. Who verifies the work was done, and can it be the same person who did it?
7. Does it ever say the fix caused the change?

Replio answers these with the rules on this page. It reads Google and Yelp reviews plus DoorDash and Uber Eats feedback from merchant reports, groups repeated complaints into one issue, assigns it to a named person, and measures the next reviews. Nothing posts to a review site until a person on your team approves it. It is $149 per location per month, billed monthly, with self-serve checkout up to 10 locations. If you have not narrowed the problem yet, start with how to find recurring complaints in customer reviews and how to assign guest complaints to store managers.

$149/mo per location. Whole team included.