How do I explain the limits of a comparison when the options were assessed using different evidence?
Quick answer
State that the options were assessed using evidence that differs in quality, coverage, or recency, then name the differences that matter. Explain how they limit a fair side-by-side judgment, distinguish observed evidence from assumptions, and avoid presenting an uncertain ranking as precise. End by saying what the comparison can support and what additional, comparable evidence would help resolve the uncertainty.
The question, the details, the next step.
I’m preparing an options paper, but the evidence isn’t like for like. One option has a recent, detailed assessment; another has older, partial information. How can I explain that without making the comparison useless?
Make the limitation visible, then define the comparison’s useful scope. You can compare what the available evidence supports, but avoid implying that every option was measured on the same basis.
A practical caveat has four parts:
- Name the difference: for example, evidence varies in recency and coverage.
- Explain its effect: say which aspects of the comparison may be less reliable as a result.
- Set a boundary: state what the paper can and cannot conclude.
- Identify a next step: say what evidence would make the assessment more comparable, if that information is needed.
For example: “The options are compared using evidence of different ages and coverage. Option A has a recent assessment of all three criteria; information for Option B is older and does not cover one criterion. The comparison indicates where the available evidence points, but does not support a confident overall ranking. A current assessment of Option B against the same criteria would help clarify the relative position.”
Treat those details as placeholders: use them only if they accurately describe your evidence.
How do I describe the evidence itself so readers can see why the caveat matters, rather than just seeing a vague warning?
Add a short evidence-quality note next to the comparison, using the same dimensions for each option. For each relevant claim, record its source or basis, date or period, coverage, and any important gap. Keep this descriptive: don’t invent a numerical quality score unless your organization has a defined method for one.
A compact note could read:
“Evidence basis: Option A—[source or assessment type], [date or period], covers [criteria]. Option B—[source or assessment type], [date or period], covers [criteria]; [criterion] is not covered. Because the evidence differs in recency and coverage, findings are not directly equivalent.”
If you have many claims, put the detail in a small evidence note or appendix and summarize the most decision-relevant difference in the main text. Distinguish documented findings from assumptions or estimates, and label each plainly. A reader should be able to tell whether a blank means “not assessed,” “not available,” or “not applicable”—those are different situations.
The paper still needs to help people make a decision. How can I present a conclusion without pretending the evidence is stronger than it is?
Make the conclusion conditional and proportionate. Say what the available information suggests, name the main uncertainty, and explain whether that uncertainty could change the decision. Avoid false precision such as a close numerical ranking when the underlying evidence is uneven.
For instance: “On the evidence currently available, Option A appears better supported on [criterion]. This is not a firm overall ranking: evidence for Option B is older and does not address [criterion]. If the decision depends on that criterion, obtain comparable information before treating the options as settled.”
If a decision must proceed before the gap is resolved, state that the choice is being made with the stated limitation, and identify who will revisit it or what information would prompt a review. Don’t imply that more evidence will necessarily favor either option.
Can you give me a reusable wording template that covers both the caveat and the note without becoming a long disclaimer?
Use this short structure and replace every bracket with verified details:
Comparison caveat: “The options are assessed using evidence that differs in [recency, coverage, or other relevant respect]. [Option A] is supported by [evidence basis and period]; [Option B] is supported by [evidence basis and period]. This limits [the specific comparison or conclusion]. The available evidence supports [what can be said], but does not establish [what cannot be concluded]. [Comparable information or follow-up] would help address this gap.”
Evidence-quality note: “For [option], the evidence [covers/does not cover] [criteria] and is from [period]. [Finding] is [documented / an estimate / an assumption].”
Keep the main caveat close to the comparison, not buried in a general limitations section. Check that the note names a real, decision-relevant difference and that the conclusion does not quietly overstate what the caveat says.
A practical checklist
Name the specific evidence differences that affect this comparison.
For each option, note the evidence basis, date or period, and coverage.
Label missing, unavailable, and not-applicable information distinctly.
Separate documented findings from estimates and assumptions.
State what the comparison supports and what it cannot establish.
Identify comparable follow-up evidence if it could affect the decision.
A few more useful answers
Should I remove an option if its evidence is weaker?
Not automatically. Explain the evidence gap and its effect; exclude an option only if the paper’s criteria or decision process gives a reason to do so.
Where should the caveat go?
Put a concise caveat beside the comparison or conclusion it qualifies. Use an appendix for fuller evidence details, while keeping the key limitation visible in the main paper.
Can I still recommend one option?
Yes, if you state the basis and limits of the recommendation. Make clear whether it is conditional on current evidence and whether a specific gap could change it.