We write down the analysis before we look at the data
The questions, the measures, and the tests are agreed with you and recorded before anyone examines a dataset. An analysis designed after the data has been seen will find something, and what it finds cannot be trusted.
An analysis designed after the data will always find something
A dataset arrives. An analyst explores it, notices a pattern, and builds the analysis around what they noticed. The finding is reported with a measure of confidence calculated as though the question had been asked in advance. It was not.
This is not fraud and it rarely feels like a choice. Exploring data and forming hypotheses from it is what analysts are trained to do. The problem is reporting the result as though the hypothesis came first.
Any sufficiently large dataset contains apparent patterns. Cut it enough ways and something will look significant. Without a record of what was planned, there is no way for a reader to know how many ways it was cut.
So the plan is written first. Questions, measures, comparison groups, and how the result will be interpreted, agreed with you and recorded before the data is opened. Anything discovered afterwards is reported separately and labelled as exploratory.
Cut a large dataset enough ways and something will look significant. Without a record of the plan, nobody can tell how many ways it was cut.
Confirmatory and exploratory kept apart
Findings that answer a pre-recorded question are reported as confirmatory. Everything else is reported as exploratory and described as a hypothesis for future work, not a conclusion.
Both appear in the report. Only one carries the weight of a planned test.
Null results are delivered, not buried
Where a pre-recorded question produces no effect, that is the finding and it is reported with the same prominence as a positive one.
A programme that only ever reports positive findings is not measuring anything.
Research and analysis
Research Programme Design
Study design, sampling, and the written analysis plan agreed before any data is collected or examined.
Survey and Field Research
Instrument design and fieldwork across English and Spanish-speaking populations on both sides of the border.
Programme Evaluation
Assessment of whether an intervention worked, including when the honest answer is that it did not.
Statistical Analysis
Analysis executed against the recorded plan, with any departure from it documented and explained.
Data Quality Assessment
An honest account of whether existing records can support the question being asked of them.
Reporting and Briefing
Written findings and in-person briefing for boards and funders, in language they can act on.
What this costs us
We deliver findings nobody wanted. A pre-recorded question producing a null result is reported as a null result, including where a funder was hoping for something else.
Our reports contain fewer headline findings. Separating confirmatory from exploratory means most of what an analysis turns up is labelled as provisional rather than presented as a conclusion.
Planning takes time before anything is measured. The design phase is real work, it is quoted, and a competing proposal that begins collecting immediately will look faster and cheaper.
Our evaluation found no effect. They reported it plainly and helped us explain it to the funder rather than looking for a subgroup that worked.Programme Director · Community health organization
The analysis plan was signed before fieldwork started. When the results arrived, nobody could argue the questions had been chosen to suit them.Research Lead · Regional foundation