Broadstone Analytics
A research and analysis practice on South Desert Boulevard, working with organizations across West Texas and southern New Mexico.
Why the plan comes first
Broadstone Analytics was founded to run research programmes whose findings could be relied upon by the people funding them.
Our analysts had all produced work where the question was settled after the data had been examined. Nothing was falsified. An interesting pattern was noticed, an analysis was built around it, and the result was reported with statistics that assume the question came first.
The distortion is invisible in the finished report. A finding arrived at after twenty comparisons looks identical on the page to one arrived at after a single planned test, and the reader has no way to distinguish them.
Nobody in the chain is acting badly. Exploration is a legitimate and necessary part of analysis. The failure is in reporting exploratory findings as though they were confirmatory.
So we record the plan before the data is opened. Questions, measures, comparison groups, sample, and how a result will be interpreted, all agreed with the client and dated.
And we separate the two categories in every report. Confirmatory findings answer recorded questions. Exploratory findings are labelled as hypotheses for future testing, however interesting they look.
What every client can expect
Plan recorded first
Questions, measures, and interpretation agreed and dated before data is examined.
Two categories, kept apart
Confirmatory findings and exploratory findings reported separately and labelled.
Null results reported
With the same prominence as positive ones, including where a funder hoped otherwise.
Departures documented
Where we deviate from the plan, the change and the reason are recorded in the report.
Data handled properly
Personal data minimised, retention agreed in writing, and deletion confirmed at close.
Bilingual fieldwork
Instruments designed and tested in both English and Spanish rather than translated afterwards.
Working across two populations
A substantial share of our fieldwork covers communities on both sides of the border, and the methodological difficulties there are practical rather than theoretical.
Instruments cannot be translated at the end. A survey written in English and rendered into Spanish afterwards produces two instruments measuring subtly different things. We design and cognitively test in both languages, in parallel.
Comparable data frequently does not exist. Administrative records, census categories, and reporting periods differ on each side, and a comparison presented without stating that is precise about something untrue.
Sampling frames are uneven. Coverage that is adequate in one community may be poor in another, and a pooled result can conceal that one population is badly represented.
We state these limits in the report rather than in a methodological annexe nobody reads, because the limits determine how far a finding can be relied upon.
Who you will work with
Rhett Anderson
Research Programmes Lead
Designs research programmes and writes the analysis plan recorded before data collection begins.
Managing Director
Leadership
Founded the practice and reviews every analysis plan and every report before issue.
Head of Fieldwork
Fieldwork
Fourteen years in bilingual survey research across the border region.