Do low survey response rates bias results? Evidence from Japan

Ronald R. Rindfuss, Minja K. Choe, Noriko Tsuya, Larry L. Bumpass, Emi Tamaki

Research output: Contribution to journalArticle

42 Citations (Scopus)

Abstract

Background: In developed countries, response rates have dropped to such low levels that many in the population field question whether the data can provide unbiased results. Objective: The paper uses three Japanese surveys conducted in the 2000s to ask whether low survey response rates bias results. A secondary objective is to bring results reported in the survey response literature to the attention of the demographic research community. Methods: Using a longitudinal survey as well as paradata from a cross-sectional survey, a variety of statistical techniques (chi square, analysis of variance (ANOVA), logistic regression, ordered probit or ordinary least squares regression (OLS), as appropriate) are used to examine response-rate bias. Results: Evidence of response-rate bias is found for the univariate distributions of some demographic characteristics, behaviors, and attitudinal items. But when examining relationships between variables in a multivariate analysis, controlling for a variety of background variables, for most dependent variables we do not find evidence of bias from low response rates. Conclusions: Our results are consistent with results reported in the econometric and survey research literatures. Low response rates need not necessarily lead to biased results. Bias is more likely to be present when examining a simple univariate distribution than when examining the relationship between variables in a multivariate model. COMMENTS The results have two implications. First, demographers should not presume the presence or absence of low response-rate bias; rather they should test for it in the context of a specific substantive analysis. Second, demographers should lobby data gatherers to collect as much paradata as possible so that rigorous tests for low response-rate bias are possible.

Original languageEnglish
Pages (from-to)797-828
Number of pages32
JournalDemographic Research
Volume32
Issue number1
DOIs
Publication statusPublished - 2015

Fingerprint

Japan
trend
evidence
regression
lobby
survey research
analysis of variance
multivariate analysis
econometrics
logistics
community
literature

ASJC Scopus subject areas

  • Demography

Cite this

Do low survey response rates bias results? Evidence from Japan. / Rindfuss, Ronald R.; Choe, Minja K.; Tsuya, Noriko; Bumpass, Larry L.; Tamaki, Emi.

In: Demographic Research, Vol. 32, No. 1, 2015, p. 797-828.

Research output: Contribution to journalArticle

Rindfuss, RR, Choe, MK, Tsuya, N, Bumpass, LL & Tamaki, E 2015, 'Do low survey response rates bias results? Evidence from Japan', Demographic Research, vol. 32, no. 1, pp. 797-828. https://doi.org/10.4054/DemRes.2015.32.26
Rindfuss, Ronald R. ; Choe, Minja K. ; Tsuya, Noriko ; Bumpass, Larry L. ; Tamaki, Emi. / Do low survey response rates bias results? Evidence from Japan. In: Demographic Research. 2015 ; Vol. 32, No. 1. pp. 797-828.
@article{02234b2bf9bc4f41be4943579af66f5e,
title = "Do low survey response rates bias results? Evidence from Japan",
abstract = "Background: In developed countries, response rates have dropped to such low levels that many in the population field question whether the data can provide unbiased results. Objective: The paper uses three Japanese surveys conducted in the 2000s to ask whether low survey response rates bias results. A secondary objective is to bring results reported in the survey response literature to the attention of the demographic research community. Methods: Using a longitudinal survey as well as paradata from a cross-sectional survey, a variety of statistical techniques (chi square, analysis of variance (ANOVA), logistic regression, ordered probit or ordinary least squares regression (OLS), as appropriate) are used to examine response-rate bias. Results: Evidence of response-rate bias is found for the univariate distributions of some demographic characteristics, behaviors, and attitudinal items. But when examining relationships between variables in a multivariate analysis, controlling for a variety of background variables, for most dependent variables we do not find evidence of bias from low response rates. Conclusions: Our results are consistent with results reported in the econometric and survey research literatures. Low response rates need not necessarily lead to biased results. Bias is more likely to be present when examining a simple univariate distribution than when examining the relationship between variables in a multivariate model. COMMENTS The results have two implications. First, demographers should not presume the presence or absence of low response-rate bias; rather they should test for it in the context of a specific substantive analysis. Second, demographers should lobby data gatherers to collect as much paradata as possible so that rigorous tests for low response-rate bias are possible.",
author = "Rindfuss, {Ronald R.} and Choe, {Minja K.} and Noriko Tsuya and Bumpass, {Larry L.} and Emi Tamaki",
year = "2015",
doi = "10.4054/DemRes.2015.32.26",
language = "English",
volume = "32",
pages = "797--828",
journal = "Demographic Research",
issn = "1435-9871",
publisher = "Max-Planck Institute for Demographic Research/Max-Planck-institut fur Demografische Forschung",
number = "1",

}

TY - JOUR

T1 - Do low survey response rates bias results? Evidence from Japan

AU - Rindfuss, Ronald R.

AU - Choe, Minja K.

AU - Tsuya, Noriko

AU - Bumpass, Larry L.

AU - Tamaki, Emi

PY - 2015

Y1 - 2015

N2 - Background: In developed countries, response rates have dropped to such low levels that many in the population field question whether the data can provide unbiased results. Objective: The paper uses three Japanese surveys conducted in the 2000s to ask whether low survey response rates bias results. A secondary objective is to bring results reported in the survey response literature to the attention of the demographic research community. Methods: Using a longitudinal survey as well as paradata from a cross-sectional survey, a variety of statistical techniques (chi square, analysis of variance (ANOVA), logistic regression, ordered probit or ordinary least squares regression (OLS), as appropriate) are used to examine response-rate bias. Results: Evidence of response-rate bias is found for the univariate distributions of some demographic characteristics, behaviors, and attitudinal items. But when examining relationships between variables in a multivariate analysis, controlling for a variety of background variables, for most dependent variables we do not find evidence of bias from low response rates. Conclusions: Our results are consistent with results reported in the econometric and survey research literatures. Low response rates need not necessarily lead to biased results. Bias is more likely to be present when examining a simple univariate distribution than when examining the relationship between variables in a multivariate model. COMMENTS The results have two implications. First, demographers should not presume the presence or absence of low response-rate bias; rather they should test for it in the context of a specific substantive analysis. Second, demographers should lobby data gatherers to collect as much paradata as possible so that rigorous tests for low response-rate bias are possible.

AB - Background: In developed countries, response rates have dropped to such low levels that many in the population field question whether the data can provide unbiased results. Objective: The paper uses three Japanese surveys conducted in the 2000s to ask whether low survey response rates bias results. A secondary objective is to bring results reported in the survey response literature to the attention of the demographic research community. Methods: Using a longitudinal survey as well as paradata from a cross-sectional survey, a variety of statistical techniques (chi square, analysis of variance (ANOVA), logistic regression, ordered probit or ordinary least squares regression (OLS), as appropriate) are used to examine response-rate bias. Results: Evidence of response-rate bias is found for the univariate distributions of some demographic characteristics, behaviors, and attitudinal items. But when examining relationships between variables in a multivariate analysis, controlling for a variety of background variables, for most dependent variables we do not find evidence of bias from low response rates. Conclusions: Our results are consistent with results reported in the econometric and survey research literatures. Low response rates need not necessarily lead to biased results. Bias is more likely to be present when examining a simple univariate distribution than when examining the relationship between variables in a multivariate model. COMMENTS The results have two implications. First, demographers should not presume the presence or absence of low response-rate bias; rather they should test for it in the context of a specific substantive analysis. Second, demographers should lobby data gatherers to collect as much paradata as possible so that rigorous tests for low response-rate bias are possible.

UR - http://www.scopus.com/inward/record.url?scp=84929405992&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84929405992&partnerID=8YFLogxK

U2 - 10.4054/DemRes.2015.32.26

DO - 10.4054/DemRes.2015.32.26

M3 - Article

AN - SCOPUS:84929405992

VL - 32

SP - 797

EP - 828

JO - Demographic Research

JF - Demographic Research

SN - 1435-9871

IS - 1

ER -