Introducing the Census Bureau’s Household Pulse Survey
The Household Pulse Survey is an ongoing federal survey that provides near real-time information about the impact of the COVID-19 pandemic on health, social and economic factors in Kansas and across the United States. This valuable resource allows for a deeper, data-driven understanding of the impact that the pandemic is having on our communities. The Kansas Health Institute (KHI) will report periodically on important impacts of the COVID-19 pandemic on Kansas. The first two issues of Pulse on Kansas are being released simultaneously with this overview of the survey methodology:
- No. 1: Mental Health During the COVID-19 Pandemic
- No. 2: Uninsured Rate During the COVID-19 Pandemic
Expect to see roughly one Pulse on Kansas per month throughout 2021.
Background
The Household Pulse Survey questionnaire was constructed using validated questions from other long-standing surveys from five federal agencies. Responses were gathered through an online questionnaire sent by text message or email. The survey is designed to provide reliable week-to-week estimates on employment status, income, food security, housing, general and mental health, access to health care and education disruption during the pandemic.
In contrast to most existing federal surveys that provide annual results with a lag of one to two years, the Household Pulse Survey aims to capture rapid week-to-week changes and make data readily available to inform policy makers and the public. The Household Pulse Survey is experimental, and the data quality is being closely monitored and assessed by federal agencies. Phase 1 of the Household Pulse Survey was issued weekly from April 23, 2020, through July 21, 2020.
Phase 2 of the Household Pulse Survey lasted from August 19, 2020, through October 31, 2020. In Phase 2 the cycle expanded from a one-week period to a two-week period. Phase 2 included many of the same questions as Phase 1 with the addition of receipt of federal benefits, spending patterns, financial resources, post-secondary education disruptions, capacity to telework and travel practices.
The Census Bureau approved the collection of the Household Pulse Survey in Phase 3 through 10/31/2023. The Phase 3 questionnaire is the same as Phase 2 through the end of 2020 (12/21/2020). Content is being reviewed and changes will be considered in early 2021.
Survey Administration
The Household Pulse Survey was designed to detect changes over time since the survey was implemented. While the pandemic began at varying times across the country, all states had already been impacted by the pandemic by the time the Household Pulse Survey was started. Thus, we can detect the direction of change since the survey began on the measures included as well as evaluate the effects of subsequent policies or actions during the pandemic, but not the impact of the policies or practices that were enacted to slow the spread of the virus before May 2020.
The Household Pulse Survey was administered online to households that had both an email address and phone number on file with the Census Bureau. Asking respondents who had both a phone number and email address on file to fill out a survey online is different from other federal surveys which usually involve phone calls to random households. This creates a potential limitation of the data since administering the survey online may have limited who was able to respond, and therefore affected the representativeness of the data to the population as a whole. For example, we know that in Kansas many rural communities lack ready access to the internet and thus people living in those communities may not have been as likely as others to complete the survey.
Sample Size
The weekly sample in Phase 1, or bi-weekly sample in Phases 2 and 3, was chosen at random from the Census Master Address List and appropriately represents each state or city in the survey. Phase 1 Household Pulse Survey respondents were included in the survey sample for up to three weeks in a row to track changes over a relatively short period of time. Starting in Phase 2, each household unit was interviewed once so that there is an independent sample for each bi-weekly estimate.
Sample sizes in the Household Pulse Survey were determined such that a two-percentage point difference in weekly estimates for the population would be detectable with a 90 percent confidence interval within each sampling area (e.g., the state or the U.S.) in Phase 1. Starting in Phase 2 the sample size was decreased, increasing the detectable difference to three-percentage points for the bi-weekly estimates. The overall sample sizes within the sampling areas were adjusted for an anticipated response rate of five percent in phase one and nine percent in phase two.
Response Rate
The Household Pulse survey adjusts for people in the sample who do not respond by deriving estimates from weighted responses, similar to what is done in other representative surveys. Weighting the responses allows for a relatively small number of respondents to represent the whole population, as well as the subpopulations within an area, and ensures we can utilize survey data to produce estimates that align with the known population totals. However, if the response rates are lower than expected, or certain populations are disproportionately underrepresented, the weighting must account for more of the estimate and the results may be more sensitive to error. Thereby increasing the range or confidence interval that we believe the true estimate to be in.
A survey response rate that is lower than the sample target of 5 percent in Phase 1 and 9 percent starting in phase 2 requires additional caution when interpreting results.
Validation Methods
While the Household Pulse Survey methodology is sound, the relatively low response rates may change the estimate week to week based on who responds, especially in areas with a relatively small population like Kansas. In Pulse on Kansas, we use two methods for validating the data depending on the source of the Household Pulse Survey question.
First, for survey questions from other national surveys that cannot report data for Kansas alone due to a small sample size, we compare the Kansas estimates to estimates for a larger sample size, like the U.S. overall. Even though the response rates for Kansas and the U.S. are similar, many more people are responding at the national level minimizing the weekly fluctuations from changes in the response rates.
Second, for Household Pulse Survey questions from other federal surveys that have reported data for Kansas, we can compare estimates between the surveys. The comparison is not to determine the magnitude of the change since before the pandemic, but rather to check that the Household Pulse Survey estimates are within a reasonable range.
Both methods assume that the U.S. and Kansas have similar trends, or that historical patterns have not changed. Kansans, like people living in all other states, are subject to the financial uncertainty, changing work environment and disrupted schooling that has been caused by the pandemic. Although some findings may be unique to Kansas or deviate from historical patterns found in other surveys, caution is needed when interpreting results with consideration of the confidence interval of the estimate, as well as additional evidence that the impact of the pandemic in Kansas may be different than other states.
Conclusion
The Household Pulse Survey can be a reliable tool for understanding the effects of the COVID-19 pandemic in a timely fashion. We provide two methods for validating the data presented in the Pulse on series: comparing the results for Kansas to the national results; and confirming that the results for Kansas are consistent with historical patterns or trends. As COVID-19 continues to disrupt our lives, these data allow for a deeper understanding of the current impact of the pandemic on our communities.
