As the coronavirus pandemic continues to alter our way of life, more than ever we need valid and reliable data to support decision-making at every level of society. When used responsibly, data analysis helps our country’s leaders determine what policies to implement and can even guide our individual actions.
Donald Rumsfeld eloquently said there are known knowns, known unknowns, and unknown unknowns. The pandemic highlights all three categories. Unfortunately, what we don’t know today about how the coronavirus is impacting the American people – the known unknowns -- is vast.
We all have questions about the virus, its implications, and its effects on our neighbors, our friends, and our families. While there are some questions that can’t be definitively answered today, believe it or not, there is much that we should be able to answer with good research if we start now.
When we consider the collective impacts of coronavirus, we will all be searching for ways of understanding how our incomes changed relative to just weeks ago, how our anxiety increased in the midst of health uncertainty, and even how our food security changed as result of ongoing uncertainty. The natural and reasonable tendency is to compare your own experience to a recent and familiar benchmark. This might be appropriate for individuals, but we have to do better when looking across our entire population.
The data and research communities are rapidly filling identified information needs for decision-makers, ensuring that we can operate on more than mere anecdote. As additional information is published and released in coming weeks, we must approach the insights with humility, recognizing limitations to even the most well-conducted studies.
The Census Bureau’s new Household Pulse Survey, which is a collaborative effort with other Federal Statistical Agencies, is to be applauded. We look forward to the information it will provide to fill in the gaps in knowledge in the months ahead. It will survey one million households per week by email and ask a range of questions about economic conditions, housing stability, and food security. But the Census Bureau’s survey will rely on a new email outreach strategy. The lack of knowledge for how representative this email approach will be, poses challenges for interpreting the validity and reliability of the information.
Facebook’s new tracking survey in partnership with university researchers asks health surveillance questions of the platform’s users. This approach has limits to how generalizable results will be for the American population based on the Facebook community demographics.
There are also a number of nationally-representative opinion surveys currently in the field, which aim to provide a sense of collective mood and direction. Opinion surveys are less useful for regional officials trying to make localized decisions based on the intensity of the outbreak in their states and localities.
Several universities launched promising health surveillance surveys distributed on social media and by email, generating responses from those who stumble across the website. These types of surveys may have the ability to produce regional insights, but also have unknown biases since we cannot be sure that those responding are representative of the region’s population. This means the insights could even be wrong or misleading.
Multiple projects are now underway – and that’s a good thing. We don’t aim to be alarmist, but instead aim to raise awareness and educate about the limits and biases in research that can affect how results will be used. We need to be honest and forthcoming about those limits as researchers and decision-makers sift through the findings.
One especially promising effort, which we are both affiliated with, has emerged to address existing limitations in understanding impacts from the pandemic. It is both a government and philanthropic effort called the COVID Impact Survey. This survey randomly selects individuals for participation across the country based on their addresses. From a research perspective this ensures the results are broadly representative of the population. It also seeks to rapidly provide underlying data for public health and economic modeling as our country’s decision-makers are working to determine next steps for responding to the pandemic. Importantly, because this survey is based on a random sample not one drawn from convenience, it is also a potential vehicle for other surveys to adjust calculations to reduce bias and provide decision-makers with increasingly more accurate information. In the longer term, projects like the COVID Impact Survey will be invaluable to future planning and modeling.
We need more projects like this one. We also need to bring together the leaders of these projects so that the results from these studies can be displayed in a comprehensive and understandable manner. By building on the strengths of each project, more usable information will help inform decisions. This is the time for the private and public sector, along with the research community, to come together to reduce the known unknowns.
Now is not the time to dither or shirk in our collective responsibility to ensure decision-makers have good data to make informed decisions. Evidence-based policymaking is needed now, perhaps more than ever. But we cannot have useful evidence without good data. The COVID Impact Survey will be a productive resource for determining which other projects truly have good data.
Nick Hart is President of the Data Foundation, which is leading the COVID Impact Survey. More details are available at covid-impact.org. Charlie Rothwell is a former director of the National Center for Health Statistics at the Centers for Disease Control and Prevention. He is a project advisor to the COVID Impact Survey.