The Crisis of Survey Data

February 20, 2019

A pillar of social science research in North America and Europe is representative survey data. This is particularly the case for questions regarding social policy, income distribution, and employment. However, the central surveys aimed at accomplishing these essential tasks are increasing biased. 


Meyer, Mok, and Sullivan (2015) show that there has been an increasing incidence of unit non-response (i.e. an unwillingness to respond) and item non-response (i.e. lack of response for particular questions). Figures 1 and 2 show this increasing incidence of item and unit non-response.  



When comparing aggregate government transfer income and recipiency data from these surveys to administrative data, it is clear that there is sizeable bias across surveys. Meyer, Mok, and Sullivan measure bias in two different manners: 'Dollar Bias' and 'Month Bias'. 


Meyer, Mok, and Sullivan (2015) provide evidence that the root of the bulk of this bias is not incorrectly reporting the length of time and income received from a government program, but rather unit non-response. Otherwise said, bias is primarily the result of survey respondents incorrectly reporting that they are not recipients of some state transfer, rather than incorrectly reporting the amount or length of time they receive from a state transfer.



Amongst all of the surveys, though, the Survey of Income and Program Participation seems to stand out as the least biased in terms of both 'Dollar Bias' and 'Months Bias'.



Please reload