Testimonials

“The SSPP is an invaluable tool for improving the quality of social science. Our team was interested in how much social scientists from different disciplines work on issues of race and ethnicity, and why such differences exist. We used the platform to test the notion that differences are based only on different priorities across disciplines. Instead, we identified that social scientists are unaware of the current level of research into race-related topics, overpredicting both how much work there is and the rise in such research. The SSPP made it possible for us to quickly and easily elicit beliefs from a niche but important audience of both producers and consumers of social science research. Given that our work was unfunded and produced quickly for a special session of the Econometric Society World Congress three months after the killing of George Floyd, we would simply not have been able to answer these questions without the SSPP.”

“We ran a 10-year follow up of a really successful anti-violence program. I’m glad we surveyed experts for their predictions ahead of time. Neither the results of the survey nor our program were what we expected, and the prediction survey really helped demonstrate the importance of the research findings”

“In the space of just a few years, the collection of expert predictions alongside an empirical study has become an important part of the social scientist's toolkit. But it quickly became apparent that a centralized platform is the optimal way to organize this work, and the Social Science Prediction Platform provides that service. I have used the platform for two projects so far. I've also participated as a predictor in a number of studies. Both as a researcher and as a participant I've found the platform to be user-friendly and powerful (e.g., by providing the ability to field multiple versions of a survey under the same project). The Platform team provided excellent support to help maximize data quality. I plan to use the Platform often in the future.”

“The SSPP is an amazing platform for those of us who are interested in predictions of scientific results. Everything is very smooth and easy to handle – once you have a survey design, uploading it to SSPP is trivial and data collection can be very fast with large samples of participants. The predictions we get from these projects allow us to infer beliefs or priors about the hypotheses tested in our studies, which can be very helpful in terms of understanding for example which results are likely to replicate or are reliable. The predictions also allow us to study when the “wisdom of crowds” works, and whether certain groups are better at predicting results. All of this is important when we want to understand for example which interventions are the most likely to have large effects if we are choosing between several potential ones, or which results are more surprising and should be replicated.”

“We used the SSPP to obtain expert predictions about which types of decision-making biases are likely to get filtered out in aggregate outcomes. The expert predictions provided a clear and surprising message: Forecasters, while highly attentive and internally consistent, didn’t have well-honed intuitions about which biases “survive” in markets. We had an incredibly positive experience with the team of SSPP and the platform itself. Qualtrics is seamlessly integrated with the platform; there are no frictions in the process of implementing a given survey on SSPP. The instructions on the webpage both about designing and about implementing a prediction survey were amazingly helpful, and the entire website is easy to navigate both as someone running the survey and as a user providing forecasts. The SSPP team was extremely helpful and responsive in figuring out all the details. In our view, the SSPP provides a much-needed tool to obtain predictions, which we think is increasingly important for research in economics and the social sciences more generally.”

“The Behavioural Economics Team of the Australian Government (BETA) is based in the Australian Department of Prime Minister and Cabinet. We see several potential benefits from using the SSPP in our future work. First, the SSPP could provide a valuable (and unique) source for gathering priors for use in frequentist analyses (null hypothesis significance testing) and especially for Bayesian analyses, which we are starting to add to our repertoire. Second, we frequently face situations where we have to provide advice on policy design, implementation, or delivery without the opportunity to conduct an impact evaluation. In these circumstances, it would be very helpful to gather accurate predictions about the likely impact (or not) of proposed interventions.”

“I have found eliciting the predictions of experts and policymakers about the effects of new policies to be useful in multiple projects. It has helped in documenting policymakers’ and academics’ ex-ante beliefs about the likely effects of interventions, and in showing where beliefs are especially heterogeneous. This has helped address the concern that it is easy for people to say “we never thought that would work” if they see a zero or small result. Even the process of deciding which outcomes to ask predictions for has been a useful conversation with policymakers since it can lead to refinements of the interventions if these goals do not line up with the main mechanisms we see the policy working through. The Social Science Prediction Platform provides an excellent public good by simplifying the work of collecting these predictions, and by providing a pool of forecasters. The team behind the platform provided excellent support in setting up the survey, which was easy to use as both a researcher and a forecaster.”

“Understanding the predictive power and the limitations of expert opinion in social science is likely to be crucial to developing new methods for inference on big questions in economics, where in my view using Bayesian methods to combine expert knowledge with data is one of very few promising ways forward. For Bayesian work to become practical on a larger scale, analysts will benefit greatly from tools like the Social Science Prediction Platform to elicit and aggregate expert forecasts of policy impacts. As far as I'm aware there is no other platform that comes close to what the SSPP has achieved thus far, and its relevance can only increase. In short, there is a serious need for a rigorous and systematic collection of this information, and aggregation platforms like the SSPP are only going to become more important as Bayesian methods become more relevant and widely-used in economics.”

“SSPP proved invaluable to me in a project where we needed a quantitative benchmark for a scholarly consensus on an effect size. The published literature provided some estimates, but no published study had done exactly what we planned. The SSPP allowed us to collect predictions from scholars which provided the benchmark we needed. This made it possible to be specific about how our results ought to inform consensus opinion. Moreover, it reduced the powerful effect of the hindsight bias that tempts readers to imagine, "I could have predicted that." Showing that our results contradicted scholarly consensus helped us make the case for our study's value.”

“We used the platform to gather predictions about a data collection on abortion public opinion in response to Supreme Court decisions on abortion. We found that scholars and non-scholars on the platform predicted shifts in abortion attitudes and perceptions of social norms in response to an abortion decision made by the Supreme Court, whereas our primary data collection found no evidence of such change. This discrepancy between the SSPP predictions and the primary findings help underscore the theoretical contribution of our findings of no change. As we enter the next phase of this project, we continue to draw from the insights gained from our SSPP data collection.”

“The Social Science Prediction Platform has the potential to serve as an essential and fundamental aspect of research across various social science fields. It has already become vital for my own research. I recently used the platform to obtain forecasts on a COVID-related project that my coauthors and I published in Science. Peer predictions about the results of a research project provides value in many different ways. For me, the highest value is that it helps the reader better understand and interpret the results of a project – especially if the project finds a null result. Researchers have been talking about file-drawer issues for a long time (e.g. null effects don’t get published). While the SSPP won’t completely solve the problem, it is a tool that can help alleviate this bias. The platform is easy to use and serves as an incredible public good. I hope to continue using it for years to come.”

“We elicited predictions about the economic and behavioral effects of increased sleep from both economists and sleep scientists for our paper "The Economic Consequences of Increasing Sleep Among the Urban Poor", now published in the Quarterly Journal of Economics. These predictions were central to the paper and helped establish that our findings were entirely contrary to the priors of scientists. The referees and editor noted the predictions in their reports and encouraged us to feature them prominently in the paper. Some of the sleep scientists we contacted were intrigued by the exercise and expressed an interest in using such predictions themselves. Since then, I have used the Social Science Prediction Platform for another project studying the long-run effects of psychotherapy for depression and I've encountered them a number of times in others' work as well. The Social Science Prediction Platform is a terrific public good for the profession”

“We used the platform to gather predictions about attitude towards loss, a fundamental preference underpinning economic behavior. We wanted to gather predictions from experts who were well-versed on these topics so the Social Science Prediction Platform was the first place we turned to. Their research support was absolutely superb in helping us get this survey up and running quickly and smoothly, and their suggestions based on expertise in survey design and implementation definitely improved the final outcome. We found that the expert panel accurately assessed the extent of loss aversion in the student sample, but underestimated differences between students and the general population.”

“The Social Science Prediction Platform has been an essential complement to our work on health decision-making in Mozambique. In one project, experts we surveyed via SSPP expected the treatment effect of a public health intervention would substantially raise rates of HIV testing. In fact, we found that the intervention actually lowered HIV testing rates. In another study on the complementarity between two treatments to raise learning about COVID-19, SSPP-surveyed experts predicted far less complementarity than we actually found. In both cases, being able to point to expert predictions bolsters our claims that our empirical findings were not "obvious" or predictable in advance. We plan to gather expert predictions via the SSPP in all our future field experimental projects, and I encourage all my collaborators and students to do the same.”

“The SSPP helps bring the important benefits of forecasting to the social sciences in a very accessible and user-friendly way. Other forecasting platforms often suffer from the questions not being particularly interesting, but all of the surveys on the SSPP have been about interesting and important topics. Getting feedback on the accuracy of my predictions has been a great incentive to keep making them and it's helpful to test whether the years of studying social science have given me a better understanding of the world. I'm looking forward to hosting my own surveys on the platform soon.”

“The SSPP represents an engaging way to be part of the experimental community and spark discussions. I have provided forecasts for several studies, and highly appreciate the possibility of going back to the predictions after the survey is completed, comparing my forecasts with the original study’s results, and eventually having the opportunity to read the final paper.”

“I read Superforecasting right before SSPP launched, so I thought SSPP would be fun to try. I appreciate when studies share their results and forecast distributions, and was pleasantly surprised to see I made relatively accurate forecasts. The Amazon gift card also swayed me to make many forecasts.”