Economists and other social scientists are increasingly collecting forecasts of research results. Forecasts are typically collected before research results are known. These forecasts can be used to contextualize research findings, mitigate publication bias, and improve the design of experiments. See
this Science article
for a short summary.
In the Survey Design section of this page, we outline some key features to include when designing research forecasting surveys, though not all features will be relevant for each study. We also provide an overview of how to use the platform to elicit forecasts. A template Qualtrics survey can be found under the Qualtrics Template header.
Forecasts can be collected for both experimental and nonexperimental studies, and for treatments effects and other study parameters. For example:
Predicting treatment effects
- Bessone et al. (2020) collect forecasts of the effects of a sleep intervention in India.
- Casey et al. (2018) collect forecasts of results form a community-driven development intervention in Sierra Leone.
- Cohn et al. (2019) collect forecasts of results from a field experiment testing the return of lost wallets around the world.
- DellaVigna et al. (2020) collect forecasts of results from three registered reports from the Journal of Development Economics.
- Groh et al. (2016) collect forecasts of the effects of soft skills training in Jordan.
- Vivalt and Coville (2019) examine how policymakers update their beliefs about the effectiveness of development interventions based on new information.
Predicting horse race comparing many treatments
- DellaVigna and Pope (2018) collect forecasts of the effects of behavioral interventions on a costly effort task.
- Predicting non-causal estimates
- Yang et al. (TBD) examine effects of a community health intervention on HIV testing in Mozambique.
- At the time we collected forecasts, baseline was complete, but the fidelity of the intervention was not known.
- View annotated survey here
- Bouguen and Dillon (TBD) run an experiment examining effects of a multi-armed cash, assets, and nutrition intervention in Burkina Faso.
- We collected forecasts after midline had been completed, and we had some information on how treatment implementation was.
- View annotated survey here
- Blimpo and Pugatch (TBD) evaluate the effects of a teacher training intervention in Rwanda.
- We had information from the control group at endline when we collected predictions.
- View annotated survey here
When designing a forecasting survey, it is important to consider:
- The target population (e.g. nonexperts may require an explanation of random assignment).
- Survey length (to reduce survey fatigue, we recommend surveys take 10-15 minutes to complete).
A typical forecasting survey has three parts: (1) a description of the study, (2) a description of the predicted outcome, and (3) elicitation of the predicted outcome. In the menu below, we provide examples highlighting the design decisions for three studies predicted in DellaVigna et al. (2020).
Part 1: Describing the study
The study description is a one to two page summary highlighting key study details. A link to a more complete study description can be provided for interested respondents. Generally, a study description should include information on the target population, sample size, randomization, interventions, and study timing.
Part 2: Describing the predicted outcome
Respondents should be aware of how the outcome they are predicting is being measured (e.g. through an online survey) and how the outcome is constructed (e.g. we are interested in a composite measure of x,y, and z).
Part 3: Forecast elicitation
There are a number of ways to elicit predictions of experimental results. For example, are respondents providing predictions of a treatment effect, or of a conditional mean? Are predictions made using a slider scale, or numeric entry? Are predictions in raw units, or standard deviations?
When should forecasts be collected?
Forecasts can be collected at any phase in the research process before experimental results are known. This includes:
(1) After an initial study design has been developed but before data collection has begun.
(2) After baseline or midline has been conducted (for experimental studies).
(3) After endline has been completed, but before results have been examined.
In general it is better to collect forecasts before experimental results are known (unless forecast outcomes were pre-specified) to avoid collecting predictions on “unusual” results (e.g. null effects or very large effects).
Modifying variable names: You may want to give each variable a short and intuitive name. For example, a question eliciting forecasts of consumption could be named "ForecastConsumption" or "ForCon" for short. Avoid using symbols such as periods or underscores in names, as these can introduce errors into some data structures. To modify a variable name, simply click the automatically generated name:
Modifying variable labels: Each variable is given an automatic label containing the beginning of the question text. You may want to change the question label to provide a brief description of the question. The example above could be labelled "forecasts of consumption".
More information can be found on the Qualtrics website under Editing Question Labels.
Modifying variable values: Qualtrics sometimes assigns values to responses on multiple choice questions that you might not expect. You may want to check what numbers Qualtrics has assigned and modify the number associated with the responses accordingly. For example, in the figure below, you may want to label the values 1 and 2, as opposed to 4 and 5. To view and/or modify the value labels, first click the gear icon, then select "Recode Values".
More information can be found on the Qualtrics website.
Numeric response bounds: You may want to bound the range of forecasts respondents can provide for text response questions. Bounding can reduce the risk of entry errors. In the example below, we bound responses at +-2 (standard deviations), since the literature suggests that effects outside this range would be very unlikely.
USING THE PLATFORM
- Register your survey, and upload your .qsf file.
- Select your key prediction questions.
- Provide an initial distribution list.
- Wait for approval from platform staff.
- Distribute your survey.
- Completion Time
- Close Date
- Have data been collected for this study?
- Location of Study
- Invite Only: Only the respondents you list will receive the survey.
- Researchers Only: This survey will be open to all respondents with a Researcher account.
- Public Only : This survey will be open to the general public.
In order to reduce burden to platform users, we ask that you answer a few short questions about a key forecasting question, such as the main outcome of your study. After the study is completed, we will ask that you report the results for this outcome to us. If you select multiple key questions, we will ask you to report the results for each of them.
To select key questions, navigate through your survey using the built-in preview function of the platform:
When you select a key question, you will be asked to provide the following information: What is the subject of the question?
- A summary statistic (e.g. a conditional mean)
- A causal estimation (e.g. an average treatment effect)
- A first stage estimate (e.g. for an IV regression)
- Standard deviations
- Percentage points
- What is it?
- Is the statistic provided to forecasters
Under no circumstances will we provide the suvey elicitor with a respondent's name or email address. However, it is possible that a respondent could be re-identified based on other information they provide. Platform administrators will also have access to the information respondents provided during registration.
There are there are two ways to send out your surveys. The first option is to use the anonymous link we provided, which you can then distribute yourself. The second is to option is to distribute the survey through the platform. In both cases you will be able to access your survey responses through the platfrom.
(1) Distributing surveys through an anonymous linkTo distribute your survey through an anonymous link, simply select Get Anonymous Link. The single anonymous survey link can be distributed to your target audience.
(2) Distributing surveys through the platformOnce you are at the Survey Dashboard, you can send the survey to your distribution list by selecting Distribute Survey.
Here, you will be provided with a template email that will be sent to your survey respondents. You can modify this template, but you must keep the text “[[ SHARE LINK ]]”, which will be automatically filled with the survey link when the survey is distributed.
There are two options to distribute your survey through the platform:
(1) Single use links: If you select to use single use links, each respondent on your distribution list will receive a unique survey link that can be completed without requiring the respondent to create a researcher account.
(2) If you do not select single use links, responses will depend on the visibility option you have selected for your survey:
- If you selected public, anyone with a SSPP account can respond to the survey.
- If you selected researcher only, anyone with a SSPP researcher account can respond to the survey.
- If you selected invite only, the survey will only be open to individuals on your distribution list and these individuals will be required to create an account before responding to your survey.