Forecasting Survey Guide

Economists and other social scientists are increasingly collecting forecasts of research results. Forecasts are typically collected before research results are known. These forecasts can be used to contextualize research findings, mitigate publication bias, and improve the design of experiments. See this Science article for a short summary.

In the Survey Design section of this page, we outline some key features to include when designing research forecasting surveys, though not all features will be relevant for each study. We also provide an overview of how to use the platform to elicit forecasts. A template Qualtrics survey can be found under the Qualtrics Template header.

INTRODUCTION

Forecasts can be collected for both experimental and nonexperimental studies, and for treatments effects and other study parameters. For example:

  • Predicting treatment effects
    • Bessone et al. (2020) collect forecasts of the effects of a sleep intervention in India.
    • Casey et al. (2018) collect forecasts of results form a community-driven development intervention in Sierra Leone.
    • Cohn et al. (2019) collect forecasts of results from a field experiment testing the return of lost wallets around the world.
    • DellaVigna et al. (2020) collect forecasts of results from three registered reports from the  Journal of Development Economics.
    • Groh et al. (2016) collect forecasts of the effects of soft skills training in Jordan.
    • Vivalt and Coville (2019) examine how policymakers update their beliefs about the effectiveness of development interventions based on new information.
  • Predicting horse race comparing many treatments
  • Predicting non-causal estimates

SURVEY DESIGN

  • Yang et al. (TBD) examine effects of a community health intervention on HIV testing in Mozambique.
    • At the time we collected forecasts, baseline was complete, but the fidelity of the intervention was not known.
    • View annotated survey here
  • Bouguen and Dillon (TBD) run an experiment examining effects of a multi-armed cash, assets, and nutrition intervention in Burkina Faso.
    • We collected forecasts after midline had been completed, and we had some information on how treatment implementation was.
    • View annotated survey here
  • Blimpo and Pugatch (TBD) evaluate the effects of a teacher training intervention in Rwanda.

When designing a forecasting survey, it is important to consider:

  • The target population (e.g. nonexperts may require an explanation of random assignment).
  • Survey length (to reduce survey fatigue, we recommend surveys take 10-15 minutes to complete).

A typical forecasting survey has three parts: (1) a description of the study, (2) a description of the predicted outcome, and (3) elicitation of the predicted outcome. In the menu below, we provide examples highlighting the design decisions for three studies predicted in  DellaVigna et al. (2020).

Part 1: Describing the study

The study description is a one to two page summary highlighting key study details. A link to a more complete study description can be provided for interested respondents. Generally, a study description should include information on the target population, sample size, randomization, interventions, and study timing.

Part 2: Describing the predicted outcome

Respondents should be aware of how the outcome they are predicting is being measured (e.g. through an online survey) and how the outcome is constructed (e.g. we are interested in a composite measure of x,y, and z).

Part 3: Forecast elicitation

There are a number of ways to elicit predictions of experimental results. For example, are respondents providing predictions of a treatment effect, or of a conditional mean? Are predictions made using a slider scale, or numeric entry? Are predictions in raw units, or standard deviations?

When should forecasts be collected? 

Forecasts can be collected at any phase in the research process before experimental results are known. This includes:

(1) After an initial study design has been developed but before data collection has begun.

(2) After baseline or midline has been conducted (for experimental studies).

(3) After endline has been completed, but before results have been examined.

In general it is better to collect forecasts  before experimental results are known (unless forecast outcomes were pre-specified) to avoid collecting predictions on “unusual” results (e.g. null effects or very large effects).

Here are some tips to improve the quality of data from your Qualtrics survey.

Modifying variable names: You may want to give each variable a short and intuitive name. For example, a question eliciting forecasts of consumption could be named "ForecastConsumption" or "ForCon" for short. Avoid using symbols such as periods or underscores in names, as these can introduce errors into some data structures. To modify a variable name, simply click the automatically generated name:

Modifying variable labels: Each variable is given an automatic label containing the beginning of the question text. You may want to change the question label to provide a brief description of the question. The example above could be labelled "forecasts of consumption".

More information can be found on the Qualtrics website under Editing Question Labels.

Modifying variable values: Qualtrics sometimes assigns values to responses on multiple choice questions that you might not expect. You may want to check what numbers Qualtrics has assigned and modify the number associated with the responses accordingly. For example, in the figure below, you may want to label the values 1 and 2, as opposed to 4 and 5. To view and/or modify the value labels, first click the gear icon, then select "Recode Values".


More information can be found on the Qualtrics website.

Numeric response bounds: You may want to bound the range of forecasts respondents can provide for text response questions. Bounding can reduce the risk of entry errors. In the example below, we bound responses at +-2 (standard deviations), since the literature suggests that effects outside this range would be very unlikely.



Slider centering and decimal places: Often you will want your slider responses to be centered around a specific number. In the example below, Qualtrics does not allow the slider to be centered around zero when responses are provided to three decimal places. In this example, changing the number of decimal places from three to two allows the slider to be properly centered at zero.



Note that too much rounding can introduce forecast error (for example, if I want to predict an effect of .5 percentage points, but am limited to whole numbers). The number of decimal places you allow will depend on the scale of the outcome being predicted.

We have designed a Qualtrics template to help you design your forecasting survey.

  • You can view the survey here.
  • To download the .qsf file, click here.

USING THE PLATFORM

There are 5 steps to eliciting responses using the platform:
  1. Register your survey, and upload your .qsf file.
  2. Select your key prediction questions.
  3. Provide an initial distribution list.
  4. Wait for approval from platform staff.
  5. Distribute your survey.
Details on each of these steps are provided in the sections below.
To register a survey, first click All Studies, then click Begin under Create New Submission.

Step 1 of registration requires that you provide the following information for your study:
  • Title
  • Completion Time
  • Close Date
  • Have data been collected for this study?
  • Field
  • Location of Study
  • Abstract
  • Visibility:
    • Invite Only: Only the respondents you list will be able to access the survey.
    • Researchers Only: This survey will be open to respondents logged in with a Researcher account.
    • Public: This survey will be open to the general public.
You will also be required to upload your survey (.qsf) file.
Some surveyors may wish to provide financial incentives to respondents. These could be flat-rate incentives for participation, or could vary based on forecast accuracy. While we do not offer a mechanism to distribute payments, surveyors can have respondents opt in to provide an email address that can be used to coordinate payments. If you choose to offer incentives, we ask that you provide some basic information on the type of incentives. We provide examples of these questions below, which you will be asked when registering your survey.
When respondents take the survey, they will be able to opt in to provide you with additional information, as in the example below (where the surveyor has set different incentives from the example above).
This option allows researchers to prevent themselves from viewing their forecasts before a designated time. This option can be selected if, for example, you would like to guarantee that forecasts will not be viewed until after a study is completed. This can serve as a way for researchers to credibly signal that they did not modify their analysis after seeing the forecasts, mitigating the worry that they could have engaged in specification searching to find "surprising" results.
In addition to coordinating forecast elicitation efforts across researchers, we plan to use data collected on the platform to improve our understanding of forecast accuracy. This will help academics and policymakers improve their research and decision-making.

In order to reduce burden to platform users, we ask that you answer a few short questions about a key forecasting question, such as the main outcome of your study. After the study is completed, we will ask that you report the results for this outcome to us. If you select multiple key questions, we will ask you to report the results for each of them.

To select key questions, navigate through your survey using the built-in preview function of the platform:

When you select a key question, you will be asked to provide the following information:

What is the subject of the question?
  • A summary statistic (e.g. a conditional mean)
  • A causal estimation (e.g. an average treatment effect)
  • A first stage estimate (e.g. for an IV regression)
  • Other
Does this question use a text entry box or slider bar?
  • Text
  • Slider
What units are the forecasts elicited in?
  • Standard deviations
  • Percentage points
  • Percent
  • Other
Is the standard deviation of the outcome known? (please use the same units as the outcome)
  • Yes
    • What is it?
    • Is the statistic provided to forecasters
  • No
When you have selected all of your key prediction questions, click Next to continue to the next page.
Some researchers may be interested in collecting measures of forecaster confidence, though such measures are not required for a forecasting survey.

Measures of forecaster confidence can serve several purposes:
  • They allow forecasts given with different degrees of confidence to be treated differently.
  • They enable us to learn about how confidence correlates with accuracy.
  • They can help measure the extent to which forecasters are well-calibrated; a perfectly calibrated forecaster is able to accurately assess the proportion of their forecasts that fall within some bounds.
Examples of two popular types of questions measuring forecaster confidence are provided below:
  • Likert scale: How confident are you in this prediction? {Not at all confident, Not very confident, Somewhat confident, Very confident}
  • Percent within bounds: What percent of your forecasts do you think will fall within 50% of the observed experimental effect?
    • Note that in the latter example, we have selected a large bound (+-50%) due to the prevalence of overprecision among forecasters.
    • Bounds need not be provided in percent. For example, you could choose to provide bounds in standard deviations.
    • The bounds you decide on will depend on your research questions and context.
To distribute surveys by invitation, you need to upload a distribution list. You can do this either by directly uploading a list of (first name, last name, email), or through a .csv template we provide.
You can upload a maximum of 100 recipients, and can also modify this list before the surveys are distributed. Note that if a contact has recently completed a forecasting survey or has opted out of receiving forecasting surveys, they will not be contacted.

Under no circumstances will we provide the survey elicitor with a respondent's name or email address. However, it is possible that a respondent could be re-identified based on other information they provide. Platform administrators will also have access to the information respondents provided during registration.
After your survey is submitted, platform staff will review it for completeness. After approval, you will need to select "Publish Survey" before you are able to distribute your survey.
To view approved surveys, click All Studies and select Manage:


There are there are two ways to send out your surveys. The first option is to use the anonymous link we provided, which you can then distribute yourself. The second is to option is to distribute the survey through the platform. In both cases you will be able to access your survey responses through the platfrom.

(1) Distributing surveys through an anonymous link
To distribute your survey through an anonymous link, simply select Get Anonymous Link. The single anonymous survey link can be distributed to your target audience.
(2) Distributing surveys through the platform
Once you are at the Survey Dashboard, you can send the survey to your distribution list by selecting Distribute Survey.

Here, you will be provided with a template email that will be sent to your survey respondents. You can modify this template, but you must keep the text “[[ SHARE LINK ]]”, which will be automatically filled with the survey link when the survey is distributed.

There are two options to distribute your survey through the platform:

(1) Single use links: If you select to use single use links, each respondent on your distribution list will receive a unique survey link that can be completed without requiring the respondent to create an account.

(2) If you do not select single use links, responses will depend on the visibility option you have selected for your survey:
  • If you selected public, anyone with a SSPP account can respond to the survey.
  • If you selected researcher only, anyone with a SSPP researcher account can respond to the survey.
  • If you selected invite only, the survey will only be open to individuals on your distribution list and these individuals will be required to create an account before responding to your survey.