Forecasting Survey Guide

Forecasts can be collected for both experimental and nonexperimental studies, and for treatments effects and other study parameters. For example:

• Predicting treatment effects
  • Bessone et al. (2020) collect forecasts of the effects of a sleep intervention in India.
  • Casey et al. (2018) collect forecasts of results form a community-driven development intervention in Sierra Leone.
  • Cohn et al. (2019) collect forecasts of results from a field experiment testing the return of lost wallets around the world.
  • DellaVigna et al. (2020) collect forecasts of results from three registered reports from the  Journal of Development Economics.
  • Groh et al. (2016) collect forecasts of the effects of soft skills training in Jordan.
  • Vivalt and Coville (2019) examine how policymakers update their beliefs about the effectiveness of development interventions based on new information.
• Predicting horse race comparing many treatments
  • DellaVigna and Pope (2018) collect forecasts of the effects of behavioral interventions on a costly effort task.
• Predicting non-causal estimates
  • Card et al. (2019) collect predictions about gender differences in the association between citations and paper quality.
  • Dreber et al. (2015) and  Camerer et al. (2016) elicit predictions of the probability that experiments would replicate.

In this section, we highlight mistakes in forecasting surveys, as well as recommendations we have received from forcasters.

Reliable image hosting: If you include images in your survey, make sure that they are hosted reliably. We recommend stable image hosting platforms such as https://imgbb.com/ or github. If you host survey images on your personal website, be aware that changing the directory where images are stored will result in failure to display images in your survey.

Remind respondents to finish survey: It is fairly common for respondents not to click the arrow on the final page of the survey that officially marks their response as complete. If respondents fail to do this, their response will not be logged in the count of completed survey responses we display (once the survey has closed, you will still be able to view these "partial" responses). We recommend that you remind participants on the last page that they need to proceed to the next page to officially end the survey.


Requiring responses: If you want to ensure that your participants answer a particular question, you check the Force Response box.


Modifying variable names: You may want to give each variable a short and intuitive name. For example, a question eliciting forecasts of consumption could be named "ForecastConsumption" or "ForCon" for short. Avoid using symbols such as periods or underscores in names, as these can introduce errors into some data structures. To modify a variable name, simply click the automatically generated name:

Back button: A number of forecasters have requested that surveys include a back button. You can select this option by going to Survey Options and selecting the Back Button box.
Note that conditional logic and randomization within surveys can limit the functionality of back buttons. We recommend that you test your survey to ensure that it is functioning as desired.

Provide a reminder of the study description: Many forecasters want to revisit the study description when providing their forecasts. We recommend that you provide an external link to your survey description that respondents can have open in a separate window while providing predictions (For example, we provide an annotated survey annotated survey in which we provide an external link to the study overview. You can find more examples here here). Alternatively, a short review of the study details can be provided on each page.

Preview before publishing: We strongly recommend that you preview your survey within the platform before publishing it. This option is available for surveys that have been "approved", and can be found on the Manage Survey page:

Changing a published survey: In some circumstances, you may wish to modify a survey that you have already launched. There are currently two ways to do this. First, you can contact us at support@socialscienceprediction.org, and our team will make and publish the desired changes to your survey. Alternatively, you can upload a new version of your survey onto the platform that includes the desired changes. Note that within a project, the procedure for uploading a second survey is the same as for uploading the initial survey--simply select the "Create New Survey" option, after which we will review and approve the survey.


Additional recommendations for Qualtrics surveys can be found in the Qualtrics Tips section of the guide.

Some researchers opt to pre-register that they will collect predictions. For instance, in their pre-registration documents, researchers might state, "We intend to gather predictions about our results on the Social Science Prediction Platform." Additionally, some researchers may wish to specify what they are collecting predictions of or how they plan to use the forecasts.

When designing a forecasting survey, it is important to consider:

• The target population (e.g. nonexperts may require an explanation of random assignment).
• Survey length (to reduce survey fatigue, we recommend surveys take 10-15 minutes to complete).

A typical forecasting survey has three parts: (1) a description of the study, (2) a description of the predicted outcome, and (3) elicitation of the predicted outcome. In the menu below, we provide examples highlighting the design decisions for three studies predicted in  DellaVigna et al. (2020).

Part 1: Describing the study

The study description is a one to two page summary highlighting key study details. A link to a more complete study description can be provided for interested respondents. Generally, a study description should include information on the target population, sample size, randomization, interventions, and study timing.

Part 2: Describing the predicted outcome

Respondents should be aware of how the outcome they are predicting is being measured (e.g. through an online survey) and how the outcome is constructed (e.g. we are interested in a composite measure of x,y, and z).

Part 3: Forecast elicitation

There are a number of ways to elicit predictions of experimental results. For example, are respondents providing predictions of a treatment effect, or of a conditional mean? Are predictions made using a slider scale, or numeric entry? Are predictions in raw units, or standard deviations?

When should forecasts be collected? 

Forecasts can be collected at any phase in the research process before experimental results are known. This includes:

(1) After an initial study design has been developed but before data collection has begun.

(2) After baseline or midline has been conducted (for experimental studies).

(3) After endline has been completed, but before results have been examined.

In general it is better to collect forecasts  before experimental results are known (unless forecast outcomes were pre-specified) to avoid collecting predictions on “unusual” results (e.g. null effects or very large effects).

Outliers

Often we are interested in the average forecast, which can be influenced by extreme values.

A common strategy to reduce the likelihood of outliers is to bound the response scale in Qualtrics. Many project teams choose to bound response scales at +-1 standard deviations, though this will depend on the outcome and context.

A complementary strategy is to consider transformations like winsorization that can limit the influence of outliers. Another alternative is use measures of forecast centrality like the median forecast which are less susceptible to outliers than the mean prediction.

Here are some tips to improve the quality of data from your Qualtrics survey.

Modifying variable labels: Each variable is given an automatic label containing the beginning of the question text. You may want to change the question label to provide a brief description of the question. The example above could be labelled "forecasts of consumption".

More information can be found on the Qualtrics website under Editing Question Labels.

Modifying variable values: Qualtrics sometimes assigns values to responses on multiple choice questions that you might not expect. You may want to check what numbers Qualtrics has assigned and modify the number associated with the responses accordingly. For example, in the figure below, you may want to label the values 1 and 2, as opposed to 4 and 5. To view and/or modify the value labels, first click the gear icon, then select "Recode Values".


More information can be found on the Qualtrics website.

Numeric response bounds: You may want to bound the range of forecasts respondents can provide for text response questions. Bounding can reduce the risk of entry errors. In the example below, we bound responses at +-2 (standard deviations), since the literature suggests that effects outside this range would be very unlikely.



Slider centering and decimal places: Often you will want your slider responses to be centered around a specific number. In the example below, Qualtrics does not allow the slider to be centered around zero when responses are provided to three decimal places. In this example, changing the number of decimal places from three to two allows the slider to be properly centered at zero.



Note that too much rounding can introduce forecast error (for example, if I want to predict an effect of .5 percentage points, but am limited to whole numbers). The number of decimal places you allow will depend on the scale of the outcome being predicted.

Sliders and mobile devices We have received numerous reports that slider scales are difficult to use on mobile devices. If you think your targeted respondent pool will contain many users who complete the survey on their mobile device, we would caution you against using slider scales. For more information, see Qualtrics Mobile Survey Optimization page.

Image hosting: We are not able to host images on our Qualtrics account. Therefore, images referenced in your Qualtrics file need to be externally hosted. After you upload your survey, be sure to go through it using the "preview" function to check for formatting issues before the survey is distributed.

We have designed a Qualtrics template to help you design your forecasting survey.

• You can view the survey here.
• To download the .qsf file, click here.

These are the steps to eliciting responses using the platform:

1. Register your survey, and upload your .qsf file.
2. Select your key forecasting questions.
3. Wait for approval from platform staff.
4. Distribute your survey.

More detailed directions are provided in the sections below.

This option allows researchers to prevent themselves from viewing their forecasts before a designated time. This option can be selected if, for example, you would like to guarantee that forecasts will not be viewed until after a study is completed. This can serve as a way for researchers to credibly signal that they did not modify their analysis after seeing the forecasts, mitigating the worry that they could have engaged in specification searching to find "surprising" results.

In addition to coordinating forecast elicitation efforts across researchers, we plan to use data collected on the platform to improve our understanding of forecast accuracy. This will help academics and policymakers improve their research and decision-making.

In order to reduce burden to platform users, we ask that you answer a few short questions about a key forecasting question, such as the main outcome of your study. After the study is completed, we will ask that you report the results for this outcome to us. If you select multiple key questions, we will ask you to report the results for each of them.

To select key questions, navigate through your survey using the built-in preview function of the platform:
When you select a key question, you will be asked to provide the following information:

What is the subject of the question?

• A summary statistic (e.g. a conditional mean)
• A causal estimation (e.g. an average treatment effect)
• A first stage estimate (e.g. for an IV regression)
• Other

Does this question use a text entry box or slider bar?

• Text
• Slider

What units are the forecasts elicited in?

• Standard deviations
• Percentage points
• Percent
• Other

Is the standard deviation of the outcome known? (please use the same units as the outcome)

• Yes
• What is it?
• Is the statistic provided to forecasters
• No

When you have selected all of your key prediction questions, click Next to continue to the next page.

Some researchers may be interested in collecting measures of forecaster confidence, though such measures are not required for a forecasting survey.

Measures of forecaster confidence can serve several purposes:

• They allow forecasts given with different degrees of confidence to be treated differently.
• They enable us to learn about how confidence correlates with accuracy.
• They can help measure the extent to which forecasters are well-calibrated; a perfectly calibrated forecaster is able to accurately assess the proportion of their forecasts that fall within some bounds.

Examples of two popular types of questions measuring forecaster confidence are provided below:

• Likert scale: How confident are you in this prediction? {Not at all confident, Slightly confident, Somewhat confident, Very confident, Extremely Confident}
• Percent within bounds: What percent of your forecasts do you think will fall within 50% of the observed experimental effect?
• Note that in the latter example, we have selected a large bound (+-50%) due to the prevalence of overprecision among forecasters.
• Bounds need not be provided in percent. For example, you could choose to provide bounds in standard deviations.
• The bounds you decide on will depend on your research questions and context.

To distribute surveys by invitation, you need to upload a distribution list. You can do this either by directly uploading a list of (first name, last name, email), or through a .csv template we provide. You can upload a maximum of 100 recipients, and can also modify this list before the surveys are distributed. Note that if a contact has recently completed a forecasting survey or has opted out of receiving forecasting surveys, they will not be contacted.

Under no circumstances will we provide the survey elicitor with a respondent's name or email address. However, it is possible that a respondent could be re-identified based on other information they provide. Platform administrators will also have access to the information respondents provided during registration.

After your survey is submitted, platform staff will review it for completeness. After approval, you will need to select "Publish Survey" before you are able to distribute your survey.

Accessibility

When registering your survey, you will have to select who will be able to access it. There are 3 options:

1. Open to specific platform users: Only the respondents you list will be able to take the survey. If they do not have an account, they will be required to make one with an email that matches that of their invitation.
2. Open to any platform user: This survey will be open to anyone who has an account on the platform and a link to the survey
3. Public: This survey will be open to anyone with a link. No account on the platform is necessary. You may want to select this option if you plan to share the link over Twitter.

Distribution

These accessibility options are complemented by three different (non-exclusive) ways to distribute the surveys:

1. Anonymous link
To distribute your survey through an anonymous link, simply select Get Anonymous Link. The single anonymous survey link can be distributed to your target audience.


If you are using anonymous links, we recommend you set the accessibility options to "Public".


2. Collecting predictions through the Prediction Dashboard
The second way to collect predictions is by publicizing your survey on the Social Science Prediction Dashboard.


If you are posting the survey on the Prediction Dashboard, we recommend you set that you do not set the accessibility as "Open to specific platform users".


3. Email Distribution
Once you are at the Survey Dashboard, you can send the survey to your distribution list (see the section above) by selecting Distribute Survey.

Here, you will be provided with a template email that will be sent to your survey respondents. You can modify this template, but you must keep the text “[[ SHARE LINK ]]”, which will be automatically filled with the survey link when the survey is distributed.


If you would like to generate links for each participant on your contact list, leave the top box unchecked. If you would like the links (distributed by the platform) to be single use, check the second box.

Using the Duplicate function to track multiple distribution methods

What if you want to use multiple data collection formats? For example, you may want to post an anonymous link on Twitter, and also to send emails to a small number of experts who have published work related to your study, where only those targeted individuals would be able to use the assigned links.

This can be achieved using the "Duplicate" function (found in the "Additional options" section of the survey page), which creates a copy of your project, allowing you to adjust the survey restrictions for each target population.

You have the option to add a set of basic questions on the academic background of the respondent.

What describes you best? Please select "yes" if you are currently one of the following: {Graduate student (Master level or PhD), Faculty, Post-doc, Non-academic researcher}

IF YES: 

• Institution
• Discipline {Economics, Political Science, Psychology, Sociology}
• Title {Professor,Assistant Professor, Post-doc, Researcher, PhD Student, Masters Student}
  • If "Researcher" is selected: What is the highest level of education you have attained? {Bacheor's degree, Masters degree, PhD, other graduate/professional degree}
• Field {Main fields of each discipline}

IF NO:

• What type of organization do you currently work for? {Academia/research institute,Government,International organization,NGO,Private sector,Other:__}
• What is the highest level of education you have attained? {Bachelors degree, Masters degree, PhD, other graduate/professional degree}
• Sector

By default, we add the following four questions to the end of the survey. If you are interested in including them, they will be automatically appended to your Qualtrics survey during the upload phase, so there is no need for you to include them in the version of the survey you upload. If you do not with to include these questions, simply uncheck them in the form.

1. How confident are you in your predictions for this study? If you are confident, it means that you believe your predictions are very accurate.
   {Not at all confident, Slightly confident, Somewhat confident, Very confident, Extremely Confident}
2. Have you conducted research on the topics covered in this survey? {No, Yes}
   We encourage you to change the question text to fit your study topic. For example, if the study topic is depression, "the topics covered in this survey" could be changed to "mental health".
3. Have you conducted research in the setting where this study takes place? {No, Yes}
   We encourage you to change the question text to fit your study setting. For example, "the setting where this study takes place" could be changed to "rural Kenya". For online studies, consider the sample pool (such as Mechanical Turk).
4. If you have any comments, please enter them below. We would love to hear your feedback. [Free response]