Geopolitical Forecasting Skill in Strategic Intelligence

PDF

Authors
  1. Mandel, D.R.
  2. Barnes, A.
Corporate Authors
Defence Research and Development Canada, Toronto Research Centre , Toronto ON (CAN)
Abstract
Extending research by the authors on intelligence forecasting, the forecasting skill of 3,622 geopolitical forecasts extracted from strategic intelligence reports was examined. The codable subset of forecasts (N = 2,013) was expressed with verbal probabilities (e.g., likely) and translated to numeric probability equivalents. This subset showed very good calibration and discrimination, but also underconfidence. There was no support for the hypothesis that forecasting skill was good mainly due to the general ease of forecasting topics. First, forecasting skill was as good among authoritative key judgments as in the general set. Second, forecasts that were assigned high degrees of certainty, indicative of ease, (P≤0.05 or P ≥0.95) did not discriminate as well as less certain forecasts (0.05 < P < 0.95), and these subsets did not differ in calibration. Sensitivity and benchmarking tests further revealed that if the 1,609 uncodable forecasts were all assigned forecast probabilities of .5 (i.e., if all followed a “cautious ignorance” rule), skill characteristics would still show a large effect size improvement over a variety of guesswork strategies. The findings support a cautiously optimistic assessment of forecasting skill in strategic intelligence and indicate that such skill is not primarily attributable to the selection of easy forecasting topics. However, the large proportion of uncodable cases suggests that intelligence forecasts could be impr
Keywords
geopolitical forecasting;intelligence assessment;calibration;discrimination
Report Number
DRDC-RDDC-2017-P091 — External Literature
Date of publication
01 Nov 2017
Number of Pages
29
Reprinted from
Journal of Behavioral Decision Making
DSTKIM No
CA045471
CANDIS No
805881
Format(s):
Electronic Document(PDF)

Permanent link

Document 1 of 1

Date modified: