Restructuring structured techniques in intelligence analysis

PDF

Authors
  1. Wang, W.
  2. Berdini, E.
  3. Tetlock, P.E.
  4. Mandel, D.R.
Corporate Authors
Defence Research and Development Canada, Toronto Research Centre , Toronto ON (CAN)
Abstract
Structured analytic techniques (SATs) are intended to improve intelligence analysis by checking the two canonical sources of error: systematic biases and random noise. Although both goals are achievable, no one knows how close the current generation of SATs comes to achieving either of them. We identify two root problems: (1) SATs treat bipolar biases as unipolar. As a result, we lack metrics for gauging possible over-shooting—and have no way of knowing when SATs that focus on suppressing one bias (e.g., overconfidence) are triggering the opposing bias (e.g., under-confidence); (2) SATs tacitly assume that problem decomposition (e.g., breaking reasoning into rows and columns of matrices corresponding to hypotheses and evidence) is a sound means of reducing noise in assessments. But no one has ever actually tested whether decomposition is adding or subtracting noise from the analytic process—and there are good reasons for suspecting that decomposition will, on balance, degrade the reliability of analytic judgment. The central shortcoming is that SATs have not been subject to sustained scientific of the sort that could reveal when they are helping or harming the cause of delivering accurate assessments of the world to the policy community.
Keywords
structured analytic techniques;intelligence analysis;scientific method
Report Number
DRDC-RDDC-2017-P113 — External Literature
Date of publication
01 Nov 2017
Number of Pages
24
DSTKIM No
CA045499
CANDIS No
805913
Format(s):
Electronic Document(PDF)

Permanent link

Document 1 of 1

Date modified: