ATP 2-33.4, page 48

Chapter 4 QUALITY OF INFORMATION CHECK 4-20. The quality of information check technique evaluates the completeness and soundness of available information, both independently and in conjunction with sources. If a major analytic assessment is planned, analysts should individually or collectively review the quality of their information and refresh their understanding of the strengths and weaknesses of past reporting on which an analytic line of thinking rests. Without understanding the context and conditions under which critical information has been provided, it will be difficult for analysts to assess the information’s validity and establish a confidence level in an intelligence assessment. 4-21. Periodic reviews of the quality of the information should be conducted after the initial check to prevent assumptions or weak judgments from becoming fact over time. Facts 4-22. Weighing the validity of sources is a key feature of any critical thinking. Moreover, establishing how much confidence one puts in analytic judgments ultimately rests on how accurate and reliable the information base is. Analysts essentially must judge the accuracy and reliability of the information. Thus, checking the relative quality of information should be a continuous process. 4-23. Determining the quality of information independently of the source of the information is important to ensure that neither unduly compromises nor supports the other. That is, an excellent source can knowingly and admittedly pass third- or fourth-hand information that may be of low quality. It is important to keep the two reviews separate. This check can— e@ Provide the most important basis of determining confidence of the assessment and judgments. e Provide an opportunity to mitigate assimilation or confirmation bias based on the source. e Provide an opportunity to catch errors of interpretation. e Identify intelligence gaps. © Help identify areas of concern of denial and deception. @ Give the analyst an opportunity to clearly convey to the customers a better understanding of the analyst’s confidence in the aspects of the problem. 4-24. Analysts can become susceptible to circular reporting and source-based bias when reviewing the quality of information. Critical information can occasionally be found in reports from sources judged to have low access or a poor record. To ignore the information on the basis of quality independent of the source could cause the information to be unduly dismissed. Where one analyst works the same subject or area for extended periods, the analyst may miss the significance of incremental changes. The use of indicators can mitigate this possibility. THE METHOD 4-25. For the information review to be fully effective, analysts will need as much background information on sources as is possible. Knowing the circumstances in which reporting was obtained is often critical to understanding its validity. With this information the analysts should then, at a minimum— © Review all sources of information for accuracy; identify any of those sources more critical or compelling. e@ Determine if they have sufficient and/or strong collaboration between the information sources. © Reexamine previously dismissed information in light of new facts or circumstances. e Ensure any circular reporting is identified and properly flagged for other analysts; analysis based on circular reporting should also be reviewed to determine if the reporting was essential to the judgments made. e Consider whether ambiguous information has been interpreted and qualified properly. © Indicate a level of confidence they can place in sources, which are likely to figure in future analytic assessments. 4-4 ATP 2-33.4 18 August 2014 Chapter 4 4-4 ATP 2-33.4 18 August 2014 QUALITY OF INFORMATION CHECK 4-20. The quality of information check technique evaluates the completeness and soundness of available information, both independently and in conjunction with sources. If a major analytic assessment is planned, analysts should individually or collectively review the quality of their information and refresh their understanding of the strengths and weaknesses of past reporting on which an analytic line of thinking rests. Without understanding the context and conditions under which critical information has been provided, it will be difficult for analysts to assess the information’s validity and establish a confidence level in an intelligence assessment. 4-21. Periodic reviews of the quality of the information should be conducted after the initial check to prevent assumptions or weak judgments from becoming fact over time. FACTS 4-22. Weighing the validity of sources is a key feature of any critical thinking. Moreover, establishing how much confidence one puts in analytic judgments ultimately rests on how accurate and reliable the information base is. Analysts essentially must judge the accuracy and reliability of the information. Thus, checking the relative quality of information should be a continuous process. 4-23. Determining the quality of information independently of the source of the information is important to ensure that neither unduly compromises nor supports the other. That is, an excellent source can knowingly and admittedly pass third- or fourth-hand information that may be of low quality. It is important to keep the two reviews separate. This check can—  Provide the most important basis of determining confidence of the assessment and judgments.  Provide an opportunity to mitigate assimilation or confirmation bias based on the source.  Provide an opportunity to catch errors of interpretation.  Identify intelligence gaps.  Help identify areas of concern of denial and deception.  Give the analyst an opportunity to clearly convey to the customers a better understanding of the analyst’s confidence in the aspects of the problem. 4-24. Analysts can become susceptible to circular reporting and source-based bias when reviewing the quality of information. Critical information can occasionally be found in reports from sources judged to have low access or a poor record. To ignore the information on the basis of quality independent of the source could cause the information to be unduly dismissed. Where one analyst works the same subject or area for extended periods, the analyst may miss the significance of incremental changes. The use of indicators can mitigate this possibility. THE METHOD 4-25. For the information review to be fully effective, analysts will need as much background information on sources as is possible. Knowing the circumstances in which reporting was obtained is often critical to understanding its validity. With this information the analysts should then, at a minimum—  Review all sources of information for accuracy; identify any of those sources more critical or compelling.  Determine if they have sufficient and/or strong collaboration between the information sources.  Reexamine previously dismissed information in light of new facts or circumstances.  Ensure any circular reporting is identified and properly flagged for other analysts; analysis based on circular reporting should also be reviewed to determine if the reporting was essential to the judgments made.  Consider whether ambiguous information has been interpreted and qualified properly.  Indicate a level of confidence they can place in sources, which are likely to figure in future analytic assessments.