Have PI’s assessments undergone any third-party reviews?

At the Predictive Index, we employ a full-time staff of qualified researchers who develop, maintain, monitor, and document the PI assessments to ensure that they are valid, reliable, and fair; however, our researchers are not the only people who have eyes on our assessments. PI’s assessments have undergone formal peer reviews to reviews where researchers check the properties of the instrument to see if it can be used in their own studies. Time and again, the PI Behavioral Assessment and Cognitive Assessment have stood up to scrutiny, which is why the assessments have endured as useful, powerful workplace tools.

What have been the most comprehensive third-party reviews?

The most comprehensive third-party review of the PI Behavioral Assessment was conducted from 2017 to 2018 through the certification company DNV-GL. In that review, Form V of the PI Behavioral Assessment was reviewed by Dag Øyvind Engen Nilsen and Sara Henrysson Eidvall, psychologists and independent DNV-GL auditors who evaluated the assessment against the guidelines published by the European Federation of Psychologists’ Associations (EFPA, 2013). These two auditors reviewed the entire assessment system, including validity, reliability, fairness, development, norms, reporting, supporting documentation and training, and even pricing and distribution. In addition to the general assessment system, the auditors reviewed and certified the PI Behavioral Assessment system for use in Sweden and Norway, meaning that technical documentation, psychometric analyses, fairness studies, and translation information were also analyzed for samples from those regions. The PI Behavioral Assessment passed this review and was certified under the EFPA model in September 2018. The PI Behavioral Assessment was successfully recertified under EFPA guidelines in 2021. PI now provides DNV-GL with annual reports on the PI Behavioral Assessment and recertifies the assessment with full reviews every five years.

 In 2020, Drs. Nilsen and Eidvall conducted a review of the PI Cognitive Assessment on behalf of DNV-GL and the assessment achieved certification for use on the overall level and for Sweden. This exhaustive review looked at technical documentation, additional psychometric analyses, fairness studies, and translation information. The PI Cognitive Assessment passed this review and was initially certified under the EFPA model in June 2020. The PI Cognitive Assessment was successfully recertified following EFPA guidelines in 2023. PI now provides DNV-GL with annual reports on the PI Cognitive Assessment and will undergo additional reviews every five years.

What other third-party reviews have been conducted? 

The first formal third-party peer review of the PI Behavioral Assessment was in 1983, when, at PI’s request, Form III of the PI Behavioral Assessment was reviewed by Christopher Perry and Philip Lavori from Harvard Medical School.

At the time, Dr. Perry was an assistant professor of psychiatry, and Dr. Lavori was a statistics lecturer in the psychiatry department. Using samples of 120-260 results that we provided, Perry and Lavori conducted their own independent analyses of the internal consistency reliability and test-retest reliability of the PI Behavioral Assessment Form III. They also conducted a construct validity study, validating PI Behavioral Assessment factor scores against the 16PF scales (e.g., Cattell & Mead, 2008). Their results showed that the PI Behavioral Assessment was both valid and reliable (Perry & Lavori, 1983). 

Some years later, Lennart Sjöberg of the Stockholm School of Economics wrote two reviews of the psychometric properties of Form IV of the PI Behavioral Assessment (the first in 2000 and a follow up in 2003), specifically focusing on the psychometric equivalency of the instrument when translated. At the time of these studies, Sjöberg was a professor of economic psychology. Sjöberg’s analyses were a more advanced review of the PI Behavioral Assessment, leveraging confirmatory factor analysis techniques that were not as accessible to researchers when Perry and Lavori (1983) conducted their study. In his first analysis, Sjöberg (2000) looked at a sample of 343 responses collected in Sweden using Form IV of the PI Behavioral Assessment. He studied the internal consistency reliability of the factor scores and found the assessment to be reliable. He also used exploratory factor analysis to check the dimensionality and construct validity of the assessment. Based on these results, Sjöberg hypothesized that there may be an opportunity for scale development to differentiate between social attitude and social behavior, but he ultimately concluded that the factor structure matched the intended factor structure reported by the assessment. 

In his second study, Sjöberg (2003) analyzed a sample of 273 Swedish respondents, 107 of which were recruited by one of our regional partners, and 166 of which were recruited by Sjöberg through an employment agency. Again, Sjöberg found acceptable levels for his internal consistency reliability estimates. Sjöberg conducted a confirmatory factor analysis, finding an acceptable model fit for the four primary factor model used by the PI Behavioral Assessment. Through that study, he identified a possible underlying general factor, which he linked to measures of social desirability for some traits; however, he concluded that the effect of social desirability did not meaningfully change the outcomes or interpretations of the scores. Sjöberg also noticed that raw scores correlated with emotional intelligence across all four factors, and he suggested that this be reviewed in future research (this was later explored in Fossey’s [2016] factor analysis of response rate in the PI Behavioral Assessment Form V, which found no meaningful relations between emotional intelligence and response rate). Sjöberg also reported on a small criterion validity study of 30 salespeople (Sjöberg, 2003), and he later used the PI Behavioral Assessment as a comparison in his critique of another similar assessment product (Sjöberg, 2005). 

In addition, several doctoral students from accredited university programs have used the PI Behavioral Assessment as a measure of personality in their dissertations. 

Are there any blind, academic journal reviews? 

Although the studies above are all third-party peer reviews, they are not articles from scientific journals (which also use peer review). Journals in our field (e.g., Journal of Applied Psychology) typically publish original research or experimental results. One can find journal reviews of psychometric properties of clinical instruments, but commercial assessments like ours are commonly reviewed by third-party accreditation programs against a set of published standards, such as the EFPA review mentioned above. The American Psychology Association (APA) and the Society for Industrial Organizational Psychology (SIOP) both publish assessment standards (to which we adhere), but they do not have their own review or accreditation programs like EFPA. 

What does this mean in practice? 

For our clients, we hope that these third-party reviews and collaborations further bolster your confidence in the PI Behavioral Assessment. We invest heavily in the research and development behind our assessment systems because we know that clients depend on the accuracy and validity of the results to drive their talent optimization decisions.

In many regions, there is no requirement that an assessment be reviewed by third parties or meet more than the bare minimum threshold for validation, but we feel that third-party reviews and collaboration help build a strong case for the value of the PI Behavioral Assessment. Our own PI researchers also learn from these experiences, and the results and feedback obtained from third-parties help inform and guide PI’s own research as we continuously improve our assessment systems. We are proud that the PI Behavioral Assessment continuously holds up to scrutiny, but we are even more proud to be able to provide such high-quality assessments to our clients. 

References 

• Cattell, H. E. P., & Mead, A. D. (2008). The Sixteen Personality Factor Questionnaire (16PF). In G. J. Boyle, G. Matthews, & D. H. Saklofske (Eds.), The SAGE handbook of personality theory and assessment, Vol. 2. Personality measurement and testing (pp. 135-159). Thousand Oaks, CA: Sage Publications, Inc. 
• Elsevier (2018). What is peer review? Amsterdam, The Netherlands, Elsevier. Available from https://www.elsevier.com/reviewers/what-is-peer-review. 
• European Federation of Psychologists’ Associations (EFPA) (2013). EFPA review model for the description and evaluation of psychological and educational tests: Test review form and notes for reviewers (v 4.2.6). A. Evers, C. Hagemeister, A. Høstmælingen, P. Lindley, J. Muñiz, & A. Sjöberg (Eds.). Brussels: EFPA. Available from: http://www.efpa.eu/professional-development/assessment. 
• Fossey, A. (2016). Project M: Construct validity study on the interpretations of Factor M and Morale in the PI Behavioral Assessment [technical report]. Available from: https://piworldwide.atlassian.net/wiki/display/LINK/Science+Whitepapers?preview=/110666709/110667094/Project%20M%20-%20Results%20Document%202.pdf. 
• Perry, J. C., & Lavori, P. W. (1983). The Predictive Index: A report on reliability and construct validity [technical report]. Wellesley Hills, MA: The Predictive Index. 
• Sjöberg, L. (2000). The psychometric structure of the Swedish version of the Predictive Index (PI). Stockholm, Sweden: Stockholm School of Economics. 
• Sjöberg, L. (2003). Properties of the new Swedish version of the Predictive Index. Stockholm, Sweden: Stockholm School of Economics. 
• Sjöberg, L. (2005). The Swedish version of Thomas PPA: Critical comments. Stockholm, Sweden: Stockholm School of Economics.