Making the most of the Job Assessment
What is the JA?
The PI Job Assessment (JA) helps users to build a Job Target that accurately reflects the behavioral and/or cognitive requirements for a role. It’s not a psychological assessment and does not measure characteristics of a person. Instead, it establishes the skills and abilities necessary upon entry to the role and serves as a conversational aid in aligning interviewers on what to look for in candidates.
Why are Job Targets important?
Setting job targets can help to eliminate bias in the hiring process. If a client doesn’t set job targets in advance, they’d sort through candidates using an assumption of who would fit the role best when they’re reviewing assessment results. There would be very little objectivity in the process. By setting job targets, clients can use fit ratings to more objectively determine which candidates are suited for the role based on predetermined requirements. Inherently, there is little bias in this process, resulting in better decisions made by the client.
The Job Assessment asks a set of questions about what behaviors are needed for the job, and each of these behaviors is tied to a factor. As with the PI Behavioral Assessment, raw factor scores on the PI Job Assessment are tied to pattern scores. This means that if you choose many Dominance (A) behaviors and not many Patience (C) behaviors, you will see the same type of pattern in the PI Job Assessment as you would see in the PI Behavioral Assessment.
The best (i.e., most accurate and inclusive) Job Assessment results come from a combination of the opinions of multiple subject matter experts (SMEs). The results should be used as the basis for discussion amongst SMEs rather than as a finalized target. The Job Assessment is intended to serve as a robust aid in decision making but it should not be used in isolation of human judgment and conversation.
Completing the Job Assessment is highly recommended so that users establish a job target that accounts for key stakeholder inputs and adequately reflects the requirements of the role, which reduces the level of bias that occurs when determining candidate fit.
How was the JA developed?
The original version of the JA was comprised of 90 items that represented behavioral competencies in working adults. These competencies were aligned with the four factors that are measured with the Behavioral Assessment – Dominance (A), Extraversion (B), Patience (C), and Formality (D).
Upon receiving feedback that the original JA was arduous, long, or difficult for users to complete on their own and also that many items are rarely applicable to jobs, the PI Science team conducted research and shortened the JA to the 20 items we see today. Their goal was to create a new version that is more efficient and easier to complete for stakeholders, and that would subsequently motivate users to think critically about the requirements for the role and complete the JA.
The PI Science team collected multiple samples of responses to the job assessment and followed three major steps to develop the Short Form BA Job Assessment.
Over 600,000 responses to the Original JA were collected and analyzed. Selection rates, inter-item correlations (i.e., how well each item correlated with other items of the same factor), and item loadings (i.e., how well each item “loaded” onto its respective factor) were analyzed to determine whether each item was measuring the factor it was supposed to measure.
This analysis permitted the team to remove any item that was being selected at unusually high or low rates – specifically at a rate of less than 5% or greater than 95% – items that were not correlated with other items measuring the same factor, and items that were loading onto factors to which they should not load.
After these item statistics were examined, internal subject matter experts, including our science team and content team, reviewed the items for redundancies and peculiar wording. They identified themes for each factor, removed items that exhibited too much conceptual overlap, and lightly rephrased items for clarity. This qualitative review, in conjunction with the item analyses described above, allowed the team to reduce the original 90 items to 48 items.
As a second step, the Science team asked 150 PI employees to complete the 48 item JA with their own role in mind. Once responses were gathered, the Science team examined item statistics including selection rates, inter-item correlations, and item factor loadings. A handful of items exhibited correlations with multiple behavioral factors (i.e., they appeared to measure multiple behavioral factors) and our subject matter experts identified conceptual redundancy in some items. Removing items that appeared to measure multiple factors and reducing the conceptual overlap (while ensuring we retained at least 5 items per factor) allowed us to reduce the number of items to 20.
The third step was to test the final list of 20 items with a sample of working adults. The team gathered over 300 responses to this version of the JA from respondents on MTurk using a survey administered via Qualtrics. At this stage, the analyses confirmed that all of the items were performing the way we expected them to. To ensure that the 20 item version of the JA produced results in line with the 90 item version (i.e., to establish relative equivalence), the team collected a sample in which respondents completed both versions of the assessment for the same role. This step was meant to confirm that resulting job targets from the long and short version resembled each other.
- We compared selection rates for items in each factor – for example, are people selecting more A (Dominance) words in the original version?
- We compared the distributions of factors in the resulting targets to ensure normalcy – what is the distribution of A sigmas for each version?
- We compared the distributions of resulting reference profiles – for example, did the original version result in more targets recommending Mavericks?
Results of this analysis indicated that the targets resulting from both the 20 and 90 item versions were indeed comparable. Specifically, the item selection rates for each factor were equivalent.
The proportions of selection rates for factors A, B, C, and D were similar for each version. The distributions of targeted sigmas for each factor were also equivalent – the 20 item version did not result in targets that more frequently recommend significantly higher or lower sigmas for any given factor.
How to take the Job Assessment
You don’t need to invite every person who is on the interview team to take the JA. We recommend selecting three to five people who know the role best. Typically, this is the hiring manager, a top performer currently in the role, and someone from your recruiting team.
Once you select individuals to complete the JA, you can send an email notifying them in PI Hire. Alternatively, you can complete the JA independently. Your stakeholders will receive an email asking them to select the behaviors and cognitive abilities required for the role.
Ensuring success
To make the JA experience the most successful, we recommend the following process:
Ensure each stakeholder understands the role.
Hopefully each person is already well acquainted with the role, but there will be times that the hiring team is look for specific characteristics. Make sure each stakeholder has the job description and is aware of key characteristics or criteria that the hiring manager is looking for.
Reflect on the role and what it requires.
What kinds of behaviors and abilities have led to success in this role in the past? What are the strongest performers doing frequently? What kinds of behaviors will be required most often?
Be discerning in selections on the Behavioral JA.
After reflecting on what the role requires, only select behaviors that are the most critical for success in the role. If someone could be successful without a behavior, you probably don’t need to select it.
Only select the minimum level required of each cognitive ability of the Cognitive JA.
The Cognitive JA is intended to set a minimum threshold for cognitive ability required. For that reason, only select the minimum level required and identify if certain abilities aren’t required at all.
Remember that you will have a conversation to finalize the target.
Don’t worry about making the perfect selections. If you feel stumped, go with your gut and you can make final decisions when you align with other stakeholders.
Landing on the final target
When establishing a collaborative job target, we strongly recommend facilitating a conversation about the final job target once everyone has completed their JAs instead of simply accepting the target as it is. It’s possible that there are both points of agreement and disagreement so you should work with your team to determine what the final job target should look like. You may consider using the following process to finalize your target:
- Thank your contributors and schedule a time to discuss results. Once all stakeholders have completed their JAs, thank them for their input and ask if they have some time to discuss as a team.
- Review results and identify areas of alignment. Are there factors or specific behaviors that your team is aligned on? If so, you can likely accept the target for that factor.
- Identify areas of misalignment and facilitate discussion around it. When you look at results, are there behaviors that exhibit disagreement? For example, does one stakeholder think high Extraversion (B) is required, while another thinks it is not required? If so, pose questions to the group that probe their thinking. How outgoing or persuasive does the candidate need to be? Is the role more reserved and analytical?
- Consider exploring unique responses to aid discussion. You can review each person’s input on both the Behavioral and Cognitive JAs. It might be helpful to see what behaviors each person selected so they can explain their thought processes and disagreements.
- Once you feel aligned, confirm with each stakeholder and manually adjust the target as needed. After landing on an agreement for each targeted behavioral factor and your cognitive target, slide the target to your desired levels and set the target as active.
If you have additional questions or are struggling to land on a final job target when using the JA, you may consider reaching out to your PI Consultant to help facilitate conversation. Remember, this process is intended to provide a reliable concept of what kinds of behaviors and abilities are required for and most critical to success in the role.
When in doubt, come back to the question of “What will make a candidate the most successful in the role?”
Additional resources
To inform us of a typo or other error, click here. To request a new feature, click here.