PI Diagnose: How questions and categories were developed

PI Diagnose empowers users to collect detailed, actionable feedback about the employee experience. As with any engagement platform worth its salt, Diagnose was built on science and diligent research.

In this article, you’ll learn more about the science behind PI Diagnose and the studies that helped shape the software.

We’ll cover:

• The PI Employee Experience (XP) Survey
  • XP question categories
  • XP item benchmarks

• Pulse surveys (new to Diagnose)
  • Pulse survey questions
  • The Overall Pulse Check

The PI Employee Experience Survey

The centerpiece of Diagnose—the PI Employee Experience (XP) Survey—is a 50-item survey designed to measure a wide variety of employee engagement drivers. The survey was created by a group of Science PhDs with backgrounds in organizational psychology, psychometrics, and education.

To develop the survey, these scientists wrote a pool of nearly 100 questions, which they then administered to a sample of approximately 3,000 U.S. working adults. This sample—which included respondents representing a variety of backgrounds, ages, titles, and industries—was chosen to reflect the greater U.S. working population.

The data contained the following industries:

• Business/Professional Services
• Computer Software
• Construction
• Consulting
• Education
• Energy
• Finance/Insurance
• Government
• Healthcare/Life Sciences
• Hospitality/Food Services
• Manufacturing
• Other
• Real Estate
• Retail
• Sports/Entertainment
• Technology/Telecom
• Transportation/Distribution

Determining the XP categories

Once PI’s science team administered its ~100 developmental questions, the team examined the results.

Through exploratory factor analyses, the team determined that the possible drivers of engagement fell into four categories: job, manager, people, and organization (JMPO). Once these categories were identified, the team refined the list of survey items to 50 questions:

• 10 questions that measure the “Job” factor
• 10 questions that measure the “Manager” factor
• 10 questions that measure the “People” factor
• 16 questions that measure the “Organization” factor
• 4 additional questions that measure general engagement

Learn more about each of the JMPO categories below.

Creating XP item benchmarks

Each item in the XP Survey is designed to be actionable. The software allows users to see how their team or organization “scored” for each item, and how that item compares to external benchmark data. That way, users can see where their biggest strengths and caution areas lie—and take steps to level up the employee experience.

When PI Diagnose launched in 2019, its external benchmark data consisted of the same sample used during development (i.e., approximately 3,000 U.S. working adults). Benchmarks were calculated by taking the average of each item and category score across the full sample. 

In 2021, these benchmarks were updated to include the original sample, plus any user data collected since the release of Diagnose. To date, the software pulls from a sample of approximately 25,000 respondents from across 17 industries.

Note: As our sample data becomes more robust, the Diagnose team plans to roll out industry-specific benchmarks. As of now, however, we do not have adequate data across all relevant industries to formulate a holistic series of benchmarks.

Diagnose pulse surveys

While planning the latest iteration of PI Diagnose, our development team wanted to provide the same powerful insights existing users are accustomed to, while creating a much more flexible survey experience. 

The team quickly landed on a pulse survey experience, in which users can send out shorter, more targeted surveys, each dedicated to a specific JMPO driver (Job, Manager, People, Organization). 

Not only does this allow users to finetune the ways in which they measure engagement drivers, but it lets them administer surveys in more frequent intervals (e.g., weekly, monthly, or quarterly). Users can now track how specific elements of the employee experience fluctuate over time—and how those experiences may improve or degrade in response to certain events or actions.

Developing these pulse surveys

To create Diagnose’s pulse surveys, our science team examined the pre-existing questions for each JMPO factor and came up with “sub-factors,” which provide added insight into the drivers of engagement. In total, the team identified seven to 10 sub-factors per JMPO factor.

For example: When examining questions that measure “Job,” the team identified sub-factors such as role clarity, flexibility, and career development.

Once these sub-factors were determined, we assembled a team of PI experts—spanning science, product, and people operations—to refine this list based on timeliness and potential impact. The final list consisted of the following sub-factors:

• Job: Role Clarity, Meaningful Work, and Career Development
• Manager: Growth, Manager Relationship, and Recognition
• People: Accountability, Communication, and Trust
• Organization: Culture Health, DEI, and Transparency

After finalizing these categories, our scientists wrote a pool of 120 questions (10 per sub-factor). They then administered these questions to a sample of approximately 1,000 U.S. working adults, all working full-time. This sample—which included respondents representing a variety of ages, titles, locations, and industries—was chosen to reflect the greater U.S. working population.

The team examined the results, conducting confirmatory factor analyses to ensure:

1. That each question measured its respective sub-factor
2. That each question was indeed related to employee engagement

Based on these analyses, the team refined the list of questions and created 12 surveys, each four to six questions in length. From those surveys, the team selected five—Trust, Manager Relationship, Culture Health, Meaningful Work, and DEI—as Diagnose’s first series of pulse surveys.

Note: The Diagnose team eventually plans to release surveys based on all 12 sub-factors. For the latest release, we chose to focus on a smaller list of surveys to gather preliminary data and ensure we’re providing a high-quality experience for our users.

Developing the Overall Pulse Check

In addition to these new pulse surveys, Diagnose offers a longer, 15-question pulse survey known as the Overall Pulse Check. The Overall Pulse Check is based off the PI Employee Experience Survey, but can be completed in a shorter period of time. This makes it perfect for measuring engagement between larger XP surveys.

Unlike Diagnose’s other pulse surveys, which focus on a specific JMPO theme, the Overall Pulse Check includes questions that touch on all aspects of the employee experience.

The Overall Pulse Check is based on a sample of 25,000 U.S. adult workers, with respondents representing a variety of backgrounds, ages, and titles, and spanning 17 industries.