Employee assessments are neither universally used nor always trusted by hiring and training professionals. This seems to be partly due to misconceptions about the strengths, and weaknesses of assessments. Therefore, we present insider information that debunks common beliefs about assessments and can help companies decide whether the use of assessments is in their best business interests. 

Have you ever been faced with a recruiting, employee relations or development situation that posed the following challenges....

• How can I identify and retain top talent?
• When doing succession planning, who are the best candidates for promotion?
• What are the strengths and weaknesses of a given work team, and how can I maximize that team's performance?
• How can I hire in a way that will reduce turnover?
• How do I tell whether a candidate will perform well in our company culture?

Small and large businesses alike do not consistently resolve these issues in effective ways.  Although assessments can significantly streamline and improve recruitment and training, they are neither universally used nor always trusted by professionals. From our experience, this is partly due to misconceptions about the strengths and weaknesses of assessments. Instead of fiction, this article presents the inside "FAQs."

Myth #1: Employee assessment is a fad

Reality: Informal "testing" has become a fad in many ways, such as the wealth of "quizzes" online and offline that are offered for entertainment purposes, self-awareness or finding compatible, romantic partners. But, employee assessment in one form or another has a long history.

In particular, the use of psychological instruments for screening, selecting and training employees has its formal roots in the early 1900s, and it achieved considerable momentum during the emerging corporate culture of the 1940s and 50s. Despite clear benefits from legitimate testing, many companies fail to conduct any type of assessment during the recruiting process. In doing so, these companies miss out on a valuable opportunity to gauge candidates' intellectual abilities and the likelihood that their personality and skill set are suited to the job and a company's culture.

Myth #2: Psychological profiling is not a real science

Reality: A profile refers to a configuration of variables and traits that are known to be important in certain situations. For instance, being a good manager surely requires one to "know one's business." But, what role do other factors play, such as a person's leadership qualities, ability to guide group dynamics or sense of fairness? When all such variables are considered simultaneously we have a profile.

Of course, profiling can be done wrong. When based on no more than intuition or conviction, profiling is no better than flipping coins (and often worse). However, when based on relevant, representative, and reliable data obtained via valid tests, profiles have much to recommend for themselves as they capture the best combination of skills, training, education and personality that will maximize job performance.

Myth #3: Assessments are not fair to all test takers

Reality: Sad to say: this is often true. Even when tests are constructed with the best of intentions, subtle (and sometimes not so subtle) biases can occur. Such biases may be related to test takers' sex, ethnicity or age simply because of a choice of words – some of you may remember the days when "cool" meant (almost) cold, and "phat" sounded like a derogatory term. Also, tests can be unfair because they require respondents to know things that have little to do with the purpose of the test.

For instance, it is one thing to frame mathematics questions in terms of sports (e.g., "A team of eleven players loses two members, how many remain?"). But, unless one is selecting a new coach it is not fair to assume that all test takers will know the details about a particular sport (e.g., "A soccer team loses two members, how many will remain?"). By the way, the preceding is one of the main reasons why most assessments cannot simply be translated into another language for use in different cultures.

But, the good news is that tests can be made fair, and that there are statistical methods to detect and correct for test biases. Unfortunately, too few test constructors use such methods to date. Right now, bias testing (and bias removal) is familiar mostly to those using advanced test construction methods such as Item-Response Theory (IRT).

Myth #4: The use of employee assessments increases the legal liabilities in the recruitment and hiring process

Reality: Actually, legitimate assessments help ensure that organizations do not illegally discriminate against candidates and incumbents as required by the Uniform Guidelines on Employee Selection Procedures. Unlike more subjective evaluations such as interviews or even reference checking, valid assessments are objective appraisals of a person's knowledge, skills, abilities, attitudes or personality traits. Moreover, every time a company hires without practicing due diligence, that firm may be accepting liability for the employees' actions - even when they are "off the clock." Thus, arguably there is the potential for more liability in the absence of legitimate assessment.

Myth #5: Assessments require a psychologist to interpret properly

Reality: Trained professionals are indeed required to interpret statistical findings accurately, but not all psychologists or social scientists have adequate training in statistics – much less tests and measurements. Furthermore, customers repeatedly report that they find the most useful employee assessment reports to be ones that avoid psychological and statistical jargon and instead explain results in terms of applied contexts and business-related goals.

Myth #6: Assessment feedback is too general or vague to be useful

Reality: If by "feedback" one means a mere summary of test scores – perhaps accompanied by a qualitative label or two – this is often not a myth: it is actually true. Fortunately, the advent of modern techniques derived from advanced test theory (i.e., IRT) also brought new and surprising ways to look test takers' answers. In particular, when tests are computerized, IRT can be used to find "outliers" among the answers, and such outliers can be exploited for diagnostic purposes.

For instance, in addition to knowing that Candidate A scores "high" on "leadership," is not also valuable to know that Candidate A seems to abhor all forms of conflict and that he or she lacks in decisiveness? Well, these are the kinds of questions that can now be answered.

In our experience, many employers actually value such detailed information as much as (and often more than) the typical information provided by most standard assessments. Diagnostic feedback will also point to applicants' or employees' strengths and weaknesses of importance for future training purposes.

Myth #7: Assessment is a cost-effective replacement for the traditional job interview

Reality: Assessment is generally a cost-effective addition to any recruitment effort, considering that assessment outperforms traditional behavioral interviews 4:1 in predicting job performance. However, due diligence requires more than just giving an employee assessment, regardless of its quality.

Accordingly, assessments should never be used in isolation, but always in conjunction with a wider process whereby a candidate's experience, education, qualifications, competence and trainability can accurately be assessed in a larger context. This wider process of due diligence may include background checks, proper reference checking, structured behavioral interviewing and job-specific skill testing.

Myth #8: Pencil-and-paper assessments are no different from online assessments

Reality: Actually they can be markedly different, and frequently vendors do not properly adapt paper-and-pencil tests for online use. For instance, web versions almost certainly have different layouts and response contingencies than do paper-and-pencil tests. As well, when older tests are used, the meanings of its wordings may have changed over time (e.g., consider how the vernacular versions of "cool" and "hot" have fluctuated over the last decades).

Finally, differences in the method of administration (i.e., offline vs. online) may systematically affect test takers' reactions to the questions. It is mandatory, therefore, that web adaptations of paper-and pencil instruments be re-administered and recalibrated based on online administration. Moreover, the exact same item format and layout should be used during pilot testing and operational use. Finally, online tests should not rely on norms that were established by paper-and-pencil methods.

Myth #9: All assessments are basically the same

Reality: Different testing products may draw on similar psychological theories, but products vary greatly in reliability and validity due to their methodological and mathematical approaches. There are two common methodologies: a self-referential versus an inter-individual approach.

Some of the most popular products like the Myers-Briggs Type Indicator, Predictive Index and the DISC assessment are actually the least sophisticated, because such instruments describe individuals only in a self-referential way, i.e., against themselves. Feedback from these assessments is limited to what the test taker wants that measure to say. Normative instruments, by comparison, are inter-individual. This means they describe test takers against a reference group.

But "norms" per se are not enough to ensure reliability and validity of an assessment; instruments must also have a mathematical basis in modern test theory. This is the same statistical gold standard used in well-known assessments like the GRE, MCAT and LSAT. Unlike outdated methods used by most assessment vendors, true scaling approaches generate proper normative instruments that take into account response biases related to age, gender, cultural background and employment level of the test taker. Besides greater technical precision and the protection of meeting legal requirements, modern testing theory also elicits richer information about test takers.

Myth #10: Customized assessments are better than standardized assessments

Reality: Because assessments are always written with a target audience in mind, they all are "customized" to some extent. In principle, customization could continue down to the lowest level of an organization. Unfortunately, the return on investment will diminish quickly. For instance, customizing an assessment designed for businesses "in general" to be one that deals primarily with service organizations will improve this assessment's quality for the service industry. However, there is considerable expense involved in doing so.

As the answers to the preceding myths indicate, the content of the assessment will have to rewritten and then recalibrated on new samples of respondents. Likewise, new scaling runs will have to be performed, together with extensive analyses for potential response biases. Those who can afford all this are indeed in the enviable position of having a high-quality, customized assessment. Often, however, customization involves little more than a face-lift where the questions are reworded, but little else is done. In such cases one ends up with an impoverished version of the original, with unknown norms, doubtful psychometric qualities, and possibly, biased results. If so, it would have been far better not to customize at all.

To be fair, it is important to point out that not all of the myths above stem from the general public or assessment critics. Some myths are actually perpetrated by assessment vendors in order to sell their particular products.  Can you spot which myths? – contact the first author for the answer.

To protect consumers and test takers alike, professional testing standards require that all assessments and surveys have a Technical Manual or other supporting documentation that describes their psychometric properties, intended purposes and score interpretation.

We highly recommend that you disregard any employee assessment you are considering for your company, if that product is not based in modern test theory.  Indeed, the stated benefits and effectiveness of such outdated and unsophisticated "assessments" tend to be myths, not realities. 

Notes 

This article was based on material presented in:

Embretson, S. E., & S. L. Hershberger (Eds.) (1999). The new rules of measurement: what every psychologist and educator should know. Mahwah, NJ: Lawrence Erlbaum.

Houran, J., & Lange, R. (2006). State-of-the-art in measurement in human resource assessment. HVS Journal. 28^th Annual NYU Hospitality Industry Investment Conference, New York, NY, June 4-6.

About the Authors 

James Houran holds a Ph.D. in Psychology and recently joined HVS to head the 20â"‚20 Skills™ assessment business. He is a 15-year veteran in research and assessment on peak performance and experiences, with a special focus on online testing. His award-winning work has been profiled by a myriad of media outlets and programs including the Discovery Channel, A&E, BBC, NBC's Today Show, Wilson Quarterly, USA Today, New Scientist, Psychology Today, Forbes.com and Rolling Stone.

For information on the Best Practice 20â"‚20 Skills™ assessment system and industry surveys and analytics, contact:

James Houran, Ph.D.
jhouran@2020skills.com

Rense Lange holds a Ph.D. in Psychology and a Masters' in Computer Science, both from the University of Illinois at Urbana-Champaign. He is one the world's foremost expert in tests and measurement and applied Item Response Theory and Rasch scaling, and Computer Adaptive Testing (CAT) in particular. In addition to serving on the faculty of the University of Illinois, the Southern Illinois University School of Medicine, and Central Michigan University, Dr. Lange has worked for ten years as the lead psychometrician at the Illinois State Board of Education.

www.2020skills.com