This is not a test: interpreting the results of psychometric tools in coaching
Coaching Today, October 2022

My career in executive coaching and coaching supervision includes 12 years’ experience working with personality assessment tools, during which time I developed a special interest in the use of these instruments. Over the years, I have spoken to many coaches and human resources/learning and development professionals about the use of personality assessments in the context of personal and professional development, which led to the ideas expressed in this article.

My co-author Rink Hoekstra is an assistant professor in the field of behavioural and social sciences, with a special interest and expertise in statistics and psychometrics. He contributes here with particular regard to the clarification of some common misunderstandings around psychometric instruments.

Together, we explore the different ways personality assessment instruments are used by coaches in relation to how such instruments should be regarded, based on their characteristics.

Three ways of deploying personality instruments in coaching

In coaching, as in a variety of other employee development initiatives in organisations, personality assessment instruments are often used as tools for understanding and positioning someone’s personality and behaviour. Although some coaches might initially start working with a personality assessment tool for practical reasons (such as client requirements), in my experience, the majority of coaches who use a certain instrument do so because they believe it helps them to understand their clients better or more quickly, thus improving the coaching process. 

The value an instrument has for a coach is often related to the way the instrument is positioned in their coaching work. During my years in executive coaching and coaching supervision, I have seen different ways personality assessment instruments are deployed, and which for the sake of clarity I have divided into three categories: unconditional/explanatory, utilitarian/contextual or open/explorative. The adjective before the slash refers to the coach’s perspective, and the subsequent one to the approach taken. In practice, elements of different approaches are frequently combined.

Unconditional/explanatory

When a coach unconditionally accepts a personality assessment instrument, they presume the instrument reveals their client’s true personality. The coach discusses the topics the client brings to the table in relation to their profile scores or types, in order to clarify and explain things for the coachee. In general, a coach with this approach does not question the behaviours predicted by the client’s personality profile. If the client does not directly recognise the behavioural associations mentioned, the coach, for instance, would assume these behaviours simply might not yet have manifested themselves, or the client might not be fully aware of their own behaviours.

Coaches with this approach tend to use conclusive language, such as ‘According to your profile, you are at risk of…’ or ‘You seem to have a talent for…’, referring to what the instrument is supposed to measure and predict. Such an approach sits well with a more directive coaching style and may come across as very convincing, assuming the coach knows the instrument inside out and has a thorough understanding of human behaviour.

Utilitarian/contextual

For a coach with a predominantly utilitarian attitude towards a personality assessment instrument, the instrument is not necessarily very meaningful, though it may resonate with them to some extent. Reasons for using it might be that the instrument helps to structure the session and/or that it provides a model and language that make it easier for a coachee to make sense of their experiences. A coach with this approach would probably employ slightly cautious language that tones down the importance of the instrument, such as ‘According to this instrument…’, ‘Other people with your profile generally have…’, ‘You might recognise…’ etc. When a coachee does not directly recognise a tendency or behaviour, the coach may emphasise that some behavioural tendencies apply in certain contexts, but not in others. This approach may coincide with a variety of coaching styles.

Open/explorative

A coach with an open/explorative deployment of a personality assessment instrument may even prefer not to use them at all, because they don’t add any particular value to their way of working. However, they adopt an open attitude towards applying them when requested by their clients. A coach working in this way would be hesitant to explain the meaning and interpretation of the personality profiles of their coachees, and focus instead on the way the coachee interprets their profile: what resonates with them and what comes to mind when they try to make sense of it all. The instrument here, then, is nothing more than a catalyst to inspire a meaningful, helpful conversation. Whether this approach is convincing fully depends on the coach’s ability to hold the possibly uncomfortable state of not having an opinion about the interpretation of the profile that is provided by the publisher of the instrument.

Although there is no sound empirical evidence that work-related personality feedback interventions positively impact job performance,¹ according to how coaches generally explain the use of a personality assessment instrument, they believe that it does add value to the coaching process to include personality feedback in coaching. This is generally based on indicators observed during the coaching work that a coachee is improving in areas seemingly related to job performance, like self-awareness or self-confidence. But as we have learned from coaching research, the tools and interventions used by coaches are not the major determinants of the extent to which coaching works,² and the improvements could just as well have happened without the intervention of a personality instrument. Having said that, it is perfectly understandable that coaches sometimes choose to work with personality instruments when these instruments have a good reputation and seem to add value to coachees, as well as to the coaches themselves. However, surprisingly, even personality instruments with excellent reputations are not the robust predictors of behaviour they claim to be, in particular at the individual level. In the following paragraphs, we will try to make our case for greater caution with regard to the use of personality instruments in coaching, based on aspects we believe are not always covered in certification training.

What does it mean when a personality profile has changed?

We assume that reliable personality assessment instruments are useful, but there is a lot to be said about reliability that we may not be aware of.³

Here, we will limit this discussion to a few important aspects. We expect a reliable instrument to produce similar outcomes when used again with the same person, but what exactly do we expect and why? For instance, what time frame do we have in mind within which we expect to see similar results? And how much change do we find acceptable to consider the profile as similar enough?

In practice, a coach can be confronted with questions regarding reliability when a coachee has already received a profile produced previously by the same personality instrument, and their profile has now changed. In fact, at least four factors can contribute to such a change:

• The quality of the instrument
• Changes with regard to the instrument, usually psychometric improvements
• Changes with regard to norm groups used
• Real personality changes.

With respect to the first three factors, publishers should provide sufficient clarity and transparency to users to assist them in their interpretation of profile changes. Unfortunately, this is not always the case. With regard to the latter factor, within the field of personality psychology it is now believed that personality is relatively stable, but not fully. For instance, it is demonstrated that even across a relatively short time period, people can change with respect to the five traits most accepted by psychologists, the so-called ‘big five’: extraversion, openness, agreeableness, conscientiousness, and neuroticism.⁴

Lately, some publishers have changed their stance with regard to the changeability of personality profiles. A few years ago, I spoke with coaches who were convinced the particular instrument they used measured stable personality characteristics; however, now the publishers state that within a year most profiles change, and they recommend that people ‘rediscover themselves’ by taking the questionnaire again.

So, when asked by a coachee if it is possible that their personality has changed, the most honest answer is ‘yes’, but we still do not know exactly which aspects of personality change and to what extent, and what factors influence these changes. A coachee might also come up with their own explanation, related to situational changes or to their personal and professional development. They might even dismiss one of the two profile results because they are convinced that on one of the two occasions they used the tool, they were in a particular circumstance or mood. It would be arrogant to disregard such an explanation given the actual state of our knowledge.

How is validity relevant to the use of personality instruments?

Of course, we are not only interested in reliability, which is no more than the basic requirement of any well-developed instrument. In particular, we want to know whether a personality instrument assesses something meaningful and tells us something new about a person, for instance, behavioural tendencies that might jeopardise careers, or talents they may not yet have explored. Most coaches therefore prefer to work with personality assessment instruments that are based on sound psychological theory and have been tested for validity. But what does validity entail? There is no one single indication of overall validity. All aspects of validity have different meanings and are all relevant, but coaches are predominantly interested in predictive validity: what do the scores tend to predict about a person that we do not yet know?

We therefore usually rely on correlation tables, representing statistical relations of certain personality characteristics as measured by an instrument with other relevant variables (the things we would like to predict, like performance or behavioural risks). We are often informed that we need to look for significant correlations, but this is quite a misleading term. The word significance, as it is used in statistics, only tells us that if there was no correlation whatsoever, the odds of finding the effects we found, or more extreme ones, are relatively small. It suggests the representation of a real, structural connection between the two variables, even though it does not tell us anything about the strength of this connection. In the area of personality assessments, a correlation of, for instance, .30 between two characteristics is considered quite good, but we should take into account that this means it only explains about 9% of the variance of the behaviour we are interested in. The other 91% is simply not known to us! And even the word explain should not be taken literally, because correlation and causality are by no means the same. So, saying that one causes the other is merely a thought based on a theory that might be plausible, but cannot be proved (unless it were in a controlled experiment in which we would randomly assign personalities to people, something that is virtually impossible).

Reliability and validity certainly matter, but…

To what extent do reliability and validity matter? I would say they do, but that depends most of all on what the instrument is used for. First of all, in the context of hiring decisions (which is what personality assessments were developed for in the first place), well-developed personality assessment instruments help to make better decisions on a group level than some other commonly used instruments, such as unstructured interviews. Also, correlations between personality traits and certain behaviour may be useful for showing trends in the general population. But we should keep in mind that this does not automatically translate into statements about individuals.⁵ Knowing the correlation between two variables (such as neuroticism and leadership success) only marginally increases our prediction about someone’s chances of being or becoming a good leader, and even more so when the person scored average on a scale for neuroticism.

Other things that matter on an individual level (but which are statistically accounted for at a group level) are related to the way in which someone completed the questionnaire. For instance, individual answering tendencies, wanting to make a good impression, or a feeling of having answered most of the questions arbitrarily because one could not decide which answer best described them, cannot be fully ‘corrected for’ on an individual level, even though some people believe this is the case. This makes the question of how someone filled in the questionnaire relevant, especially when a coachee does not recognise their own profile description. However, the question of how the coachee filled in the questionnaire is not always asked, because some coaches more often tend to look for substantive, meaningful explanations before considering technical explanations. But both are equally relevant.

How do we deal with all this?

So where does this leave us, knowing that even when working with personality instruments that are considered to be reliable and valid, we cannot fully rely on a given interpretation of a profile with regard to current or future behaviour? Depending on a coach’s stance regarding the use of personality assessment instruments in coaching, one option is to consider a personality profile resulting from a questionnaire as a possible way to look at the self, which needs to be reflected on, in order to see in what way this makes sense to someone (or not) and to discuss this against the objectives of the coaching. If work-related behaviour is associated with the coachee’s profile, this needs to be checked within the work environment of the coachee, taking into account the fact that different people can have different perspectives on the same behaviour. Another option would be for coaches to consider the profiles as constructed stories about their coachees, to which a coachee can relate in many ways. The focus of the coach’s work then becomes how people relate to these stories, make meaning of them and create their own. With this approach, a coach would use the profile merely as an almost arbitrary stepping stone for meaning-making. 

Nevertheless, whatever one’s stance on personality assessment instruments, the nature of these instruments never justifies the idea that a personality profile resulting from a questionnaire – no matter how well developed – represents the (only) truth about a person.

Even if we use these tools cautiously, we owe it to our clients to be transparent about the psychometric qualities of any given instrument, and we should be able to tell them where this can be verified. Preferably, these qualities are reviewed by an independent authority, such as the British Psychological Society (BPS), and not merely conducted and published by the owners. We would encourage coaches who use personality assessment instruments to actually read these reports. Also, coaches would benefit from reading up on recent studies with regard to the instruments they use, critical as well as favourable, as a coachee might also have read these studies.

Ideally, we believe it would be preferable to be circumspect regarding the use of personality assessments in coaching. That said, we are aware of the positive experiences many coaches, as well as coachees, have had using them, and we expect that coaches will continue to use these instruments for years to come. With this in mind, let us at least try to adopt a little humility with respect to the value of these instruments, however sophisticated they might appear. After all, the best predictor of success in coaching is how we relate to our clients, not the tools we use.