Provoking behaviour: training roleplayers at assessment centres

Assessment days for evaluating work-relevant behaviours ofapplicants or job incumbents often draw on actors to perform as difficultteam-members or curious clients in meeting simulations. A recent study hasshown that these role-playing actors can be trained to effectively weave pre-writtendialogue prompts into the improvised simulations. However, whether this helpsmeasurement of participant behaviours is less clear.

The study authors Eveline Schollaert and Filip Lievens gave19 role-players training, which in one condition included explicit guidance onusing behaviour-eliciting prompts during assessment exercises; for example,"Mention that you feel bad about it" in order to provoke behavioursrelating to a dimension of interpersonal sensitivity. Such prompts are often provided in prep material, but actual usage was unknown. The authors wondered whetherrole-players could realistically increase their prompt usage through training, or whether this istoo much to ask an actor in the thick of a dynamic interaction.

At a subsequent assessment centre, the role-playersinteracted in simulations with 233 students from Ghent University. Role-playerswith prompt training were able to incorporate four to five times more promptsthan those without such training, an increase from about two prompts perexercise to 10-12.

More prompts ought to elicit more relevant behaviours, so theauthors expected observers to get a better picture of true 'candidate'performance. But this isn't clear. In the high-prompt condition, pairs ofraters watching the same role-play didn't agree any more on their ratings,suggesting the behaviours remained just as obscured as without prompts. Thatsaid, there was better correspondence of some of the ratings to other measurementsyou would expect to be related - for instance, interpersonal sensitivitycorrelated better with an Agreeableness personality score acquired pre-centre.But half of the predicted increases in correlation weren't observed.

Regarding their unsupported hypotheses, the authors wonderwhether the rating assessors should also have been trained on prompt use toencourage sensitivity to candidate reactions. I have additional concerns on thenature of the assessors -minimally trained masters students - used to drawconclusions about a professionalised domain. Nonetheless, this rare examinationof role-player impact on face to face assessments suggests training cangenerate more dimension-focused contributions, which in turn may result inmeasurements with more predictive power.

Schollaert, E., & Lievens, F. (2011). The Use of Role-Player Prompts in Assessment Center Exercises International Journal of Selection and Assessment, 19 (2), 190-197 DOI: 10.1111/j.1468-2389.2011.00546.x

Can we get away with using lo-fi assessment to recruit advanced positions?

In recruitment, the promise of comparable results for less effort is understandably tempting. It's offered by the offsetting of costly assessments with alternative measures that use pencils, screens and standardised questions instead of expert assessors. However, as some sources suggest a bad hire can cost twice or more that position's annual salary, the stakes are high. A new study kicks some assessment tyres to see whether that bargain is actually a banger.

Researchers Filip Lievens and Fiona Patterson looked at recruitment into advanced roles which typically seek the skills and knowledge to hit the ground running. They took their sample of 196 successful candidates from the UK selection process for General Practitioners in medicine (GPs). To get here, you've completed two years of basic training and up to six years of prior education, by which stage you're after someone ready to go, not a future 'bright star'. Lievens and Patterson were specifically interested in how much assessment fidelity matters, meaning the extent to which assessment task and context mirror that in the actual job.

Three types of assessment were involved, all designed by experienced doctors with assistance from assessment psychologists. Written tests assessed declarative knowledge through diagnostic dilemmas such as “a 75-year-old man, who is a heavy smoker, with a blood pressure of 170/105, complains of floaters in the left eye”. Assessment centre (AC) simulations meanwhile probe skills and behaviours in an open-ended, live situation such as emulating a patient consultation; these tend to be more powerful predictors of job performance, but are costly.

The third was the situational judgement test (SJT), a pencil and paper assessment where candidates select actions in response to situations, such as a senior colleague making a non-ideal prescription. SJTs are considered by many to be “low-fidelity simulations”, losing their open-endedness and embodied qualities, but hanging on to the what-would-you-do-if? focus. The authors were interested in whether its predictive power would be in the same class as the AC simulations, or mirror the more modest validity of its pencil and paper counterpart.

The data showed that all assessments were useful predictors of job performance, as measured by supervisors after a year spent in role. Both types of simulation - AC and SJT - provided additional insight over and above that given by the rather disembodied knowledge test – each explaining about a further 6% of the variance. But in comparison with each other, the simulations were difficult to tell apart, with no significant difference in how well they predicted performance.

It should be noted that the AC simulations did capture some variance over and above the SJT, notably relating to non-cognitive aspects of job performance, such as empathy, which is important as such areas are less trainable than clinical expertise. However, this extra insight was fairly modest, just a few percentage points of variance. More expensive AC assessments can provide additional value, but the study suggests that at least in this specific recruitment domain, you can get away with a loss of fidelity if the assessments are appropriately designed.

Lievens, F., & Patterson, F. (2011). The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. Journal of Applied Psychology, 96 (5), 927-940 DOI: 10.1037/a0023496

How much should we trust job applicant ratings of their own emotional intelligence?

Self-rating is a popular way to measure emotional intelligence in the workplace. Under lab conditions it's been shown that these ratings vary depending on what your (imaginary) objective is: to give a 'true' picture or to successfully win a job. A new study translates this lab finding to the workplace, finding that applicants for jobs really do rate themselves higher on EI than counterparts already working in that organisation.



The study compared scores for 109 job applicants with 239 volunteers, matched by department and managerial level. They rated themselves on four classic components of EI: self emotion appraisal, others emotion appraisal, use of emotion, and regulation of emotion. Applicants significantly outscored incumbents in all areas, on average rating themselves more than a standard deviation better. The areas of greatest divergence were in use of emotions and regulation of emotions, which have much in common with the Big Five personality traits conscientiousness and emotional stability, which we know job applicants have a higher tendency to inflate.



On all but one of the components, applicant scores were significantly more bunched together than incumbent scores, which could be seen as additional support that they were manufactured, with candidates homing in on scores that were solidly good, avoiding suspicious high or unhelpful low scores.



The study is important because in other areas of research, score discrepancies can be found in the lab, due to different explicit instructions, that don't seem to surface in the real world, suggesting the overt nature of lab conditions can exaggerate or even manufacture differences. Yet here the effect is found again, suggesting that if we do want to rely on self-report to assess EI we should recognise that this inflation may take place, and that relying on the normative data that accompanies these tests may lead us to unrealistically high appraisals of candidates.





Lievens, F., Klehe, U., & Libbrecht, N. (2011). Applicant Versus Employee Scores on Self-Report Emotional Intelligence Measures Journal of Personnel Psychology, 10 (2), 89-95 DOI: 10.1027/1866-5888/a000036

Are job selection methods actually measuring 'ability to identify criteria'?

While we know that modern selection procedures such as ability tests and structured interviews are successful in predicting job performance, it's much less clear how they pull off those predictions. The occupational psychology process – and thus our belief system of how things work - is essentially a) identify what the job needs b) distil this to measurable dimensions c) assess performance on your dimensions. But a recent review article by Martin Kleinman and colleagues suggests that in some cases, we may largely be assessing something else: the “ability to identify criteria”.



The review unpacks a field of research that recognises that people aren't passive when being assessed. Candidates try to squirrel out what they are being asked to do, or even who they are being asked to be, and funnel their energies towards that. When the situation is ambiguous, a so-called “weak” situation, those better at squirrelling – those with high “ability to identify criteria” (ATIC) - will put on the right performance, and those that are worse will put on Peer Gynt for the panto crowd.



Some people are better at guessing what an assessment is measuring than others, so in itself ATIC is a real phenomenon. And the research shows that higher ATIC scores are associated with higher overall assessment performance, and better scores specifically on the dimensions they correctly guess. ATIC clearly has a 'figuring-out' element, so we might suspect its effects are an artefact of it being strongly associated with cognitive ability, itself associated with better performance in many types of assessment. But if anything the evidence works the other way. ATIC has an effect over and above cognitive ability, and it seems possible that cognitive ability buffs assessment scores mainly due to its contribution to the ATIC effect.



In a recent study, ATIC, assessment performance, and candidate job performance were examined within a single selection scenario. Remarkably it found that job performance correlated better with ATIC than it did with the assessment scores themselves. In fact, the relationship between assessment scores and job performance became insignificant after controlling for ATIC. This offers the provocative possibility that the main reason assessments are useful is as a window into ATIC, which the authors consider “the cognitive component of social competence in selection situations”. After all, many modern jobs, particularly managerial ones, depend upon figuring out what a social situation demands of you.



So what to make of this, especially if you are an assessment practitioner? We must be realistic about what we are really assessing, which in no small part is 'figuring out the rules of the game'. If you're unhappy about that, there's a simple way to wipe out the ATIC effect: making the assessed dimensions transparent, turning the weak situation into a strong, unambiguous one. Losing the contamination of ATIC leads to more accurate measures of the individual dimensions you decided were important. But overall your prediction of job performance measures will be weaker, because you've lost the ATIC factor which does genuinely seem to matter. And while no-one is suggesting that it is all that matters in the job, it may be the aspect of work that assessments are best positioned to pick up.



Kleinmann, M., Ingold, P., Lievens, F., Jansen, A., Melchers, K., & Konig, C. (2011). A different look at why selection procedures work: The role of candidates' ability to identify criteria Organizational Psychology Review, 1 (2), 128-146 DOI: 10.1177/2041386610387000