Book on Employee Surveys By Peter Hutton Revised edition, 2009
ISBN 978-1-4452-4688-8 Retail price £14-90, 174 pages paperback
Available from www.Lulu.com and all leading booksellers
In this revised edition of his book, first published in 2008, Peter has taken the opportunity to correct a couple of inaccuracies that emerged after publication, sharpen up some of the arguments and add additional information particularly on how to respond to survey findings.
As before, the book challenges some of the key assumptions underlying employee surveys and argues that most surveys fail to identify the key issues for the business because of the way their questions are constructed. Many measures of ‘employee engagement’ are misleading, and the claims made by some leading consultancies for the superiority of their questions are without foundation.
What this book says about…
- Questionnaire design
- Agree/disagree scale questions
- Gallup’s ‘Q12’ questions
- Best Companies’ questions
- The use of multivariate analysis
- Norms and KPIs
- Measuring employee engagement
Questionnaire design
Good questions reveal what is going on. Bad questions obscure it. Good questions point to solutions, bad questions do not. Good questions resonate with staff. Bad questions bemuse them.
There are literally thousands of different questions that can be asked in an employee survey. However, they actually reduce to just three types: Scale questions, Lists questions or Open-ended questions.
Instead of applying the full range of different question types available to the researcher, much of the industry has defaulted to using just one kind of question – the agree/disagree scale
Agree/disagree scale questions
Many of the largest international consultancies employ surveys that consist of ‘agree/disagree’ scale questions and virtually nothing else.
Consultants like agree/disagree scales as they lend themselves to compiling normative databases.
Questionnaires that list dozens of statements against the one agree/disagree scale are necessarily constrained in what they can ask.
If the agree/disagree scale is the only tool used in your employee survey, you will be excluded from understanding many of the things that are most critical in managing your organisation’s performance. Standardised statements were developed so that they made some kind of sense in any organisation, not so that they made very profound sense in any particular organisation.
While agreement with positive statements is generally reassuring, problems often arise with understanding what disagreement with those statements mean.
The use of the agree/disagree scale in employee surveys is often defended on the grounds that staff don’t have to think too much. To my mind, that defeats the object of asking your employees for their views.
Norms and KPIs
The more general kind of normative questions are likely to be the most useful and can be used as KPIs.
The danger is that a company can become so obsessed with improving its KPI measures that it stops fulfilling its true purpose.
The key thing to remember about KPIs is that they should only be a guide as to whether you are going in the right general direction.
Underlying the consultant’s package of questions is the major assumption that there is an ideal profile of attitude scores to which every company should aspire Your key measures may alert you to the fact that you have issues that need to be addressed, but rarely do they tell you what these issues are.
Multivariate analysis
There are major problems associated with multiple regression analysis that can render the outcomes quite worthless.
I have a real problem with the kind of model that says that one set of attitudes is driven by another set of attitudes.
I have seen ‘key driver’ analysis demonstrating one set of priorities and more conventional analysis throwing up a completely different, and conflicting, set of priorities.
Although the factors (generated by from factor analysis) may help you to make sense of the kinds of patterns that appear to lie beneath the surface of your data, they are actually rarely very useful.
Gallup and its ‘Q12’ questions
All non-attitude statements had to be discarded, and so too did all questions measuring attitudes in any way other than with the agree/disagree scale.
Although the authors claim that Gallup analysed ‘one hundred million questions’ in reality it seems that only a small proportion of these were ever eligible for inclusion in the statistical analysis to find the final ‘Q12’ questions.
Perhaps the greatest weakness of the analysis is not so much in the analysis itself but the implied assumption of causality
Although high scores on these questions might be an indication that the company has ‘arrived’, that does not mean that these are the measures they need to tell them how to arrive.
The problem with asking only those 12 statements is that there is no further information to shed light on what you can do about any one of them.
There seems no substance in the claim that the ‘Q12’ questions ‘capture the most information and the most important information’ about the strength of your workplace.
Best Companies’ questions
Its questionnaire is very bad at measuring behaviours, motivations, and knowledge.
Because Best Companies only included statements that correlated highly with other statements, many important statements were probably excluded.
Despite Best Companies’ assertion that the factor analysis ‘determines the factors that employees filling in the surveys regard as most important to their engagement’, in my view, it does not.
A major concern with measuring tools that are used to draw up ranked listings of companies is that the measures totally ignore what the management of those companies are aiming to achieve.
Best Companies’ claim to have ‘the most accurate and valid survey instrument in the UK for measuring employee attitudes to their work and their organisation’ just does not hold up.
Measuring ‘employee engagement’
All the major consultants working in this area talk about ‘employee engagement’. But are they really talking about ‘engagement’ or is it just a convenient hook on which to hang their wares?
None of them provides any convincing proof that … their measures are the best ones to help organisations manage their engagement.
The failure of many consultants to customise their questions for individual organisations imposes a major constraint on their ability to understand employee engagement in any particular organisation.
To understand the current level of engagement, and develop appropriate improvement initiatives, it is necessary to break out of the agree/disagree scale measurement straightjacket.
‘Employee engagement’, as used by Best Companies, appears to be no more than a convenient label linking its analysis to a term that is widely used by HR managers.