By Gangasudhan
Ask anyone who deals with statistics or who has been doing polls long enough and the chances are that they will let slip how utterly useless statistics are most of the time. While it’s a good mathematical tool in theory, often times these tools can be abused to the whims and fancies of the persons or organisations who commission them.
In the latest installment, a survey on the attitudes of Singaporeans towards National Service had been conducted by the Institute of Policy Studies (IPS), at the behest of the Committee to Strengthen National Service (CSNS).
Before we get into the survey itself, which is available directly from the IPS website for inspection, let’s consider the types of news headlines the survey generated. TODAY understood the survey as a call to “Let PRs, women contribute to National Service: Survey”, while it was a deceptively positive sounding “One in 10 S’porean women willing to do full-time NS: IPS” for Channel NewsAsia. The Straits Times meanwhile went for a rather convoluted “Primary mission of NS? To teach discipline and values to young, survey findings show” and finally, Yahoo Singapore went in a whole other direction with “2 in 5 NSmen say employers rather hire staff with no NS commitments: survey”.
As can be seen, reporters at different news outlets had varying points to highlight about the survey. The principal researcher, Dr Leong Chan-Hoong, also had much to add in the various news reports, to the extent of discounting some less-positive findings (i.e. the fact that 40% of NSmen polled felt that employers would rather hire someone who did not have NS commitments) with a “Two out of five is not a majority, and this is a preference rating” remark.
Indeed, taking the “this is a preference rating” qualifier by Dr Leong, the Straits Times’ headline and assertion in its article – that the “primary mission for National Service (NS) is to instill discipline and values among the young, according to a new study” – is actually incredibly misleading because the statement did not come from those polled but by the researchers who framed the study. What the study ultimately established was that most Singaporeans preferred this statement over the others in the entire list of 10 statements that was provided to them.
But over-simplistic generalization to sensationalize is not a new practice and this sort of varied interpretation is par for the course when it comes to drafting news reports on surveys. Surveys and polls are also often the choice of delivery system when trying to influence readers on various issues.
This issue of what parts of the survey get highlighted in the news is just one part of the problem where such official surveys are concerned. The bigger – and sometimes alarmingly misrepresenting – element comes from how the survey is designed.
Wrong scale used, or simply inadequate?
In this survey, the most telling is how the statements were crafted to elicit a response along the scale of Strongly Agree, Agree, Slightly Agree, Slightly Disagree, Disagree and Strongly Disagree. But the discussion of the findings largely lumped together the first 3 levels of ‘agree’ or the last 3 levels of ‘disagree’ (e.g. while the actual responses for one statement was 18.8% for Slightly Agree, 53.9% for Agree and 16.3% for Strongly Agree, the discussion simply adds them together as ‘89% agree’).
What this means is that when posed the statement of ‘My employer is supportive of my NS commitments’, the respondent would have deliberated across 6 different levels when in fact all that finally mattered was whether he or she generally agreed or disagreed.
Of course, the final analysis should really be an “unknown” until the ratings of the scale have been obtained and their leanings ascertained. However, in this instance of employer support for NS, the answer would clearly be a yes or no, and the variable more a matter of the conditions that demonstrate this support. For instance, beyond the simple act of allowing me to go for In-Camp Training, does my employer insist that I fulfill all my work commitments before I leave? Am I penalised during performance review for work that I missed during ICT? Does my supervisor allow time to catch up when I return from ICT? Do we have a corporate culture where covering for a colleague for ICT is a norm?
When we include such a breakdown of conditions, it becomes clear that even a scale of one to six is not sufficient to reflect the diversity that goes into how each individual makes a decision to rank their work-versus-NS status, let alone sufficient to be lumped into two extreme categories, as the final analysis has done.
It is not enough to just get a sizeable number of respondents “weighted by race, age and gender” and have face-to-face interviews alone. The statements themselves must be meaningful to the purpose and the way they are asked and answered ought to be reflective of true opinion. In this case, the researchers seem to have committed a glaring oversight, were surprisingly incompetent or intentionally made the survey design vague. Consequently, the analysis has been skewed towards “discovering” a certain (positive) outcome that takes advantage of the fact that most NSmen would deem it reasonable for employers to simply let them off for ICT.
Interpreting the survey carefully
Indeed, much of the survey seems to suggest that the researchers have approached the issue of National Service from a ‘do you agree with my assessment’ direction and does not allow respondents to share their own take on this far-reaching issue. Instead, responses are pigeon-holed into fixed statements that would have highly predictable outcomes along an Agreement Scale.
Nevertheless, some interesting points that could have been explored further include how ‘employed servicemen’ felt that while their employers generally supported their NS commitments (89%), it appears as though workload was adjusted for significantly fewer (77%), and the confidence in hiring practices that take into account NS commitments was not that strong (58%). This could mean that employers don’t walk the talk when it comes to supporting National Service on their part, or are doing the bare minimum to support NS efforts.
But perhaps one of the most disappointing aspects of this survey is how ‘Benefits and Impact of NS’ merely listed 10 statements and (again) just asked respondents to indicate their agreement levels for each separately. This exploration of attitudes would have been best served as a ranking order task where respondents should have been required to order the statements according to their importance or choose the 3 that they felt best represented the benefits of NS.
As it stands, the predictable outcome of all 10 statements having a generally positive response from the respondents does not offer any real insight into understanding Singaporeans’ attitudes towards the benefits of National Service. But the more perplexing observation is how the researchers simply dismissed the least popular statement as still garnering 90% agreement instead of discussing why the first 8 statements have an agreement level of 96% and above (with one other statement having 94%) while the statement ‘NS improves one’s civilian employment prospects’ only garnered 88.4%.
Another area that could have been explored further (rather than issue a blanket observation that suggests ‘things are generally good’) is the area of feedback and improvements on NS. When contrasted against the level of agreement for governmental recognition (95.4%), the relatively lower level of agreement for whether feedback on NS is taken seriously (91.4%) and whether feedback channels exist to suggest improvements (92.3%) gives much reason for further exploration. This is also bearing in mind that with a sample size of over 1,200, this 3-4% variance would have some examinable implications.
While it is convenient to ask Singaporeans whether they agree with generic statements about National Service, the analysis of this issue cannot be ‘conveniently done’. Surveys and research such as the one done by the IPS here only serve to inform the authorities what they would like to hear and does nothing more. Even with the limited ability of the survey itself, some findings actually gave hints on some of the problems that may exist, but unfortunately, the researchers chose to overlook these areas for reasons that are best known to them.
