By Gangasudhan
On 22 September, the Institute of Service Excellence (ISES) which is a part of the Singapore Management University (SMU) published a report on customer satisfaction called the Customer Satisfaction Index of Singapore (CSISG). It is in its ninth year of publication and purports to use data from “13,355 unique responses covering 206 companies”.
In this report, it says that the satisfaction score for SBS Transit went up from 63.9 to 66.5 and for SMRT, the score went up from 61.5 to 66.3 (as shown below). This was picked up by the mainstream press who mentioned it as though it was some sort of improvement without drawing any context. The Straits Times was even helpful in explaining it as “Despite reports of MRT service disruptions from time to time, consumers have grown more satisfied with the MRT as well as the overall land transport sector in the past year.”
However, on closer inspection, there are dubious misgivings about the survey methodology and the way it has been portrayed in the press. First of all, the findings are based on surveys done in a narrow timeframe – data on MRT satisfaction is collected between April and June of each year and thus if there is a lull period in the ever-reliable breakdown schedule, the survey findings would be impacted. This was actually noted by ISES as a possibility for the improved score, since data was collected “before the Jul 7 SMRT train disruption, which ISES said “may otherwise have negatively impacted commuter satisfaction”.”
Questionable Respondent Numbers
The bigger issue may be the actual numbers used to derive the customer satisfaction of SBS Transit and SMRT. The survey methodology described by ISES states that “Typically 100-200 respondents per company would have answered the CSISG questionnaire”, which means that at best, only 200 people were actually surveyed on their experience with SMRT and another 200 on SBS Transit. Essentially, the customer satisfaction of the MRT system in 2016 could be based on as little as 200-400 respondents.
Another issue with the survey methodology is that it is unclear whether these 400 respondents were from the 7,076 face-to-face surveys, 2,295 tourist surveys or 3,984 online surveys. While the face-to-face surveys were weighted to be representative of the Singapore population, the others were unlikely to be so and if a large number of the 400 respondents were tourists, for example, then the impressions formed would be quite different as compared to Singaporeans who have to rely on public transport daily.
When TOC wrote in to ask ISES if the respondent breakdown could be confirmed, we did not receive a response. Only when a second email was sent to SMU’s corporate communications department was there a reply. Unfortunately, ISES has chosen to ignore our questions altogether and merely referred us back to the published findings on their website (see the email transcript at the end of this article for the full exchange).
The decision to leave questions that go towards the validity of their customer satisfaction index deliberately unclarified is troubling to say the least. Coupled with the fact that ISES has been forthcoming in replying queries from the mainstream media, there is no option but to assume that there must be some truth to the possibility that the number of respondents and the profile of those respondents are not representative of regular Singaporeans.
Misleading Index
It appears as though the CSISG might be nothing more than an amateurish attempt to present some sort of findings in the mainstream media to appear relevant. This might explain the disconnect between anecdotal evidence from actual Singaporeans and the CSISG report that tells us the exact opposite, as far as the customer satisfaction on the MRT system is concerned. It might also explain why the customer satisfaction for SMU is noticeably higher than all the other universities in Singapore (ISES is part of SMU).
Unless further information is made available on the methodology and the breakdown of the respondents for each cohort, we can only imagine that the 13,355 questionnaires collected are just smaller sets of surveys of 100-200 each per company. When placed in that context, it is hard to take it seriously or believe it to be representative of ground sentiment. Just as we should not hand over money to someone claiming to be the Police without any proof, we should also not hand over our trust to surveys that claim to report findings without any proof.
Email Sent by TOC to ISES on 25 September 2016:
Hi, we noticed in the summary you provide on the CSISG 2016 Q2 Results at http://ises.smu.edu.sg/sites/default/files/ises/CSISG%202016%20Q2%20Media%20Deck.pdf that 13,355 questionnaires were completed, 206 entities were measured and findings for 40 entities were reported. It also mentions that 100-200 respondents would have responded for each company.
We would therefore like to clarify if this is an accurate description:
- a) about 8,000 questionnaires were collected covering the 40 entities reported
- b) further 5,000 questionnaires or so covered another 166 entities
- c) at least 100 and not more than 200 questionnaires covered the entity SMRT
- d) at least 100 and not more than 200 questionnaires covered the entity SBS Transit
- e) the breakdown for the satisfaction scores for the specific MRT/LRT Lines are based on the questionnaires covering the entities SMRT and SBS Transit, which number at least 200 and not more than 400 together
We would be pleased to receive the exact numbers for the following:
1) Total number of questionnaires used to analyse the 40 entities
2) Number of questionnaires used to analyse the entity SMRT
3) Number of questionnaires used to analyse the entity SBS Transit
Finally, we would like to understand if these sub-groups were adjusted to represent the Singapore population demographics? For example, does the analysis guarantee that all those who responded for the entity SMRT were not just tourists at Changi Airport?
Second Email Sent by TOC to SMU on 27 September 2016:
Hi, we emailed the department involved on Sunday regarding their reported findings and have yet to hear from them. We are preparing an article to describe the context of the survey findings, and would thus appreciate if the clarifications in the preceding email could be addressed. Otherwise, we will have to indicate that there was no reply from SMU regarding valid queries. Thank you for your assistance.
Reply from ISES to TOC on 27 September 2016:
Apologies for a delayed response. Thank you for your query. We are unable to offer more info than what’s available in the public domain. Kindly refer to the Executive Summary of the study which has been posted on our website.
Best Regards,
CHEN Yongchang,
Head, Research & Consulting
