Software bug in security system caused outages to SingPass and CorpPass

Senior Minister of State for Communications and Information Janil Puthucheary told Parliament on Monday (19 March) that investigations shown that a software bug in the server of a vendor was behind the two outages to SingPass last month, which disrupted hundreds of essential e-government services in what was the longest disruption to SingPass since it was set up in 2003.

He was responding to questions filed by Dr Tan Wu Meng (Jurong GRC) who asked the Prime Minister on the outcome of investigations into the interruptions and service degradations in SingPass and CorpPass services on 8 and 9 February 2018 and what redundancy mechanisms and systems are in place to maintain uninterrupted SingPass and CorpPass service provision.

The minister stated that the software bug in the system of the vendor, Dutch cyber-security firm Gemalto, manifested itself only after an enhancement to the SingPass and CorpPass system in January this year.

He added that CorpPass, used by businesses, was also affected by the outages.

Dr Janil said that the enhancement complied with all technical specifications and was properly tested, however, the interaction between the enhancement and the software bug caused some records to persist in the system instead of being automatically removed 30 days after they expired, which was the root cause of the slowdown.

He said that while the bug itself was elusive, the symptoms, slowdown in system performance, could have been detected earlier, adding, “Our early detection and warning capabilities can be improved. We intend to do so by enhancing the software checks and diagnostics so that in such cases, the engineers can act before the system condition worsens to a state that would affect users.”

Dr Janil also said that while the system had the hardware backup to deal with hardware and infrastructure failure, such redundancy did not address unknown internal software bugs of this nature, adding, “We will review the system design to improve all-round resiliency, beyond just hardware resiliency.”

The minister also said that the Government is reviewing its contracts with commercial providers to ensure that they adequately cover service outages.

The two outages which lasted about 10 hours in total, affecting many people as it disrupted hundreds of essential e-government services such as the filing of employees’ Central Provident Fund (CPF) contributions and work permit applications.

It was said that some Malaysian workers had to return home as their work permit could not be processed, while, companies risked fines because they could not file their employees’ CPF contributions on time.

According to experts, the main concerns are that the SingPass authentication systems are not robust enough and could potentially dent the public’s trust in the national digital identity system, a key Smart Nation project.

SingPass has more than 3.3 million registered users and supports 57 million e-government transactions, including the filing of income taxes, parking fine payments and foreign domestic worker applications. CorpPass, which was rolled out in September 2016, is meant for corporate transactions, including the filing of corporate taxes and work permit applications.