Post Office Horizon crash disconnected branches from datacentre

[TrueBlueTerrier]

https://www.computerweekly.com/news/366 ... datacentre

The Post Office has apologised to subpostmasters affected by a major IT outage, which stopped them from doing business

By
Karl Flinders, Chief reporter and senior editor
Published: 21 Aug 2024 10:29

The Post Office has revealed a disconnect between branch systems and a Fujitsu datacentre is what left subpostmasters unable to authenticate and log onto their systems yesterday morning. A backup datacentre was used to redirect traffic and restore branch operations by around 9.30am.

As reported by Computer Weekly, about 3,000 Post Office branch counters were unable do business at opening time on 20 August because they could not log into branch Horizon systems.

A spokesperson said the Post Office was first made aware of the issue at 6.45am, at which point its major incident manager engaged with Fujitsu on early investigations. Then at 8.00am, its own IT teams and those of its partners Fujitsu, Verizon and DXC worked to establish the root cause of the outage.

At about 9.00am, Fujitsu identified that certain network reverse proxies (NRPs)/load balancers were prevented from passing traffic to the datacentre from branches. All NRPs and traffic were transferred to a secondary datacentre, which brought traffic back to all branches at around 9.30am. 

The Post Office Horizon system and its problems are under the spotlight following the airing of a drama and documentary about the Post Office Horizon scandal, which raised public awareness. Today, many eyes are on the Post Office and its support of subpostmasters.

Mark Baker, former subpostmaster and Communications Workers Union (CWU) representative, said the Post Office fixed the issue “surprisingly quickly”.

“They did get things back up and running quickly, but the failure of the system which authenticates branch connections to the datacentre has happened before,” he added.

The Post Office apologised to those affected. A spokesperson said: “We apologise to any postmasters who were not able to log on to their counter this morning. We ensured postmasters were kept informed with the progress to resolve the issue by placing messages on Branch Hub and a text blast was also sent via one of our partners informing postmasters of the issue. We appreciate some customers may have been inconvenienced, but with many branches open until 5.00pm or later, we hope customers were able to come back later in the day to purchase our products and services.”

But Sheffield subpostmaster Richard Trinder, a representative at the Voice of the Postmaster campaign group, said communication from the Post Office was “non-existent”. “They said they sent out a text blast, but none of us received it,” he said.

The Post Office is in the midst of an IT crisis as it attempts to replace the Horizon system, which has been at the centre of a major scandal that saw thousands of lives ruined. Many subpostmasters lost their livelihoods and about 900 were wrongly convicted of financial crimes based on data from the error-prone system used in about 12,000 branches.

A project to replace Horizon was due to be completed in 2025, but will now not be fully in place until 2030. Supplier Fujitsu is likely to receive another five-year contract worth up to £180m to continue support for Horizon.

Earlier this month, the Post Office announced it is bringing in a new technology leader as it awaits nearly £1bn additional government funding for the troubled project. The organisation’s chief transformation officer, Chris Brocklesby, is leaving on 6 September at the end of his one-year contract, and will be succeeded on an interim basis by Andy Nice, formerly transformation director at Camelot, the company that ran the National Lottery until January 2024. Nice starts on 23 August.