Skip to Main Content

Find Case LawBeta

Judgments and decisions from 2001 onwards

Bates & Ors v the Post Office Ltd (No 6: Horizon Issues) (Rev 1)

[2019] EWHC 3408 (QB)

Case No: HQ16X01238, HQ17X02637 and HQ17X04248

Neutral Citation Number: [2019] EWHC 3408 (QB)

THE POST OFFICE GROUP LITIGATION

IN THE HIGH COURT OF JUSTICE

QUEENS BENCH DIVISION

Rolls Building Fetter Lane London, EC4A 1NL

Date: 16 December 2019

Before :

THE HONOURABLE MR JUSTICE FRASER

- - - - - - - - - - - - - - - - - - - - -

Between :

Alan Bates and Others

Claimants

- and -

Post Office Limited

Defendant

-

- - - - - - - - - - - - - - - - - - - -

Judgment (No.6) “Horizon Issues”

-

- - - - - - - - - - - - - - - - - - - -

Patrick Green QC, Kathleen Donnelly, Ognjen Miletic and Reanne MacKenzie for the Claimants (instructed by Freeths LLP)

Antony de Garr Robinson QC, Simon Henderson, Owain Draper and Rebecca Keating for the Defendant (instructed by Womble Bond Dickinson LLP)

Hearing dates: 11, 12, 13, 14, 18, 19, 20, 21 and 27 March;

3, 9 and 12 April; 4, 5, 6, 7, 11, 13 and 14 June 2019;

1 and 2 July 2019

Draft distributed to parties on 28 November 2019

-

- - - - - - - - - - - - - - - - - - - -

Mr Justice Fraser:

1.

This judgment is part of the long-running legal dispute between the claimants, who are all either former or serving sub-post masters and sub-post mistresses (“SPMs”), and the Post Office. All the SPMs in the litigation had contracted with the Post Office at different times to run branch Post Offices in different locations across the country. The Post Office introduced a computer system called Horizon in 2000 across all its branches. That was changed in 2010 to an online version called Horizon Online or HNG-X, and the former version is now called Legacy Horizon. The claimants’ case is essentially that both Legacy Horizon and Horizon Online (which used many elements of the existing system) were or are unreliable, and this led to unexplained shortfalls and discrepancies in their branch accounts. The Post Office denies this, asserting that the systems were and are robust, and extremely unlikely to be the cause of such matters. There are numerous different causes of action brought against the Post Office, and the Post Office counterclaims against the claimants, including seeking damages for fraud. This is group litigation under CPR Part 19.

2.

This judgment concerns the operation and functionality of the Horizon system itself. It follows the lengthy Common Issues judgment, which is Judgment (No.3). That judgment was extremely long, for two main reasons. It concerned six different claimants, each of whom had contracted with the Post Office at different times, and each of those six claimants’ cases concerned very different facts. Very few of the material facts relating to contract formation were agreed for any of these six, and therefore factual findings had to be made for each of them in this respect. Approximately 500 paragraphs were necessary to do this. Although first instance judgments must be reasoned so that parties understand the outcome (and so that any judgment can be subject to appropriate review), very lengthy judgments can be difficult to follow. If a judgment becomes too long, or too technical, it can be counterproductive to wider understanding on the part of those not immersed in the fine detail of every aspect of the case.

3.

I have therefore included technical detail about the Horizon system, its operation, and some aspects of the technical evidence, in a Technical Appendix to this judgment. The contents of that appendix should not be seen as being of subordinate effect to the contents of the judgment itself; this is done simply for the convenience of readers. It is also intended that readers who do need to immerse themselves in technical matters to a very fine level of detail may not find it necessary to study the appendix, and reading the judgment alone may be sufficient. Notwithstanding that approach, however, this judgment too is extremely long. This is due to the complexity of the Horizon system, which even for computer systems is extraordinarily complicated for the reasons explained below; the period of time over which the complaints range (which starts with the introduction of Horizon in 2000 and runs to date); and due to the way that the litigation has unfolded. There are some matters in this judgment that go to issues affecting the group litigation going forwards, such as disclosure, which have increased its length. It is also the case that this litigation is being very strongly contested on both sides. I have endeavoured to provide a reasonable level of detail to explain my findings on the Horizon Issues to assist both sides as much as possible. Finally, the only findings that are made in this judgment are those necessary to come to conclusions on the Horizon Issues. All other matters in all the claims of the claimants and the counterclaims by the Post Office remain for decision in later judgments.

4.

This judgment is in the following parts:

Paragraph no.

A.

Introduction 5

B.

The Horizon Issues 18

C.

Features of this Group Litigation 57

D.

Evidence of Fact: The Claimants 76 E. Evidence of Fact: The Post Office 202

F.

Documents and Available Information 559

G.

The Experts’ Agreements 654

H.

The Period 21 March 2019 to 4 June 2019 706

I.

The Expert Evidence 731

J.

Conclusion on Expert Evidence 870

K.

Audit Data 905

L.

Overall Conclusions 925

M.

Answers to the Horizon Issues 965

Appendices:

1.

Technical Appendix

2.

Summary of bugs, errors and defects

3.

Glossary

A. Introduction

5.

These proceedings are being conducted pursuant to a Group Litigation Order (“GLO”) made on 22 March 2017 by Senior Master Fontaine, and approved by the President of the Queen’s Bench Division. A more comprehensive introduction to the issues generally between the parties is contained at [2] to [43] of Judgment (No.3) “Common Issues”, which is at [2019] EWHC 606 (QB). Other judgments concerning procedural rather than substantive issues, are the first Judgment at [2017] EWHC 2844 (QB) and Judgment (No.2) “Strike Out” at [2018] EWHC 2698 (QB). During the Horizon Issues trial which is the subject of this judgment, the Post Office issued an application that I recuse myself as Managing Judge in this group litigation, and stop the Horizon Issues trial, so that it could be recommenced at some later date in the future (before a replacement Managing Judge). That application was refused and the judgment is at Judgment (No.4) “Recusal Application” at [2019] EWHC 871 (QB). Permission to appeal was refused by the single Lord Justice on 9 May 2019. There is also Judgment (No.5) “Common Issues Costs” at [2019] EWHC 1373 (QB) which made various orders in respect of the costs of the Common Issues trial, a hearing in respect of that having taken place on 23 May 2019.

6.

This trial concerns what the parties referred to at the case management stage as the Horizon Issues. The intention behind the case management of this litigation was that the contractual issues (which were called the “Common Issues” and affected all the claimants) and the computer issues relating to the operation, functionality and reliability of the Horizon system (which were called the “Horizon Issues”, and also affected all the claimants) would be resolved first. The parties needed time to prepare for these two trials, which took place in late 2018 (for the Common Issues) and into

the spring of 2019 (for this one), in particular to perform disclosure, and prepare and serve evidence. For the Horizon Issues, this included expert evidence, for which I gave permission, from two IT experts, one for the claimants (Mr Coyne) and one for the Post Office (Dr Worden).

7.

The recusal application caused significant delay and disruption to the Horizon Issues trial. Originally, that trial had been programmed to finish (including delivery of oral closing submissions) by 11 April 2019, a date which became unachievable as a result of that application. Following Judgment (No.4) the remaining witness of fact for the Post Office to be called in person, Mr Parker, a senior Fujitsu employee, was called on 11 April 2019.

8.

The two IT experts could not be called until 4 June 2019 onwards. The two experts therefore had a period of some weeks, following on from the evidence of fact, before they were called to give their oral evidence. There was therefore an interval between 11 April 2019 and 4 June 2019 when the experts were called. Their evidence collectively took two court weeks.

9.

The effect of this interruption upon the trial timetable meant that there was therefore time available for each of the experts to consider the full scope of the crossexamination of the factual witnesses that had taken place some weeks before, and reflect (with more time than is usual for experts whilst a trial is underway) upon whether that impacted upon their views. Experts are supposed to consider the factual evidence advanced by both sides in litigation neutrally, and if witnesses are crossexamined, then – at least potentially – other evidence that might emerge ought, if it is important, to be considered by those experts in arriving at their final opinions given in evidence. In a conventional trial, expert evidence will usually follow on almost immediately after the evidence of fact has been heard. This means that, in practical terms, most experts will only have a very short time in which to consider any potentially important factual developments in any case before they are themselves cross-examined. Here, there was no such restriction.

10.

The Horizon Issues trial timetable was therefore different to a conventional trial in this respect, because the vast majority of the oral factual evidence, with the single exception of Mr Parker’s evidence, was completed by 21 March 2019. That is 75 days, or almost 11 weeks, before Mr Coyne was called by the claimants, and even longer before Dr Worden was called by the Post Office. The approach of the experts to the opportunity presented by that interval further to assist the court with their evidence is notable, in my judgment. I will deal with this further in Parts H and I of this judgment. For a more detailed description of the architecture of the Horizon system (both Legacy Horizon and Horizon Online) reference should be made to the Technical Appendix. For a general understanding of the issues in this litigation, an overview is sufficient.

11.

Horizon is both an accounting system, and also supports a large number of what are called business applications. Some transactions that a customer might wish to carry out in their local Post Office branch are what are called retail activities, such as buying a Post Office product. One example of this is buying a book of stamps. However, the Post Office has a large number of business customers whose products are offered to the public through branch Post Offices. The Post Office calls these companies “clients”. For every kind of activity which a customer wishes to transact in their Post Office branch, Horizon needs to have the functionality to perform it. This functionality means supporting the counter activity of carrying out the transaction, which is another way of saying Horizon enables the SPM or their assistant to transact the business over the counter. That business may be a combination of Post Office retail activity, and purchasing services or products offered by the Post Office on behalf of its clients. However, in conjunction with the ability of a SPM to serve a customer (what is called counter or front office), there is the associated “back office activity” of settling with Post Office's 'client' organisation, who has provided some service to the customer - such as the DVLA, or a bank. Accounting is a theme or thread that runs through all of these business requirements of Horizon, but it is only a part of them. The number of services provided by Post Office branches is large and has increased steadily from 1998 to the present day. The number of clients has almost certainly increased, and the functionality of Horizon has expanded in line with the growth in service, both on the counter and in the back office.

12.

For those activities where the Post Office branch is acting like a retail outlet (such as selling stamps), both the hardware and software is provided by Horizon to support this activity. This is the Electronic Point of Sale Software component of Horizon, that is referred to as “EPOSS”. EPOSS allows the SPM or assistant to record that some goods have been provided to a customer, compute the price of those goods, and allow the customer to pay the money required for all their purchased goods, using either cash or a credit/debit card. Often, a customer may wish to carry out two or more different activities in one visit to the counter - for instance, to settle a utilities bill and to buy some stamps. This can be done in the same activity and so Horizon has the concept of a customer carrying out a 'basket' of activities and settling the total amount due for the basket in several ways - by one single credit card transaction, by a cheque, or by cash.

13.

Baskets of Post Office activities and non-Post Office activities are not supported by Horizon. Often, a local Post Office branch will be a retail outlet too, selling non-Post Office goods such as groceries, newspapers or even (as with Mr Bates at Craig-yDon) a haberdashery. If a customer wishes to buy a newspaper and some stamps, the newspaper is not sold by the Post Office; it is simply sold by the associated, though in accounting terms separate, retail outlet run by the SPM which uses the same premises. So, the customer has to settle in two parts. In some premises, a customer may queue up to purchase (say) greetings cards and pens (from the retail side of the branch Post Office). If they then wish to perform some activities with the Post Office, they may then need to queue up at a separate counter position to do that; that separate and second activity would be transacted through Horizon. In this respect, the National Lottery is an exception and spans the two businesses.

14.

The Post Office’s clients include high street banks, Camelot, gas and electricity companies (for paying utility bills), DWP (for paying benefits and pensions) and DVLA (for paying road fund tax). Because of the different nature of the services provided through the Post Office for all of these many hundreds of client organisations, the services provided through the Post Office will be different from the service provided for other clients. There is therefore the need for some unique software functionality within Horizon. This must be provided both in the branch and the back office to support the activities for that client. This is a part of what makes Horizon such a large and complex system. The other reason that Horizon is so complex is because it has evolved over a long period. As will be seen below, the original concept and design for Horizon was for a joint Post Office/DWP project so that welfare benefits could be paid to benefit claimants in a certain way. That changed and the DWP withdrew from the project, but the need to offer other and an increasing number of products has led to many additions being added on to the system, increasing its complexity over time.

15.

As well as the counter activities necessary to run a branch Post Office, Horizon also supports the periodic process of balancing and rollover for each branch; essential elements in the accounting to the Post Office performed by the SPM. Every branch operates in Trading Periods, which are either four or five weeks (according to a timetable published periodically by Post Office). At the start of each Trading Period the branch is supposed to be 'in balance'. This means that the physical stock and cash in the branch agrees with the data regarding the stock and cash held in Horizon. Then, during the Trading Period, Horizon records all customer transactions made at the branch, so it records the changes in cash and stock. It is also required to record any replenishments or remittances of cash or stock in the branch. Thus, Horizon records all changes in cash and stock held at the branch during the Trading Period, and should be able to compute, from the starting amounts and the changes, the expected amounts of cash and stock at the end of the period.

16.

Without reciting the entire background to the group litigation between the Post Office and the claimants, it is the accounting and functional accuracy of Horizon that is at the heart of the current disputes, which have run for a great many years. The claimants maintain that the Horizon system in operation threw up numerous discrepancies and shortfalls in their branch accounts, for which the Post Office unfairly held them responsible. The Post Office dispute this, and maintain that these occurrences are explicable by carelessness, fault or criminality on the part of the claimants.

17.

Horizon, whether in its first incarnation as Legacy Horizon until 2010, or what is called Horizon Online since then (originally HNG-X, now since 2017 HNG-A running on a different windows system), is at the centre of this group litigation. It should also be noted that it is not a system that is operated by the Post Office. It was originally designed, “rolled out” and operated by ICL, a company that was partly owned by Fujitsu, and which is now fully absorbed within Fujitsu, and has been for many years. There is a contract between Fujitsu and the Post Office in respect of Horizon, and perhaps in its current form it is rather different than the original contractual relations between those two entities. The contractual relationship between the Post Office and Fujitsu is not of direct relevance to the Horizon Issues, but arises tangentially only in respect of what is a sub-issue, namely the charging structure operated by Fujitsu for what are called ARQ requests for audit data and whether this inhibited the Post Office in this respect. The operation, functionality and accuracy of Horizon is an extremely thorny issue (or bundle of issues), although given the breadth of allegation and counter-allegation in the group litigation they are not the only issues. Providing the answers to the Horizon Issues will not lead to complete resolution of the litigation. It should however resolve one of the central issues about Horizon’s accuracy and functionality.

B. The Horizon Issues

18. The Horizon Issues are as follows. In the 3rd Case Management Order of 1 March 2018, leading counsel for the parties were ordered to meet and seek to agree the Horizon Issues to be tried in March 2019. This was done, the issues were agreed by the parties and approved by the court. I take the following from the Case Management

Order dated 23 March 2018, which was the next order made, and was a Consent

Order. The paragraph references to the pleadings were included in the list of Horizon Issues, which were themselves appended to the Order of 23 March 2018 itself. These include at the beginning an agreed definition of “the Horizon System”.

“AGREED LIST OF HORIZON ISSUES

definitions for the purpose of this list of issues

“the Horizon System” shall for the purposes of this list of issues mean the Horizon computer system hardware and software, communications equipment in branch and central data centres where records of transactions made in branch were processed, as defined in the Generic Particulars of Claim (“the GPOC”), at §16 and as admitted by Post Office in its Defence at §37.

BUGS, ERRORS AND DEFECTS IN HORIZON

Accuracy and integrity of data

(1)

To what extent was it possible or likely for bugs, errors or defects of the nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to 56 of the Generic Defence to have the potential (a) to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions as alleged at §24.1 GPOC?

(2)

Did the Horizon IT system itself alert Subpostmasters of such bugs, errors or defects as described in (1) above and if so how?

(3)

To what extent and in what respects is the Horizon System “robust” and extremely unlikely to be the cause of shortfalls in branches?

Controls and measures for preventing / fixing bugs and developing the system

(4)

To what extent has there been potential for errors in data recorded within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data in Horizon?

(5)

How, if at all, does the Horizon system itself compare transaction data recorded by Horizon against transaction data from sources outside of Horizon?

(6)

To what extent did measures and/or controls that existed on Horizon prevent, detect, identify, report or reduce an extremely low level of risk of the following:

a.

data entry errors;

b.

data packet or system level errors (including data processing, effecting, and recording the same);

c.

a failure to detect, correct and remedy software coding errors or bugs;

d.

errors in the transmission, replication and storage of transaction record data; and

e.

the data stored in the central data centre not being an accurate record of transactions entered on branch terminals?

OPERATION OF HORIZON

Remote Access

(7) Were Post Office and/or Fujitsu able to access transaction data recorded by Horizon remotely (i.e. not from within a branch)?

Availability of Information and Report Writing

(8)

What transaction data and reporting functions were available through Horizon to Post Office for identifying the occurrence of alleged shortfalls and the causes of alleged shortfalls in branches, including whether they were caused by bugs, errors and/or defects in the Horizon system?

(9)

At all material times, what transaction data and reporting functions (if any) were available through Horizon to Subpostmasters for:

a.

identifying apparent or alleged discrepancies and shortfalls and/or the causes of the same; and

b.

accessing and identifying transactions recorded on Horizon?

Access to and/or Editing of Transactions and Branch Accounts

(10) Whether the Defendant and/or Fujitsu have had the ability/facility to (i) insert, inject, edit or delete transaction data or data in branch accounts; (ii) implement fixes in Horizon that had the potential to affect transaction data or data in branch accounts; or (iii) rebuild branch transaction data:

a.

at all;

b.

without the knowledge of the Subpostmasters in question; and

c.

without the consent of the Subpostmaster in question.

(11)

If they did, did the Horizon system have any permission controls upon the use of the above facility, and did the system maintain a log of such actions and such permission controls?

(12)

If the Defendant and/or Fujitsu did have such ability, how often was that used, if at all?

(13)

To what extent did use of any such facility have the potential to affect the reliability of the Branches’ accounting positions?

Branch trading statements, making good and disputing shortfalls

(14) How (if at all) does the Horizon system and its functionality:

a.

enable Subpostmasters to compare the stock and cash in branch against the stock and cash indicated on Horizon?

b.

enable or require Subpostmasters to decide how to deal with, dispute, accept or make good alleged discrepancy by (i) providing his or her own personal funds or (ii) settling centrally?

c.

record and reflect the consequence of raising a dispute on an alleged discrepancy, on Horizon Branch account data and, in particular:

(i)

does raising a dispute with the helpline cause a block to be placed on the value of an alleged shortfall; and

(ii)

is that recorded on the Horizon system as a debt due to Post Office?

d.

enable Subpostmasters to produce (i) Cash Account before 2005 and (ii) Branch Trading Statement after 2005?

e.

enable or require Subpostmasters to continue to trade if they did not complete a Branch Trading Statement; and if so, on what basis and with what consequences on the Horizon system?

Transaction Corrections

(15) How did Horizon process and/or record Transaction Corrections?”

19.

Given the nature of the proceedings, and the disputes about the Horizon system which were apparent on the face of the pleadings, I had indicated to the parties at an early stage that the generic disputes about the operation of the Horizon system, which would need expert evidence, would be resolved after the contractual issues (which became called the Common Issues, and led to Judgment (No.3)). The issues were not foisted on the parties by the court. The court approved the wording of the issues agreed by the parties during the case management stage of the litigation. There was no difficulty about this at the time.

20.

However, the meaning of issues 1 and 3 in particular proved, at the Horizon Issues trial itself, to be controversial. This is regrettable. Nor was any controversy aired with the court prior to the actual trial. That too is regrettable. The court is well used to parties who disagree over the answers to certain issues in litigation generally. It is unsatisfactory when they also disagree about what any particular issue itself actually

means, or the question that is being posed (including being posed to experts) by an issue, particularly once that issue has already been finalised and ordered.

21.

Here, the two main areas of dispute over the meaning of the Horizon Issues were as follows (there were others, but these are the most important ones). The claimants approached Issue 1 as it was worded, requiring consideration of whether bugs, errors or defects had the potential to cause discrepancies or shortfalls in SPMs’ branch accounts or transactions. In other words, this issue was not limited to consideration of whether bugs, errors or defects had in fact actually caused discrepancies or shortfalls. The Post Office, on the other hand, approached Issue 1 as requiring consideration of whether bugs, errors or defects had in fact actually caused discrepancies or shortfalls, and by reference to the claimants’ accounts specifically. This approach by the Post Office sought, in my judgment, to narrow the scope of the Horizon Issues. This was not raised at the case management stage.

22.

The second main area of dispute concerned Issue 3. This includes consideration of the concept of “robustness”. This word has been at the heart of the Post Office’s defence of the Horizon system for many years. The Post Office has said publicly, and on many occasions, that no computer system is 100% accurate and/or perfect but Horizon is “robust”. This approach by the Post Office pre-dated the commencement of the group litigation and was at the heart of the Post Office’s response to the increasing criticisms from different quarters about Horizon. I deal with this important concept below in more detail at [36] below. What the Post Office effectively means by this (in outline terms only) is that the Horizon system can properly and safely be relied upon by the Post Office for the purposes for which it is designed and intended. The parties – and this is clear on the pleadings, as well as in the terms of Horizon Issue 3 – disagree over whether Horizon is robust. In the Reply, the claimants challenged this in the following terms:

“It is therefore denied that Horizon 'is robust and [...] is extremely unlikely to the cause of losses in branches' (paragraph 16). In fact, the relatively small chance of errors admitted by the Defendant, would be likely to produce the very picture reflected in the Claimants' case.”

23.

Mr de Garr Robinson for the Post Office was somewhat critical at the trial of the drafting of both Issues 1 and 3. He was not involved in the drafting of the Horizon Issues, as the Post Office had used different leading counsel. However, that criticism overlooked that the Post Office had agreed to the wording of all the Horizon Issues through its other leading counsel, and the wording had also been approved by the court. Although there have been a total of three leading counsel altogether instructed for the Post Office thus far at first instance, with a fourth leading counsel used by the Post Office to try and appeal the Common Issues to the Court of Appeal, simply because different counsel are used does not mean that the issues ordered to be tried become different. The Horizon Issues themselves were appended to the Case Management Order itself.

24.

Lists of issues, whether in group litigation or generally, are very important. I can do no better than quote Longmore LJ in Scicluna v Zippy Stitch Ltd [2018] EWCA Civ

1320, who said at [14] “ever since the Woolf reforms, parties in the High Court have been required to agree lists of issues formulating the points which need to be determined by the judge. That list of issues then constitutes the road map by which the judge is to navigate his or her way to a just determination of the case.” In this litigation, the Horizon Issues are not only important, but are vitally important. The whole of the proceedings concerns the operation and reliability of the Horizon system. It could quite easily be called the Horizon Post Office Case. Although there are a great number of different causes of action pleaded in both directions, both by the claimants and by the Post Office, against one another, the central core of the case is about Horizon. I consider that the wording of both Horizon Issues 1 and 3 – indeed, all the Horizon Issues – to be clear and had they not been, I would not have approved the agreed wording.

25.

However, some explanation is necessary given the disagreement to which I have referred.

26.

The phrase “bugs, errors or defects” is sufficiently wide to capture the many different faults or characteristics by which a computer system might not work correctly. The parties in this litigation, usually but not always for convenience, would often refer simply to “bugs”, and one part of the 2nd Experts’ Joint Statement became known as “the Bug Table”. Computer professionals will often refer simply to “code”, and a software bug can refer to errors within a system’s source code, but “software bugs” has become more of a general term and is not restricted, in my judgment, to meaning an error or defect specifically within source code, or even code in an operating system. Source code is not the only type of software used in a system, particularly in a complex system such as Horizon which uses numerous applications or programmes too. Reference data is part of the software of most modern systems, and this can be changed without the underlying code necessarily being changed. Indeed, that is one of the attractions of reference data. Software bug means something within a system that causes it to cause an incorrect or unexpected result. During Mr de Garr Robinson’s cross-examination of Mr Roll, he concentrated on “code” very specifically and carefully. There is more to the criticisms levelled at Horizon by the claimants than complaints merely about bugs within the Horizon source code.

27.

The wording of Horizon Issue 1, “to what extent was it possible or likely for bugs, errors or defects…..” is therefore very wide. Although that wording was proposed and agreed by the parties, I considered at the case management stage when the Horizon Issues were ordered, and still consider now, that these words clearly cover the whole range of criticisms levelled at Horizon by the claimants, and the matters that could potentially be wrong with the Horizon system without restriction. Bugs, errors or defects is not a phrase restricted solely to something contained in the source code, or any code. It includes, for example, data errors, data packet errors, data corruption, duplication of entries, errors in reference data and/or the operation of the system, as well as a very wide type of different problems or defects within the system. “Bugs, errors or defects” is wide enough wording to include a wide range of alleged problems with the system.

28.

I also consider that the words “possible or likely” are also wide enough to cover both ends of the spectrum of what the parties would have the court decide. In other words, it is a neutrally worded issue of wide effect which does not, by its phrasing, indicate any particular starting point or end point. “Possible” means something potentially could happen; or whether it could happen or not. “Likely” means that something is, on

the balance of probabilities, more likely to have happened than not. The issue uses both terms, and therefore poses a wide question so that the answer provided by the court will be of maximum utility in the group litigation.

29.

Finally, the word “potential”, in the clause “have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon…..” clearly means, on its express terms, that this issue is dealing with possible or prospective effects, rather than whether bugs, errors or defects have in fact caused actual or alleged discrepancies or shortfalls. The way the wording of this issue was dealt with in the expert evidence will be dealt with below.

30.

If there were any ambiguity in Issue 1, and I do not consider that there is, the issue specifically refers to certain passages in the pleadings. It would therefore be of assistance to consider the pleadings. The relevant passages from the Particulars of Claim as amended are as follows:

“23. However, the Claimants aver that there were a large number of software coding errors, bugs or defects which required fixes to be developed and implemented. There were also data or data packet errors. There was a frequent need for Fujitsu to rebuild branch transaction data from backups, giving rise to the further risk of error being introduced into the branch transaction records. The Claimants understand that Fujitsu maintained a 'Known Error Log' relating to some or all of these issues which was provided to the Defendant but which has not been disclosed.

24. Further, the Claimants aver and rely upon the following:

24.1.

Insufficient error repellency in the system (including sufficient prevention, detection, identification and reporting of errors), both at the data entry level and at the data packet or system level (including data processing, effecting and reconciling transactions, and recording the same);

24.1A bugs and/or errors and/or defects in Horizon and any data or data packet errors had the potential to produce apparent shortfalls which did not represent a real loss to the Defendant;

24.2.

Horizon is imperfect and has the potential for creating errors (as the Defendant has admitted in pre-action correspondence, in the Letter of Response, dated 28 July 2016, at paragraph 1.3);

24.3.

bugs and/or errors have on some occasions produced discrepancies and/or apparent shortfalls (as the Defendant has admitted in pre-action correspondence, in the Letter of Response, Schedule 6) and such shortfalls may also have arisen from data or data packet errors; and, further

24.4.

the Defendant sought and/or recovered such alleged shortfalls from Subpostmasters (as is presently understood to be admitted by the Defendant in the Letter of Response, Schedule 6, paragraphs 4.1 to 4.5).”

31.

The passages in the Defence which are identified in the Issue are paragraphs 49 to 56. This is a lengthy series of paragraphs but as all are referred to expressly within Issue 1, I will reproduce them all. These state the following:

Bugs, errors or defects in Horizon

49. As to paragraph 22:

(1)

If and to the extent that the Claimants wish to assert that any of the shortfalls for which they were held responsible were Horizon-generated shortfalls, it is for them to make that distinct allegation and seek to prove it. Post Office notes that they do not make the allegation in the GPoC. It further notes that, in paragraph 20 of their solicitors' letter to Post Office's solicitors dated 27 October 2016, the Claimants make it clear that they do not allege that there is a systematic flaw in Horizon or indeed any flaw which has caused any Claimant to be wrongly held responsible for any shortfall.

(2)

It is denied that Post Office has unreasonably or otherwise failed to provide "obviously relevant disclosure" in relation to bugs, errors or defects in Horizon. There has been no order or application for disclosure and, in the premises set out above, there appears to be no basis for providing such disclosure.

50. Paragraph 23 is embarrassing for its lack of particularity, in that (amongst other things) it does not identify the errors, bugs or defects on which the Claimants rely or how "large" their number was or the period in which they are said to have occurred and nor does it identify the transaction data that Fujitsu is alleged to have rebuilt, how "frequent" was the need to rebuild it or the extent of the ''risk of error" which is said to have been introduced. In the premises, Post Office cannot plead to the first three sentences of this paragraph. However:

(1)

All IT systems experience software coding errors or bugs which require fixes to be developed and implemented. As is noted in paragraph 53 and 54 below, there are robust measures in place in Horizon for their detection, correction and remediation.

(2)

All IT systems involving the transmission of data over the internet experience data or data packet errors during transmission and such systems routinely have protective measures in place to prevent such errors creating any difference between the data transmitted and the data received and retained by the recipient. Horizon has robust controls making it extremely unlikely that transaction data input in a branch would be corrupted when being transferred to, and stored in, Post Office's data centre in a manner that would not be detected and remedied.

(3)

Like all IT systems, Horizon has backups to guard against any loss of data due to local hardware failure. Where hardware fails, the data on that hardware is recovered from the backup. Post Office takes the term "rebuild" to refer to the situation before the introduction of Horizon Online where a new terminal was introduced to a branch and the data stored on the other branch terminals (or on a disc where it was a single counter branch) was restored to the new terminal. In this context, Post Office does not accept that there was a "frequent" need to "rebuild" data from back-ups.

(4)

It is admitted that Fujitsu maintain a "Known Error Log". This is not used by Post Office and nor is it in Post Office's control. To the best of Post Office's information and belief, the Known Error Log is a knowledge base document used by Fujitsu which explains how to deal with, or work around, minor issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide fixes have not been developed and implemented. It is not a record of software coding errors or bugs for which system-wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief, there is no issue in the Known Error Log that could affect the accuracy of a branch's accounts or the secure transmission and storage of transaction data.

51.

In paragraph 24, the Claimants again bundle many ambiguous and/ or misleading allegations together. Post Office separates out and addresses those allegations in paragraphs 52 to 56 below.

52.

As paragraph 24.1 does not explain what is meant by "error repellency", what sorts of errors are referred to, what is meant by "data entry level", what would constitute "sufficient" prevention, detection, identification or reporting of these errors, or in what respects the error repellency of Horizon was insufficient, Post Office cannot plead to this paragraph. However, the general thrust of paragraph 24.1 is denied and the robust controls, procedures and practices pleaded in paragraphs 53 and 54 below are noted.

53.

As to paragraph 24.1A, it is a truism that errors or bugs in an IT system and data or data packet errors have the potential to create errors in the data held in that system. However, Horizon has at all material times included technical control measures to reduce to an extremely low level the risk of an error in the transmission, replication and storage of the transaction record data. These have varied from time to time and they currently include the following:

(1)

Horizon creates, transmits and stores trans action data in the form of "baskets". A basket is a complete transactional session between a customer and Post Office and may include one, several or many individual transactions taking place within the same session. Horizon will not accept a basket of transactions that does not net to zero (i.e. the value of any sales is set off by the value of any payment made or received). This reduces greatly the risk of any error in the data entered within any given basket.

(2)

If a basket of transactions fails properly to complete its transmission to the central database (because, for example, of a power loss), the system rejects any partial transmission and requests the full basket from the branch terminal. This reduces greatly the possibility of baskets of transactions failing to be recorded.

(3)

At the point of a basket being accepted by Horizon, it is assigned a unique sequential number (a "JSN") that allows it to be identified relative to the other baskets transmitted by that branch. This reduces greatly the risk of recording duplicate baskets or there being a missing basket.

(4)

Each basket is also given a digital signature, i.e. a unique code calculated by using industry standard cryptography. If the data in the basket were to change after the digital signature was generated, this would be apparent upon checking the digital signature.

(5)

Initial data integrity checks are undertaken when baskets are received at the Post Office data centre from a branch. Baskets are then copied from the central database to the Audit Store where a digital seal is then applied (the "Audit Store Seal"). If the baskets and/ or the data within the baskets were altered after the application of the Audit Store Seal, this would be apparent when the baskets are extracted from the Audit Store.

(6)

Horizon and the above controls are themselves subject to various audits and checks including audits carried out by third parties.

54. Further as to paragraph 24.1A, in addition to the technical controls referred to above, there are several operational procedures and practices conducted by Post Office and Subpostmasters that serve to increase the reliability of the data stored in the central data centre as an accurate record of the transactions entered on branch terminals. These currently include the following:

(1)

For many transaction types, Post Office compares its own transaction record against the corresponding records held by Post Office clients. If an error in Horizon were to result in the corruption of transaction data, this should be revealed by the comparison.

(2)

There are detailed procedures in place to address the risk of data loss resulting from interrupted sessions, power outages or telecommunications failures in branches. These are set out in the "Recovery - Horizon Online Quick Reference Guide" and Horizon guides the system user through the recovery process (which include completing any transactions that are cut short). These procedures should prevent any data errors arising from interrupted sessions, power outages and telecommunications failures.

(3)

The display of the transactions being effected on-screen at the branch terminal allows the user of the system to identify any inconsistency between the information shown on the screen and the transaction that the user has keyed into the system. If, for example, a hypothetical bug in the terminal were to cause a key-strike on number S to be recorded as an input of number 6, this would be detected rapidly by system users, given the large number of system users and the huge number of transactions effected on Horizon.

(4)

The accounting and record-keeping obligations placed on Subpostmasters reduce the risk of any errors going undetected. For example, there is an obligation for each branch to count and declare to Post Office the cash it holds on a daily basis, which increases the likelihood of promptly detecting any overstatement or understatement of the cash position on Horizon. If a Subpostmaster detects that an error has been made at an early stage, its cause is more likely to be identified.

(5)

Fujitsu operates industry standard processes for developing and updating Horizon and for investigating and resolving any identified potential system errors.

55.

As to paragraph 24.2, Post Office admits that, like all other IT systems, Horizon is not a perfect system which has never had any errors or bugs. However, as indicated in paragraphs 53 and 54 above, it has robust systems in place to identify them, fix them and correct their consequences (if any).

56.

As to paragraphs 24.3 and 24.4:

(1)

There have been occasions on which bugs or errors in Horizon have resulted in discrepancies and thus shortfalls or net gains in some branch accounts, as outlined in Schedule 6 of the Letter of Response. It is denied (if it be alleged) that such bugs or errors have affected any of the Claimants.

(2)

On each occasion, both the bugs or errors and the resulting discrepancies in the relevant branch accounts were corrected. Post Office took steps to ensure that it had identified all branches affected by the bugs or errors and that no Subpostmaster was ultimately held responsible for any resultant shortfalls.

(3)

Paragraphs 4.1 to 4.5 of Schedule 6 to the Letter of Response relate to the socalled Suspense Account Bug. Without prejudice to the burden of proof, none of the branches affected by the Suspense Account Bug are branches for which the Claimants were responsible.

(4)

None of the Subpostmasters whose branches were affected by the Suspense Account Bug were ultimately held responsible for the shortfalls that it generated. The Claimants are therefore wrong to understand Post Office as having admitted that it "recovered such alleged shortfalls from Subpostmasters". Where Subpostmasters in the affected branches had made good or settled centrally shortfalls that were later corrected, those Subpostmasters received a payment or credit in the amount of the shortfall.”

32.

It can be seen that the word “robust” is used throughout these passages of the Defence, and “robust systems” in paragraph 55. Given “robustness” forms part of Issue 3 there is no need separately to consider it at this stage when analysing the differences between the parties about what Issue 1 means.

33.

The passages in both parties’ pleadings which I have reproduced above show that it is the potential, in general terms, of the Horizon system to have the effect complained of that is the subject matter of Horizon Issue 1. In order to read Issue 1 in the manner contended for by the Post Office, one would have to delete entirely the phrase “to have the potential” from the issue. One would also have to substitute the term “Subpostmasters’ branch accounts or transactions” with “the claimants’ branch accounts or transactions”. That is not what Issue 1 states, it is not what the parties agreed the issue would be, and it is not the issue that the court approved and ordered. In my judgment, the approach of the Post Office to what Horizon Issue 1 actually means is too narrow.

34.

In my judgment, the correct construction of Horizon Issue 1 is that contended for by the claimants. In other words, it involves a two-stage process. Firstly, consideration of whether there were, or are, bugs, errors or defects in the Horizon system as alleged by the claimants. Secondly, if the answer is that there were, or are, such bugs, errors or defects, the second stage is to consider whether these have (or did have previously) the potential to cause apparent or alleged discrepancies in SPMs’ branch accounts generally. The issue is not whether such bugs, errors or defects did in fact cause such discrepancies or shortfalls in the claimants’ accounts specifically. That separate or different issue – the effect upon claimants’ branch accounts - is a more claimant specific one. It will have to be determined at some stage, for any of the claimants whose individual claims come to trial in the future. It may require expert forensic accountancy evidence. It was not ordered to be dealt with in the Horizon Issues trial. The Horizon Issues were intended to be, and in my judgment on their wording are, generic issues relating to Horizon and its operation. However, if a bug, error or defect is shown to have had an actual impact on any SPM’s accounts, then by definition it has had the potential to have such an impact. Actual impact on branch accounts can therefore be of assistance in considering Issue 1 as ordered, namely including potential.

35.

The passages in the pleadings quoted above do show that the Post Office had, in

Schedule 6 to its Letter of Response, accepted the existence of two bugs or errors in Horizon, one of which was called the Suspense Account Bug, but denied that the latter had affected any branch accounts of any of the claimants. The passages in paragraph 56 of the Defence that explained that no SPMs had been “ultimately held responsible for the shortfalls that it generated” reads a little differently with the hindsight provided from evidence in two lengthy trials. This is because the Post Office’s Defence effectively accepts that “Subpostmasters in the affected branches had made good or settled centrally” the sums in question – that is the shortfalls in their branch accounts that Horizon showed – but states that these were later corrected. Both the expressions “making good” and “settled centrally” means that the Post Office had originally held the SPM responsible for the losses, as the shortfalls formed part of their branch accounts. These terms mean that the SPMs had either paid the money (making good) or been given time to pay sums which the Post Office treated as a debt, even if they were disputed sums (settling centrally), meanings given to these terms that were confirmed by Ms Van Den Bogerd in her evidence for the Common Issues trial. However, the pleading means that the Post Office had then, on the Post Office’s case, subsequently corrected this impact on branch accounts which was why it pleaded no SPMs were “ultimately held responsible”. Therefore there might prove not to be as much between the parties on this point in reality, as the group litigation unfolds, as there appears to be on the pleaded case. The Suspense Account Bug undoubtedly had actual impact upon SPMs’ branch accounts.

The meaning of “robustness”

36.

Turning to the disagreement about Issue 3, given the parties disagreed about whether the Horizon system is (or was) “robust”, it is a fairly elementary step to consider the meaning of that term, and how it is being used by the parties. Context is important so far as the meaning of the word “robust” is concerned. If someone is in robust health, it usually is taken to mean that they are healthy, or even very healthy. A robust exchange of views can be a polite way of referring to an argument. Given the importance of the concept to the Horizon system, its prominence in the Post Office’s defence of the system, and its express inclusion (admittedly in inverted commas) in the Horizon Issues, I asked each side in the litigation during oral closing submissions for a reference from their pleadings or submissions for the meaning which they ascribed to the word. I referred to this as their benchmark definition. Robustness was referred to by both sides in the litigation in numerous places, but not always in the same precise terms, and clarity is to be welcomed.

37.

The claimants answered this by reference to the remainder of Issue 3, namely “extremely unlikely to be the cause of shortfalls in branches” and explained that the claimants had found the word robust “difficult to define” other than by reference to this. This would mean therefore that it had no separate independent meaning other than as a summary of the longer second part of Issue 3. In other words, a robust system would be one that is extremely unlikely to be the cause of shortfalls in branches. The claimants also implicitly, if not expressly, criticised use of the term both by the Post Office in its pleadings and written submissions as being more aligned to public relations than as a performance standard.

38.

The Post Office asked for some time to provide the reference that I requested. Given the meaning of “robust” is so central in the Post Office’s defence of the Horizon system, I granted the Post Office the time that was requested.

39.

The Post Office subsequently, after the trial ended, submitted a short document entitled “the Post Office’s case on the meaning of robustness”. This was not what was intended when I sought a reference from the Post Office to their definition, and the document submitted went rather further and made wider ranging submissions. The document did state, so far as the meaning of the word is concerned, the following: “In Post Office’s submission, the meaning of robustness is a matter for expert opinion. Robustness is a well-established concept in the IT industry and is the subject of academic study: see para. 361 of Post Office’s written closing”.

40.

I do not consider that the meaning of words is a matter for expert opinion. The two experts in this case are IT experts, not experts in linguistics or the meaning of language. However, the meaning of robustness within the field of IT is, arguably, a matter upon which the experts’ opinions should be considered, not least because they were applying that term to their expert exercise. The Post Office also relied upon the 1st Experts’ Joint Statement which in respect of Issue 3 stated the following as agreed:

“There are different dimensions of robustness, such as robustness against hardware failure, software defects and user error. The robustness of the system also depends on the processes around it.

Robustness does not mean perfection; but that the consequences of imperfection must be managed appropriately. If the extent of imperfection is too high, this would be very difficult to do which would imply less robustness.

Horizon has evolved since its inception. Therefore, its robustness may have varied throughout its lifetime. The level of robustness may have increased or decreased as the system was changed.

The existence of branch shortfalls is agreed. The experts do not agree at this point as to whether this indicates any lack of robustness.”

41.

In the areas of disagreement in this Joint Statement, each expert provided the following. Mr Coyne stated (inter alia):

“For the purposes of addressing the robustness of Horizon, I have applied the following definition of robustness:

‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling.’ ”

In consideration of the likelihood of Horizon to be the cause of shortfalls in branches, Horizon is not determined to be robust in this regard because:

(a)

it contained high levels of bugs, errors and defects as set out under Issue 1 above which created discrepancies in the branch accounts of Subpostmasters;

(b)

it suffered failures of internal mechanisms which were intended to ensure integrity of data;

(c)

the system did not enable such discrepancies to be detected, accurately identified and/or recorded either reliably, consistently or at all;

(d)

the system did not reliably identify ‘Mis-keying’, which is inevitable in any system with user input, and did not reliably have in place functionality to restrict users from progressing a mis-key;

(e)

it required numerous processes and workarounds to be in place to allow Fujitsu to modify data already recorded by Horizon, which would not be required in a “robust” system; and/or

(f)

there were weaknesses and risks of errors and other sources of unreliability within Horizon.”

(italics present in original)

42.

Dr Worden stated in the same Joint Statement:

“The definition of 'robust' proposed above by Mr Coyne is not adequate, for reasons given below. The term 'robust' is not, as implied in para 3.1 of the outline, either illdefined or a piece of IT public relations. Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.

In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?

In my report I shall survey the evidence I have found that Fujitsu paid sufficient attention to the dimensions of robustness, and that they did so successfully. I shall also address evidence from Mr Coyne implying that Horizon fell short of its robustness objectives.

In my current preliminary opinion, Horizon is a highly robust system, and this has important implications for the other Horizon issues, notably issue 1.”

43.

It can be seen therefore that Dr Worden in the Joint Statement did not agree Mr Coyne’s definition, and expressly said it was not adequate. In any event, the meaning of any word – even “robust”, or “robustness” – ought to be capable of description by the parties themselves. Although on its face it did not appear that Dr Worden agreed with Mr Coyne’s definition, a footnote in the Post Office first set of post-hearing submissions suggested that Dr Worden was not disagreeing with the first part of Mr Coyne’s text, in other words that part of the text that contained his definition of robustness (which was in italics in the 1st Joint Statement). Obviously if the parties (or their experts) could agree the definition to be applied so far as the Horizon System is concerned, that ought to be identified. I therefore asked the Post Office via email whether it agreed with the definition adopted by Mr Coyne, and if not, what its alternative definition was.

44.

This led to a further document being received from the Post Office dated 18 July 2019. It referred to the passage in the 1st Joint Statement (which is quoted at [40] above) as “the agreed definition”. That rather overlooks that Mr Coyne identified the definition of robustness which he was applying, and Dr Worden expressly disagreed with this in the same Joint Statement under the heading “Areas of Disagreement”, and stated “the definition of ‘robust’ proposed above by Mr Coyne is not adequate, for the reasons given below”. It also overlooks that in the 3rd Joint Statement, paragraph 3.1 had an agreed entry which stated the following:

“Irrespective of how you define the detail of robustness, in line with most other largescale computer systems, Horizon's robustness has generally improved.

From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events.

This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.”

(emphasis added)

45.

Later in the same document of 18 July 2019 the submission was made by the Post Office that “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. It was also submitted that “As Post Office understands it, this is what Mr Coyne meant when in his comments in [the 1st Joint Statement] he defined robustness as “the ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.

46.

This was precisely the definition which Dr Worden, in his areas of disagreement on the 1st Joint Statement, described as “inadequate”. The end position therefore is as follows.

47.

The claimants found “robustness” difficult to define in the abstract and tied it in with the other wording of Horizon Issue 3; a robust system would be “extremely unlikely to be the cause of shortfalls in branches”. That however is a consequence of how a robust system would operate, not a definition of what robustness means.

48.

The Post Office defined it as follows: “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. The Post Office was also prepared to accept Mr Coyne’s italicised definition in the 1st Joint Statement, namely ‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.

49.

Mr Coyne applied the definition he set out in italics in the 1st Joint Statement, quoted in the immediately preceding paragraph of this judgment and at [41] above.

50.

Dr Worden’s definition was as follows:

“Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.

In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?”

51.

The Post Office also submitted that Mr Coyne’s definition was not “materially different” to that of Dr Worden.

52.

The Post Office made submissions in paragraph 3(b) of the written submissions dated 18 July 2019 on robustness that stated that Mr Coyne cannot have intended to exclude the effect of countermeasures when he considered the concept of robustness. I shall return to this topic when dealing with countermeasures. This is because some of the countermeasures considered by Dr Worden are not parts of the Horizon System at all, such as SPMs noticing adverse entries in their branch accounts, and the manual issuing of Transaction Corrections (TCs) by the Post Office (which both parties agree are outside of the Horizon System).

53.

I do however accept the Post Office’s submissions that there is not a great or material difference in the definitions of robustness adopted by the parties’ experts. I do not accept the claimants’ submission that robustness is difficult to define. Dr Worden defined robustness by using what he termed as “the dimensions of robustness”. It is rather circular to describe the meaning of robustness as being “robust against” some particular risk. Although Mr Coyne provided his definition in the 1st Joint Statement, the statement by Dr Worden that this was “inadequate” may only have been aimed at the entirety of Mr Coyne’s entry in the areas of disagreement, as effectively accepted by the Post Office in their most recent written submissions on the subject. Whether that is an explanation of the lack of agreement in the Joint Statement, I also agree with the Post Office that Mr Coyne’s definition is not materially different to that used by Dr Worden.

54.

Robustness is indeed an engineering concept. It means the ability of any system to withstand or overcome adverse conditions. A robust system is strong and effective in all or most conditions. The robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences; this is the same meaning as how robustness was described in the Post Office’s written submissions dated 18 July 19. Robustness does not mean perfection.

55.

The exercise necessary above, to arrive at the definition of robustness in [54] above, is not judicial pedantry. Given the central importance of robustness to the disputes about the Horizon System, and the Horizon Issues, it is in my judgment essential. It is mildly surprising, given how central the assertion of robustness has been to the Post Office’s defence of the Horizon System, that Dr Worden’s interpretation of the term has been relied upon so heavily by the Post Office, given the term was used by the Post Office for some years prior to his involvement.

56.

However, regardless of that passing observation, I find that both experts correctly understood what robustness in fact means, and applied the definition at [54] above in considering their expert evidence. I will return to the expert evidence in some detail later in the judgment, including in the Technical Appendix.

C. Features of this Group Litigation

57.

There are many different ways of managing group litigation. The subject matter of such litigation is different from group to group, and what is appropriate in one set of proceedings will not necessarily be the best approach in another set. Group litigation may involve many claimants – even tens of thousands, in some cases – all with what is essentially the same type of claim, governed by a number of common issues that apply across all or most of the cases. Some group litigation may involve similar numbers of claimants, all with similar claims that are factually different. In this case, there are 584 claimants, which in the context of some group litigation, is not very many, although it is still in the many hundreds, and the period of time over which the relevant events are said to have taken place is about 16 or 17 years. There are different aspects to each individual case, but to deal with the litigation efficiently, cost-effectively and proportionately it is simply not feasible for the same judge to try all the claims, one after the other, in full on all of their respective merits. Such a process would take several years. It is not what group litigation is intended to achieve.

58.

The concept of group litigation is that the Managing Judge, whoever he or she may be, with the assistance of the parties insofar as that may be available, selects the most suitable mechanism for that particular set of proceedings in order to achieve compliance with the overriding objective. That will – or should – lead to cost and time saving. That does however require a high degree of co-operation from the litigants. Here, my intention of holding a substantive trial each judicial term onwards to resolve as much of this group litigation as possible, as quickly as practicable, became simply unachievable as a result of the issuing of the recusal application by the Post Office, explained at [5] above. A further trial in the autumn of 2019 also became undesirable as a result of the parties wishing to have time to consider mediation, and to find out whether the Court of Appeal would grant the Post Office permission to appeal on the contractual issues in Judgment (No.3). Over the numerous hearings and two full substantive trials that I have conducted, I have gained the distinct impression that the Post Office is less committed to speedy resolution of the entire group litigation than are the claimants, but it is not possible to state with finality whether that is correct.

59.

All litigation is important to those involved in it. In this group litigation, all the claimants are of the view summarised at [10] in Judgment (No.3) in terms of the behaviour of the Post Office. Recitation that the claimants are of that view does not mean that any similar views are held by the court, or that any decisions have yet been made by the court on that point one way or the other. The total sums claimed by way of liquidated sums (excluding heads of general damages not yet quantified) are approximately £18.7 million. The Post Office counterclaims for certain sums against the claimants and also alleges fraud. Reputations are plainly at stake on both sides. That is the case in many types of proceedings, particularly high-profile ones that attract press interest. However, that sum of money is not large, in the context of large scale and protracted litigation. The parties’ joint costs are approximately now £27 million, and because of the notification provisions in one of the earlier Case Management Orders I made, regular notifications of the total sum of costs are made to the court. The Post Office alone spent over one million pounds in little more than a month earlier this year; the notification letter of 13 May 2019 from its solicitors stated that its costs were in excess of £12,800,000; by the time of its letter of 25 June 2019, the Post Office’s costs were in excess of £13,900,000. Both this level, and rate, of expenditure is very high, even by the standards of commercial litigation between very high value blue chip companies.

60.

I have already explained that the Post Office has now used four different leading counsel, and it also engaged a second commercial firm of solicitors to act for it on its appeal. The claimants are funded by means of litigation funding, explained in Judgment No.3 so far as it is relevant. The claimants are ordinary individuals who ran branch Post Offices, and the Post Office is publicly funded, so none of the parties are high value FTSE 100 companies. Not all of the Post Office’s costs will be recoverable costs regardless of the outcome, not least due to the existence of two Costs Management Orders which are in smaller amounts than the Post Office’s rapidly growing grand total, but the total figures still represent real expenditure of actual money.

61.

These proceedings also have the additional feature of criminal convictions on the part of some SPMs. As explained at [10] in Judgment (No.3), there is a Criminal Cases Review Commission (“CCRC”) review underway in respect of the convictions of

some claimants. I have been told that these are subject to a stay pending judgment upon the Horizon Issues. I have explained this before in open court, but matters such as criminal convictions are no part of this group litigation. This court has no jurisdiction over such matters.

62.

Although criminal convictions are no part of these proceedings, examples of alleged software bugs, errors and/or defects affecting branch accounts, which led to certain adverse consequences for SPMs, did form part of the subject matter of the Horizon Issues trial. Given the impact (actual or potential) upon branch accounts, with the potentially adverse implications for SPMs generally in the background, and given the way that the SPM witnesses in this trial were cross-examined (as with the Common Issues trial, some being accused expressly of criminal offences), criminal convictions were not part of the trial, but were part of the background. In the claimants’ oral closing submissions, Mr Green QC for the claimants identified a high level chronology of what he called “doubling up”, by which he meant bugs, errors and/or defects that led to certain entries in branch accounts being “doubled” incorrectly. He did so by specific reference to allegations by a particular SPM who had been convicted of a criminal offence.

63.

Mr de Garr Robinson QC for the Post Office objected to this, complaining of what he said was jury advocacy. In the course of what became an increasingly vigorous exchange between counsel at the very end of the trial he stated, by reference to what he said were the rules of commercial litigation, that “one of those rules is that one doesn't say things incautiously that might have an impact on evaluations being done in another place in relation to different proceedings.”

64.

Prior to considering the “rule” to which he was referring – it is not one of the Civil Procedure Rules, so far as I am aware – it was therefore necessary to identify exactly what “different proceedings” were being referred to in this disagreement between the parties. There are five elements to this matter.

(1)

Firstly, it had never been raised by any party at any stage during this group litigation that there were any criminal prosecutions currently underway such that it was necessary to consider any reporting restrictions for the group litigation. I expressly raised this with the Post Office on the occasion referred to at [62] and [63] above, who confirmed there were no such prosecutions underway. The “different proceedings” to which Mr de Garr Robinson referred were the proceedings already before the CCRC.

(2)

Secondly, the possibility of future (as opposed to current) criminal prosecutions, or the potentially criminal impact upon individual SPMs, did more than hover in the background to the Horizon Issues trial. Some claimants who gave evidence in this trial were expressly accused by the Post Office of criminal offences in crossexamination in this trial, something which had also occurred in the Common Issues trial. It should be clearly understood that any findings I make in respect of any witnesses do not determine with finality any issues to be tried in any particular trial to follow, including the trial(s) of their individual claims.

(3)

Thirdly, the rule to which leading counsel for the Post Office referred was not identified. Journalists must be alive to the risk of serious prejudice when criminal proceedings are live, as a result of the Contempt of Court Act 1981, and are thereby restricted in reporting matters generally in that respect. That statutory provision replaced what had been referred to as the sub judice rule, but this did not relate to statements made in evidence in open court. The principle of open justice is a most important one and there were no restrictions imposed on any reporting of the Horizon Issues trial (nor the Common Issues trial) at all, nor were any sought by the claimants or by the Post Office.

(4)

Fourthly, there was no specific restrictions imposed on the parties by the court in terms of the evidence they could adduce and what could be said by way of submissions (written or oral), due to potential impact upon any “different proceedings”. I have not heard any argument on the matter, but I do not consider it would be proper to do so in any event. It was also not argued before me that the CCRC proceedings, which are subject in any event to a stay pending at least part of the outcome of the group litigation, are in any way of such a character that reporting of the Horizon Issues trial ought to be restricted in any way, or such that they could be prejudiced by the group litigation. Indeed, the presence of a stay of those proceedings pending the outcome of some of the group litigation issues would suggest directly to the contrary. The objection therefore by the Post Office to the way in which the claimants sought to make closing submissions on the actual impact upon different SPMs of what were said to be bugs, errors and defects in the Horizon System was, so far as it could be understood, not a well-founded one.

(5)

Finally, the claimants by their counsel were trying to draw parallels between what had happened to one particular SPM regarding “doubling up”, and numerous entries in internal Fujitsu and Post Office documents prior to that of similar “doubling” occurrences being caused by bugs, errors or defects. Those parallels are obvious on the face of the documents. The degree to which the CCRC find such parallels of assistance, if at all, is a matter entirely for them. The Post Office submitted most strongly that these different references were not to one single bug, but were references to a number of different issues or bugs that had been experienced in Horizon over the years. Whether that makes the claimants’ point for them or not, it is important to remind all the parties that the issues in this litigation are not going to be decided with sympathy, or lack of it, coming into account in the analysis in any respect whatsoever.

65.

Mr Green for the claimants submitted that the Post Office was “trying to distract from the stinging nature of the underlying documents by objecting” to the submissions and that the Post Office may be finding it “uncomfortable” to have its own documents, in a particular date range, highlighted by reference to particular criminal cases, prosecutions mounted by the Post Office, and the experiences of SPMs in those cases. Certainly the Post Office and Fujitsu have been exposed to a degree of scrutiny in this litigation which does not appear to have occurred before; however, there has not been litigation of this type on these issues before.

66.

The claimants and the Post Office doubtless know this already – certainly their legal representatives will - but given the high profile nature of this dispute and the fact that this judgment may be read by those other than the parties themselves, I will make the position very clear:

(1)

The Senior Courts of England and Wales (who acquired this name by reason of the Constitutional Reform Act 2005) comprises the Court of Appeal, the High Court and the Crown Court.

(2)

The Court of Appeal has two divisions, Criminal and Civil. Appeals from the High Court are to the Civil Division. Appeals from the Crown Court are to the Court of Appeal Criminal Division, also known as the CACD.

(3)

The High Court is the highest court of first instance in civil cases.

(4)

The Crown Court is the highest court of first instance in criminal cases. Some criminal matters are dealt with by the High Court (by way of Divisional Court) but these are narrow in scope and do not arise in any respect concerning the group litigation. All appeals against both convictions and sentence from the Crown Court are dealt with by the CACD.

(5)

The Criminal Cases Review Commission, or CCRC, is a statutory body responsible for investigating alleged miscarriages of justice in England, Wales and Northern Ireland. It was established by section 8 of the Criminal Appeal Act 1995. It is an independent non-departmental government body, and has the power to refer cases to the CACD.

(6)

This group litigation is concerned only with the issues arising in the civil claims being brought against the Post Office by the claimants, and the Post Office’s counterclaims. It will result in a series of judgments on those issues which are public. What, if anything, the CCRC do in any respect following any of the judgments is entirely a matter for the CCRC and forms absolutely no part whatsoever of the group litigation.

(7)

This court has no jurisdiction in respect of any of the convictions of those SPMs who were successfully prosecuted by the Post Office. Although the presence of criminal convictions does have evidential effect in respect of individual claims by individual claimants who have been convicted of false accounting, these have not arisen in either of the two substantive trials held to date (Common Issues and Horizon Issues) nor will they arise in either of the next two (the principles governing Heads of Loss, and then some individual claims).

67.

There has been no restriction imposed on any party, or any witness, in this group litigation by the court in terms of the evidence that could be adduced, or submissions that could be made, with one exception. Mr Henderson of Second Sight was subject to a restriction upon his evidence by reason of the terms of an agreement that he had entered into with the Post Office when Second Sight were engaged in what was originally intended to be consensual resolution of the claims. This was called the Second Sight Mediation Scheme. This feature of the evidence of Mr Henderson is a matter which is dealt with in more detail in Part D of this judgment. It was a restriction imposed upon him, or agreed, by the parties, and the court was not involved in that restriction in any way.

68.

Turning to a different matter, I have already made certain criticisms of the Post Office in Judgment (No.3) in terms of how it had conducted itself in the litigation, and I had

also made criticisms of some of its senior witnesses. One of those witnesses, Ms Van Den Bogerd, was also called as a witness in the Horizon Issues trial. So far as I am concerned, she came to the court for the Horizon Issues trial with a clean slate in terms of whether her evidence on the Horizon Issues would be accepted or not. A different way of putting the same point is that simply because certain adverse findings had been made concerning the evidence she had given in the Common Issues trial, this did not mean that I adopted any particular starting position so far as her factual evidence was concerned for this trial. The court was entirely neutral in terms of starting position. Although it was certainly not a point that went to her credit that I had already made adverse findings of the accuracy of her evidence in the Common Issues trial, that did not mean that I started with any fixed view of the likely accuracy of her evidence in the Horizon Issues trial. Her evidence in the Horizon Issues trial is dealt with at [203] below.

69.

Further, before turning to the detail of each side’s evidence, an approach was adopted by the Post Office on occasion of seeking to adduce what was (or should have been) in reality evidence of fact, but by way of submission, or points made “on instruction”. Sometimes, depending upon the nature of the subject matter, such an approach is understandable or unavoidable, and may be unobjectionable. It is not therefore sensible to state that this should never be done in any conceivable circumstance in any trial. However, on important points that have been dealt with by a particular witness in their evidence of fact, it is not a suitable device to adopt. This was particularly done in terms of the cross-examination of Mr Coyne concerning evidence already given by Mr Godeseth in his cross-examination, about alteration by Fujitsu of a particular branch account. I deal with that in detail at [376] to [379] below.

70.

This was also done in Appendix 2 of the Post Office’s Closing Submissions, where (sometimes detailed) factual explanations were given in respect of bugs in the Bug Table. That appendix was compiled by different teams of solicitors and counsel, something explained by the Post Office when, some months after it was submitted, they discovered that three pages were missing and sought permission to serve them rather late. I granted permission for them to be added, as they had been prepared before the deadline for service and omitted due to an administrative oversight.

71.

Submissions should not contain evidence, or positive evidential assertions, that are not present in the evidence served in the trial. This is a fundamental point. I provide some examples in the Technical Appendix by reference to specific entries for specific bugs. Blurring (or ignoring) the lines between submission and evidence is entirely unhelpful. Evidence is something that comes from a witness (lay or expert) and which the opposing side is entitled to test by way of cross-examination. It is not appropriate for detailed factual assertions to be made in closing submissions that are not directly referable to evidence in the case. There is no way such factual assertions can be tested; if they come in closing submissions, there is no way that the opposing party can deal with those assertions in their own evidence, or even put relevant points to witnesses for the other party in cross-examination.

72.

Further, this is not a case that is being tried in a Specialist List, such as the Technology and Construction Court – it is a general Queen’s Bench Division case – but it readily could have been tried in such a list. It contains a great deal of technical subject matter, particularly in this trial dealing with the Horizon Issues. The two IT

experts have each given evidence in other computer litigation before. Such subject matter, and such expert evidence, is readily suited to analysis by the parties and precision, which is the usual approach of courts generally. In my judgment, this is particularly important in technical matters such as these. Bluster, unfounded assertion in cross-examination, detailed technical explanations of fact given “on instruction” and submission are rarely helpful.

73.

Finally, there has been a vast amount of highly detailed material deployed by both sides, not simply evidence of fact, and expert evidence, but also reference to a great many documents. The use of an electronic bundle has made this easier to manage than otherwise, but written submissions alone were in excess of 1100 pages in total from both sides. It is neither possible nor desirable to recite in this judgment, or in the Technical Appendix, or resolve, every single disputed item, or every single disputed fact, no matter how minor. I only make findings in this judgment that are necessary to enable me to resolve the Horizon Issues themselves. Simply because I do not specifically refer to a particular submission or piece of evidence, it should not be thought that I have not had regard to it. I have considered all the material, evidence (both factual and expert), submissions, and passages in contemporaneous documents, multiple times. The experts’ agreements in particular have been of great assistance, but everything has been considered. This judgment, together with the Technical Appendix, will be of substantial length, and to recite everything would very probably make it of unmanageable proportions. It is in any event far longer than is ideal, but that is partly explained by the fact it deals with the life of a complex computer system that spans some 18 years.

74.

In adopting this approach, I have borne very much in mind the overriding objective in the Civil Procedure Rules, the need for proportionality, but also the obvious need to provide a reasoned judgment. I have also taken specific account of the dicta of Males

LJ in Simetra Global Assets Ltd and another v Ikon Finance Ltd and another

[2019] EWCA Civ 1413. That case concerned foreign exchange trading, and claims for dishonest assistance and damages for deceit and conspiracy against a total of 12 different defendants, both personal and corporate. Although it concerns those matters, plainly very different to the Horizon Issues, it does state generally what ought to be included in a judgment. The requirement for a judge to give adequate reasons in a judgment is analysed at [37] to [46], with the expression “the building blocks of the reasoned judicial process” (used by Henry LJ in Glicksman v Redbridge Healthcare NHS Trust [2001] EWCA Civ 1097) approved at [42] by Males LJ. Four points are summarised at [46] of Simetra:

“Without attempting to be comprehensive or prescriptive, not least because it has been said many times that what is required will depend on the nature of the case and that no universal template is possible, I would make four points which appear from the authorities and which are particularly relevant in this case. First, succinctness is as desirable in a judgment as it is in counsel's submissions, but short judgments must be careful judgments. Second, it is not necessary to deal expressly with every point, but a judge must say enough to show that care has been taken and that the evidence as a

whole has been properly considered. Which points need to be dealt with and which can be omitted itself requires an exercise of judgment. Third, the best way to demonstrate the exercise of the necessary care is to make use of "the building blocks of the reasoned judicial process" by identifying the issues which need to be decided, marshalling (however briefly and without needing to recite every point) the evidence which bears on those issues, and giving reasons why the principally relevant evidence is either accepted or rejected as unreliable. Fourth, and in particular, fairness requires that a judge should deal with apparently compelling evidence, where it exists, which is contrary to the conclusion which he proposes to reach and explain why he does not accept it.”

75. I am acutely conscious that the first of those points, succinctness, is not likely to be achieved in this judgment. This is not only due to the nature of the subject matter, the Horizon Issues, but also the fact that both Legacy Horizon and Horizon Online are involved, and the system was brought in some time ago, namely the year 2000. Due to the nature of the claims brought (and the limitation issues certain to arise), the functionality of the Horizon system(s), robustness, and the other Horizon Issues had to be dealt with at an early stage in the group litigation. I will return to the other three of Males LJ’s four points after my review of the evidence, as well as his comments on the importance of contemporaneous documents at [48] and [49] of Simetra. I do this at [937] and following below.

D. Evidence of Fact: The Claimants

76.

The claimants originally served statements from a greater number of witnesses of fact than were in fact called at the trial. This was because, just before the Pre-Trial Review on 14 February 2019, the Post Office objected to the evidence of two particular individuals. Very shortly before the Common Issues trial, the Post Office had issued an application to strike out a significant number of passages in the six Lead Claimants’ witness statements, and that application had been dismissed in Judgment (No.2) “Strike Out”. On this occasion, although no application to strike out was issued, the Post Office sought a ruling at the PTR in respect of the evidence of Mr McLachlan and Mr Henderson.

77.

Mr McLachlan had been called as an expert witness in the criminal trial of Mrs Seema Misra, a SPM at West Byfleet in Surrey who was charged both with theft from her branch, and also false accounting. He was called for the defence. The sums in question were approximately £74,000. Mrs Misra pleaded not guilty, and her defence was that the Horizon system was to blame. She was convicted by a jury after a trial in late 2010 at Guildford Crown Court, and was sentenced to a term of imprisonment of 15 months. She was pregnant at the time of her conviction and imprisonment. Expert evidence was called by the Post Office at her trial from Mr Gareth Jenkins from Fujitsu. Mr Jenkins was not called as a witness by the Post Office in the Horizon Issues trial before me, but a large amount of the evidence from the Fujitsu witnesses was attributed to information directly given to them by Mr Jenkins. This was a controversial matter between the parties at the Horizon trial. The Post Office did not proffer an explanation for Mr Jenkins’ non-appearance as a witness; they were not obliged to do so. However, in their closing submissions and after all the evidence had been called by both parties, they did so. The explanation provided was by way of submission and not evidence. This is a matter with which I deal with further at [509] to [513] below. Prior to the PTR, the Post Office objected to Mr McLachlan’s evidence on the basis that it contained opinion evidence and the claimants did not have permission for it, which is something required under the CPR. This was resolved by the claimants deciding not to call Mr McLachlan, and explaining at the PTR itself

that they would not call him. On the face of it, the claimants were wise to concede this point as if his statement had included opinion evidence, the Post Office would have had a valid point of objection. I have not therefore considered the evidence of Mr McLachlan which forms no part of the case.

78.

The other witness whose statement was subject to objection was Mr Henderson. Mr Henderson is a Director of a company called Second Sight Support Services Limited (“Second Sight”). Second Sight was appointed by the Post Office to conduct a review into problems with the Horizon system in July 2012. A number of reports were produced by Second Sight, and the Post Office responded to these. These reports were as follows:

1.

Interim Report dated 8 July 2013;

2.

Briefing Report Part One dated 25 July 2014;

3.

Briefing Report Part Two version 1 dated 21 August 2014;

4.

Briefing Report Part Two version 2 dated 9 April 2015.

79.

The Post Office submitted at the PTR that I should give a ruling concerning those parts of Mr Henderson’s witness statement upon which the Post Office was required to cross-examine, and those parts upon which it was not. This was said to be justified by a concern the Post Office had, namely that if it were required to cross-examine upon the correctness or accuracy of the contents of the Second Sight reports themselves, the Horizon Issues evidence would not be capable of being completed within the time estimate for the trial, which at that stage was 16 days. Mr de Garr Robinson also expressed concern that any failure by the Post Office specifically to cross-examine upon any particular point in any of the accompanying Second Sight material, rather than the actual witness statement of Mr Henderson, would lead to the claimants submitting that such a point was not challenged, and that any such point would therefore be taken by the court as being formally agreed by the Post Office. I declined to give a ruling directing the Post Office upon those parts of Mr Henderson’s statement which should be subject to cross-examination. This was for three reasons.

80.

Firstly, the answers to the different Horizon Issues would not be determined based on the accuracy or correctness of the contents of the Second Sight reports themselves. The court was to hear detailed expert evidence from two IT experts whose evidence would go directly to the Horizon Issues. Whether the Second Sight reports were, or were not, correct in their conclusions (which were generally critical of Horizon) would not form part of that process, and would not even qualify as a sideshow. Certainly their contents would not be determinative of the Horizon Issues in the group litigation. I have had no regard to the contents of the different Second Sight reports, nor to the Second Sight conclusions, in arriving at the answers to the Horizon Issues. I am aware that all of the Second Sight conclusions were challenged by the Post Office.

81.

Secondly, it is not for the court in civil litigation to identify in advance to any litigant that it need not cross-examine upon particular evidence of fact of the opposing party. This is particularly so in a time-limited trial such as this one, but in my judgment is a general point of principle. The court was effectively being asked to direct the Post Office as to how it should conduct its case, and also consequentially, about how much of its time at the trial should be used for particular witnesses. This is not the function of the court. I was aware that Mr Henderson’s evidence was challenged by the Post

Office. It is for this reason that he was to be cross-examined. How the Post Office chose to do that, the trial time its advisers chose to allocate to that exercise, and which (if any) parts of the statement were to be challenged and how, were all matters for the Post Office to decide.

82.

Thirdly, it would in any event be wholly unconventional in a time-limited trial dealing with the subject matter of this group litigation for the court to give any weight to any submissions by the claimants that particular points of criticism or detail in such detailed documentation as the Second Sight reports were essentially “accepted” by the Post Office because they had not been specifically challenged in cross-examination. There is never sufficient time, in any time limited trial, to cross-examine upon everything. A time limited trial in the 21st century is not conducted in the same manner as a trial would have been many years ago, particularly in a detailed technical dispute such as this one. There was no forensic route available to the claimants whereby a failure to cross-examine upon the Second Sight conclusions or reports, whether due to the length of the trial or otherwise, would or could lead to a conclusion by the court that some aspect of the Second Sight reports was agreed or admitted by the Post Office, when it plainly was not. The period in question in the group litigation spans some 18 years. If attention is to be paid to other court users, the overriding objective and the Civil Procedure Rules, it is simply not possible (nor desirable) for any trial judge to permit the parties to have unlimited time to cross examine upon everything. The quid pro quo of that is a party cannot be expected to cross examine upon everything. There should have been no concern on the part of the Post Office that the Second Sight report(s) would be taken as not challenged by the Post Office unless their contents were subject to cross examination. Given there appeared to be such a concern, I explained the position at the PTR. The claimants also expressly said at the PTR that no such point would be taken. That latter element was unnecessary, as whether the point was taken by the claimants or not, it would not have been accepted.

83.

The position was therefore clarified or explained at the PTR and the Post Office, notwithstanding that no formal ruling was made on those specific parts of Mr Henderson’s statement that it had to challenge in cross-examination, appeared to be content. A transcript of this hearing is available. The contents of the Second Sight reports were not cross-examined upon when Mr Henderson gave evidence, but in my judgment there was no need for them to be. As it happened, Mr Henderson was not cross-examined for very long, but that does not matter. I have not taken any account of the contents of the Second Sight reports in deciding the Horizon Issues.

84.

The claimants therefore called the following witnesses of fact. For the reasons that I explain below in relation to Mr Roll for the claimants, and Mr Godeseth for the Post Office, I consider these two witnesses in particular to have been of the greatest assistance in resolving the Horizon Issues. However, all of the factual witnesses contributed to my understanding of the system over the years, both generally and specifically. The specific experiences of the claimant witnesses who gave evidence about what occurred in their branches are, if I accept them, specific examples of the working of the Horizon system at the branch Post Office end in practice. I will deal with each of the witnesses in turn.

Mr Latif

85.

Mr Adrees Latif was the SPM at Caddington Post Office in Caddington, Bedfordshire, since 2001 until late September 2018. His appointment with the Post Office therefore ended after the litigation commenced, and he was subjected to an audit in September 2018. He was cross-examined by video link from Islamabad, as it had been necessary for him to travel to Pakistan as a result of a family bereavement. He left the UK on 19 February 2019, and he travelled from Kashmir to Islamabad to give his evidence by video.

86.

He gave evidence about two specific incidents. One occurred in July 2015 and related to the transfer of £2,000 from the AA stock unit to the stock unit designated SP1. Originally his statement had said it was the SJ1 stock unit but this was a typographical error which he readily accepted. He successfully transferred the £2,000 from AA, but when he went to the SP1 unit, the same sum had not transferred into that unit successfully. There was no explanation for this that he could come up with, including having checked his own CCTV, and he was sure he had carried out the transaction correctly – it was not an unusual transaction. He described the sum of £2,000 as having “simply disappeared from Horizon” and explained that this would lead to a shortfall in the branch account for that sum.

87.

The second incident occurred in January 2018 in relation to Camelot and scratch cards. His heading for this he corrected to “Transaction Acknowledgement Issue”. He explained that certain information was sent by Camelot to Horizon twice, and the Post Office sent out a notice stating that due to this mistake by Camelot, a Transaction Correction or TC would be issued. The correction occurred and he accepted it, but the stock figure for scratch cards in his branch on Horizon remained unchanged. This therefore showed his branch as having £1,000 more in scratch cards than was actually present in the branch. This issue was still outstanding as at the date of his amended statement which was 1 March 2019, and remained outstanding as at the date of his cross-examination which was 12 March 2019. This is, obviously, a period well in excess of one year.

88.

The Post Office’s case on the £2,000 was put methodically to Mr Latif. That regarding the stock transfer is best summed up in the evidence of Ms Van Den Bogerd, who stated that provided certain steps or action were carried out correctly, what Mr Latif had said happened simply would not occur. She said, inter alia “providing these two actions are completed, the stock unit from where the cash is transferred should not show a discrepancy" (emphasis added). As she put it, her “strong belief is that Mr Latif has recalled these events incorrectly”. She also said that “the records that Post Office has reviewed do not support what Mr Latif has said and I believe that he may have mis-recollected events from 3 years ago”. The Post Office’s case was essentially that Mr Latif had not done the steps correctly, because had he done so, what he said happened could not have happened.

89.

Mr Latif’s response to that was both consistent, considered and credible, and was best summarised in one of his answers:

“A. I'm experienced -- I have been running a post office for 17 years, sir. I have also worked for the Post Office on training other offices how to run a post office. I was also involved in running and introducing the new Horizon software changes in 2006 onwards, where I went to several offices on behalf of the Post Office to give them training. So I'm an experienced, trained subpostmaster and I ran my business successfully for 17 years. So I may have been a bit brief in the statement but obviously I can run through those -- exactly those steps that we would take to make sure that there is no operator error on our behalf.”

90.

The conflicting evidence on this particular point is a good illustration of the “poles apart” position to Horizon by the Post Office and the claimant SPMs in this litigation. Because of the Post Office’s position on Horizon, almost all and any of the criticisms or accounts of factual events which the claimants made, or make, about how this system worked in practice are attributed to fault or carelessness by the SPM or their assistants. Indeed, without fault or carelessness by an SPM, the Post Office simply cannot explain these occurrences. The Post Office’s position is therefore to challenge the factual account – which it is entitled to do – because if the factual account by an SPM is accepted as truthful and accurate, then the Post Office would have to accept that there must be a fault or faults within Horizon. Therefore, the Post Office cannot accept that the factual account is truthful and/or accurate. Thus the dispute goes around and around in endless circles. This litigation is aimed at breaking that deadlock.

91.

It is also the case that Mr Latif had the following point positively put to him about why he had checked the CCTV that he had within his branch. “So you now say you looked at the CCTV because your colleagues were concerned that you hadn't done the transaction properly?” even though Mr Latif had said no such thing. Mr Latif was subject to a fairly robust attack, not only on his account, how it matched up with other records which the Post Office said contradicted it, what he and his assistants had or had not been doing, and indeed upon the full scope of his evidence and his credibility – as shown by the question I have reproduced. That question was framed as though even his own colleagues had concerns about what he had done. It was positively put to him by the Post Office that he had not even complained to the Post Office, although he provided the name of his Area Manager Mr Navjot Jando and said he had complained to him many times. The Post Office did not call Mr Jando to rebut this. One exchange will suffice as an example of the type of attack upon Mr Latif:

“Q. You don't say anywhere in your witness statement that the £2,000 physical cash also somehow disappeared, but that seems to be what you are now saying, is that right?

A.

Well, the system gave a shortfall of £2,000 and that's been my statement all the way through, sir, so I don't know what you're trying to confuse me, but there's a shortfall of £2,000 in stock unit AA and there should not be a stock shortfall. The money is physically there.”

92.

Mr Latif’s evidence had never been, so far as his witness statement and evidence orally before the court, that £2,000 in cash had physically disappeared. His evidence was that there was a shortfall of that amount shown in Horizon as a result of what he had done. There were some aspects of the cross-examination of Mr Latif which were simply unhelpful. Firstly, extensive spreadsheets were put to Mr Latif which he had not seen before. They are plainly not in chronological order, and had in any event been what was called “filtered” by the Post Office legal team; they were not agreed by the claimants, nor was the “filtering” process explained at any point. There is a limitation on the degree of assistance to be obtained by such an exercise. Further,

there was no agreed exercise by the experts whereby the two of them had gone through all the records directly relating to this specific instance and agreed what the records did, or did not, show. The Post Office’s case amounted, literally, to a pure challenge of fact that what Mr Latif said had occurred, simply did not occur in fact.

“Q. On the basis of that, Mr Latif, Post Office says there was no failed transfer such as that described in your witness statement and that you are simply wrong about that, it never happened.

A.

So you are calling me a liar?

Q. Mr Latif, you may be mistaken or you may be lying. I put the question that it didn't happen.

A. Well, I state that they did.”

93.

So far as the other issue experienced by Mr Latif in January 2018 was concerned, the Post Office accepted, as Ms Van Den Bogerd’s original written evidence (prior to later correction) put it, that its data showed “that the branch received two TAs (meaning Transaction Acknowledgement) on 18 January 2018. However, due to an error by Post Office, instead of increasing the scratch-card stock, the TAs decreased the stock. To be clear, this was a data entry error by Post Office and not an issue with Horizon. Horizon processed the TAs accurately. I note that the TAs were accepted by the branch, which could have been challenged at that point if the user had noticed that the TAs were not for a positive number, as they should have been.”

(emphasis added)

94.

However, this is notable for the following reasons. The first was that the blame is shifted back to Mr Latif for not spotting the mistake and challenging the TA. His evidence was that it was not possible to challenge a TA. They simply have to be accepted. This was accepted by Ms Van Den Bogerd when she came to give evidence orally. Therefore the notable point made by Ms Van Den Bogerd, a director of the Post Office, in her written evidence, to shift the blame back onto Mr Latif, is simply wrong in fact. TAs cannot be challenged, they have to be accepted.

95.

The second is that the Post Office evidence entirely omits any reference to the “memo view” – a type of communication used by the Post Office on Horizon which leads to a “pop up” on the terminal the next time each user logs on. There was simply no reference to this at all by Ms Van Den Bogerd, and Mr Latif said that this was sent out by the Post Office “to everyone” saying the error on the Lottery would be corrected by means of the issuing of a TC. Finally, Ms Van Den Bogerd corrected this passage in her evidence in chief by way of printed corrections prepared before she was called. She said that these corrections had been handed to her solicitors prior to the trial. If that is correct, it is surprising that Mr Latif was cross-examined on the basis of the original evidence in her statement, as that was not going to be her evidence in chief. However, regardless of that, she readily agreed that there is no choice regarding the acceptance of TAs – they simply have to be accepted by the SPM. In those circumstances, there is no explanation as to how the emphasised sentence in [93] above came to be in her statement at all, and it was plainly incorrect. Also, the call logs (about which Ms Van Den Bogerd was cross-examined) concerning this incident entirely support Mr Latif’s evidence.

96.

Finally, although this is a minor point compared to the ones in the preceding paragraph, the tenor of her witness evidence that Horizon processed the TAs in this respect gave the impression that there were no problems with the Lottery and TAs so far as Horizon were concerned.

97.

Not only did the steps taken by the Post Office – the TAs – not correct the issue that occurred in Mr Latif’s branch, but in cross-examination Mr Latif explained further:

“A. Can I just confirm, there was an audit done in September of this year, an audit by a Post Office trained auditor, and my stockholding was still showing negative. And a Jane Lawrence is the auditor and she has – still could not resolve this matter, so the problem hasn't gone away, the problem is still there. And there have been a number of calls to the helpline to resolve that negative stock and it hasn't worked. They haven't come back with a response.”

Mr Latif was very clear about this:

“A. I still state that an independent audit was done by the Post Office, a Mrs Jane Lawrence, in September 2018 the stockholding was still negative and as the branch was handed over to another subpostmaster that is going to be investigated, it's going to be investigated. If you are now coming back with this evidence, I still say that there is a problem somewhere and I don't know what's happened but we have still got a negative stock figure within our branch.”

He returned to this again, in the light of being pressed yet further on the point that it was human error at the branch that had caused the discrepancy or shortfall.

“So the fact remains there was an independent audit happened by your Post Office -- by Post Office's auditor, her recommendation was that they look into it as is (inaudible) the strategy is still there. And there were a number of calls to the helpline pleading with them to resolve this issue before the audit and there will be a complete trail of that, sir.”

He also referred to the what had happened as being “a glitch”.

98.

I accept the evidence of Mr Latif, who struck me as a reliable and careful person, and who had personally been the one who had tried to perform the transfer from one stock unit to another. He had personally experienced what he explained to the court in this respect. I accept his direct evidence on this in preference to that of the Post Office, which effectively was from people who were not there, who maintained, more or less, that it simply could not have happened, and who had nothing to substantiate or corroborate the challenge made to Mr Latif’s primary evidence. I find as a fact that it did happen as Mr Latif explained. I find that Mr Latif performed the required steps correctly in respect of the stock transfer between units, as one would expect of someone who had 17 years of experience, and was sufficiently skilled at his role such that the Post Office had, prior to the litigation, been sufficiently satisfied of his competence that he was used by the Post Office as a trainer for training other SPMs. I also accept his evidence, which is direct primary evidence of the state of the accounts at this branch, that neither he nor the auditor for the Post Office whom he named, have resolved the issue concerning the Lottery, the incorrect TAs and the effect of the

TC, which remains an unexplained shortfall or discrepancy. Ms Van Den Bogerd’s evidence is a number of steps removed from the branch, and is little more sophisticated than assertions that there must have been other matters to blame, alternatively reliance upon records which did not, due either to their contents or to the way that they were deployed in cross-examination of Mr Latif, demonstrate the points that the Post Office maintained they demonstrated. I make further findings in respect of Ms Van Den Bogerd’s evidence in the section below dealing with the Defendant’s Evidence in Part E. Mr Latif’s branch was subject to an audit, and his appointment as a SPM ended then or shortly afterwards.

99.

In due course Mr Latif will have his own individual claim, and the Post Office with have its individual counterclaim against him, tried. The only findings I am making in respect of his evidence are those necessary for me to resolve the Horizon Issues. All other issues remain to be tried in those later proceedings. Those later proceedings may explore in some detail not only the two specific matters in respect of which he gave evidence in the Horizon Issues trial, but any others which are relevant to both claim and counterclaim. They will do so by reference to other documents, as explained in the section of this judgment Part K, Audit Data. The degree to which the findings of fact that I make affect my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.

Mr Tank

100.

Mr Jayesh Tank was the SPM at Fleckney Post Office, in Fleckney Leicestershire, from 4 May 2006 to 15 March 2017 when it was closed as part of the Network Transformation Programme. He gave evidence regarding certain issues with Horizon that he had experienced, which included the effects of a complete power failure to the building, which occurred mid-transaction. This occurred whilst a customer was making a withdrawal in the branch from her Post Office card account. On his evidence, this led to a shortfall in his accounts of about £600. Mr Tank paid this sum to the Post Office, and the letter from the Post Office stating how it came to be owed described it as a “branch discrepancy”; it was in fact £660. He paid it by way of a deduction from his remuneration.

101.

The second issue was one concerning mail labels, when Horizon would (as he put it) “jump ahead” to the end of the transaction and no label (which is what would or should be affixed to the packet to be posted) would be printed. This would cause a loss in the branch accounts.

102.

Ms Van Den Bogerd again gave evidence about these specific instances. Her witness statement led Mr Tank to correct the date when he remembered the power failure occurring (he accepted it was 2014, not earlier in 2010 or 2011 as he had initially thought), and he corrected this in a supplemental statement. Her statement had helped him pin down the date. This incident became the subject of a PEAK at Fujitsu, although Mr Tank did not know this at the time. This supplementary statement also exhibited to it old “posts” he had made contemporaneously on an internet forum on the website Yahoo in respect of another loss, £195, which he had suffered in 2011.

103.

Mr Tank had been interviewed by the Post Office investigators in 2015 and in the

Horizon Issues trial it became clear that certain matters extremely critical of his conduct were going to be put to him in cross-examination. He was therefore given the warning against self-incrimination under the Civil Evidence Act in the same way that the two claimants in the Common Issues trial, who were accused in cross-examination of criminal offences, were given it. Mr Tank answered all the questions that were put to him. Mr Tank had, eventually, used the icon on Horizon in his branch for “Official Postage” incorrectly, and his explanation for this was that this was deliberate and was done in order to get the attention of the Post Office. He said at his 2015 interview that all his attempts by way of phoning the helpline and contact with his area manager had simply not resolved the numerous complaints he had made. He frankly accepted that he had not sent the post by official means which he had entered on the Horizon system as official postage, doing what he did by way of protest. It was put to him in cross-examination that he had “helped himself” to official postage, which on the evidence available was a suggestion that did not seem to have any basis in fact, if he had been doing what he said he was doing. There was certainly no evidence before the court to substantiate this assertion.

104.

In his interview on 5 November 2015 by the Post Office investigator Mr Bridges, it had been accepted by Mr Bridges (who interviewed him) that Mr Tank had reversed the entries for official postage in any event. This acceptance at the time by the Post Office investigator was directly contrary to the way that the points were put to Mr Tank in his cross-examination, which were that he had “helped himself” to postage and “improved his financial position” by acting as he did. Detailed findings on this will have to wait for the full trial of his claim and counterclaim, but it is notable, in my judgment, that the attack on the credibility of this witness in the Horizon Issues trial, and positive allegations to him of criminality, were not consistent with the contemporaneous record of the acceptance by Mr Bridges for the Post Office in the investigation interview in 2015. Mr Bridge clearly accepted that the entries had been reversed. Indeed, Mr Bridges introduced this subject in that interview as follows:

105.

“Okay thank you. Let's move on to the postage claims for the moment. So in my letter I gave details of the claims and the reversals that you completed since 25 August. I think my first question would be that I know that you reversed them but why undertake them in the first place bearing in mind these are, in effect you are stating the transactions which took, well say the transactions have taken place which you have used official postage for which in effect did not take place.”

106.

The questions in cross-examination to Mr Tank, which must have been put on instruction, did not take account of what the Post Office had accepted at that time in the interview. Another feature of the interview is that Mr Tank was told by the interviewer “there are no issues with Horizon”. That exchange is as follows:

“KB (Mr Bridges) I can confirm that there are no issues with Horizon and as my letter stated there is no issue with the production of labels.

Jay (Mr Tank) Then obviously you conducted your investigation into my concerns without asking me for my evidence. I've got the evidence here to show you. I'm happy for you to take copies but I'm going to hang on to it but how can you say something is fully investigated if you've not looked at all of the evidence.”

On the face of it, that is a valid point that Mr Tank made. Had there been an investigation, his evidence would plainly be something that ought to have been taken into account by any investigator.

107.

The Post Office relied, again, upon evidence from Ms Van Den Bogerd. Mr Tank was cross-examined by reference to this and also by reference to a technical explanation concerning the fact, it was said in cross-examination, that the power outage must have occurred at a particular point in time. The cross-examination was as follows:

“MR HENDERSON: So what appears to have happened was a transaction from the Post Office card account was in the middle of being processed and so it was in the stack presumably.

A.

Yes.

Q. But had not yet been posted to Horizon. So you hadn't cashed out on that transaction, you hadn't completed everything to do with that transaction?

A. On the stack – Q. It's on the stack.

A. It's on the stack, but the stack has a balance of zero, so to clear the stack you just press "enter" and it goes straight –

Q. But you hadn't got to the point of clearing the stack?

A. I'm not sure.

Q. Okay. My suggestion is that there was probably an outage at just the point where the money had been taken from Post Office card account but had not been processed onto Horizon. That's my suggestion to you. A. Okay.

Mr Justice Fraser: Well, is the witness going to be in a position to agree or disagree?

Mr Henderson: Well, he might be if he recalled.

Mr Justice Fraser: Do you recall that happening when there were outages? A. No. I cannot recall.

Mr Justice Fraser: Were you aware of when outages would occur like that? A. Not all the time.

Mr Justice Fraser: Do you want to put the question again?

Mr Henderson: Yes. What I'm suggesting is that the cause of this problem was that an outage occurred at a particular point in time. A. Yes.

Q. You were in the process of effecting a transaction from POCA?

A. Yes.

Q. It was in the stack and it had cleared from POCA?

A. Actually you mentioned the word "outage". I'm not -- was there a power outage?

Q. I'm not sure if it was a power outage, but I think it may have been a problem with the system.

A. Ah, okay.

Q. The system went down in some way.

A. So -- yes, because you said that if there's a power outage then there's evidence when you have to log back in, so did that happen on this occasion?

Q. Okay, I want to come to all this and I'm doing this clumsily. What I'm suggesting is that what may have happened -- and if you don't recall, you don't recall, but what may have happened is that the transaction was in the stack, the money had been taken from the Post Office card account and before you cleared the stack there was an outage . A. Possibly.

Q. Okay.”

(emphasis added)

108.

I have reproduced this passage because it demonstrates the following. The Post Office put a possible explanation, based on technical grounds, to the witness who did not have either at the time, or at the trial, the technical expertise, or the relevant technical records, or the data, to enable him to agree or disagree. Even that explanation itself accepted “there may have been a problem with the system” and “the system went down in some way”. The witness’ answer, which in my judgment is the only sensible answer that could sensibly be given in these circumstances, was “possibly”.

109.

This type of evidence obtained on cross-examination is of no assistance in resolving the Horizon Issues, other than an implicit acceptance by the Post Office (on this occasion) that outages could, potentially, in technically terms lead to what Mr Tank experienced. In any event, the Post Office’s explanation accepted that “there may have been a problem with the system” which is equally consistent with the claimants’ case, as it is with the Post Office’s case, if not more consistent with the claimants’ case than with that of the Post Office. Power outages do happen, and there is a process to be followed. Given Mr Tank was using Horizon Online, he was then in his crossexamination taken to Version 5 of a HOL quick reference guide. The version to which he was taken was not, however, the version available to him at the time, as that was a single A4 double-sided version in his branch, and the one put to him at the trial was longer than two pages and available online. The date attributed in the electronic trial bundle to Version 5, the version put to him in cross-examination, is 30 July 2015. His forum posts and his supplementary witness statement made it clear that the incident occurred in September 2014, a later date admittedly than given in his first statement, but still somewhat earlier than July 2015. This was however the version of the guide that Ms Van Den Bogerd relied upon in paragraph 78 of her 2nd witness statement to challenge Mr Tank’s account. I find the version that he would have had available at the time was not Version 5, and was the single A4 version that he had available to him in his branch in September 2014. However, and regardless of this, Mr Tank accepted that recovery receipts would or should be generated and set out the procedure. He also accepted that giving all the receipts to the customer was not the proper procedure.

110.

In the PEAK dealing with the shortfall of his accounts of £195, which is PEAK PC0214226 and is headed “Failed Recovery Transaction(s)” Fujitsu recorded:

“Date:14-Dec-2011 10:43:53 User: Wayne Bragg

Summary:

The banking transaction had completed (A3 rec'd and authorised @ 13:33:37), including the receipt print (13:33:42), and money should have changed hands.

The basket settlement failed from 13:35 with 'No response received from data centre', and the two retries also failed, and the attempt CANCELLED at 13:37.

The Disconnected Session receipts show "Cash TO CUSTOMER 195.04" so the customer's account should be correct but the branch will have a shortage (for a withdrawal) because the session hasn't been recorded.”

(emphasis added)

111.

In my judgment this entry clearly supports Mr Tank’s evidence about how Horizon operated on the occasion to which he referred. The heading of the PEAK was “failed recovery transactions”. The PEAK clearly states that the customer’s account would be correct – they were given the cash of £195.04 as they expected, but “the branch will have a shortage (for a withdrawal)” as the session was not recorded. A transaction correction was issued. There were related PEAKs, and also there was an entry “KEL acha959T may be relevant” on the PEAK, and Anne Chambers was referred to. The PEAK also recorded “Sending to SSC for investigation.” That KEL was raised by Anne Chambers on 28 February 2010, and the title was “HNGX banking reconciliation – state 4”. This is an important KEL, and I deal with its further detail in the Technical Appendix.

112.

There was a dispute between Mr Tank and counsel for the Post Office about whether this was an example of “Horizon working as it is supposed to”, which Mr Tank did not accept, and also whether he would have been refunded the sums (which he accepted had occurred) had he not phoned in and reported the problem. It was put to him that he would have been refunded even had he not called in to report, although how this would have occurred given the way PEAKs are initiated was not explained to him. In any event, the PEAK demonstrated that what Mr Tank had said occurred, had indeed occurred. Transaction Corrections are issued outside the scope of the Horizon system.

113.

The position regarding the procedure for spoiled labels was, again, a situation whereby the Post Office’s explanation was this simply could not have happened. Ms Van Den Bogerd’s evidence was that there was a procedure available that Mr Tank should have used, yet the document put to Mr Tank that was said to support this stated expressly in terms that “the label could only be spoiled if the label was on hand” and Mr Tank’s point was that no label was “on hand”, given the problem he had was that the label did not print at all. Using the procedure in the document therefore would not be possible, and indeed would be contrary to the Post Office instructions that it could only be used if the label was on hand. These instructions also required the SPM to write “spoilt” on the label and keep it with the Horizon receipt for two years. This could not be done if the label had never printed. The explanation by Ms Van Den Bogerd, put to Mr Tank, was also directly contrary to the contents of a letter dated 7 September 2015 from the Post Office to Mr Tank which stated:

“Your enquiry has been investigated and I can confirm there are no issues with Horizon online or the production of postage labels which would cause the situation you described. However if a user either pressed the yes or return key quickly before screen messages appear this can lead to a user confirming a postage print has happened when in rare circumstances it may have failed and put the cost of the failed print into the basket. In this circumstance Subpostmasters should contact Network Business Support Centre to arrange a credit for the spoiled postage.”

This means that the Post Office in that letter accepted that in some circumstances, depending upon how quickly a key was pressed, the printing of a postage label could fail and cause an impact to branch accounts. The SPM must make a telephone call to the helpline, to correct the fact that the branch account would show a charge for the postage even though no label had been printed and hence could not be provided to the customer. This letter suggests not that Mr Tank had not followed a particular procedure for dealing with spoiled (rather than non-existent labels) but that the system would, depending upon how quickly a user pressed a particular key, not print a label, even though the cost of the failed print would go in the basket. By going “in the basket” this means the branch accounts would include the cost of that label as a debit to those accounts. In other words, this letter supported Mr Tank’s direct evidence.

114.

The Post Office in cross-examination put the following to Mr Tank:

Q: There were procedures built into Horizon to cater for the situation that you explained -- I have to say in the vaguest of terms, but as I understand what you are saying, there were procedures in place which ensured you could deal with the situation, weren't there? A. No.

Q. We will have to differ.”

I would express this rather differently. If the speed of pressing a key could lead to the cost of a printed label being added to the basket (and hence branch accounts) even though that label had not printed, it is difficult to see that there is a “procedure built into Horizon” as defined to cater for this. On the contrary, it appears from the Post Office’s own letter in 2015 that there was no such procedure. For the avoidance of doubt, I find that there was no such procedure built into Horizon. The evidence demonstrates to the contrary. Phoning the helpline and asking for a refund is not “a procedure built into Horizon”.

115.

Further, it is difficult to see how the explanation proffered by Ms Van Den Bogerd regarding how to deal with a spoiled label could apply to Mr Tank for two reasons. Firstly, I accept that as a matter of common sense interpretation and language, a label cannot be “on hand” if it was never printed. Mr Tank did not consider he could properly use the procedure in the guide for the problems he experienced, and I agree with him. Secondly, the Post Office’s own explanation to him in 2015 was very different to what he was told in cross-examination in 2019 that he should have been doing. Even were I to assume (in the Post Office’s favour) that there must have been instructions from the Post Office somewhere regarding how a SPM should behave when a label failed entirely to print, as print failures (which would not produce a label “on hand”) were plainly not unheard of, it remains to be seen what those instructions were, and whether those instructions conflicted with what he was told in the letter in 2015. No such instructions have been produced by the Post Office in any event.

116.

I consider Mr Tank to have been a credible witness. I find that Mr Tank’s experience in branch was not an example of Horizon working “as it was supposed to”. I find that the creation both of a PEAK, and reference to a KEL within that PEAK, to be consistent with my conclusion. The proposition that Horizon worked as it was supposed to is an obviously flawed one. Indeed, that proposition is also inconsistent with common sense. PEAKs and KELs are not created for situations where Horizon is working “as it is supposed to”. They are created to deal with errors. The acronym KEL is for Known Error Log. Nor are matters referred to the SSC for investigation when Horizon is working correctly. Whether what occurred afterwards was an example of Horizon working correctly depends on the categorisation of Transaction Corrections and whether they are part of the Horizon system or not. I explain later in this judgment that they are not, therefore the issuing of a TC cannot be Horizon working as it should. Mr Tank also gave evidence, which I accept – and this is made out in the contemporaneous documents – that he had some difficulty in advancing this matter through the correct channels. The helpline operator with whom he was dealing initially refused to “pass up” the matter to a more senior person. Whether this was an isolated instance of unhelpfulness in this single case, or a more generally obstructive approach across the helpline, will have to wait for future trials in this litigation, and I recite it for completeness only in respect of Mr Tank’s experience. It does not form part of my consideration of the Horizon Issues.

117.

Mr Tank was accused of criminal offences and it was said that he had “helped himself” to official postage and had “taken official postage”. These accusations were not put to him at his interview in 2015 and indeed the text of that interview shows that the Post Office interviewer expressly accepted that he had reversed the transactions, which is not consistent with Mr Tank “helping himself”. There are 10 pages of detailed closing submissions by the Post Office about Mr Tank’s evidence, and although he is criticised for being vague, confused, imprecise and for not having prepared his evidence with care, the allegation of criminality is not raised. However, the point was positively put to him in open court, and Mr Tank is entitled to a finding on it. In my judgment, it would be quite wrong to leave such an accusation hanging in the air, unresolved in this way. I find that on the material before the court in this trial and on the evidence available and put to him, these serious accusations are not made out. That is not to say I have made binding findings on the full details of his claim which remains to be tried in due course. The Post Office submitted, both in respect of Mr Tank and other of the claimants’ witnesses, that they confined their evidence to what was described as “a small sub-set of his complaints, presumably because it was felt that this threw light on the Horizon issues – and Post Office’s evidence was similarly confined”.

118.

However, there are two important points that must be borne in mind about that submission, both as it is made in respect of Mr Tank, and indeed the other witnesses called for the claimants. This has not been a trial of their individual claims. There may be other issues about Mr Tank’s branch accounts, raised both by him and by the Post Office, at the full trial of his claim and the Post Office’s counterclaim. That does not mean that this specific evidence, about specific incidents experienced on Horizon, should be discounted. Indeed, this specific evidence is not only directly relevant to the Horizon Issues, but is, in my judgment, very important evidence going to those issues. The second point is the parties’ agreed wording of the Horizon Issues does not allow for any wider issues going to Mr Tank’s branch accounts to be considered in this trial. All the other issues between Mr Tank and the Post Office remain to be tried in the later proceedings. The only point of difference between them on one of his two issues concerning Horizon was whether what occurred when a transaction correction was issued could properly be described as “Horizon working as it is supposed to”, and certainly the KEL referred to in the PEAK appeared during the Horizon Issues trial more than once. The creation of a PEAK, and the KEL to which reference was made

within it, in my judgment demonstrates that this is not Horizon working as it should, as does the issue of a TC. The wider consideration of these matters that are dealt with both by the experts and later in this judgment. I reject the suggestion that Horizon was working as it was supposed to on this occasion. So far as Mr Tank’s experience with the £195 shortfall is concerned, there was a Horizon Online failure; three identical receipts were printed, which should not have happened; the receipts all showed a disconnected session; and there was a loss in the branch accounts that evening for the amount paid out to the customer. This was even though Mr Tank had settled with the customer for the amount specified on the receipt. There was no record of the transaction at all on the transaction log that was produced for the period of the Horizon Online failure. None of these, in my judgment, are examples of “Horizon working as it was supposed to”.

119.

The operation of the helpline is not part of the Horizon Issues, so it is not necessary to consider and make findings on what Mr Tank said the helpline told him, which he explained in his posts was “that the loss is mine unless I can sort out with customer directly.”

120.

The later proceedings concerning Mr Tank’s claims (and any counterclaim) will probably hear further evidence which are relevant to both claim and counterclaim. They will do so by reference to other documents, as explained in the section of this judgment Part K, Audit Data. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.

Mr Anup Patny

121.

Mr Anup Patny became the SPM of Spencefield Post Office in Leicester in October 2014. He is currently suspended by the Post Office, having been suspended on 17 August 2016. His son, Mr Aakash Patny, also gave evidence. I shall refer to Mr Anup Patny as Mr Patny Senior for this reason. I shall also deal with his evidence, and that of his son, together for reasons that will become apparent.

122.

There was what he described as a “major system outage” on 9 May 2016 in his branch, which was a single counter branch run in conjunction with his retail business in the same premises. This was just a few months before he was suspended. This outage led to the closure of his branch and he believed that this had affected the whole network. At the end of the next trading period, which was on 11 May 2016, there was a shortfall of almost £17,000, predominantly made up of a shortage of 16,000 coins of £1 denomination. This was a very large number and he said that he knew he would never have such a large amount of £1 coins in his branch. He was also the person in the branch who had dealt with the cash delivery that took place on the relevant morning, which is called “remming in” the cash. This means accepting cash into the branch from the cash delivery that Post Office makes to the branch, and entering the cash received into Horizon. His son, who also gave evidence, followed this matter up with the Post Office. On 19 May 2016 the Post Office contacted him about a discrepancy in respect of stamps, which so far as he was concerned related to the £16,000 regarding coins.

123.

The Post Office accepted that there had been an outage but disputed that this either did, or could have, resulted in the discrepancy which Mr Patny said occurred at his

branch. Ms Van Den Bogerd gave evidence about this, and the Post Office’s position is best summarised in the explanation put to Mr Patny Senior in his crossexamination:

“Q. Well, what I'm suggesting is that the most natural explanation for this, whatever adjustments were made, is that at some point on 11 May someone hadn't counted a big pile of £10 notes. They had been put in a safe and forgotten about, which is understandable, and that they were found the next day, or located, and there was an accurate cash declaration on 12 May. A. I don't know about that, sir.

Q. It's perfectly plausible, isn't it?

A.

I can't say anything to that.”

124.

This suggestion by the Post Office is, in my judgment, somewhat fanciful. A “big pile of £10 notes” is not the sort of thing that ordinarily gets “put in a safe and forgotten about”, and then suddenly “found”. It was also put to the witness that “things were pretty chaotic in your branch when it comes to these sorts of things” and “the cash declarations look like they are all over the place”. Mr Patny Senior did not agree with this characterisation of the way his branch was run. He certainly did not agree with the “big pile of £10 notes” theory the Post Office put forward. His evidence was also that it was his son who had contacted the helpline, and who had made the adjustments to Horizon that he was advised to do by the helpline. Initially the Post Office sought to challenge this account through cross-examination of Mr Patny Senior only. Counsel for the Post Office did not initially intend to put questions on this to his son, even though his son was plainly the correct witness to answer the detailed questions, as he was the person who said he had done certain things, and he was about to be called as a witness. The reluctance of the Post Office to put questions to the person who was plainly the relevant person to ask about this was said to be because “the records are the records” and the Post Office did not intend to repeat cross-examination that had already been done. I was not prepared to permit this. I required questions about what Mr Patny had, or had not, done, to be put to him directly, and not have the crossexamination conducted by proxy through his father. Mr Patny Senior’s son had given a witness statement in respect of these matters, and was plainly the correct witness to whom the questions should be put. He was the next witness listed on the trial timetable, and was going to be called next for the claimants. The notion that questions challenging what he had done, or not done, should more properly be directed to his father is contrary to how cross-examination should be conducted.

Mr Aakash Patny

125.

Mr Aakash Patny is the son of Mr Patny Senior, and worked in the branch and retail premises with him, as did Mrs Patny, the wife of Mr Patny Senior and mother of Mr Patny. He is not a claimant. He was given the warning against self-incrimination under the Civil Evidence Act. He did however answer all the questions put to him.

126.

Mr Patny assisted his father, particularly with balancing. He would usually arrive at lunch time, and so was not present when the national outage occurred on the morning of 9 May 2016, which had been resolved by the time he arrived that date. He gave evidence concerning the same amount of money, which both he and his father considered a shortfall, and which the Post Office had claimed was due to it and was a

branch discrepancy for which the branch was responsible. It had been settled centrally but had not been paid to the Post Office.

127.

So far as 11 May 2016 was concerned, when Mr Aakash Patny had arrived at the branch he became aware of the shortfall of over £17,000, which was a “complete surprise” to him and which he considered could not be explained. He had contacted the helpline, and had followed the steps he was advised to take, which he thought had resolved the issue. These steps were things he did on the Horizon terminal, and sequences of entries he was talked through over the phone. The documents available supported his evidence that he had made such a call, although some of the entries did not match what he said he had told them. There was no record of what he was told to do, nor was there any record put to him showing what key strokes had been inputted at the branch.

128.

The Post Office’s case on the shortfall was that the outage on 9 May 2016 did not lead to any discrepancy. It was not put to Mr Patny that he had not made key strokes as advised over the telephone, or that he made different keystrokes to those advised. All that was put, somewhat unclearly, was that “the problem” was resolved, cash had been physically “found” that had been missed at the time of the previous cash declaration, and also that a figure showed on a cash declaration of “plus £17,000 odd” had the effect of “cancelling out” the previous discrepancy and showed that “there was no longer a cash problem”. Mr Patny denied any money had been found that had been lost; denied that if a discrepancy had been cancelled out the figure would have been plus £17,000, rather than zero; and said that although he thought following the key strokes he was told to input had resolved the issue, it had not. He was told to “readjust” the cash stock figure, something he had never done before, and relied upon the instructions he was given by the helpline. The Post Office’s cross-examination was done by reference to the cash management report, not the audit data. I will return to audit data later in this judgment in Part K, but the audit data would have showed what keys were pressed, in what sequence and when. Such data would have been of great assistance, but was not produced by the Post Office.

129.

On 19 May 2016 the branch was contacted by the Post Office, in respect of an overdeclaration of stamps. The amount of this was approximately the same value as the amount of the discrepancy of cash had been a few days earlier. He had done the declaration for stamps. A person called Debra Lambley phoned the branch. He was struck by the co-incidence in the amount of the stamp discrepancy, and told Ms Lambley the stamp declaration was impossible due to the number of stamps this represented. To assist a reader of this judgment on that latter point, but something that was not explored in the evidence, to obtain a broad idea of how many stamps that would represent, the following points are relevant. On 28 March 2016 the Royal Mail announced an increase to take effect that week in the price of stamps. 1st class stamps became 64p each, and 2nd class stamps became 55p each. This shortfall would therefore have represented 25,000 1st class stamps, or over 29,000 2nd class stamps. It goes without saying that this is a great deal of stamps.

130.

He was instructed to re-adjust the stamp stock figure, followed her instructions as to how to do this, and the system then showed the £16,000 shortage. It was by now past 7.00pm, he phoned the helpline, they contradicted Ms Lambley and he then had to re-

declare the following day, being unable to do so that day as it was after the 7.00pm cut off.

131.

The Post Office’s case on this, as it was put to Mr Patny, was firstly that he had made a mistake in declaring the stamps. He denied this. The circularity of the Post Office’s case can be seen from a lengthy passage of evidence, which I will not reproduce, from page 26 on that day’s transcript to page 35. Mr Patny explained, basically, that the helpline gave instructions to resolve the cash discrepancy; this resulted in the figure for stamps being wrong by the same amount; Ms Lambley was not prepared to have this, and following her instructions this resolved the stamps situation, but the cash discrepancy reappeared. Then the sequence happened a second time. Mr Patny was prepared to accept he may have made a mistake once, but denied he could have made exactly the same mistake the same way a second time.

132.

It was also then positively put to him that one way of disguising a shortfall in cash would be to over-declare the number of stamps. A positive allegation of dishonesty was put to him, which he rejected.

133.

The premise behind this suggestion of dishonesty was that it was said by the Post Office that if the stamps figure went up, the cash figure would automatically go down by the same amount. He did not agree with this. It was put as a positive statement in the following terms:

“Q. I suggest finally, Mr Patny, it is rather surprising, given that you do all the balancing, or did all the balancing in the branch, that you don't understand that the effect of increasing the declaration of stamps is to have a corresponding effect on the declaration of cash?

A.

I wasn't aware of that, no.”

134.

There was no direct evidence to support the proposition that by increasing a declaration of stamps, it would have a corresponding effect in the opposite direction upon the declaration of cash held in the branch. In any case, in the Common Issues trial, there was evidence that making a cash declaration required specifying how many notes of each denomination was physically held in the branch. An SPM had to declare both the cash and the stock that was held in the branch. It is difficult to see how changing the number of stamps held in a branch would, or could, affect the physical counting and declaration of the amount of cash held in a branch by way of specific denominations. However, and in any event, the allegation of dishonesty against Mr Patny was a line of cross-examination that was not supported by evidence from Ms Van Den Bogerd, who was expressly asked about it, and who accepted that she had not made an allegation of dishonesty in her witness statement and said she did not offer an opinion on whether Mr Patny was dishonest or not. That means that there was no evidence from any witness called for the Post Office to support the allegation of dishonesty.

135.

The second problem in relation to which Mr Patny gave evidence was a MoneyGram transaction in February 2016. This related to a failed payment by card by a customer, who tried to send £3,100 by MoneyGram but whose card was declined twice. However, the branch accounts still showed two debits in that amount in the branch accounts that led to a loss showing at the branch of £6,200. A TC was issued but only for one of these, which led to a £3,100 loss. Mr Patny accepted that he had only cancelled the transaction at the time, and not reversed it as well (which was required) until later, in the evening, when advised to do so by the helpline. As he put it, the amount had “doubled up”, in that the MoneyGram transaction, which was cancelled and reversed later on (which was for £3,100) had led to a loss showing of £6,200. If that is correct, then it is indeed double the value of the transaction. Ms Van Den Bogerd said that this was “not supported by the data” and also relied upon the fact that the cash shortfall on the day was greater than £6,200, namely about £500 greater than the cash shortfall which Mr Patny said had been caused by MoneyGram. She could not proffer an alternative explanation, however, and in her witness statement said:

“On 23 February, the branch declared cash holdings of £25,803.87, a net downward movement of £8,601.59. The net value of transactions during this period resulted in a £1,806.71 decrease in cash. The Moneygram transaction described above would account for a further £3,100, bringing the total explainable cash movement to a £4,906.71 decrease. However, this leaves £3,694.88 of cash movement unaccounted for. I cannot say for certain what caused this additional loss of cash but there is nothing in the accounts that suggest a problem with Horizon. It appears more likely to me to be a problem with cash handling in the branch or a user error when making cash declarations.”

(emphasis added)

136.

This statement does not answer the evidence of Mr Patny on the point, and the more that counsel for the Post Office tried to press the point(s) in cross-examination, the more stark it became that this evidence by the Post Office was no answer. Firstly, it assumes that the MoneyGram transaction was only responsible for £3,100, not the figure of £6,200 which Mr Patny stated. It assumes that £3,694.88 does not include the lower figure of £3,100, which in my judgment, on the documents deployed, it plainly does. Secondly, Ms Van Den Bogerd cannot say what caused the additional loss of cash. Thirdly, it states that that there is “nothing in the accounts that suggest a problem with Horizon” (emphasis added). That is rather stating the obvious. If all the problems complained of by the claimants with Horizon could be seen in the branch accounts, this litigation would not be occurring. Fourthly, she retreats to criticism of the way cash is handled in the branch, or other mistakes – “user error”. It is a perfect summary of the Post Office’s case, but it does not provide an answer. Other than vague “user error”, there is no explanation provided. I find that there is nothing in any of the documents to suggest user error.

137.

Additionally, Mr Patny explained that he had fully explained all of this to the area manager Mr Irwin, who agreed with him that the MoneyGram transaction had appeared to “double up”. Further disclosure by the Post Office when the trial of Mr Patny Senior’s claim comes to be tried, together with evidence from Mr Irwin if he is called, will provide further information. Finally, the exercise undertaken by the Post Office in cross-examination failed to take account of the fact that one had to consider both the cash figure, and the shortfall, from day to day to make sure that one is comparing apples with apples. I consider the exercise undertaken in crossexamination to be flawed.

138.

I cannot resolve this purely on the factual material put before the court by the Post Office. The audit data showing the actual key strokes inputted will be required in order to do that, together with findings on the expert evidence and my findings on the existence of bugs, defects and errors within Horizon. However, I do find that Mr Patny came across as a careful person, and as a credible witness, and I accept that he took the steps he did as explained in his evidence. He accepted that he had not reversed the transaction until later, rather than immediately as he ought to have done. The resolution of the movement of the cash position, and what if any explanation is available, simply cannot be done with finality at this stage on the basis of the factual evidence before the court alone. The Post Office had, according to Mr Patny, failed to provide the Credence data which would make it clear what had happened and how. The helpline logs show that Mr Patny had chased this on numerous occasions, and also the entries suggested that a Credence report was being prepared. The audit data showing what key strokes he had undertaken and when, was not produced by the Post Office.

139.

There was no evidence before the court to support the allegation of dishonesty that was expressly made against him, and it was not supported by Ms Van Den Bogerd. On the material before the court in this trial, I find the allegation of dishonesty against him not to be made out and I reject it. That is not to say that the subsequent trial dealing with Mr Patny Senior’s claims will automatically accept all the evidence given by his son, if he calls him as a witness. All future issues in individual claims will be dealt with, and resolved, on the evidence led by the parties in those subsequent trials. I am only deciding matters in this judgment that are necessary to resolve the Horizon Issues. However, in circumstances where positive allegations of criminality are expressly put to a witness in a trial such as this, it is only fair to deal with those allegations. They cannot as a matter of fairness be left hanging in mid-air, particularly where there is no evidence provided in support from the party making the allegation in any event.

140.

The later proceedings concerning Mr Patny Senior’s claims will explore in some detail not only the specific matters in respect of which he and his son gave evidence in the Horizon Issues trial, but any others which are relevant to both his claim and counterclaim. They will do so by reference to the evidence advanced in that later trial and the other documents, as explained in the section of this judgment Part K, Audit Data. The degree to which the evidence of Mr Patny Senior and his son affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.

Mrs Burke

141.

Mrs Angela Burke had worked for the Post Office in various roles since she was 16. Most recently, she worked in the Newport Post Office in Brough where her husband was the SPM. She was a Branch Assistant and had worked as an assistant for about 15 years, and had also previously been a SPM herself. She and her husband had an associated greeting cards and stationery business, and the branch was closed in October 2017 as part of Network Transformation. The card business has also closed.

142.

Her evidence related to events of 9 May 2016 when the national outage occurred, the same date as the occasion about which both the Mr Patnys had given evidence. She gave evidence about the impact upon the branch business, the way she was serving customers and how Horizon was being very slow that day, with a sand timer appearing on the screen for a very long time. She served one customer, who was making a cash withdrawal, as she obtained the relevant messages and approvals on the screen. However, after they had left, a receipt printed saying “Recovery failed” and the withdrawal of £150 was not shown. She then later studied the transaction log and this latter transaction did not appear.

143.

She realised what was happening and decided to close her branch that morning. She sat down and worked through all the transactions in her branch, using what was available to her. She was given an explanation by the helpline which she did not accept, as she knew and could remember that the payment to the customer had been authorised, which is what led to her handing over that sum. She also was engaged in a contemporaneous exercise whilst it was very fresh in her memory. She said there was no alert from Horizon, and “there was no means through the Horizon system for the discrepancy to be identified or for its cause to be established in my situation”. The transcript of this call was available – she and her husband had obtained it using a subject access request from the Post Office.

144.

Mrs Burke then went to extraordinary lengths. She also proved herself very tenacious, as many people may well have simply given up on the sum of £150. She identified the customer, and she tracked him down. She went to his house and explained what had occurred. He happened still to have the receipt from the transaction at her Post Office. It entirely matches her account. She went with the customer to the customer’s bank, which was the TSB in Goole. She explained with the customer to the bank cashier what had happened, and the cashier printed out the bank statement and showed that the sum had been withdrawn from the customer’s bank account. The customer permitted Mrs Burke to have this.

145.

She and her husband then pursued this through the helpline and their call was escalated. They did receive a TC for £150, but actually that referred to an amount in respect of Lloyds Bank and not to TSB. This was pointed out to the Post Office who claimed there was no code on Horizon for the TSB. If that is correct, I find it highly surprising that there should be no separate code for the TSB. The bank currently known as Lloyds used to be known as Lloyds TSB – there are cases on bailii that use that name in relation to other litigation, for example – but the TSB was split from Lloyds TSB, which then became simply Lloyds Bank. This split occurred some years before the events of which Mrs Burke gave evidence. TSB and Lloyds Bank had been separate for some time by May 2016.

146.

Mrs Burke’s evidence was explored in very considerable detail in cross-examination, including the fact that she chose to keep the stack open in order to serve more than one customer at once, how she had served them and the other details of what had occurred. She was challenged on what was said to be a “relatively small” point, of whether she would have received a TC from the Post Office without having done all the work that she had done, in tracking down the customer and going to his bank. I reject that categorisation of the point by the Post Office. It is not a small point. It is, in the context of the overall litigation, a major point. It is difficult to see, given the evidence collected by Mrs Burke which was provided to the Post Office, how they could have failed to issue her with a TC. Whether this TC would have issued absent the work that she did to demonstrate the fault and its effect is not a “small point”.

147.

There are really two points that are relevant. The first is a broader one: is the process for issuing TCs part of the Horizon system? The second is a case specific one: would, on the balance of probabilities, the Post Office have issued a TC in her case without the evidence that Mrs Burke herself collected and provided to the Post Office?

148.

So far as the first point is concerned, the process for issuing TCs is not part of the Horizon system. That is addressed elsewhere in the judgment. So far as the second point is concerned, the documents put to Mrs Burke demonstrate that Fujitsu had worked out what had occurred; that the internal document attached to an internal email of 12 May 2016 did not accept that Newport branch had in fact paid the money to the customer, even though Mrs Burke had told them this on 9 May; and that by 16 May 2016 the decision had been taken that the branch would receive a TC anyway. Her response to the “small point” was “possibly, yes”.

149.

The conclusion that I draw from this is that the Post Office required something more from Mrs Burke than her word that the money had been paid out by her branch to a customer – the internal documents refer to “not knowing” if the money had been paid, even though she had already told the Post Office this. I find that the fact that this is a Fujitsu document rather than a Post Office document does not matter for this purpose. That something more that was required, in her case, was the evidence she had personally compiled that was of great weight, and was accepted by the Post Office, as the TC was issued. Whether evidence of lesser weight would have been accepted by the Post Office is not possible to say, and is entirely hypothetical. Fujitsu had, though, identified this particular problem experienced at the Newport branch by the national outage. The information of what occurred, and the problem, was not visible to the SPM or their assistants in the branch.

150.

Counsel for the Post Office’s “relatively small point” put to Mrs Burke, that the Post Office would have issued her with a TC anyway, is in fact not entirely consistent with the Post Office’s own evidence for this trial. Mrs Van Den Bogerd stated in paragraph 110 of her witness statement that “Once Post Office was presented with evidence that the customer had received the cash and the customer's bank had recorded the withdrawal, a transaction correction was issued to bring the branch.” She makes it clear, in my judgment, that it was the work that Mrs Burke did personally that had led to the issuing of the TC. She also makes it clear that she believes that what happened occurred as a result of what Mrs Burke did:

“Following Mrs Burke's investigation, Post Office generated a transaction correction for the £150 withdrawal that had not been recovered at the time. If Mrs Burke had not done this herself, there is a process built into Horizon for flagging non-recovered transactions which would have prompted an investigation and I'm sure would have led to the same outcome.”

151.

She explains that had this not been done, more would have been required from the Post Office – namely an investigation – before the Post Office would have reached the decision to refund the £150, although she does say that she is sure the same outcome would have been reached. This is rather different to the way the case was put to Mrs Burke, which did not include the requirement for any such further investigation by the

Post Office being required. The impression sought to be given was that on the Fujitsu documents already available, a TC would have been issued anyway to Mrs Burke a few days later. I do not accept that. Something more at the Post Office end would have been plainly required, as Ms Van Den Bogerd explains. Mrs Burke was the person who had in fact provided proof of payment out, and so no investigation was undertaken.

152.

It is in my judgment likely, on the basis of all the evidence in the litigation to date, that the sum of £150 would have showed up as a shortfall at the end of the next branch trading period, as any further investigation was not likely to have been completed by the end of that period. However, if Mrs Burke had not acted as she had, and if an investigation had been done by the Post Office, if that had been resolved in her favour, and if that had led to a TC being issued, then the shortfall in the branch accounts would have been corrected in a later branch trading period, namely the period during which the TC was issued. No loss flowed to Mrs Burke in respect of this incident, as a result of her own investigation. In any event, this is not a point of difference between the parties that it is necessary to resolve in order to resolve the Horizon Issues. There was plainly a potential impact to her branch accounts in any event, and that potential impact was caused by the Horizon system.

Mr Roll

153.

Mr Richard Roll had worked at Fujitsu between 2001 and 2004. This was during the days of Legacy Horizon and he had no experience of Horizon Online. He provided two witness statements, the first of which was dated 11 July 2016. This was obviously very much in the early days of the group litigation. The second statement addressed certain factual matters in Dr Worden’s first expert report. I consider Mr Roll to be an important factual witness in this group litigation. He has no personal interest in the litigation and is not a claimant. He has never worked for the Post Office, although whilst at Fujitsu working on Legacy Horizon he obviously had a great amount of involvement in the Horizon system. However, given the expert evidence and particularly the degree of agreement between Mr Coyne and Dr Worden on the number of accepted bugs in Horizon, which even on Dr Worden’s position was 11 different ones, in addition to Mr Roll’s evidence there was ample other evidence in relation to my findings on the Horizon Issues. Mr Roll’s career has started in the Royal Air Force, which he joined in 1976 and left in 1989. Whilst in the RAF, which he had joined as an avionics engineer, he worked on mainframe computer systems and was selected for a software development team working on aircraft control and attack systems. After he left the RAF, he worked in various roles in development and support, and joined Fujitsu in January 2001. There, he worked in the Software Support Centre, or SSC, in Fujitsu at Bracknell in 3rd line support.

154.

After he left Fujitsu he changed career direction entirely and attended the University of Southampton, from where he obtained a BSc in Podiatry. He worked in the NHS until 2011 and then went into private practice, where he remains, now with his own clinic. The claimants described Mr Roll as a “whistleblower”, effectively making public allegations about how Fujitsu dealt with Legacy Horizon and the access which was available to Fujitsu to SPMs’ branch accounts. This was access of a far wider nature than admitted to by the Post Office or Fujitsu publicly.

155.

It was Mr Roll’s evidence that was substantially behind the eventual acceptance by Fujitsu witnesses of the ability to obtain remote access without SPM knowledge or permission, and the injection of messages into the counter. In particular Mr Parker had, in his first statement, described that Fujitsu lacked this “power” and had said that Mr Roll’s evidence was inaccurate and misleading. Mr Parker later corrected his 1st statement, and I deal with his explanation about this change – and what he said he meant in his 1st statement – in my summary of Mr Parker’s evidence below at [464] to [498].

156.

Mr Roll described himself as an IT specialist, a description with which I agree, and one which is demonstrably correct given the number of years he performed his role at Fujitsu in the SSC in 3rd line support. Problems or issues would only reach 3rd line support if 1st or 2nd line support did not resolve them. His IT experience includes, in my judgment, a considerable degree of expertise in software. Indeed, although I am critical of Fujitsu elsewhere in this judgment, that criticism does not extend to any suggestion that they employed personnel within SSC who were not experienced and sufficiently expert in software matters to perform that role. Had Mr Roll not been sufficiently qualified to work in 3rd line support, he would not have been there for the period that he was.

157.

The Post Office in cross-examination attacked both his recollection, and seemed also to challenge his expertise. This the Post Office is entitled to do. Given he was 3rd line support, in my judgment he obviously had a high degree of expertise. There was a debate at one stage about whether he was “elite” or “super elite”, and he put himself in the former category. The Post Office also concentrated on a narrower band, or subset, of the issues regarding which he gave evidence, focussing more on potential problems with code and Mr Roll’s experience of those, rather than the wider issues generally in respect of Horizon of which he gave evidence. He had made clear very early in his cross-examination that, so far as he was concerned, data corruption was an issue in Legacy Horizon as well as software issues which were related to the code.

158.

Mr Roll had left Fujitsu in 2004 and there will inevitably be difficulties in recollection by any witness of detailed points after such a passage of time. He was willing to indicate when he could not remember something, and would agree with points put to him by the Post Office when relevant. These were often on the basis of hypotheses which he was asked to accept, often in relation to a spreadsheet which Mr Parker had prepared showing how relatively few software issues Mr Roll was said to have worked upon. Mr Roll also, again inevitably, was not submerged in the fine details of the subject matter of the litigation, in the same way as some of the Fujitsu witnesses such as, for example, Mr Parker himself, who is the Head of Post Office Application Support and the SSC, and who has worked on the Post Office account at Fujitsu since 1997, before Horizon was even introduced. Given Mr Roll left Fujitsu in 2004, it is 15 years since he worked on Horizon.

159.

His two statements were fairly short (4 and 8 pages respectively) and given at a high level. One example can be provided from his cross-examination. He was of the view that problems referred to 3rd line support, which by definition would mean that 1st and 2nd line support had not been able to deal with them, would not include problems caused by mistakes by SPMs. Mr Parker disagreed with this and this was put to Mr Roll.

“Q: Then the next unnumbered paragraph, Mr Parker says:

"If NBSC were unable to identify the cause of a discrepancy they would often fall back on a default statement along the lines of 'this looks like a software issue' so that the SSC would investigate it. However, Mr Roll's statement that 'if an error was referred to us then it was extremely unlikely to be due to a mistake made by a postmaster' is not correct. The vast majority of discrepancies investigated by the SSC as pseudo 'software issues' were (and are) not caused by software issues."

I would like to suggest to you, Mr Roll, that that's true, isn't it?

A.

The way that I remember it, it was issues to do with the software that were causing the problems. Whether that was the programme that had been written or data corruption, that's what I remember as our problems being.

Q. You remember there being problems with data corruption?

A. Yes.

Q. I'm not asking you about data corruption, Mr Roll, I'm asking you about software issues. The claim I would like to put to you again is the last sentence of that paragraph:

"The vast majority of discrepancies investigated by the SSC as pseudo 'software issues' were (and are) not caused by software issues." Are you in a position to agree to that?

A. From my recollection I would disagree with that, but it was a long time ago. Q. I'm grateful.”

160.

This passage of cross-examination, and there are a number of other similar examples, shows the following:

1.

Mr Roll’s conclusion was a general one, with which Mr Parker disagreed. Assertion and counter-assertion are not, of themselves, of assistance to resolving the Horizon Issues. His conclusion, as with that of Mr Parker, is simply their point of view.

2.

If a problem made it to 3rd line support, then by definition it had not been capable of being resolved by 1st and 2nd line support.

3.

Fujitsu would, even at the 3rd line support level, attribute some problems which they could not understand or resolve to SPM error, and Mr Parker’s evidence is consistent with this approach by Fujitsu. This is made clear in the text of PEAKs put to Mr Parker. There is a very stark example in one of the PEAKs where the Post Office’s own auditors have specifically ruled out user error, yet Anne Chambers chose to close the PEAK with the defect cause being user error.

4.

The Post Office would try to keep the evidence in cross-examination of Mr Roll within very narrow bounds. This passage shows that Mr Roll expressly gave data corruption as one of the problems of which he had experience. Horizon Issue 1 included the expression “bugs, errors and defects”. In my judgment data corruption is included within that phrase. Mr Roll was kept very close in his cross-examination to evidence concerning code alone. Bugs, errors and defects are far wider than simply code. The Horizon Issues were not specifically restricted to software issues as narrowly re-defined by the Post Office, nor are they restricted to code.

5.

Mr Roll would sensibly concede that the events of which he gave evidence were a long time ago.

6.

As shown by the answer “the way that I remember it, it was issues to do with the software that were causing the problems”, even at this remove of time, Mr Roll had a specific recollection that there were “issues with the software” that were “causing the problems”. This is as long ago as 2001 to 2004.

161.

Some of the cross-examination was of no assistance in resolving the Horizon Issues, for example an exercise with a spreadsheet which Mr Parker had been involved in preparing, showing the percentage of calls to 3rd line support and the broad categories into which they had been placed (by Mr Parker, as it turned out, and inaccurately at that). Mr Roll was in no position to give any helpful evidence in respect of this, and the categories were in any event misleading as headline descriptive terms. A witness who had left Fujitsu’s employment some 15 years ago would not, sensibly, be in a position to agree or disagree with such detailed points or collation of data in such a detailed spreadsheet in any way that would provide the court with useful evidence, particularly when it emerged (as it later did when Mr Parker was cross-examined) how misleading the Fujitsu headings were.

162.

Mr Roll had looked at Mr Parker’s statements, although he said only briefly, and he did not know how many statements Mr Parker had made. He could not remember some of the details, such as the detailed descriptions of 1st and 2nd line support at SSC, and was frank about this. He could not recall the split between 1st line support by Fujitsu and the Post Office’s own 1st line help desk called the National Business Support Centre (or NBSC). He acknowledged that he worked in 3rd line support, rather than “third/fourth line support” as he had put it in his witness statement. He did not however over-state his abilities; for example he said some members of 3rd line support were “super elite”, but he would not count himself as falling within that description. Part of his cross-examination which I consider fairly encompasses his approach to giving evidence is the following, where parts of Mr Parker’s witness statements were being put to him.

“Q. And if we can pick it up at paragraph 28 of Mr Parker's witness statement, you worked in third line for between -- well:

"Between 1 January 2001 and 31 December 2014 ...” and those are the years that you worked for Fujitsu "... the SSC received a total of 27,005 calls, meaning that on average 563 calls per month were dealt with over this 4-year period."

And he refers to a spreadsheet setting that out and he then analyses the data in that spreadsheet. Would you accept that that is a fair reflection of the amount of calls coming in, the amount of incidents coming in to SSC when you were there, third line?

A.

I can't really remember. I know there were periods when it was very busy and periods when it wasn't so busy. Sometimes we had three or four jobs on the go at once, other times we were given other work to do from the manager. Q. And paragraph 29, Mr Parker says:

"Transferred calls (ie those not resolved by the SSC) are of interest." He says:

"A very small proportion of calls transferred to 4th line support would have concerned software errors requiring resolution ..." Stopping there, Mr Roll, that's true, isn't it? A. Yes.

Q. So he then says:

"... it would be interesting [therefore] to know the number of calls transferred to fourth line."

Would you agree to that? It would give some indication of the extent to which incidents coming into the SSC properly, genuinely represented software areas that required fixing? That would be a useful way of -- a touchstone of trying to work out -?

A. Yes.

Q. Thank you. He then says that, unfortunately:

"... while the SSC have records of the volume of transferred calls, we do not retain records of where they are transferred to and it is not the case that all of these would have been transferred to 4th line support. For example incidents would often arrive at SSC from internal teams for routing back to help desks." Do you remember that?

A. I don't remember that particular ...

Q. But would it be right to say that of the calls coming into third line support, a significant proportion would go -- of calls that would then be transferred out, would go to places other than fourth line support? Would that be fair? Does that accord with your recollection?

A. The way I recollect it is that calls would come in and we would work on them, either fix them, in which case they would go back to the originator, or we would pass them on.

Q. To other people?

A. To other people.

Q. Depending on the nature of the problem?

A. Yes.

Q. So if there was an infrastructure problem you would probably pass the call on to the infrastructure team, it wouldn't go to fourth line support, would it? A. Probably.

Q. And there are a number of other teams that calls coming into third line would be passed to who would not be fourth line support?

A. Probably. I can't remember the full details.”

163.

This can be seen all to be at a very high level. Mr Roll is not likely to have known the total number of calls that came into SSC during a 4 year (or indeed any) period; the average number of calls per month; or of how these were dealt with. He would also not be likely to be able to recall these details, even had he known them at the time. The analysis which was put to him would have been “useful to know” – the percentage, or even total number, of calls transferred from 3rd line support to 4th line support – would not have been something he would have known at the time either. Interestingly, not only would such information have been something that Fujitsu could have readily recorded at the time, as both 3rd and 4th line support at SSC are both Fujitsu functions, and not only was this plainly accepted as something that would be of assistance, but Fujitsu “did not retain records of where they are transferred to”. Thus the absence of a useful record in this respect is a result of how Fujitsu retained its records.

164.

Mr Roll was then taken to an analysis of calls coming in, and calls going out, which had been done by Mr Parker. This related to some 3,764 calls. Mr Roll was never going to be in a position to dispute that figure; to dispute or agree the percentages given for how the calls were dealt with; or provide any meaningful evidence about Mr Parker’s analysis at all. It was presented to him for the first time in the witness box. Sometimes unreasonable or unhelpful witnesses take issue with matters of which they have no knowledge; Mr Roll did not do so. If he did not know, or was not in a position to answer substantively, he would simply say so.

165.

Another passage of his evidence was as follows, when a passage of Mr Parker’s evidence was put to him that stated "From the SSC, only a tiny proportion of incidents were escalated to the 4th line support team. It follows that only a tiny fraction of incidents raised actually needed to be looked at by the only team who might potentially effect changes in software." He was asked if he accepted the conclusion:

“A. When you take it as an average then yes I suppose so, but the system evolved from -- I started there in 2000 I think it was and left in August 2004. The nature of the work changed over time and the ones that stick in my mind are the ones where there was sort of the fire-fighting efforts where it was -- there were difficult periods where there were software issues, so ...

Q. Yes, Mr Roll. We could by all means go back to the graphs because you will have seen from the live PEAKs into and out of the SSC it actually gave monthly figures, but what I'm suggesting to you, Mr Roll, is that -- although you may well have been busy on all sorts of things, the fact of the matter is that software problems requiring a software fix represented a tiny fraction of the work that was handled by the SSC third line support.

A.

Yes.”

166.

This theme of the way that the Post Office put its case carried on into the evidence of Dr Worden and his Section 8 statistical exercise, which I address below. The expressions “tiny proportion”, “tiny fraction” and other terms used sought to demonstrate, basically, that Horizon worked very well most, if not all, of the time. Given the use to which the Horizon system was put by the Post Office, namely monthly balances which governed the branch accounts of SPMs, with shortfalls and discrepancies being “made good” – which means paid – by the SPMs in question, this approach can be seen to have limited utility. Firstly, these are subjective terms. Secondly, given the PEAKs show that a SPM may report a single unexplained discrepancy for (say) £25,000, or even £1,000 or £500, the fact that this may be only a tiny fraction of the number of calls the SSC worked on that month might come as cold comfort to that specific SPM. The number of calls, and how they were dealt with, is of some relevance to robustness, but such high level and subjective points were not likely to be of enormous assistance, one way or the other, in challenging Mr Roll’s evidence.

167.

It was put to Mr Roll that his role, which he described as “product specialist”, was “the junior level of people working” in SSC, which he disputed. He explained that most people in SSC were “product specialists”. 25 of the 30 or so total number of people were, according to Mr Roll, all product specialists.

“Q. -- the SSC? Of that 30 people how many people were at your level?

A.

When I worked there I -- it was two or three people were senior levels, I think Mr Parker was one, a couple of others, then there was Mike Peach and I believe the rest of us -- 25 or so.

Q. The figures I would like to ask suggest to you, Mr Roll, are about 25 people had the junior level and there were about five people who were true specialists who I think you fairly described as specialist earlier on in your evidence. Would you accept that that might be the case?

A. Junior level seems to -- it doesn't fully explain the complexity of the system or the knowledge of the system required, but yes, I suppose ...

Q. Well, let's agree on ordinary -- I'm not seeking to cast any imputations and it is right that you should – words do have implications. So the ordinary level was your level – A. Base level.

Q. -- and there was this perhaps five or so people above -- the senior people above you and what did they do? Did they do the more challenging work?”

An interesting post script to this line of questioning is that later in the trial it emerged that all of the SSC personnel had the very powerful APPSUP privileges at this time. It is highly unlikely that this was given to those at the “junior level”.

168.

The Post Office attempted at the trial to dilute Mr Roll’s experience and cast him as someone who was not sufficiently skilful to understand, or give evidence about, the matters he explained in his witness statement. It is clear to me, and this was effectively confirmed by the documents put to Mr Parker later in the trial, that everyone at 3rd level support in SSC was a specialist. The notion that these 30 people, in a department separate from the normal Fujitsu work areas, to which access was security restricted, were all at “the junior level” or “ordinary” with 5 people managing them who were “the true specialists” is simply misleading. There were a handful of people at the senior level, and the rest of 3rd line support were product specialists. A Fujitsu internal document, put to Mr Parker by the claimants (but not to Mr Roll by the Post Office) explained it rather more comprehensively. This is from a 2011 document but there was no evidence to suggest that there was a wholesale reorganisation of SSC 3rd and 4th line support between Mr Roll’s time there and this document.

3rd line support

3rd line support groups within RMGA include:

SSC – 3rd line support for RGMA written application code.

MSS – 3rd line support for software distribution and event management

3rd line support staff apply analytical skills to the symptoms and evidence gathered by 1st and 2nd line and undertake in-depth investigation into incidents. They have detailed knowledge of the system based on documentation and source code inspection.

Trained on operating systems, COTS packages that underlie the application and the coding languages used within the application. They are also expected to self train by examination of support guides, design documentation written for the components of the end user application. They will also have access to development and package management tools to allow the production of specialised diagnostic code, scripts and support tools.

It is incumbent upon the 3rd line support unit to produce a work around and on 4th line to produce the final code solution to any software problem. This does not preclude the production of a workaround by other units or negate the requirement for 4th line to provide assistance in the generation of a workaround.

The SSC are responsible for the implementation of any workarounds that require data changes to the live system. They are the only unit with authorisation and sufficient physical security controls to perform this function.”

(emphasis added)

169.

This gives a more comprehensive picture of the type of expertise required to be in 3rd line support, which is where Mr Roll worked whilst he was at Fujitsu. This is of more assistance than the terms deployed, such as “the five true specialists” (for the senior managers) and “the junior level” for Mr Roll. It shows that 3rd level support have a high level of necessary expertise. They are the only unit with the necessary authorisation and security controls to be permitted to implement workarounds that require data changes to the live system. This is not the role of junior personnel.

170.

This area of questioning also focused on code, and introduced another concept, the “core specialist team”.

“Q. I'm really trying to grope towards what the five true specialists, what kind of work they were doing. They were doing the more challenging kind of work, weren't they?

A. I find it difficult to answer that from what I remember of the way we worked. There were some areas where some of the senior people where I would perhaps have been more -- had more experience because of previous work and previous programming that I would have been better off and had more knowledge than they would have done, but in other areas then they would be far superior to me.

Q. Well, let me suggest one area where they would have been far superior to you, Mr Roll, or at least it was perceived within the organisation -- I'm not trying to have a debate with you about your own perceptions of yourself, that would be completely unfair, but the perception at the SSC was that were there software errors or potential software errors that required large amounts of code to be examined, the people who would generally do that examination would be those five people, people like Mr Parker. Would you accept that?

A. Generally I suppose, yes, although –

Q. And it would be relatively rare for someone outside that core specialist team to be doing that kind of work -- not impossible but relatively rare, yes?

A. Several of us looked at the code on occasions. I was a C programmer in that and other languages before I moved to Fujitsu so there were areas of knowledge we had from previous areas, but I suppose most of it would then be passed on to those people.

Q. I see. So would you accept from me then that generally it would be them that would look at lots of code but occasionally there might be occasions when someone else would look at lots of code?

A. From a PinICL perspective yes, but some of us looked at code more often, just out of interest.”

171.

This series of questions introduced another subjective variable, “large amounts of code”. Mr Roll, in his answer above, explained that he had been a C programmer before he moved to Fujitsu. C is a general purpose programming language which is widely used. Mr Roll, as someone who had been a programmer both in C and other languages before he went to Fujitsu, had both written code himself (as a C programmer), and I find would have been very familiar with code in its different guises. Indeed, given the Fujitsu documentation describing the expertise of 3rd line support in [168] above, which stated 3rd line support would have “detailed knowledge of the system based on documentation and source code inspection”, it is clear to me, and I find, that Mr Roll was amply experienced to give evidence to this court on the matters that he did concerning the Horizon system (which was at that point Legacy Horizon), the problems that were experienced with it, and what occurred at SSC. Source code is a term used to describe a version of software, or a description, as it is originally written. There are different definitions of source code in different computer authorities (such as the Linux Information Project) but the precise definition is not important for the purposes of the Horizon Issues trial. The Horizon Issues do not concentrate solely on problems either with code, or source code. The phrase is “bugs, errors and defects” and the Horizon System is defined in the Horizon Issues.

172.

What is important, in my judgment, is that Mr Roll used to programme in C programme language and other languages before he went to Fujitsu; he was required to understand and have “detailed knowledge of the system based on documentation and source code inspection” by virtue of being a member of 3rd line support at SSC; all the evidence demonstrates that he did his job effectively and competently; and he did not leave Fujitsu under any sort of a cloud. It was effectively accepted by the Post Office that he had a responsible position and a responsible role, and was well trained. The Post Office also sought to draw the following conclusion from Mr Roll’s evidence in favour of Fujitsu: namely that “….a clear picture emerged of Fujitsu as an organisation which was thorough, professional and conscientious and which took considerable care to ensure that matters were properly investigated and dealt with.” I do not accept that this picture emerged from Mr Roll’s evidence, but there was a considerable amount of other evidence in respect of Fujitsu in the trial other than that of Mr Roll. I will return to this claim by the Post Office at the end of my review of all the evidence, factual and expert, which is in Part L “Overall Conclusions”. That section of the judgment begins at [925] below.

173.

Mr Roll’s evidence was that errors made by SPMs were relatively easy to pick up at 1st and 2nd line support level; that most errors he dealt with were coding errors or data corruption; that issues were identified that required software “fixes” to be written by Development; and – importantly in my judgment - that the type of issues that were routinely encountered at SSC could and did cause financial discrepancies to branch accounts. He also stated that:

“If we were unable to find the cause of the discrepancy then this was reported up the chain and it was assumed that the postmaster was to blame.”

174.

He said that even if software fixes were developed the problem would sometimes reappear several weeks later. He also stated that remote access to branch systems was possible; the ability was extensive; that this was done without the SPMs being aware; that data and transaction information could be changed by Fujitsu; and that sometimes SSC would log into a branch system whilst it was switched on but not in use.

175.

The final paragraph of his 1st statement said:

“In summary, the issues with coding in the Horizon system were extensive. Furthermore, the coding issues impacted on transaction data and caused financial discrepancies on the Horizon system at Branch level. It was those issues that I, and other colleagues at Fujitsu, were routinely working on daily. Furthermore, remote access to the Horizon system at Branch level was extensive, as was the ability to change data and change transaction information, even while the postmaster was working, without the postmaster being aware of this.”

176.

He had also said that “during the course of resolving the software issues, we would frequently access a Post Office counter IT system remotely”.

177.

His use of “frequently” and “routine” are, in my judgment, subjective, and as explained above in terms of “tiny fraction”, subjective terms are not entirely helpful. What he meant by this is it was not unusual for this to occur. It is difficult to judge, at the remove of 15 years from when Mr Roll left Fujitsu, just how often something that he remembers as frequently or routine in fact occurred. He sensibly accepted that the more regular or mundane matters in which one is involved tend not to be remembered, with rather more unusual events sticking in the memory. In terms therefore of how often this occurred whilst he was there, it is not possible to come to a concluded view. However, I consider it important that the occasions to which Mr Roll refers were not isolated incidents on his evidence, nor were they unusual, and that financial discrepancies on the Horizon system at branch level was something which he recalls working on daily. It is also important that he gave evidence that if Fujitsu could not track down the cause then it was assumed the SPM was responsible, in other words user error would be used as a default setting for investigations. This matches the evidence of an enormous number of PEAKs in the Technical Appendix.

178.

He also gave evidence about hardware failures, and gave specific responsive evidence in respect of Dr Worden’s expert report.

179.

I accept Mr Roll’s evidence, which is supported by the contemporaneous documents. The Post Office in its closing submissions accepted that he was a careful and precise witness, and also that “he was at pains to assist the court and to give accurate answers”. It also, however, submitted that his evidence “was, unsurprisingly, hazy in many respects”.

180.

Notwithstanding the limitations on his evidence due to the passage of time, I found Mr Roll to be a reliable and helpful witness. I do not consider it was hazy in any important respects. I also found his evidence to be very important. The Post Office set out in cross-examination to demonstrate that he did not have the expertise that he claimed to have; that he was at too low a level in 3rd line support to have been involved in the matters which he described, certainly so far as software issues were concerned; and that Mr Parker’s evidence in particular about what he had been doing should be preferred. On those two former points the Post Office failed, in my judgment. The success or failure of the latter point also, but not exclusively, depended upon the evidence of Mr Parker. After Mr Parker was cross-examined, it was clear to me that Mr Parker’s exercises that were put to Mr Roll could not be relied upon to demonstrate what they sought to demonstrate. I prefer and accept the evidence of Mr Roll to that of Mr Parker and the other Fujitsu witnesses (with the exception of Mr Godeseth) by some considerable margin. I find that during the years when Mr Roll worked at Fujitsu, 2001 to December 2004, SSC were not infrequently involved in attempting to remedy unexplained shortfalls and discrepancies in branch accounts reported by SPMs. If they were unable to find the cause of the discrepancy then the assumption would be made that it must be the SPM to blame. This is clearly shown in my analysis in the Technical Appendix particularly on the different entries in the Bug Table, and my findings on the number of bugs present in the system.

181.

There is one area of Mr Roll’s evidence, again a wholly subjective one, to which I do not attribute any weight, not that I disbelieve him. This was concerning the pressure of work within the SSC and the fact that members of the SSC were under pressure to “close calls”, which basically means record them as having been completed. The relevant passages are as follows:

“15. During my time at Fujitsu I know that there were budget pressures and redundancies which impacted system development and testing. The test team felt they were under enormous pressure to complete the testing within certain timescales which negatively affected the test regime. Meanwhile, the development team had to balance time spent on fixes, with time spent on developing new features for Legacy Horizon and time spent developing a new system which I believe later became Horizon Online.

16. In my first statement, I refer to the pressure that the SSC team and Fujitsu were under generally due to an awareness of the financial penalties imposed by the service level agreements between Post Office and Fujitsu (paragraph 12 of my first statement). I believe that although individual penalties were quite modest, when applied across multiple counters/post offices the cumulative figures involved were very high, potentially amounting to tens of millions or more. I disagree with Stephen Parker’s statement that these potential financial penalties were not a factor for the SSC (paragraph 43 of Stephen Parker’s witness statement) as we were aware of them and often commented on them, e.g. “That’s saved Fujitsu another £25 million”.”

182.

I do not attach much weight to this as pressure in the workplace is such a subjective matter. Some members of SSC may have felt under great pressure; others less so. This is a subjective account of what it was to work at SSC at that time from the point of view of Mr Roll. The type of statement to which Mr Roll refers in terms of millions of pounds saved does not advance matters one way or the other. Given the type of problems that 3rd level support was involved in attempting to resolve, there would have been an inevitable pressure involved in attempting to resolve such issues speedily. Mr Roll chose to leave Fujitsu, and therefore he left those pressures behind. Putting it at its mildest, it would be extraordinarily disappointing if PEAKs were closed, attributing fault to a SPM, simply because that was the easiest and quickest way for a SSC product specialist to keep on top of their workload. Without separate cross-examination of each product specialist who chose to close PEAKs where prior entries suggested user error had been ruled out, it is not possible to determine the degree to which this was their motivation in any individual case. However, it is not relevant. It is no part of the Horizon Issues to determine why, or whether, individual personnel at SSC acted in a particular way – budgetary concerns for their employer; exercises of judgement; even laziness; or any of the other many possibilities. Mr Parker gave evidence that “the possibility of financial penalties was never a factor for the SSC”, and Mr Roll said that it was. Mr Parker’s evidence on this suffers from a wholesale attribution of this lack of motive to every member of SSC, as indeed does Mr Roll’s, in the other direction. Everyone is different, and there will have been a

range of different reasons operating on each member of SSC each day. Any sensible business will, in any event, always have at least part of its attention on financial performance and this is understandable. The Horizon Issues trial is not an inquiry into how Fujitsu manages its personnel, or its business. I do question the allocation of Category A, B or C to some of the PEAKs that were used in the trial, but this is something that was pursued (though not very far) with Mr Parker and I deal with it there.

183.

The way that Dr Worden was simply not prepared to accept Mr Roll’s evidence at face value, and set out to disprove his factual evidence, something which in my judgment an independent expert should never do, is dealt with in the section of this judgment where I deal with Dr Worden’s evidence. The degree to which the evidence of Mr Roll, and his experience of Legacy Horizon between 2001 and 2004, affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.

184.

The fact that I find Mr Roll reliable does not mean that I automatically accept all his conclusions. I have given a specific example at [181] about financial pressures. However, Mr Roll’s evidence was supportive of the claimants’ case and it was necessary for the Post Office to render his evidence unreliable, insofar as they could, in order to damage the claimants’ case and bolster their own. In my judgment, this the Post Office failed to do. Mr Roll’s evidence was also what directly led to the true picture emerging of remote access. However, of itself this does not provide an entire answer to the Horizon Issues.

Mr Henderson

185.

As explained in [78] and [79] above, Mr Henderson is a director of a company called Second Sight which was involved in a mediation scheme, until the Post Office withdrew from it. Initially Second Sight was engaged to perform an investigation, but after its interim report its scope of involvement changed. It ran from 2012 to 2015, when Second Sight’s appointment was terminated. The investigation and subsequent mediation scheme were set up in the light of disquiet about the complaints of various SPMs, the operation of Horizon, and concerns that were publicly discussed, not least by Members of Parliament. All of the documents and emails which Second Sight had at that time were returned to the Post Office. Mr Henderson estimates these as 16,500 emails and 18,500 other documents, which he said were sensibly organised in 1,700 folders. They were in good order.

186.

The documents in that set that were provided by the Post Office to the claimants for this litigation were not, according to Mr Henderson, provided in the same way. He criticised them for having no folder structure, having had metadata removed, and lacking the original date and time stamps which were present on the original documents with which Second Sight had been provided by the Post Office. Some important documents, such as the Fujitsu XML transaction reports, were described by the Post Office as “unreadable”. He had however reviewed some, but not all, of the XML data when inspection was provided of some of the unreadable documents. He did not believe that the documents provided in the Horizon litigation were the complete set of data provided to Second Sight, which he had returned. I accept his evidence on this. He was clear in his recollection about how the documents were organised, and approximately how many there were and their type. I do not know why the claimants were not given, in disclosure, the same documents, prepared and collated in the same way, as the Post Office themselves had received them from Second Sight.

187.

He described Mr Jenkins, whom he had met, as the Fujitsu lead engineer on the Post Office contract. He had met him in September 2012. He explained in his witness statement that he had been given sample XML data by Mr Jenkins, and had used this to see a level of detail that neither SPMs nor, interestingly, the Post Office, had available to them to view. He had also been told at the same meeting that remote access was “occasionally used” by Fujitsu. He asked for complete email records for 2008 for the 7 employees known to be working at Bracknell at the time, so he could satisfy himself as to whether such remote access was done without the SPM’s permission or knowledge. He was not given these, and the partial records he was given were inconclusive. His reverse engineering of the XML data he was given gave him grounds for concern but Second Sight’s engagement was terminated before he could reach any concluded view.

188.

He had also identified what he considered to be particular difficulties that Horizon had with foreign currency – it is (or was) a single currency system and only used pounds sterling – and also the National Lottery. In view of the evidence of Ms Van Den Bogerd in the Common Issues trial and the Ping fix, that latter point may not be particularly controversial in any event as this litigation progresses. Foreign exchange is the subject matter of bugs 14 and 23 in the bug table. Bug 14 is now accepted by the Post Office in Appendix 2 of its Closing Submissions as one of a number of “bugs with lasting impact (although they were resolved)”. I find in the Technical Appendix that Bug 23 is a bug. The contents of the Technical Appendix are therefore consistent with Mr Henderson’s views on this.

189.

A working group was involved in the Second Sight scheme, and the progress of cases all the way through the mediation scheme was slower than the working group wanted. Second Sight’s appointment was terminated before Mr Henderson could reach finalised conclusions on most of the issues that he said had been uncovered. I have already stated the approach that I adopt in relation to the technical nature of the Horizon Issues, and how the evidence of the experts in IT who gave evidence before me effectively takes precedence over observations by other witnesses on technical issues, which must include Mr Henderson.

190.

Mr Henderson explained in his oral evidence that he considered his evidence to the court not to be entirely unrestricted. He considered it was subject to a restriction that had been imposed upon him. There was a letter of engagement in this respect, which

was dated 1 July 2014 and accompanied by a document which was signed by someone called Chris Aujard for the Post Office. The letter of engagement set out several pages of terms governing Second Sight’s engagement. Clauses 2.1 onwards set out what Second Sight were to do:

“2.1 The Scheme has been set up to try to achieve the mutual and final resolution of a Subpostmaster's concerns about Horizon and any associated issues.

2.2.

Second Sight Support Services Limited ("Second Sight" or "you") has agreed to be a member of the "Working Group" whose role it is to oversee the Scheme and to assist in investigating individual Subpostmaster complaints.

2.3.

This letter and its schedules will form the basis of the terms of Second Sight's engagement by Post Office Limited ("Post Office") to provide Services to the Working Group in relation to the Scheme.”

191.

Clause 6.2 stated:

“6.2 Second Sight will not, and will ensure that the SS Directors and any SS Personnel will not, act, directly or indirectly, in any capacity (whether for any former or current Subpostmaster or a competitor of Post Office or otherwise) against Post Office or any of its officers, directors or employees save to the extent a) that it is expressly agreed in writing by Post Office that the work proposed to be undertaken will not have a material adverse effect on Post Office's commercial or financial interests or reputation, or b) as required by applicable law or by the mandatory rules or requirements of any regulatory authority, government department or agency to which Second Sight is subject or c) as required by an order of a court of competent jurisdiction.”

Clause 6.3 imposed this for a period of 15 months.

192.

The shorter one page document which Mr Henderson and his co-director were asked to sign and return contained different terms. The letter of engagement set out the terms upon which Second Sight would act in the mediation scheme. The shorter document stated the following:

“I will not act, directly or indirectly, in any capacity (whether for any former or current Subpostmaster or a competitor of Post Office or otherwise) against Post Office or any of its officers, directors or employees save to the extent a) that it is expressly agreed, in writing by Post Office that the work proposed to be undertaken will not have a material adverse effect on Post Office's commercial or financial interests or reputation, or b) as required by applicable law or by the mandatory rules or requirements of any regulatory authority, government department or agency to which Second Sight is subject or c) as required by an order of a court of competent jurisdiction.”

The period of that was stated to be 15 months, corrected by hand from the period initially typed in of 12 months. This obviously therefore, as signed, represented if not the same then almost identical terms as those contained in paragraph 6.2 of the letter of engagement.

193.

It is a curiosity that the detailed terms in the letter of engagement were produced by the Post Office – as Mr Henderson explained – about half way through the process. It is also the case that the services to be provided by Second Sight were to be provided to the Working Group, which also included members of the Justice for Subpostmasters Alliance or JFSA, although the services were to be paid for by the Post Office. I have outlined these terms above to make it clear, on a transparent basis, what restrictions were imposed upon Second Sight. The Post Office’s wish to avoid any “material adverse effect on Post Office's commercial or financial interests or reputation” is expressed in very wide terms. Its concern about its reputation is similar to a provision included in its funding agreement with the National Federation of Subpostmasters which is referred to in Judgment (No.3).

194.

The restriction to which Mr Henderson referred in his oral evidence, which he considered limited his evidence, was an agreement between the Post Office and the claimants about what Mr Henderson could state in evidence in this litigation, which he referred to as the “protocol agreement”. The Post Office was therefore in the position that it could, to a certain extent, control the scope and extent of Mr Henderson’s evidence. He also thought that there was a restriction on the length of the witness statement he could provide. For clarity, there was no such restriction imposed by the court.

195.

The Post Office, in its closing submissions, submitted the following:

“Mr Henderson was asked whether confidentiality restrictions had caused him any inhibition in answering the questions put to him in cross-examination. He said that he had the issue in the back of his mind and that he had tried to make sure that his answers did not infringe the protocol. It is understandable that Mr Henderson would wish to be careful, but the idea that he was restricted in answering Post Office’s questions is bizarre.”

196.

The submissions then went on to deal with the questions put to Mr Henderson, and how there was no restriction that could have impacted upon his answers.

197.

The actual question posed to Mr Henderson was not as set out in the above extract of the submissions at [195] above, it was rather wider. It was posed by me and it was at the end of his cross-examination. It was in the following terms:

“I just want to be clear: is it your evidence therefore that because of that protocol agreement your evidence of fact to this court is narrower in scope than it would be absent the protocol agreement?”

198.

His answer was “yes, it is”. He was then asked by counsel for the Post Office whether it had inhibited him in answering questions, and he said it did not. I do not consider that the closing submissions are an accurate summary of both questions put to him about this, or the restrictions he considers had an impact upon his evidence. The restriction was wider than impacting upon the questions he was asked, and his answer showed that his evidence as a whole had been affected. In any event, those questions were posed to him by the Post Office, as he was called as a witness for the claimants.

199.

The mediation scheme was brought to an end by way of notice given by the Post Office to Second Sight on 10 March 2015. The cross-examination on this was as follows:

“Q:…. If we could just look at it very briefly, the termination letter to which you have just referred is at {F/13/24.1}. That's the termination letter I think to which you at least indirectly referred. A. That's correct.

Q. If we could look then to {F/1324.2}. This is a much longer letter from Post Office that came to you on the same day. Do you recall this longer letter? If I maybe tell you broadly what it is to do with. It's a letter in which Post Office sets out a plan for how Second Sight could finish its outstanding work. Do you remember that letter? A. Yes, I do.

Q. And the second paragraph makes clear that Second Sight was expected to continue working during the notice period and that even beyond the notice period there would be a proposed future role for Second Sight. That's what this document dealt with, isn't it? It's a fairly long document.

A. It is, but there's another document of that date or very close to that date which you haven't mentioned which was the press release from Post Office announcing the winding up with immediate effect of the mediation scheme itself. That I understand was the primary decision and our termination was a consequence of that, not a separate issue.

Q. Well, I think when you say mediation scheme it's fair to say, isn't it, that this winding up process, as you describe it, did not put an end to mediations; in fact –

A. It put an end to the mediation working group with, as I understand it, no consultation and the announcement was made I think the day immediately before the next planned meeting of the working group, so it was a considerable sort of shock to everybody.”

The following points can be made in respect of the evidence from Mr Henderson:

1.

Second Sight was engaged to act in a mediation process. Mediation is consensual dispute resolution, and such processes require a high degree of confidentiality within them if they are to be effective.

2.

The termination of that process unilaterally by the Post Office, with or without consultation, is, so far as this group litigation is concerned, something that may or may not in the future call for further consideration in terms of costs. It is not at this stage relevant to the answers to the Horizon Issues.

3.

Reasonable and conventional terms of confidentiality were to be expected in any terms of engagement agreed between the Post Office and Second Sight, given the scheme was a mediation scheme.

4.

Terms seeking to protect the “Post Office's commercial or financial interests or reputation”, which were included, whether justified or not in all the circumstances of the role being performed by Second Sight, are not relevant to the answers to the Horizon Issues.

5. In Farm Assist Ltd (in liquidation) v Secretary of State for the Environment,

Food and Rural Affairs [2009] EWHC 1102 (TCC) a mediator, who had been issued with a witness summons compelling her attendance to give evidence, failed in her bid to have the summons set aside. Ramsey J gave a judgment refusing to do so, having considered the position on confidentiality, without prejudice privilege and other privilege. He stated, in his conclusions at [44]:

“The court will generally uphold that confidentiality [of mediation proceedings] but where it is necessary in the interests of justice for evidence to be given of confidential matters, the Courts will order or permit that evidence to be given or produced.” The point has not, however, been argued in this case in any respect, and it may never arise.

6.

In this case the restriction has been imposed by the Post Office and agreed by the claimants. It is regrettable, in my judgment, that any witness of fact feels their evidence to be restricted by any existing agreement with a party to that litigation. Apart from anything else, it is something of a contradiction for a witness, who swears or affirms that their evidence is “the truth, the whole truth and nothing but the truth” to then add that there is such a restriction. This appears to contradict the requirement to tell the whole truth. However, the court has never been asked to become involved in resolving any dispute between the parties in this respect.

7.

I do not consider that any such restriction – the scope of which I am in any case unaware – will have had any effect upon my consideration of the correct answers to the Horizon Issues, or the answers themselves.

200.

Mr Henderson seemed to me to be a careful and honest witness. Although his involvement in the Second Sight scheme may have led to his having particular personal views about Horizon, he was measured about how he expressed these. His views about concerns over issues in particular affecting the National Lottery and foreign currency exchange are, to a certain extent, corroborated both by the Ping fix and the position of the Post Office on bug 14 in the bug table. However, whether they are corroborated or not, it is not necessary to come to a concluded view on Mr Henderson’s evidence about these, as it is the two IT experts whose evidence will be of most assistance in this respect.

201.

The degree to which the evidence of fact by the claimants affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.

E. Evidence of Fact: The Post Office

202.

The Post Office called the following witnesses. For the reasons explained further below, I consider Mrs Van Den Bogerd and Mr Godeseth to be the most important witnesses called by the Post Office.

Mrs Van Den Bogerd

203.

Mrs Van Den Bogerd gave evidence in the Common Issues trial, and indeed her witness statement for the Horizon Issues trial was served during that trial in November 2018, and shortly before she was called as a witness. She had become the Business Improvement Director of the Post Office shortly before the Common Issues trial. She is a senior director at the Post Office and is, so far, the most senior member of Post Office personnel to have given evidence in the group litigation. I had made certain adverse findings about her evidence in Judgment (No.3) on the Common Issues, which was sent out to the parties in draft form under the usual embargo shortly before the Horizon Issues trial started, and was formally handed down at the end of the first week in the that trial, just after the claimants’ evidence of fact had been completed.

204.

I have already explained that Ms Van Den Bogerd’s evidence was considered completely afresh in this trial, and that simply because I had made the findings that I had about her previously, did not mean that I would adopt the same conclusions in this trial. Her written evidence for the Horizon Issues trial was still substantially of the same tenor in relation to individual SPMs, in terms of widespread attribution of fault to SPMs as a default setting, but I did detect a change of approach in Ms Van Den Bogerd in this trial, compared to the Common Issues trial.

205.

Originally, for example, in her statement Mrs Van Den Bogerd had specifically attributed Mr Patny’s loss on MoneyGram to user error, based on what she said the data showed. This passage, in paragraph 72, was corrected by her on a sheet of corrections of a number of the Post Office’s witnesses, and in that correction (there were a number of others) she explained that Mr Patny had cancelled the transaction but had not reversed it. She therefore accepted that the transaction had been cancelled.

206.

She generally demonstrated in her cross-examination a more realistic approach to the accuracy of her evidence than she originally demonstrated in the Common Issues trial. By the time she was cross-examined she would have had the benefit of reading Judgment (No.3), and that might explain her new approach.

207.

She had also amended, in her sheet of corrections, certain statements that had been included in her Horizon Issues witness statement that were simply not sustainable on the facts. One example of this was in relation to Mrs Stubbs (a claimant and witness in the Common Issues trial, though not in the Horizon Issues trial) whom Mrs Van Den Bogerd had said “chose to settle centrally” items that were, in fact, obviously and plainly disputed by Mrs Stubbs; and in respect of which no SPM had any real “choice”. Their choice, such as it was, was either paying immediately or settling centrally, which meant not paying immediately, but seeking time to pay. Mrs Van Den Bogerd’s explanation for this was that she had not known these sums were disputed by Mrs Stubbs. Quite how that could be, given the extended saga in relation

to these sums, the involvement of Mrs Stubbs’ MP on her behalf (Sir John Redwood, a former Cabinet Minister), the Post Office’s promises both to Mrs Stubbs and her MP of an investigation (the results of which, if one was ever done, have still not emerged, so far as I know), and indeed Mrs Stubbs’ own evidence in the Common

Issues trial, is entirely unclear.

208.

She also explained that in relation to Mrs Burke “I have looked at other evidence in relation to Mrs Burke and what was very clear to me is that Mrs Burke had done absolutely nothing wrong in that situation.” This was not at all how Mrs Burke had been cross-examined. Mrs Van Den Bogerd said that the corrections she had made to her statement had been communicated to the Post Office’s solicitors before the Horizon Issues trial had started. If that is true, I do not see how counsel for the Post Office would have cross-examined on the basis of her un-corrected statement.

209.

She gave evidence about out of hours transactions and so-called phantom sales, the latter of which she explained (in her written evidence) as follows. “I am informed by Post Office’s solicitors that in the course of investigating this matter, Fujitsu have advised that ‘phantom sales’ were reported in around 2000 which appeared to be caused by hardware issues”. There is a master PEAK in relation to this from 2001, and even though Mrs Van Den Bogerd was very closely involved in the issues on Horizon, she had not known about this until some time later. Indeed, she could not remember even the approximate year when she had become aware of it. She did not even recall, in the witness box, having seen the master PEAK before.

210.

I am most surprised that Mrs Van Den Bogerd could not remember seeing this PEAK before she was shown it in cross-examination. It is a very important PEAK. It is PEAK number PC0065021, dated 17 April 2001. The reason it is important is as follows. It relates to multiple branches. It concerns phantom transactions. It identifies dissatisfaction from more than one SPM as to how phantom transactions are being investigated and resolved, or more accurately, how they are not being. It shows calls being “closed” – ie brought to an end - without the permission of the SPM, even though that should not be done. It also shows at least one SPM threatening not to comply with their contractual obligations due to lack of confidence in the system, and also threats of legal action. Further, in one branch, where items had been the subject of phantom transactions (according to the SPM) ROMEC, the Royal Mail’s own engineers, attended that branch to fit suppressors and other equipment in an effort to rectify this.

211.

The PEAK plainly records the involvement of ROMEC, the Royal Mail’s own engineering personnel, as follows. “ROMEC have been to site and state that they have actually seen the phantom transactions, so it is not just the PM's word now.” (emphasis added). The significance of this entry is obvious, and notable. Mrs Van Den Bogerd agreed that this was “independent site visit corroboration of the problem by Royal Mail’s own engineers at the branch”, and she also agreed that this was “clearly not user error any more”. I do not understand how the master PEAK containing such important information could not have been at the forefront of Mrs Van Den Bogerd’s mind. It is, in my judgment, important corroboration from those with experience of Horizon (the Royal Mail’s own engineers) who stated they had actually seen the phantom transactions.

212.

However, the conclusion reached by Fujitsu and recorded in the PEAK was as follows:

"Phantom transactions have not been proven in circumstances which preclude user error. In all cases where these have occurred a user error related cause can be attributed to the phenomenon"

The PEAK also concludes “No fault in product".

213.

This conclusion by Fujitsu is only not made out on the factual evidence, including the contemporaneous entries in the PEAK itself, but it is, in my judgment, simply and entirely unsupportable. It wholly ignores the independent support of the ROMEC engineers, who have reported that “they have actually seen the phantom transactions” and it arrives at a conclusion that, in my judgment, entirely contradicts the evidence available to Fujitsu at the time, and indeed contradicts common sense. Given the entry that “it is not just the PM’s word now”, this conclusion ignores two entirely different sources of actual evidence. One, what the SPM reported. Two, what the ROMEC engineers visiting the branch actually saw.

214.

Another PEAK Mrs Van Den Bogerd was asked about demonstrated the lack of accuracy in Fujitsu’s characterisation of the type of problems that made their way to the SSC (and which formed the basis of the Post Office’s cross-examination of Mr Roll based on the Fujitsu/Parker spreadsheet). This was PEAK PC0208335 of 14 February 2011, also headed “Phantom Stock Declaration”. This related to withdrawn stock discrepancies, and was summarised as "Branches will be forced to declare stock when they don't want to. Apparent reappearance of withdrawn stock may cause spurious discrepancies." It was recorded in the PEAK that this could affect 10 branches per week over the next few months. The PEAK showed that a SPM was told to declare the correct stock, which that SPM disagreed with, and which Mrs Van Den Bogerd agreed would mean the SPM entering an account into the Horizon system with which the SPM disagreed. Mrs Van Den Bogerd had not seen this PEAK either, before she was asked about it in the witness box. She agreed that this PEAK appeared to be a software problem, and also that it was in a sense easier for the SPM because the phantom transaction related to stock that was not even any longer in use. The SPM in question simply could not have dealt in that stock.

215.

In any event, I found this PEAK of great assistance, not only due to the content I have summarised above. A fix was developed by Fujitsu but Anne Chambers, a Fujitsu employee who had also given evidence in legal proceedings in court before (in at least one case, the Castleton case) had entered the following in the PEAK which stated that “this fix and the MSC already applied doesn't remove all old declarations”. Further, Fujitsu chose to categorise this PEAK as “Administrative Response”. That term does not begin, in my judgment, properly to describe or summarise the problems to which the body of the PEAK referred. I find that this PEAK clearly related to a software problem, regardless of the misleading way that Fujitsu chose to categorise it as “Administrative Response”. I consider that anyone reading this PEAK at the time could only sensibly conclude that this was a software problem.

216.

Mrs Van Den Bogerd was taken through a number of examples of real-world situations, recorded in a variety of contemporaneous documents, where a wide variety

of SPMs reported a very wide range of problems. In one an internal Post Office email reported that a SPM had “found sensitive issue with Horizon when the system put a phantom cheque on the cheque line in July 2013. Claims to have evidence to support his claim. Although he himself did not suffer a loss, thinks that Horizon is flawed. Did not ask to be contacted about this. Just wanted to say that he had this information and threatened to go to MP as a result.”

217.

The question was posed internally at the Post Office:

"Given the current media and in particular the BBC's attention on Horizon, do you think it is worthwhile looking into this 'alleged flaw' with Horizon that this SPMR has highlighted to pre-empt any enquiries from his MP?"

218.

The ultimate response from Andrew Winn of the Post Office was that the claim could not be investigated without further details and Fujitsu involvement, that Mr Winn did not understand the purpose of the call by the SPM, and also stated:

“My instinct is that we have enough on with people asking us to look at things.”

219.

Mrs Van Den Bogerd agreed that this was an inadequate response. She said it would have been very easy for Mr Winn to have contacted the branch and obtained further details, and he should have done so. In my judgment, the stance taken by the Post Office at the time in 2013 demonstrates the most dreadful complacency, and total lack of interest in investigating these serious issues, bordering on fearfulness of what might be found if they were properly investigated. This SPM, whose branch was known to the Post Office, should obviously have been asked for further details (if further details were required for an investigation), and the Post Office and/or Fujitsu should plainly have investigated the matter as a matter of some importance. By 2013 Horizon was an extraordinarily controversial subject; there can simply be no sensible excuse for the Post Office’s failure to try and understand this particular subject. This is particularly reprehensible given that an internal Post Office document in August 2013 showed that Mr Winn’s involvement in this was because his area of responsibility was as follows: “also responsible for resolving specific branch accounting issues.” It was his specific job to resolve specific branch accounting issues, yet he decided at the time that “we have enough on”. I agree with Mrs Van Den Bogerd that this is inadequate – that is putting it at its most favourable for the Post Office. Somewhat stronger terms are also justified.

220.

Mrs Van Den Bogerd had only learned of the ability of Fujitsu in terms of remote access, namely the insertion of transactions at the counter under Legacy Horizon “within the last year or so”, which given she is central to the Post Office’s position in the group litigation shows, in my judgment, a remarkable situation. It is not necessary, in order to resolve the Horizon Issues, to go further into why she has found this out so belatedly, or why that might be.

221.

Mrs Van Den Bogerd would accept changes to her evidence in cross-examination where these were justified. One example was in relation to Mr Latif and corrections issued in respect of the lottery. Her written statement had said:

“However, due to an error by Post Office, instead of increasing the scratch card stock, the TAs decreased the stock. To be clear, this was a data entry error by Post Office and not an issue with Horizon.”

Actually, given the process of issuing TAs for the lottery was entirely automated, and done in conjunction with the Camelot system and by Horizon, she accepted that the passage should be corrected. The cross-examination was as follows:

“Q: So would it be fair to correct that part of your statement – A. It would be actually, yes.

Q. -- to say "To be clear, this is an issue with Horizon and not a data entry error by Post Office"; is that a fair correction to make?

A.

I have made a mistake in that the way I have worded that, absolutely, yes. So, yes, it is fair.

Q. Would you agree with the formulation I have given, or would you prefer something slightly different? What would you want the court to note as your evidence?

A. So the automated -- my understanding is the transaction -- the TA is the information that comes from Camelot to us and then it is passed through into Horizon, so in that respect Horizon just conveys it, is my understanding, and the information that's come from Camelot in that respect would be incorrect.

Q. So the point is that either way, it is not a manual data entry by Post Office?

A. No, it's not.

Q. Whatever it is, it's definitely not that, you agree with that?

A. Yes, absolutely not. Agree, yes.

Q. What it could be is some problem with the information somewhere between the terminal in the branch – A. The lottery terminal, yes.

Q. The lottery terminal in the branch and the matters showing up on the face of the Horizon terminal in the branch? A. Yes, I agree that.

Q. Somewhere there?

A. Yes.

Q. So it's definitely not a user error, is it?

A. No, that's not user error.

Q. Right. And on the face of it, it at least suggests some doubt as to the robustness and integrity of the Camelot data coming through in that automated system, doesn't it?

A. I would say yes.”

222.

It can therefore be seen that her written evidence – “To be clear, this was a data entry error by Post Office and not an issue with Horizon” – could more accurately be stated, after she answered these questions, as “to be clear, this was not a data entry error by Post Office and was an issue with Horizon”. There is a world of difference between those two statements, which are, in my judgment, poles apart both in their content and effect. Indeed, the accurate statement is the exact opposite of how it had been put in her witness statement.

223.

She also accepted that whereas her written evidence relied, so far as Mr Latif’s experience was concerned, upon the fact that TAs had been “accepted by the branch,

which could have been challenged at that point”, in fact a branch had no choice but to accept TAs. They simply had to be accepted by the SPM, and could not be challenged. She also accepted that there were problems with TAs not just for the Lottery, but for Post and Go, and also Paystation, including in Crown Offices (which were not branch Post Offices but which also used Horizon).

224.

She had also said that Mr Coyne had relied upon extracts from the Rose Report which were “taken out of context and mistakenly claim[s] that the relevant reversal was issued in error by Horizon and not the” SPM. She then went on to explain why this was not correct. The Rose Report is explained in more detail both below and at [940] onwards where I consider contemporaneous documents. However, in her crossexamination, she accepted that this part of her statement was not correct. The Rose Report, in fact and in express terms, stated that the reversal was done by the system, something Mrs Van Den Bogerd accepted; she said that it was “done by the system, absolutely yes”, that it was “definitely generated by the system” and there was “no question” it was generated by the system. She stated she was not “making herself clear” in her witness statement.

225.

Mrs Van Den Bogerd was re-examined on this specific point and the question and answer are as follows:

“Q: The extracts taken from the report by Helen Rose referred to by Mr Coyne are taken out of context and mistakenly claim that the relevant reversal was issued in error by Horizon not the subpostmaster."

I just want to give you an opportunity -- Mr Green was pressing you to accept that actually what you had said in that sentence was wrong. I would like to give you an opportunity to explain to the court what you actually meant.

A.

What I meant was that the actual reversal was part of that recovery and it had actually taken place as it should have taken place, which is what I meant in that. So it wasn't a failed reversal because it actually had happened as it should have happened, but I accepted in there that the -- it wasn't obvious to the postmaster at the time that what had happened -- that he hadn't -- because it didn't show that he had actually -- it showed that he had done it and he knew he hadn't done what we referred to earlier was an explicit reversal. That's what I meant in that.”

226.

I do not accept Mrs Van Den Bogerd’s characterisation of the evidence she had originally given in her witness statement. She had clearly stated that Mr Coyne, having taken passages out of context, “mistakenly claim[ed] that the relevant reversal was issued in error by Horizon, not the Subpostmaster”. In fact she accepted that the reversal had been generated by the system, which in my judgment it plainly had. Not making oneself clear is a curious way of describing that her own statement had said the exact opposite of the factual situation. In my judgment her witness statement had, on the face of the statement, stated the exact opposite on this part of the Rose Report.

227.

In brief terms, the Rose Report dealt with a situation whereby the data being used made it appear that a reversal had been done by the SPM, when in fact it had been done by the Horizon system. Mrs Van Den Bogerd was asked about one particular feature of the Rose Report (named after its author, Helen Rose), which dealt with the fact that no separate code was used such that this could be identified. This arose as a result of an occurrence at Lepton when the SPM engaged a forensic accountant as although he paid the shortfall shown on Horizon (about £80) he was adamant that he had not done the reversal that the system showed he had done. In other words, a reversal could appear in the Credence data as though it had been done by the SPM, yet in fact it had been the system (and not the SPM) that had done the reversal. I consider this to be important, given that the Credence data was used by the Post Office in any investigations. An extract from the report was put to her in her crossexamination (because she had given the evidence above about Mr Coyne, the Rose Report) together with a quotation from an email exchange with Mr Jenkins:

"Q:

“Just one question from my part - if the reversal is system created but shows as an existing reversal, could this not be reflected with a different code ie SR (system reversed) to clear up any initial challenges. My feelings at the moment are not questioning what Horizon does as I

fully believe that it is working as it should, it is just that I don't think that some of the system based correction and adjustment transactions are clear to us on either Credence or ARQ logs."

That's what she is saying, yes? A. Yes.

Q. And that was a fair observation, wasn't it, by her?

A. That was a fair observation, yes.

Q. About the shortcomings of Credence and ARQ logs, yes?

A. Yes.

Q. And Gareth Jenkins' answer:

"I understand your concerns. It would be relatively simple to add an extra column into the existing ARQ report spreadsheet, that would make it clear whether the reversal basket was generated by recovery or not. I think this would address your concern. I'm not sure what the formal process is for changing the report layout. Penny, can you advise as to the process: is this done through a CR?"

Do you know what a CR is? A. Change request.

Q. Change request, okay. Then at the bottom:

" I do believe that the system has behaved as it should and I do not see this scenario occurring regularly and creating large losses. However, my concerns are that we cannot clearly see what has happened on the data available to us and this in itself may be misinterpreted when giving evidence and using the same data for prosecutions.

My recommendation is that a change request is submitted so that all system created reversals are clearly identifiable on both Fujitsu and Credence."

Do you know if that change request was acted on after that?

A. I don't believe it has been acted on.

Q. You don't believe it has been?

A. I don't believe so.”

(emphasis added)

228.

Another express passage or conclusion in the report was also put to her:

Q. "The reversal was due to recovery (Counter Mode Id= 8 118) so this was not an explicit reversal by the clerk. This scenario is fairly rare so it is certainly quite easy for the clerk to have made a mistake and either he or the customer could be in

pocket/out-of-pocket (depending on exactly what happened!)." Then this:

"The system is behaving as it should."

So there were issues, weren't there, where a problem could arise for a subpostmaster by design of the system; were you aware of that? There was a whole category of PEAK codes for faults which are agreed between Fujitsu and Post Office to just stay like that as part of the design?

A. Okay.”

These two questions were then put individually:

“Let's take it in stages. You can see there it says "The system is behaving as it should"? A. Yes.

Q. That would be problematic for a subpostmaster, wouldn't it?

A. The way Gareth [Jenkins] describes it here would be, yes, because what he says is that it would have printed the session receipt but it doesn't seem as if it did, when actually disconnection transaction receipts were actually printed in this example and a recovery receipt was printed. But that's not referred to in here.

Q. Well, let's just have a look at that. Let me just ask you my second question and then we will go on to probe that with more care. The second point is are you aware of a closure code for Fujitsu for PEAKs which refers to faults which are known in the Horizon system but agreed between Post Office and Fujitsu to stay there? A. I'm not aware of a closure code.

Q. You didn't know about that?

A. No, I don't know about that.”

229.

The following important points arise in respect of this:

1.

Mr Jenkins thought that “the system had behaved as it should”. Given Mrs Van Den Bogerd accepted that this situation could lead either to an SPM, or a customer, either being in pocket or out of pocket, I disagree that an accounting system should work in that way. This is not only an optimistic description, it is in my judgment entirely wrong.

2.

The conclusion was that the scenario might not occur regularly and create “large losses”. The size of the potential losses was plainly relevant to Fujitsu. Given how the Post Office used the Horizon system vis à vis its SPMs, sums that were not “large” to Fujitsu does not mean that smaller losses would not still be appreciable sums to individual SPMs, who would have to pay for them if there were branch shortfalls. There is also no way of discerning what Mr Jenkins meant by “regularly” in terms of the risk of re-occurrence. Both regularity, and whether potential losses were large, are subjective. The Lepton incident showed that the Post Office required the SPM initially to pay the shortfall, which he did.

3.

The Credence data was inadequate to show what had actually happened. Indeed, not only was this data inadequate, it made it look as though the SPM had done something that he or she had not done, and which the system had done.

4.

It was expressly accepted that more was required in terms of the required accuracy for “giving evidence and using the same data for prosecutions”. It was concluded that system created reversals had to be identifiable, which they were not as at the date of the Rose Report and these emails.

5.

A change required a formal Change Request, and that was neither initiated nor acted upon.

6.

Finally, and in my judgment also importantly, any risks that Fujitsu and/or the Post Office “cannot clearly see what has happened on the data available to” them and “this in itself may be misinterpreted when giving evidence and using the same data for prosecutions” is a serious matter. I do not understand how a report containing such a reference to such a serious matter could be mis-summarised by Mrs Van Den Bogerd in her witness statement dealing with Mr Coyne’s analysis of this.

230.

A large number of documents, including operational documents, were put to her, many of which it appeared she had not seen before. For example, she was shown documents in respect of MoneyGram dealing with a software release in January 2016 which showed, to use the IT speak of the document, “multiple instances of system latency” – or, to use more normal language, problems in the operation of the system. These persisted through 2016 until nearly the end of that year. The internal summary of the problem was:

"For the last several months Post Office has experienced a live operational issue with MoneyGram transactions across the branch network . In the event of a transaction timing out at the counter, a system error message is displayed to the user ... and the transaction is aborted. This leaves no record of the transaction at the counter and the transactions and funds may or may not have been committed in the MoneyGram domain. This causes significant issues for Post Office and MoneyGram and for customers."

(emphasis added)

231.

Mrs Van Den Bogerd said that she had not seen the internal document setting all this out until she gave evidence in the witness box. One of Mr Patny’s issues in his evidence was about MoneyGram.

232.

She had also been involved in a meeting on 17 May 2012 with James Arbuthnot MP, Oliver Letwin MP, and others from the Post Office, including the Chairman and the Chief Executive of the Post Office. The documents, including the briefing pack, demonstrate that the MPs were told that the Post Office wished to be “open and transparent”. This was not accurate. The MPs were told that the Horizon system had undergone an upgrade in 2010 and had the full support of the NFSP. In reality, its functionality for SPMs had not been upgraded, and the NFSP had privately expressed its concern to the Post Office about Horizon. There is nothing open and transparent about telling these MPs information to the contrary. James, now Lord, Arbuthnot had been the Minister for Defence Procurement under the Conservative government of Sir

John Major. Oliver, now Sir Oliver, Letwin was Minister of State for Government Policy under the Coalition government, which had come to power on 11 May 2010. These gentlemen had become involved in this matter on behalf of their constituents. They were entitled to expect accurate information from the Post Office. They did not receive it.

233.

A large number of different issues with both Horizon Legacy and Horizon Online were also put to Mrs Van Den Bogerd, only some of which she knew about. She has been closely involved with the defence of Horizon for some years. To be fair to Mrs Van Den Bogerd, she has also been responsible for some changes over time which have made Horizon easier to use, more reliable and (to use the wording of Horizon Issue 4), more robust. Horizon as at the date the experts were looking at it is a more robust system than it was, say, 5 or even 3 years ago. A great deal of this is likely to be due to the efforts of Mrs Van Den Bogerd.

234.

Equally, however, the contents of some of the internal Post Office documents that were put to her were, in my judgment, very damaging to the Post Office’s case on the Horizon Issues. In an “Extracts from Lessons Learned Log” document of 11 November 2015, which was heavily redacted (including the redaction of single words), one entry under “issues identified” was as follows regarding the Post Office:

" Failure to be open and honest when issues arise eg roll out of Horizon, HNGx migration issues/issues affecting few branches not seemingly publicised."

(emphasis added)

This frank expression in a document authored by Mrs Van Den Bogerd herself as recently as 2015 is consistent with the general tenor of the claimants’ case and their criticism of the Post Office. This entry in this contemporaneous internal Post Office document is, in my judgment, inconsistent with the picture which the Post Office continually seeks to portray, namely that it wishes to be open and transparent about issues with Horizon, and is as interested in getting to the bottom of any problems with Horizon as anyone else. It is an internal recognition of a “failure to be open and honest when issues arise.”

235.

There were two periods that she was asked about which clearly showed the problems experienced by Horizon almost from the outset. These were the introduction of Legacy Horizon, in late 1999 and 2000 (the national roll-out); and the migration to Horizon Online in 2010 (including the pilot scheme in 2009). Mrs Van Den Bogerd gave some evidence in her statements about both of these, in terms complimentary of Horizon, although she was far more involved in the latter than the former. At the time of the introduction of Legacy Horizon she was only responsible for a smaller number of branches in her then role.

236.

In PEAK PC0033128 an entry in November 1999 shows that the branch at Dungannon experienced a discrepancy of £43,000. The PEAK states:

“PM - Dugannon PO £43k discrepancy

Outlet has a discrepancy of £43,000 after balancing SUs and doing office snapshot.

Phil Turnock POCL BSM has advised outlet on this week’s balance. Steve Warwick development is investigating why this mis-balance occurred.

Immediate impact of this week’s balance has been addressed but POCL are concerned that the cause is still unknown and this will affect this and other outlets.”

237.

After some investigation the following entry appears:

"I have talked with development ref this problem. It is seen as a one-off. No fault can be found and developments do not expect to be able to find a fault with the evidence available. There is no additional information available as evidence. I suggest this call be placed on monitor for 1 month."

The approach to this discrepancy mirrors so much of the case. A discrepancy occurs; Fujitsu cannot find a fault; and say they do not expect to be able be find one “with the evidence available.”

By February 2000 Fujitsu stated “counter 5 looks suspect” and continued:

“Further examination of the event logs for these two counters indicate that counter 5 looks suspect (C drive nearly full and big gap of no messages).

Calls from PO into HSH for period between 30-Oct and 10-Nov indicate a reboot (counter not specified, but would tie in with counter 5 event log) on Saturday 30-Oct1999.

The evidence in the message store was that messages continued to be written to the message store but that all the 'Payment' transactions which should have been recorded in the rollover trailer messages failed to appear (although others did, such as the Rem OUT and Transfer OUT totals).

This indicates that the problem was not one of running out of Disk space but of failing either to retrieve, or write out, transaction totals for one particular node in the node hierarchy.

Given that there were known problems with corrupted Persistent Object indexes at about this time, it is possible that an update to an EPOSSNodes object failed to be registered correctly at the outlet, causing the node accumulation to fail.

It was decided to prove this out by deleting the 'Payments' node in the node Hierarchy and then running the SU balance, to attempt to identify the root cause of the problem. Call passed to testing to be scheduled.

Update 18th Feb 2000

The test was carried out on 16th February as follows: delete the Payments EPOSSNodes object before producing a SU balance, on a version of the current live system (CI2_2R).

When trying to print the Payments part of the SU balance, the missing node is detected by the system and an error tablet with message "A system error has occurred whilst printing. Please ring the helpdesk. Error at 67640." is generated. So the balance could not be finished.

This type of error trapping error trapping was introduced at the end of last year when resolving AI298 issues and we are investigating if the outlet did not have such error handling when the problem occurred.

Certainly, with the current system, a missing Payments node now would not go undetected.

The problem is currently back with development for further investigation.”

238.

On 17 March 2000 there is another entry that says there has been another incident “of a very similar nature” at another office. This can therefore be seen to be something that was not, as recorded initially, a “one-off”, even if that were acceptable given the discrepancy of £43,000. Mrs Van Den Bogerd agreed that this showed that the underlying information was not checked until February 2000, even though the discrepancy occurred in November 1999. She was asked about a later entry in April 2000 that referred to another occurrence, this time in Appleby for £9,000 approximately. The cause for this was recorded in July 2000 as “data trees failing to build” and that entry states “there have been a number of calls relating to this kind of issue. A fix has been put in at C14 which will prevent this happening”. Mrs Van Den Bogerd accepted that this was an example of an error occurring, and the system itself failing to spot that the error had occurred.

239.

She was shown PEAK PC0027887 which was dated July 1999 and related to “receipts and payments misbalance”. This showed an escalating misbalance that increased, in my judgment dramatically, in steps up to over one million pounds:

“On week 9 receipts and payments misbalance of £1337.05, week 10 misbalance of £24000, week 11 misbalance of £12000, week 12 misbalance of £1051111.48 and week 13 misbalance of £17426.05, she has a difference on week 11 of balance due to post office and balance brought fwd on week 12 of £1082544.32

Overall these weeks net out a difference of £27343.84. she needs business support (reconciliation) to look into this.”

240.

Mrs Van Den Bogerd agreed that £1.05 million was a large amount for any branch and could not be a “real amount of cash” that “the SPM has put in her pocket”. A later entry in the same PEAK of 27 July 1999 was:

“Balance brought forward was multiplied twice due the known software error. The initial balance brought forward for this CAP was £1196622.72. This was multiplied twice to give a total BBF of £2279189.04.

The discrepancy was therefore £1082540.28. This was due a known software error which has no been resolved.”

241.

She did not know if “no resolved” meant “now resolved” or “not resolved”. That PEAK was not closed until 31 August 2000, and was categorised as “Administrative Response”. One of the latest entries in this PEAK, which was not put to Mrs Van Den Bogerd but in my judgment is relevant to Fujitsu’s approach generally, in particular to categorisation of PEAKs (as this had already stated there was a “known software error”), was dated 31 August 2000 (one year after the original issue occurred) and it is most convenient to reproduce here in this judgment. This stated:

“I have now taken over analysing this problem from Steve Warwick who requested additional information on 13/10/1999 17:49:21. I have looked at the information and I am not convinced it can possibly be complete.

For example he said he wanted information from 20th May until 26th May. I opened the spreadsheet mc20may.xls and observed it only had 6 counter 32 transactions in - is this correct?

In any case my team is not used to dealing with data in this form (spreadsheets). We have developed techniques to look at problems using a full message store plus audit and event logs from the failing counter. Even if all the relevant spreadsheets could be obtained I do not think it would be worthwhile me getting to grips with this new method of analysing problems.

I see this is a very old problem (21/07/1999) and there have been many software updates since then.

May I suggest we discontinue investigation of this particular problem but that if a similar problem occurs again you send full message store plus audit and event logs from the failing counter.”

And

“Closing call on basis of insufficient evidence. As this is such an old call I have not contacted the call originator. I suggest that this call remains closed!

INSUFFICIENT EVIDENCE - ADVICE GIVEN”

(block capitals present in original)

To close such a PEAK as “Administrative Response” is, in my judgment, remarkable.

242.

The number of different incidents recorded in the monthly incident review increased in November and December 2000, which is shortly after (for example) the date when Horizon was installed in Mr Bates’ branch office at Craig-y-Don, as set out at [105] of Judgment (No.3). By then the Horizon System had been rolled out nationally. As the ICL Management Summary that was put to Mrs Van Den Bogerd states:

“The most frequently occurring incidents in November were both types of Receipts and Payments Incidents (Migration and Post Migration), with 31 incidents per category. The Migration incidents have remained at the same level, whereby the Post Migration occurrences have increased. This was followed by 17 Transactions Polled by TIP but not by HAPS, these were due to delayed transactions as reported on APSS 2133c. These transactions are added back into normal processing.”

243.

Mrs Van Den Bogerd was not aware of these problems going on at the time; she was a Regional Network Manager and did not have a national role then. No background information was provided to her then about problems that branches other than her own were experiencing, which were being passed back to ICL at the time. She therefore had no knowledge of the type of problem experienced by Mr Bates in June 2001, which was recorded in a Post Office audit document of his branch that was put to her which stated “A correct assessment of cash holdings could not be made because the Horizon system intermittently adds the previous days cash holdings to the daily declaration.”

244.

By 2010 and the introduction of Horizon Online, she did have a national role and broader responsibility. She agreed that Horizon Online did not upgrade functionality, and documents were put to her that showed, as Mr Green put it “a litany of issues” with Horizon. The pilot scheme went sufficiently badly that Fujitsu put it on “red alert” and the high volume pilot was suspended in April 2010. Mrs Van Den Bogerd

had not even mentioned this in her witness statements, and said she did not know this had occurred. The NFSP are recorded as “raised concerns but remain supportive”. All migrations were suspended as from 25 March 2010. There were database errors recorded in a Post Office document at F/614 called “Horizon Online Programme Update” which required an Oracle patch set to be supplied by Oracle, and on one occasion for a limited period 50% of MoneyGram transactions failed.

245.

A Network Functional Report of May 2010 was put to her. Over 5 pages of the 6 pages are redacted. Mrs Van Den Bogerd could not recall if she had seen this document, and said that given the extent of redactions, it would be hard for her to recognise it, an observation with which I agree. She said that she was not aware of there being issues with Horizon over Easter 2010, which the document recorded.

246.

Redactions had been incorrectly applied by the Post Office and/or its solicitors to some of the relevant documents. This can be seen from the following example. I asked leading counsel for the Post Office to review certain redactions and having done so, an unredacted version was disclosed of a document dated 25 June 2014 entitled “Branch Support Programme”. This had been co-authored by Mrs Van Den Bogerd. The purpose of the paper was to “Update the Post Office Executive Committee on the progress of the Branch Support Programme.” Each one of the 6 key performance indicators against which the programme would be tracked had initially been redacted by the Post Office, and following the review by counsel as requested by the court they were unredacted. They were as follows:

“* Reduction of operating costs by £3m per annum

*

Reduction in net agent debt by £1m

*

Reduction in subpostmaster suspensions as a result of audit shortages to a level of 60 per year

*

Reduction of calls into NBSC by 25%

*

Reduction of audit losses of £10k of over by 50%

*

Satisfaction with on-line training models of 95%”

247.

Apart from the first bullet point, three of the other five are consistent with an aim by the Post Office to make improvements to reduce the amount of debt incurred by, and suspensions, of SPMs, and reduce the audit losses. The other two are related to training. This is not consistent with a view that the debt/suspensions/audit losses are incurred by carelessness on the part of SPMs or criminal activity. It is also hard to see how it could be justified that these had been redacted originally.

248.

So far as her evidence about Mr Latif was concerned, after Mrs Van Den Bogerd had been taken through the features of his experience with the failed stock transfer and the similarities with accepted bugs in Horizon, namely the printing out of two receipts very close to one another (in his case, 4 seconds apart), and asked if this worried her, she said she “would want to go back and have a look at” it. In my judgment, part of the way through the Horizon Issues trial in 2019 is rather too late for that. I conclude

that there was no real answer available to Mrs Van Den Bogerd to the points put to her concerning Mr Latif’s experience. Had there been, I have no doubt she would have provided that answer in evidence.

249.

Mrs Van Den Bogerd was in the witness box for in excess of one day, the longest period of any of the witnesses of fact for either the claimants or the Post Office. Her cross examination led to a far greater understanding of the Horizon Issues on the part of the court, although her written evidence was, as originally drafted, extraordinarily one-sided. She minimised any reference to problems or issues with Horizon, and reverted to potential user error whenever possible as a potential explanation, an approach which she explained in her written statement as providing “plausible” explanations. Her witness statement also stated, in terms, the exact opposite of what the reality of the situation was, and I have given examples at [221], [223] and [226] above. Witness statements are supposed to be factually accurate, and care must be taken in future rounds of this group litigation that they are drafted in accordance with the rules. Making statements that are the exact opposite of the facts is never helpful, to put it at its mildest. It is also the opposite of what witness statements are supposed to be.

250.

However, during her cross-examination Mrs Van Den Bogerd readily conceded a number of points which were put to her. Her accuracy in her oral evidence was also more satisfactory than it had been during the Common Issues trial. She accepted her written evidence suggesting a causative user error by Mr Tank was not correct, for example. I do not consider that her written evidence had provided plausible explanations. It provided explanations that the Post Office wished to advance, as these explanations would, if accepted, provide a defence to the claimants’ factual case. These explanations were not based on the facts.

251.

There were no evident attempts on this occasion to mislead me in her oral evidence, or avoid uncomfortable points through a claim of ignorance (with the possible exception of the PEAK referred to at [210] above) or of not having seen a document before. When she told me that she had not seen a particular document before, I accept that. However, she also told me that she had been assisted in preparing her evidence by a team of ten people, and the Post Office had devoted “high level resources” to her evidence, as one would expect. It is therefore very surprising that she had not seen many of the important documents that Mr Green put to her in cross-examination. Some, such as the master PEAK on phantom sales, one of the recurring issues on Horizon, would have been the obvious place for anyone, still less a team of ten, to start when considering preparation of evidence for a witness statement. She explained that there was some pressure of time in terms of how long was available to her to prepare the statement, and this was further explained in supplementary reexamination, but I do not accept insufficient time as a valid explanation for her lack of knowledge on such important points. For example, she told me that at the time of preparing her witness statement, she had not even heard of the Callendar Square bug, one of two bugs that the Post Office accepted some time ago had been present in the Horizon system. This is an extraordinary gap in her knowledge. She did not know that there was a KEL dealing with failed recoveries, originally raised by Anne Chambers in Fujitsu as long ago as 2010, which is called KEL acha959T and which was updated most recently in 2017. This described failed recoveries, and seemed on its face to accept that these would recur, and was very close to the experience of both Mr Tank

and also Mrs Burke. I do not see how Mrs Van Den Bogerd (assisted by her team of ten, and with the benefit of the Post Office’s considerable resources) could seek to give accurate evidence in the Horizon Issues trial without referring to this KEL, still less without even knowing about it. I am also somewhat disappointed – putting it at its very best for the Post Office – that a team of ten could have assisted Mrs Van Den Bogerd in preparing a witness statement that was so inaccurate on such important points as I have identified above.

252.

Although my findings on her evidence in the Common Issues trial cannot be ignored, I am of the view that her approach in the Horizon Issues trial to answering questions was far more constructive and aligned to what is expected of any witness giving evidence in court, particularly a senior witness of an organisation such as the Post Office. I do however consider that this litigation, and indeed her cross-examination, is a very expensive way for a senior director at the Post Office to become educated about the myriad issues contained in the documents that were put to her. Either the team of ten people assisting her with her evidence had the aim of producing entirely one-sided evidence in chief, or they were unaware of all the documents relied upon by the claimants. Either alternative is highly regrettable.

253.

Finally, and this is a point in Mrs Van Den Bogerd’s favour, there is no doubt that the Post Office is not as sufficiently close to the detail of what has occurred over the years on Legacy Horizon and Horizon Online, as Fujitsu. As will be seen from my analysis of the Fujitsu evidence of fact, I have certain views about the lack of accuracy on the part of Fujitsu witnesses in their evidence. If that lack of accuracy has also been included in reporting to the Post Office by Fujitsu, then that goes some way to explaining the Post Office’s lack of grasp of so much material that is consistent with the claimants’ case. As at late 2019, the date of this judgment, the Post Office also has the added benefit of the views of both the IT experts in the litigation, their four joint statements, the agreed number of bugs in the bug table (12, if one takes the number accepted by Dr Worden), the total number of bugs Mr Coyne says he has discovered (which so far as Horizon Issue 1 is concerned became 21), and collectively my findings on the Horizon Issues. These are not restricted solely to the number of bugs which I have found to exist which relate to Horizon Issue 1, but also those that relate to Horizon Issue 4. There are multiple bugs, errors and defects in both Legacy Horizon and Horizon Online in its HNG-X form.

254.

The degree to which the evidence of Mrs Van Den Bogerd sits with my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.

Ms Phillips

255.

Ms Phillips is a Team Leader at the Post Office for Agent Accounting and Santander Banking. She was originally engaged by the Post Office in 1999 through an employment agency, and has worked in a number of roles since then. In her current role, she oversees the processes of recovering losses from SPMs that they have declared in their branches. The department used to have “debt” rather than “accounting” in the name, but its function remained the same.

256.

Ms Philips confirmed what had been in dispute for so long during the Common Issues trial, namely that SPMs had no option but to accept the figures provided to them, even

though they may have been “settled centrally”. This is notwithstanding that her terminology in her witness statement was that SPMs “chose to accept”. This terminology was used throughout the witness statements for the Post Office, and it has already been seen that Mrs Van Den Bogerd accepted TAs had to be accepted, and there was no choice in this respect, as Mr Latif had explained. SPMs clearly did not have a choice, and the wording “chose to accept” is not an accurate description. It obviously states or suggests that SPMs had a choice, when no such choice was available to them. The only choice they had was whether to pay the amount immediately, or settle centrally which means obtaining time to pay. It had been necessary for me to make findings on what “settling centrally” meant in the Common Issues trial, and the Lead Claimants in that trial had been directly challenged on this very point in their cross-examination.

257.

There is no reference in the letters sent to the SPMs in question relating to any ability to dispute the sums which the Post Office treated as debts. Ms Phillips’ evidence went to the way that the Post Office seek to collect sums from SPMs who have settled centrally, even though they may have earlier raised a dispute (or sought to raise a dispute) with the helpline. Sometimes amounts above £5,000 would be checked by reference to the helpline call logs, but the amount was not a set threshold and it depended upon how many sums had been settled centrally as to whether this happened. This checking did not therefore always occur, even above that figure.

258.

Her evidence demonstrated that there was no dispute function within Horizon. Essentially, her task was debt collecting, as the Post Office saw amounts “settled centrally” as debts. Her concern was to obtain payment, or agreements to pay, and if not, then to escalate the matter to a SPM’s contract adviser. Usually, she said, the first or even second letter would lead to payment by an SPM. Neither of those standard letters referred to disputes. Mr Godeseth confirmed that the lack of any dispute button or function on Horizon was, as he put it, “by design”. This shows that the Post Office had been advancing a case, at least for a substantial part of the Common Issues trial, which was directly contrary to the evidence of its own witnesses of fact in the later Horizon Issues trial. I find this difficult to understand or explain. However, Ms Phillips’ role was fairly clear and she explained the process of branch dispute forms which would be used when a SPM raised a dispute directly with her team. This elementary system seems sensible, but it was only introduced in early 2018, and it would lead to a seven day period after which (if the form was not returned) the “block” placed on the amount would be lifted. She estimates that it would be used at least 5 times a month.

259.

Ms Phillips gave useful evidence on the system used by her team, and I found her credible. The evidence did however go more to the Post Office’s accounting functions, and how it pursued sums that it considered to be due from SPMs. The reforms introduced in 2018 in terms of the branch dispute form are part of that process, but are not part of the Horizon system.

260.

She had also provided some evidence to Mr Smith (who was also called as a Post Office witness) in terms of the number of Santander TCs that were issued, which had not been described correctly in his witness statement and which had been misunderstood by both the experts. This was clarified, and also she explained that of the number of TCs issued in one year in her particular area, 2016-2017, which was

3,968, the Post Office could not say how many were disputed because that data was not kept. This would appear to be a fairly elementary piece of information for the Post Office to monitor or record. The failure to do so is surprising.

261.

Ms Phillips’ evidence in her witness statement did not directly impact upon the Horizon Issues to any appreciable degree. Her evidence went to wider issues in the group litigation. Her figures for Santander TCs, which had been given in evidence (for some reason) by Mr Smith and not by her, were considered by the experts.

Mrs Mather

262.

Mrs Tracy Mather is a Team Leader of the Finance Service Centre or FSC. She started working for the Post Office in 1987 and has been in the FSC throughout, becoming a team leader in 1999. She has managed teams in Cheques, Postal Order, Pay-out and MoneyGram. Her witness statement was served in response to Mr Coyne’s 1st Expert Report, and explained why the Post Office used Credence to investigate discrepancies. She explained the difference between Credence, which records the transactional data, and POLSAP, which is what is called a “back end” accounting system. She explained the latter is on a higher level. Credence is used by her team to investigate differences between what a branch says has happened to a transaction through Horizon, and as she put it, what a different source of information might say. Credence was adopted in 2009.

263.

She confirmed that Credence does not record key strokes, it records “transactional data as in sales and non-sales”. Mrs Van Den Bogerd had stated that Credence records actual key strokes by stating in her written evidence that Credence “records all key stroke activity performed in that branch by the user ID, date and time.” This was not correct, and was corrected by Mrs Mather in chief. Her team does not have access to the ARQ data, which has to be obtained from Fujitsu. This is also called audit data. Credence is not used for MoneyGram. I accept her evidence on this point and find that Credence does not record all key stroke activity.

264.

Doubts expressed in the Ernst & Young Management Letter for the year ended 27 March 2011 about the integrity of data in Credence were put to her, but she was unaware of any such doubts having been expressed and did not know about them. She had not been in a team that had requested audit data, and said that would be the fraud or security team who would do this. Passages from internal documents suggesting the audit data came at a high cost to the Post Office were not something of which she seemed to be aware. She started using Credence in 2016.

265.

She gave written evidence about what was called the Rose Report, although she did not actually know about it herself, and she also was unaware of anyone being deterred from making ARQ requests due to the cost to the Post Office (charged by Fujitsu), which was a point Mr Coyne had raised. Given her team did not make ARQ requests, her evidence on this was unsurprising. She was aware that a certain amount of requests were allowed, and above that amount a charge was raised by Fujitsu; however she did not know what the charge was.

266.

There was one passage of her witness statement which went plainly beyond any direct evidence she could usefully give. It was in paragraph 20 and gave vague evidence

about her understanding of some confirmation given by Fujitsu to the Post Office’s solicitors about the contractual limit on ARQ requests, when or if that had been exceeded, a “new commercial term” between Fujitsu and the Post Office about what was agreed and the cost of these. Her conclusion was “any requests above the agreed limit are chargeable but I understand that the terms depend on the details of the requirement.” She was not cross examined about this, and given her answers in cross examination about her understanding of ARQ requests (which was minimal) and her experience of having made such requests (which was never), that is understandable. However, in terms of providing guidance to the parties on the contents of witness statements, to assist the efficient conduct of these proceedings going forwards, witnesses should not have paragraphs of this nature in their witness statements about matters in which they have had no involvement at all. Mrs Mather – and there is no criticism of her in this – has simply never been involved either in making ARQ requests, or in the contractual arrangements between Fujitsu and the Post Office, or the “new commercial term” between those two entities in relation to the extra data required for the litigation. She stated that she had “no idea” about the charges that would be raised.

267.

I found Mrs Mather to be a credible and helpful witness. Her evidence was very useful regarding Credence, and otherwise of less than central relevance to the Horizon Issues. The degree to which the evidence of fact supports my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.

Mr Smith

268.

Mr Paul Smith is the Operations Support Manager at the FSC and has been since November 2016, having started at the Post Office in 1996. Originally he started in pensions and then moved to human resources and payroll, until in 1999 he moved to the Network Business Support Centre helpline, or NBSC. In 2000 he moved to work as a Problem Manager, dealing with problems that came through the helpline. He led the team that updated the knowledge base for the helpline, and was Lottery Manager in 2004. After performing different roles he became a Change Analyst in 2011, and then the Santander manager. He returned to Lottery within the Post Office in 2013 and worked there as a Team Leader. His existing role has a number of different activities including handling specific product based problems. His evidence dealt with volume of TCs and a suggestion made in Mr Coyne’s report that TCs for a branch in Potters Bar were initially made in error.

269.

One passage in his statement is of particular importance in resolving the Horizon

Issues. It states:

“TCs are issued by FSC. I understand from Post Office’s solicitors that the processes by which FSC determines whether a TC is required is outside the scope of the Horizon Issues trial.”

270.

The reason that this is important is as follows. Dr Worden relies upon TCs as one of his “countermeasures” that goes to the robustness of the Horizon system. Mr Coyne does not, and the claimants submit that the whole process of TCs is entirely outside the scope of the Horizon system. This evidence from one of the Post Office’s own witnesses makes it clear that the claimants’ point of view is shared (or at least was

when Mr Smith’s statement was prepared) by the Post Office’s solicitors, and the decisions regarding whether TCs are issued are outside the scope of the Horizon Issues. This evidence by Mr Smith supports the claimants’ position on this, and also supports the approach of the expert evidence of Mr Coyne, and is contrary to the position adopted by Dr Worden.

271.

A case management system was introduced by the Post Office in September 2018 to record each individual challenge to a TC. This is called the Dynamics System, and is still in what is called “roll out”. Individual challenges to TCs were not recorded prior to that. Further information about how this system was working was obtained in cross-examination, and leads me to the conclusion that the way that TCs are recorded now is far improved from what it was before the Dynamics System was introduced. That does not however take one very far. Mr Smith knew of reports that were available from that system but he had not produced any. Those reports would be the best evidence of the type of points Mr Smith was seeking to make for the most recent periods at least, but given he had not used them, it is not possible to speculate on what they might show.

272.

Mr Smith had obtained almost all the data contained in his statement from others, and some was plainly incorrect, and other data so vague as to be wholly unhelpful. For example, for Santander, originally he had provided a table with a column stating that 2,890 disputes were received from branches in the 2016/2017 financial year. In fact, that column should have been disputes received by Santander from the Post Office. This was corrected in his second witness statement, but Mr Smith was unnecessarily combative, in my judgment, about the need for the correction, maintaining that “the facts and figures were correct” in the first statement when they plainly were not. The figures were correct as figures but the descriptions of what the figures were, were plainly wrong.

273.

Even after the correction, the various figures extracted by him from information provided by others was not particularly useful. For example, the figures for the numbers of TCs disputed, and for compensating TCs, for BOI Retracts and the Lottery, were both estimated at 1,500 and 500, for both line entries. This estimate was plainly of the most vague type, and even Mr Smith could not comment on the reliability of either of those estimated figures for either line entry, although he maintained the figure for issued TCs would be reliable. This evidence from him, was of the most general type, and was not of great, or indeed any, assistance. For example, in respect of MoneyGram his evidence was that one or two disputes a month were received but they were rarely if ever accepted. Given he had no idea at all of the overall total, this evidence does not assist me in resolving the Horizon Issues one way or the other.

274.

His explanation for the TCs at Potters Bar was that a cheque that was deposited for £90,000 was mis-keyed, and entered for £900,000. Accordingly TCs were required to correct this at the branch, and the need for two was because the payment was made by cheque.

275.

I found Mr Smith’s evidence, with three exceptions, to be of no assistance whatsoever. The two exceptions are his evidence I have summarised in [269] which is relevant to the scope of the Horizon Issues and Dr Worden’s countermeasures, and his explanation about Potters Bar and the two TCs. I accept his explanation about the Potters Bar TCs. Both of those exceptions are relevant to the Horizon Issues. The third exception is his evidence about the Dynamics System. This goes to the robustness of Horizon as of 2019 and is therefore relevant. However, the absence of such a system also goes to robustness prior to its introduction in 2018, and is also therefore relevant. However, his wider evidence is of limited relevance at best. His knowledge of the figures contained in his statement, and the vagueness of his evidence generally, were such that the utility of the exercise which he presented in his statement is de minimis. In any event, the substance of that evidence went to an attempt to demonstrate that some TCs are challenged by branches, and fewer challenges are upheld than are made. The figures vary inevitably from year to year, and this case is not about general statistical trends in any event. In some areas of business TCs have fallen to a very low level, for example the DVLA, which Mr Smith explained was only 4 in the last year, none of which have been upheld. However, the records that were put to him show that there were 2,717 in the period 2010/2011. All this evidence does is show that far fewer TCs are now issued by the Post Office in relation to the DVLA business, which as Mr Smith explained, has fallen considerably in any event. None of these points are of particular relevance to resolving the Horizon Issues. The experts are agreed that Horizon as it is in mid-2019 is more robust than it was in the past. One would expect, therefore, fewer TCs to be issued now.

Mr Johnson

276.

Mr David Johnson is a Training and Audit Advisor at the Post Office. He started his career at the Post Office in 1984 as a counter clerk in a Crown Branch, and worked his way up until he was the Branch Manager at the Barry Crown Office. He was in that post when Horizon Online was introduced in 2010. He moved to his current role in 2012, and his role used to be called Field Support Adviser. Two thirds of his time is spent training SPMs; the other third is spent auditing them. He had also been involved in training some of the legal representatives for both sides involved in the group litigation, and the experts, on the Horizon system. As well as his first witness statement, he had provided a supplementary one dealing with some points in Mr Henderson’s witness statement which was served for the claimants.

277.

He was cross-examined about screen layout and design, the risk of mis-keying (which did not seem to concern him, although there were internal Post Office documents showing it was a known problem) and also the reports and functions available to SPMs to investigate shortfalls and discrepancies, rather than those which the Post Office itself used such as Credence. He agreed that the ways available to SPMs to investigate in branch were “not the most user-friendly way of investigating”.

278.

He agreed with a statement in a 2016 internal Post Office document entitled “Network Development Programme Operation Simplification” that stated:

"There are a number of branch operations processes, especially around branch accounting and reconciliation, which operate using legacy processes. They are unnecessarily complex and detract Post Office branch resources from serving customers. Stock Unit Management and accountability is very poorly controlled and is operated on very complex business rules. The lack of accountability and visibility of cash and stock transfers between Stock Units can lead to errors, rework and provides opportunities for fraud."

(emphasis added)

279.

He also stated that the following entry was “not completely unreasonable” although he said that his experience of suspense accounts in branches was that they worked and did what they were supposed to do:

"Similarly, Suspense Accounting is based upon legacy cash accounting practices, illdefined and out of date processes. Inefficiencies lead to poor utilisation of resources both in Post Office branches and Support Services."

280.

Mr Johnson is a highly experienced person, as one would expect given that the Post Office chose him to train the legal teams and experts on Horizon. All his experience, so far as using Horizon is concerned, is in Crown Offices, which are not the same as branch Post Offices, in that the Post Office operates Crown Offices itself. This means that a SPM in a branch Post Office is personally responsible for any losses in that branch; whereas a Crown Office manager is an employee of the Post Office. This therefore gives him a somewhat different angle of approach to the use of Horizon, but I accept him as an accurate and helpful witness.

281.

His evidence was very useful in terms of how Horizon works. The Horizon Issues are not about the actual design of the Horizon terminals in branch, and whether (say) this could be improved by having larger buttons or the screen layout adapted differently. In some of Mr Johnson’s evidence, the claimants’ counsel strayed into the areas of potential improvements to functionality in branch and design. These are not part of the Horizon Issues. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.

Mr Dunks

282.

Mr Andy Dunks is at IT Security Analyst at Fujitsu Services Ltd, and was the first Fujitsu witness to be called for the Post Office. He has been employed at Fujitsu on the Post Office account since 2002. He has working knowledge of the Horizon System. His evidence went to audit data extraction and the integrity of the data during that process. His evidence related to the process as it was when he made his statement, although given he had been involved for some 17 years his experience also related to Legacy Horizon and his evidence about data extraction covered that period too. As emerged in his cross-examination, he had also given a witness statement in the criminal trial of Mrs Seema Misra.

283.

He agreed with the definition of data integrity put to him, which is that data integrity is the overall completeness, accuracy and consistency of that data which you can measure by comparing between sources. He said the 12 controls that he had listed in his statement explaining these he had compiled as a list from his recollection, and that he had not worked off a document to come up with the list in his statement.

284.

Mr Dunks gave every indication, in the first part of his cross-examination, of being helpful and frank, and explained the process of data extraction and answered questions put to him openly. However, this encouraging start came to an end when he was asked about two curiously worded paragraphs in his statement. These stated as follows:

“8. There is no reason to believe that the information in this statement is inaccurate because of the improper use of the system. To the best of my knowledge and belief at all material times the system was operating properly, or if not, any respect in which it was not operating properly, or was out of operation was not such as to effect the information held within it.

9. Any records to which I refer in my statement form part of the records relating to the business of Fujitsu Services Limited. These were compiled during the ordinary course of business from information supplied by persons who have, or may reasonably be supposed to have, personal knowledge of the matter dealt with in the information supplied, but are unlikely to have any recollection of the information or cannot be traced. As part of my duties, I have access to these records.”

285.

Before I come to his evidence about this, it is obvious that the wording of paragraph 8 is almost that of a legal disclaimer (or a legally worded claim of accuracy, to be more precise), rather than a witness’ actual evidence. It would be very curious for a witness of fact to decide to put such a formally (and rather clumsily) worded paragraph in their witness statement. However, he was asked about these passages. He was asked what he meant by “the system”. The transcript records the following:

“Q: What is the "system" there? Is that the system of the process of extracting audit data, or is it something else?

(Pause).

A.

Good question. There's no -- I'm not sure what I was meaning by that, "There is no reason to believe ..."”

286.

He was asked if, in relation to the part which states “the system was working properly or, if not, any respect in which it was not operating properly, or was out of operation” the following:

“Q: Again, pausing there, it is slightly confusing. Are you aware or not aware of any instances where that system was not operating properly?

A.

No, not really, no.”

287.

He was asked if that part of the statement had been written for him by someone else, and he said he could not remember. He was also asked if it was a “Fujitsu party line” to put this in statements when it comes to extracting data and he said no, and that he was not aware of a Fujitsu party line. He accepted that gaps and duplicates in audit data were problematic, and he explained that they had to be investigated by audit support. It was clear to me that Mr Dunks considered audit data that had gaps and duplicates should not, in that form, be used. He said he would contact audit support if these occurred.

288.

However, some of Mr Dunks’ evidence, in particular what he had to say about paragraph 8 of his own statement, proved in my judgment to be somewhat misleading.

289.

He was shown a witness statement from Mr Gareth Jenkins from the Misra criminal trial in 2010. He said he had not seen this statement before. The statement put to him is some 8 years older, approximately, than his own statement. Mr Jenkins’ statement stated at the end:

“There is no reason to believe that the information in this statement is inaccurate because of the improper use of the computer. To the best of my knowledge and belief at all material times the computer was operating properly, or if not, any respect in which it was not operating properly, or was out of operation was not such as to effect the information held on it. I hold a responsible position in relation to the working of the computer.”

290.

The similarities between this passage, and the one in paragraph 8 of Mr Dunks’ witness statement almost 9 years later, are somewhat striking. Given he had not seen the statement before, there was no obvious explanation. His answer was as follows:

“Q. And you're not sure why it would largely replicate your paragraph 8?

A.

No, I mean, we do have a standard witness statement that we produce for ARQs. When we supply ARQs we are sometimes asked for a witness statement to go through the process and verify as far as I'm aware that the data I supplied is accurate. Now, we use that quite a lot and it may actually be in that statement.

Q. I see. So when I asked you earlier whether it was something of a Fujitsu party line you said you didn't think it was, but actually it looks as though it is on the basis of what you have just told me and on this document as well.

A. It could be. It may be part of our standard witness statement that we supply.”

291.

Mr Jenkins had given evidence in his witness statement from 2010 about duplicate entries in the audit data that had had to be removed. Mr Dunks was then taken to the witness statement of another Fujitsu witness in the Misra trial, called Penelope Thomas. She had given evidence about data extraction and listed a number of controls, which were remarkably similar to the ones that Mr Dunks gave evidence about (although Ms Thomas had omitted one). This is hardly likely to be a coincidence, if Mr Dunks really had compiled his list of controls from memory or experience. She also had the identical passage in her witness statement as Mr Dunks’ paragraph 8.

292.

Mr Dunks also said in respect of this, which in my judgment painted a rather different picture to that portrayed in his witness statement, the following:

“A. The statement again is to verify the integrity of the data once extracted and given. We don't control what's in the data. Our process is about extracting it and securely passing it over to the Post Office. It's not our concern of what's in the data. Q. It's not your concern what's in the data?

A.

No, it's what we're -- we're process-driven to extract certain types of data for certain requests.

Q. And so whether or not that might match another record or not, or replicate or duplicate or have gaps, that's not part of your remit, that's not really part of your concern?

A. No, it's not.

Q. I see. So reading that and reading the paragraph 8 as it was in your witness statement, it doesn't give much comfort to somebody that's then trying to rely on ARQ data as being a gold standard to compare and investigate anomalies, does it? A. Possibly not.”

293.

The reference to “gold standard” was the expression that had been used by Dr Worden in his report to describe the quality of the audit data that was held by Fujitsu. It should be a very great concern to anyone tasked with extracting audit data, should that data prove to have gaps and duplicates in it. This is because gaps and duplicates in the data affects the accuracy of that data. I consider that such a cavalier approach to whether audit data has gaps and duplicates, as evidenced by Mr Dunks’ saying that such things were not part of his concern, to be entirely contradictory to a statement verifying the accuracy of such audit data.

294.

My findings in relation to this are as follows:

1.

Mr Dunks expressly sought to mislead me by stating that there was no “Fujitsu party line” when it came to the contents of drafting witness statements about audit records for legal proceedings. There plainly is; it was used in the Fujitsu statements in 2010 and it was used by him in his statement for the Horizon Issues trial.

2.

The passage included in paragraph 8 of his statement is plainly a standard form of words, and reads as though it is a legally drafted assertion of accuracy. The witness’ assertion that the system was at all material times working properly, or if it wasn’t, this did not affect accuracy, does not sit consistently with Mr Dunk’s acceptance of gaps and duplicates in audit data.

3.

That Mr Dunks did not really know very much about paragraph 8 of his own witness statement was confirmed by the way he was puzzled when he was asked about it.

4.

However, Mr Dunks’ acceptance of standard wording provided by others (probably within Fujitsu) is less important than the fact that he originally tried to mislead the court.

5.

I do not accept that he identified the 12 controls in his statement from his recollection. He must have been working off another document, which he was not prepared to identify. I draw this conclusion because of the marked similarity with the controls included in Mrs Thomas’ statement from 2010; the wording of the controls themselves; and the fact he was prepared to mislead me about the paragraph 8 point.

6.

In re-examination he asserted that the statements in his paragraph 8 were true, and that he would not have signed his statement had this not been the case. I find that he cannot have known whether those statements were true or not. There is no evidence of any steps he took to check whether or not they were true or not. I find they were simply standard sentences supplied to him, and I find that he signed his statement without any independent knowledge of whether they were true or not.

7.

I find that passage in his witness statement simply to be meaningless assertion.

295.

He said that he was unaware of Post Office internal concerns that there was “a lack of system audit trail” in a document of 22 October 2016. I find his lack of knowledge about this surprising, given his role.

296.

I found Mr Dunks very unsatisfactory as a witness. He was both plainly aware of the Fujitsu “party line”, or corporate position, regarding the words asserting accuracy of audit data, and he was very anxious to keep to it, whilst initially denying that there was one. He sought to mislead me about both his paragraph 8 wording, and the way he had compiled his list of controls. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.

Mr Godeseth

297.

Mr Torstein Godeseth is employed by Fujitsu and is the Chief Architect on the Post Office account. He had provided three witness statements, as had Mr Parker. After graduating from Oxford with a degree in physics in 1974, he worked for Rolls Royce as a combustion engineer and then joined the Royal Navy in 1977 as an Instructor Officer. Whilst in the Royal Navy, he started his career in IT working in systems programming. It will be remembered that in the 1970s computing and IT was a relatively young field, and Mr Godeseth has therefore been involved in that industry since its relative infancy. He left the Royal Navy in 1981 and worked in systems programming and technical support for the IT at Forward Trust Ltd. He joined the Post Office IT department in November 1987. Initially this was to work on a project to introduce technology into Post Office branches.

298.

He was working with the Post Office as a technical advisor when the Post Office and Benefits Agency procured the Horizon system originally. He remained involved in different roles, including being outsourced from the Royal Mail IT department to Xansa in 2003, and contracted to the Post Office to act as the technical adviser interfacing with Fujitsu amongst others. Although an independent contractor, he worked between 2005 and 2010 with the Post Office as the technical advisor on IT projects including the change from Legacy Horizon to Horizon Online. He joined Fujitsu as a full employee in 2010. He described himself in cross-examination as the designer of Horizon.

299.

He had a greater knowledge of Horizon Online than of Legacy Horizon, although given his involvement in Horizon from its early days he had significant knowledge about both. Horizon was initially, in the late 1990s, called the Pathway project and was a joint effort between the Post Office and the Benefits Agency, including a way to computerise the payment of benefits to those entitled to them, through payments to benefits claimants which were to be made through the Post Office by means of a swipe card method. This project ran from approximately 1996 to 1999 and was to be provided by ICL (which Fujitsu partly owned and later fully acquired). At some point during the development of this project the Benefits Agency withdrew, and what had been the Pathway project, essentially a tri-partite venture (the parties being ICL, the Post Office and the Benefits Agency) became Horizon, now called Legacy Horizon.

300.

Although this litigation is not about the way that the Horizon system was designed, it is about its functionality and robustness in use, Mr Godeseth said that it was a specific Post Office decision not to have any dispute button/function for SPMs built into the Horizon system.

301.

He said:

“I think the basic argument was that disputes -- we wanted the flow of data through the system as quickly as possible because that keeps our books tidy and it was an inference that there was always the -- you had to press a button to take things through, but then you would pick up the phone to NBSC and say that wasn't right.” (emphasis added)

302.

I accept this factual evidence. It is, however, directly contrary to a major part of the Post Office’s case in the Common Issues trial, namely that by reason of an SPM accepting items at the end of a branch trading period, that gave the branch trading statement the effect in law of being what is called a “settled account”. I rejected that argument by the Post Office then, unaware when I did so of later evidence of fact that would emerge of the specific decision taken at the time by the Post Office that expressly did not include any feature within the Horizon system for a SPM to dispute items with which they disagreed. My finding in the Common Issues trial is consistent with Mr Godeseth’s evidence on this point.

303.

I deal with Mr Godeseth’s evidence within this judgment in more detail than any of the other witnesses of fact from either the claimants or the Post Office. This is because his evidence was considerably more detailed, and of more direct assistance in resolving the Horizon Issues, than any other witness of fact. He has been involved in Horizon in both its iterations for 20 years and has a vast amount of knowledge of its operation. I found the majority of his evidence reliable. I provide my conclusions regarding him as a witness at the end of this section below at [453] to [463].

304.

In Legacy Horizon, a messaging system was responsible for storing all data in Post Office branches and replicating it to data centres. This was called Riposte. Mr Godeseth decribed himself as being “on the other side of the fence” during this period and he had consulted Mr Jenkins for the section of his evidence dealing with when Riposte was in use. However, Mr Godeseth did describe himself as “having a pretty good knowledge of Riposte”.

305.

When it was brought in, Horizon Online was aimed, not at improving functionality of Horizon, but at reducing cost, and re-used (the term put to Mr Godeseth was “recycled”) many application components of Legacy Horizon. The data was no longer to be held at the branches (on the counters), it was to be held centrally in the branch database, or BDRB. Accordingly, harvesters were required to extract transactions from the BDRB. The branch database receives information from different sources. Data in Horizon Online is obtained from different sources.

306.

As an example with the lottery, if a person buys some lottery tickets in their branch Post Office, the information about a particular transaction starts at the lottery terminal in a branch; goes directly from there to Camelot; from Camelot it goes to Credence; and then from Credence it goes to the branch database, which is effectively that information arriving in Horizon. It is then transmitted to the terminals (in the branch)

and the SPM would see a series of TAs on the terminal in the branch, which that SPM would have to accept; these TAs would relate to the lottery tickets sold at the beginning of this short history. The transaction of the purchase of the lottery tickets by this route enters the branch accounts when the TAs are accepted, and the data is captured by the audit system. This is how lottery transactions work after what was called the Ping fix. Prior to that it was somewhat less streamlined, although given the process I have described is the post-Ping fix streamlined version, it is obvious that pre-Ping fix, the route for the data was even more convoluted.

307.

Other data for other transactions for other products goes into the branch accounts from what is called the basket. That basket is compiled by, as an example, a customer at the counter purchasing various Post Office and other client products; the SPM or assistant serving that customer and pressing an icon on the screen (for example for stamps, and other purchases); or for other products, from a PIN pad, scanning a barcode or weigh scales. The cost of a book of stamps is taken from the reference data table and the terminal shows the SPM or assistant how much money the total number of transactions are worth and what the cost to the customer is to be. It is basically the shopping basket which is put together by the SPM (or assistant) serving the customer and adding, by means of pressing buttons on the screen, different items to the basket as the customer goes through the items or products they wish to buy. At the end of this hypothetical transaction, if the cost of the basket is (say) £21.98, the customer will pay that amount (say by cash) which will be taken by the SPM and put into the cash amount held in the branch.

308.

In Legacy Horizon, the data was held in the branch on the counters in a message store. The message system called Riposte was how the replicating of the data to data centres occurred. The data was then stored on the correspondence server message store. This was explained by Mr Godeseth in the following way.

309.

All counter data was held in a bespoke message store (which was part of the Riposte product supplied by another company called Escher Inc.). This data was replicated within each branch to all counter positions and from each branch to the data centres where it was held in the correspondence server message stores. Similarly, any data inserted into the message store at the data centre (for example reference data or authorisations for banking transactions) would be replicated back to the branch counters. Selected data was then extracted from the correspondence servers to update Post Office's back end systems.

310.

Users with sufficient access permissions could inject additional messages (i.e. data) at the correspondence server. Any additional messages injected at the correspondence server by users with sufficient access permissions included information including the identity of the user. That information would not be visible in the standard audit extracts, but it would be visible in a detailed examination of the raw audit data. An SPM did not have access to audit extracts.

311.

There was some disagreement between Mr Godeseth and internal Post Office documents about whether the audit data from Riposte was held in Riposte attributable language (his view) or whether an audit conversion tool was required to convert existing audit data from Riposte to another readable/searchable format. This ultimately may not matter, and it may be that the author of the internal document suggesting otherwise did not understand that Riposte attributable language was a readable/searchable format. This difference in view does not matter for the purpose of resolving the Horizon Issues. The Riposte software was provided by another entity called Escher. There was no evidence about the contractual relationship between Fujitsu and Escher, and that does not matter for the purpose of resolving the Horizon Issues.

312.

Mr Godeseth had studied a number of problems in Horizon, and knew there were problems with Riposte, having seen numerous PEAKs referring to this. He had also been responsible for some testing (for example potential failures of controls, such as a basket not balancing to zero). He was not aware of items with the same Journal Sequence Number (or JSN) having been committed to the BRDB, and agreed that this should not be possible. This is because the whole way the BRDB works (or one of the principles of it, at least) is that each item committed there should be given a unique JSN. It should not be possible for two different items to be given the same JSN.

313.

Mr Godeseth explained that “the legacy version of Horizon was far more susceptible to communication glitches” and agreed that there were two improvements made when Horizon Online was introduced in that respect. These were that the susceptibility to communication glitches was reduced; and also the quality of the communications infrastructure was improved. It should also be added that because the storage of data was moved to somewhere not in branch (somewhere which had not existed in Legacy Horizon, namely the BRDB), and because Horizon Online was an online system, the communications infrastructure had to change substantially in any event.

314.

One internal document from as long ago as 2008 stated: “In addition, in common with many elderly systems that have been subjected to a succession of major changes, it [ie Legacy Horizon] has become increasingly difficult to make those changes, and expensive to operate." This was put to Mr Godeseth:

“Q: Now, that's a fair description of how the system originally was perhaps designed jointly with the DSS at the beginning and launched and then over the years, between then and 2008, there have been lots of sort of bolt-ons and additional things that have been changed on the system, haven't there?

A.

Yes, I think that's fair. The major one probably in my mind would be banking.”

The reference to the DSS meant the Benefits Agency. In other places this was referred to as DWP (or Department of Work and Pensions) but all three references – DSS, Benefits Agency, DWP – meant the department involved in paying welfare benefits. Different documents and references would use different acronyms to refer to the same entity.

315.

He explained that a small group of Fujitsu users from the SSC (about 30 users) had the ability to inject additional transactions into a branch's accounts in Horizon Online, using a designed piece of functionality called a Balancing Transaction. This was approximately the whole number of personnel who worked in SSC.

316.

He had given information in his first witness statement, namely that in Legacy Horizon, any transactions injected by SSC would have used the computer server address as the counter position which would be a number greater than 32, so it would

be clear that a transaction had been injected in this way by someone other than the SPM. This is important because it would be consistent with the case originally advanced by the Post Office that any such injections would be entirely visible as having been done externally (ie, not within the branch) due to the counter number used.

317.

However, this important information was simply incorrect, and was corrected both by Mr Godeseth and Mr Parker in subsequent statements before they were called, and as a direct result of Mr Roll’s evidence. The information that was incorrect, and therefore had to be corrected, had come directly from Mr Jenkins. This shows that Mr Jenkins did, in at least one very important respect, give Mr Godeseth directly incorrect information about the visibility of injected transactions, which not only could have an effect on branch accounts, and whether this would show (or rather, not show) that the impact on those accounts had come from injections made outside the branch. Mr Godeseth only found out the true position when Mr Parker was preparing his subsequent witness statement in the weeks prior to the commencement of the Horizon Issues trial, in other words in 2019. He had not known that before. His explanation about this was as follows.

“Q. You were finding out a detail that you didn't know before in quite a controversial area, weren't you?

A.

It was clearly an area that was going to be of interest because of the fact that we were inserting transactions into Riposte. It was an operational necessity and it was done in a controlled way. I had believed that the way that transactions were being injected would give them a counter position greater than 32 because the correspondence servers basically had nodes or addresses which were above 32, there was a special address for the gateway server, there was a special address for the extra disc in a single position branch and I had basically expected messages to be introduced using a different counter position and having read a whole number of PEAKs, I can quite clearly see that the standard practice in Fujitsu was to label something which was being inserted into Riposte so as to make it as clear as possible that it was not being done -- it was being done as something out of the ordinary, it was being inserted because of a problem.

So we had techniques for doing that. You could put in an attribute because this wouldn't be visible to a subpostmaster, I fully understand that, but it would be visible in the audit trail, when you ever come back to pull out the audit trail, you could put in an attribute to say "This was done under PEAK 75". Subpostmasters would never see that. They would not see it in their account in their branches and I'm fully aware that that is the case. It was a better audit than Mr Roll was alluding to when he said that it was left in a PINICL, because that would have been an audit which is separate from the actual data that we would be looking at should we ever need to pull stuff out of the audit trail and the intention was always to make it as clear as possible that this had been done under exceptional circumstances.

The techniques used to make it as visible to the subpostmaster as possible would be to put in references which referred to a counter that didn't exist in the branch, such as -- I saw a technique described in a number of cases which said put in a -- you know, if you are correcting something for counter 1, call it counter 11; if you're correcting something for counter 2, call it counter 12. These things would have been visible to a subpostmaster and the reason that you had to do it that way was to make sure that these transactions also got picked up and dealt with because these were legitimate counter numbers.

If I start to put in data with a number which is not a legitimate counter number then it's going to be ignored by systems further down the track. Q. So which were legitimate counter numbers?

A. Up to 32.”

(emphasis added)

318.

He also said in relation to Riposte that “Riposte was responsible for actually wrapping the message that we were looking to insert at the counter, and in doing that, Riposte will tell you the counter ID, or technically it was a stream, it would pick up the user ID, it would pick up the time, so this was effectively the envelope which wrapped the payload that we were looking to inject. If there was no user logged on at the counter then Riposte would introduce a blank user ID and that would be picked up in later processing.” This therefore meant that if, say, a SPM (or an assistant) was logged on at the time that the message was inserted at the counter, it would appear next to that SPM’s user ID (or that of the assistant). It would look as though the SPM had been responsible.

319.

This was made crystal clear in different passages of cross-examination:

“Q: You realised for the first time that it was possible to inject or insert a transaction with a counter position less than 32 when Mr Parker was preparing his second statement?

A.

Correct.

Q. And you knew that that was a contentious issue in this litigation, yes? A. Yes.”

320.

And further:

“Q. Now, it would be possible, would it not, to use a counter number of 1, or 2, or 3? A. It would.

Q. And if that counter number was a counter number actually in use by the SPM, it would appear to the SPM, from the records they could see, that it was a transaction which had been done in their branch, by them or their assistants?

A.

Yes.”

(emphasis added)

321.

I consider this to be extremely important evidence, both in resolving the Horizon Issues, and indeed in the whole group litigation. Its import is obvious. It means that Fujitsu could remotely insert a transaction into the accounts of a branch using a counter number which was the same as a counter number actually in use by the SPM (or an assistant). This would appear to the SPM from the records that they could see (and anyone else looking at those records) as though the inserted transaction had been performed in the branch itself. This information was only disclosed by Fujitsu (and therefore the Post Office) in this group litigation in January and February 2019. Even Mr Godeseth, a very senior person in Fujitsu so far as Horizon is concerned, said that he did not know this before.

322.

Mr Godeseth was also shown a Fujitsu document which was "Fujitsu restricted" and "Copyright Fujitsu Limited 2017". The title of the document was "Post Office Account – Customer service problem management procedure" and its purpose is described as "To describe and document the customer service problem management process." He was not sure he had seen the document before, and he was certainly not familiar with its contents. Both experts had seen the document and proceeded on the basis that what it recorded had in fact been adopted. “The customer” for the purpose of the document was the Post Office.

323.

He sensibly accepted that to have a robust system it was important to make informed assessments of where problems lie, based on the relevant information that was available, and also that it was important to capture and track that information in a way that could be readily analysed. Indeed, Mr Godeseth’s role included awareness that changes being made to the system were implemented without prejudicing the continued operation of the system. The document itself expressly defined “problem” as the unknown underlying root cause of one or more incidents. That, to me, seems a sensible definition. An unknown root cause of even one incident is a problem in a system such as this one. This is accepted by the document’s definition of “problem”.

324.

The document identified the metrics that were to be used within Fujitsu to measure this. Mr Godeseth agreed that they were professional and sensible metrics. They are stated as follows:

“The following metrics, to be reported monthly, will be used to measure effectiveness of the process and drive performance of the process and overall service in general:

resolved

-active actions and trend analysis

arising from proactive actions

CP) activities

services/major releases.”

325.

However, when the claimants had sought to obtain reports that would be expected to exist, which included or demonstrated the reporting against these bullet points which

Fujitsu’s own document stated were required, and which were supposed to be prepared every month, Fujitsu stated (through the Post Office’s solicitors) that “Fujitsu believes that it does not record problems in such a way that would allow this to be determined without retrospectively carrying out detailed analyses” and that it would require “a disproportionate effort and cost” to provide these. Mr Godeseth also gave written evidence that the reporting system identified in Fujitsu’s own document had not been implemented and that the records did not even exist. The passage in his statement said:

"I have spoken to my colleague Steve Bansal, Fujitsu's senior service delivery manager, who has informed me that the Post Office account customer service problem management procedure document was introduced by Saheed Salawu, Fujitsu's former Horizon lead service delivery manager and that Saheed Salawu left the Fujitsu Post Office account in around February 2013, before the new procedure had been implemented. I understand from Steve that Saheed Salawu's replacement did not wish to implement the changes and therefore the records referred to by Mr Coyne in paragraphs 5.157 to 5.159 of his report do not exist, as we continued to follow the previous existing reporting methodology.”

(emphasis added)

326.

Mr Godeseth did not know who Saheed Salawu’s replacement was, although he thought it might be a Mr Wicks. He did not know what the “previous existing reporting methodology” to which he referred in his own witness statement in fact was. He could not explain why the report which was said to have been abandoned, or at least not pursued after February 2013 when Saheed Salawu had left Fujitsu, was issued for approval in July 2014, over one year later. Indeed, he could not really explain why the records sought by the claimants did not exist at all. He thought the reporting might be done at meetings with ATOS, but he could not remember going to one. In fact, his evidence on this was so very vague as to be somewhat alarming, given his role within Fujitsu. He did not know if there were any records of the particular type apart from between the years 2014 to 2017, and he only knew that because Mr Bansal had told him.

327.

In another document, he was shown an entry which recorded that First Rate, a joint venture between the Post Office and the Bank of Ireland dealing with foreign exchange (also called bureau de change or bureau services), had identified an anomaly over the way that Horizon reversed transactions that were recorded and polled through to them. This was fixed in something called counter release 9, but although he said this happened “on his watch” this had not been brought to his attention.

328.

In the 2015 Problem Management Problem Review, the Dallmellington bug (which is an acknowledged and agreed software bug) was referred to, but without identifying how many branches were affected, for how long they were affected, or the amounts by which their branch accounts were impacted. He accepted that it was fair to describe the report as “not a particularly rigorous or robust treatment of recording the problem, its extent and duration and effect.”

329.

The following conclusions can be drawn from this evidence by Mr Godeseth. Fujitsu should, had it been interested in providing a robust system to the Post Office, been collating and reporting problems, which should obviously have included software

bugs, errors and defects, against the metrics contained in the Customer service problem management procedure document, or very similar ones. On the Fujitsu evidence before the court, it did not do so. The discrepancies in the dates of distribution and issue of the document entitled "Post Office Account – Customer service problem management procedure" and its "Copyright Fujitsu Limited 2017" do not tally with the account in Mr Godeseth’s witness statement that this process was abandoned or not pursued when Mr Salawu left Fujitsu in 2013. The dates just do not match up. The document was issued for approval one year after that gentleman left Fujitsu. Even the evidence that it was not pursued or implemented after Mr Salawu left was second hand from Mr Bansal, the source of Mr Godeseth’s knowledge.

330.

Had Fujitsu done what its own document, copyrighted in 2017, states was required, the experts in this litigation could, and probably would, have used those reports. Those reports would have included comprehensive records of precisely the sort of matters both experts had to investigate in order to provide their evidence to the court on the Horizon Issues. It is, in my judgment, a stark deficiency in the robustness both of Legacy Horizon and Horizon Online that such records were not kept by Fujitsu. This deficiency must inevitably also have an adverse impact upon the quality of information that the Post Office itself would have had.

331.

In the Post Office’s closing submissions, an explanation was given in submission in relation to the explanation given by Mr Godeseth in his witness statement which I have set out in [325] above about why the records did not exist, that explanation being that the procedure was not implemented. The Post Office submitted that it was only section 1.4 of the document that was not implemented, and not the whole document. This was in paragraph 147 of its written closing submissions, which dealt with what the Post Office submitted were “errors” made by Mr Godeseth which it submitted (and which the Post Office accepted) were “significant” and which it also stated had taken the Post Office “by surprise”. It sought to correct these errors, by way of submission rather than evidence, and one of those corrections was in relation to the implementation of the document in question. The claimants objected to this, and said the explanation that was provided was nowhere in the evidence, and the Post Office should not be permitted to give evidence by way of submissions.

332.

Leading counsel for the Post Office was asked about this specific part of the submissions at the end of oral closing submissions and said that the explanation in paragraph 147.4 of the closing was “my instructions but, my Lord, it is based upon a previous version of the document and then an amended version of the document. And I will undertake to give your Lordship the two references.” In an email of 10 July 2019, which provided some references, further explanation was given which were said to support the submission that it was only section 1.4 of the document that was not implemented. The Post Office relied upon the following four points in support of that submission:

1.

The passage in Mr Godeseth’s witness statement I have reproduced at [325] above.

2.

That there were different versions of the document that were disclosed, but only one had been put in the trial bundle. The first version containing section 1.4 was dated 6 June 2010 and was authored by Mr Salawu.

3.

This indicated therefore that the change was introduced by him. Most of the procedure document had been in existence for a number of years before that.

4.

Earlier versions had been disclosed, none of which contained section 1.4, and none included section 1.4 and none were distributed to or even mentioned Mr Salawu.

333.

The following points can be made about this.

1.

Witness statements are expected to be factually correct.

2.

Each party in this litigation who has submitted witness statements had ample opportunity to ensure those statements contain accurate evidence before they were served. Further, each witness is asked, at the beginning of their evidence when they confirm the contents of their statements in their evidence in chief. Apart from qualifications that were sometimes made, in this case (and in the great majority) the statements are usually stated to be true to the best of that witness’ knowledge and belief.

3.

If a witness’ written evidence is shown to be incorrect (as Mr Godeseth’s, along with other of the Fujitsu witnesses, was) then the appropriate way to correct this is either in re-examination or by way of supplementary evidence. Here, Mr Godeseth was asked about this document and gave evidence in respect thereof on 20 March 2019. Mr Parker, also a senior witness from Fujitsu, was not called to give evidence until 11 April 2019. That was, again, ample time for this self-contained point to have been dealt with by supplementary evidence from Mr Parker, if that was thought by the Post Office to be important. In any event, there is no reason for it having been incorrect in the first place. Whether a procedure was, or was not, adopted by Fujitsu internally, is not something that ought to be capable of misinterpretation in any event, let alone on a subject as important as recording problems in the Horizon system.

4.

I do not accept a matter such as this is suitable to be corrected by submission “on instruction” in any event.

5.

However, even if it were, I am afraid that my view of information emanating from Fujitsu is, based on numerous other instances of such information being wrong in the actual evidence, to be somewhat sceptical of its reliability. It would also be potentially unfair to the claimants to allow this type of “correction” as they would be deprived of the opportunity of testing the new explanation. Absent some corroboration of this type of information coming from Fujitsu that supported it, I would not accept it in this case on this issue.

6.

Further, and even if I were to take a contrary view of the points at 1 to 5 above, and accept that the Post Office should be given the opportunity of correcting Mr Godeseth’s evidence in this way, the explanation given by way of closing submissions does not assist the Post Office, for the reasons explained in the following paragraphs, [334] and [335] below.

334.

This is for the following reasons. Mr Coyne’s point, to which Mr Godeseth was responding, was that the Fujitsu document identified metrics and KPIs to measure/control and reduce the risk of failure to detect, correct and remedy Horizon

errors and bugs. He was basing this on section 1.4 of the document. His conclusion was that “From the above, it is my opinion that Post Office should be aware of all recorded bugs/errors/defects in addition to those previously acknowledged by them, from the process metrics compiled above.” These records were sought by the claimants and the Post Office explained that Fujitsu’s position was that the records did not exist. They were not therefore provided to the experts. Mr Godeseth’s crossexamination showed that the Fujitsu explanation was wrong (and the Post Office’s closing submissions accepted it was wrong). The records however were not produced.

335.

Finally on this, after receiving the email of 10 July 2019 to which I refer at [332] above, I asked in an email of 12 July 2019 for hard copies of the documents referred to in the relevant paragraph of that email (and hence relied upon by the Post Office in this respect) to be delivered to me. This was done, and a file served on the court containing four versions of the Post Office’s Customer Service Problem Management Procedure. They are version 0.1 dated 13 November 2007; version 2.0 dated 22 April 2008; version 2.1 dated 6 June 2010; and the version put to Mr Godeseth which was in the trial bundle, dated 5 September 2017. Given the additional versions produced all pre-date Mr Salawu leaving Fujitsu in 2013, this means that the points I have identified at [329] and [330] are still valid ones, and are still in the claimants’ favour.

336.

Even on Dr Worden’s evidence and his views on the lower number of bugs in the Bug Table (he accepts that there are 11, although Mr Coyne’s figure is somewhat higher), there are plainly more bugs in Horizon than Fujitsu itself was aware of. Over the years of both Legacy Horizon and Horizon Online, the total number of software bugs, defects and errors in Horizon considered by the experts is far greater than the number to which Fujitsu have admitted. This is shown in the appendix to this judgment summarising the number of bugs, errors and defects, and their years of operation. The total number of software bugs, defects and errors in both versions of Horizon is very important information. Why Fujitsu chose not to collate and report these in the manner that even a Fujitsu internal document stated would (or should) be done, is wholly unclear. A comprehensive record, whether using the metrics identified by Mr Salawu or other similar ones, would surely only assist both Fujitsu and the Post Office in its administration of Horizon. Given that such a comprehensive record has not been produced to the experts for their consideration, notwithstanding the express request made on behalf of Mr Coyne, the court is entitled to conclude that no such comprehensive record exists.

337.

Mr Godeseth was also asked about the BRDB transaction correction tool. This applies to Horizon Online. The original design document for this stated that it was to allow the SSC to correct transactions by inserting balancing records to transactional/accounting or stock tables directly into the BRDB system, and also to audit the changes made. It was not designed to delete or update the records. However, in cross-examination the claimants’ leading counsel analysed, and put to him, some careful points on how this tool had been used in practice.

338.

The tool works by means of an SQL insert into a variety of tables in the BRDB. SQL is an abbreviation for Structure Query Language, and is a domain-specific language used in programming and designed for managing data. It is very widely used. Although Mr Godeseth would refer simply to “Oracle” as the programme, Oracle Database or Oracle is a database management system which is produced and marketed

by the US corporation which is also called Oracle. Although that is the name of the company, and SQL was developed (or invented) before the Oracle Corporation was founded, in this trial the terminology sometimes became a little shorthand on the part of those IT professionals very experienced in such matters. The Oracle Corporation is very well known, not only in the computing field, but through its co-founder Mr Larry Ellison who, amongst other achievements, has backed and competed in various America’s Cup races, including winning the 33rd America’s Cup in 2010 in the trimaran USA 17 with Oracle BMW Racing. Mr Godeseth, and the experts, are all highly experienced in the field and shorthand references to well-known languages or applications such as SQL and Oracle are to be expected. This use of shorthand did not lead to any lack of clarity in the evidence, and I refer to it only for completeness.

339.

Information provided by the Post Office’s solicitors prior to the trial stated that the transaction correction tool had been used 2,297 times. However, all save one of these usages were said to be “Type 1” which were for the most part used to unlock stock units and which had no impact on branch accounts. There was one admitted instance of what was called “Type 2”, which was where a Balancing Transaction was inserted where it changed transaction data in the main transactional tables. This will have affected branch accounts. Mr Godeseth had not been involved in drafting that information supplied by the Post Office’s solicitors, but he knew about it and it matched his view of the intended use of the transaction correction tool. The 10 database objects, or tables, and sequences into which the tool was permitted to write are identified in a table at 2.4.1 of the low level design document for the tool. They are identified in terms of their object names, and were the BRDB operational exceptions table; the system parameters table; the FAD hash outlet mapping table; the process audit table; the process audit sequence; the transaction correction tool journal table; the FAD hash current instance table; the transaction correction tool control table; the branch information table; and finally the branch operators (or operational) exception sequence.

340.

The design document made clear that:

“The following transaction tables have been granted INSERT privileges to OPS$SUPPORTTOOLUSER.

The transaction correction statement is only allowed to insert into these tables.”

These 9 tables were then listed. The process would use a "Transaction file containing an SQL INSERT statement that creates the required balancing transaction.", which is an Oracle command, and the SQL INSERT statement would provide a missing half of a transaction, where only one half was present. As it was put to Mr Godeseth, “the "SQL INSERT" statement effectively goes in and puts in the missing other side of that transaction?”, a point with which he agreed. The SQL INSERT command would put that missing other side into the records.

341.

The SSC members would log into their own UNIX user, and then change directory and place their transaction file into the sub-directory. The document then records “They [ie the SSC member using the tool] will then invoke BRDBX015 manually.

The shell script module will be owned by the UNIX user 'supporttooluser'."

342.

However, the explanation that was provided by the Post Office’s solicitors, of which Mr Godeseth said he was fully aware, about the use of the tool was as follows:

"Each document is associated with a single SQL statement which made a database correction. There are two different types of correction shown in the files - the SQL statements for each are of the form:

"1. Update OPS$BRDB.brdb_rx_recovery_transactionsSET_ settlement_complete _time stamp = ..." And then the "INSERT INTO" command.

343.

The explanation of Type 1 was "Type 1 reflects the action taken to reset the recovery flag on a transaction. This will have no effect on branch accounts….”. It was also stated expressly that "Type 2 reflects the action taken to insert a Balancing Transaction where it changes transaction data in the main transactional tables. This will affect branch accounts."

344.

Mr Godeseth expressly accepted, however, that this showed that the command used for what was being called Type 1 was not an insert command, it was an update command. Mr Godeseth described himself as not expert on Oracle, but he had a working knowledge of it, and he is plainly more experienced with using Oracle than most people. I find his experience in Oracle more than sufficient for him to be able to answer questions on this subject. Mr Godeseth’s evidence shows that Type 1 was an update – something which the low level design document expressly said the tool would not be used for. The design document did not contain the necessary database object fields table for performing the unlocking function, which the Post Office’s solicitors said represented all of the occasions, save one, when the tool had been used. This was all put carefully and clearly to Mr Godeseth in his cross-examination, and he agreed with all the points put.

345.

This was further confirmed – the mandatory use of insert, and the fact that an update command was not included in the design intent of the tool – elsewhere in the design document where it stated under method “The module will read the contents of the input transaction file, which will be in the form of an SQL insert statement. Only a single insert statement is allowed and (after an optional introductory comment) it must start with the 'insert into' clause." The insert would be in the SQL script, and Mr Godeseth expressly confirmed this. This is another example of Fujitsu failing to observe its own design intent for the use of a tool that can have an impact upon branch accounts, and in my judgment is a factor that goes to the robustness of Horizon as well as the accuracy of data. As was put to Mr Godeseth, the use of the tool had gone “way beyond” the intent in the design document, and he agreed. He said that “there is tooling which is based on this” – by which he meant based on the transaction correction tool - “which has two aspects to it, certainly, so I think I’m agreeing with you”. It is agreed that there was only one occasion when what was called Type 2 was used. This evidence goes to the wider point that I have explained, namely Fujitsu failing to observe its own design intent.

346.

Mr Godeseth was taken, very carefully, through a specific use of the transaction correction tool in 2010. In PEAK 0195561, a problem was reported to the SSC on 4 March 2010 where a SPM had tried, on 2 March 2010, to transfer out £4,000 (referred to in the PEAK as 4,000 pds, which means either pounds (plural) or pounds sterling)

from an individual stock unit into the shared main stock unit when the system crashed. The SPM was then issued with 2 x £4,000 receipts. These two receipts had the same session number. The PEAK, as one would expect, records various matters in note form and also uses informal shorthand. However, the main thrust is that when the SPM did the cash declaration, although the main stock unit (into which the £4,000 was being transferred) “was fine”, the unit from which the cash was taken “was out by 4000 pounds (a loss of 4000 pds)”. This is very similar to what Mr Latif said had happened to him, although the transfer in July 2015 to which he referred was £2,000. The PEAK related to Horizon Online and was the admitted occasion when the Balancing Transaction tool had been used.

347.

Mr Godeseth had obtained the information in his witness statement dealing with this from Mr Parker. He also said he thought it was puzzling, and he was not sure that the PEAK “would be a totally accurate reflection of what happened” and was “an interpretation put together by a developer who is investigating the problems.” He suggested the original log files would be better (by which he meant better sources of information) and still exist, but they were not before the court. These log files would be held by Fujitsu, so any failure to have them before the court was no fault of the parties in this litigation, and that can only be laid at the door of Fujitsu. If a senior Fujitsu witness thought they would be better records, they should have been accessed and evidence about them could have been given directly. Certainly Fujitsu witnesses could have given evidence that relied upon them, had they so wished.

348.

The PEAK records the following:

“After discussion with Gareth Jenkins, the suggested correction is to negate the duplicate transfer out by writing 2 lines to the BRDB_RX_REP_SESSION and BRDB_RX_EPOSS_TRANSACTIONS tables, with:

1)

Product 1, Quantity 1, Amount 4000.00, Counter mode id 7 (TI)

2)

Product 6276, Quantity -1, Amount -4000.00, Counter mode id 7 (TI)

This should be done using the Transaction Correction tool. An OCP approved by POL will be needed.”

349.

This concerns Horizon Online, due to the references to OCP and OSR, and shows extensive efforts being made at Fujitsu to try to understand why this has occurred, and general doubt that the efforts to reproduce the fault in testing has worked. Some at Fujitsu wanted the priority to be downgraded; other entries resisted this, shown by some of the entries such as:

“What is missing from this Peak is an explanation of the events in terms of the requests, how they were ordered and when any was committed. Only then can we qualify the priority. The assumption is that we have a fix. The facts are –

1: A settlement request to timed.

2: A retry of request timeout occurred.

3: According to the DB entries both later succeeded.

Now unlike other reconciliation Peaks this stands out…..

We can’t reduce the priority unless we understand what is going on.”

350.

The PEAK is lengthy and demonstrates a degree of frustration within Fujitsu at their own failure to get to the root cause. One entry on 24 March 2010 states inter alia “…..this shouldn’t have happened”. One might observe that this is stating the obvious. A later one on 25 March 2010 states “advised on the latest update in the call she states she was going to check if it was the same thing that happened before. [The Post Office] are want to know why this has happened? - why does it keep happening? - can you advise on this.”

(emphasis added)

351.

One of the reasons or concerns expressed in the PEAK is that JSN entries – which should be unique and in respect of which no duplicates should be permitted in the journal table - appear to have occurred. Mr Godeseth confirmed that this should not have happened. Duplicate entries should not be permitted in the database. The documents showed that this did occur. Mr Godeseth said that the “JSN is part of the primary key into the message table, I cannot see how an Oracle database would allow that to happen” and “that would have to be a bug in Oracle, and it was certainly not the bug that I was looking – it was not the bug that I was looking at for the red alert”. He also said: “If this in fact happened as written down here [ie in the PEAK] then it would have had to have been a bug in Oracle and I certainly don't remember any such bug.”

352.

The PEAK also shows computer script being used, both in an attempt to correct, and understand and analyse, what occurred. On 23 April 2010 an entry states:

“I think we have done as much as we can on this one. In conclusion although we haven’t been able to totally explain the behaviour, the risk of this type of PEAK occurring again has been minimised in live due to a change of behaviour in the BAL with respect to transactions…..This may now be marked as a duplicate of PC0194893”.

Rather concerningly – but entirely consistently with what I consider to be Fujitsu’s general approach to closure codes - the following entry appears that same day in another entry stating that the “Defect cause” is “updated to 40 – General – User”.

353.

However, later the same day the following entry is made:

“I am sending this call back with Response Rejected.

Closing a call as 'Duplicate Call' results in a black mark against me. It basically means that I should not have sent the call over since the same problem has already been sent over in a previous call.

PC0195561 (duplicate transfer of 4000.00 cash) may have been caused by the same underlying fault as PC0194893 (banking reconciliation), however I could not have been reasonably expected to link the 2 calls and take the decision that it was not necessary to send PC0195561 over for further investigation.

Please close this call with category 'Advice After Investigation'

[End of Response]

Response code to call type L as Category 52 -- Pending -- Response Rejected

Response was delivered to Consumer.”

354.

Eventually KEL cardc262s was updated with the information in the PEAK on 4 May 2010 and the call was closed. One of the final entries states that day “Category 60 – Final – S/W Fix Released to Call Logger”. “S/W fix” means software fix. The call is closed the same day. This is 2 months after the incident. The PEAK also referred to other PEAKs. Mr Godeseth was asked about those too. PEAK 0195962 records that the transaction tool has “been used in live”. That led to an Operational Change Process or OCP 25882 and stated:

Due to a system fault, the branch did a Transfer Out of £4000 and a corresponding Transfer In of £8000

Justification: Correct a loss of £4000 at the branch due to a system fault

When: Planned for 10/03/2010 16:00 with a duration of 30 minutes

Extra detail: The Transfer In details were incorrectly doubled up when they were written to the BRDB. This needs to be corrected using the Transaction Correction tool.”

(emphasis added)

355.

The actual printed address of the OCP document is as follows:

http://deathstar/SSC2/SSC_OCP/viewocp.jsp?OCPRef=25882&SID=f630192836137

356.

This may show that the folder in which the document was kept was named by someone with an interest in a particular series of well-known films, or it may not, and this does not matter. It does, however, show that it was plainly generated or kept by the SSC.

357.

Mr Godeseth agreed that the OCP showed at least one use of the transaction correction tool for one balancing transaction. Part of Mr Godeseth’s explanation for what had occurred, even though there had been two duplicate JSN entries, was as follows:

“In this situation the symptoms, as I'm reading them, are that because there was a bug in Horizon and this is in the new system, it was pretty early days of the Horizon Online, because there was a problem, we had something coming through which got through the journal filter but then failed at the branch database and so therefore, as far as the branch database is concerned, it has not happened.

MR JUSTICE FRASER: I understand that. I think what Mr Green is putting to you is that this shouldn't have got past the journal filter.

A. It's a bug, so certainly the way that the system should have worked, a JSN -- the same JSN coming up would be just a simple repeat of the message for -- because there was some sort of glitch.”

(emphasis added)

358.

His final evidence in cross-examination on this was the following:

“A. Somebody may have been trying to look for a duplicate JSN entry. I can't really comment on what the guy was doing at the time who was trying to investigate. I am simply asserting that barring an Oracle bug, which would have been huge, you cannot have two entries with the same JSN.”

359.

In my judgment, the Horizon Issues are sufficiently wide and costly to resolve concerning the Horizon System, both Legacy and Online, without widening them to include some attempt at including specific bugs within Oracle too. The Oracle Corporation is not even a party, and has had no opportunity to provide evidence or submissions. I simply recount Mr Godeseth’s evidence as he attempted to explain the PEAK. He did, as can be seen, refer to potential bugs in Oracle as an explanation for something that he believed to be puzzling. He accepted it was a bug and/or “a glitch”. In any event, other than Mr Godeseth’s evidence expanded in re-examination that he was involved in 2010 when establishing himself in Fujitsu (and in this he referred to an Oracle bug, which involved nodes in the database going down) there is no evidence before me that there were any bugs in Oracle in respect of this doubling up, and I reject the suggestion that this can be laid at the door of Oracle. What this PEAK shows, in my judgment, is that Fujitsu itself, with a great deal of time to prepare for the Horizon Issues trial, simply could not explain then, and cannot explain now, what caused this event to occur. That is relevant to my consideration of the Horizon Issues, and in my judgment shows that there was in that respect a bug, error or defect within Horizon. It also shows that there were those in Fujitsu who wished to close the call with the error put down to the user, as shown by the “Defect cause” being “updated to 40 – General – User”. There is nothing in any of the material on this incident to suggest it was in any way caused by the user.

360.

He was taken to another PEAK, PC0175821 which was dated 19 February 2009. This is during the period of Legacy Horizon.

361.

PEAK 01320275 was in relation to an incident on 21 December 2005 which showed a SPM, at a five counter site with seven stock units, had rolled over a particular stock unit called BB. He rolled it over in what was called “an effectively empty state” but having declared the correct amount of cash, and also adjusting the stock levels to the correct volumes, his branch account showed “a gain of approximately £18,000”. This was an obvious impact upon his branch account, and it was investigated. One entry in the PEAK stated:

We are unable to correct the system figures safely. We can however provide accurate figures for what should have been in the Final Balance for BB, to enable POL [ie the Post Office] to make the correction perhaps by using a Transaction Correction.

POL need to make a decision on whether they are able to correct the problem in this way, however we do not see any other alternative. Corrective action should be taken before 11th January when the branch is due to roll into TP10.

The cause of the problem is unknown and is under investigation.”

(emphasis added)

362.

This accepts in terms that there is a problem, and accepts that its cause is unknown. The suggestion by Fujitsu later in the PEAK on 3 January 2006 was to generate a negative figure effectively by changing the relevant data in the messagestore to try and cancel out the £18,000 which the SPM had showing as a positive figure in his branch accounts. It is obvious to me that this would only remedy the effect of the problem, rather than identify what had caused the discrepancy in the branch accounts in the first place. However, the entry is illuminating in any event:

“If we get to the problem before the office is rolled we are able to change objects in the messagestore to reset the stockunit back to the CAP (TP) rollover trailer. The PM can then rollover. PM should get a large shortage which cancels out the large gain.

We don't want to be having to do this as making manual changes to the messagestore is open to error and each time we have to seek authorisation from POL to make the changes.

If we get to the problem after the office is rolled (as in this call) then we are unable to correct the system figures safely. Its not been decided how we get the PM sorted out.

All in all, we want this fixed asap.”

(emphasis added)

363.

Mr Godeseth accepted that generating the artificial negative figure in the message store could be done, but said he was “not involved interfacing with Horizon” at the time and was not at the operational level shown in the PEAK. He was not involved in obtaining authority from the Post Office for such matters. In my judgment, the way that Fujitsu anticipated resolving this, by artificially seeking to engineer “a large shortage which cancels out the large gain” is not exactly grasping the nettle. It would have the effect of disguising why the large gain had occurred in the first place, in other words a bug, error or defect.

364.

Another PEAK 0152014 dated 10 December 2007 showed a problem that was described in the PEAK as “POLFS Incomplete Summaries Report” which related to a foreign currency transaction. At this time, £484 sterling could buy $1,000 (the PEAK did not specify US or Canadian, but the associated OCP identified US$ specifically) and the PEAK recorded that:

“This is due to a single SC line written for $1000 (£484) with no settlement in the middle of two RISP transactions.

On call PC0151718 the harvester exception was corrected and now the transaction for the day don't zero, hence this issue with the incomplete summaries report.

Am currently retrieving the messagestore for this branch, we will then be inserting a new message on the counter to remove the effects of this. OCP 17510 has been raised.”

365.

Again, this was being done to correct the effect of what had occurred, rather than discovering what had led to the issue in the first place. In two places in the PEAK, the following entry appeared:

“**Again, this may also have caused a receipts and payments error, can EDSC please confirm whether this is a gain or loss at the counter and the amount.**”

366.

Mr Godeseth had been asked to look at this PEAK in advance (his evidence went over two days and this was agreed by the parties), and therefore he had been given a chance to familiarise himself with its contents, and those of an associated KEL. The PEAK stated "Worth noting that the branch did not have any issues with the mismatched transactions because this was fixed before they did the roll. The branch is not aware of this and it's best that the branch is not advised." (emphasis added)

367.

When it was put to Mr Godeseth that this showed that on not all occasions were SPMs advised of impacts on their branch accounts of particular problems, he said that was “a fair inference”. An inference is a common sense conclusion, and I agree with Mr Godeseth that it is indeed fair to draw that inference. He did however then add “I think I would say that Post Office were well aware of this and I would argue that it's a Post Office decision whether or not to tell a subpostmaster.” This shows that Mr Godeseth was not immune from arguing the case or making suggestions not supported by the evidence. Nothing was produced by either side to show that the SPM in this case was told. Certainly the person at Fujitsu who wrote “….it’s best that the branch is not advised” cannot have shared Mr Godeseth’s view at the time. Mr Godeseth did however also say, in my judgment again importantly, that “in the background to this there was a dialogue with [the] Post Office” and he agreed with counsel who put to him that the “Post Office would have been aware of what was being done by Fujitsu”.

368.

The associated OCP however, whose subject was “write corrective bureau message for FAD 183227” – bureau meaning foreign currency – gave a great amount more detail. It stated:

“A single SC message 183227-7-1101211 was written in error on 26th November at 12:43:17, selling 1000 US dollars, with no corresponding settlement line. To remove the effects of this message at both the branch and on POLFS, we will insert a new message to negate the effects of the original message.

Justification: If the change is not made in the counter messagestore (before the stock unit is balanced on Wednesday), the branch will have an unexpected gain of £484 (or thereabouts - depends on exchange rate), and a receipts and payments mismatch. This gain would have to be resolved at the branch. There would also be an inconsistency between the branch and POLFS to be resolved. By correcting the problem locally, the branch may not be aware of the problem, and there will be no inconsistency between the branch and POLFS.”

And:

“The message will include a comment to show it has been inserted to resolve this problem (this will not be visible to the branch).

This change will first be applied to a copy of the messagestore within the SSC environment, and the stock unit then rolled over to make sure there are no unexpected consequences.

Neither the new nor the old message will be included in data sent to POLFS.

Gary Blackburn (POL) is already aware of this issue.”

(emphasis added)

369.

This shows that the Post Office did know, as Mr Godeseth had said. It also shows that the branch would not be able to see what was done. It also showed that the intention was to reverse, by means of a specific insert into the messagestore by Fujitsu, an entry or data in order to correct a discrepancy in the branch accounts caused by an error or problem within Horizon Legacy. This was approved by the Post Office as shown by a later entry. The OCP had been raised by Anne Chambers. The “extra detail” – effectively what the message needed to contain or achieve – was:

“Extra detail: The original message had ProductNo:5129, Qty:1, SaleValue:484, PQty:1000. The new message will have Qty:-1, SaleValue:-484, PQty:-1000, with other attributes (including exchange rate) as before.”

370.

The specific message that was inserted into POL FS, as shown in the OCR, was as follows:

“Extra detail: This OCR is being raised so that EDSC is authorised to amend the txn details for branch 183227 and insert these into tps_pol_fs_summaries_incomp table on the host.

Comments

Andy Keil (POA SSC Support) wrote at 12/12/2007 15:07: Updated POLFS feed for branch 183227 product 5129 mode SC with SaleValue=1014.73 and PQty=2080”

Mr Godeseth agreed that this showed “just over $2,000 being inserted in the Post Office system”.

371.

However, the PEAK clearly demonstrated that the fix or corrective action that had been applied had simply made matters worse. This is because it stated for 14

December 2007, two days after the message had been inserted by Andy Keil of Fujitsu, the following:

“The counter problem which caused the first issue has been corrected by inserting a message into the messagestore, for equal but opposite values/quantities, as agreed with POL (OCP 17510).

As a result of this corrective action, the net effect on POLFS is zero, and POLFS figures are in line with the branch. POLMIS received both the original message and the corrective message.

Once the problem was corrected, there should have been no impact on the branch. However it has been noted that the stock unit BDC had a loss of $1000, which was generated after the correction was made. We have already notified Gary Blackburn at POL (email attached). This appears to be a genuine loss at the branch, not a consequence of the problem or correction.”

(emphasis added)

372. On 17 December Anne Chambers also stated in the PEAK:

“Summary for development:

A single SC line was written for $1000 (£484), with no settlement, in the middle of two RISP transactions.

The line was missing some AdditionalData so it wasn't harvested properly, but the main problem was the lack of settlement. POL authorised us to insert an equal but opposite message, to prevent a discrepancy (in theory anyway) and to avoid problems on POLFS. Please note that this is exceptional and must not be seen as a convenient avoidance in place of a fix.

Subsequent investigations showed that a second packed pouch barcode had been scanned, before the receipt for the first had finished processing. See the audit log / messages on counter 7 26th Nov at 12:43 / 12:44.

PC0147357 is already with development for what I think is a similar problem - Mode reverts to what it was prior to RISP.

Please also see update on that call.

Routing to EPOSS-Dev via QFP.

[End of Response]

Response code to call type L as Category 40 -- Pending -- Incident Under

Investigation.”

(emphasis added)

373.

The PEAK was finally closed on 17 April 2008, the same day that the “defect cause” was “updated to 14: Development – Code”.

374.

There are two important points to make in relation to the entry in this PEAK which I have quoted at [371] above. Firstly, the discrepancy in the branch accounts, which

was, after the “corrective action”, in deficit, was broadly in the amount of the difference between the two corrections that were made, namely the one by the OCP and the OCR. These corrections should, in my judgment, have been in the same amount, as one relates to “front end” (or the branch messagestore) and the other “back end” of the accounts so far as Horizon is concerned. The statement in the PEAK that “this appears to be a genuine loss at the branch, not a consequence of the problem or correction” is simply insupportable. That is a statement made by Anne Chambers, which in my judgment flies in the face of the documents. I am supported in my conclusion by two things. Firstly, Mr Godeseth’s evidence, when it was put to him that this was a fair possibility when he said “having read this PEAK in more detail overnight then yes, clearly that is what appears to have been the case.” I accept that evidence. It is supported by the documents and by the text of the two different corrections. Secondly, plain and obvious common sense.

375.

The second important point is that the PEAK makes it clear this, or something very similar, has happened before. It states “PC0147357 is already with development for what I think is a similar problem - Mode reverts to what it was prior to RISP.” The reference to “development” is the department within Fujitsu where software is written to correct issues with the system. This is supported by the closure of the PEAK and defect cause stating “Development – Code”.

376.

There is an associated issue that arises on this PEAK, the number of which I will again recite for clarity, PEAK PC0152014. I will deal with this here, rather than in the section dealing with Mr Coyne’s evidence or within the Technical Appendix (although the latter place could be a more suitable home for it). This is because it goes to the weight to be given to Mr Godeseth’s evidence.

377.

A detailed explanation was put to Mr Coyne on this very matter, which occurred on the third day of his evidence on 6 June 2019 and went from page 103 of the transcript, ran until page 115 when the OCP was then put (which went to page 120) and then the OCR which was put between page 120 and 126. Leading counsel for the Post Office stated “I can tell you on instruction what that figure means”; “So on instructions I can tell you than the sale value of £1,0114.73 and the quantity of 2,080……” and “I appreciate that I’m telling you this on instructions because I have to say nobody knew that this suggestion was going to be made until it was put to Mr Godeseth….” It was made clear that there was what was being put to Mr Coyne as a detailed explanation of what had occurred, was “on instruction”, and it was nowhere in the evidence.

378.

Mr Godeseth was cross-examined on 20 and 21 March 2019. When Mr Coyne was cross examined, about 2 ½ months later, a great deal was put to him “on instruction”, as I have identified. It was a detailed factual explanation that was not included in any evidence at all. Firstly, the OCR that went with it showed that “This OCR is being raised so that EDSC is authorised to amend the txn details for branch 183227 and insert these into tps_pol_fs_summaries_incomp table on the host.” This was required precisely because Mr Godeseth was correct, in my judgment, in his understanding of what had occurred. I accept his evidence on this. Secondly, the explanation put to Mr Coyne “on instruction” was (broadly) that there were two problems. A careful attempt was made to explain to Mr Coyne how (basically) Mr Godeseth had misunderstood, and to downplay the effect of what had occurred.

379.

I do not accept that this subject was capable of being corrected – even if that is what the documents showed, which in my judgment they plainly did not – in the way adopted by the Post Office by attempting to give evidence by way of submissions from counsel “on instruction”. This deprived the claimants of any ability to challenge the explanation, if explanation it was, by cross-examination. Further and in any event, there was ample time for the Post Office to have provided a short witness statement from a Fujitsu witness, such as Mr Parker (who was not called until 11 April 2019) dealing with this, if that was the evidential explanation. If that had been done, permission for that statement would have been required, but given the matter arose from something Mr Godeseth was asked about, that application could have been dealt with at the time and if this genuinely was evidence that the Post Office could not have known in advance would be required, it would have been allowed in. Even supplementary questions in chief could have been used as a method of obtaining actual evidence on this point. It simply is not procedurally acceptable, or fair, for evidence of this nature to be given by way of submission “on instruction”. However, and in any case, I have considered the Post Office’s case on this notwithstanding these points, and I find that the documents do not substantiate the explanation given on Day 16 to Mr Coyne when he was being cross-examined.

380.

The explanation did not make sense on the face of the documents, and they did not support the points that were being put “on instruction”. Firstly, as Mr Coyne pointed out, the PEAK identified that a message was to be inserted into the messagestore. None of the documents put to Mr Coyne “on instruction” identified what that message was. This point was made by Mr Coyne during the cross-examination:

“Q. Yes. So what they are talking about here, what this change here is, is a change being made by using the TIP repair tool into the TPS, correct?

A. Right, so this doesn't relate to the creation of the message then.

Q. It doesn't relate to the branch accounts, Mr Coyne, does it? This is an OCR which involves an exercise -- well, the use of the TIP repair tool to change data that is in the TPS system, yes?

A. Yes, but the PEAK refers to the insertion of a message into the messagestore.”

381.

The critical point, that the explanation skipped over, was what the message that was inserted in fact was, in actual terms. Had that message been produced, it could have been looked at, and deciphered. Understanding such a message is not an insuperable task beyond mortals. The second point is that the PEAK refers to a KEL, namely KEL obengc3120K. At the beginning of the trial, I asked for hard copies of all PEAKs and KELs put to witnesses to be prepared in hard copy too, so that I had a working file that I could use during the trial for PEAKs and KELs already referred to (in addition to the three OPUS trial bundle screens that I had to work from). This was done. That KEL is at tab 21 of volume 2 of the file entitled “PEAKs and KELs referred to in Days 5-8 and 12”. The KEL states the following, and I will reproduce the text in full, including the relevant table references.

“KEL obengc3120K

Title:

Not harvested: Could not update database: Updating table TMS_RX_BDC_ TRANSACTIONS, ORA-02290: check constraint

(OPS$TPS.BT23B_BUREAU_REGION_CHK) violated.

Summary:

Harvesterd did not harvest message: Could not update database: Updating table

TMS_RX_BDC_ TRANSACTIONS, ORA-02290: check constraint (OPS$TPS.BT23B_BUREAU_REGION_CHK) violated

Symptoms

The Harvester tried to insert into the BdC table TMS_RX_BDC_TRANSACTIONS a message with missing ‘Blackbox’ data. This caused Oracle [ORA-02290] to detect a check constraint (OPS$TPS.BT23B_BUREAU_REGION_CHK) violation, hence the appearance in the TPSC254.<br><br>Harvester exceptions do not normally cause entries on TPSC250, but this one seems to. <br><br>TPSC257 displays the contents of the TPS Incomplete Summary table where the summarisation program detects that the total harvested transactions for the day don’t NET to zero.

Problem

Due to some data in the ‘Blackbox’ or AdditionalData of the messagestore not being populated, the bureau product was missing key data including<br>BUREAU_REGION<br>MARGIN<br>MARGIN_PRODUCT<br>EF FECTIVE_EXCHANGE_RATE<br>and so when the Harvester attempted to insert the message into the BdC table TMS_RX_BDC_TRANSACTIONS, however Oracle [ORA-02290] detected a check constraint

(OPS$TPS.BT23B_BUREAU_REGION_CHK) being violated.

Solution – ATOS

Warning: Check the AMOUNT (or sum of AMOUNTs) for the branch on the TPSC254 report.<br><br>If the total AMOUNT is NOT ZERO, and the branch is NOT on TPSC257 (same day), <br>or the total AMOUNT IS zero and the branch IS on TPSC257 (same day) then there is probably a problem with the messages written on the counter – a bureau SC message with no corresponding settlement. Use KEL <ahref=kel._view_kel.jsp?KELRef=acha3159Q>acha3159Q</a>.<br><br>Check the messages anyway, just to be on the safe side. If the session was settled properly….<br><br>MSU must raise an OCR so that SSC can use the TIP Repair Tool to populate the missing columns, using values from another transaction for the same currency, same branch, same day if possible.<br><br>NOTE: By repairing the txn, the TPS_POL_FS_SUMMARIES_INCOMP will be corrected automatically after that txn has been successfully harvested. <br><br>*** In the past, we have just repaired these, on the assumption that they are genuine SC transactions which have been corrupted. However I think it may not be a genuine sale, but is related to an attempted pouch reversal (this is certainly the case for the more extreme instances where the settlement is missing). If the SaleValue and the PQty have opposite signs, and there is no reversal (mode ER) for the transaction, this may have caused a loss or gain at the branch which they can’t resolve themselves. Let Anne know if there is another occurrence like this.

(emphasis added)

382.

This KEL shows that there was a potential impact on branch accounts – it states “this may have caused a loss or gain at the branch which they can’t resolve themselves”, which is very clear. It shows that the explanation put to Mr Coyne “on instruction” is, in my judgment, completely wrong. It also shows that it has happened before, as “in the past we have just repaired these”. I would also note for completeness that this KEL is not dealt with at all by Mr Parker in the table accompanying his 1st witness statement. Two KELs from Ms Obeng are, namely CObeng1123 at number 24 in the table at E2/11/40; and obengc5933K at number 31 in the table at E2/11/43. However, this one is not.

383.

I accept Mr Godeseth’s evidence on this point, which is wholly consistent with the documents, in particular the PEAK and the KEL. The text of those documents shows that a message was intended to be written into the messagestore; the actual message used is nowhere available; the effects of all this clearly being an impact on branch accounts.

384.

A further PEAK was PC0175821, which was dated 22 February 2009. This also related to foreign currency transactions. This PEAK related to two problems, or a problem with two elements to it. It is at F/485/1 in the trial bundle, and it clearly and expressly refers to the same KEL, obengc3120K.

385.

The two aspects were explained by Mr Godeseth as a change using the transaction repair tool to “get the feed into POLFS correct” and also “a change to the messagestore to get the branch aligned.” The repair “had to go two ways” because the branch and the POLFS had to be aligned. If they were not, there would be a discrepancy in branch accounts. This showed the two problems, or two sides to the same problem. It was stated as follows in the PEAK:

“The first is where all five SC transactions missing core data as described in the above-mentioned KEL." This means obengc3120K, referred to at the beginning of the PEAK.

And the “Second is absence of equal but opposite 2 (ie settlement) lines. See PC0152014 for a similar problem and how problem was resolved."

“There are two sides to the problem relating to these txns. The first is where all five SC txns missing core data as described in the above mentioned KEL. Second is absence of equal but opposite [i.e. settlement] lines. See PC0152014 for a similar problem and how problem was resolved.

For the first problem, I have used the TRT [the transaction repair tool] to insert the missing data i.e. Region, Margin, Margin Product and EffectiveExRate.”

The reference to “txns” means transactions. PC0152014 is the very PEAK referred to in [376] above.

386.

Mr Godeseth agreed with both of those points in the PEAK as they were put to him by the claimants. Basically, half the data was missing. The transaction repair tool or TRT was used for the first problem.

387.

Numerous other PEAKs and some KELs were put to him with very specific details. Both Gareth Jenkins and Anne Chambers were involved in raising, or investigating, a great many of them and they are referred to in the text of both classes of documents. In all of them, Mr Godeseth’s answers to questions were, in my judgment, consistent with the entries within the documents, and also consistent with the claimants’ case on the Horizon Issues. In particular, those in the PEAKs dealing with foreign currency which I have explained above was clear, and settlement lines were missing, which Fujitsu sought to correct by injecting lines into both the messagestore and into POLFS.

388.

Finally, a series of questions were put to him demonstrating what are called APPSUP permissions. This is a very powerful permission (which really means level of access) and PEAK PC0208119, which runs from February 2011 onwards into 2012 and beyond (the final entry being in 2015), made the following relevant comment:

“As per the previous PEAK comments, the role 'APPSUP' is extremely powerful and should only be used under extreme circumstances and under MSC supervision. As such the Branch Database design was that 3rd line support users should be given the

'SSC' role, which is effectively read access, ie. 'select_any_table + select_catalogue'. SSC team members should only have to [access] BRSS for normal support investigations, unless the information has not replicated in time. SSC should only given the optional role 'APPSUP' temporarily (by Security Ops authorisation/emergency MSC) if required to make emergency data amendments in BRDB Live.

It is a security breach if any user write access is not audited on Branch Database, hence the emergency MSC for any APPSUP role activity must have session logs attached under the MSC. Host-Dev previously provided scripts, such as the Transaction Correction Tool, are written to run under the SSC role and also write to the audit logs.

SSC users created on BRDB should only have the SSC role, and the user creation script should be amended by Host-Dev to reflect this. A separate script giving/revoking emergency MSC access via APPSUP can be delivered, logging this to the hostaudit directory. In parallel Host-Dev should investigate any Host-Dev delivered script to ensure they are all executable by the SSC role. SSC should investigate any of their own scripts to ensure they have sufficient permissions under the SSC role, taking into account they should primarily perform their work on BRSS.

Any day to day scripts should not access BRDB directly. Any scripts needing more than the SSC role should be questioned, except those that would run under MSC APPSUP. Once the investigation is complete, all BRDB SSC users with APPSUP should have the role removed by ISD (via MSC) and ensure they do have the SSC role.

If anyone is in disagreement with the above course of action, then I'll set up a meeting with yourselves and Security when I'm down in BRA01 next week.”

(emphasis added)

389.

This entry was from Andy Beardmore, Senior Software and Solution Design Architect Application Services. The experts are agreed that the APPSUP role would, effectively, permit anyone who had that permission to do almost anything on Horizon. It was available to 3rd line support at SSC, the level at which Mr Roll was employed by Fujitsu. This PEAK further substantiates the evidence of Mr Roll and is consistent with it. APPSUP was described by Mr Parker as “the more technically correct name for a type of privileged access to the BRDB”. It is a very powerful permission.

390.

A later entry in the same PEAK states:

“The Business Impact has been updated:

1.

Cost: There is currently no "cost" to this issue. The users affected have more access than is required.

2.

Perceived Impact: The customer is not aware of this problem or change.

3.

Scope: No actual impact/incidents of problems relating to this issue have been experienced yet (and not expected)”.

(emphasis added)

The customer was the Post Office. The “issue” referred to was Fujitsu SSC users having had far greater access, namely the powerful APPSUP permissions, on more than the intended basis, which was supposed to be in extreme circumstances only and temporarily. This is shown in an entry for 1 February 2011 by Mark Wright who stated “….SSC users have the APPSUP role” and “When we created SSC users for BDB/BRS etc. we used “appsup” as that is what SSC have always been and what they migrated as on Horizon databases.” (emphasis added)

It is clear that all the members SSC had always APPSUP, but were not supposed to have this powerful role all the time.

391.

Anne Chambers said, in an entry on 1 February 2011:

“Unfortunately development write their scripts explicitly to use SSC. So I think we're stuck with it unless they deliver new scripts (which would not be a popular or quick option).

When we go off piste we use appsup. Can we have both??”

(emphasis added)

392.

Mr Godeseth said that by “off piste” he was very confident that she meant “having to fix a problem that was not catered for by a script that is available to these people” by which he meant SSC. It was clear that SSC were using APPSUP and wished to continue to do so. It was equally clear that all the members of SSC were not supposed to have it.

393.

The Post Office’s auditors for the year ended 27 March 2011 were Ernst & Young (“E&Y”) a very well-known global firm. In the Management Letter for that year, E&Y had identified the following:

“The main area we would encourage management focus on in the current year is improving the IT governance and control environment.

Within the IT environment our audit work has again identified weaknesses mainly relating to the control environment operated by POL’s third party IT suppliers.

Our key recommendations can be summarised into the following four areas:

(emphasis added)

394.

The “third party IT suppliers” referred to were Fujitsu. Whether it was the work performed by E&Y that had led to the decision at Fujitsu that the APPSUP permissions were not being sufficiently restricted or controlled, or something else, does not much matter. A company such as E&Y would not lightly refer to “again identified weaknesses” in the Management Letter for a particular year unless that conclusion had been reached after a high degree of professional and in depth investigation, of a type that even a lengthy adversarial trial such as this one cannot hope to replicate.

395.

Indeed, the Post Office’s solicitors confirmed in a letter in this litigation dated 11 February 2019 that Fujitsu SSC users who were identified as PERSON-POA UNIX (also sometimes referred to as POA Unix users) had a range of privileges and were granted the UNXADM role. This may be a descriptor for Unix (hence UNX) Administrator (hence ADM) but this does not matter. The UNXADM role contains, to quote from the letter of 11 February 2019 itself, the “DBA role” which is stated to be:

“This is an Oracle supplied role for use by database administrators (DBAs). Lots of privileges are granted to this role so users have the ability to update/delete/insert into any of the Branch database tables.”

(emphasis added)

396.

POA Unix users also are:

“Granted the ability to execute the following executable functions:

(a) OPS$SUPPORTTOOLUSER.PKG BRDB TXN_CORRECTION —framework to

allow the user to insert fully audited balancing records into a BRDB transaction table (made against node ID 99).”

397.

Both of these are powerful roles. Mr Godeseth was asked specifically about the DBA role explained in [395] above. His specific answers merit quotation verbatim:

“Q. You would agree that those people have the role which allows them privileges to update, delete, or insert into branch database tables whether they are using the correction tool or not?

A. Those people could log on to the database and do an awful lot of damage.

Q. And the only audit of that that we have prior to 2015 was log on and log off; that's correct, isn't it? A. Correct.

(emphasis added)

398.

Mr Coyne had given evidence that he had identified 2,175 occasions when it had been used. Mr Parker in his 3rd witness statement had said that figure was wrong, but also said:

“I have not examined the privileged user logs, but based on my experience my expectation is that these uses of APPSUP, or at least the vast majority, are for support work that does not involve changes to transaction data. I cannot recall any cases in which it has been used to change transaction data, but I cannot state unequivocally that there are no circumstances in which it has ever happened.”

399.

Mr Godeseth said in his written evidence that “Privileged users can, theoretically, inject, edit or delete transaction data in Horizon Online. As far as I am aware, this has never happened.” His evidence above about prior to 2015 makes it clear that he could not work out the number of times this may have happened in any event. Therefore the fact that he was not aware it had ever happened is neither here nor there, and does not assist the Post Office.

400.

In his re-examination Mr Godeseth was asked about an entry in an OCP in relation to a fix that stated UNIX user brdb. He said he did not know the detail, it meant logging on to the database, and the “UNIX user” referred to “would be one of the guys in Ireland”. These were not members of the SSC. When asked to clarify “who are the ‘guys in Ireland’ exactly?” he said “the people who support the hardware, so UNIX is an operating system so they work at a pretty low level on the systems” and by “low level” he meant “they have pretty powerful user rights but they are very much driven by process as to how they use them”. He also said there was complete and utter control on the process they go through before they did any of this type of activity, which in that re-examination example was "Logon to BRDB Node 1 as UNIX user

'brdb'."

401.

He was also cross-examined about some of the bugs, which appear in the Bug Table considered and partly agreed by the experts. The main admitted bug by the Post Office (in distinction to other bugs admitted by its expert, Dr Worden) is called the Callendar Square bug. Mr Godeseth gave evidence in his witness statement about this, partly in response to a statement that had been given by Mr McLachlan, who was not

in the event called by the claimants. However, Mr Godeseth’s evidence about Callendar Square on this was still called by the Post Office. It is a Legacy Horizon issue. Mr Godeseth had no first hand knowledge of this but had reviewed documents and spoken to Mr Jenkins.

402.

One of the KELs to which he was taken, with the reference JSimpkins338Q, related to problems with Riposte and what was sometimes called the Riposte lock or unlock problem. This was when an unexpected error occurred in Riposte while attempting to insert a message. The KEL shows it ran from 2002, with other events occurring in 2003, 2004 and 2005. The entry for September 2005 expressly states that “this problem is occurring every week, in one case at the same site on 2 consecutive weeks”. A PEAK was sent to development. Under “solution” in the KEL the following entry appears:

“SMC: If the event is seen at a multi-counter office during the working hours of the PO, or up to 18:00 on a weekday (in case they are balancing out-of-hours), RING THE OFFICE AND GET THEM TO REBOOT the eventing counter. If they are in the process of balancing, it is strongly advised that they reboot before continuing with balancing as they are at risk of producing an incorrect balance. Warn the PM that if transactions appear to be missing, they should not be re-entered - they will become visible after the counter has been rebooted.

If a reboot/Cleardesk does not resolve this problem, send the call over for further investigation - SSC can rebuild the messagestore on the affected counter.”

(Block capitals in original; emphasis by underlining added)

403.

The risk of a branch producing an incorrect balance as a result of this problem with Riposte is clearly recorded. An incorrect balance is another way of referring to discrepancies in branch accounts, a central issue in this litigation.

404.

The Callendar Square issue or consequences (now admitted by the Post Office to be a bug) were discovered in 2005 at the Callendar Square branch (hence the name). It is also sometimes called the Falkirk bug, but I shall refer to it as Callendar Square. It undoubtedly caused discrepancies in the accounts. The SPM in question had support from their Area Manager and the documents demonstrate the following, from the Area Intervention Manager Visit Log:

“This office had severe problems balancing on Wk 25, resulting in a shortage of £6,414.46. After checking various reports I am satisfied that the error is made up of:

£3,489.69 – Transfers

£2,870 – Giro Deposits

£54.52 – unidentified (however due to all the coming and going with re- keying entries, then this could come back as an error).

The Spmr claims that there was a Horizon software problem on 14.09.05 from 15.30 onwards. This was picked up when a member of staff noticed that a transaction, which had been taken by another member of staff, had not been entered onto the system, so therefore she put the transaction through. She checked at the time with her colleague who said that she thought she had put it through already however she accepted that she could have made a mistake. Following on from that, it was picked up that other giro business deposits that had been entered had not come up on the system, so they were re-keyed.

There was also a problem with transfers from one stock to another, in that they had doubled up. The Spmr made several telephone calls to the NBSC, telling them about his problems and he was advised to carry on with balancing and produce his Cash Account. Whilst doing this a warning came up, however the NBSC told the staff to continue to roll over. The result was that the office balanced £6,414.46 short.

The Spmr spoke to the Horizon Support Centre (ref E0509150123) who investigated and agreed that there had been a navigation problem that had now been rectified. They told the Spmr that they would report to NBSC that they had identified and rectified the problem and that the amount could be held in the suspense account. However, as part of the shortage relates to transfers, and no error notice will be issued, then the Suspense Account Team are not prepared to authorise the entry.

I telephoned The Suspense Account Team (Ann Wilde), who told me that checks could be made with Girobank after next Wednesday, and if that shows that duplications have been made, then they will authorise the amount to be moved to the suspense account, until the office receives an error notice. However, Ann stands by what she said about the transfer problems, and that they would not move this amount to The Suspense Account.

I went back to the Horizon Support Centre and spoke to a supervisor (Ken). He said that the call had now been closed as the problem had been rectified. I asked what was to happen about the resulting shortage and he referred me back to NBSC, who they said would do various checks. I then contacted NBSC, spoke to Rob Hughes and told him the story – he said he would put a call through to Service Support. No follow up was received from Service Support regarding this call.”

(emphasis added)

405.

The important points from this are as follows. This problem turned out to be an admitted bug. At the time, the SPM noticed the problem – and could only identify it as a “problem with Horizon” and reported it. The branch accounts were plainly affected by a sizeable amount, over £6,400. Horizon Support Centre, after investigation, agreed that there was a “navigation problem” – this is a problem with the Horizon system, and was caused by a bug. The Suspense Account Team would not authorise the shortfall in the branch accounts being moved to the Suspense Account; in other words, they were not prepared not to hold the SPM responsible. Mr Godeseth also accepted that “the genesis of it appears to go back to February 2003 and similar lock agent problems back in November 2000”, a point that he accepted as “that’s fair”. The history of the problem was therefore somewhat longer than something that just occurred in 2005 for the first time. During that five year period, branch accounts were usually being produced every four weeks.

406.

Mr Godeseth admitted that it was “a horrible position” for the SPM to have been in. He said “Service Support” was not Fujitsu, it was the Post Office. The story at [404] and [405] read like a mini-summary of many of the factual issues in the litigation generally, with the exception that this is an admitted bug by the Post Office.

407.

The subsequent documents in this respect show that the matter went on for some considerable time. In November 2005 the following is stated:

“1/ This problem is the root cause of the reconciliation error closed in PC0126042

2/ Presumably the root cause is deemed to be software not hardware

3/ The Postmaster has a workaround in place which is not to duplicate transactions (e.g. Transfer In) just because the original attempts were successful but not showing on all nodes

4/ POA CS MSU have a workaround in place which is that if 3/ above is not followed & PC0126042 reoccurs, a BIMS will be issued advising POL to issue a Transaction Correction

5/ There is no SLT for software fixes as they are delivered based upon the priority or severity of the issue and could remain open until both businesses decide a fix is necessary or the work around is adequate.”

This shows that the entry in the PEAK identifies this as a software problem; that is effectively now accepted, but this entry from 2005 did not lead to any public acceptance of a bug in Legacy Horizon either then, shortly afterwards, or for some years after that.

408.

The acronym SLT means Service Level Target. What this entry means is that this problem had occurred before, and was known to have occurred before; there was something called “a workaround” in place, but that required the SPM physically deciding not to duplicate transactions if the original transaction did not show up as successful on all the nodes; if that were not done, TCs would be required; both Fujitsu and the Post Office had to decide a software fix was required in order for this to be corrected by fixing the software. No such decision had been taken at this point.

409.

At Callendar Square itself, in January 2006 the SPM phoned one of his superiors. The record of that call in the log states the following:

“Issue Raised: PM WANTS TO SPEAK TO SANDRA MCKAY URGENTLY REGARDING SERIOUS PROBLEMS WITH HORIZON IT IS REPEAT PERFORMANCE AS TO WHAT WAS HAPPENING BEFORE AND HSH

FAILING TO PICK IT UP

Response by SPM: Telephoned the office and Allan said that he was having problems again with transfers. He has contacted the Horizon helpdesk who have subsequently come back to him to say that there is no system problem and that he should contact

NBSC. He did this and from what I can understand the NBSC have told him that he is

trying to balance on two different terminals. Allan disputes this and is adamant that there is a system error.”

(Block capitals in original; emphasis by underlining added)

410.

The statement that “there is no system problem” which is told to the SPM evidently conflicts with what the PEAK at [407] had recorded, that “presumably the root cause” was the software.

411.

Emails in February 2006 to and from Anne Chambers stated expressly the following:

“…. I notice that in the early guise of this problem in the call it states the PM as Female:

Wed 12 October 2005 17:39 by UK956078 / HSH1 Saved: Wed 12 October 2005 17:39 Pm was trying to transfer £2490 from node 7 onto node 2. She states that she has accepted the transfer on node 2 but the system is not showing this. On node 7 it is showing pending transfer but it is not showing on node 2. It appears on her transfer sheet as completed.

At the bottom of this email re a magical £43k appearing and disappearing the PM is Male He reports:

You may recall that in September the above office had major problems with their Horizon system relating to transfers between stock units.

The Spmr has reported that he is again experiencing problems with transfers, (05.01.06) which resulted in a loss of around £43k which has subsequently rectified itself. I know that the Spmr has reported this to Horizon Support, who have come back to him stating that they cannot find any problem.

Clearly the Spmr is concerned as we have just spent a number of months trying to sort out the first instance and he doesn't want a repeat performance. He is convinced that there is something wrong with his Horizon kit. I would be grateful if you could investigate this and give him any support that you can. I'm due to visit the office tomorrow to have a look at his paperwork and discuss the situation with him.

So apologies for the long windedness but I have been given this by Liz as a problem so:

1.

Is there a problem at this branch? is it Horizon kit or is there an issue with staff there?

2.

If there is an issue is this S90 release the cure? how confident are you/we it will fix the problem?

3.

S90 counter release due week 4th March. Getting Sarah to check if this site is in the pilot 24th or just part of the general release 4th March.

Appreciate your comments please.”

(emphasis added)

412. In the answer to this question posed in February 2006, Anne Chambers stated the following:

“Haven't looked at the recent evidence, but I know in the past this site had hit this Riposte lock problem 2 or 3 times within a few weeks. This problem has been around for years and affects a number of sites most weeks, and finally Escher say they have done something about it. I am interested in whether they really have fixed it which it why I left the call open - to remind me to check over the whole estate once S90 is live - call me cynical but I do not just accept a 3rd party's word that they have fixed something!

What I never got to the bottom of, having usually had more pressing things to do, was why this outlet was particularly prone to the problem. Possibly because they follow some particular procedure/sequence which makes it more likely to happen? This could still be worth investigating, especially if they have continuing problems, but I don't think it is worthwhile until we know the S90 position.

Please note that KELs tell SMC that they must contact sites and warn them of balancing problems if they notice the event storms caused by the held lock, and advise them to reboot the affected counter before continuing with the balance. Unfortunately in practice it seems to take SMC several hours to notice these storms by which time the damage may have been done.”

(emphasis added)

413.

This shows the following important points. At least Anne Chambers in early 2006, and all those with whom she was corresponding, knew that this problem – now admitted to be a software bug – had been around “for years”. Horizon Support were telling the SPM, whose branch accounts were affected by discrepancies, that “they cannot find any problem”. The SMC – the part within Fujitsu responsible for providing corrective action for the “event storms” – would not always notice these had occurred in time and by then “the damage may have been done”. I find that by “the damage” this can only mean impact upon branch accounts.

414.

Mr Godeseth gave evidence about S90, the software release intended to remedy this, which was distributed in March 2006. This was 6 years after Legacy Horizon became in use. His evidence was that this release was one year before what he called “the relevant period” for Ms Misra’s branch, one of the SPMs convicted of criminal offences, which he was asked about. He also gave evidence that the Callendar Square bug affected 30 branches, and had caused mismatches in the branch accounts at 19 (corrected in supplementary evidence in chief from 20) of them. He said in his witness statement that this information had come from Matthew Lenton, Fujitsu’s Post Office Account Document Manager and that “Fujitsu has established this for the purposes of this statement using the event logs described above.” He accepted that this part of his witness statement was not correct. The figures that he had provided had not come from an exercise specifically for the purposes of his evidence, or for the Horizon Issues trial. It had actually come from “the spreadsheet [which] is something that was done at the time, as I understand it by Anne Chambers, to actually investigate what had happened when we had had these locks.”

415.

This was an important correction by Mr Godeseth, for reasons that I will now explain.

416.

It is a matter of public record that Anne Chambers gave evidence before the High Court in the case of Post Office Ltd v Castleton [2007] EWHC 5 (QB) before HHJ Havery QC. Her evidence in that case is summarised at [23] of that judgment. Callendar Square is referred to, but as a single branch affected. The reason that I have referred to this case is because it was specifically referred to by Mr Jenkins in an email of 8 March 2010 which was referred to by Mr Godeseth in paragraph 14 of his witness statement. Mr Godeseth said that he agreed with that email, which stated:

“I’ve been asked about the issue at Callender Square, Falkirk (I’m not quite sure about the spelling) that came up at the Castleton Trial.

I thought I’d better keep you in the loop on this.

I’ve now dug back into the archives to provide the following summary:

1.

The problem occurred when transferring Cash or Stock between Stock Units. Note that West Byfleet does operate multiple Stock Units so the issue could have occurred.

2.

It manifests itself by the Receiving Stock Unit not being able to “see” the Transfer made by the “sending” Stock Unit and is compounded by attempting to make a further transfer. Note that such transactions usually reappear the next day.

3.

It is clearly visible to the User as a “Receipts and Payments mismatch” at the time that one of the Stock Units is Balanced. This usually results in the Branch raising a call. There are no such calls in Andy Dunks’ Witness Statement which summarises the calls raised by West Byfleet. Also this can be checked on any Balance Reports or Branch Trading Statements that are available from the Branch which should show that Receipts and Payments do match and that the Trading Position is zero.

4.

The problem is also visible when looking at system events associated with the Branch. The System events from 30/06/2005 to 31/12/2009 for West Byfleet have been checked and no such events have been found.

5.

The problem was fixed in the S90 Release which went live in March 2006 and so would not have been relevant at the time of the detailed Transaction Logs obtained for West Byfleet between December 2006 and December 2007.

Therefore I can conclude that the problems identified in Calendar Square, Falkirk are not relevant to West Byfleet.”

417.

This email does not provide anywhere near the same degree of information about the Callendar Square bug as was available at the time of the Horizon Issues trial, or as recorded in the PEAKs above. There has obviously been further investigation at Fujitsu into this specific issue since then, as made clear by Mr Godeseth’s evidence about the information coming from Mr Lenton and Ms Anne Chambers. The email does not use the term “software bug”. It does not refer to the fact that it is accepted by

Fujitsu, following an investigation, that the admitted bug had affected at least 30 branches. Nor does it refer to the fact that the Callendar Square bug caused mismatches at 19 of those 30 branches.

418.

The spreadsheet to which Mr Godeseth referred as having been “prepared at the time” by Anne Chambers was also put to him. It included textual notes and a list of call references and branches affected by approximate date, which were all in 2004 and 2005. The notes were:

“NB many other branches had multiple events, preventing replication, but these are the majority of those which came to Peak, having either reported a problem or it caused a reconciliation report entry.

From Sept 2005, cash accounts were replaced by branch trading statements and the TPSC256 report was no longer populated. I can't remember how we then knew about receipts and payments mismatches and if we would have picked up on further issues.”

(emphasis added)

419.

These notes make clear that there were other branches potentially affected, and the list of 30 branches compiled in the spreadsheet were “the majority”. It does not state that it is an exhaustive list and the notes suggest it is not. The summary for each branch listed gives a variety of consequences or effects. They include:

1.

Receipts and Payments Mismatch.

2.

Receipts and Payments Error.

3.

Host Generated Cash Account Line.

4.

Unable to roll Stock Units.

5.

Reconciliation Report.

6.

Slow running counter.

7.

Pension and Allowance Report printouts.

8.

Screen freezes.

9.

Problems with declarations.

10.

Unable to create AA.

11.

Loss due to double transfer.

12.

Transfers not showing on nodes.

13.

CAC Lines not matching.

420.

The spreadsheet was disclosed to the claimants on 27 February 2019, or very shortly before the Horizon Issues trial, and the properties shown on the electronic document identify Anne Chambers as the author, with a date of 22 December 2015.

421.

Mr Godeseth also, entirely accurately in my judgment, accepted that the Callendar Square bug had been present for a considerable amount of time prior to 2005. He said “I would agree that the underlying bug had been there for a considerable time, probably since the Horizon went in”. This is made clear by, amongst other documents, Anne Chambers’ entry in February 2006 stating that the problem had “been around for years”. In another of the many PEAKs that dealt with this issue, the entry for 22 September 2005 states “Note - a few of these errors seem to occur every week at different sites.”

422.

Mr Godeseth was taken to the contents of two letters from the Post Office’s solicitors to the claimants’ solicitors. The first was dated 28 July 2016. This was the Post Office’s solicitors’ substantive response to the claimants’ letter of claim, is very lengthy (as was the letter of claim itself) and ran to 99 pages. The second letter was dated 11 January 2019. He agreed that certain important elements of the paragraphs in those letters explaining Callendar Square were incorrect. They were:

1.

That the bug was only discovered in 2005.

2.

That the bug had only affected one branch. The second letter stated in very clear terms that “The Falkirk/Callendar Square issue was only known to have affected that one branch.”

423.

Not only were both those statements in the Post Office’s solicitors’ correspondence completely wrong, but the numerous documents at [398] to [421] above demonstrate that there were a great number of references, including in both KELs and PEAKs (documents of a type that had been at one stage claimed to be wholly irrelevant to the issues in this litigation) to the direct contrary.

424.

Mr Godeseth’s witness statement about this, prior to his cross-examination, presented a very different picture to the one that eventually emerged. Indeed, I would go further, and I find that his witness statements omitted some very important headline points in respect of the Callendar Square bug, presented a chronology very different to the real one, and had the effect (whether intended or accidental) of giving a misleading impression of the Callendar Square bug and its impact.

425.

I find that the headline points omitted by Mr Godeseth are as follows:

1.

The Callendar Square bug had existed “probably” (Mr Godeseth’s word in crossexamination) since the inception of Horizon in 2000. There were numerous incidents of it occurring prior to 2005.

2.

By February 2006, Anne Chambers and others at Fujitsu knew that this bug “had been around for years”.

3.

It had an impact “most weeks”. Preparing a software fix was not seen as a priority.

4.

The bug had directly impacted the branch accounts of at least 19 branches, and possibly more, as the Anne Chambers’ spreadsheet listed the 30 branches as “these are the majority”.

5.

There were numerous effects of this bug, and at least 13 different ones identified in the spreadsheet.

6.

Even in 2006, Horizon Support was telling at least one SPM that they “could not find any problem” with Horizon when the bug’s impacts were reported to it, and this was recorded in documents linked by Fujitsu directly with the Callendar Square bug.

7.

Fujitsu had not established the total number of branches affected by consulting the events logs for the purposes of preparing Mr Godeseth’s witness statement, or otherwise. To represent the spreadsheet prepared by Anne Chambers as being the totality of branches affected by the bug is to ignore the text of the notes.

426.

Finally on this point, the Post Office’s solicitors’ letters were obviously not factually inaccurate. Nor can the Post Office’s own legal team have known about the Anne Chambers’ spreadsheet until very early 2019, otherwise disclosure of such an important document would surely not have been given only on 27 February 2019, only seven working days before the Horizon Issues trial began. I do not know why the Anne Chambers’ spreadsheet of 22 December 2015 was only disclosed to the claimants a mere 7 working days before the start of the Horizon Issues trial, and it is not necessary to speculate.

427.

Although the Callendar Square bug in Legacy Horizon could, amongst other effects, cause a receipts and payments mismatch, there is another bug in Horizon Online that also had the same effect. This was called the Receipts and Payments Mismatch bug and is the first entry in the Bug Table, Bug 1.

428.

Mr Godeseth gave evidence about the Receipts and Payments Mismatch bug. This was a problem at the cache. His information came both from Mr Jenkins and his own investigations. He did not know about it at the time in 2010, and he did not in his witness statement refer to what is in my judgment a very important document concerning this bug, namely the Receipts/Payments Mismatch issue notes dated in the trial bundle index 17 October 2012, although it is likely to be a 2010 document as an associated document from Mr Jenkins (trial bundle reference F/1000/1 and F/1001/1) is dated 29 September 2010. I will reproduce certain parts of the issue notes, as I consider them to be highly relevant to the Horizon Issues. There were 10 attendees, including Mr Jenkins. The document was marked Commercial in Confidence and is, according to the trial bundle index, a memo of a meeting. Another document with the same date is headed “Correcting Accounts for “lost” Discrepancies” and states “any branch encountering the problem will have corrupted accounts”. The issues notes document stated:

“What is the issue?

Discrepancies showing at the Horizon counter disappear when the branch follows certain process steps, but will still show within the back end branch account. This is currently impacting circa 40 Branches since migration onto Horizon Online, with an overall cash value of circa £20k loss. This issue will only occur if a branch cancels the completion of the trading period, but within the same session continues to roll into a new balance period.

At this time we have not communicated with branches affected and we do not believe they are exploiting this bug intentionally. (emphasis added)

The problem occurs as part of the process when moving discrepancies on the Horizon System into Local Suspense.

When Discrepancies are found during Stock Unit rollover into a new Transaction Period, then the User is asked if the discrepancy should be moved to Local Suspense. If the branch presses cancel at this point the Discrepancy is zeroed on the Horizon System.

Note at this point nothing into feeds POLSAP and Credence, so in effect the POLSAP and Credence shows the discrepancy whereas the Horizon system in the branch doesn't. So the branch will then believe they have balanced.

If at the next screen the rollover is completely cancelled, then no harm is done. However if the Rollover is re-attempted at this point, the rollover will continue without any discrepancy meaning Horizon doesn't match POLSAP or Credence.

This has the following consequences:

There will be a Receipts and Payment mismatch corresponding to the value of Discrepancies that were "lost"

Note the Branch will not get a prompt from the system to say there is Receipts and Payment mismatch, therefore the branch will believe they have balanced correctly.

When the Branch begins the new Branch Trading period the discrepancies will show at Zero, however the Receipts and Payment mismatch will carry over into the next period.

Impact

The branch has appeared to have balanced, whereas in fact they could have a loss or a gain.

Our accounting systems will be out of sync with what is recorded at the branch

If widely known could cause a loss of confident in the Horizon System by branches • Potential impact upon ongoing legal cases where branches are disputing the integrity of Horizon Data

It could provide branches ammunition to blame Horizon for future discrepancies.

Identifying the issue and forward resolution

The Receipts and Payment mismatch will result in an error code being generated which will allow Fujitsu to isolate branches affected this by this problem, although this is not seen by the branches. We have asked Fujitsu why it has taken so long to react to and escalate an issue which began in May. They will provide feedback in due course.

Fujitsu are writing a code fix which stop the discrepancy disappearing from Horizon in the future. They are aiming to deliver this into test week commencing 4th October. With live proving at the model office week commencing 11th October. With full roll out to the network completed by the 21st of October. We have explored moving this forward and this is the earliest it can be released into live.

The code fix will on stop the issue occurring in the future, but it will not fix any current mismatch at branch.

Proposal for affected Branches

There are three potential solutions to apply to the impacted branches, the groups recommendation is that solution two should be progressed.

SOLUTION ONE - Alter the Horizon Branch figure at the counter to show the discrepancy. Fujitsu would have to manually write an entry value to the local branch account.

IMPACT - When the branch comes to complete next Trading Period they would have a discrepancy, which they would have to bring to account.

RISK- This has significant data integrity concerns and could lead to questions of "tampering" with the branch system and could generate questions around how the discrepancy was caused. This solution could have moral implications of Post Office changing branch data without informing the branch.

SOLUTION TWO - P&BA will journal values from the discrepancy account into the Customer Account and recover/refund via normal processes. This will need to be supported by an approved POL communication. Unlike the branch "POLSAP" remains in balance albeit with an account (discrepancies) that should be cleared. IMPACT - Post Office will be required to explain the reason for a debt recovery/ refund even though there is no discrepancy at the branch.

RISK - Could potentially highlight to branches that Horizon can lose data.

SOLUTION THREE - It is decided not to correct the data in the branches (ie Post

Office would prefer to write off the "lost"

IMPACT - Post office must absorb circa £20K loss

RISK - Huge moral implications to the integrity of the business, as there are agents that were potentially due a cash gain on their system.”

(bold present in original)

429.

I find this to be a most disturbing document in the context of this group litigation. It is a 2010 document and issues between the Post Office and many SPMs concerning the accuracy of Horizon had, for Legacy Horizon, gone on for a decade (2000 to 2010) and these continued under Horizon Online (introduced in 2010). Under “Impact”, some of the bullet points incorporate a summary of these issues.

“• The branch has appeared to have balanced, whereas in fact they could have a loss or a gain.

Our accounting systems [ie Horizon or the Post Office’s] will be out of sync with what is recorded at the branch

If widely known could cause a loss of confident in the Horizon System by branches • Potential impact upon ongoing legal cases where branches are disputing the integrity of Horizon Data

It could provide branches ammunition to blame Horizon for future discrepancies.” (emphasis added)

430.

The attendees at this meeting included at least one member of Post Office (rather than Fujitsu) personnel, Andrew Winn of POL Finance. There were obviously legal cases going on at the time, hence the reference in the underlined bullet point to “ongoing legal cases”. If these were criminal cases, the Post Office would be the prosecuting authority, with certain important duties. If these were civil cases, the Post Office would be a party with disclosure obligations. An affected branch would believe it had balanced its accounts correctly; it would not have done so. There is an evident concern amongst those at the meeting which is recorded in this document that this issue should not become “widely known” in order to avoid causing “a loss of confidence in the Horizon System”. Fujitsu do not seem to have been particularly prompt in either identifying the problem or reacting to it.

431.

Of the three solutions considered, the one that was adopted was issuing a debt recovery/refund even though there would be no discrepancy at the branch. Given one of the Common Issues was the status of a Branch Trading Statement, which the Post Office argued should legally have the status of a settled account with the SPM as agent, I note the suggestion adopted with more than passing interest. The solution explains “even though there is no discrepancy at the branch”. This means the Branch Trading Statement would be correct, or in balance. The discrepancy would not be in the Branch Trading Statement, it would be in POLSAP or Credence. That this is correct is shown in an associated document from Mr Jenkins (trial bundle reference F/777/2) where he stated that “the data used for the BTS will also have a zero value for Discrepancies at the end of the period.” BTS means Branch Trading Statement. A similar entry is in the document at F/1000/1 which states at /2 “Note that if the bug was not present, then the Discrepancy would have been transferred to Local Suspense and that would have been cleared, so there are a number of things wrong with the BTS.” (emphasis added). In other words, the Post Office itself was not considering the BTS as having the status of a settled account, based on these entries.

432.

Further, an associated document entitled “Receipts and Payments Mismatch” states:

“1. The purpose of this note is to document a request that we have had from Post Office in terms of presenting details of what happened as a result of a bug in HNG-X

in September 2010 which caused a Receipts and Payments mismatch and also resulted in Discrepancies being lost.

The background to this is the fact that there was a BBC documentary broadcast on Monday 7 th February 2011 reporting on postmasters being unhappy about being pursued for losses by Postmasters on Horizon.

It should be noted that the issues described here relates to HNG-X (Horizon Online) and that the implementation of the accounting mechanisms in the two systems is totally different (but they do produce the same reports and support the same business process).”

(emphasis added)

433.

The two references – one to “ongoing legal cases”, the second to a BBC documentary – show that there was a distinct sensitivity within both the Post Office and Fujitsu about keeping this information to themselves in order to avoid a “loss of confidence” in Horizon and the integrity of its data. A less complimentary (though accurate) way of putting it would be to enable the Post Office to continue to assert the integrity of Horizon, and avoid publicly acknowledging the presence of a software bug. The solution adopted was the issuing of TCs to the relevant branches. There was no publicity given to the SPMs at the time about the presence of this software bug in Horizon. An entry at F/1000/3 states:

“It should be noted that as Discrepancies are normally Losses, then a Lost Discrepancy would normally work in the Branches favour and so there is no incentive for the branch to report the problem. Also if we do amend the data to re-introduce the Discrepancy, this will need to be carefully communicated to the Branches to avoid questions about the system integrity.”

(emphasis added)

434.

Mr Godeseth was asked about the lack of communication of this.

“Q. Are you surprised that this had not been communicated so people would just be warned about it?

A. There was obviously a fear that subpostmasters may be looking to exploit this because it gave -- there was a fear that people could see this as a way of defrauding the Post Office.

Q. So concealing it from SPMs who were honest was justified because of the expectation of dishonesty of subpostmasters in the network, in a nutshell?

A. In my view, this was a decision made by Post Office on how to manage this particular bug. You could interpret it the way that you have put it. Q. Do you agree with the way that I have put it to you?

A. I think I'm agnostic. I can see the -- I can see a rationale for not broadcasting this, but equally, if the objective is to be totally open and honest and take the risk of causing more chaos in the network then yes, I would have to agree with that.”

435.

Indeed, the approach of the Post Office to notification of the existence of bugs to SPMs is an illuminating one. The Post Office communicates with SPMs generally in a number of ways. Methods that have been referred to in both of the Common Issues trial and also the Horizon Issues trial include “Branch News”, a type of newspaper that was used, and also “Memo View”, a pop up message that appears on Horizon terminals. During the preparation of this judgment, in view of the fact that judgments are required to be factually accurate, because dates can sometimes become lost in the midst of huge trial bundles and very detailed submissions, and also because the simplest way to arrive at a date when something has been done is (sometimes) simply to ask the parties for a date or a reference, on 7 October 2019 the parties were jointly asked the following:

“For each of the Dalmellington and Callendar Square bugs, could the parties please indicate (by means of an agreed fact but only if that is possible, if not each party’s contended for answer) their answers to the following two questions:

1.

The date when the existence of each of these was communicated by the Post Office to SPMs;

2.

A trial bundle reference to a contemporaneous document, email or communication from the Post Office demonstrating question 1.”

436.

The parties could not agree on a date when the existence of these two bugs – Callendar Square being an acknowledged bug in the Bug Table dealt with below (from the 2nd Joint Statement of the two experts) and Dalmellington, which is now accepted to be a bug – was communicated to SPMs. The claimants’ answer to the two questions was the same for each bug, namely that there was “no date in any contemporaneous document from the Post Office in the trial bundle showing that such communication in fact took place and identifying the actual content of any such communication.” The Post Office’s answer was as follows:

437.

For Callendar Square: “It has never been Post Office’s case that, once it was discovered, the existence of the Callendar Square bug was communicated by Post Office to all SPMs or communicated to SPMs who had been affected. Post Office's case is that it was unnecessary and would have been inappropriate to do so.” A number of references and further explanation then followed, stating that when “an SPM was affected by the bug, this was detected and corrected in the ordinary course” and also that “In some cases, SPMs who were affected by the Callendar Square bug may have been made aware that there was a problem in Horizon which had caused an error in their accounts which required correction.”

438.

For Dalmellington: “It has never been Post Office’s case that, once it was discovered, the existence of the Dalmellington bug was communicated by Post Office to all SPMs or communicated to the SPMs who had been affected. Post Office's position is that it was unnecessary and would have been inappropriate to do so.” The Post Office also stated that “when an SPM was affected by the bug this was detected and corrected in the ordinary course” and that the Dalmellington bug “was prevented, by effective counter-measures, from causing even a single impact on branch accounts”.

439.

The Post Office also stated that there were real disadvantages to notifying SPMs of this. The Post Office relied upon passages in Dr Worden’s report that summarise reasons why users of an IT system should not be given information about parts of that system which they did not encounter in their daily work; that there was “no point in trying to educate all the users in details and terminology of the system which will never concern them”; that the best thing to do is try and fix a bug or defect, rather than “create some new error message to the users”; that “automated messages from the system are only of limited help to users”; and other points in similar vein. The Post Office also provided documentary references to instances where “an SPM might have been made aware”, although these documents were not deployed in the evidence.

440.

I will reproduce two entries from one of these documents, a PEAK with reference PC0103864. This shows that on 3 June 2004 an SPM reported “that he had a problem

with some transfers yesterday, he was transferring stock and cash between the aa main stock unit and the bb shared stock unit and although only one transaction shows for the transfer out the transactions were transferred into the bb stock unit twice giving the pm a discrepancy.”

441.

The discrepancy in the cash account was £22,290 in total (shown in the entry for 8 June 2004) although the reconciliation error reported by the Host was £44,580. This was finally resolved on 5 August 2004 with the issue of an error notice. There is nothing to suggest that the SPM was told that this had been caused by a software bug.

442.

The Post Office’s approach to this, in my judgment, entirely misses the point. In my judgment, the above passages are simply extraordinary. Two of the documents above are dated 2010, some 9 years ago. The PEAK is from 2004, 15 years ago. Their contents support the claimants’ case on the Horizon Issues. Fujitsu knew, to take Callendar Square as an example, that this bug existed in Horizon. They knew that it had affected branch accounts. It was not, as the Post Office puts it “unnecessary and inappropriate” to notify SPMs of this. I have listed the points on this bug at [425] above omitted by Mr Godeseth from his written evidence. Those same points all lead to the same conclusion in my judgment, namely that the Post Office ought to have notified, at the very least, all those SPMs whose branch accounts had been impacted by this bug that this had occurred, and that it had occurred as a result of a software bug. The fact that the integrity of Horizon data was a live issue at this time should not have influenced the decision to notify SPMs of a software bug. Further, the Post Office’s explanation in its submissions that SPMs had their accounts “corrected in the ordinary course” is not a suitable phrase, unless by “ordinary course” one means keeping the cause or reason for the correction secret and therefore hidden from the other party in the accounting transaction, namely the SPM. Also, one is not educating users in the details and terminology of the system (as suggested by Dr Worden) if one informs them that there is a software bug in the system and its symptoms are as follows. This is relevant to Horizon Issue 2, namely whether the Horizon IT system itself alerted SPMs of such bugs, errors or defects as described in Issue 1 above and if so how, and also Horizon Issue 9, which concerns transaction data and reporting functions available to SPMs. It is also relevant to Dr Worden’s consideration of countermeasures, which he considers includes vigilance by users (which means SPMs).

443.

Mr Godeseth gave express evidence in his witness statement that the Receipts and Payments mismatch bug occurred in September 2010. That date too was factually incorrect. The issue notes refer to Fujitsu knowing about it far earlier, and Mr Godeseth accepted he had seen this document before his cross-examination. That document even records Fujitsu being taken to task by the Post Office about how long it had taken to react, as in “We have asked Fujitsu why it has taken so long to react to and escalate an issue which began in May. They will provide feedback in due course.” Not just the impression, but the express text in Mr Godeseth’s witness statement, was to the effect that the bug was discovered in September 2010 and almost immediately dealt with. That was far from the case, and that written evidence was simply wrong.

444.

Mr Godeseth gave evidence about another bug, called the Local Suspense Bug, also in Horizon Online. His witness statement quoted almost word for word from a contemporaneous document prepared by Mr Jenkins. Almost all his information on

this bug had come from Mr Jenkins. The bug caused entries from local suspense accounts in 2010 to be reproduced in Horizon in two successive years. What was supposed to be only temporary data was retained, not deleted as it ought to have been, and the system then used it again. It affected certain tables in the branch database and the archiving strategy of deleted Stock Units. This affected 15 branches. The effects of the bug were brought to the attention of the Post Office by two SPMs “with the largest discrepancies” and Fujitsu became aware of it in January 2013. However, the note by Mr Jenkins of 15 May 2013 summarising it states that the problem was known about far earlier than that:

In April 2011 a problem was found with the archiving strategy related to Stock Units that have been deleted in a Branch. A consequence of this is that some changes were made to the archiving strategy on 3rd July 2011. An unintended consequence of this change was that any Branch that deleted a Stock Unit at the end of 2010 which had local suspense transaction in that Stock Unit before it was deleted were left in the table used for constructing the BTS. This meant that as Trading Periods cycle around each year, these BTS records became visible in 2011 when the same Trading Period was reached.

The effect of these old records was that after the BTS was produced an incorrect figure was generated for the Opening Balance of the Local Suspense Account for the following period. This amount corresponded to the value of the historical record. These orphaned records were created between 16th November 2010 and 9th December 2010.

When the next Trading Period was balanced, then this incorrect Opening Figure would result in the total value for Local Suspense being calculated incorrectly and the SPMR being asked to make good an incorrect amount. It is at this point that transactions would be generated into the audit trail reflecting the fact that the SPMR had cleared the Local Suspense account for an incorrect amount. The audit trail operated correctly in the sense that it accurately recorded the transaction on the system.

This problem was not reported to Fujitsu in 2011/12 and only affected a small number of Branches and only for a single Trading Period. However the two branches with the largest discrepancies did report the issue to Post Office Ltd who could see the impact of the problem in their back end system and wrote off the loss or gain for the branch but did not ask Fujitsu to investigate further.

At the same Trading Period in 2012/13, the problem re-occurred and this time one of the affect Branches reported the problem to Fujitsu on 25th February 2013 (PEAK 223870) resulting in a detailed analysis of this issue and finding the orphaned BTS records. The root cause was determined by 28th February 2013 and a preliminary report was sent to Post Office Ltd. A further update was sent on 14th March 2013 with a full analysis of the issue and all the affected branches.”

(emphasis added)

445.

Another software bug which Mr Godeseth gave evidence about was called the Dalmellington Bug. This is also known as the Branch Outreach Issue or Bug, as it affected what are called Outreach branches, which are those that have a core branch

(such as a rural or small branch post office) and an outreach branch, such as a mobile post office van, or village hall somewhere else (where post office services would be provided, say, one day a week). This “outreach” element is used, particularly in very rural areas, to provide post office services to remote communities. A SPM needs to scan or record a pouch from one branch (the core) into the outreach branch (the van or village hall) to record that the contents of the pouch are not in the core branch, but are in the outreach branch instead. This is a branch to branch cash remittance, if the pouch contains cash. Mr Godeseth had no first hand knowledge of this bug. As the claimants put it in opening submissions, this is a useful way of examining certain features in Horizon as the same person, the SPM, controls “both ends” of the transactions.

446.

The Dalmellington Bug related to Horizon Online and actually included two potentially separate issues, and the combination of these. One related to what is called Forced Log Out, when the Post Log On script was not correctly closed down and was left on the stack of incomplete processes. The other related to the way the Pouch Delivery script operated; because the Post Log On script had not correctly closed down meaning the stack was not empty, the Pouch Delivery script thought it had not finished and attempted to repeat the last part of the script. This had the effect of recording the remittance transactions and printing of receipts. This would lead to duplicate pouch IDs. Pouches have bar codes that are scanned and each is supposed to have a unique ID. The duplicate pouch ID would have a value attached to it – the sum in the original pouch. It would however be recorded twice as a result of this.

447.

What happened at Dalmellington was the SPM scanned £8,000 from her core branch into her outreach branch. She was lucky in this sense, in my judgment, in that she “controlled” both ends of the transaction. The transfer replicated four times. She had Horizon receipts in her outreach branch of £32,000 (4 times £8,000) and therefore a discrepancy of £24,000, as she had only transferred £8,000 in cash out of her core branch into it. This was investigated by SSC and she was issued with a TC a few weeks after it had occurred. This was caused by a software bug, and the bug is now admitted by the Post Office, although it describes it as having “transient impact”. That simply means it was corrected.

448.

When investigating this Fujitsu found 112 occurrences affecting 88 different branches in the previous 5 years. They were as follows, taken from an internal Fujitsu presentation dealing with this:

Feb 2010 to Jan 2011 65 incidents

2011

6 incidents

2012

9 incidents

2013

7 incidents

2014

9 incidents

2015

16 incidents

449.

The same document showed fixes applied in April 2010, January 2011 and January 2016. It also showed only one call to SSC at Fujitsu in 2015, and none for any of the years 2010 to 2014. The presentation also showed that some branches that were affected were so-called “mediation branches” – in other words the SPMs would have been in the Second Sight mediation scheme that was brought to an end – but that none of the dates when those branches were affected matched the “mediation dates”, which means the periods for which ARQ data was to be used in the mediations to analyse exactly what had happened at each branch. There is nothing in the Fujitsu document used by Mr Godeseth for his evidence, to suggest that this analysis led to each branch being told of these software bug occurrences outside the mediation date range, and the items were usually corrected by means of TCs.

450.

Mr Godeseth said that “Unfortunately this particular error is not subject to receipts and payments problems because it could be -- it could be the user doing something twice, it's -- the bug had the effect of making it look as though a user was simply doing something multiple times.” (emphasis added) He said that “the issue was not raised with Fujitsu until 2013”, it was put to him it was 2015 (based on the slides) but regardless of which of these dates is right, the Fujitsu slides show only one call to Fujitsu in 2015, none for the other years, although the reference to fixes in 2010 and 2011 suggests it was reported to Fujitsu earlier. If those fixes were intended to correct the bug they cannot have worked. Fujitsu maintained that they investigated and found all occurrences, and the Post Office maintained that they had corrected any financial impact of this bug. Whether this is correct or not in all cases, the evidence demonstrated that the effects of this bug were experienced for the period from 2010 until 2016.

451.

In his re-examination, some of which I have already referred to in the context of UNIX user and privileges above, Mr Godeseth also said that having seen the OCPs and OCRs in cross-examination, he accepted data had been deleted but did not consider it to be transaction data. Initially he described it as “operational data” but said that was too wide a term, and:

“There possibly ought to be a different definition for it because equally data in the database which is telling you which stock unit, you know, what -- which TP [ie trading period] you're in, whether a stock unit is in use, all of those are operational data which do not go down and impact the transaction data, so for me, the definition of transaction data -- I should be able to get from one opening -- one set of opening figures to the end opening figures by going back to the transaction data in its absolutely raw form.”

452.

He said it was what one went through to bring different totals together “to get to your end position in a trading statement.” Regardless therefore of whether the data which he accepted was being deleted is, or is not, properly described as “transaction data”, it would feed into the end position in a branch trading statement. It would therefore, in my judgment, have an impact upon branch accounts, regardless of the term used to categorise it.

453.

By the end of Mr Godeseth’s evidence, and by virtue of the way he answered questions, the reality of the development and operation of both Legacy Horizon and Horizon Online was far clearer than it had been before the Horizon Issues trial. I

found Mr Godeseth to be a truthful, credible and helpful witness in his oral evidence. His evidence was of considerable assistance; he did not always answer questions the first time he was asked, but the final answer would be reached. Given his role throughout the whole of the life both of Legacy Horizon and Horizon Online, I found his evidence in cross-examination to be of central relevance to the Horizon Issues. His evidence in cross-examination was also, in my judgment, broadly supportive of the claimants’ general case.

454.

There are, however, two aspects of Mr Godeseth’s evidence – though not his performance as a witness in the witness box - that were highly unsatisfactory and cannot pass without comment. The first aspect relates to his written evidence. The totality of his evidence, after his cross-examination, bore very little semblance to the picture that had been portrayed in his written witness statements. On some extremely important factual matters, such as the dates when Fujitsu had become aware of a particular bug, or the spreadsheet exercise by Anne Chambers prepared in 2015 (and disclosed in 2019), his written evidence was simply directly wrong. On others, such as the headline points omitted on the Callendar Square Bug that have been set out in [425] above, very important central elements detrimental to Fujitsu were simply omitted.

455.

Legacy Horizon had, as has been explained, started life originally as something rather different to what it became, having initially been intended as a tri-partite project involving payment of benefits. It did not unfold in this way and became rather different. Horizon Online also did not have a happy birth. The pilot for it had to be stopped, and Fujitsu put it on what was called “red alert”. Mr Godeseth described this as “very serious”. The biggest issue was with Oracle, which was what Mr Godeseth was working on and hence knew the most about, but he explained that there were other problems going on at the same time. Some of these problems were put to him – and it must be remembered that this was a pilot scheme, with some problems to be expected – and they included cash being short on one day by £1,000 because a transaction for £1,000 did not show up on the online report facility; cash withdrawals being authorised on screen yet the printed receipt being declined (the customer very honestly brought the cash back next day having noticed the receipt wording); a similar problem with a cash deposit; and remming in figures all being doubled up. These are all somewhat – and indeed markedly - similar to some of the problems alleged by the different claimants’ witnesses in this litigation. These all occurred during the pilot scheme.

456.

Not one of these different problems was referred to in Mr Godeseth’s witness statements, of which there were three. Witness statements are supposed to be accurate, and in a case such as this one with such centrally important issues, accuracy is clearly important. Quoting only selectively from, or wholly ignoring, contemporaneous documents prepared by (say) Mr Jenkins, who was the extensive source of much of the evidence, is not only unhelpful, it presents an entirely misleading evidential picture. It is not necessary to consider further how many personnel at Fujitsu may have assisted Mr Godeseth in producing such documents. Their content was wholly misleading in their original written form. Fortunately the cross-examination of Mr Godeseth led to a far clearer picture in so far as his evidence is concerned.

457.

The second unsatisfactory aspect which arose from his evidence is the approach of Fujitsu as demonstrated in various documents, including the PEAKs and KELs, but also in particular in the Receipts/Payments Mismatch issue notes. To see a concern expressed that if a software bug in Horizon were to become widely known about it might have a potential impact upon “ongoing legal cases” where the integrity of Horizon Data was a central issue, is a very concerning entry to read in a contemporaneous document. Whether these were legal cases concerning civil claims, or criminal cases, there are obligations upon parties in terms of disclosure. So far as criminal cases are concerned, these concern the liberty of the person, and disclosure duties are rightly high. I do not understand the motivation in keeping this type of matter, recorded in these documents, hidden from view; regardless of the motivation, doing so was wholly wrong. There can be no proper explanation for keeping the existence of a software bug in Horizon secret in these circumstances.

458.

The degree to which either, or both of, Fujitsu and/or the Post Office, expressly or constructively, knew exactly what and when, is for future trials in this litigation, and I make no findings in that respect in this judgment. They are not necessary in order to resolve the Horizon Issues and I do not speculate.

459.

In my judgment, however, there are sufficient entries in the contemporaneous documents to demonstrate not only that Fujitsu has been less than forthcoming in identifying the problems that have been experienced over the years, but rather the opposite. The majority of problems and defects which counsel put to Mr Godeseth, and which were effectively admitted by him, simply would not have seen the light of day without this group litigation.

460.

I do not know if this is because of concerns about the future of Horizon going forwards; or for Fujitsu to protect its corporate reputation; or for some other reason. It is unnecessary to speculate, and I do not do so. An internal Post Office IT risk management document from 2017 stated that “the HNG-X platform is end of life and is running on unsupported Windows software”, that it needs replacing, and also that the "Branch counter technology is aged and unreliable, with frequent hardware failures, resulting in branch disruptions." Mr Godeseth agreed with all of this. The unsupported platform is Windows NT4. As Mr Godeseth put it, “any technologist would tell you that was too old”. Windows NT4 is an operating system that was released in 1996, and Microsoft ended mainstream support for it in 2002 and extended support in 2004. Horizon now runs, not on NT4, but on Windows 10 and it is called HNG-A rather than HNG-X. This change to Windows 10 occurred in February 2017 and the roll out was administered by Computacentre, with Fujitsu providing the software. The Horizon system of 2019 is therefore a very different system than both Legacy Horizon and the earlier version of Horizon Online, HNG-X.

461.

The original Horizon Online system was introduced in 2010, and was even then substantially based on the existing system, Legacy Horizon, itself by then at least 10 years old. That had been introduced in 2000, and therefore it should not be controversial to describe Horizon Online in its HNG-X form as old technology. There have of course been upgrades and other improvements since 2010, and other software additions and add-ons, but Horizon will by necessity require replacement at some point. I detected a degree of sensitivity in the litigation generally about any plans for replacing Horizon Online, as though the Post Office feared that any hint that it might

be replaced would be seized upon by the claimants (and/or assumed by the court) as accepting that the criticisms of Horizon Online by the claimants would be corroborated by any Post Office plans for its replacement. Such fears, if there are any, are groundless, at least from the court’s point of view. It is inevitable that technology systems, particularly complex systems from as long ago (in technology terms) as 2010 will need replacing at some point. Technology moves at an increasingly rapid pace. There is reference in some of the documents in the trial bundle, but not in the evidence, to something called “Thin Client”, about which Dr Worden said he had no knowledge. He thought it was a cloud based system. It is unnecessary to speculate about this either. It is sufficient to state that, in my judgment, any plans there may be for replacement of Horizon Online in the ordinary course of the Post Office’s wishes to improve its business are not something that I need to consider in order to resolve the Horizon Issues.

462.

The degree to which Mr Godeseth’s evidence affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.

463.

Very slightly before the end of Mr Godeseth’s cross-examination – about 30 minutes or so – the Post Office sought to bring the Horizon Issues trial to an end by issuing their recusal application seeking to remove me as the Managing Judge, and to have the trial re-started at some indeterminate point in the future before another judge. There was therefore a substantial interruption in the trial, and a period of 22 days before the Horizon Issues trial resumed and the next factual witness for the Post Office, Mr Parker, was called.

Mr Parker

464.

Mr Stephen Parker is also a Fujitsu employee, and is now the Head of Post Office Application Support. He is therefore a very senior person. He first started work on what was then called the Royal Mail Group Account in 1997, which was before the introduction of Horizon. He has continued to provide support to the Post Office Account in the various roles he has occupied at Fujitsu throughout the whole of Horizon’s life, by which he meant both Legacy Horizon and Horizon Online.

465.

He was originally a support consultant within the SSC, and stated that he was effectively the deputy manager of the SSC, providing 3rd line support. He was lead designer and part of development team for the internal website providing the support knowledge database (which is also called KEL), technical documentation management and operational change control. He assisted the SSC manager in the provision of the support service and operational management. He became the Manager of the SSC in March 2010. The SSC later expanded and provided support services to other Fujitsu customers, but the Post Office Horizon system is still the largest. He has between 25 and 40 staff in the SSC whom he manages. He said that he was deputy manager of the SSC when Mr Roll worked there, although he did not have that title formally, and that people in the SSC would know this because he would stand in for the manager in his absence, and make decisions on approving actions for him.

466.

He had provided three different witness statements for the Horizon Issues trial, and his 2nd and 3rd statements were further to, and ultimately corrected, his earlier evidence, which arose in respect of what Mr Roll said had been done whilst he had been at the SSC, including injecting transactions during the time of Legacy Horizon. These were done using Riposte, the language then in use. During the Horizon Issues trial and just before the issuing of the recusal application, the Post Office’s solicitors wrote a letter to the claimants which stated the following. This letter was dated 20 March 2019, and was sent the night before Mr Parker was originally due to be crossexamined, his evidence then becoming delayed due to the recusal application:

“We understand from Fujitsu that the SSC has been carrying out further work to identify any PEAKs that show transactions being injected at the counter in Legacy Horizon in addition to those referred to in paragraphs 29 and 30 of Mr Parker's second statement. On Monday [ie the 6th day of the Horizon Issues trial] we learned that an SSC technician has:-

searched for all KELs that mentioned "RiposteMessageFile", "Ripostelmport" and "RiposteMessage';

collated the responsive KEL references (AChambers2226M, CObeng1029162824, DSeddon822M, MYoung5043M, RColeman1250R, acha2340K, ballantj498J, dsed344J, AHolmes3343J, DSeddon1753N, GMaxwell46141, PCarroll12541, RKing5135L, 81111, pcar847S, wbra716s);

re-searched the Peak system for any PEAKs which contained those KEL references; and

identified the following PEAKs: PC0105560, PC0106885, PC0063599, PC0063871, PC0065796, PC0066061 and PC0083998 (which cross refers to PC0076029).

We understand that the PEAKs referred to above relate to either:- • correcting configuration data after a PinPad change; or

reference data.

We also learned that during this exercise the SSC technician has also identified three examples of the marooned transaction scenario described in paragraph 38.2 of Mr Parker's second statement: PC0068495, PC0099141 and PC0079196.”

467.

The search terms identified in the letter were different to the ones that Mr Parker had used in his 2nd witness statement. These were "RiposteMessageFile", "RiposteMessage", "LPO Delete", "Marooned" and "RiposteObject put". The reason this is important is the command “RiposteImport”, used by the SSC as a search term during the week commencing 18 March 2019, but not used for the purposes of Mr Parker’s 2nd witness statement in January 2019, is actually one of the commands that was in fact used to inject transactions. In other words, the exercise carried out for the purposes of the witness statement was wholly deficient, in that it did not search for one of the specific terms used to inject transactions.

468.

When asked to explain this in his cross-examination, and the corrections that he made, Mr Parker stated the following:

“A I notified the legal team that Mr Simpkins had done some more work and as a result of that we changed -- we -- this letter was actually generated.

Q And you didn't think it was important to correct paragraph 29 of your second witness statement accurately to reflect the situation as it would have been on the day you were due to give evidence, or indeed today?

A That wasn't a choice I made personally. I was advised that we generated this letter.”

(emphasis added)

He did not say who made the choice to which he referred, namely not to correct his witness statement.

469.

It certainly seems to me highly curious – at best - that the exercise performed for Mr Parker’s 2nd statement, by personnel at SSC who are expert in such things, should have omitted using the search term “RiposteImport”. Given the purpose of paragraph 29 and 30 of Mr Parker’s 2nd witness statement was to identify how many times data had been in fact injected, a search exercise that did not use “RiposteImport” was never going to give the full picture. That initial search exercise was plainly inadequate. It is also rather curious that, having done that, the exercise that was described in the letter of 20 March 2019 from the Post Office’s solicitors that correctly used “RiposteImport” as a search term should have been done so very closely before Mr Parker was originally due to be called as a witness, namely 21 March 2019. In re-examination, Mr Parker said that “the thought occurred to him that he could add some other search terms into the work that he had done previously” and “he came to me with the data”. Why this thought should have suddenly occurred to the person it occurred to, who was not a witness, and at the time that it did, could not be pursued. Mr Parker denied that he was trying to conceal anything, and said he was just trying “to get the right data for the court which can be difficult sometimes when you are going back fifteen years.” This explanation overlooks that the decision not to use “RiposteImport” was one taken for his witness statements. His first witness statement was dated 16 November 2018; his second was dated 29 January 2019, and his third was 28 February 2019. None of these are “going back fifteen years”.

470.

Mr Parker had also given evidence in his 1st witness statement in respect of Mr Roll’s written evidence that Fujitsu could inject and affect branch transaction data without the knowledge of the SPM. He had originally said the following:

"As I explain below, those suggestions are incorrect and Mr Roll's account of Fujitsu's actions and powers is inaccurate and misleading".

471.

He explained this paragraph as follows in his cross-examination.

“A. In that paragraph I am trying to make the point that the suggestion that we frequently changed branch transaction data without informing the branches that such actions were being actually taken is not correct. "Frequently" is a subjective term but I would not have described the rate at which we were changing branch transaction data as "frequently".”

472.

One has only to read the original paragraphs as drafted to see that Mr Parker was not originally taking issue with frequency, or how often such actions were taken; he was stating in express terms that this could not be done by Fujitsu at all. For someone who has effectively spent the best part of 20 years in SSC, rising to a very senior post, such evidence must have been – and I find that it was - quite obviously incorrect. His explanation of this was as follows:

“Q Now, given the change of your evidence in your later statements about the ability remotely to access and inject transaction data, can you explain to the court what you were saying you believed about that sentence? What should the court now read for those words in the light of the three statements that it has before it?

A If I take the last sentence in isolation, which is what I think you are asking me to do, then I don't understand how I apply it, because I am -- I have been simply trying to say there that the frequency was not high, and that we would always involve the

Subpostmaster wherever possible if that sort of action was actually being taken. That's what I'm trying to say by that.”

473.

This was a very poor attempt by Mr Parker at explaining away quite directly incorrect factual evidence in a witness statement on a very important area in the litigation, namely Fujitsu’s ability to inject transactions remotely. Mr Parker’s only recourse was to claim he had not been saying what he plainly had been saying in his 1st witness statement – that Fujitsu did not have the power to do it, and that Mr Roll’s evidence to the contrary was inaccurate and misleading. The power to do something cannot be equated with frequency.

474.

Mr Parker also accepted that at the time he signed his 1st witness statement he had known that Fujitsu had the power to insert transactions into individual branch counters by using the correspondence server, a process Mr Roll described as “to piggy back through the gateway”. It was put to him that he had been giving his evidence initially from a position of not being very well informed about what could or could not be done. He denied that; he said “that would be wrong” and that in general terms he was confident of the information he gave. The obvious conclusion that I draw from this is that Mr Parker chose specifically to give the impression in his 1st witness statement that Fujitsu did not have the power (the word Mr Parker expressly chose) to inject transactions into the counter at branches, even though he knew that it did. This paints him in a very poor light as a credible witness. There is also no adequate explanation for why Mr Parker’s colleague should have decided, in the middle of the trial, to use the “RiposteImport” search term; equally, there is no adequate explanation for why this search term was not used for the exercise for Mr Parker’s witness statement. Injection of messages into the counter has been an extremely central issue in the long running dispute between the Post Office and its SPMs for a long time, and Mr Parker’s reliance in his re-examination upon the passage of time and the difficulty of “getting the right data” for the court is, in my judgment, a poor excuse. In reality, he had no credible explanation.

475.

A series of propositions was put to him about the degree of expertise expected of all the members of SSC 3rd line support. He did not agree with all of the points put, and in particular did not agree that all those in the 3rd line support role would be expected to have detailed knowledge of the system, based on both documentation and the inspection of source code; or that they would be trained in the coding languages used within the application. He was, however, left rather exposed in terms of the accuracy of this evidence when it was shown that the points that he had been addressing in this part of the cross-examination – some of which he would not accept - were taken from a Fujitsu internal document, entitled “End to End Application Support Strategy” marked Fujitsu Restricted and Commercial in Confidence which he had himself drafted. Indeed, it is page 1 of that document that specifically identifies Mr Parker himself as the author of the document, and it was approved by the Head of Application Services at Fujitsu, an obviously very senior role. This refusal by Mr Parker to accept his own previously drafted points in what I consider to be an important contemporaneous document also paints him in a very poor light as a credible witness.

476.

He accepted that Mr Roll was conscientious, skilled and experienced, and that he had given Mr Roll a reference when he left Fujitsu.

477.

He, and others not identified in his statement (although he said who they were when he was asked, Mr John Simpkins and Mr Mark Wright), had prepared a spreadsheet that had been put to Mr Roll showing the total number of calls to SSC between 2001 and 2004. Mr Simpkins, for what it is worth, appears to have been the person behind the exercise referred to in the letter at [466] above, so he was obviously involved both in preparing evidence and doing searches. Mr Parker agreed that SSC during Mr Roll’s time needed to address coding issues, even if they did not necessarily have an estate wide impact, and he also agreed that the list of PEAKs that had been identified as involving Mr Roll was only those where he had been the person who put the final response on the PEAK, rather than where he had worked on the PEAK at all.

478.

He also accepted that in categorising “software errors” he had excluded from the count those that were categorised as Avoidance Action, Administrative Response and Solicited Known Error.

479.

He was taken to a PEAK in 1999 where an initial balance was multiplied twice “due to a known software error” in the amount of a discrepancy of over one million pounds - £1,082,544,32 to be exact. The PEAK ran to 14 pages. 10 people worked on it including those in Development as well as the SSC. He accepted that this would have involved people looking into code. It was, however, categorised as “Administrative Response”. That PEAK alone demonstrates to my satisfaction the lack of inaccuracy in the Fujitsu closure codes, but there were many others. Other PEAKs to which he was taken showed reconciliation errors, and a note by the developer identifying that what were “up to 10 unnecessary reconciliation errors each week” and that the PEAK was a regression of another PEAK; the investigation into it led to the conclusion that a subsequent software release had not caught a fix. It stated "Risks (of releasing and of not releasing proposed fix): Without this fix, there will be possibilities of system errors at counter and while doing reversal transaction". This software fix was “released to live”, which means the fix was actually written and introduced into the system. Mr Parker had not included this in his software category either, because the final person working on it had categorised this as Administrative Response.

480.

In another PEAK, it was recorded that “the system is still playing up in that the screen is hanging in the middle of transactions -- PM did transaction ... but left office for 1 hour -- when he came back the monitor had 141 first-class stamps on screen totalling £38.07". Mr Parker accepted that this was not how the system was supposed to work, but the same PEAK also recorded the SPM becoming upset because the SSC were advising it was user error. The SPM wished to make a complaint about the person at SSC and the PEAK records “we both feel that this PM is complaining unjustly”.

481.

The same PEAK recorded that ROMEC engineers had been to the site, and had actually seen the problem themselves, which were phantom transactions. It recorded

“it’s not just the PM’s word now”. Indeed, this PEAK is particularly concerning, in terms of the accuracy of PEAK closures by Fujitsu. It records in one of the Fujitsu entries the following:

“I now have pressing evidence to suggest that unwanted peripheral input is occurring, the likely source being the screen. This has been seen at Old Isleworth ...When the PM has been asked to leave the screen on overnight I have observed system activity corresponding to screen presses happening with no corresponding evidence of either routine system activity or human interference. The way forward now is to correlate this with Microtouch complied monitoring software and to this ends Wendy is arranging for installation of the kit on Friday..."

However, notwithstanding this, the subsequent entries include “Phantom transactions have not been proven in circumstances which preclude user error" and “In all cases where these have occurred a user error related cause can be attributed to the phenomenon?”.

482.

The use of the phrase “in all cases” in respect of attributing user error wholly ignores that the ROMEC engineer is said to have seen the problem themselves, and it is not simply the word of the SPM in this instance. It ignores the activity observed overnight by the Fujitsu employee who made the entry in the PEAK, when the branch was not even open. Also, there is no reason, given the lengthy entries and studying the text throughout the PEAK in detail, why user error should be reached as a conclusion at all. This appears to be been used simply as an excuse to avoid an adverse conclusion to Fujitsu.

483.

Notwithstanding the detailed entries in the PEAK itself that in my judgment plainly point to the contrary, the PEAK was still categorised as No Fault in Product.

484.

In another, dealing with phantom transactions, one branch was described as “a problem office” that was accordingly being “monitored”. The PEAK recorded that “there have been incidents which showed a possible correlation between system activity and phantom transactions. These pointed to a touch screen problem and as a result the screen was replaced with a resistive model. As this produced no measurable improvement it has to be assumed that the problems were user related". This was also closed as No Fault in Product.

485.

A KEL on phantom transactions stated "There have been several calls over the last few months where Postmasters have reported phantom sales. Items appear by themselves for which the PM has not pressed an Icon. These may be individual items or several of the same item. Sometimes when no one has been near the screen items may appear". These are similar problems being reported by a number of different SPMs. That alone should have suggested, not as a matter of computer expert evidence but simply basic common sense, that something was amiss. If a number of different and unconnected people all report the same symptom, then that could well suggest a common problem to them all. However, the KEL, which ran from 2000 to 2004, suggested that the cable between the screen and the base unit was the root cause. The dates on this KEL show that, even on the face of the document, the problem was manifested for some years.

486.

Another PEAK that showed a problem described as “non-zero trading position on office rollover” was put to Mr Parker. This had been given a priority C status by Fujitsu, a status he accepted meant it was not taken account in the Service Level Agreement with the Post Office in liquidated damages thresholds (a type of financial penalty), which was a feature of those software PEAKs given higher priority status A and B. Status C also was non-critical. The description of the problem was described as “PM states that he has rolled over but the system is telling him that he hasn't -- PM states that he is in balance period 7 and he states he is getting the message 'wrong trading period MSG 31318 office balancing error'" and "Non-zero trading position ... on rollover of branch by user WMC002 to trading period 8".

487.

Mr Parker described himself as “hesitant” to describe this as a payments mismatch issue, and said it was an “office balancing error”, even though later in the PEAK senior personnel at SSC recorded “NBSC have ruled out user error” and also that "The problem occurred on 15/09/10 when stock unit 02 rolled over. This was originally reported, as per KEL, BALLANTJL759Q, in call PC0204537 ... but for some reason the call was closed without being investigated. There is a known problem with the use of the Cancel button during the stock unit rollover. This is fully described in KEL WRIGHTM33145J".

(emphasis added)

488.

He said he had not recognised the KEL number as being the payments mismatch

KEL. He accepted that in another entry in the PEAK the matter was stated to be “Severity: Critical” – which meant it could not be status C, defined as non-critical – and also included the entry “the branch accounts will need to be corrected.” This PEAK was however, on 1 November 2010, closed as category “Avoidance Action Supplied”. Mr Parker’s spreadsheet did not include “Avoidance Action Supplied” as constituting a software issue. He claimed that he had used “the definition my experience dictated rather than that in the document” and that “I forgot that that was exactly what I had done in that circumstance when I prepared that a few months ago.” Finally, after the question was put a third time, he accepted that it was a deliberate decision on his part to depart from the definition in Fujitsu’s own documentation.

489.

Another PEAK also identified as category C, non-critical, related to pouches being remmed in twice, the duplicate pouch being £25,000 in value. The PEAK stated “this might be a software issue” but also “PM is unsure if this is user error if it is a software fault. NBSC have not confirmed if it was user error or if they can reverse it.” One entry stated "POL have been informed of the error. Hopefully they'll issue a TC to correct loss at the branch. The underlying problem caused by using previous button during or just after scanning pouch barcodes, is still under investigation". This was closed as a Solicited Known Error.

490.

One PEAK which I found most illuminating, in terms of Mr Parker’s response to it when he was asked questions, is PEAK PC0229446. This is dated 17 November 2013, hence is Horizon Online, not in the period of the pilot project and sufficiently after the introduction of it, in 2010, for it not to be put down as the sort of issue that might occur in very early days of a system whilst minor issues were ironed out. The problem

was described in the PEAK as “PM doing cash declarations every now and again has major loss.” Entries in the PEAK included "PM has had cash declaration problem throughout the year and is losing a lot every now and again"; "He phoned up helpline told him can't [have] declared properly. He states that he loses £2,000 then jumps suddenly to £4,000, one point they lost £8,000 and is always losing money. PM stated that he has three post offices, only happens on this site"; and then "Done a declaration this morning and had a £6,000 also. It shows no error message when doing it. No report prints out only print-out of cash declarations". This passage of his crossexamination followed this:

“Q. Pausing there, if the PM is correctly reporting that, then that would be very serious for the Postmistress or Postmaster, wouldn't it? A. If it is being correctly reported, yes.

Q And it would not be the system working as it should.

A If we attribute it as a system fault, yes.”

491.

This again was categorised by Fujitsu as Category C, non-critical. In a sense, those two questions and answers sum up a large part of the case. The reason that this plainly, in my judgment, does demonstrate a system fault is Fujitsu’s own PEAK recorded:

“Voiced NBSC quoted ref H18174172 to see what checks they have done themselves before transferring call to Horizon.

They stated they had trainers come into the office and ruled out user error.”

And

“NBSC states the user error checks were carried out by Auditors at the site and not over the phone.” (emphasis added)

492.

This matter was assigned to Anne Chambers, whose name appears very often in many of the documents. The final part of her conclusion was (having said that “there were no known issues that would result in the variance being incorrect”):

“I can't tell why the declared cash doesn't match the expected cash figure, the branch need to make sure that what they have recorded on the system is correct, and investigate the anomalies.”

(emphasis added)

This is simply passing the buck back to a SPM in a branch. How an SPM could “investigate the anomalies” given they appeared to be generated by the Horizon system is not explained, nor could it sensibly be.

493.

She also closed the response with the category “Final – no fault in product” and closed the call by defect cause “General – User”. I consider this PEAK to be an ideal illustration of what I consider to be the most extraordinary situation at SSC, and one which on the face of it is difficult to explain, given the function of SSC was to investigate faults:

1.

Fujitsu routinely assigned non-critical Category C to matters that were really very important in their own right in any event, but of extreme importance to SPMs whose branch accounts were being directly affected. Mr Parker accepted that only Category A or B attracted financial penalties. It is not possible, on the evidence before the court, to conclude one way or the other whether it was this that affected the categorisation adopted by Fujitsu personnel.

2.

Fujitsu would ignore information directly from the Post Office itself that demonstrated that a SPM was not at fault. ROMEC engineers observing specific matters occurring, or in this case the Post Office’s own auditors ruling out user error, were simply ignored. This effectively amounted to the Fujitsu SSC personnel positively over-ruling the Post Office’s own non-SPM personnel, who were giving them essential information about failures in the Horizon system.

3.

Fujitsu would also close PEAKs in a variety of ways that entirely mischaracterised the issue. This particular one was “no fault in product” and defect cause “general – user”, in other words, it was caused by the SPM. This had been specifically ruled out by the Post Office’s own auditors who had visited the branch.

4.

On the PEAKs that were used in the Horizon Issues trial, Fujitsu would routinely assign lower categories of importance to reported matters that were directly impacting SPMs’ branch accounts. They would also routinely assign user error to reported matters, not because they had uncovered user error, but because they could not explain what had occurred. It seems to have been used as a default setting. Given my findings in the Technical Appendix, this approach by Fujitsu simply cannot be justified in technical terms.

494.

Fujitsu do not, on the face of these documents shown to Mr Parker, appear to me to have properly and fully investigated these myriad problems, nor did Fujitsu categorise such incidents correctly. They also seem to have moved away, in their investigations, from concluding that there were any issues with the software wherever it was possible for them to do so, regardless of evidence to the contrary, an approach that has been carried into the Fujitsu evidence for the Horizon Issues trial. In re-examination Mr Parker was asked about the way he had done his exercise, and also the point was put to him that the PEAKs to which he was taken were outwith the period when Mr Roll was employed at SSC. The implication of this was that Fujitsu may, or would, have been more accurate in the period in question rather than in later periods. Mr Parker said he could not possibly read all the PEAKs, that he considered his data to be accurate, and although “you may find a few PEAKs where the response code does not tally with the document” given the number of PEAKs involved “human beings will make those errors”.

495.

I reject that explanation, such as it is. I do not consider that Mr Parker was interested in accuracy in any of his evidential exercises, and I do not consider that he was objective in the way he presented his evidence, although he sought to give the impression that he was. I consider that Mr Parker, and the team who assisted him, sought to portray the Horizon system – Legacy Horizon and Horizon Online – in a light as favourable as possible to Fujitsu, regardless of its own internal evidence to the contrary, and regardless of the facts. Mr Parker’s spreadsheet plainly did not calculate all the categories that were used by Fujitsu for problems with the system, and sought

to give the impression that Mr Roll had been involved in far fewer incidents of this type than were the case.

496.

My conclusion about Mr Parker’s lack of accuracy, is supported by the changing story of Fujitsu about remote injection of messages; the need for correcting supplementary statements, including from Mr Parker himself; his admission that he knew Fujitsu had the power to do this, even though his 1st statement expressly said otherwise; the attempt to shift the blame for this on to Mr Roll; the assertions in his subsequent statement that Mr Roll had been unclear; and also by his attempt to portray his original evidence as dealing with frequency, not power or ability. My conclusions are also supported by his “deliberate decision” to depart from Fujitsu’s definition in its own internal document, which I have explained at [488] above. I also consider that the fact that someone such as Mr Parker had been on the Post Office Account for such a very long time, in his case throughout the whole life of Horizon, means he was hardly best placed to be objective in this type of evidence. But even making allowance for the natural reaction of an employee to wish to protect his employer’s interests, which many people may have sub-consciously, I find that Mr Parker’s evidence to the court was inaccurate to a significant degree.

497.

In my judgment, the exercise done by Mr Parker to demonstrate the number of PEAKs that Mr Roll worked on involving software, included in the spreadsheet of categorisation he and others prepared, is of no evidential value whatsoever. It relies upon the categorisation of the PEAKs performed by the Fujitsu personnel themselves, which in the vast majority of documents used in the trial were simply wrong and/or misleading; and it also used a different definition to the one included in Fujitsu’s own documents such as the PEAKs above. He accepted that he had made a deliberate decision to use a definition other than the one in the Fujitsu documents, which he justified by saying was based on his experience. That deliberate decision alone is remarkable. I find that he was, by means of the exercise he was involved in, trying to downplay the extent of Mr Roll’s involvement in software issues, and also trying to downplay the extent of software problems that were experienced on Legacy Horizon during Mr Roll’s employment at SSC.

498.

Although Mr Parker agreed, as it was put to him more than once, that accuracy is important, I do not consider his evidence in his witness statements to have been remotely accurate, even though he stoutly maintained that it was. He continued to maintain this in his re-examination, even though by then he can have been in no doubt that he had departed from Fujitsu’s own definition – which he had said he had forgot about. I found him a very unsatisfactory witness, who presented in his witness statements a misleading and one-sided sanitised version of actual problems and events that Fujitsu had experienced throughout his time at SSC. Although he provided tables with one of his witness statements that provided a detailed commentary on a large number of PEAKs and KELs, that commentary appearing in a column headed “Fujistu’s Comments” with two columns, “Response to Mr Coyne” and “Financial Impact on branch accounts” – that latter almost always “no impact”. The entries in those columns are almost entirely self-serving by Fujitsu and I find them to be of no evidential value. The text of the actual PEAKs and KELs themselves are far more useful, obviously entered at a time when the Fujitsu/Post Office position in the litigation was not at the forefront of the different authors’ minds, and these are what for the most part the experts have considered. Mr Parker’s table is of no assistance.

499.

The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.

Mr Membury

500.

Mr William Membury is another Fujitsu employee, namely a Fujitsu Central Quality Partner and he is specifically focused on the Post Office account. He was taken ill during the early part of the Horizon Issues trial, and when (after dismissal of the recusal application) the trial resumed with outstanding factual evidence on 11 April 2019, he was not well enough to be called in person. His witness statement was dealt with as hearsay evidence and given the circumstances the claimants agreed to extend time for the relevant notice to be given under CPR Part 33.2.

501.

Mr Membury has worked for Fujitsu since 1998 and became the Quality Risk and Compliance Manager for the Post Office account in 2011. He has overseen multiple audits of the Horizon system. Given the timing of this, these would have all been on Horizon Online. In 2014 he became a Central Quality Partner specifically focusing on the Post Office as well as the Payment Card Industry (PCI) Security Standards Subject Matter Expert (SME) for Fujitsu UK and Ireland. In 2015 he became the PCI SME for the Europe, Middle East, India and Africa regions. He returned to the Fujitsu Post Office Account as Head of Quality and Compliance in October 2018.

502.

He described his statement as an overview of the auditing and development regimes in place in relation to Horizon to the extent that those points were not covered by the disclosed technical documents. The statement was said to provide a high-level overview of the audits to which Horizon had been subject since 2000; Fujitsu’s audit methodology and his role in the audits; and to describe the development of changes to Horizon and how that tied in with or into the audit process.

503.

Mr Membury’s statement was very brief; it ran to only 5 pages of text. It also omitted some highly material matters. For example, he stated that the Post Office and Fujitsu agreed in 2010 that Horizon Online would be audited against the International Standard on Assurance Engagements (ISAE) No. 3402 assurance standard (Horizon ISAE 3402 Audit), entitled "Assurance Reports on Controls at a Service

Organization", which was issued in December 2009. This was developed to provide an international assurance standard for allowing public accountants to issue a report for use by user organisations and their auditors (user auditors) on the controls at a service organisation that are likely to impact or be a part of the user organisation's system of internal control over financial reporting. It provides assurance that the solutions in place to manage financial transactions are appropriate.

504.

However, he completely omitted any reference to the Ernst & Young Management Letter for the year ending 27 March 2011 which has been referred to in [264], [393] and [524], which included concerns about privileges and other critical matters in relation to IT. Indeed, he made only two references to Ernst & Young. One was to state that:

“Ernst & Young have carried out the Horizon ISAE 3402 Audit since the 2012/2013 financial year (preparations for the audit began with Post Office in 2011)”

505.

The second was a reference to how Ernst & Young performed the ISAE 3402 audit for the period 1 April 2014 to 31 December 2014. His failure to mention the 2011 Management Letter is, in my judgment, a serious omission.

506.

He could not be asked any questions about his evidence; although that was not his fault, this goes to the weight which I attach to his evidence in any event.

507.

I consider that Mr Membury’s evidence is of limited, if any, assistance in resolving the Horizon Issues. It does however continue the very one-sided picture presented by all the Fujitsu witness statements, which was to omit any reference to important contemporaneous documents that criticise or demonstrate any deficiencies with Horizon.

The absence of Mr Gareth Jenkins

508.

It is entirely a decision of the parties which witnesses they choose to call in any proceedings in respect of any evidence. The position of one person, however, who did not appear in the Horizon Issues trial, must be considered in more detail than would be usual, as the claimants make considerable complaint about this. The person in question is Gareth Jenkins, a senior Fujitsu employee who, although he retired recently, was obviously widely available to the Post Office and the source of a great amount of information both to the Post Office’s witnesses of fact, and also to Dr Worden (although he was not separately identified in Dr Worden’s “sources of information” paragraphs in his 1st Report). The fact that he provided information to Dr Worden emerged during the latter’s cross-examination. Mr Jenkins had previously given expert evidence for the Post Office in some of the criminal prosecutions of SPMs, in particular that of Ms Misra, to whom I have referred above, who was convicted of criminal offences in Guildford Crown Court in 2010.

509.

When the Post Office served its evidence of fact, there was no witness statement from Mr Jenkins, although many of their witnesses relied upon him as their source of information, he was referred to very often, and he obviously knew a great deal about Horizon. The extent and way in which Mr Jenkins had been closely involved was explained by Mr Godeseth in his cross-examination. Mr Godeseth had, in respect of the receipts and payments mismatch, originally stated in paragraph 42 of his 2nd witness statement that 60 branches were affected. He had corrected this to 62, a factual correction that was specifically made by him. The following passage of evidence is relevant to Mr Jenkins’ involvement in this.

“A. No, no, I didn't do a calculation to come up with the 60. I was quoting from other people.

Mr Justice Fraser: Someone just gave you the 60, did they?

A. I thought I was quoting from other people. I -- Gareth even said to me that in my statement I had said "approximately 60", so I was not -- clearly I didn't because the statement here doesn't contain that word. I had rather hoped it had when this was first brought to my attention, but no, I certainly did not do any specific calculation to come up with the 60 that I put into my original statement.

Mr Green: Did Gareth explain the change to you from 60 to 62? A. No.

Q. But how did he come to tell you what was in your original statement? What was that conversation?

A. I picked up the number -- to be -- my objective in this was to explain to the court the symptoms of the bug and how -- the technical aspect of it. I did not pay particular attention to getting the detail on how many branches were affected, correct. Q. Okay. So you have spoken to Gareth since your statement?

A. I don't think I have spoken to him about this in particular. I was -- as I say, when Gareth had said I had originally said "approximately 60" I was thinking that was quite neat, but that's not the case.

Q. Well, you said "Gareth even said to me that in my statement I had said 'approximately 60'", so he must have said that to you after your statement had been filed?

A. It was a comment in a document that we were exchanging.

Q. But you hadn't spoken to him about remote access since your first statement?

A. No.

Q. Why have you stayed off that topic with him?

A. Oh, sorry, this was just a comment. We have been exchanging documents, we have been commenting on documents, so it was not a particular conversation. It is merely a case of Gareth had commented on this when it was pushed back to us that I had originally said 60 and actually the answer was different.”

510.

There is nothing wrong with Mr Godeseth correcting the number of affected branches from 60 to 62. There is also nothing wrong with him reproducing a number given to him by someone else, as long as it was properly identified that the information came from someone else, who ought to be identified (which it wasn’t in his statement, but was, in at least outline terms, in his cross-examination). The reason for reproducing these passages is simply to identify the extent to which Mr Jenkins was so closely involved in the litigation, a point relied upon by the claimants. There were a great many references throughout Mr Godeseth’s evidence, written and oral, about information he had obtained from Mr Jenkins.

511.

When the Post Office served their evidence of fact, the claimants had asked the Post Office why there was no statement from Jenkins, whether Mr Jenkins was available to give evidence, and also whether he was involved as one of a team of what the claimants referred to as the “shadow experts”. This description was challenged by the Post Office, and the question of shadow experts is addressed further at [556] below. No explanation was given for Mr Jenkins’ absence in response to these requests, or in evidence in the trial, although it was confirmed that Mr Jenkins was not one of the team of so-called “shadow experts”.

512.

There the matter might have rested. However, in the Post Office’s written closing submissions, an explanation of sorts was for the first time provided. This was in the context of two matters: firstly, by way of explanation of Mr Godeseth’s evidence, and potentially to downplay its impact; secondly, in relation to the claimants’ complaints about the second hand nature of some of the Post Office’s factual evidence because in large part this had emanated from Mr Jenkins. This explanation by the Post Office included the following passages in its written submissions:

“144. [The claimants] understandably complain that Mr Jenkins and the other source of Mr Godeseth’s information could have given some of this evidence first hand. However:

144.1

Taking into account that Mr McLachlan’s evidence specifically addressed things said or done by Mr Jenkins in relation to the Misra trial, Post Office was concerned that the Horizon Issues trial could become an investigation of his role in this and other criminal cases.

144.2

Moreover, Post Office was conscious that if it only adduced first hand evidence in the trial, it would end up having to call more witnesses than could be accommodated within the trial timetable.

144.3

Furthermore, so far as Post Office was aware, the relevant parts of Godeseth 2 were most unlikely to be controversial. For example, the Misra trial was a matter of public record, the four bugs were covered by contemporaneous documentation and Post Office had no reason to doubt Fujitsu’s account of the documents it held.”

513.

In a footnote to paragraph 144.2 of the closing submissions, the Post Office added “…..As noted above, had its witnesses only given first hand evidence, Post Office estimates that some 34 additional witnesses would have been required.”

514.

The following are relevant in my judgment:

1.

Of primary importance is the principle that it is for each party to decide whom to call as a witness, and what evidence they may seek to obtain from any particular witness.

2.

Mr Jenkins is an important and central person so far as the operation, efficacy and robustness of Horizon is concerned, and also in respect of the number of incidents over the years that have led to PEAKs, KELs, problems and fixes. That is reinforced by the Post Office’s own closing submissions that stated, at paragraph 138, that there were two possible candidates for the witness giving the overview of Horizon, namely Mr Godeseth and Mr Jenkins. It also stated that taking into account Mr Jenkins’ involvement in cases before the CCRC (including the Misra case) the decision was taken to use Mr Godeseth. For that decision to have been taken, it is implicit that Mr Godeseth must know enough about the system to give the evidence that he did.

3.

The point in the footnote about a potential further 34 witnesses is not relevant to whether Mr Jenkins was, or was not, called by the Post Office. The Post Office did not at any stage apply to the court and explain that further time was required for the length of the Horizon Issues trial for the specific reason that relevant evidence of fact from the Post Office would require a particular minimum number of witnesses. However, given the primacy of point (1) above, it was entirely for the Post Office to decide whether to call Mr Jenkins or not, how many witnesses to call, whom those witnesses were to be and what the content of their evidence should include.

4.

The Post Office chose to proffer a reason for Mr Jenkins’ absence in closing submissions. They were not obliged to explain. However, the reason in paragraph 144.1 quoted above is not a valid reason for his absence. The claimants would have been entitled, in cross-examination, to put to Mr Jenkins any previous inconsistent statements he had made on the same subject, but obviously only if there were any. These could, potentially, have included previous statements he may have made in earlier proceedings, but in order to be allowed to do that, such statements would have had to be inconsistent with his evidence in the Horizon Issues trial. Putting a previously inconsistent statement on a particular fact is permitted as part of crossexamination if a witness’ evidence on that fact is in issue. However, the Horizon Issues trial would not have become “an investigation of his role in this and other criminal cases” – the Horizon trial was about the Horizon Issues. Also, this type of cross-examination would only have arisen if any statement(s) in the Horizon Issues trial was or were not consistent with those he had made in previous cases in which he had been involved.

5.

Regardless of the validity of the explanation given by the Post Office, Mr Jenkins was not called as a witness, and it merits repetition that it is entirely a matter for the Post Office which witnesses it called, and which it did not. No speculation is permitted as to what evidence he might have given in the Horizon Issues trial. It ought to be recorded however that no limit on the number of witnesses was imposed by the court on any party, and the Post Office could (had it wished) have called both Mr Godeseth and Mr Jenkins. Nobody forced the Post Office to choose one or the other.

6.

There is established authority that in certain circumstances the court can draw adverse inferences from the absence of a witness. The claimants do not invite me to draw any adverse inferences from the absence of Mr Jenkins, and I do not do so.

7.

The fact that the Post Office chose to advance certain evidence in the Horizon trial, that emanated from Mr Jenkins, by means of another witness saying “Mr Jenkins told me that…..” means that the claimants were deprived of the opportunity properly to test that evidence by asking Mr Jenkins about it. It also emerged in cross-examination, although not in his written report, that Dr Worden has also obtained certain information directly from Mr Jenkins. By dealing with the material in this way, and having deprived the claimants of the opportunity to cross-examine Mr Jenkins, the weight to be given to evidence emanating from Mr Jenkins is less than it would be otherwise.

515.

There are two consequences of the Post Office choosing to adduce such evidence in this way. The first is that Mr Jenkins did not have to undergo any cross-examination on such evidence; the second is that such evidence would not be given the same weight by the court as though Mr Jenkins had given it himself. The Post Office would have known this in advance, as this is wholly conventional. How these different factors were weighed by the Post Office and its legal advisers in deciding whether or not to call him, is solely a matter for the Post Office. The end result is that Mr Jenkins did not appear and was not cross-examined.

516.

However, I reject the suggestion that Mr Godeseth was insufficiently knowledgeable of the matters upon which he was cross-examined for his evidence to be of significant evidential weight. The Post Office’s Closing Submissions sought to dilute the effect of Mr Godeseth’s cross-examination by, at the same time, implicitly suggesting that

Mr Jenkins (or someone else, but not Mr Godeseth) would have been a far better

person to have answered those questions; yet explaining why Mr Jenkins had not been called. These submissions, though understandable from the Post Office’s point of view in forensic terms only, have little force. A clear choice was made by the Post Office not to call Mr Jenkins as a witness; that was their right. A clear choice was also made to call Mr Godeseth. The Post Office was are entitled to call such evidence as deemed necessary by both the Post Office and its advisers. They cannot, however, have matters both ways, and try to downplay Mr Godeseth’s evidence because he was called to deal with certain matters, and Mr Jenkins was not. The Post Office has to abide by the consequences of their choice of witnesses, in terms of the evidence now before the court to resolve the Horizon Issues trial. I have already explained my view of Mr Godeseth’s evidence in Part E above, and it need not be repeated.

Inaccurate Statements by the Post Office

517.

Litigation in this jurisdiction is adversarial. What that means is each side advances their own case, and challenges or rebuts the case of the other side, and the court assesses the evidence before it adduced by the parties, makes necessary findings on the facts and applies the law. Parties in civil litigation will usually advance their own evidence of fact, and although mechanisms are available for compelling other witnesses to attend and answer questions, there can be some pitfalls in doing so and issuing a witness summons (what used to be called a subpoena) is a relatively rare step. The court can only therefore resolve the case on the evidence before it, although it can draw inferences, that is to say common sense conclusions, on the evidence it does have.

518.

The Post Office has no obligation to assist the claimants in advancing their case against it. The Post Office has, however, maintained publicly that it was seeking to be “transparent” about Horizon, and prior to the litigation it made certain public statements in relation to the increasing disquiet on the part both of SPMs, and others who became involved either on their behalf (such as some Members of Parliament) or in an investigative way (such as the BBC Panorama programme and other journalists). These statements by the Post Office routinely and strongly insisted that there was nothing in the criticisms being levelled at the accuracy of Horizon, and that losses that were shown in SPMs’ branch accounts were caused either by carelessness or dishonesty on the part of the different SPMs who experienced what they considered to be unexplained discrepancies and losses. In 2015 a Parliamentary Select Committee held hearings into the mediation scheme that had been set up to attempt to address the claims by SPMs in respect of Horizon. I have not considered any evidence submitted to the Select Committee, or any details of the mediation scheme (which was being conducted by Second Sight) and refer to them here simply as a matter of chronological record.

519.

The claimants rely upon a public statement released by the Post Office in 2015 after the BCC Panorama programme. Part of the statement was headed “The Horizon System” and stated:

“The Horizon Computer System

Horizon is robust and effective in dealing with the six million transactions put through the system every day by our postmasters and employees at 11,500 Post Office branches. It is independently audited and meets or exceeds industry accreditations. There have been 500,000 users of the system since it was introduced.

Nevertheless, rigorous re-investigations were undertaken into claims made by 136 mainly former postmasters that the system caused losses in their branches.

There is overwhelming evidence that the losses complained of were caused by user actions, including in some cases deliberate dishonest conduct. The investigations have not identified any transaction caused by a technical fault in Horizon which resulted in a postmaster wrongly being held responsible for a loss of money.

There is also no evidence of transactions recorded by branches being altered through ‘remote access’ to the system. Transactions as they are recorded by branches cannot be edited and the Panorama programme did not show anything that contradicts this.”

(emphasis added)

520.

Another document from 2015 upon which the claimants rely is an internal email chain, which originated from Paula Vennells, then the Chief Executive of the Post Office, on 30 January 2015. This was prior to her appearance before the House of Commons Select Committee in February 2015. She posed the following question in an email sent internally to Mark Davies and Lesley Sewell, both of the Post Office:

“Dear both, your help please in answers and in phrasing those answers, in prep for the SC:

1) "is it possible to access the system remotely? We are told it is."

What is the true answer? I hope it is that we know this is not possible and that we are able to explain why that is. I need to say no it is not possible and that we are sure of this because of xxx and that we know this because we have had the system assured.”

(emphasis added)

521.

Ms Vennells obviously needed to know whether the answer matched her understanding, which was – as she put it, both “I hope” and “I need” -- that it was not possible to access the system remotely. This query was passed on through various people, including at one stage from James Davidson who has both a Fujitsu and Post Office email address, who answered to Mark Underwood:

“As discussed, can you hook up with Kevin to review what answers have already been provided to second sight as this should form the Post Office response.”

522.

The answer was provided by Mark Underwood on 30 January 2015 in an email which is part of the same email chain or string:

Can Post Office or Fujitsu edit transaction data without the knowledge of a Subpostmaster?”

Post Office confirms that neither it nor Fujitsu can edit transaction data without the knowledge of a Subpostmaster.

There is no functionality in Horizon for either a branch, Post Office or Fujitsu to edit, manipulate or remove a transaction once it has been recorded in a branch's accounts.

The following safeguards are in place to prevent such occurrences:”

(various matters are then listed in the remainder of the email)

(bold present in original)

523.

This then was subject, in the email chain, to a degree of refinement. Kevin Lenihan forwarded the email onwards to Mark Underwood and others, and stated:

“Mark / Mel,

James has had a look at your answer to Q1. And thinks there’s too much detail for Paula – this was written for a different type of audience. He has captured the same points but in a more appropriate format :- He states:-

Having looked again at the request from Paula, it appears that the fundamentals around this question (remote access) are not understood. I suggest that Paula is briefed along the lines of the following.

1)

No transaction data is held locally in any branch. Transactions are completed and stored in a central database and copies of all data is sent to a secure audit database.

2)

Sub-postmasters directly manage user access and password setting locally so system access (to create transactions) are limited to approved local personnel only who are responsible for setting their own passwords. Users are only created following an approval process which requires authorisation by the sub-postmaster. All subsequent transactions are recorded against the id used to log on to the system.

3)

Once a transaction has been completed, there is no functionality (by design) for transactions to be edited or amended. Each transaction is given a unique number and ‘wrapped’ in a digital encryption seal to protect its integrity. All transactions are then posted to a secure and segregated audit server.

4)

On approval, there is the functionality to add additional transactions which will be visible and have a unique identifier in the audit trail. This is extremely rare and only been used once since go live of the system in 2010 (March 2010)

5)

Support staff have the ability to review event logs and monitor, in real time, the availability of the system infrastructure as part of standard service management processes.

6)

Overall system access is tightly controlled via industry standard ‘role based access’ protocols and assured independently in annual audits for ISO 27001, Ernst and Young for IAS 3402 and as part of PCI audits.”

(emphasis added)

524.

I do not see how the statement that “I do not know how the fundamentals around this question (remote access) are not understood” can sensibly be made. Nor do I know what the expression “the fundamentals around this question” in fact means, in plain English. The question from Ms Vennells was very straightforward. It was as follows, using her words, and separating out each clause of the enquiry:

1.

“Is it possible to access the system remotely? We are told it is.”

2.

“What is the true answer?”

525.

Both of these are really very simple questions. Question 1. means can it be done, because Ms Vennells is being told it can be done. Question 2, “what is the true answer?” means she is seeking the true answer to question 1. This is not complicated, either in linguistic, computing or even business terms, nor is it difficult to understand. She then expressed her aspirations in terms of what the true answer might be. “I hope it is that we know this is not possible and that we are able to explain why that is. I need to say no it is not possible.” (emphasis added)

526.

This trial has shown that the true answer to the enquiry she made in early 2015 was “yes, it is possible.” It has taken some years, and many tens of millions of pounds in costs, to reach that answer. Kevin Lenihan, who wrote that he did not understand “the fundamentals around this question (remote access)”, seems to me to have avoided providing a simple answer to a simple question. It is not necessary to consider why that was, whether from a lack of understanding on his part, or otherwise.

527.

I am unaware of whether there is any other final answer to Ms Vennells internally to her very straightforward question, or of what she in fact told the Select Committee. None of the people through whom this email chain passed were called as witnesses in the Horizon Issues trial. However, the point in bold in the email quoted at [522] above is now known, as a result of this litigation, simply to be incorrect. Such editing can take place without the knowledge of the SPM. All and any previous statements by the Post Office stating that this cannot be done are simply factually wrong. The more detailed points at (1) to (6) of the email quoted at [523] provide only selected information and, in my judgment, are entirely off the point. They have the effect of obscuring what ought to be a simple answer, with a level of detail which makes the actual answer highly unclear. The answer at (1) applies only to Horizon Online. For Legacy Horizon, transaction data was held locally in the first instance, on what was called the counter. This was explained very clearly by the Post Office in its opening, including in its oral opening submissions. The first sentence of (3) in the list in the email is not correct. The statement in (4) – which may refer to the Transaction Correction tool – is correct (or at least, not incorrect) but only so far as it goes, and in my judgment crucially omits the ability to edit transactions without visibility to the SPM. It also fails to consider the existence of APPSUP permissions at all, which every member of SSC had. The experts are agreed that the APPSUP permission was very powerful and enabled an SSC employee to do pretty much whatever they wanted, to paraphrase the Post Office’s own expert. It can also be seen, particularly by the evidence that emerged in the cross-examination of Mr Godeseth, that in a great many cases the PEAKs show that user error was attributed to problems that were not user error at all, and that others within the Post Office (for example its own auditors and ROMEC engineers) had either witnessed for themselves and/or satisfied themselves that user error was not the cause of specific issues, but Fujitsu decided they were user error in any event.

528.

The extent of permissions enjoyed by SSC was the subject of some express concern by Ernst and Young (the Post Office’s own auditors) in 2011, and although the position may have been regularised by 2015, certainly prior to 2011 (which covers the

Legacy Horizon years) the situation was very different. The email also states “he has captured the same points” yet the most important point to which an answer is given, or certainly the answer given in bold in the answer at [522] above, is not corrected, even though it is plainly wrong. The overall tenor of what Ms Vennells received, if it reflected the full content of this email chain, would have been that the Post Office and Fujitsu could not edit transaction data without the knowledge of the SPM. The evidence in this trial is directly to the contrary, and Fujitsu could do so. It is important to bear in mind the distinction between the way Legacy Horizon, and Horizon Online, work. Referring to keeping data on the “counters” is a Legacy Horizon expression. This refers to the way the data was held at the branch; these were discs (including mirror discs) at the branch which contained the data. The discs were contained in terminals on the counter. In Horizon Online, the data was held at a central database or server (the phrase used by the Post Office in its oral opening).

529.

This tenor of these public statements made by the Post Office was maintained into, and as part of, the group litigation. For example, certain statements were made in the Generic Defence in these proceedings. A party’s Defence is the formal pleading document which sets out a party’s legal position in the litigation. Since the reforms in civil litigation adopted after what are called the Woolf Reforms, which led to the introduction of the Civil Procedure Rules in 1998 which govern all civil litigation in the jurisdiction, such pleadings must be accompanied by a Statement of Truth. These are, for larger companies, usually signed by directors or very senior personnel, which occurred in this case. That statement attests to the truth of the matters stated in the pleading.

530.

This issue was dealt with in the pleadings in the following way. By this point the Post Office had accepted that there was some limited ability to do that which it had previously stated was not possible. In the Generic Statement of Case, the claimants set out the following part of their case:

“25. Further, the Defendant was, by itself and/or via its agent Fujitsu, able to alter branch transaction data directly and carry out changes to Horizon and/or transaction data which could affect branch accounts.

26. However, the Defendant has made public statements in the following terms:

26.1.

"Horizon does not have functionality that allows Post Office or Fujitsu to edit or delete the transactions as recorded by branches" (Defendant's published reply to Second Sight's Briefing Report Part Two, concerning a review of the Horizon system); and

26.2.

"Transactions as they are recorded by branches cannot be edited" (Defendant's published reply to the BBC Panorama documentary in relation to Horizon).

27. These statements were untrue, as the Defendant now admits.”

531.

These paragraphs were pleaded to by the Post Office in its Generic Defence in the following terms:

“57. Paragraph 25 appears to be concerned with the editing or deletion of transaction data input by or on behalf of a Subpostmaster without his or her consent. Accordingly, Post Office assumes that it is not concerned with transactions such as Transaction Corrections which are sent to branches but must be accepted by or on behalf of the Subpostmaster before forming part of his or her branch account. As to the circumstances in which such transaction data can be edited or deleted without the consent of the Subpostmaster:

(1)

Neither Post Office nor Fujitsu has the ability to log on remotely to a Horizon terminal in a branch so as to conduct transactions.

(2)

A Post Office employee with "global user" authorisation can, when physically present at a branch, use a terminal within the branch to add a transaction into the branch's accounts. The purpose of "Global User" authorization is to allow access to the systems for during training and/ or audits. Any transactions effected by a Global User are recorded against a Global User ID and are readily identifiable as such.

(3)

Fujitsu (and not Post Office) has the ability to inject transactions into branch accounts (since the introduction of Horizon Online in 2010, transactions of this sort have been called "Balancing Transactions"). These transactions do not involve any removal or amendment of the transactions entered at the branch. Their intended purpose is to allow Fujitsu to correct errors or bugs in Horizon by cancelling the effect of an error or bug on a branch's data. They may be conducted only by a small number of specialists at Fujitsu and only in accordance with specific authorisation requirements. They are rarely used. To the best of Post Office's information and belief, only one Balancing Transaction has ever been made so as to affect a branch's transaction data, and this was not in a branch operated by a Claimant. A Balancing Transaction is readily identifiable as such.

(4)

There are a small number of Fujitsu specialists who have certain privileged user access rights which they could in theory use to amend or delete the transaction data for a branch. The intended purpose of privileged user rights is system support, not the alteration of branch transaction data. To have abused those rights so as to alter branch transaction data and conceal that this has happened would be an extraordinarily difficult thing to do, involving complex steps (including the writing of sophisticated computer programmes and circumvention of sophisticated control measures) which would require months of planning and an exceptional level of technical expertise. Post Office has never consented to the use of privileged user rights to alter branch data and, to the best of its information and belief, these rights have never been used for this purpose.

(5)

Post Office cannot conceive of a reason why any Fujitsu personnel would have sought to add, inject, amend or delete any transactions in any branch accounts so as to create a false shortfall. It would for all practical purposes be impossible for any of them to generate significant shortfalls without detection and, even if they were able to do so, they would be unable to take the benefit of such shortfalls for themselves.” (emphasis added)

532.

This stance was maintained by the Post Office in the evidence served on its behalf for the Horizon Issues trial, until service of Mr Roll’s 2nd witness statement. To be fair to the Post Office, its origin was the witness statements served by Fujitsu employees, rather than Post Office employees. The position within the Fujitsu witness evidence,

prior to its correction by the later statements from Mr Parker, was that what Mr Roll said was possible on Legacy Horizon, and what he had himself done, was simply not possible. Indeed, Dr Worden considered it sufficiently clear that as an IT expert he felt able confidently to assert in his 1st Expert Report that he, Dr Worden, had “established” that Mr Roll’s evidence of fact in this respect was wrong. After service of Mr Roll’s 2nd witness statement, Fujitsu finally came clean and confirmed (via Mr Parker) that what Mr Roll said was correct. Data could be altered by Fujitsu on Horizon as if at the branch; under Legacy Horizon, transactions could be inserted at the counter in the way Mr Roll described. This could be done without the SPM knowing about this. Mr Godeseth also confirmed that it would appear as though the SPM themselves had performed the transaction. This is directly contrary to what the Post Office had been saying publicly for many years.

533.

Therefore some of the earlier public statements made by the Post Office, and the important one contained in the Post Office’s pleaded Defence to which I have referred above, were factually untrue in at least one highly important respect. This concerns the ability of Fujitsu to insert transactions into a branch’s accounts remotely, without the SPM being aware of this, and without the transaction being identifiable in the transaction data as having been inserted remotely in this way. The term “remotely” means from a location elsewhere than within the branch itself.

534.

Although the phrase “remote access” was used during the Horizon Issues trial, the parties did not always use the expression in the same way as one another. The Post Office’s definition was explained in oral opening in the following way:

“When I talk about remote access I'm talking about action taken remotely to either inject new transactions or to edit existing transactions or to delete existing transactions in a way that could change the accounting position of the relevant branch.”

For the purposes of this judgment, I too use the term “remote access” in this way. The true picture is now that, following the Horizon Issues trial, there is evidence of transactions recorded by branches being altered through remote access by Fujitsu; and transactions as they are recorded by branches being edited; that this can be done without a SPM’s knowledge or permission; and without being identifiable as having been done remotely. Dr Worden was asked about the 2015 Post Office statement in response to Panorama, which he said he had not read, and he explained that it was the audit records in the audit store that could not be edited. He was not prepared to say the 2015 statement was factually wrong. The BRDB records could be edited, and he also referred to the joint statement in which the two experts had agreed that they could not say that anything could not be done. He was very hesitant, if not reluctant, to state that the 2015 statement was wrong.

535.

In my judgment the 2015 Post Office statement plainly is wrong. I find that the 2015 statement was not true. Again, to be fair to the Post Office, this was accepted in the litigation and at the hearing before Senior Master Fontaine. The quotation from the transcript of that hearing when this was addressed by the Post Office in oral submissions is as follows:

“Master, first of all, could I just deal with the remote access point? The letter to which my learned friend took you was, as you might expect, written by people who thought it was correct. The Horizon system is a very complicated system. It involves lots of departments in ... both in Fujitsu and in the Post Office. And the people who are responsible for the correspondence didn't know that, in fact, there were these two other routes. Very few people at Post Office knew that there were these two other routes. They were ... they were routes that are under ... essentially under the control of Fujitsu who's the expert independent contractor that is involved in the operation of the system. And it is a matter of enormous regret that the people who wrote that correspondence and made those submissions weren't aware of that but, you know, we are where we are; the point is that, the point having been discovered, the Post Office wasted no time in ... in bringing the truth ... the accurate ... and accurate set of facts to the knowledge of the claimants.”

(emphasis added)

536.

The claimants rely upon the way this sequence unfolded, and the admission by the Post Office that its previously public statements were untrue, as making it particularly important that the true and accurate position was then provided by the Post Office. Indeed, the submissions I have quoted at [535] above make it clear that the court and the claimants were being told that the Post Office had wasted no time in bringing the truth, “the accurate set of facts”, to the knowledge of the claimants. As the analysis of the Defence and the evolution of the Fujitsu evidence of fact on remote access for the Horizon Issues trial demonstrates, the accurate set of facts did not emerge at that time either. The truth, the accurate set of facts, has only emerged as a result of the final supplementary statements for the Horizon Issues trial that were served, and only finally (so far as Fujitsu is concerned) after their witnesses had initially stated clearly in their witness statements that Mr Roll was wrong. Eventually these witnesses accepted that he was right. The supplementary witness statements that accepted this were dated 29 January 2019. Prior to that date, the opposite position had been maintained by the Post Office.

537.

The Post Office in its opening submissions stated the following:

“Taking remote access as an example, the need for remote intervention affecting branch accounts will obviously be rare. On any view, the occasions on which privileged users at Fujitsu have exercised their ability to remotely inject, edit or delete branch transactions or accounting entries will represent a tiny percentage of the relevant transactions/accounting entries. And the occasions on which they have done so negligently or dishonestly will, in turn, represent a very small percentage of those occasions. So, compared with the volume of business recorded in branch accounts, the number of cases in which false data will have been remotely introduced will be extremely small (multiplying a small chance by a small chance). This is a “second order effect” (a small proportion of a small proportion) which is, by definition, extremely unlikely to have any significant impact on the robustness of Horizon.” (emphasis added)

538.

That submission misses the point, in my judgment. It elides two matters, namely whether something is technically possible, and the number of times that it has in fact been done. The former, whether it was possible, had been expressly denied, and that denial is now shown to be wrong. The latter, the number of times it was done and

with what effect, is a different matter. There is also very little, if any, evidence that is relied upon by the Post Office to justify the assertion that it was a “tiny percentage” of times. That is a subjective assertion of very limited weight.

539.

I consider the significance of the previously factually untrue statements to be considerable. The statement was made publicly by the Post Office, turned out not to be factually correct, and the Post Office gave an explanation and said the full set of facts was now available. The situation was pleaded to by the Post Office in its Generic Defence, with a statement of truth. That too turned out not to be correct, and the true position has only emerged in the Horizon Issues stage of the litigation as a result of the evidence of Mr Roll, which I have dealt with above. It was only following his written evidence that Mr Parker, and Mr Godeseth – both senior Fujitsu employees – prepared their supplementary witness statements correcting their first statements. These first statements, as I have explained above, were simply untrue in that important respect. These witnesses had previously stated that this was not possible. Mr Parker said Fujitsu did not have the power to do this.

540.

I find as a fact that Fujitsu do have the ability to insert transactions into branch accounts on a remote basis (in other words, remote access as the Post Office defined it in opening submissions existed) and this could be done without these being visible to the SPM in that branch in question, either at the time or subsequently. It also follows that this could be done without an SPM in this position having knowledge of this, and therefore without consenting to it. Someone who does not know something is happening cannot give permission for it to happen. Mr Godeseth also gave important evidence about how such transactions would appear. It is not possible to say, in blanket terms, that permission was not sought and/or given in some cases. Indeed, Dr Worden accepted that the APPSUP permissions meant that – as he put it – Fujitsu personnel could do “pretty much anything”. On Day 18 he said in cross-examination the following:

“Q. Dr Worden, you knew that a central issue, not only a central issue legally but a very high-profile issue in the case, was the extent to which Post Office had remote access to the counters, didn't you? You knew that?

A. Yes, and what I'm talking about, what I was talking about was the extent to which this could happen without the knowledge of the subpostmaster.

Q. And that's the –

A. And we agreed in the joint statement that more or less Fujitsu or Post Office could do anything.”

541.

He was asked about this answer two days later.

“Q. It is Day 18, page 67. I think it is lines 5 and 6 actually. It says: "Answer: And we agreed in the joint statement that more or less Fujitsu or Post Office could do anything."

A. Yes. We can see precisely what we agreed in a minute, obviously.

Q. In relation to the –

A. We agreed the experts couldn't demonstrate that they couldn't do everything. I mean, that's sloppy wording by me there. I think the joint expert statement says it better.”

542.

The exact wording in the joint statement in respect of repairing transactions is as follows at 10.2 of the 4th Joint Statement:

“Certain facilities and procedures used by Fujitsu to repair the more common issues which arose in Horizon were standardised, and evidence of them persists. However, to repair less common issues which arose from time to time, standard tools and procedures might not have been sufficient, and evidence might not persist of what was done at the time. Even when evidence does persist, it may be extremely difficult for the experts to interpret it today, because of the scale and complexity of Horizon.

Therefore, it is usually difficult for the experts to make categorical negative statements of the form: ‘X or Y never happened’.”

543.

This litigation is between the claimants and the Post Office. Fujitsu is not a party. Although one of the contracts between the Post Office and Fujitsu was uploaded to the electronic trial bundle on the penultimate day of evidence (by the Post Office’s solicitors) the litigation is not concerned with the details of that contractual relationship, other than (tangentially only) the cost to the Post Office of making ARQ requests, as the ARQ data is maintained by Fujitsu and charges are raised to the Post Office in this respect. However, Fujitsu’s involvement does cast something of a shadow. That contract has been addressed in terms of whether certain documents were, or were not, in the control of the Post Office. I have already dealt with the issue of PEAKs and KELs, and how the Post Office maintained initially it was not obliged to disclose these. Fujitsu personnel who were not witnesses were involved in providing information to the Post Office’s expert, Dr Worden. It was the Fujitsu witnesses who originally, and in my judgment unequivocally and directly, flatly contradicted what Mr Roll said could be done on the subject of remote access. I have also identified that in my judgment, the majority of the Fujitsu witnesses were more interested in following some sort of Fujitsu party line than they were in answering questions in cross-examination wholly frankly, although I exempt Mr Godeseth from that criticism. From what was said by the Post Office’s leading counsel to Master Fontaine at the very first case management hearing, those within the Post Office had relied upon Fujitsu in obtaining relevant information for its earlier public statements, and it plainly relied upon Fujitsu in terms of evidence of fact in the Horizon Issues trial, because it called a number of Fujitsu witnesses. I am unaware of the degree to which the Post Office has depended upon Fujitsu in the background to the Horizon Issues trial, and there is no reason why I should be, and I make it clear that I do not speculate on this. This information must have originated from Fujitsu at some point, but the Post Office must bear some responsibility for such incorrect statements having been made before, both publicly and in its pleadings.

544.

Nor could it be said, in the Post Office’s and/or Fujitsu’s defence, that remote access was a side issue. It has been a very important central element of the whole dispute between SPMs and the Post Office over the Horizon system for many years. Mr Green for the claimants adopted a non-computing analogy, namely that of “a back door”. The origin of this was a phrase of Mr Roll, who referred to the back door to the system. The branch accounts can be likened to a locked room, to which the SPM has the key. This development in the case is comparable, after many years of the Post Office and Fujitsu maintaining that only the SPM has the key, that only the SPM could perform anything within the locked room, and if the Post Office and/or Fujitsu

needed access to the room the SPM would know because they would have to borrow the key, to now being an admission by the Post Office (and Fujitsu) (and a finding of fact by the court) that there is a back door which they could, and did, use without the SPM even knowing such a door existed. This analogy is not entirely apt, because software and remote access to edit transactions is not completely the same as physical access through a door, but it is an attempt to capture the general idea in lay terms.

545.

The truth concerning remote access has now emerged in 2019, in group litigation that started in 2017. I find it notable that the truth did not emerge in the first Fujitsu witness statements that were originally served for the Horizon Issues trial. Such statements stand as the evidence in chief of witnesses of fact. They are supposed to be accurate. Minor corrections are not unusual and indeed are almost expected, as a trial approaches, as witnesses either research further or remember (when preparing for trial) other minor details. This topic, however, did not undergo that type of correction, and is a subject far more central and important than that. The truth only finally emerged in later statements, which were required to correct what I find were directly inaccurate statements in the first witness statements of Mr Godeseth and Mr Parker. There has been no adequate explanation for the contents of those first statements, which not only omitted this important fact, but contained evidence directly to the contrary. Those first witness statements were misleading. The statement in the Defence was misleading too. It ought also to be noted that the truth did not emerge internally within the Post Office in the email answers provided to internal inquiries in 2015 by senior Post Office personnel, such as the Chief Executive, who posed the specific question in preparation for providing evidence to a Select Committee and asked: “What is the true answer?”

546.

She also said in the same email “I hope it is that we know this is not possible and that we are able to explain why that is”. The true answer is that, contrary to her aspiration, it was possible.

547.

She also stated “I need to say no it is not possible and that we are sure of this because of xxx and that we know this because we have had the system assured.” The true answer to that was also “yes, it is possible”.

548.

It is also difficult to believe that the signatory of the Statement of Truth in the Generic Defence would have signed that statement if she did not believe the contents of that pleading to be true. There is another relevant feature in this case, which is that the Generic Defence also includes a counterclaim seeking damages against the SPM claimants, including damages for fraud. Fraud is the most serious allegation that can be brought in civil litigation and there are special rules in relation to pleading it, which means that a pleading containing a fraud allegation should be subject to particular scrutiny before it is served.

549.

It may therefore be that the Post Office itself fell into error as a result of information provided to it by Fujitsu on this important matter. It may be that some within the Post Office were themselves surprised by these revelations prior to, and during, the Horizon Issues trial. There is no need for me to speculate on this, and I do not do so. Certainly Mr Godeseth did not appear to have known about this for very long. Whatever the origin of this, and whether it came from Fujitsu, internally, being less than frank with the Post Office or not, the effect is that the Post Office has made specific and factually incorrect statements about what could be done with, or to, branch accounts in terms of remote access without the knowledge of the SPM. The evidence in this trial has made it clear that such remote access to branch accounts does exist; such remote access is possible by employees within Fujitsu; it does exist specifically by design; and it has been used in the past.

550.

It follows that the previously stated public position of the Post Office to the contrary, in the statements to which I have referred above, is specifically wrong in fact.

551.

Further, and as part of the Post Office serving supplementary witness statements from both Mr Godeseth and Mr Parker in respect of Mr Roll’s evidence, the Fujitsu witnesses adopted a position in the litigation, the rationale of which is somewhat difficult to understand. Responsibility for the incorrect and potentially misleading evidence of Mr Godeseth and Mr Parker (in their first statements) was effectively laid at the door of Mr Roll for giving what was said to be vague evidence, or evidence that, it was said, had not been fully understood by Fujitsu. Infallibility is a rare commodity, and everyone is capable of making mistakes. However, it is how one reacts to mistakes that is telling. In this instance, the initial reaction of the Fujitsu witnesses was to seek to shift the blame for their own misleading written evidence upon someone else. In this case, that “someone else” was their former Fujitsu colleague whose very evidence was responsible for exposing the full picture. This is not something that could have been arrived at lightly. To be fair to Mr de Garr Robinson, it was not a point he adopted on the Post Office’s behalf during the Horizon Issues trial. In my judgment, he was entirely sensible in not doing so. That it was done at all in the written Fujitsu statements speaks volumes, in my judgment, about Fujitsu’s reaction to being caught out.

552.

Mr Parker’s supplementary statement blaming Mr Roll said the following:

“In paragraph 20 of Roll 2, Mr Roll describes a process by which transactions could be inserted via individual branch counters by using the correspondence server to piggy back through the gateway. He has not previously made this point clear. Now that he has, following a discussion with colleagues who performed such actions I can confirm that this was possible. I did not mention it in my first witness statement because, when faced with a less clear account in Mr Roll's first statement, my recollection was that if it was necessary for the SSC to inject a transaction data into a branch's accounts, it would have been injected into the correspondence server (injecting via the server was the default option which was followed in the vast majority of cases).”

553.

The recusal application was issued the day after Mr Godeseth’s cross-examination had made it clear, not only that this remote access existed, but after he was taken in careful cross-examination through specific examples of Fujitsu personnel manipulating branch accounts, and leading to discrepancies in branch accounts. I am aware that criticism of the Post Office and Fujitsu in this respect may prove to be controversial, as earlier criticism of certain aspects of the Post Office’s case in Judgment (No.3) was not well received by it. However, if criticism is justified, I consider it would be detrimental to proper resolution of the group litigation if that criticism were to be withheld simply because it might lead to a further negative reaction by the Post Office. It is also an inherent part of the judicial function in any litigation to make findings, which may include criticisms where justified, that may be contrary to a litigant’s own view of the merits of their case. Some litigants are so convinced of the righteousness of their own position that they consistently refuse to accept any possible view of the litigation other than their own. Such a blinkered view is rarely helpful, and would be particularly unhelpful from a publicly owned institution.

554.

I consider that criticism is justified of the Post Office’s incorrect previous statements – which included public and high profile statements such as the response to the BBC Panorama programme, whatever may have been said on the subject to the Select Committee, as well as in its own pleadings - about remote access without the knowledge of SPMs. The Post Office should have done its best to discern whether such remote access was possible when this subject first arose, and whether it had occurred, before these statements were made. Fujitsu should have been frank and unequivocal, internally, with the Post Office, so that there could be no possibility of incorrect statements on this important point being made publicly by the Post Office. The Fujitsu witnesses should not, in their first witness statements, have made the incorrect statements that they did. Had those initial statements been factually accurate, there would have been no need for the supplementary statements from them that eventually led to the true position being accepted by Fujitsu, and therefore by the Post Office. It is highly regrettable that such a situation as this should have developed. I am not making any findings on any future issues, whether of fact or law, yet to be tried.

555.

There is however considerably more to resolving the Horizon Issues than simply making findings about remote access to branch accounts.

So-called “shadow experts” instructed by the Post Office

556.

I return to the issue of so-called “shadow experts.” The parties in this case agreed that costs management would apply, and the Post Office in its costs budget for the Horizon Issues trial included, for the costs management hearing on 5 June 2018, an item for incurred expert costs in its costs budget in the sizeable amount of approximately £800,000. This was in addition to the amount incurred by that point in terms of the fees of Dr Worden. His costs at that stage were only £58,000. The sum of £800,000 broke down to about £300,000 paid to Fujitsu, and £500,000 paid to other experts, who were not being instructed to give expert evidence. Dr Worden was the expert who would be giving evidence to the court, which meant he would owe the relevant duties of independence to the court under the CPR. The description of these other experts was given in a skeleton argument for the Post Office in the following terms:

“The Defendant has spent around £500,000 on investigations by internally appointed experts for the purposes of determining its litigation strategy. The resulting material - which is privileged - has not been provided to the Defendant's expert for the purposes of this litigation.”

557.

The claimants adopted the term “shadow experts” for these, a term with which the Post Office did not agree. At the cost management hearings, the claimants pointed out that by “internally instructed” this meant these experts were not instructed by the Post Office’s solicitors. It was also pointed out that there was no corresponding entry in the budget for any conference with any of their counsel, and this meant that this was “an entirely ring-fenced operation”. The Post Office did, however, reserve the right to recover their costs as costs in the litigation, and therefore this item was included in its costs budget. All of the material produced by these other experts was said to be privileged.

558.

It is a highly unusual situation that entirely separate experts, instructed directly by a party, without the involvement either of that party’s solicitors or their counsel (all the more so when those experts are not even identified), are instructed on such a task, whatever that task might actually be. I recorded an adverse comment in the costs management order of 23 July 2018 stating that the Post Office’s incurred costs for experts were extraordinarily high, unreasonable and disproportionate; and that Fujitsu’s costs of assisting with the litigation, and the costs of these internally appointed experts did not, on the face of it, appear to be properly recoverable sums in the litigation. I should clarify that this adverse comment should not be taken as applying to Dr Worden’s costs that had been incurred at that stage, which were modest. It is not necessary to deal with this matter any further to resolve the Horizon Issues.

F. Documents and Available Information

559.

There are certain categories or descriptions of classes of documents that have featured heavily in the evidence at the Horizon Issues trial. The path to disclosing them has not always been smooth. The majority, if not all, of the technical documents that relate to how Horizon was actually operating in fact in IT terms are in the possession of either the Post Office or (more usually) Fujitsu. The two most important categories, in my judgment, are Known Error Logs (also known as “KELs”) and PEAKs. The first of these records or logs known errors, which means errors with the Horizon system. The latter is a browser-based software incident and problem management system used by Fujitsu for the Post Office account, in other words for incidents and problems associated with Horizon that occur.

560.

Disclosure had previously been a troublesome topic between these parties. I had in Judgment (No.3) on the Common Issues, been somewhat critical of certain aspects of the Post Office’s approach to the disclosure of some documents. These observations were not well received by the Post Office and some had been categorised, incorrectly, by Mr Parsons in his 14th witness statement supporting the recusal application as “critical invective”. Whether the contents of Judgment (No.3) encouraged the claimants to mount, or continue to advance, criticisms of the Post Office’s disclosure in the Horizon Issues trial (in the sense that they may have felt they had a “following wind”) is not clear. The Post Office, in the Horizon Issues trial (predominantly in solicitors’ correspondence, but also in some submissions), also criticised the claimants’ disclosure. For the sake of clarity, I wish to make it clear that the claimants had no such following wind.

561.

The Horizon Issues are decided in this judgment based on all the evidence and the submissions. I deal with disclosure in the subsequent paragraphs of this judgment in order to assist and guide the parties concerning the future conduct of the group litigation. Some issues, such as whether certain Fujitsu documents were within the Post Office’s control, are likely to crop up again in respect of other types of documents in the future. The court’s interest in disclosure is to promote compliance

with the rules, and also to ensure cost effective progress of the litigation. The answers to the substantive Horizon Issues are no different because, for example, the experts were not given particular documents relating to the specific bugs in good time. For the most part, large numbers of KELs and PEAKs were provided to them, and the experts had an abundance of material. Some 5,000 KELs emerged from Fujitsu well after the trial ended, in October 2019. The experts did not consider these as they were not given them. Mr Parker had given evidence (at paragraph 61.9 of his 1st witness statement) that 1,491 KELs had been deleted at the time of writing that statement, but there was no reference to the existence of this far greater number that emerged after the trial was over.

562.

The Post Office’s written closing submissions also included a separate section on disclosure as follows, which I will quote verbatim:

“I6. The Court’s interventions on disclosure

1145.

Post Office is concerned that the Court may have lost sight of the nature of the Orders made and of the approach which the Court ordered. It is striking that Cs have not themselves made any applications for specific disclosure nor have they advanced any complaint that particular disclosure orders have not been complied with.

1146.

The recent judgment of the Court of Appeal in Serafin v Malkiewicz & ors

[2019] EWCA Civ 852 @ para 118 is relevant:

“We are also highly troubled by the repeated demands and criticisms by the Judge regarding the Claimant’s disclosure, in circumstances where pre-trial disclosure had been completed by both sides at a time when both the Claimant and Defendants had been represented by solicitors and counsel, and no application for further disclosure had been made by the Defendants…” ”

563.

These submissions identified what were said to have been “the Court’s interventions on disclosure” in the Horizon Issues trial. The authority identified by the Post Office, and stated to be relevant to that submission, concerned a defamation case in which the judge had demanded disclosure of a litigant in person, and made criticisms of him, both during the trial of that litigant’s libel claim, and in the judgment itself. In some instances the court in that case had, of its own volition, demanded production of certain documents from the litigant in person, and within a very short timescale. This had been part of findings made by the Court of Appeal in respect of procedural irregularity and the successful appeal against the judgment.

564.

There were no “interventions on disclosure” by the court during the Horizon Issues trial, nor were there “demands and criticisms”. When asked during oral closing submissions about this passage in the Post Office’s written submissions citing Serafin, Mr de Garr Robinson did not demonstrate a great deal, if any, enthusiasm in advancing any claims of procedural irregularity having occurred during the Horizon Issues trial. He submitted that “this is not that case” – which does raise the question of why the passages from Serafin were included in the Post Office’s submissions in the first place.

565.

There were, however the following matters that arose in this litigation during the Horizon Issues trial, which given paragraphs 1145 and 1146 of the Post Office’s closing submissions, should be recorded:

1.

Redactions. In Judgment (No.3) I had identified the issue of unjustified redactions made by the Post Office of some contemporaneous documents that were used in the Common Issues trial. During the early part of the Horizon Issues trial some documents were referred to which again had redactions, which due to their nonredacted parts did not seem to me to be of a type that would necessarily attract privilege. In view of the previous history of this subject, I asked leading counsel for the Post Office to perform his own review of the redactions that had at that point been made of documents deployed in the Horizon Issues trial. This review resulted in some of the redactions in some of the documents being removed, and unredacted versions of those documents becoming available to the claimants, some of which were then used in cross-examination. Redactions of 2 documents were maintained on the grounds of legal professional privilege, which is entirely conventional. He also provided a helpful two page note explaining the approach that had been adopted. Asking counsel to do a review of this nature in these circumstances is entirely conventional. There was no short timescale imposed for this to be done. I asked for a summary of the position on 4 June 2019, which was after the interval which occurred as a result of the recusal application, and about 2 ½ months after I had asked him to perform the review. It was an outstanding matter, of a type sometimes referred to in a trial as “housekeeping”, which had not been entirely forgotten, but which had been overtaken in the latter weeks of March when the trial was temporarily halted.

2.

A witness statement was ordered from the Post Office to explain the express, and factually incorrect, submissions made to the court by the Post Office about the Royal Mail’s refusal to produce, at the Post Office’s request, audit documents. The Royal Mail had, contrary to what the court had expressly been told by the Post Office, not even been asked by the Post Office for these. The Post Office’s leading counsel accepted that he had, entirely unwittingly, and on instruction, provided misleading information to the court and explained and apologised as soon as he discovered this. A witness statement from the Post Office, which was provided by Mr Parsons, was entirely justified in these circumstances.

3.

A witness statement was also ordered from the Post Office in respect of the production of over 2,000 documents in disclosure by the Post Office in the interval in the trial period between 11 April 2019 (completion of the factual evidence), and the experts commencing their oral evidence on 4 June 2019. Given the timing and quantity of these, in the circumstances a witness statement (again, being provided by Mr Parsons) was justified. The first explanation given to the court, again on instruction, again proved to be wrong. These documents were originally said to have all been of some age (ten years or so) and discovered at Fujitsu on a “long-forgotten” server. One deployed the day after this explanation in cross-examination turned out to be dated 21 August 2018. A witness statement was again called for.

566.

None of the above, in my judgment, fall into the category of the court losing sight of the nature of the orders that had been made concerning disclosure, or of demands or criticisms by the court, unjustified or otherwise. Further, the claimants made numerous complaints of the Post Office about disclosure. Indeed, the factually incorrect submissions made to the court about the audit documents and the Royal Mail came at the end of a court day when the claimants, by their leading counsel, expressly attempted to make a specific disclosure application. I was not prepared to hear that application at that point. Royal Mail was the correct respondent to such an application, not the Post Office. Royal Mail was not represented, nor even present in court, nor had any notice been given by the claimants to the Royal Mail of such an application. Each of these features is a far from promising start. I did however make directions for the making of a third party disclosure application against the Royal Mail, but this did not ultimately prove to be required. Once those at the Royal Mail were actually asked for these documents, the Royal Mail readily produced them. The claimants also made regular complaints about disclosure and would often point out that documents had been produced during the hearing. This is because documents were often produced during the hearing. This is something that had also occurred during the Common Issues trial.

567.

The Post Office is, however, correct to point out that there have been no specific disclosure applications mounted by the claimants, with the exception of the one to which I refer in [566] above in respect of the Royal Mail. General complaints about disclosure only take a party so far, and that is not very far, if deficiencies are known about in advance of a trial, and no applications are made.

568.

The parties had also voluntarily adopted Model C of the Disclosure Pilot, which I approved. This was done prior to the coming into force of the Pilot, the full title of which is Disclosure Pilot for the Business and Property Courts (“BPC”) which became mandatory in all BPC cases where the first CMC is held after 1 January 2019. This group litigation is not a BPC case and the first CMC was held before that date in any event. The Pilot is designed to run for a two year period. The Pilot is found at Practice Direction 51U, which is made under CPR Part 51.2. The features of the Pilot that are particularly relevant here are the Principles included in paragraph 2 of the Pilot. These state:

2. Principles, “document”, “adverse” and “known adverse documents”

2.7 Disclosure extends to “adverse” documents. A document is “adverse” if it or any information it contains contradicts or materially damages the disclosing party’s contention or version of events on an issue in dispute, or supports the contention or version of events of an opposing party on an issue in dispute.

2.8 “Known adverse documents” are documents (other than privileged documents) that a party is actually aware (without undertaking any further search for documents than it has already undertaken or caused to be undertaken) both

(a) are or were previously within its control and (b) are adverse.”

3.1 A person who knows that it is or may become a party to proceedings that have been commenced or who knows that it may become a party to proceedings that may be commenced is under the following duties (“the Disclosure Duties”) to the court—

(1)

to take reasonable steps to preserve documents in its control that may be relevant to any issue in the proceedings;

(2)

once proceedings have commenced against it or by it and in accordance with the provisions of this pilot scheme, to disclose, regardless of any order for disclosure made, known adverse documents, unless they are privileged;”

(emphasis added)

569.

The fact that the parties chose Model C does not go to answer the complaints that the claimants have made about the Post Office’s disclosure. The Post Office had an obligation to disclose known adverse documents, as do the claimants as well. “Adverse” as defined plainly includes any documents that refer to bugs, errors or defects, or the operation of the Horizon system that led to potential impact upon branch accounts. This obligation is “regardless of any order for disclosure made”, in other words it exists whether any specific classes of documents are, or are not, already the subject of any order. The rational for this is obvious. Reliance upon the adoption of Model C is not therefore the answer to the claimants’ complaints in this respect. However, the fact that the claimants did not make applications for disclosure is relevant to the criticisms made at the Horizon Issues trial.

570.

The Post Office has, essentially, submitted that the breadth of disclosure requests has been too wide. Particular criticism is that “Freeths’ [the claimants’ solicitors] did not control Mr Coyne’s requests”. I am not persuaded that it is the correct approach to matters of independent expert evidence to seek to “control” the documents that an independent expert wishes to see. The claimants did not issue any opposed applications for disclosure of documents which their expert wished to see, so a filter of some sort was applied in that sense to the documents Mr Coyne wished to see, and was not given.

571.

Finally, disclosure is a particular feature of litigation in this jurisdiction. However, there was a clear perception in commercial litigation generally that the then-current disclosure regime had become inadequate and also that standard disclosure was often excessive in scale, cost and complexity. This is why the Disclosure Working Group was created in May 2016 at the initiative of Sir Terence Etherton, then the Chancellor of the High Court and now the Master of the Rolls. This led to the Disclosure Pilot.

572.

It would be highly regrettable if disclosure in this case were to become yet another battleground between the parties. There are more than enough points in issue between the parties without adding endless disagreements about disclosure to that list. There are the following areas, however, of disclosure which should be recorded. In my judgment, the background to the evolution of the following categories does throw light on the parties’ different approaches to the litigation.

The Known Error Log or KELs

573.

KELs are sometimes referred to in the singular – for example, in some places reference is made to the Known Error Log, as though it were a single Log. In practice, the Log as a whole includes multiple KELs, as they were referred to during the trial. If an incident or problem occurs, a PEAK is raised, and if the problem or incident refers to an existing issue which is known to have occurred before, reference will be made to that KEL entry, or more usually, it is said that the PEAK will refer to the KEL. Both

PEAKs and KELs have reference numbers; the former are usually all numbers after the prefix PC, whereas KELs have letters (taken from the identity of the employee who raised the KEL followed by four digits and a letter). Therefore, when a problem (which I shall refer to generally as an example as problem x) occurs for the first time, a KEL will be raised which will be given a reference. Subsequent PEAKs which identify the same problem x again, over time, afterwards will all refer to the same KEL. Sometimes a PEAK will be identified as including a problem which is similar to problems identified in more than one KEL, in which case both the KELs will be referred to within that PEAK. Any creation of a new KEL, or updating of an existing KEL, must be authorised by SSC before it can be seen by all users.

574.

The experts agreed the following in the 2nd Joint Statement about both PEAKs and KELs: “KELs and PEAKs together form a useful source of information about bugs in Horizon but are a limited window on what happened. It is sometimes necessary to use evidence from both to try to understand, but even so they are not a comprehensive picture. It is to be expected that both KELs and PEAKs are incomplete in various respects.” It is also agreed that KELs will often give information about the impact of a bug or user error and they may also provide information about causes. There are other documents referred to as OCPs and OCRs. They are Operational Change Proposals and Operational Change Requests.

575.

The subject matter of this litigation spans many years, starting from the first implementation of the Horizon system (what is now being called Legacy Horizon) in 2000. In the Letter of Claim from the claimants dated 28 April 2016, the Known Error Log was sought from the Post Office, that letter stating:

“We understand that Fujitsu maintained a 'Known Error Log' for Horizon and that such reports will have been provided to Post Office. Please see the list of the categories of documents relating to Fujitsu referred to below, that we request disclosure of.” Item 22 in the list of documents sought was “The 'known error log' kept by Fujitsu and provided to Post Office as referred to above, and all correspondence relating to the same.”

576.

The answer in a letter from the Post Office’s solicitors against the specific item 22 was:

“In circumstances where you have not particularised any factual basis on which Horizon is defective, disclosure of these documents (if they exist) is not relevant, reasonable or proportionate.”

577.

The suggestion in that letter that the Known Error Log was not relevant, is simply wrong, and in my judgment, entirely without any rational basis. The further suggestion, viewed with the hindsight now available, that the “known error log” may not exist, is disturbing. The claimants’ request used the precise title – “known error log” – and this clearly did exist. To suggest in an answer “if they exist” is somewhat misleading.

578.

Item 23 in the same list of documents sought was “Internal memoranda from Fujitsu and POL referred to by Second Sight as identifying a `Horizon bug' within Horizon Online.”

579.

The answer against that item was:

“We do not recognise the document to which you refer. Please provide further details.”

580.

In my judgment, the documents sought in that entry must clearly include any PEAKs that identified a bug within Horizon Online. “Internal memoranda” is a plural reference, yet it was interpreted by the Post Office’s solicitors as though it were singular, and the request was for a single document, or a document with the title “internal memoranda”. This is, in my judgment, obstructive.

581.

The claimants were not to be dissuaded, and sought the Known Error Log or KELs again. A reply from the Post Office’s solicitors on 13 October 2016 is relied upon by the claimants as showing that the Post Office was denying the relevance of the Known Error Log. This reply stated:

“The claims which you have particularised concern errors with the Core Audit Log. Following a review of the Known Error Log, Fujitsu have confirmed that there have been no logs in respect of Core Audit Log. The remainder of the Known Error Log does not relate to the claim which you have particularised and as such disclosure of this document is not relevant.”

(emphasis added)

582.

Existence of the Known Error Log was at that stage accepted, but its relevance to the proceedings was now challenged. The Post Office’s solicitors stated that its contents “did not relate to the claim” and that “disclosure of this document is not relevant”. Disclosure of it was plainly resisted. The claimants did not therefore have it when the Generic Particulars of Claim was pleaded on 6 July 2017. In the Generic Defence, which is dated 18 July 2017, the Post Office changed its position, and now pleaded that the Known Error Log was not in its control. At paragraph 50(4) of the Generic Defence, the Post Office stated:

It is admitted that Fujitsu maintain a "Known Error Log". This is not used by Post Office and nor is it in Post Office's control. To the best of Post Office's information and belief, the Known Error Log is a knowledge base document used by Fujitsu which explains how to deal with, or work around, minor issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide fixes have not been developed and implemented. It is not a record of software coding errors or bugs for which system-wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief, there is no issue in the Known Error Log that could affect the accuracy of a branch's accounts or the secure transmission and storage of transaction data.”

(emphasis added)

583.

This was expressly challenged in paragraphs 3 and 4 of the Generic Reply.

584.

The two main points to consider within that passage in the Defence are that the Post Office stated that it did not have control of the Known Error Log, and that entries within it related to minor/trivial matters and could not affect the accuracy of a branch’s accounts. This position was maintained, notwithstanding the terms of the

Post Office’s skeleton argument for the first case management hearing on 19 October 2017. That had stated:

“Cs’ response to the criticism of their case on Horizon is to argue that they do not have the access to information and documents that would allow them to plead any properly articulated case as to the “bugs” that they wish to blame for the shortfalls in their branches. Without prejudice to its position that the case is not properly pleadable, even at a generic level, Post Office is prepared to take steps to help Cs investigate Horizon, take an informed view as to whether they really wish to maintain their claims and, if so, decide how to proceed with them. Post Office wishes to put an end to the speculative attacks on Horizon and the related allegations as to its own conduct. It puts forward its proposals in paragraphs 2 to 8 of the Draft Order as a pragmatic compromise between the parties’ competing interests and concerns.” (italics present in original, emphasis by underlining added)

585.

At the first case management hearing itself, upon specific enquiry by the court, this position was further maintained.

586.

Disclosure of the Known Error Log was resisted by the Post Office on the grounds both of control, and content. So far as control is concerned, Mr Parsons had provided his 4th witness statement for that hearing, the first before me as the managing judge, which expressly stated at paragraph 35 “Despite Post Office explaining the irrelevance of the Known Error Log and that it was not within Post Office’s control…” and also at paragraph 37 “Due to the large amount of information involved and the fact that the Known Error Log is not in Post Office's control….”. This position was confirmed by the Post Office through its leading counsel in the following exchange with the court at that hearing, the transcript of that showing the following exchange.

“Mr Justice Fraser: Do you still maintain it is not in your control?

A: My Lord, yes. It’s a Fujitsu document.

Mr Justice Fraser: No. Just because it’s a Fujitsu document doesn’t mean it’s not in your control.

A: That’s right. At no point has Post Office ever suggested that it’s in its control so far as I’m aware in any of the letters that I – Mr Justice Fraser: Okay.

A: (Reading from pleading)

"To the best of Post Office's information and belief the known error log is a knowledge based document used by Fujitsu which explains how to deal with or workaround minor issues that can sometimes arise in Horizon for which often because of their triviality system-wide fixes have not been developed and implemented. It is not a record of software coding errors or bugs for which system-wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief there is no issue in the known error log that could affect the accuracy of branch accounts or the secure transmission and storage of transaction data."

587.

This was the relevant passage in the Generic Defence that was read out and Post Office’s leading counsel then continued:

“It will be clear from what I've just read to your Lordship that Post Office thinks that this is a complete red herring.”

588.

The position was then, again, expressly confirmed as follows.

“Mr Justice Fraser: I think it's going to be useful for me, certainly, for you to define all the different grounds upon which you currently resist disclosure of the known error log. Number (1) is you say it's not in your control.

A: Control.

Mr Justice Fraser: Number (2) you say the subject matter is not relevant. Is that right? A: My Lord, yes. An order that Post Office disclose documents relating to bugs and errors causing the branch account errors, if Post Office had control of the document and it dutifully complied with that order it would not be disclosing the known error log because, as far as it is aware, the known error log is not that kind of document.”

589.

So far as the content of the Known Error Log is concerned, when asked by the court “does it contain any errors at all?” the following answer was given by the Post Office’s leading counsel:

“It contains things like there's a problem with printers. There's a printer. You have to kick it on the left-hand side to make the printer work. I mean there's a vast range of hardware problems of that sort and maybe some software problems …. but not the kind of bugs, errors and defects that the claimants are wishing to pursue in their particulars of claim so far as Post Office is aware.”

590.

This exchange with the court then continued.

“Mr Justice Fraser: Well, that's the rider which slightly might concern---

A: Well, my Lord, unfortunately, that's all that Post Office can say because it's not Post Office's document. It's Fujitsu's document. Fujitsu are the experts.”

591.

Given there was no witness statement from anyone at Fujitsu available on the subject at that hearing, that was as far as the matter could be taken on that occasion in terms of making any specific order relating to the Known Error Log. The Post Office had made clear that any order for disclosure that did not specify the Known Error Log, but was more general requiring disclosure of documents in relation to bugs and errors causing branch account errors, would not lead to disclosure of the Known Error Log in any event. A practical way forward, without requiring a fully argued separate application which would have included Fujitsu, was adopted whereby the IT experts were permitted to inspect the Known Error Log. The status of the Known Error Log as a disclosable document was resisted both on the grounds of control and relevance. It has to be acknowledged that this practical solution was one suggested by Mr de Garr Robinson.

592.

It must therefore have come as a surprise to the Post Office, given the contents of its pleadings and the express submissions that it had made to the court, that both experts considered the Known Error Log to be highly relevant. This should also have led to the Post Office beginning to doubt what it was being told by Fujitsu, given the source of what the court was told about this was what Fujitsu had told the Post Office, as set out above. The explanation of what the Known Error Log was, what it contained, and its lack of relevance, was not remotely accurate.

593.

Although a certain number of entries in the Known Error Log, which led to a significant number of different entries which are each called KELs being deployed in the trial and examined closely, were thereafter available, the issue of control of the Known Error Log and the Post Office’s earlier position resisting disclosure of this, did not go away. An entire appendix to the claimants’ opening submissions was devoted to criticism of the Post Office on disclosure, including its shifting position on the Known Error Log. In closing submissions, at the very end of the final day of the Horizon Issues trial, the claimants handed up a document entitled “Claimants’ Points in Reply”. Part F of that was a short document headed “KELs said not to be in the Post Office’s control” together with a variety of references. It ought also to be recorded here that some 5,000 KELs were later disclosed by the Post Office in October 2019, well after the trial ended, once the Post Office was told by Fujitsu that previous entries, which Fujitsu had previously told the Post Office were not retained, were in fact retained.

594.

The subject of whether the Post Office had control of the Known Error Log therefore remained somewhat live. It simply was not, therefore, going to go away. The Post Office’s legal team had loaded version 12 of its contract with Fujitsu (“the Fujitsu contract”) onto the electronic bundle database on the last day of evidence, Day 20, for re-examination of its expert witness, Dr Worden. The contractual situation between the Post Office and Fujitsu was therefore before the court, which it had not been for the first case management hearing on 19 October 2017. It is also the case that “control” of documents (other than the KELs) held by Fujitsu could potentially crop up again in this litigation going forwards. It is cost effective and efficient to resolve that now, for the assistance of the parties.

595.

I therefore gave both parties the opportunity to lodge further written post-hearing submissions restricted to whether the Known Error Log was in the Post Office’s control. It was a point relied upon by the claimants and it has to be resolved. There were two reasons for this. One is that in group litigation of this type, the parties’ approach to disclosure goes beyond a single document, or type of document. Guidance as to the court’s general approach will be of assistance in the litigation going forwards. There may well be other documents held by Fujitsu that need to be disclosed later, and the issue of control needs resolving. Secondly, the court is entitled to expect accurate evidence from parties on interlocutory matters, and accurate submissions.

596.

The Fujitsu contract that was loaded onto the electronic bundle was a recent one. This document has a version history at the front which shows that it runs, with amendments therein identified, from 31 August 2006 to version 12 on 3 July 2017. Version 12 pre-dates the date of the Generic Defence, but in any event, clause 25 (which appears below at [602]) does not appear against any of the later version numbers in the version history of the Fujitsu contract. It is likely therefore that clause 25 in substantially the same (if not identical) form appeared in all the earlier versions of the Fujitsu contract prior to Version 12.

597.

The claimants made the requested post-hearing written submissions based on the meaning of ‘control’, for the purposes of CPR Part 31.8, which is uncontroversial.

598.

That rule provides as follows:

“(1) A party’s duty to disclose documents is limited to documents which are or have been in his control.

(2) For this purpose a party has or has had a document in his control if – (a) it is or was in his physical possession;

(b)

he has or has had a right to possession of it; or

(c)

he has or has had a right to inspect or take copies of it.”

599.

Sub-paragraphs (a) to (c) are not exhaustive, but collectively those two subparagraphs mean, in this case concerning this document or documents, that if the Post Office has, or had, a right to possession of the Known Error Log, and/or a right to inspect or take or be provided with copies of it, it is within the Post Office’s control. Control will be established if a party has a contractual right to inspect or take copies of the document or documents in question. Given the terms of the Fujitsu contract, the claimants submitted that the Known Error Log is in the Post Office’s control and has been at all material times.

600.

The Post Office’s written submissions on this were also received. These conceded, as it was put in paragraph 3, that the KEL was in its control. That paragraph stated at sub-paragraph (1):

“As at July 2017, Post Office understood the KEL to be a relatively unimportant internal working document produced and used by Fujitsu to assist in the performance of some services under the contract. On that basis, the KEL would not properly have been characterised as a “record…relating to the performance” of Fujitsu’s services under the Fujitsu Contract and would thus not have been within Post Office’s control. But, on the facts now known to Post Office, it would not contend the KEL to be outside its control.”

(emphasis added)

601.

The title of the log – the Known Error Log, or known error log – includes rather obviously the two words “known” and “errors”. The Horizon Issues use the expression “bugs, errors and defects”. The presence of the word “error” in both might give an early indication of likely relevance of the log. Further, as shown in [603] below, in the Fujitsu contract the term records includes the phrase “full and accurate records” (emphasis added). Although records is used in the definition of Records – a point correctly identified by the Post Office, which it says is largely circular – it is clear that “full and accurate records” are included.

602.

Clause 25 in version 12 of the Fujitsu contract states the following:

“25.8 In addition to its obligations under Clauses 25.2 and 25.3, Fujitsu Services shall provide the Court Case Support Services to Post Office in relation to prosecutions and other disputes (whether civil or criminal) with any third party including but not limited to any fraud, theft, breach of contract or impropriety (the “Court Case Support Services”). The Court Case Support Services shall include any matters whether they relate to Horizon, HNG-X or any other system provided by or on behalf of Fujitsu Services to Post Office, its agents or its subcontractors (including Post Office Service

Integrator and any Tower Contractor). Fujitsu Services shall provide the Court Case Support Services within the timeframes required by Post Office or the relevant court or other authority.

25.9 Without prejudice to Clause 25.3, the Court Case Support Services shall comprise:

25.9.1

the provision of copy reports;

25.9.2

the provision of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data) where such data is held by or in the control of Fujitsu Services;

25.9.3

the compilation of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data);

25.9.4

the interpretation of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data);

25.9.5

the provision of technical reports regarding technical aspects of any system (whether Horizon, HNG-X or otherwise);

25.9.6

live witness evidence at Court if any of the information provided (including without limitation that provided pursuant to Clauses 25.9.1 to 25.9.5) is challenged to the extent to which Fujitsu Services provided said information; and

25.9.7

the right of access to Records, including but not limited to information, reports and data, held by or in the control of Fujitsu Services, and the assistance of Fujitsu Services’ personnel with appropriate knowledge of the applicable Records (to the extent any such personnel remain employed or contracted to Fujitsu Services) for any independent experts and/or legal advisors instructed by Post Office and/or any other claimant(s) or defendant(s) and the Prosecution in any mediation, arbitration tribunal, court case or dispute in which Post Office is involved in relation to the Horizon and HNG-X or any other system provided by or on behalf of Fujitsu Services to Post Office.”

(emphasis added)

603.

Records is defined in the Fujitsu contract in Schedule 1 “Interpretation”, in the following way:

“Records: means the full and accurate records relating to the performance of the Services.”

I consider that this term plainly includes the Known Error Log, as that relates to the performance of the Services, and in any event this contains data and hence also falls within clauses 25.9.2 and 3. I reject the lengthy analysis in the Post Office’s submissions on this point that the KEL is not properly characterised as being a record. It plainly is.

604.

In my judgment the Post Office clearly has, and had, a contractual right to be provided with the Known Error Log by Fujitsu, given these are civil proceedings for (inter alia) damages sought by the claimants for breach of contract, and also fraud being alleged against the claimants by the Post Office by way of counterclaim. I do

not accept that the KEL is a type of document covered by the authority cited by the Post Office at paragraph 11 of its submissions on this point, namely working papers prepared by professionals for their own assistance in carrying out expert work for their clients; Hanley v JC & A Solicitors [2018] EWHC 2592 (QB) at [42]. That point is , in my judgment, plainly wrong. Given the terms of the contract between the Post Office and Fujitsu, it is not necessary to consider that point further.

605.

I consider it verging on entirely unarguable, given the express terms of the Fujitsu contract which is now available to the court, that the Known Error Log was not in the control of the Post Office. Mr Parson’s witness statement had merely stated “it was not within Post Office’s control” and I simply cannot understand the basis for that statement, given the express terms of the contract that the Post Office had with Fujitsu. It plainly is in the Post Office’s control, given the terms I have reproduced above, and that point is now conceded.

606.

The fact that the Post Office has submitted that in July 2017, on its understanding then, the KEL was not a “record” that “related to the performance” of the Fujitsu services demonstrates a worrying lack of knowledge on the part of the Post Office, about both Horizon, and Fujitsu’s record keeping. It also means, when this is put together with what the Post Office, by its leading counsel, submitted to the court at the 1st case management conference, that Fujitsu were extraordinarily inaccurate about the information it provided to the Post Office at that stage of these proceedings. As at 2017, the Horizon system (both Legacy Horizon and Horizon Online) had been in use for about 17 years. KEL appears in a table of Abbreviations/Definitions at F/87/3 in a document entitled “CS Support Services Operations Manual” that is dated 29 January 2001. That document is referred to in footnote 34 of Mr Coyne’s 1st report, and three passages within it state the following:

4.5.1 Maintaining the Known Error Log on the SSC intranet site

The SSC generates and maintains a Known Error Log (KEL) system that uses searchable documents in HTML format. The mechanism for searching is a query entry in an intranet site. The KEL system is available to first, second, third and fourth line support units as well as SSC staff.

4.5.2 Transferring knowledge between support units

The SSC intranet site has KEL search facilities and other useful diagnostic data, documents and tools.

SSC and SMC staff raise KELs based on customer-observed symptoms. KELs are further maintained once the fault has been resolved.”

“4.7.1 Known Error Logs (KELs)

The intranet site holds known error details in Microsoft Word format, the contents of which may be searched for, in full text form. Documents are created to a defined template wherever possible. An application has been generated which limits the properties of the document to a subset of possible values, for clarity and ease of search. This application is made available to all support units.

The process for creating KEL entries outside of the SSC has not yet been formulated, but it is expected that no KEL will be allowed onto the system before it has been authorised by SSC staff.”

(bold present in original)

607.

The Post Office plainly did not know this when making the submissions that were made in 2017. The Services Operation Manual had been in existence for 16 years by then, and this ignorance about the Known Error Log, which is clearly explained within that document, is most surprising.

608.

Even without such an express obligation in the Fujitsu contract itself, I find it very difficult to believe that Fujitsu could mount any sort of coherent or legitimate objection to producing it at the request of the Post Office. The assertion that the Log was not in the control of the Post Office is, in my judgment, not a valid one.

609.

Similarly unsustainable is the submission that was made to the court in 2017 that its contents were not relevant to the Horizon Issues, and that it did not contain references to bugs and errors having an impact on branch accounts. On that latter point, the Post Office’s submissions were based upon what the Post Office had been told by Fujitsu. Fujitsu were, again, being clearly inaccurate about what they had told the Post Office was actually contained in the Known Error Log.

610.

Nor is the passage quoted above in the Generic Defence an accurate description of the Known Error Log. The Known Error Log does far more than “explain how to deal with, or work around, minor issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide fixes have not been developed and implemented”. The Known Error Log appears to be a comprehensive record of all the errors and defects of which Fujitsu have become aware over the life of Horizon. On the basis of the evidence that has been heard in the Horizon Issues trial, there are many issues in the Known Error Log that could affect the accuracy of a branch's accounts or the secure transmission and storage of transaction data, contrary to what is pleaded by the Post Office. Some KELs led to software fixes too. Indeed, Mr Coyne has used it in order to discover the existence of, and then investigate, the numerous different bugs from what was called “the Bug Table” in the 2nd Joint Statement, 11 of which Dr Worden agrees in his written evidence. Dr Worden himself has discovered some of these bugs, from the sample of the entries in the KELs that he had inspected. It is the KELs that are the origin of these expert conclusions, due to the text in the entries that they contain.

611.

This discovery of these different bugs by both experts has directly flowed from the entries in the Known Error Log by Fujitsu, together with the associated PEAKs.

612.

It is only fair to record that after contested disclosure had been considered at the first hearing before me on 19 October 2017 (dealt with in detail above), the Post Office’s leading counsel suggested the “pragmatic solution” that was giving Mr Coyne the opportunity to inspect it. It was because of this that the matter of control was not resolved at that point, as disclosure was permitted. Mr Coyne inspected the Known Error Log at Fujitsu’s premises. He found it highly relevant. The claimants received most of the KELs they had for the trial on 10 May 2018, with further KELs in respect of particular bugs disclosed on 16 November 2018, one month after the first expert

reports of Mr Coyne and Dr Worden had been served. Further KELs were provided in 2019, and a very substantial quantity of 5,000, well after the trial in October 2019. There has been no detailed explanation for that very late post-trial disclosure in any form of any witness statement from anyone at the Post Office or Fujitsu.

613.

The KELs provided to the experts prior to their reports featured centrally not only in Mr Coyne’s two reports, but those of Dr Worden too. They were also the subject of a great deal of cross-examination by both sides. I consider that disclosure of the KELs to the experts – the pragmatic solution suggested by leading counsel for the Post Office - has been central in the discovery and investigation of the bugs, errors and defects that the experts agree were, or are, present in the Horizon system, and also of the bugs, errors and defects that are not agreed, but upon which both experts opine and in respect of which I make findings in the Technical Appendix. Dr Worden has also analysed them extensively, and devoted a separate appendix of his report to them. The best description of the KELs is taken from Dr Worden, who stated that they were “a rich source of evidence”. The notion that they were not relevant, or did not contain relevant material, is extremely difficult to fathom, and I do not understand why, or how, Fujitsu would or could sensibly choose to inform its own customer of many years, the Post Office, directly to the contrary. If Fujitsu’s description of the contents of the KELs had been taken at face value and not challenged, then the knowledge now available to the two experts and to the court about the extent of bugs, errors and defects would not have been available, with the obvious detrimental impact upon the fair resolution of the Horizon Issues.

614.

The further developments, after the trial, that led to disclosure of another 5,000 KELs by the Post Office after the trial in October 2019 are dealt with at [627] below and following under the heading “Further post trial disclosure”.

PEAKs

615.

Turning to PEAKs, upon inspecting the KELs at Fujitsu’s premises in 2018, Mr Coyne discovered the existence of a different category of documents called PEAKs, as these were referred to within different KELs. These documents were requested from the Post Office in July 2018. The claimants maintain that the Post Office were obstructive over this, but whether that is correct or not, over 218,000 different PEAKs were disclosed by the Post Office’s solicitors on 27 September 2018. This was two weeks before the date for exchange of experts’ reports. A further 3,866 were disclosed on 25 October 2018 – two weeks after the experts’ first reports. OCP documents (which are mentioned in PEAKs) were disclosed on 24 January 2019, and a number of PEAKs were identified by Mr Coyne in his supplementary report which he had identified but which had not been disclosed. These were provided (or extracts of them) on 18 February 2019, which is only three weeks before the Horizon Issues trial started.

616.

A large quantity of further documentation was then disclosed by the Post Office, on dates that were not only well after the dates ordered for expert evidence to be served, but on dates that were even after the intended end date of the Horizon Issues trial itself. 2,590 OCRs were disclosed on 18 April 2019, and other PEAKs were disclosed on 31 May 2019, literally one working day before Mr Coyne was called to give evidence on 4 June 2019. This highly unsatisfactory situation led to my seeking an explanation. I consider an explanation was plainly required given the timescales. This is a large quantity of documents, dealing with central elements of the Horizon Issues, and the stage at which they had been provided was very late. PEAKs are important documents. I was told in submissions that Fujitsu had discovered an old database that had been copied more than ten years previously, and as soon as Fujitsu had told the Post Office this, matters had moved speedily and the Post Office had provided the contents of this old database to the claimants.

617.

However, the day after this explanation was given, and during cross-examination, a PEAK dated 21 August 2018 was put to Dr Worden and the court was told this was one of the documents that had been disclosed on 31 May 2019. This plainly could not have come from “an old database” in excess of ten years of age which Fujitsu had only recently discovered. A further and more accurate explanation was plainly called for.

618.

I had already ordered another witness statement to be provided by the Post Office’s solicitors, this one being the 18th witness statement of Mr Parsons, which was served on 19 June 2019. The contents of that witness statement were highly unsatisfactory. Firstly, it showed that the existence of the forgotten Fujitsu database came about because of questions raised by the Post Office’s solicitors “during preparation for the Horizon Issues trial at the beginning of April 2019”. The Horizon Issues trial in fact had an enforced interlude, from 21 March 2019 when the recusal application was issued, until that application was dismissed on 9 April 2019. In other words, those questions were asked very late in the trial process. Further, Mr Parsons’ witness statement stated, so far as the latter disclosure of PEAKs on 31 May 2019 is concerned, the following:

“20. In the course of investigations into the bugs referred to by Mr Coyne and/or Dr Worden, including in the Second Joint Statement, on 29 March 2019 Mr Lenton identified PCO273234 to Ms Simmonds as being a document relating to the Drop and Go Bug. For the next couple of months my firm continued to investigate these bugs and a number of further documents relating to these bugs came to light. PCO273234, along with 17 other documents (excluding redundant image files) that had come to my firm's attention when preparing for the Horizon Issues trial were disclosed in a single batch on 31 May 2019.”

(emphasis added)

619.

Mr Lenton is a member of Fujitsu personnel. The 2nd Joint Statement was signed by the two experts on 25 February 2019. This passage means that one month after that document, Mr Lenton identified the specific document relating to the Drop and Go Bug. This is one week after Mr Godeseth had completed his evidence. I find the rather glossed-over reference in the witness statement to “continuing investigations” that went on “for the next couple of months” extraordinarily opaque, in the context of a trial that commenced on 11 March 2019, and a document identified on 29 March 2019. I also find the wholesale lack of any explanation of when the “further documents relating to these bugs came to light” – and indeed how – puzzling, again in a witness statement ordered specifically to explain what was already a highly unusual situation. Further, the PEAK relating to the Drop and Go bug was plainly identified by Mr Lenton, a Fujitsu employee, on 29 March 2019. It is in my judgment verging on the extraordinary, in the context of any trial that was actually underway, that it

took two months more for this PEAK to be disclosed to the claimants. This is particularly concerning in the context of this trial between these parties on these issues. It is also obvious from the witness statement that the PEAK in question dealing with the Drop and Go bug was not disclosed immediately, until “for the next couple of months my firm [ie the Post Office’s solicitors] continued to investigate these bugs.” There is no reason that investigations, whether ongoing or not, should delay disclosing a document of this nature to the other side, particularly in the middle of such a trial as this.

620.

The conclusion to be drawn from this is that disclosure was given in a manner that could only have disrupted and delayed proper investigation of the issues contained in the documents. In this specific case, that includes the Drop and Go bug. That document should have been provided to the claimants very rapidly once it came to the attention of the Post Office’s solicitors, not kept “for the next couple of months”, as Mr Parsons puts it. In subsequent trials, if relevant documents of this nature are discovered by either side or their legal advisers, it is a matter of some importance that disclosure is given very rapidly. This is particularly important whilst the trial is underway. A failure to do so risks disruption, delay and increased costs.

621.

The experts agreed the following about PEAKs and their content. “PEAKs record a timeline of activities to fix a bug or a problem. They sometimes contain information not found in KELs about specific impact on branches or root causes – what needs to be fixed. They are written, by people who know Horizon very well. They do not contain design detail for any change. They are generally about development activities and timeline rather than about potential impact. PEAKs typically stop when development has done its job, so they are not likely to contain information about follow-on activities, such as compensating branches for any losses.” It is also agreed, and indeed can be seen from the actual PEAKs themselves, that some of them record observations of financial impact. PEAKs do not however always use the symbol “£”, which was a search item used by Dr Worden to create a set of 259 KELs that contained that symbol. Even where they referred to a financial discrepancy, PEAKs sometimes just recorded the figure, or sometimes used “pds”, and did not use the symbol “£”.

Release Notes

622.

A spreadsheet listing the Release Notes was produced which shows that there have been a great number of changes and updates both to Legacy Horizon and Horizon Online over the years. The total number of Release Notes is 19,842 at final version. Each expert provided an estimate of how many changes per week there have been to Horizon. Mr Coyne’s estimate is approximately 19 changes per week. Dr Worden’s estimate is 5 changes per working day. Given the number of working days per week, these figures are broadly the same. No details of the actual Release Notes were provided. The spreadsheet was not, in my judgment, complete, nor was it in useful usable form. No Release Notes detrimental to the Post Office’s case have been produced. The Post Office adopted an unhelpful approach to Release Notes, requiring the claimants to identify from the very difficult spreadsheet which Release Notes they would like to see, with an offer to provide those. On 14 February 2019 the claimants’ solicitors indicated which ones they did want to see. On 20 February 2019, the Post Office’s solicitors stated “we are currently taking instructions in relation to the requests”. Both the request from the claimants, and the response, are wholly unhelpful

and far too late. To be fair to the Post Office, a late request will undoubtedly lead to a late response. Further, the spreadsheet itself is almost wholly unusable.

623.

I have commented upon the approach of the Post Office to disclosure in this litigation before, and not in favourable terms. At the first CMC in October 2017, I said in a short ruling on disclosure that the Post Office had been “obstructive” in its pre-action behaviour in this respect, and that its attitude to disclosure up to that point had been somewhat less than ideal. In Judgment (No.3) I made some further criticisms, which it is not necessary to repeat. It is regrettable that the Post Office’s approach to disclosure, admittedly this time in conjunction with Fujitsu, has continued in different respects to fall short of what is required. However, it does seem to be improving and it is to be hoped that such improvement will continue. Particularly in litigation such as this, with enormous distrust and suspicion on both sides, it is simply counterproductive to have that atmosphere added to by an unnecessarily combative stance on disclosure.

624.

The claimants submitted in their opening that the Post Office’s approach to disclosure had the effect of impeding the claimants from obtaining a full view of the documents and the totality of the Horizon system. I agree with that submission. However, balanced against this, it does have to be remembered that the claimants did not issue any applications for disclosure, as I have already explained.

625.

The Post Office’s case in this litigation will be subject to exactly the same degree of measured and independent assessment, and weighed in the same objective way, as the claimants’ case, and as it would be were these failures not present. However, there must be a change in approach to this group litigation. Disclosing documents late, and in the middle of trials, in such sizeable numbers, and failing (as clearly identified from the passage of Mr Parsons’ own witness statement referred to in [618] above) to act with expedition when clearly relevant documents are unearthed actually during a trial, has two distinct effects. It is disruptive to the proceedings, and it leads to an increase in the costs and/or causes delay. It is the antithesis of co-operation, which the Civil Procedure Rules expressly require. I consider that these passages within this judgment should provide sufficient encouragement to a more constructive approach to prompt disclosure of obviously relevant documents when (or if) they are discovered shortly before, or even during the course of, forthcoming trials.

626.

A PEAK discovered on 29 March 2019 relating to the Drop and Go bug ought to have been disclosed to the claimants within a period of about 48 to 72 hours, not two months. With modern electronic communication, that is a readily achievable timescale.

Further post-trial disclosure

627.

There was then yet further post-trial disclosure given by the Post Office during September 2019. However, the solicitors for the Post Office also notified the court in a letter of 3 October 2019 that further previous information given to the claimants and to the court in the Post Office’s e-disclosure questionnaire was, again, wrong. The letter to the court stated inter alia:

“We write further to the email sent by the Claimants' solicitors to the Managing

Judge's clerk on 27 September 2019 at 12:29, which brought to the Managing Judge's attention that 32 documents relating to the Horizon Issues Trial were disclosed by Post Office on 25 September 2019.

During the course of responding to a number of queries that the Claimants' solicitors had raised in relation to the disclosure of these documents (which can be located at [H/358] of the Horizon Issues Trial Bundle), it has come to Post Office's attention that a statement within its EDQ is incorrect and that there may be previous versions of the Known Error Log that have not been disclosed to the Claimants. Please find enclosed a letter which was sent to the Claimants' solicitors today explaining the background to these documents coming to light and the proposals which have been made in respect of providing any disclosure which may be sought by the Claimants.”

(emphasis added)

628. The actual statement that was wrong was further explained in the accompanying letter to the claimants, which stated:

“In respect of your query regarding the fact that intervening changes to KELs are not captured on the face of the KELs disclosed, we have made further enquires with Fujitsu to confirm our understanding that previous versions of the KELs are no longer held. We regret to say that these enquiries have revealed that our understanding was wrong.

As you will be aware, Post Office's EDQ stated that "[t]he KEL only contains the current database entries and is constantly updated and so the current version will not necessarily reflect the version that was in place at the relevant time. The previous entries /versions of the current entries are no longer available".

This statement was based on info provided by Fujitsu. In response to our recent enquiries, however, we were informed by Fujitsu on 30 September 2019 that this is incorrect and that previous versions of KELs are available.

This takes Post Office greatly by surprise. It relied on the information provided by Fujitsu at the time of filing its EDQ that such documents were not available. It is extremely sorry that this information was incorrect.

Regardless of whether these documents may or may not be "adverse", Post Office has taken immediate steps towards arranging for the previous versions of these KELs to be extracted and has instructed Fujitsu to begin producing a script to extract the documents into a readable HTML format. Unless you disagree, once this script has been produced we will ask Fujitsu to extract all of these versions.

When this has been done, again unless you disagree, we propose to take immediate steps to disclose previous versions of the KELs which were referred to by either Dr Worden or Mr Coyne in their expert reports or the joint statements or the bug table, or were for any other reason included in the trial bundle.

Disclosure of the previous versions of these KELs (where there are previous versions) will be given as a matter of urgency.

We invite you to tell us whether the Claimants also wish to be provided with disclosure of previous versions of the KELs which were not referred to by the experts or included in the trial bundle.

Given the seriousness of this matter, we propose to notify the Managing Judge of this development immediately and will send him a copy of this letter under cover of an explanatory email.”

(emphasis added)

629.

This was correctly characterised by the Post Office’s solicitors as a serious matter. Further, when these KELs were produced and disclosed, it turned out that there were approximately 5,000 of them. This is an extraordinary number, and far more than were available to the experts for the trial.

630.

There was no explanation of the way that what was obviously now incorrect information – that previous versions/entries of KELs were not retained by Fujitsu – had been communicated to the Post Office’s solicitors, or by whom, and there was no explanatory witness statement provided on a voluntary basis. I did not order one, nor did this disclosure interrupt preparation of the draft judgment, or lead to any application for further evidence or cross-examination.

631.

I include it therefore for completeness. There are two points that arise from this.

1.

It would have been a matter for argument, and an exercise in discretion, as to whether any further submissions or evidence (including cross-examination) would be permitted, had such an application been made. The Post Office were pragmatic and sensible, and essentially left this decision to the claimants. The claimants made no such application. However, had they done so, my preliminary view (without hearing any argument) is that I would not have permitted the Horizon Issues trial to be resumed and further evidence adduced. This is for three reasons. Firstly, there has to come a point when the parties are given a decision on the Horizon Issues. The trial had already been interrupted to a substantial degree by the recusal application. This trial was supposed to be completed in April 2019. Resuming it would have led to that date becoming sometime in early 2020. The over-riding objective requires consideration of other court users in other cases. Secondly, subsequent stages of the group litigation would, once again, become consequentially delayed. It is with a sense of wistfulness that I recall a statement at [16] in Judgment No.3 that this litigation would be speedily progressed. Thirdly, the parties have told me that they wish to mediate before Christmas 2019 and would like to know the outcome of the Horizon Issues trial to assist in that process. This would not be possible if the trial were resumed.

2.

This extensive further disclosure does, however, rather reinforce the criticism which the claimants level at the Post Office and Fujitsu. The Post Office are the party in the litigation, and they are the party with the disclosure obligations. They obviously rely upon Fujitsu in this respect. Quite how Fujitsu, a reputable company with a worldwide reputation who have been involved in Horizon for nearly 20 years, could be so wrong on yet another important point going to Horizon is not explained, and will remain unclear. Given the statements made by Fujitsu, relied upon by leading counsel for the Post Office at an earlier hearing and repeated to the court, that the KELs did not contain any material relevant to bugs, errors and defects, a pattern has emerged on the part of Fujitsu. This is that its statements about KELs have been shown to be clearly wrong on more than one occasion, and in important respects.

Criticisms of the disclosure of the claimants

632.

I turn therefore to the criticism that the Post Office has levelled at the claimants. It should be borne in mind that the bulk of the disclosure burden fell upon the Post Office, simply because of the nature of the Horizon Issues. However, that does not mean that the claimants were not themselves obliged to give relevant disclosure to the Post Office. This disparity is obvious in the number of documents disclosed by each party, which the Post Office in its Closing Submissions stated was some 517,965 documents by the Post Office, and only 1,525 by the claimants. Obviously that previous number does not include the 5,000 KELs disclosed by the Post Office in September and October 2019, after the submissions. The comparison in the figures demonstrates the different burden upon the claimants and the Post Office. It is also correct to identify that for many categories of very important documents, the Post Office is reliant upon Fujitsu. The claimants are not.

633.

The Post Office submitted that despite it making appropriate and narrow requests for Model C disclosure of documents held by those claimants who were witnesses in the Horizon Issues Trial, the claimants have not engaged in discussions concerning the scope of the claimants’ disclosure nor provided disclosure of the documents sought by the Post Office.

634.

The Post Office, in the context of its complaints about the claimants’ disclosure, submitted that “it was therefore not appropriate for claimants to serve claimantspecific evidence from current or former SPMs” because of the nature of the Horizon Issues and the terms of the Order that ordered witness statements, and that the Post Office sought to avoid making disclosure requests nearer to the trial. However, once the claimants did so, and served their nine witness statements in evidence in September 2018, the Post Office’s solicitors wrote to the claimants on 22 October 2018, 7 November 2018, 30 November 2018 and 20 December 2018 and did not receive any response. Even though the Common Issues trial was underway for part of this time, I do not see why such letters should have gone unanswered by the claimants’ solicitors. When the response did come, it was to the effect that there would be no further disclosure as this was not envisaged by existing orders.

635.

The Post Office contrast this with the disclosure it was prepared to give in respect of specific witnesses, and also stated that “given that claimants had flouted that Order, Post Office sought to understand what searches (if any) had been conducted by claimants so that it could understand whether further disclosure would be required in light of claimants’ evidence. A further request for details by claimants was made by Post Office on 17 January 2019.”

636.

I am invited to contrast this lack of response by the claimants, which it is said should be viewed in light of the approach adopted by the Post Office. One example of the further disclosure that Post Office had agreed to give which was outside that ordered by the Court was disclosure made in respect of the “operation of branches by Angela Burke, Akash Patny, Anup Patny, Jayesh Tank, Setpal Singh and Adrees Latif….”

637.

The Post Office complains that the claimants “clearly intended disclosure to be a onesided exercise”, seeking disclosure of this nature but refusing to provide their own claimant-specific exercise. Mr Tank did search, in February 2019, within an archived Subpostmasters “Yahoo Groups” forum where he used to post (which means to make comments on the site) and after doing so, Mr Tank identified a post he made on 13 December 2011 relating to the £195.04 shortfall referred to by Ms Van Den Bogerd,

and also a post he made on 29 September 2014 relating to the £600 shortfall that he refers to in paragraphs 6-11 of his witness statement.

638.

Mr Tank has also found a letter he received from Post Office’s Agents Accounting Team in Chesterfield dated 13 October 2014 which related to this shortfall (and which Mr Tank referred to in his forum post).

639.

Mr Tank was cross-examined about this, and explained that at the time of his first witness statement he “hadn’t fully researched the whole background” and he did not realise at that time that the post he had made previously was as important as it had subsequently become, by which he meant in the trial. He also said he had only become aware of the £195 loss to which it related once he had read Mrs Van Den Bogerd’s witness statement. He also explained that he kept a box file and that was where he found the documents. Complaint is made of the fact that the documents that were disclosed should have been disclosed on 17 July 2018, the date for disclosure. I accept that, and even if one makes the assumption in the claimants’ favour that they did not know at that date that Mr Tank was to be a witness, certainly no later than the date of his first statement (which was 28 September 2018) disclosure ought to have been given. Mr Tank’s lack of understanding of relevance and disclosure should not have arisen, and the existence of his box file should have become known to the claimants’ solicitors far earlier than it was.

640.

Further disclosure was also necessary in relation to Mr Latif. This came about because he served an amended witness statement on 1 March 2019, which came about following enquires with Mr Latif's former branch manager, who consulted the documents within the branch to confirm the date on which particular transaction corrections had been issued.

641.

He explained this in his cross-examination:

" Q. First question about that: by amendment to your statement you now say that this was in around January 2018 rather than March 2018. A. Correct.

Q. What caused you to make that correction?

A. I had a look at the -- we hold the records for the information in the office, so I had my assistants look at the records, transaction logs and that's when I confirmed that it was January rather than March. (Inaudible) was logs in March as well. We made calls effectively every month to Horizon help desk concerning this issue.

Q. When do you say you asked your assistants to check about the date?

A. Yes.

Q. Sorry, when do you say that happened?

A. It was after I made the initial statement, I was checking.

Q. Roughly when, Mr Latif?

A. It would have been a few weeks ago, sir.

Q. So is it right that you didn't check those records from the branch before making your witness statement?

A. No, I thought I was correct but I double checked and made sure that actually in fact they were correct, those (inaudible), so I was right but initial incident happened in January when TA (inaudible) transaction acknowledgement, the TC, the corrections, they came in March.

Q. Okay. So you say, do you, that the TCs in relation to these transaction acknowledgements came in March 2018? A. Yes.

Q. And you say that's something you have checked from your records?

A. Yes."

642.

The Post Office made requests for disclosure during the trial, on 15 March 2019 (which was the end of the first hearing week) requesting disclosure of various documents relating to Mr Latif and Mr Tank and their answers in cross-examination. This also included requests of CCTV recordings which Mr Tank said he had looked at. These requests were refused on 27 March 2019, and the claimants stated that this refusal was on the basis that they were either outside their control, no longer existed or were not relevant to the claimants’ evidence and were not relied upon at trial.

643.

Documents that were held in Mr Latif's former branch were plainly outside of his control, even though his former branch manager had accessed them previously. However, that does not mean that the former branch manager could not at least have been asked for them. Neither the claimants did that, nor indeed did the Post Office (so far as I can tell). I consider the claimants’ attitude in this respect unhelpful, although it should be noted that for at least the documents in respect of Mr Latif, the Post Office could have taken its own steps in this regard.

644.

There are some unsatisfactory aspects to the Post Office’s submissions about the CCTV footage. Mr Latif clearly said in his first witness statement at paragraph 7 “I also checked the CCTV to make sure that I had performed the transaction correctly.” Secondly, the Post Office’s closing submissions refer to requests for “the CCTV recordings which were consulted by Mr Tank for the purposes of drafting his witness statement”. He was asked about the CCTV during his evidence and did not say that he had consulted it for the drafting of his witness statement, he said that he had checked it at the time. Had the Post Office wished, it could have asked for that footage shortly after receiving that statement. It is also the case that the event in respect of which he referred to the CCTV occurred in July 2015. It may be that the CCTV images do not still survive, so long after the event in question, but this point was not pursued in cross-examination.

645.

There is, however, one major difference between the claimants and the Post Office in this respect. The Post Office were entitled to, and did, explore with specific witnesses certain issues in respect of disclosure by asking the relevant person (in the above example, Mr Latif) questions about it. The claimants did not have this opportunity for the following reason. The majority of complaints made about the Post Office’s disclosure arise as a result of Fujitsu. Save for rare circumstances, such as Mr Parson’s specific witness statement about the Royal Mail issue, all that the claimants and the court are usually told is “Fujitsu say X” or “Fujitsu told the Post Office” such and such. Specific individuals are not identified, and they are certainly not called as witnesses to be asked questions about their approach, or (for example in respect of KELs) how they could have been so factually wrong in their explanation of the KELs’ contents. I have formed a dim view of the accuracy of information of this nature coming from Fujitsu, regardless of the person in that organisation responsible for communicating it.

646.

Notwithstanding this, I wish to make one matter completely clear. If a witness refers to a particular document in their oral evidence, and the other party in this litigation wishes to see a copy of that document, then (absent issues of privilege) that ought to be disclosed, even more so if a specific request is made. This is by far the most sensible and proportionate way forward. It is in no party’s interests to engage in squabbling about relevance of a specific document in those circumstances. It costs far more to argue back and forth in solicitors’ letters than it does simply to give disclosure. In any event, the relevance of the document is made clear by the fact that it has been referred to orally in evidence.

647.

It is also correct to say that notwithstanding the different burden upon the different sides in terms of disclosure in the Horizon Issues trial, which has undoubtedly been far heavier for the Post Office, this burden will not always be borne in these proportions. The Post Office also has the benefit of a far higher number of legal advisers. As at the date of this draft judgment, simply in terms of the total number of leading counsel engaged in the group litigation, the Post Office outnumbers the claimants by 4:1. In terms of the number of solicitors’ firms, the Post Office currently outnumbers the claimants by 2:1; although only one firm was acting for the Post Office on the Horizon Issues, and the second firm acted for the Post Office on its unsuccessful attempt to appeal the Common Issues.

Summary on disclosure

648.

The Post Office is, in my judgment, somewhat at risk of downplaying certain unsatisfactory aspects of its own disclosure that have emerged during the Horizon Issues trial. For example, the incident concerning the incorrect statements made to the court about the Royal Mail refusing disclosure without an order of the court was described in the Post Office’s closing submissions as a “mix up” which was “regrettable but should not be the subject of criticism of Post Office.” I do not consider telling the court something in express terms which is factually wrong can properly or correctly be identified as merely a “mix up”. Two points that were expressly explained by its leading counsel were simply not factually accurate. The Royal Mail was not refusing disclosure, and it did not require an order of the court before it would provide certain documents. It had not even been asked for them.

649.

Similarly, in its closing submissions the Post Office referred to the interlocutory discussions about the Known Error Log as a “debate about relevance”. That again downplays what had happened, including the positive submissions made, that the Log was specifically not relevant and even if an order were made in respect of documents showing bugs, errors and defects, the Log would not be disclosed (even if it were in the Post Office’s control, which it was stoutly maintained it was not). The submissions made by the Post Office on the adequacy of its own disclosure have been rather undermined, since the trial ended, by the production (emanating from Fujitsu, once again) of over 5,000 KELs, something rightly described by the Post Office’s solicitors as “a serious matter”.

650.

The claimants have not been entirely co-operative on their own disclosure, however. I consider that the criticisms of the claimants for serving what was described as “claimant specific” evidence are unjustified. At least one of the experts, Mr Coyne, adopted a specific approach and investigated specific issues or occurrences, and there is no reason that both of the experts could not have done so. Dr Worden did not, but

that does not mean that the claimants were wrong to do so. Some of the criticisms made by the Post Office regarding the claimants are unjustified, therefore, but not all of them are. Although I do harbour the suspicion that the Post Office may have decided that, on disclosure at least, attack may be the best form of defence, some of the criticisms made in the Post Office’s closing submissions are justified. For example, for the claimants’ solicitors simply to state that “at present, it is not the Claimants’ intention to make another round of disclosure in relation to the Horizon Issues trial” as was done in July 2018, is not co-operative. Further, the reason that documents which Mr Tank had in his box, that were plainly relevant, were not disclosed was because Mr Tank had not been asked for them. Had he been asked – which is something that the claimants’ solicitors should have done, once it was known he would be a witness – these could and would have been disclosed. They certainly should have been.

651.

The following points are of general application to both parties in this litigation. At some point a party who seeks to complain to the court about disclosure ought either to shelve its general disquiet on this subject, or make an application.

652.

It is important to identify, so that the parties in this group litigation are aware, the court’s approach to disclosure. This is given for the purposes of guidance, and costeffective and efficient progress of the group litigation going forwards.

1.

Care must be taken when explanations or submissions are made to the court. If further time is needed in order for accuracy to be achieved, that time should be sought. Those giving instructions to counsel must be astute to ensure that they are giving correct information.

2.

All the substantive issues will be resolved on the evidence, the applicable law, and taking account of submissions. A party does not obtain a preliminary advantage or “head start” on the substantive issues because of any criticisms about disclosure.

3.

Criticisms about the other party’s disclosure should be proportionate. Depending upon the circumstances and the nature of the document, the fact that disclosure is given late might be relevant, but simply because continuing disclosure is given by one party does not mean that there is anything sinister lurking in the background.

4.

The court can always be asked to resolve disputes about documents. Mr Coyne sought a large number of different types of documents, but no applications were made by the claimants even though these were not provided. Criticisms of the Post Office’s disclosure by the claimants has to be seen in the light that the claimants did not consider it sufficiently important at the time to make any applications to the court. This is not to encourage applications. Sensible and well-advised parties who cooperate will usually deal with almost all disclosure issues consensually.

5.

The parties are expected to comply with their disclosure obligations. If a document comes to light that should have been disclosed but has not been, it should be provided promptly. If this is discovered actually during a trial, it must be disclosed very rapidly indeed. As happened with the document referred to at [618] above, a document concerning the Drop and Go bug was discovered actually during the Horizon Issues trial, then (as Mr Parsons put it in his explanatory witness statement) “for the next couple of months” kept by the Post Office’s solicitors while “research” was done. It was then disclosed just before resumption of the trial “in a batch”. This is not working to the time scale either required, or expected, by the court, and it is not the correct way to deal with disclosure during a trial. This is a laissez-faire approach to disclosure of an important document during the trial itself, and the document should have been disclosed very promptly.

6.

Both parties will be held to the same standards. The Post Office has, so far, and given the issues that have been tried, borne the bulk of the disclosure burden. The claimants are expected to observe their disclosure obligations in the same way, and will be expected to do so in the future when further issues and the claimants’ different cases are tried in 2020.

7.

Witness statements will be ordered from a party when the court requires an explanation. Nothing else should be read into the ordering of a witness statement, other than it means precisely that -- an explanation is required.

8.

It is obvious that the Post Office has had to rely upon Fujitsu to a large degree. However, given it was Fujitsu who told the Post Office what the Known Error Log contained – see further [586] above – Fujitsu has, so far, shown itself not to be entirely reliable in this respect. Fujitsu are also responsible for the Post Office making a directly incorrect important statement in its EDQ about retention of KELs, which led to the disclosure of about 5,000 of these some months after the trial closed.

9.

Adoption of the Model C procedure does not mean that parties can keep adverse documents to themselves and not disclose them until a specific request is made. Such adverse documents should be disclosed promptly.

10.

Finally, disclosure is very expensive. The court will be astute to guard against it becoming either satellite litigation or a weapon in the interlocutory arsenal.

653.

I have exhorted the parties to co-operation many times before in this group litigation. I will continue to do so. Eventually, it is to be hoped that they will emerge into the broad sunlit uplands of the over-riding objective on disclosure matters.

G. The Experts’ Agreements

654.

There were four experts’ agreements in this case. Experts’ agreements are very important. Permission to rely upon IT expert evidence was granted in the First CMC Order dated 25 October 2017. The two experts met on a number of occasions and agreed certain points.

655.

As will be seen from Part H of this judgment, the experts approached their respective exercises in very different ways. However, they did reach some areas of agreement that were very helpful. Their joint statements or experts’ agreements did however also set out a considerable number of areas of disagreement. As the trial approached, I requested through the parties that the joint statements of the experts focus more on areas of actual agreement, rather than recitations of position and disagreements.

656.

The 1st Joint Statement was dated 4 September 2018 and set out the following, inter alia, at the very beginning of the statement:

“Each expert's approach to writing his report, and to this joint memorandum which foreshadows their reports, could broadly be one of three possible approaches:

a)

To focus mainly on negative points found in the disclosed documents about where Horizon may have fallen short.

b)

To focus mainly on those aspects of Horizon which were intended to achieve robustness and reliability, and the evidence implying that they succeeded.

c)

To provide the court with a clear foundation for understanding the design and operation of Horizon; then, building on that foundation, to provide a balanced assessment of the ways in which Horizon succeeded, whilst addressing any disclosed issues where Horizon may have fallen short.”

657.

This was the first area of disagreement. Mr Coyne stated in the section immediately following, entitled Areas of Disagreement:

“In my opinion the technical issues can be answered without reference to the Claimants high level allegations document. The issues are about how Horizon and its interactive components operated and the processes employed by Post Office and Fujitsu in supporting these systems and the data within.

Whilst my report will take a balanced approach, it is the case that many of the issues require a deep focus on the occurrences of bugs, errors and defects as well as the potential for modification of transactional data. Whilst context will be provided as to how Horizon should work in typical circumstances, it is the non-typical operation where focus will be placed.”

658.

This approach was not agreed by Dr Worden. He stated:

“I intend to take the balanced approach (c).

To provide a joint statement at this stage which will be useful to the court, under each Horizon issue the experts need first to agree what is worth agreeing having regard to the case which is being alleged. For instance, it would not help the court for the experts to list a number of detailed points as agreed, if one or the other expert thinks that those points are irrelevant or unimportant to the case alleged; or if collectively the points imply a view which, in the opinion of either expert, is not a balanced view. Therefore, the areas of agreement may at this stage be limited and more high level.

My approach will also focus on the Horizon system. It will consider Horizon within its wider business context, but I do not believe that the Horizon issues extend to offering any opinion on the wider business practices of Post Office.”

659.

I consider that the balanced approach, which both experts stated they would be adopting, was the correct one. However, I do not find the following statement by Dr Worden in this 1st Joint Statement as helpful:

“under each Horizon issue the experts need first to agree what is worth agreeing having regard to the case which is being alleged. For instance, it would not help the court for the experts to list a number of detailed points as agreed, if one or the other expert thinks that those points are irrelevant or unimportant to the case alleged.”

660.

It is a matter for the court to consider how helpful a particular point of detail, agreed by the experts, is or is not to the issues to be decided. Obviously the parties can make submissions about that, but it is not for an expert to decide that a point of detail, upon which he may be agreed with his opposite number, should however not be agreed because that point will not be helpful to the court; or that it should not be included in a joint statement. It is also not for experts to direct the court in any particular direction. The Horizon Issues were carefully drafted by the parties, approved by the court, and specifically included in an Order. They were the issues that the experts were to agree, and detailed points of an IT nature that arose within those Horizon Issues should, where possible, have been agreed.

661.

Because Mr Coyne and Dr Worden approached their tasks so differently, the former inspecting a number of PEAKs and KELs about 10 times greater in number than the latter, this meant that a great amount of very detailed matters shown within those documents were not agreed by Dr Worden. He approached his expert exercise from what he described as “a top down” approach, a methodology that found its most stark manifestation in his statistical analysis of likelihood in his Section 8.

662.

I will include within this judgment, not the full text of all four of the different Joint Statements, but some lengthy passages. This is to identify the large degree of agreement between the experts. I will however omit (with some isolated exceptions) the passages where the experts set out their separate, not-agreed, views.

The 1 st Joint Statement

663.

I also quote at this stage from the 1st Joint Statement, signed on 4 September 2018, and the points of agreement and disagreement in respect of Horizon Issue 1. This is because in my judgment it is an important point of agreement, in respect of an important Horizon Issue, but also it is useful to set out the approach within the Joint Statements.

“2. Issues and Position

2.1 Issue 1

Issue 1 – To what extent was it possible or likely for bugs, errors or defects of the nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to 56 of the Generic Defence to have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions as alleged at §24.1 GPOC?

Areas of Agreement:

Evidence exists that bugs/errors/defects have caused actual discrepancies or shortfalls relating to Subpostmasters’ branch accounts/transactions.

Each time any IT system (including Horizon) is changed there is the potential to introduce new bugs/errors/defects.

Once bugs/errors/defects are discovered, they take time to resolve and therefore systems such as Horizon often continue to operate with bugs/errors/defects with or without workarounds in place.

Theoretically, bugs/errors/defects that existed within Horizon have the potential to cause apparent or alleged discrepancies or shortfalls in relating to Subpostmasters’ branch accounts/transactions.

In the event that any specific bug/error/defect had such an effect, the experts have differing views as to the ‘extent’ of the impact that such bugs/errors/defects may have had on branch accounts.”

(Emphasis added)

The statement continues:

“Areas of Disagreement:

Jason Coyne: As identified by select Known Error Logs (KELs), many thousands of bugs/errors and defects occurred and many of these known errors had impacts that recurred in different circumstances, therefore the reliability of Horizon to accurately process and record transactions is questionable.

Such facts undermine the reliability of Horizon to accurately process and record transactions.

Examples:

(i)

the Calendar Square/Falkirk bug – which led to Horizon failing to recognise transfers between different stock units, thereby affecting Branch accounts;

(ii)

the Payments Mismatch defect – which affected at least 62 branches and was related to the process of moving discrepancies into the local suspense account, thereby affecting Branch accounts. This defect was not capable of being identified by Subpostmasters, who would have believed from Horizon that their account was balanced when it was not; and

(iii)

the Suspense Account bug – which erroneously replicated suspense account items for 14 branches from 2010 in the same monthly trading periods in 2011 and 2012, thereby affecting Branch accounts.

In my opinion there is possibility of bugs/errors/defects existing in Horizon that have not yet been discovered. Further, most bugs/errors/defects that exist in Horizon have existed for extended periods of time before they were discovered.

I understand that Horizon had at least 19,842 ‘releases’ of new software, each one of these could have introduced new bugs/errors/defects.

Dr Worden:

My current preliminary view is that undetected bugs/errors/defects in Horizon were very unlikely to be the cause of permanent shortfalls or discrepancies in a branch's accounts.

It is necessary to define measures for the 'extent' of issue 1. At least two measures are possible: the number of such bugs and errors, and their net expected financial impact on claimants' branch accounts, compared to the total shortfall experienced by all claimants, which is of the order of £18 million. I intend to assess both measures. The latter measure may be more useful, because if any set of bugs has expected financial impact much less than £18M, it can do little to account for the claimants' shortfalls, either collectively or individually.

I propose to assess these measures from a number of different sources of information:

(a) the defects of Horizon cited by the claimants in para 1.3 of the outline; (b) the KELs identified by the claimants in para 1.4 of the outline; (c) other KELs; (d) data on their shortfalls provided by the claimants in their schedules of information; and (e) data from other sources (if available).

I am still scoping the data needed for (e). If it is available, I will provide it to Mr Coyne as soon as I have it.

My preliminary analyses of the data on (a) - (d) imply that:

For the majority of KELs, (which record known bugs), it can be shown, from an understanding of the robustness measures built into Horizon or otherwise, that they would have no permanent impact on branch accounts.

The net financial impact on claimants' branch accounts of any other bugs, including undiscovered bugs, is very small compared to the claimants' total shortfalls. I am not yet able to quantify this, but I intend to do so.”

664.

In my judgment, this presages Dr Worden’s statistical analysis of likelihood, which I deal with as part of Section I. The following text used by Dr Worden should be considered:

“….if any set of bugs has expected financial impact much less than £18M, it can do little to account for the claimants' shortfalls, either collectively or individually” The phrase “much less than £18M” could, potentially, direct expert consideration of Horizon Issue 1 away from what it is in fact asking, towards a different issue entirely. The experts were not tasked only with investigating bugs, errors and defects that had a proven financial impact of (say) £18M, £10M, or any financial figure at all, and certainly not one in the millions of pounds. This was for the following reasons. It was never part of the Post Office’s position at the case management stage when the Horizon Issues were discussed and considered, that there should be any sort of threshold for financial impact, above which the experts should investigate, and below which they should not. The wording of Horizon Issue 1 has already been addressed above, and this uses the terms “possible or likely”. This notion of imposing a financial threshold must have come from Dr Worden.

665.

Nor is the litigation intended to be a wide-ranging enquiry into all of the Post Office’s accounts with all of its SPMs over the 15 or 16 year period in question. The Post Office’s position is that Horizon is sufficiently robust such that it cannot be the cause of the different shortfalls complained of by the hundreds of different claimants. The counter-position to that by the claimants is not that particular discrete sets of bugs had, individually, an impact of millions of pounds upon their branch accounts. If that were the claimants’ case, then Dr Worden’s approach as set out in this joint statement might be understandable. Rather, the claimants’ case is that there are a large number of different bugs, errors and defects that each affected each of their branch accounts over time, with the end result being that each of them was (in accounting terms) identified as owing the Post Office sums in the tens of thousands, or thousands of pounds (or less) with the consequences which are well known. Some discrepancies were only a few pounds, but are said to have occurred many times. Some were said to be tens of thousands of pounds cumulatively. Some occurred many times; others less often, but with greater individual effect. One bug – of which more later – is said to “double up”, which would mean an exponential effect as a loss doubles, then quadruples and so on. If one imposes, at the beginning of the expert investigation process, a filter or threshold that looks for a “set of bugs” causing losses of “millions of pounds” that will have the effect of restricting the field of investigation. There is no reason for such a restriction, in my judgment.

666.

On Horizon Issue 2, the experts were agreed, so far as that issue was concerned, that the extent to which any IT system can automatically alert its users to bugs within the system itself is necessarily limited, and while Horizon has automated checks which would detect certain bugs, there are types of bugs which would not be detected by such checks. The experts were also agreed, in respect of Horizon Issue 6, that whilst Horizon contains measures and controls for detecting system integrity concerns, the automatic mechanisms have failed in the past. The experts did not agree as to the ‘extent’ of prevention, detection, identification, reporting or risk reduction that the automatic and manual control measures delivered; however, the agreement that automatic mechanisms have failed in the past, which was contained in the 1st Joint Statement, is in my judgment important. It means that the Post Office cannot rely upon the presence of automatic mechanisms or processes as being an answer to the consequences of failures in system integrity.

667.

It was also agreed by the experts that the causes of some types of apparent or alleged discrepancies and shortfalls may be identified from reports or transaction data available to the SPMs, but that other causes of apparent or alleged discrepancies and shortfalls may be more difficult or impossible to identify from reports or transaction data available to the SPMs in question, because of their limited knowledge of the complex back-end systems. Identification required the cooperation of PO staff and the SPMs. Again, I consider this expert agreement to be important. This was, essentially, the essence of the complaints raised by Mr Bates himself in the very earliest days of Legacy Horizon, namely that there was or were insufficient information or reports available to him to enable him to identify the causes of apparent or alleged discrepancies. The experts are agreed that, for some causes of apparent or alleged discrepancies this is indeed the case, due to the complexity of the back-end systems (back-end means not at the branch) and the limited knowledge a SPM would have.

668.

The experts agreed that Horizon had evolved, and its robustness may have varied throughout its lifetime. They also agreed that the level of robustness may have increased or decreased as the system was changed. The existence of branch shortfalls was agreed, but they were not agreed at the stage of the first joint statement that this indicated any lack of robustness. They were also agreed, as at the date of their 3rd Joint Statement, that Horizon as it is in 2019 is relatively robust. However, given the span of the years under consideration is 2000 to 2010 for Legacy Horizon, and 2010 to date for Horizon Online, and given the agreement to which I have just referred that robustness may have increased or decreased, this only takes one so far.

669.

On Horizon Issue 4, the experts agreed that there are a number of actual reported errors in data recorded within Horizon arising from (a) data entry, (b) transfer or (c) processing of data. Therefore, the potential exists. The experts do not agree as to “its “extent”.” Dr Worden saw this issue as a subset of Horizon Issue 3. He felt that because of the measures “built into Horizon”, errors of data entry, transfer and processing were very unlikely to affect branch accounts. The experts agreed on Horizon Issue 6 that the automatic measures and controls for detecting system integrity within Horizon had failed in the past; they could not agree as to the extent

that the automatic and manual control measures delivered. Dr Worden also made the point that bugs, when discovered, could not be fixed instantly and what he called “trade offs” were required when dealing with practical release management.

670.

So far as remote access and Horizon Issue 7 was concerned, at the time of the 1st Joint Statement Dr Worden was unclear about this and the technical document provided to him by Mr Coyne had not been received in sufficiently timely manner such that he could consider it. By the time the Horizon Issues trial had started, the Fujitsu witnesses such as Mr Parker had corrected their earlier incorrect witness statements and accepted that this could be done. However, those statements were not served until November 2018 so would not have been in existence as at the time of the 1st Joint Statement.

671.

On Horizon Issue 9, the experts agreed that the causes of some types of apparent or alleged discrepancies and shortfalls may be identified from reports or transaction data available to SPMs. Other causes of apparent or alleged discrepancies and shortfalls may be more difficult or impossible to identify from reports or transaction data available to SPMs, because of their limited knowledge of the complex back-end systems. Identification required the cooperation of PO staff and SPMs.

672.

On Horizon Issue 10, it was agreed that by the very nature of rolling out fixes in any IT system, including those implemented by Fujitsu, these had the potential to affect transaction data or data in branch accounts. This could be seen as stating the obvious, but at least the experts agreed it. The experts were also agreed that the use of tools and facilities to do this should be auditable, however the maintenance of logs recording this would be dependent upon retention periods and the size of the logs in question. Again, this could be seen as stating the obvious. The experts were not agreed on Issue 12, how many times the ability to do this was used, if at all. Nor was there any agreement on Issue 13, the extent to which such facilities had the potential to affect the reliability of branch accounts; Issue 14, a detailed issue with 7 sub-issues concerning functionality; or Issue 15, which concerned Transaction Corrections or TCs.

673.

The experts agreed, so far as the existence and use of permission controls is concerned, which is the essence of Horizon Issue 11, that the usage of tools and platforms which existed specifically for the purpose of accessing and modifying transaction data, should be auditable. It was agreed however that the maintenance of logs would be dependent upon retention periods and size. Such tools which were identified initially in the 1st Joint Statement under Horizon Issue 10 areas of disagreement by Mr Coyne, were at that stage split into the following:

1.

Global Branches which would enable the input of transactions within Horizon as though it had come from an actual Branch;

2.

the Branch Transaction Correctional tool;

3.

the Transaction Information Processing repair tool.

674.

The best description of “Global Branch” or “Global User” is contained in a Post Office document headed “Global User Process Management” which said that it allowed people such as auditors “to be able to go out to any branch and use the same log on to access any Horizon kit in Branch without having to request a single use password at each branch. The individual requiring a Global Use account has to be approved by a Manager showing on the Global User requester and approver list”. However, the document also identified that PWC had discovered that “the current Global User list inherited from Fujitsu contained employees who had left the Business. This raised concerns over the security of access to Horizon. Although this is a low risk in terms of Horizon access (the individual would have to gain access into a Branch first) it is still a risk and needed to be mitigated.” There were 32 such former employees. It was in respect of the permission control procedures concerning these tools that the documented authorisation process was identified, using documents called the “Master Service Change” or MSC documents and “Operational Control Procedures” or OCPs. These documents were requested by Mr Coyne but no actual disclosure application was issued. The document to which I have referred explaining Global User was in the trial bundle.

The 2 nd Joint Statement

675.

The 2nd Joint Statement is dated 25 February 2019 and identified and collated the different bugs, most of which were given actual names, which had either been identified and agreed, or which Mr Coyne maintained were bugs. This statement was in landscape form and lengthy, and included what became to be referred to as “the bug table”. I adopt the terms used by the experts for the descriptions of the different bugs, or alleged bugs. The number of different bugs that were agreed by Dr Worden, by the time the trial ended, was 11. The description of this in the joint statement itself was

“The structure of the document captures i) A table of bugs/errors/defects containing evidence of financial impact upon branch accounts that both experts agree (or indicate if they do not), ii) all expert agreements grouped by Horizon issue, and iii) additional comments and observations input by the respective expert.”

676.

Prior to the commencement of this litigation, the Post Office had acknowledged the existence of only two bugs in Horizon. Mr Coyne’s evidence was that there were 29 relevant to Horizon Issues 1 and 4, and he clarified in his cross-examination that the number that he considered had lasting impact upon branch accounts was 21. I deal with that cross-examination at [794] to [798] below.

677.

The 2nd Joint Statement also said in its Introduction that:

“Because of time pressures and the complexity of the issues, we have not been able to address all the Horizon issues in this joint statement. We will issue a subsequent joint statement addressing those issues we have not yet addressed. For those issues we have addressed in this statement (issues 1, 2, 9, 14 and 15), the layout and references are not as polished as we would have wished.”

678.

It is extremely useful for the court to have Joint Statements from experts, and I am grateful to the experts for producing these statements, in particular what was called “the bug table” in the 2nd Joint Statement, in what is a very complex case, and when they were under considerable time pressure. The bug table was very widely used in both cross-examination by the parties and indeed in their submissions.

The so-called “Bug Table”

679.

The bug table, and my findings upon the bugs within it, are in my judgment central to the Horizon Issues. Evidence concerning bugs in the Bug Table formed a large part of the expert evidence that was tested in cross-examination. The different bugs, or alleged bugs, identified in the Bug Table were as follows. I have added to the following the summary of the position of the Post Office in Appendix 2 of its Closing Submissions, which in some cases accepted there was such a bug, and qualified that acceptance in most instances by references to what was said to be the impact of the bug. I deal with detailed findings on those bugs that are in issue in the Technical Appendix to this judgment. For the detailed submissions by the Post Office for each bug, the paragraph numbering in Appendix 2 started again at 1 for each bug. This did not cause particular difficulty, and because the Post Office explained that Appendix 2 had been put together by different parts having been drafted by different counsel/solicitor teams, this is understandable. I refer to it here, because otherwise a reader of the passages below might become confused as to why there seem to be so many paragraphs all (for example) numbered 2 in Appendix 2. The following summary of the Bug Table is intended to be a useful summary for this judgment, and is not restricted solely to the entries within the 2nd Joint Statement.

1.

Receipts and Payments Mis-match bug.

This was agreed in the bug table as an “acknowledged bug” which had an impact on branch accounts. Dr Worden added in his comments that “Therefore, the extent of this bug is well established, in the GJ analysis.” GJ is Gareth Jenkins. The identified year of its effect was 2010. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. In the summary in Appendix 2, the Post Office stated that “In the event, however, this bug resulted in transient impact only.”

2.

Callendar Square/Falkirk bug.

This too was agreed in the bug table as an “acknowledged bug” which had an impact on branch accounts. Its identified years of effect were 2000 – 2006. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had “transient impact”. Dr Worden added different comments, including that “the bug arose from a fault in the underlying Riposte software, so it is not surprising that it took Fujitsu some time to understand it, or that they had to rely on the suppliers to fix it. It does not show poor system design or support by Fujitsu”. That latter sentence is a little surprising, and seemingly very defensive on Fujitsu’s behalf. This is because the Horizon Issues are all about the operability and functionality of Horizon, not who is to blame for the presence of any bugs, errors or defects. Riposte software was part of Legacy Horizon. The fact that it is a product designed and supplied by Escher, and not Fujitsu, is not relevant to the dispute between the Post Office and the claimants. In any event, and as accepted and recited in paragraph 2 of the Summary of Appendix 2 dealing with this bug, “Mr Coyne asserts that Bug 2: Callendar Square is a bug with lasting financial impact and in JS2, Dr Worden appears to agree that there is strong evidence of this….”

3.

Suspense Account bug.

This was agreed in the bug table as an “acknowledged bug” and its identified years are 2010 to 2013. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. Dr Worden’s comments do not include that it had an agreed impact upon branch accounts. This is because of a concept he introduced (expanded in his reports) of “transient effect”. What this meant was that if there was an impact on branch accounts by something within Horizon, but that was then corrected by a TC, he concluded it had a “transient effect” upon branch accounts and dealt with it differently. I address this concept below. Other comments of his in relation to this acknowledged bug are:

“It was a transient effect arising not from a fault in the software, but from a change in database archiving policy in 2010. The delay in correcting it arose from a failure of communication between PO and Fujitsu. Because the bug would only manifest itself annually for any affected branch, the effects of this delay were not widespread.

Peak PC0223870 shows that Fujitsu were able to identify the branches affected, even when Subpostmasters did not report it. There is evidence that the branches were compensated, as I would expect from the normal error correction processes.”

4.

Dalmellington bug/Branch Outreach Issue.

This was not acknowledged as a bug in the bug table, but is effectively accepted by the Post Office as a bug. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had “transient impact”. Indeed, for some time during the trial the Post Office would refer to this as “an issue”. I find that it is a bug – that much does not seem to be controversial, as paragraph 2 of the detailed part of Appendix 2 relating to this bug states the following:

“This is a bug which Mr Coyne states has lasting impact on branch accounts. Post

Office submits that there was no lasting impact on branch accounts.”

The years of its effect were 2010 to 2015. It is named in respect of the branch where it was “discovered” in 2015, although the investigation undertaken by Fujitsu in that year identified previous occurrences. There were over 110 occurrences of it over this 5 year period (the actual number is probably 116, but at least 114) with 88 branches affected. Some were impacted multiple times. It is what is called a “cash remming error” and only happened in certain circumstances. It was explained further by Mr Godeseth in his evidence at [445] onwards above. It would, in my judgment, have an impact upon branch accounts – it is the extent of that impact that is in issue, whether it was lasting or transient. Dr Worden relied upon what he described as “a well-tested process of reconciliation and TCs to detect and correct errors in cash remming (used 20,000 times per year)” and said it was “straightforward for Horizon to detect any discrepancy between a “rem out” and the corresponding “rem in” (a mismatch arising either from a miscount, or a multiple count of a pouch) and then a TC can be issued.” He also added that “this process catches and corrects remming errors, whatever their cause - including if they arise from, or are provoked by, software faults.” He therefore implicitly accepted that the Dalmellington bug was a software fault, although he did not say so in terms. I find that it was plainly was. He was also reliant upon the process of TCs to correct it. Further, the bug was there for 5 years and was not discovered, although its effects doubtless were. I find that this was a software bug that impacted upon branch accounts and I will come to the nature of its effect in my findings on the bugs that are not agreed.

5.

Remming In bug.

This was not acknowledged as a bug in the bug table. However, Dr Worden accepted it was a bug or defect in his cross examination and it is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is again said to have had transient impact. The years it was said to have been present in terms of effect were “March – August 2010 and recorded as fixed approx. 2011”. Mr Coyne said there were 14 branches affected. Dr Worden’s entry in the Joint Statement relied upon TCs, and stated “As for the Dalmellington bug, above – PO had a robust process for detecting and correcting remming errors, whatever their origin. So, there were no lasting effects on branch accounts.” In paragraph 2 of the detailed part of Appendix 2 relating to this bug, the Post Office submitted that “any discrepancy would be transient as instances of this bug are caught by automatic reporting.”

(emphasis added)

The answer to this is therefore dependent upon my answer to the point concerning lasting and transient impact.

6.

Remming Out bug.

This was split into two in the bug table, 6(i) which was identified in KEL acha508S and 6(ii) which was identified in KEL GMaxwell3853P. The former was identified in February/April 2007, recorded as fixed approx. 2007, with Mr Coyne identified 57 branches affected; and the latter was in May 2005 with one branch being affected. They were both remming out issues, hence the grouping by the experts in the bug table. It was not acknowledged as a bug in the bug table. However, it is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Again, for both of these iterations, 6(i) and 6(ii), Dr Worden used the same wording as he had in the entry for bug 5, the Remming In bug, namely “As for the Dalmellington bug, above – PO had a robust process for detecting and correcting remming errors, whatever their origin. So, there were no lasting effects on branch accounts” (emphasis added). In paragraph 2 of the detailed part of Appendix 2 dealing with this bug, the Post Office stated that this “comprises two separate issues, only one of which was a bug. Any discrepancy caused by either issue would be transient as instances of both issues were caught by automatic reporting.” It was also said that Mr Coyne had conflated two unrelated issues under the heading “Remming Out”. Given this was an entry in the 2nd Joint Statement agreed by the experts, it is a little unfair to state that Mr Coyne had done this, as the entry was obviously agreed by Dr Worden. However, it was split into 6(i) and 6(ii) and I will consider each separately.

7.

Local Suspense Account issue, not the same as 3. Suspense Account bug. This was reported in 2010 and recorded as fixed in September 2010. It was not acknowledged in the bug table, but is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, although it is one of those bugs said to have had transient impact. Mr Parker had identified 33 branches affected. Mr Coyne recorded what he said were four associated KELs, namely acha5259Q (for which there were 6 PEAKs)(this KEL is mistakenly recorded twice in column 3, with acha5838T only mentioned in the text in that column); cardc2043L (10 PEAKs); PorterS199P (3 PEAKs); and acha5838T (which states there are “two different but similar problems” and appears in the text, but not in the list of KELs at the end of the text by Mr Coyne). Dr Worden’s comments were that

“The KEL acha5259Q implies that PO and Fujitsu were able to identify all occurrences of the problem, without being notified by any Subpostmaster. I would therefore expect them to have corrected any impact on branch accounts as part of normal error correction processes.

I would not expect evidence of all corrections to accounts to have survived to the present day. PEAKs and KELs are not used to record corrections of financial impact.”

He also relied upon a statement by Mr Parker that there was ‘Temporary financial impact which would have been cancelled out in the following period by a corresponding discrepancy’.

This introduces another concept, similar if not identical to Dr Worden’s “transient impact”, and that is “temporary financial impact”. Dr Worden is correct that PEAKs and KELs do not record corrections of financial impact; they do however record financial impact, because when an SPM reaches SSC (which raises the PEAK) they often start by recording “SPM says he/she has a problem in that……” and financial impact is often then recorded. The Post Office in paragraph 2 of the detailed part of Appendix 2 dealing with this bug states that “any discrepancy would not be lasting”. This does therefore accept, albeit implicitly, that it would have an impact upon branch accounts. It is also submitted that it was a “teething problem” from the early days of the Horizon Online pilot scheme.

8. Recovery Issues.

The Post Office does not accept these are bugs at all. These are not agreed as bugs by the experts, and there are four different types included in the table, with years of effect from 2010 to 2018. Mr Coyne’s entry stated that “The text within the PEAKs and KELs suggests that in each case a branch account discrepancy would be evident and would require correction by the Post Office.” Dr Worden stated that:

“The KELs and PEAKs cited by Mr Coyne are not indicative of errors in Horizon. They provide guidance on how to correct discrepancies caused by human errors or other errors in transaction recovery ('recoverable transactions')

Because there were many such errors, there were many calls to the help desk and many PEAKs and KELs. Normally, correction of errors involved back office reconciliation and issuing TCs. This was accurate and effective; I have derived an upper limit of £2 per branch per month on the mean impact of erroneous TCs.

One important KEL acha959T was guidance to the back office MSU, not for Subpostmasters”

There was a minor typographic error in the submissions in paragraph 4 of the detailed part of Appendix 2 which referred to bug 9. However, the four different types of recovery issues are addressed in the subsequent paragraphs and the conclusion paragraphs deal with bug 8. This was later corrected in a sheet of corrections, which clarified that paragraph 4 of the detailed part of Appendix 2 should have stated "Mr Coyne states that Bug 8: Recovery Issues is a bug with lasting financial impact. Post Office submits that it is not a bug at all". The relevant findings on these issues are dealt with in the Technical Appendix.

9.

Reversals.

This was not acknowledged as a bug in the bug table but is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Indeed, Dr Worden’s entry in the 2nd Joint Statement in relation to this stated “Transaction reversals are a complex area which, like recoverable transactions, are less familiar to Subpostmasters and are more prone to human error. They lead to many calls to the help line and to many KELs and PEAKs - not necessarily related to any fault in Horizon.” This can be seen as yet another example of attributing fault to the SPMs where possible, but given it is now accepted as a bug it is not necessary to consider that any further. Mr Coyne’s entry in the 2nd Joint Statement stated “In April 2003 due to a failure in regression testing, Horizon version S30 was released by Fujitsu and this introduced a bug where the value of transactions reversed by Subpostmasters was shown twice in the amount of the reversal in branch accounts.”

10.

Data Tree Build Failure discrepancies.

This is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. The PEAK dealing with this reads “Data trees have been failing to build fully, and the system has not been detecting this.” Data trees were part of Legacy Horizon, and is used to build a summary (or picture, the word used by the Post Office in its Appendix 2) of the accounts. The building of data trees is a software function, and given “the system” is Horizon and is supposed to detect failures of this nature, it is difficult to see how it could ever have been in issue that this was a bug. Mr Coyne’s entry in the bug table recites that “Dugannon branch suffered a £43,000 discrepancy but the cause was not immediately known. £52,814.29 at the Yate Sodbury Branch. £9,368.40 at the Appleby Westmoreland branch.” These are sizeable sums and arose in the branch accounts. That this is a bug was de facto accepted by Dr Worden in the bug table due to the text of his entry which states:

There was a bug which has potential impact on branch accounts, early in the lifetime of Horizon. Soon after it arose, the error was trapped and detected by DEP and was then soon fixed.

The fault was easily noticeable at branches before the error trapping which was soon introduced and would be even more noticeable after that. Only three branches appear to have been affected, as described by Mr Coyne.

Because it was so noticeable at the branch, and the Peak is concerned with a software error rather than any other cause, I would expect any discrepancies in branch accounts to have been corrected.”

(emphasis added)

This is therefore undoubtedly a bug, but findings in respect of this therefore depend upon the nature of its impact on branch accounts, which are made in the Technical Appendix.

11.

Girobank discrepancies.

Mr Coyne considered this was a bug. Dr Worden did not and stated that “the first fault concerns reports. A fault in a report is not a discrepancy in branch accounts, and only causes one if it causes a person to make a mistake.” This is now essentially accepted as a bug by the Post Office, but it is submitted it had no branch impact. It is included in paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact.” The detailed part of Appendix 2 dealing with this submits that there is no evidence of any financial impact upon branch accounts, let alone a lasting impact. This was in the early days of Legacy Horizon. There are said to be six distinct issues arising under this heading, and further detail on this is included within the Technical Appendix. I do not know if what are called “giros” are still in use in 2019.

12.

Counter-replacement issues.

There were two KELs associated with this dealt within the bug table. It is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Mr Coyne considered that when replacing a counter within a branch, the process could result in “the total loss of a transaction”. Dr Worden stated that the cause, recorded in the first KEL (which was created in December 2000 and last updated in July 2007) was that Riposte was coming online from the Recovery mode too early, and causing messages to be overwritten. The nature of the correction was stated as being “to find the overwritten transactions for reconciliation we need to look at the Ripostemirror messagestore' followed by detailed instructions”. Riposte, as has been explained, was part of Legacy Horizon and was a product provided by another company, Escher, but it is plainly part of Horizon. Dr Worden also stated that “the incident arose from a hardware replacement (probably from a hardware fault) not from a fault in Horizon. It is a different kind of recovery issue.” In my judgment the hardware is part of Horizon. Findings in respect of this are made in the Technical Appendix.

13.

Withdrawn stock discrepancies.

The Post Office does not accept these as a bug. Mr Coyne maintains that it is, and in the bug table he extracted part of a PEAK that stated these “Can cause confusion and unexpected (though hopefully temporary) discrepancies at branches by allowing them to declare stock which has already been withdrawn.” Dr Worden stated that “some impact on branch accounts cannot be ruled out, although it is small”. The Post Office detailed submissions in Appendix 2 of its Closing Submissions concentrated on the fact that withdrawn stock – which of course is part of the way that the Post Office manages its business, adding and removing types of products from time to time – this removal of products is done by means of an update to reference data, “and not a change to the core code in the system.” This again is concentrating on the code, rather than the way the system operates. Findings on this are made in the Technical Appendix.

14.

Bureau discrepancies.

These relate to foreign exchange, hence the name. It must be differentiated from bug 23 (also foreign currency, but entitled Bureau de Change). This one arose in 2017. Mr Coyne considered it to be a bug, and Dr Worden effectively agreed in that his entry stated “This appears to be a system error with impact on branch accounts. Although it is possible that a subsequent discrepancy between branch accounting and POLSAP would reveal the problem, leading to a correction (e.g. see Peak PC0265443, and Mr Coyne's para 3.146), I cannot be certain of this.” It is now accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact

(although they were resolved)”. The detailed part of Appendix 2 states that Mr Coyne has drawn together two distinct issues. Paragraph 4 of the detailed part also states the following:

“Bug 14: Bureau Discrepancies is a bug with the potential for lasting financial impact. There are two distinct issues which fall under this heading. With regards to the first issue the branch was made good and a fix was implemented. The second issue was not a bug in Horizon nor an issue which could have impacted branch accounts; it created what was essentially a cash flow problem for the branch.”

Findings on this are made in the Technical Appendix, but so far as the first issue, given a software fix was required, it is undoubtedly a bug, and it is correct to record (as the Post Office do) that there was the potential for lasting financial impact, as without that, there would be no need for the “branch to be made good” because that means the Post Office corrected the branch account discrepancy caused by the bug.

15.

Phantom Transactions.

There are three different issues grouped under this heading. The Post Office does not accept these as a bug. Mr Coyne in his cross-examination stated that this was referable to Horizon Issue 4. He also accepted that some of the issues were not bugs in Horizon. Dr Worden relied upon the fact that what he called “the master PEAK” was closed as “no fault in product”; however, the evidence of fact on this is dealt with above at [209] to [213] when the PEAK was put to Mrs Van Den Bogerd. Dr Worden stated that “There is no evidence for bugs in Horizon with impact on branch accounts.” That entry was obviously before the cross-examination, which in my judgment provided greater factual information. I do not consider reliance can be placed upon the Fujitsu conclusion in the PEAK. Findings on this are dealt with in the Technical Appendix.

16.

Reconciliation issues.

The Post Office does not accept these as a bug. There are a number of different issues grouped together in this heading. The issue was that the SPM was shown a discrepancy on his or her screen. Mr Coyne accepted that the discrepancy would not be shown in the branch accounts; Dr Worden stated that “as it concerns an issue in reporting, the software fault (which was fixed after 5 months) had no direct impact on branch accounts. The only effect of an error in this report would be to mislead or confuse the Subpostmaster - probably leading him to check his figures more carefully and costing him some time.” That accepts a software fault; the fault was in fact a miscounting of the number of files by the system. However, there are six different issues grouped under this heading and the process of reconciliation, which is effectively the comparison by Horizon of two different sets of data, is part of its design function. Mr Coyne in his cross-examination also stated that this was referable to Horizon Issue 4, rather than Horizon Issue 1. The recovery messages were held in the branch messagestore. Findings on this are dealt with in the Technical Appendix.

17.

Branch Customer discrepancies.

This was an entry in respect of Horizon Issue 4 and recorded as such in the bug table. The Post Office does not accept these as a bug. Mr Coyne accepted in his crossexamination that the entry in the PEAK did not suggest any impact upon branch accounts. The entry in the bug table originated in a single PEAK, although there were two PEAKs that relate to the same issue, which was a discrepancy between the financial records held by the Post Office (from Horizon) and a bank’s records, after a counter crashed whilst a transaction was being processed. Findings on this are dealt with in the Technical Appendix.

18.

Concurrent logins.

It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. In paragraph 2 of the detailed part of Appendix 2 dealing with this bug, the following is stated by the Post Office: “Post Office accepts that Bug 18: Concurrent Logins had a potentially lasting financial impact. There is no evidence of any discrepancy in the PEAKs referred to by the experts.” The problem was in the early days of Legacy Horizon when it was possible for users to log in to two terminals at once. Dr Worden in the bug table made the following useful summary statement:

“discrepancies could occur - manifesting themselves as a receipts/payments mismatch. This had the potential to affect branch accounts. The mismatch would bring it to the attention of the Subpostmaster, who would require it to be investigated, except possibly in the case of small mismatches, which he might pass off as an error in the branch (e.g. of counting stock).

“….Fujitsu believed it was a problem with the underlying Riposte software, and passed it to Escher. In September 2000, the problem was 'Now formally fixed in Build 223 update 19 which was released overnight.' However, the new release from Escher did not, as it was expected to, fix the problem. Escher denied that is was a bug in Riposte, but Fujitsu believed in July 2001 that 'This is clearly a bug in the Supplier's code'.”

Dr Worden considered that “these faults had to the potential to produce discrepancies in branch accounts, of small amounts, for a short period of time”. How the bug is therefore characterised depends upon my findings in relation to lasting and transient impact, which is dealt with below. There is no doubt, in my judgment, that this was a bug, and Fujitsu clearly recognised and recorded that in 2001. The fact that the bug was in the code supplied by Escher, as part of Riposte, and not that of Fujitsu is wholly irrelevant for the purposes of the Horizon Issues.

19.

Post & Go/TA discrepancies in POLSAP.

It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. The entry relates to Horizon Issue 4 rather than 1. “Post & Go” are self-service terminals, that are no longer available in branch Post Offices, and are only available in Crown and WH Smith main branches. They are basically selfservice kiosks, designed to avoid queuing, and a customer can weigh a letter or parcel, the terminal will print the relevant stamps/labels, and the item is then posted. The Post Office submitted that this issue was irrelevant to the Horizon Issues trial as ”this issue does not relate to branches that are the focus of this trial”. Findings on this are dealt with in the Technical Appendix.

20.

Recovery Failures.

The entry in the bug table identifies this as relating to Horizon Issue 4. Post Office does not accept these as a bug and states that there is no evidence of a bug in Horizon. There are three different PEAKs. Mr Coyne accepted in his cross-examination that this (or these) should be removed from the table that he used as the originator for his list of bugs (which was Mr Coyne’s Table 1). Dr Worden stated in the bug table that “there was some implication of hardware faults, with a replacement of a base unit, but the PEAK has no evidence of software faults in Horizon.” In one of the PEAKs, he concluded that there was no evidence of any fault in Horizon. Findings on this are dealt with in the Technical Appendix.

21.

Transaction Correction Issues.

This is de facto accepted as a bug by the Post Office, but it is submitted it had no branch impact. Mr Coyne’s entry was that “Transaction Correction bugs/errors and defects do not cause discrepancies with branch accounts but” and he then listed various consequences. In his report he referred to “technical flaws”. Dr Worden, in respect of some PEAKs, accepted they were a bug, such as PEAK PC0129587 where he stated “In my opinion this bug would result in an inconvenience to the Subpostmaster (inability to rollover to the next TP) but would not result in inaccurate processing of any TC, or any impact on branch accounts”. The bug is included in the list of paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact.”

These are submitted by the Post Office as there being “no evidence of any financial impact upon branch accounts”. Findings on this are dealt with in the Technical Appendix.

22.

Bugs/errors/defects introduced by previously applied PEAK fixes.

This is de facto accepted as a bug by the Post Office, but it is submitted it had no branch impact. It is included in paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact”. It is also submitted by the Post Office that “a number of PEAKs arose in testing. The PEAKs that arose in the live environment do not indicate evidence of branch impact.” It is listed in the bug table as concerning Horizon Issue 4. Mr Coyne stated in the Joint Statement that “Branch accounts would be affected by this bug which would cause a discrepancy when handling cheques where the value of the cheque would be doubled” although he accepted that the SPM in question was processing a cheque in a different manner to that recommended. Dr Worden used the term “a fault” in his entries, maintained that there was no impact on branch accounts if the SPM followed correct procedures, and also in respect of another branch relied upon the fact that the effect was an error of 2p only. Findings on this are dealt with in the Technical Appendix.

23.

Bureau de change.

These arose in 2005, 2006 and 2010, and should be differentiated from bug 14, which arose in 2017, was during Horizon Online and is referred to as Bureau Discrepancies. The Post Office does not accept this as a bug. The Post Office also states that Mr Coyne has identified three issues with the same heading, and submits that they all relate to user error. Dr Worden’s entry in the bug table in relation to one of them states “Analysing the second KEL (2010) I noted: ‘Impact small until bug fixed - rounding errors 10-5 in exchange rates.’ The Post Office submitting that this is not a bug is a somewhat bold submission, given the KEL to which Dr Worden refers expressly states that the problem is a different exchange rate appearing on the HNG-X rateboard and the Horizon rateboard. It states “Problem – there is a bug in the code” (emphasis added). Findings on this are dealt with in the Technical Appendix.

24.

Wrong branch customer change displayed.

Mr Coyne concluded this was a bug and part of his entry in the 2nd Joint Statement was “the KEL explains that “the cash amount entered is multiplied by the Qty and hence the new stack total is wrong”, this Horizon bug was due to incorrect reference data and led to an incorrect amount of change being displayed on the branch screen leading to the operator to provide the branch customer with the wrong amount of money thereby leaving a discrepancy in Branch Accounts. It is possible that the amount of change shown on screen is more than the actual money tendered by the customer.” Dr Worden’s entry was “When analysing this KEL I noted ‘Sounds like a genuine problem which may have led to giving the customer the wrong amount - i.e. not recoverable.’ It is now accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. It is also said that it is a reference data bug, and that therefore once discovered could be quickly fixed by changing the relevant reference data. Findings on this are dealt with in the Technical Appendix.

25.

Lyca top up.

It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. It is also a reference data bug, and paragraph 2 of the detailed part of the submissions on this in Appendix 2 states “Lyca Top Up is a bug with the potential for lasting financial impact. This is also [a] reference data bug. As set out above, the experts have agreed that that while reference data bugs may be a significant proportion of the bugs with financial impact, once discovered, they could be quickly fixed (by a change to the reference data) once the bug is correctly identified.” It is also submitted that it was identified through Fujitsu’s automatic reporting. Findings on this are dealt with in the Technical Appendix.

26.

TPSC250 Report.

It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. The origin of the experts discovering this bug is a KEL raised by Anne Chambers in February 2005, and last updated in April 2008. Findings on this are dealt with in the Technical Appendix. I do however find it notable that although Dr Worden says the amounts involved are what he calls “small”, he accepts it is as a “back end reporting problem” – which is nothing to do with the SPM, by definition – and also the KEL states that “the accounting tree has not handled this properly when calculating the daily recon figures and it has resulted in a mismatch…” The mismatch in that case was 66p. The accounting tree is part of Horizon. The KEL also appears to be incomplete because only the first page is present. The section that follows the heading “Evidence” is entirely missing. There is also nothing in terms of the text available that demonstrates what occurred in 2008, even though it is plain for the page that is available that the KEL was updated then. The Post Office submits that there are five separate issues drawn together within this bug. Findings on this are dealt with in the Technical Appendix.

27.

TPS.

It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. One of the Fujitsu documents referenced in the 2nd Joint Statement states that the Transaction Repair Tool or TRT is being used “to repair 1 harvester exceptions for” a particular branch and “There is no correction to be performed and hence no call for confirmtiprepair - this is just an oddity performed by that very flaky mails code.” (emphasis added) There were 40 associated PEAKs, and Mr Coyne observed that both the credit and debit sides of a transaction were doubled, so the net impact of the bug was zero, although he drew attention to the entry in two PEAKs that suggested that SSC requested confirmation of any gain or loss at the counter. Dr Worden believed it was a back-end reporting problem, although the chances of impact upon branch accounts were small. Findings on this are dealt with in the Technical Appendix.

28.

Drop and Go.

It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. This occurred in July 2017 and related to a duplicate “Drop and Go” transaction for £100. This was performed twice; the branch was debited with £200, but the customer credited with only £100 (which had been the amount “topped up” by the customer). Mr Coyne considered it a bug with impact upon branch accounts; Dr Worden stated in his entry “My analysis of this KEL was ‘Possible financial impact. Seems very visible on the counter. Script = reference data - therefore fixed easily’. Paragraph 2 of the detailed part of Appendix 2 on this bug states “Peak PC0260269 relates to an issue involving a Drop and Go transaction (a £100 mobile phone top up) that timed out on Horizon” (emphasis added). This does not appear to be correct, and I do not believe that the transaction relates to a mobile phone. However, I deal with this further in the findings in the Technical Appendix.

29.

Network Banking Bug.

This related to a KEL which was raised in 2004 and updated in 2005, with 12 associated PEAKs in the range 2004 to 2010. The Post Office does not accept this as a bug. Mr Coyne stated that “Horizon appears to mis-handle communications, leading to errors within network banking and in turn causing the potential for branch account discrepancies.” One of these was pension transactions being declined yet the customer’s bank account being debited; when the customer complained, the SPM refunded the £50 in question. Dr Worden considered it was “mainly about a communication problem from BT, outside Horizon” but also wanted to investigate further due to an entry that referred to a “CNIM own goal”. The Post Office maintained there were two separate issues and the £50 was paid out due to “user error”. Findings on this are dealt with in the Technical Appendix.

680.

The names given to the different bugs above came, sometimes, from the names applied to the bugs by the Fujitsu personnel who knew about these bugs and worked upon them over the years. These names would sometimes appear in the PEAKs themselves that identified the bugs or which represented the views reached by different personnel within Fujitsu who investigated the background, sometimes identifying that previous incidents had similar, or sometimes even identical, characteristics to earlier occurrences of the bug, and who worked to provide fixes or solutions to some of them, or came to conclusions that there was nothing wrong with Horizon and attributed blame, fault or responsibility to the SPM. Some of the names are self-explanatory. Some, such as Callendar Square or Dalmellington, are the names of places where that bug was first identified by Fujitsu.

681.

Certain terms were used, dependent upon whether a particular bug was agreed as being a bug (such as the Callendar Square bug), or something else, such as “issues” or “discrepancies.” However, as at the end of the trial, it can be seen from the above entries that the experts’ agreements, the clarification provided by each during crossexamination, and the other developments in the trial such as (one assumes) some reflection by the Post Office on the actual text of PEAKs and KELs, lead to the following situation. Of the 29 different entries in the bug table, eight were alleged by the Post Office not to be bugs at all. They are those numbered 8, 13, 15, 16, 17, 20, 23 and 29. This is set out in paragraph 4 on the first page of Appendix 2 to the Post Office’s Closing Submissions. This careful terminology therefore became a little clearer. The remaining number can properly therefore be described as bugs, even on the Post Office’s own case, although their effect was challenged.

682.

This means that the remainder of what is, in my judgment, a lengthy list, were accepted to be bugs. In my judgment, that number of accepted bugs, even on the Post Office’s own case about their effect, and the non-acceptance of the eight others, is a sizeable number. It is approximately a ten-fold increase in the number admitted by the Post Office prior to the Horizon Issues. It is correct that the Post Office did not accept lasting impact on a number – a point I deal with below under “Lasting and transient impact” – and maintained that only 9 “had the potential to cause lasting impact, but were resolved by the Post Office and Fujitsu” but, in my judgment, that is rather to miss the point. The wording of Horizon Issue 1, agreed by the parties, and (one assumes) a major issue that both parties wished to have resolved in order sensibly to progress this group litigation (and omitting the references to the pleadings) is:

“To what extent was it possible or likely for bugs, errors or defects of the nature alleged… to have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions.”

683.

The use of the words “possible” and “potential” in this issue makes it clear, in my judgment, that the number of “bugs, errors or defects” that fall to be considered is, even on the Post Office’s own submissions, 21 different bugs. I find that to be a significant total.

684.

Part only of Appendix 2 to the Post Office’s Closing Submissions dealing with bug 16, Reconciliation Issues, was, by administrative error, not submitted by the date for closing submissions, or until some time after oral submissions were delivered. The Post Office’s solicitors realised this in September 2019. There were three missing pages. I am satisfied that these were drafted prior to the date for service, and I allowed these to be submitted late, together with a clarification document, and short responsive written submissions by the claimants (as the claimants had been deprived of the opportunity to address these three missing pages orally in July). I consider that I am permitted to do this under the court’s general case management powers, and that to do this was to adopt a reasonable and proportionate course. The Post Office’s Closing Submissions was a very substantial document, and it was in my judgment sensible to admit the 3 pages only that had been omitted from one of the appendices.

685.

Other elements of the 2nd Joint Statement were also agreed. These included that Horizon had produced over 3 million sets of monthly branch accounts (a point relevant to Dr Worden’s Section 8 statistical exercise), and also that the experts had different views on “branch impact”. The relevant passages in the Joint Statement were that:

“Mr Coyne refers to any discrepancy that caused a loss (or gain) within branch accounts that needed corrective action as an “impact to branch accounts”. Dr Worden only considers an effect or impact on branch accounts where a discrepancy loss (or gain) was not rectified by a correction such as a Transaction Correction.”

686.

This in particular demonstrates an important difference in the approach of the experts. Given, as I have explained above, that the Horizon Issues do not refer to Transaction Corrections, which even the Post Office’s own witnesses explained had not been addressed because the Post Office’s solicitors said these were outside the scope of the Horizon Issues trial, I consider Mr Coyne’s approach to “impact on branch accounts” to be the correct one. Dr Worden’s approach minimises or avoids impact on branch accounts that have been caused by bugs, errors or defects, even where those that are (or have been) present in the Horizon system have in fact affected the branch accounts of SPMs. It is a feature that I take into account when I consider which of the two experts’ approaches and evidence I prefer. I also explain this further in “Lasting and Transient Impact” below.

687.

The 3rd Joint Statement continued the process of setting out the experts’ agreed position by working through the remaining Horizon Issues. This statement dealt with Horizon Issues 3 to 8 and was dated 1 March 2019. The experts agreed the following points in respect of each. The numbering is mine. One entry below relates to one passage of Dr Worden’s separate and not agreed view under Horizon Issue 3. It was not the only such entry – in different places the experts would record their own separate views.

The 3 rd Joint Statement

Horizon Issue 3

“1. Irrespective of how you define the detail of robustness, in line with most other large-scale computer systems, Horizon's robustness has generally improved. From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events. This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.

2.

Computer systems are considered more robust if access to the back-end databases is restricted tightly.

3.

It was possible for some of the Horizon support staff working at Fujitsu to modify the Horizon back-end branch database. In 2012, Post Office's auditors observed that there were inappropriate system privileges assigned to the APPSUP role (which allowed amendments to the BRDB).

4.

Post Office does not consult the full audit data (unfiltered ARQ Data) before deciding how to handle discrepancies and issuing Transaction Corrections.

5.

PEAKs show that some defects have lain undetected in Horizon for extended periods without being diagnosed and fixed.

6.

During the life of Horizon there have been 19,842 changes made to it via the Fujitsu/Post Office release mechanism.

7.

It is common modern IT development practice to make frequent incremental builds and releases of software.

8.

Specific release note detail has not been provided. Of the 19,842 changes, we would expect that many were minor changes. It is likely that others contained changes to improve the system or to fix bugs and defects.

9.

The effectiveness of various countermeasures changed throughout the life of Horizon.

10.

Countermeasures are basic elements of practical IT system design

11.

Countermeasures work by limiting the impact of Horizon bugs/error and defects on branch accounts. Countermeasures do not always eliminate the effects of adverse events (they are not perfect) but they are often effective in the area where they are deployed; that is why they have become basic elements of practical IT systems design.

12.

It is difficult to measure the extent of the robustness of Horizon, apart from how it might limit the extent of impact on branch accounts, as in Issue 1.

13.

There are indications that in its first year of operation, and in the first year after the introduction of Horizon Online, the system suffered from more problems than in other years. One might expect a higher level of problems in these early periods. The extent to which these problems were serious, or evaded countermeasures, or caused discrepancies in branch accounts, is not agreed.”

14.

Dr Worden stated the following, which was not agreed, following the entry immediately above. “The main way in which the experts have assessed the extent of robustness of Horizon is to ask to what extent failures of its robustness impacted branch accounts. I have addressed this in Horizon Issue 1. In answering this question, fluctuations over the years are of less importance than the sum over all years, if the sum over all years is small. I have found the sum of impacts on branch accounts over all years to be very small”. (emphasis added). This approach is considered further by me when I address Dr Worden’s Section 8 statistical analysis.

15.

The agreed passages continued. “The users of any IT system play a role in assuring its robustness. The designers of a system should not make unrealistic assumptions about the users of the system. Unrealistic assumptions would lead to inappropriate design, making the system less usable.

16.

As Horizon has changed throughout its lifetime, the existence and effectiveness of any countermeasures has too. To have considered the time dependence of all robustness countermeasures over 20 years, would have made the expert reports impossibly lengthy. There was not the time to do so.

17.

Many software bugs can have the same effects as a user error (as illustrated, for instance, by the Dalmellington bug, which produced a remming error)”.

(emphasis added)

Horizon Issue 4

“1. Bugs, errors and defects identified in relation to Horizon Issue 1 are often relevant to Issue 4 in that they are ultimately errors arising from the processing of data in Horizon.

2.

There is evidence within the PEAKs and KELs of bugs/errors/defects within Horizon arising from parts (a), (b) and (c) of this issue that occurred without causing financial discrepancies as well as some that occurred causing financial discrepancies.

3.

Reference data is critical to the operation of Horizon and errors in reference data have led to discrepancies in branch accounts.

(emphasis added)

4.

Of the bugs which in the experts' opinion had the potential to produce discrepancies in branch accounts there may be some involvement of reference data in Bureau Discrepancies, Bureau de Change, Wrong branch customer change displayed, Lyca top-up, and Drop and Go. (Rows 14, 23, 24, 25, and 28 of the bugs table in the second expert joint statement). It is notable that these bugs all concerned specific products (arising from the reference data defining those products).

So, while reference data bugs may be a significant proportion of the bugs with financial impact, once discovered, they could be quickly fixed (by a change to the reference data) once the bug is correctly identified.

5.

Reconciliation between transactions recorded on Horizon and transactions recorded by Post Office's clients is largely automated. Detected discrepancies were subject to manual corrective fixes and/or the issue of Transaction Corrections/Error Notices to the Subpostmasters.

6.

The adequacy of Post Office back office processes to prevent discrepancies in branch accounts can be measured by the quality of the TC process. This quality includes:

The processes of consideration of available data

The level of errors observed in the process

The level of complaints or disputes raised following a TC

The level of upheld complaints following a TC

The level of financial impact of erroneous TCs

7.

Errors in third-party data have led to discrepancies in branch accounts, through erroneous TCs being issued on Subpostmasters.

8.

PO does not control the level of errors made by its third-party client organisations (which may lead to errors in TCs), or the delays in their processes (which may lead to delays in TCs).

(emphasis added)

9.

PO can and should ensure, by careful investigation of disputed TCs, that only a small proportion of errors by PO clients lead to losses for Subpostmasters, provided that the Subpostmasters are in good control of their branches and have the required information available.”

Horizon Issue 6

“1. It is agreed that there are many measures and controls within Horizon that existed to prevent, detect, identify report or reduce the risk of varying errors.”

Horizon Issue 7

“1. Fujitsu could access all transaction data recorded by Horizon.

2. Both Post Office and Fujitsu can read data remotely, and FJ needs remote access for support purposes.”

Horizon Issue 8

“1. Post Office had access to data which would not have been available to Subpostmasters.

2.

The descriptions of facilities for PO in the two expert reports are consistent and can be taken together as a description of those facilities.

3.

Post Office were reliant upon Fujitsu for diagnosis of whether the occurrence of shortfalls was caused by bugs/errors or defects within the Horizon system.” (emphasis added)

688.

It can therefore be seen from the passages of agreement reached by the two IT experts in the 3rd Joint Statement, that a great amount of relevant material directly relevant to the Horizon Issues was not in issue. It was agreed that errors in reference data have led to discrepancies in branch accounts, and also that errors in third-party data have led to discrepancies in branch accounts too. Further, the Post Office has no control over the level of errors made by the third-party clients of the Post Office, and this is agreed. The point also ought to be made that SPMs have no control over that either. Both errors in reference data, and also errors in third party data, being agreed to have led to discrepancies in branch accounts, are points consistent with the claimants’ case in respect of the Horizon system generally.

The 4 th Joint Statement

689.

This dealt with Horizon Issues 10 to 12, and was the last joint statement agreed by the experts. It was dated 4 March 2019 and was agreed just before the trial commenced. Again, the numbering is mine, and I also reproduce one passage from each expert that

was not agreed. One entry below in relation to Horizon Issue 11 was agreed with the word “whenever”. Both experts agreed that this should be worded “usually when” and this latter agreement on that wording was reached during the expert evidence phase of the trial.

Horizon Issue 10

1. Dr Worden stated the following, which was not agreed, in relation to privileged user access, and as one of a number of passages at the beginning of the Joint Statement that were not agreed concerning remote and privileged access. “The Privileged User Access logs are not a useful source of evidence about remote access, including Balancing Transactions.”

The following were agreed:

“2. Within Horizon Online, a Balancing Transaction can be created using the BRDB Transaction Correction Tool (called BRDBX015).

3.

In the one acknowledged Balancing Transaction which has been disclosed, there were SQL INSERTS into only two tables of the BRDB. These tables were ops$brdb.brdbb_rx_eposs_transactions and

ops$brdb.brdb_rx_rep_session_data”

4.

Mr Coyne stated the following, as one of a number of passages that were not agreed. “Outside of the Transaction Correction Tool other methods were available in Horizon Online.

The APPSUP role used by Fujitsu has elevated privilege and could be used “for unenvisaged ad-hoc live amendment not covered by BRDBX015” as outlined in the referenced document.

Fujitsu also has the ability to perform balancing transactions via direct SQL operations (using a command line interface) to perform corrective transactions in Horizon Online (including deletions of operational data).

Such modifications may or may not be visible to the Subpostmaster and they could be done without their consent.”

The following were agreed:

5.

“‘Fixes’ implemented by Fujitsu had the potential to affect transaction data for branch accounts, for transactions which occurred after the fix was deployed.

6.

We have not been provided with logs or audits from the Transaction Repair Tool (TRT).

7.

Certain facilities and procedures used by Fujitsu to repair the more common issues which arose in Horizon were standardised, and evidence of them persists. However, to repair less common issues which arose from time to time, standard tools and procedures might not have been sufficient, and evidence might not persist of what was done at the time. Even when evidence does persist, it may be extremely difficult for the experts to interpret it today, because of the scale and complexity of Horizon. Therefore, it is usually difficult for the experts to make categorical negative statements of the form: ‘X or Y never happened’”.

Horizon Issue 11

All of the following are agreed entries:

“1. Evidence from several PEAKs indicates that usually when [corrected from whenever] Fujitsu needed to make any change to data which impacted branch accounts, they were concerned to seek permission from PO to do so, and to ensure that PO took responsibility for the resulting change.

2.

For any large commercial IT system, it is necessary for some technical users to have privileged access to databases, with wide-ranging capabilities, for system maintenance and problem-solving purposes rather than application-related purposes. It is important to keep the number of these users down to a minimum possible and for each action that they take whilst logged in to be recorded and audited.” (emphasis added)

“3. The logging of Privileged User Access (in PAU logs) commenced October 2009.

Between 2009 and 2015 these logs only displayed the fact that a Privileged User had logged on or off but not what actions they had taken whilst the Privileged User was logged in.

The use of the Transaction Correction Tool cannot be seen in these logs.

4.

At all times, any privileged user access log only shows what tables of BRDB were accessed for a very small minority of accesses.

There are no privileged user access logs which show access to the two tables of BRDB used to insert the one acknowledged balancing transaction.

5.

The authorisation for the use of the Transaction Correction Tool for the acknowledged Balancing Transaction can be seen in one OCP within the OCP disclosure of 25th January 2019.

The record of the uses of the Transaction Correction Tool, including the one acknowledged Balancing Transaction, can be seen in the .aud files, disclosed 22nd February 2019.”

Horizon Issue 12

The following are all agreed entries:

“1. Post Office report that there have been 2,297 uses of the Transaction Correction Tool. For each of these, the usage was logged and the SQL used to alter BRDB has been disclosed.

Only one of these uses (POL-0512131) changed transaction data in two transaction tables, in March 2010.

This Balancing Transaction consists of about 500 lines of SQL, in a pre-defined template.

None of the other disclosed uses of the Transaction Correction tool contain the relevant SQL INSERT commands such as: ‘INSERT INTO ops$brdb.brdb rx_rep_session_data ...’

2. Usage of the Transaction Correction Tool is recorded in the BRDB_TXN_CORR_TOOL_JOURNAL table within the BRDB.

The tools Low Level Design sets out that it has the capability to insert entries into the following tables:

BRDB_RX_BUREAU_TRANSACTIONS

BRDB_RX_EPOSS_TRANSACTIONS

BRDB_RX_APS_TRANSACTIONS

BRDB_RX_EPOSS_EVENTS

BRDB_RX_NWB_TRANSACTIONS

BRDB_RX_REP_SESSION_DATA

BRDB_RX_DCS_TRANSACTIONS

BRDB_RX_REP_EVENT_DATA

BRDB_RX_CUT_OFF_SUMMARIES"

690.

Of the agreed entries in relation to Horizon Issue 11, I find the following of very great relevance. The experts are agreed that for any large commercial IT system, which Horizon obviously is, it is necessary for some technical users to have privileged access to databases with wide-ranging capabilities. This however, is not for application related purposes, but is for system maintenance and problem-solving purposes. It is also agreed by both IT experts that it is important that the number of users with these abilities is kept down to the minimum possible, AND for each action that they take whilst logged in to be recorded and audited (my use of block capitals).

691.

I consider that to be entirely conventional, obviously agreed by both the experts, and also to be technically justified, particularly in a system such as Horizon which deals with accounting. It is also not requiring an unrealistic level of perfection; it is simply a matter of having properly controlled access to a system such as this one.

692.

However, when one then considers the subsequent passages of the 4th experts statement, it can be seen how far from this joint agreed (and technically justified) position the Horizon system was. The logging of Privileged User Access (in PUA logs) commenced in October 2009. For the period 2009 to 2015 – obviously a 6 year period - these logs only displayed the fact that a Privileged User had logged on or off, “but not what actions they had taken whilst the Privileged User was logged in”. Therefore the actions they were taking when logged in were being neither recorded nor audited. All that could be seen is they were logged in. Further, it has already been seen that the number of users with the relevant privileges was not, in my judgment, restricted to a minimum. Further, the use of the Transaction Correction Tool cannot be seen in these logs. Yet further, the experts are agreed that at all times, any privileged user access log only shows what tables of BRDB were accessed for a very small minority of accesses.

693.

Yet further, it is agreed by the experts that there are no privileged user access logs which show access to the two tables of BRDB used to insert the one acknowledged balancing transaction. Indeed, one of Dr Worden’s entries in the experts’ joint statements was

“The Privileged User Access logs are not a useful source of evidence about remote access, including Balancing Transactions.”

694.

What this amounts to, in my judgment, is a serious deficiency both in the required level of controls in Horizon, in the recording of what privileged users were actually doing (other than that they were simply logged on) and also a corresponding absence of recording and auditing of those activities.

695.

In my judgment, and based upon the experts’ agreement in this respect, this serious deficiency lasted up to 2015. It also means, because of the extent (or rather the power) of the APPSUP privilege, the Transaction Correction Tool is but a small subset of the way that records in the BRDB can be affected or changed by Fujitsu. It also means that the tool’s Low Level Design document that was available in the trial bundle, and considered by the experts, has been overtaken by use of the Transaction Correction Tool, because the SQL INSERT command is not present in 2,296 acknowledged uses of the tool. Both the number of uses, and the non-use of the SQL INSERT command, are in agreed passages by the experts in the 4th Joint Statement. This is also a point that was put to Mr Godeseth in his cross-examination (and accepted) and is dealt with at [339] to [344] above.

Lasting and transient impact

696.

Of the total number of bugs accepted by the Post Office, the Post Office submits that nine of them had, or potentially had, only transient impact. These are the following:

Bug 2: Callendar Square;

Bug 4: Dalmellington;

Bug 5: Remming In;

Bug 6: Remming Out;

Bug 7: Local Suspense (not to be confused with Bug 3. Suspense Account bug)

Bug 9: Reversals;

Bug 12: Counter Replacement;

Bug 19: Post & Go/TA Discrepancies in POLSAP; and

Bug 25: Lyca top up.

697.

Transient impact is a concept introduced by Dr Worden. It does not form any part of the Horizon Issues as formulated, agreed and ordered. Dr Worden agreed, in his cross examination, that he had added in the word “lasting” to Horizon Issue 1. He also said: “…for better or worse I took my role to be concerned with lasting effects on branch accounts rather than transient ones.”

698.

This is a plain and obvious departure from the wording of Horizon Issue 1, and indeed in my judgment represents a narrowing of it. I consider this departure unjustified. It is not the issue that was drafted and agreed by the parties, approved by the court and ordered to be tried.

699.

Dr Worden also said in the 2nd Joint Statement (in one of the passages that was not agreed) that “transient inaccuracies in branch accounts, which needed some form of correction, have arisen so frequently and from so many causes that to list them is not useful; and that evidence of each correction being carried out is unlikely to persist to this day.”

700.

He stated that “not rectified” means lasting, whereas “rectified” means transient. He also stated that if a bug caused something that required correcting with a Transaction Correction or TC, then that would mean (if the TC were issued) that it were a transient effect; but if the TC was never issued, that would be a lasting effect. The passages were:

“Q. So if the TC process takes a long time, then you might say something was lasting, even notwithstanding years later it might be corrected?

A. That is not how I understood lasting –

Q. How did you understand lasting?

A. There were delays in the TC process which might be due to client organisations or might be due to all sorts of things, and they could be at the outside, I believe, several months. My definition of lasting did not depend on TCs coming in within a certain timeframe. If TC never came in, that would be lasting, but if TC took several months to come in that is not what I would call a lasting effect. A lasting effect is permanent; it is at the end of the day, you know, he has lost money forever.”

701.

The use of the phrase “lost money forever” demonstrates the artificiality, in my judgment, of this distinction. Whether this was the intended purpose or not, applying such a concept is to minimise as much as possible the impact on branch accounts of bugs, errors and defects that otherwise exist. He also stated that “the branch accounts would look wrong” for the period prior a TC being issued. That is undoubtedly correct; the accounts would not only “look wrong”, they would be wrong. This makes it even more clear that Dr Worden’s approach is at odds with the wording of the Horizon Issues, as well as with the way which the Horizon System was operated.

702.

His approach is, in my judgment, wholly artificial, for at least five reasons.

1.

It ignores the express wording of Horizon Issue 1 itself.

2.

It ignores the very function of Horizon, which was used as the branch accounting mechanism between SPMs and the Post Office. A SPM would “roll over” their branch each trading period, usually a 4 week period (sometimes a 5 week period). At the end of each trading period, the SPM would produce a Branch Trading Statement (or “BTS”) using Horizon, which was the accounting statement of the branch for that period. Any shortfall or discrepancy in the accounts as demonstrated by the BTS would have to be made good by the SPM (which means the amount of the loss would have to be paid) or “settled centrally” (which means the SPM would seek time to pay). Therefore, any occurrence of any bug, error or defect within that trading period would have an impact on the branch accounts (by reason of the regular production of the BTS) at the end of that particular 4 week trading period.

3.

The issuing of TCs is done by the Post Office outside of Horizon in any event.

4.

It pays no attention to whether a TC is issued within a branch trading period (and hence before the creation of the Branch Trading Statement for that period) or afterwards, in the next or any subsequent trading period. The fact that (for example) a TC might be issued some months later does not mean that the branch accounts have not been impacted. The branch accounts specifically have been impacted by the occurrence of the bug, error or defect.

5.

The correct characterisation of a bug, error or defect within Horizon, and its impact, cannot sensibly depend upon an act (the issuing of a TC) that is done outside Horizon. If a SPM were to find themselves reporting, for the first time, an incident to SSC in relation to a hypothetical bug, bug H, and upon investigation SSC eventually were to accept that there was a bug and a software fix was required, then the nature of bug H should not depend upon how the PO chooses to correct it. If, say, a SPM had retired, closed their branch, been suspended or in any other way ceased to be a SPM, then a TC could not sensibly be used to them. One assumes that in this hypothetical scenario, the PO would refund the SPM (or their estate, if they had died) but that step should not affect the characterisation of bug H.

703.

The point at (2) above is consistent with the way that branch accounting works. This was the subject of a considerable amount of evidence at the Common Issues trial which is dealt with in Judgment (No.3). I do not know if Dr Worden had read this judgment, and he was not asked about it. However, I do not consider that any aspect of the outline accounting process operated by the SPM using Horizon at the end of each Branch Trading Period is controversial.

704.

In my judgment, if any bug, error or defect within Horizon could have an impact upon branch accounts in any particular trading period, then unless a TC were issued within the same trading period, it would have had an actual impact upon branch accounts. Similarly, if any bug, error or defect within Horizon could have an impact upon branch accounts in any particular trading period, then – regardless of the issuing of any TC in respect of that occurrence – it would have the potential to impact upon branch accounts.

705.

This can be tested by considering the Callendar Square bug, number 2 in the bug table list. The evidence I have identified at [401] to [414] above makes it clear that this bug was discovered in 2005, and Mr Godeseth accepted that it had probably been present in Horizon since its inception in 2000. It affected at least 30 branches, and was recorded in one of the contemporaneous documents as “this problem has been around for years and affects a number of sites most weeks.” Yet on Dr Worden’s somewhat artificial constraining of impact on branch accounts, it would be categorised as only having a transient impact. The dictionary meaning of transient is fleeting, passing, brief, temporary, momentary, transitory, short-lived or ephemeral, and plainly does not apply to the effect of this admitted bug. In my judgment the use of this word by Dr Worden and the Post Office is unjustified. The use of it by Dr Worden was unnecessarily to restrict his analysis.

H. The Period 21 March 2019 to 4 June 2019

706.

During this period two matters of note occurred in the group litigation (in addition to hearing the recusal application and handing down of Judgment (No.4), which it is not necessary to address in this judgment).

707.

The first was the further disclosure of documentation by the Post Office, which is explained further at [616] above.

708.

The second was service of a further supplementary expert’s report by Dr Worden directly upon the court. This occurred on 22 May 2019. It came under cover of an email from Dr Worden, copied only to the Post Office’s solicitors. It is extremely unusual, if not verging upon unheard of, for an expert witness to communicate directly with the trial judge, and directly serve material upon him or her. It is also the case that no party (whether by itself or its solicitors) in ongoing litigation should communicate with the court without copying in the other parties to the case. The only very narrow exception to this is ex parte proceedings, in respect of which there are special rules, and which is not the case here.

709.

In the interests of transparency, I reproduce the whole email from Dr Worden here. It was accompanied by some attachments: “Third Report Open”; and three appendices. The email stated:

“Dear Judge,

As was explained by Mr de Garr Robinson in court on 11 April, I have made some further analyses of Horizon Issues 1, 12, and 13. This work was done at my own instigation, and was not prompted by the Post Office or its lawyers. In my opinion this work has led to a material change to my opinions, and I am therefore obliged to inform the court of those changes in a report. With this email I am sending you the report and its Appendices, in completion of the action. I am doing this on the basis that the trial will re-convene on 4th June, and so the court may need to be aware of it then. The report is 13 pages long.

As the court is aware from my first expert report, I have interpreted ‘extent’ in Horizon issue 1 as requiring a number - the maximum proportion of the claimed losses which is attributable to bugs in Horizon. My first estimate of that number, in my first report and revised in the second expert joint statement, is 0.4%. This estimate was adjusted in a direction to favour the claimants, so that the court may better rely on it.

In my opinion, that estimate is still sound. The enclosed report contains an independent estimate of the same number, which is 0.6%. Because that estimate is based on a simpler analysis, and relies on evidence that in my view can be checked over a weekend, I believe it can be of assistance to the court. That is why I started this analysis, and why I am now sending it to the court.

Similarly, for Horizon Issues 12 and 13, the analysis in the attached report is in my opinion simple, easy to check, and leads to results which I was not able to derive before. Therefore I believe it can be of assistance to the court.

A draft version of this report was first provided to Mr Coyne on 25th April, and a final version was sent to him on 16 May. I have not yet been able to have a substantive discussion of this report with him, other than my explaining points in the report to him.

I understand that when cross examined on the Horizon issues, this report is now a part of my opinion and I may need to refer to it. It is also my understanding that Post Office does not intend to make any application to the court in relation to this report.

Yours Sincerely,

Robert Worden”

(emphasis added)

710.

The reference to what had occurred on 11 April 2019 is as follows. Following dismissal of the recusal application on 9 April 2019, the trial resumed on 11 April 2019 with the resumption of the Post Office’s final evidence of fact. This involved the Post Office calling Mr Parker to be cross-examined, and explaining that Mr Membury (another of the Fujitsu witnesses) was unable to be called, and the relevant Civil Evidence Act notice being given under CPR Part 33.2. The position of the claimants on that latter point was explained as their consenting to the necessary extension of time (rather than a waiver by them under section 2(3) of the Act, a difference that is immaterial in the circumstances). Mr Membury’s witness statement was therefore not cross-examined upon, but Mr Parker was called and was cross-examined.

711.

Upon conclusion of Mr Parker’s evidence, Mr de Garr Robinson, leading counsel for the Post Office, explained that, as he put it, “a matter of some awkwardness” had arisen. He submitted the following:

“Dr Worden has recently realised that there is a new way of looking at the evidence in this case which, in his view, could greatly assist your Lordship, assist the court, in deciding Horizon issues 1, 12 and 13.

This approach involves focusing on those PEAKs, OCRs, OCPs and MSCs which actually mention the FAD codes of one or more of the claimant branches. Just to explain, when Fujitsu did any authorised remote handling of data, to put it neutrally, which might affect branch accounts, they raised an OCP, OCR or MSC whose text was likely to include the six digit FAD code of the relevant branch. So it's therefore possible to search all the OCPs, OCRs and MSCs with a view to finding all of those which mention the claimant branches during the relevant claimant's period of tenure. This search yields a limited number of OCPs, OCRs and MSCs, and it's therefore possible to assess expert issues 12 and 13, which is how often was remote access facility exercised and what effect did it have. It is possible to assess those questions as they affect the claimants by examining that much smaller document set. My Lord, that is the first exercise that he would like to undertake, and indeed he has embarked work on -- I think this week he has embarked work on that.

Second, if a detected bug affected the accounts of any branch the Peak relating to that bug was likely to mention that branch's FAD code. Typically, it will also -- it may also mention a sum of money. It's therefore possible to search all the PEAKs in the same way that I have just outlined, looking for PEAKs which mention any claimant's FAD code during the relevant claimant's period of tenure, and again, this document could shed some light on Horizon Issue 1 to which extent is it likely that bugs have affected the relevant branches.

Now, Dr Worden has specifically asked me to offer his apologies to the parties and to the court that he didn't think of this before. In fact, frankly, he is kicking himself that he didn't do so. He believes that he and Mr Coyne would only need a short time to consider the relevant documents and to consider how it affects their views on those issues. He wishes to discuss the documents with Mr Coyne with a view to agreeing what they do or do not show.

My Lord, in the days since the recusal application was issued he started to consider how the new approach affects his views. He believes on Issue 1 it allows the parties to make a much simpler analysis of the point, and he takes a similar view in relation to the remote access issues. It makes, in his view, it possible for the experts to form a view as to how often remote access was exercised and what its likely effect was. It is Dr Worden's view that it is his duty under CPR Part 35 to inform the court of this change of view and to allow the court to consider whether or not it wishes to see it considered. That belief is based, as your Lordship will be aware, on CPR35.3 which imposes a duty on experts to help the court on the matters within their expertise, whether or not they are instructed so to do. My Lord, it's also based on CPR35 -- I should say the practice direction CPR35, paragraph 2.5, which requires experts to inform the court of any change of views.

I should emphasise this -- none of this comes at the request or instigation of my client. This has come from Dr Worden. This is his idea. My Lord, he wishes to discuss it with Mr Coyne in a further meeting between the experts, but of course it's -- it's only right that your Lordship should be aware of that. I'm not making any application for permission to put in supplemental expert reports –

Mr Justice Fraser: I don't think you have any supplemental experts' reports to apply for permission for, do you?

Mr de Garr Robinson: I'm not making any kind of application, I'm simply sharing with your Lordship the view that has been expressed to me by Dr Worden.”

712.

The Post Office, in continuing this explanation of what was happening, referred to another case which had concerned late expert evidence, the judgment in which had been referred to in the opening submissions for the Horizon Issues trial. That case is Imperial Chemical Industries Ltd v Merit Merrell Technology Ltd (No.2 Quantum)

[2018] EWHC 1577 (TCC).

713.

The exchange with the court continued:

“Mr Justice Fraser: …In order to apply for permission to adduce extra expert evidence you would have to have a draft of the report for which you would be seeking permission, wouldn't you?

Mr de Garr Robinson: My Lord not necessarily.

Mr Justice Fraser: You don't think so?

Mr de Garr Robinson: My Lord, I would submit not. It would depend on the circumstances. Often one would have such a report. I'm conscious that in the ICI case your Lordship cited as a reason for not giving the relevant party permission to put in a report which they had prepared, that the experts hadn't gone through that collaborative process and I'm quite anxious to ensure that my expert doesn't fall into the same trap, if I can put it that way.

Mr Justice Fraser: Well, depending on whichever point one reaches in terms of your actually make an application to put in a supplementary expert's report, that application will be dealt with as and when it's made, so I'm not dealing with that either positively or negatively at the moment.

Mr de Garr Robinson: Thank you.”

714.

It is not necessary to dwell upon the point, but any party seeking to obtain permission to rely upon a very late additional expert’s report should ordinarily expect to have at

least a draft of such a report available. It is difficult to see how permission could be given in general terms for late expert evidence, the content of which was not available. However, no such application was made by the Post Office in any event. The trial had commenced on 11 March 2019. Dr Worden’s first expert report was dated 7 December 2018 and his supplementary report was dated 1 February 2019. This notification to the court on 11 April 2019 of Dr Worden having “recently realised that there is a new way of looking at the evidence in this case” came over four months after the date of his first report, over two months after the date of his supplementary report, and exactly one month after the Horizon Issues trial itself had actually started. I requested a witness statement from the solicitors for the Post Office setting out the chronology of this realisation by Dr Worden.

715.

The case to which reference was made on behalf of the Post Office, ICI v Merit Merrell, concerned a supplementary report by a quantum expert, instructed on behalf of ICI, which was produced on the first day of trial without warning. This was based on an analysis of documents the expert had only recently been provided with by ICI, which it turned out had been in ICI’s possession for over three years, but which the expert had not previously been given. The application for permission to adduce that supplementary report in that case was refused. Points relevant to the application in that different case on those facts were set out in [158] of the judgment, which stated:

“Two other points relevant to the application should also be identified. Firstly, Mr Kitt embarked upon this exercise with no notice at all to his opposite number, Mr Linnett. This is not co-operative behaviour. Mr Linnett should have been told that this was being done, and the documents could have even been studied together by both experts so that an agreed position could be produced by them on what the documents did or did not show, and the use to which they could or could not be put. I conclude that the only reason for keeping the fact that this exercise was underway, until after it was ready to be served after the trial had actually started, was to cause maximum disruption to MMT. This is a much discredited approach to civil litigation. I conclude that this can be the only explanation for giving Mr Kitt the documents in a more usable (and searchable) format than Mr Mills, MMT's solicitors.”

716.

Giving permission for late expert evidence is a matter of discretion, and each case will be different. At [237] in ICI v Merit Merrell I had stated the following, as one of a number of principles that relate to expert evidence:

“5. Where late material emerges close to a trial, and if any expert considers that is going to lead to further analysis, consideration or testing, notice of this should be given to that expert's opposite number as soon as possible. Save in exceptional circumstances where it is unavoidable, no expert should produce a further report actually during a trial that takes the opposing party completely by surprise.”

717.

Here, the following distinct matters fall to be considered. Firstly, just because in the ICI case that expert performed a new exercise without telling his opposite number he was doing so (which led to criticism), does not mean that all an expert has to do is to inform his opposite number during a trial that he has decided to undertake a new exercise, or produce a new report, and that by giving such advance notice this means the new report will be admitted. The whole ethos of exchange of expert evidence well in advance of trial is so that each party (and their respective experts) have advance

notice of the opposing point of view, so that evidence can be properly addressed and surprise avoided. That is undermined by service of very late reports containing new analyses. Sometimes this is unavoidable and late material emerges. It is not possible to be entirely proscriptive, as each case is different. That is why permission of the court is required. Secondly, the stage of the trial at which this 3rd Report was produced by Dr Worden was very late indeed. It was weeks after the trial had started. Thirdly, the extent to which the court needs to consider all the circumstances depends upon whether the 3rd Report was sought to be relied upon by the Post Office. Finally, and of wider application, is the way in which the Third Report was dealt with, namely by service by the expert himself directly upon the court.

718.

Dealing with those matters in turn, the first two are effectively bound up with one another. I do not know the actual date when Dr Worden realised that this new way of looking at things appeared to him to be useful. However, the court was told he had started thinking about this on 11 April 2019, and the claimants’ solicitors had already been told of his intention by the Post Office’s solicitors the previous day on 10 April 2019 (one day after the recusal application was dismissed, and the day before resumption of the trial). Thereafter, a draft report was provided on a without prejudice basis to Mr Coyne on 25 April 2019 by Dr Worden, his “current workings” having been provided on the same basis on 11 April 2019 and the appendices, again on a without prejudice basis on 25 April 2019. There was then a degree of to-ing and froing between the solicitors, which led to the actual report itself being sent on an open basis to Mr Coyne on 16 May 2019 (after another experts’ meeting on 15 May 2019).

719.

Regardless of the exact date when Dr Worden realised there was a different way of looking at the evidence in the case, this 3rd Report emerged at an extraordinarily late stage in the proceedings. As can be seen in [721] below, it was in any case based on material which Dr Worden accepted he had had for some time. It was not based on material he had only received late, for whatever reason.

720.

Turning to the third of the matters identified in [717] above, this can be readily dealt with. The Post Office made it clear, both on 11 April 2019 in the passages which I have quoted above, and subsequently, that it did not seek permission to adduce the contents of Dr Worden’s 3rd Report in evidence. It issued no application in this respect. Therefore, it was not necessary for the court to consider the balancing exercise inherent in resolving any application in this respect (on the basis that such an application was resisted by the claimants, which was likely). It is inherent within this decision by the Post Office not to issue an application to adduce it therefore, that there was no explanation of when or how Dr Worden had experienced this “new way of looking at the evidence”, other than what has been identified above. It also meant that this report was not adduced as any part of his evidence-in-chief, and he was not crossexamined about it. He referred to its existence only once, very much in passing, and given that the contents of it had not been adduced in evidence, that was as far as it went.

721.

Had I been faced with a contested application by the Post Office to be permitted to rely upon the contents of the 3rd Report in evidence, the matter would have been considered in accordance with the principles in the CPR generally, CPR Part 35, the Practice Direction that goes with it, as well as the Guidance for the Instruction of Experts to Give Evidence in Civil Claims 2014 (“Guidance of Experts in Civil Claims”) which is in the White Book with the Practice Direction. Doubtless such an application, had it been made, would have been supported by a detailed explanation of all the relevant matters, together with argument on both sides. I do not have the benefit of either a detailed explanation or of any argument. On the material of which I am aware, but without hearing any submissions on it for obvious reasons, a substantial hurdle to the success of such an application would have been its lateness. This is because of the enormous disruption and yet further delay that would have been the inevitable consequence of permitting reliance upon such a late expert’s report. Severe disruption and delay had already been caused to the Horizon Issues trial by the time this report emerged, and by late May, the trial was shortly about to resume after an enforced interval, in very unusual circumstances, of over two months. Had this new report been admitted into the evidence, Mr Coyne would have required time to study it, and then potentially produce a further expert’s report of his own, and none of that could have been accommodated between late May (whenever such an application could have been listed) and 4 June 2019. Calling him would either have been delayed, or he would have had to be recalled later to deal with it. Further delay to the Horizon Issues trial would have been undoubtedly caused. Delay is not always fatal to such applications, but the later an application of this kind is made, the stronger the explanation for that delay has to be.

722.

The claimants had also made the point on 23 May 2019, the day after Dr Worden had served his 3rd Report upon the court, that it was based upon an analysis of documents, namely OCPs, OCRs and MSCs, that Mr Coyne himself had expressly requested in July 2018 and which the Post Office’s solicitors had then said were irrelevant. In any event, when I asked him at the end of his cross-examination, Dr Worden confirmed that he had had the documents upon which he had based this 3rd Report “for some months” and that he had had the full set of PEAKs since, as he put it, “way back in 2018”. As the Post Office’s own counsel explained on 11 April 2019, Dr Worden was “frankly, kicking himself” that he “didn’t think of this before”. The lateness seems therefore to have been the consequence of Dr Worden failing to consider this approach until very late in the day, rather than only very recently being provided with the material. That is not promising ground for the making of such an application. It is also the case, in my judgment, that there were no exceptional circumstances which made it unavoidable that this new report was produced during the actual trial. Indeed, rather to the contrary. This was entirely avoidable. Finally, the covering email from Dr Worden made it clear that his estimate, included in his first report and revised in the 2nd Joint Statement of 0.4%, was “still sound”. This new exercise simply sought to bolster his conclusions. It was not an explanation of a change of opinion by him.

723.

I turn therefore to the fourth matter, the issue of direct communication with, and service upon, the trial judge of an expert’s report by the actual expert rather than through solicitors.

724.

When the email was received to which I have referred at [708] above, my clerk immediately sent a copy of the email and attachments to all the legal advisers for both parties in the litigation, stating that witnesses should not communicate directly with the court, and also requesting that the claimants should not read any of the attachments until the matter was addressed in court the next day (when the parties were attending in any event to argue costs orders and other consequential matters following Judgment (No.3)).

725.

The legal advisers for the Post Office did not seem concerned at the hearing of 12 April 2019 that this direct service upon the court by the expert himself had occurred. An analogy was drawn with an expert seeking directions under CPR 35.14, and it was dealt with by the Post Office as though the direct service upon the court was simply the actions of an expert seeking in the normal way to fulfil his duties to the court, and was entirely usual. It is, however, not usual; it is highly unusual. Further, the covering email from Dr Worden makes it clear that the new report was not a change of opinion; in particular, the sentence that makes this entirely clear is “in my opinion, that estimate is still sound” concerning the calculation in his earlier reports. His exercise was simply a different way of reaching the same conclusion. Therefore, the provisions of paragraphs 64 to 66 in the Guidance of Experts in Civil Claims do not apply.

726.

I do not consider that any witness, lay or expert, should communicate directly with the trial judge about the content of their evidence, or indeed about any subject at all. There is a substantial risk that the efficient management of proceedings will be compromised if this is done. Communications with the court should always be done by solicitors, if they are on the record. Further, communications between an expert and those instructing him, or her, are privileged. If, as here, an expert takes it upon himself to inform the court of applications that his client may, or may not, be intending to make, there is a real danger that privileged information might be divulged, both to the court and also to the other side in the litigation. This is far less likely if communications are performed by solicitors, who have specific qualifications in legal matters and will be experienced in litigation. Finally, there is specific provision in the Guidance of Experts in Civil Claims dealing with the situation where an expert wishes to exercise their right to ask the court for directions. This is contained in paragraphs 28 and 29. These proposed directions have to be sent to the party instructing the expert at least seven days before the request is made, and to the other parties in the litigation at least four days before. They are to be made by letter in the form prescribed at paragraph 29(a) to (e). That gives the other parties the ability, firstly to have advance notice of what is proposed, as well as providing advance notice that the court will be asked for directions and what those directions are, and why; and also give the expert’s instructing solicitors the ability to consider the communication in advance so that privileged information is not included.

727.

In this case, that process was not followed. Mr Parson’s 17th witness statement explained the outline chronology, and made it clear that Dr Worden’s opposite number Mr Coyne had a copy of the 3rd Report, initially on a without prejudice basis, and then on an open basis, and that Dr Worden told Mr Coyne on 16 May 2019 (when he sent it on an open basis) that he would be sending the report to the court but not before 22 May 2019.

728.

Finally on this subject, when I asked Dr Worden at the end of his evidence whether he had ever preformed this type of direct service himself on the court before, he told me he had not. He also said that he had sent the email direct to my clerk as this was what he had been advised to do by the “Post Office’s lawyers”. He said that he had not given his instructing solicitors the opportunity to check or approve the covering email, but “had been given some advice as to what it should cover”. I assume that advice did not include telling Dr Worden that he should inform the court of the Post Office’s potential intentions on what applications it may, or may not, be minded to make in the litigation. This is a highly unusual situation. In closing submissions the Post Office invited me to provide guidance in this judgment as to what should be done for future cases, as though the position was unclear. I do not know why it should be thought necessary to tell witnesses – lay or expert - that they should not unilaterally communicate with the court. However, if it is necessary, no witnesses should communicate with the court directly in this way. Any communications from legal advisers should always be copied to the other parties in litigation. No expert should inform the court directly what applications his or her solicitors might be contemplating, or not contemplating, in the future conduct of the litigation.

729.

The points that I draw from this episode are as follows:

1.

Experts should not serve their own material, particularly new material during a trial, and whether or not accompanied by detailed covering emails, directly upon the court themselves. Service should always be done by a party’s solicitors if solicitors are on the record.

2.

A party’s solicitors will know that all communications to the court must be copied to the other parties in the litigation.

3.

Experts should not notify the court directly of the intentions of their instructing solicitors and/or ultimate client in the litigation in which they are instructed. Such matters are privileged.

4.

It is difficult to be critical of Dr Worden in this respect, given he told the court that he was advised by the Post Office’s legal team to send his report directly to the court. Advice of that type should not be given to experts.

5.

Dr Worden must have considered that the documents upon which he relied were, at least, relevant. These were documents of a type sought by Mr Coyne in July 2018, whose relevance was disputed at that stage by the Post Office. Mr Coyne’s request ought to have led to these documents being provided to both the experts in the middle of last year, a reasonable period of time after Mr Coyne specifically asked for them.

730.

I will now turn to each of the expert’s evidence, their reports and their approach to the Horizon Issues.

I. The Expert Evidence

731.

The subject matter of this group litigation spans a long period. Mr Bates was a SPM before Horizon was introduced in 2000, and it is from Horizon’s introduction that year that, on the claimants’ case, the problems and difficulties arose, and arose as a result of the functionality of the computing system introduced by the Post Office, which in this judgment is referred to as Legacy Horizon. As the Post Office pointed out regularly during these proceedings, there are thousands of branches and millions of transactions each year in all of those branches, which means that over such a lengthy period, there is a vast amount of computing activity that had to be considered by the experts.

732.

The expert evidence was therefore required to be substantial. The aspects of the litigation that require expert evidence are not, however, of unheard of proportions, so far as the High Court is concerned. The High Court often deals with matters that span a lengthy period; it also often deals with matters of great complexity that require expert evidence. By comparison with many other cases, one expert discipline alone, with only one IT expert per party, is not a particularly expert-heavy case. The experts’ reports, though lengthy, were not of an unusual size. There were two reports from each expert that were adduced in evidence, and each of them was of a type that would be expected in a trial of medium complexity, which I consider the Horizon Issues trial to have been. Nor was it necessary for the experts to examine every claimant SPM’s separate experience; the whole purpose of having generic Horizon Issues was to avoid precisely that. The Post Office would often concentrate on how vast the panorama was before the court, but this may have been done simply to emphasise how few, in relative terms, the claimants are; which also emphasised at the same time how well (so far as the Post Office generally is concerned) the Post Office business operates day by day, week by week, across so many branches. The experts agreed that Horizon has produced over three million sets of monthly branch accounts.

733.

At times, however, this focus on scale did seem to come close to an assertion that the IT experts had an impossibly vast task; or that the court faced a significant challenge even in understanding all the expert evidence; or that the parties could not practically deal with the other side’s expert evidence in the time allotted.

734.

As has been set out in Part G of this judgment, the experts reached a great deal of agreement in their different statements. Notwithstanding that, there remained large areas of disagreement. Given the modern approach to trying cases generally, it was neither possible, proportionate nor desirable to give the Post Office as much time as their advisers maintained was necessary to conduct the Horizon Issues trial. As an interesting comparison, in the case of The Ikarian Reefer dealt with further below, which was before the implementation of the CPR and concerned the US$3 million loss of a vessel, that trial took no fewer than 87 trial days. Such a lengthy trial, post the introduction of the CPR, would be highly unusual in a case of the financial scale of this one.

Time limited trials and the approach to detailed evidence

735.

Prior to the introduction of the Civil Procedure Rules, fewer trials were time-limited and some trials dealing with a large number of complex issues could run for a great many months, if not far longer. Since the introduction of the overriding objective, which is contained in CPR Part 1.1(1), this states “These Rules are a new procedural code with the overriding objective of enabling the court to deal with cases justly and at proportionate cost”. That is explained further in Part 1.1(2), which states that:

“Dealing with a case justly and at proportionate cost includes, so far as is practicable: (a) ensuring that the parties are on an equal footing;

(b)

saving expense;

(c)

dealing with the case in ways which are proportionate –

(i)

to the amount of money involved;

(ii)

to the importance of the case;

(iii)

to the complexity of the issues; and

(iv)

to the financial position of each party;

(d)

ensuring that it is dealt with expeditiously and fairly;

(e)

allotting to it an appropriate share of the court’s resources, while taking into account the need to allot resources to other cases; and

(f)

enforcing compliance with rules, practice directions and orders.”

736.

Since the Civil Procedure Rules, trials are usually set down for a fixed time estimate and are expected to be time-tabled to finish within that period. The court system does not have the luxury (if luxury it is) of giving endless periods of time to litigants to argue their cases to the nth degree. This case, as with all litigation, is very important to those involved in it – to all of the claimants, but also to the Post Office, as an institution generally, and also to its other directors, employees, SPMs who are not claimants and its clients too. The issues and different causes of action in both claims and counterclaims are of considerable importance, and the amount of money at stake is not small. However, the amount of money is not large in terms of specialist litigation, for example. It is not measured in the billions, or even many hundreds of millions. Ignoring claims for general damages, the amount of damages that are specified in liquidated sums are said to be about £18 million (although that is only an approximate figure currently). There are, of course, also reputational issues at stake as well as money.

737.

Judgment (No.5) [2019] EWHC 1373 (QB) made it clear at [78] that the total of the parties’ reported costs as at the date of that judgment was over £25 million (£12.6 million for the claimants, and £12.8 million for the Post Office). Since then, on 25 June 2019 the Post Office notified the court that its costs had exceeded £13.9 million.

738.

The reason for explaining this in detail is to put into context the following points about the scope of expert evidence, and the number of trial days available for the experts to be cross-examined. At the Pre-Trial Review, the Post Office submitted that at least four days was needed for the cross-examination of Mr Coyne. It was realistically accepted that longer than that was not likely to be permitted, although leading counsel for the Post Office did submit that he would prefer longer. The claimants submitted that they also wanted four days, but had trimmed that period to three days, for the cross-examination of Dr Worden. Given there were further experts’ meetings to come, and given my view of the scope of the expert material and considering the overriding objective, I set a trial timetable that permitted each side two and a half days to cross-examine the other side’s expert. That would have given a total number of five days of cross-examination for all the IT expert evidence. At that time, I considered that would be sufficient, and also proportionate, taking account of the overriding objective.

739.

However, the subsequent Joint Statements following the PTR did not reach as much agreement as I had (perhaps naively) anticipated. I therefore adjusted the trial timetable and set a new one, with the Post Office being given four days to crossexamine Mr Coyne, and the claimants being given three days to cross-examine Dr Worden. That trial timetable also allowed an interval of one non-sitting week, to follow the evidence of fact, for each side properly to prepare their time-limited expert cross-examination in detail.

740.

In my judgment, the period of time I had allotted at the PTR of 2½ days each was sufficient in itself; but the increased period of time allowed was undoubtedly sufficient for each party properly to cross-examine and test the expert evidence adduced by the other side in the litigation. Seven court days for the total crossexamination of one expert discipline in a case such as this is, given the Horizon Issues in this trial, generous. Counsel might not have had time to put each and every single individual point to the expert on the other side, but they had more than enough time to put the main ones (and a considerable number of subsidiary ones too, for that matter). Each side had leading counsel, well-versed in all of the issues, who have been involved in the case from at least the making of the GLO itself in early 2017, as well as a team of other counsel assisting too. Considerable pre-reading had been done by the court. As it turned out, the issuing of the recusal application on 21 March 2019 led to an even longer period of the trial not sitting, with only a half day (for Mr Parker) of the trial taking place between 21 March 2019 and 4 June 2019, a period of 75 days, or over ten weeks. The Post Office had a different counsel team acting on the recusal application than the team acting on the Horizon Issues trial. That 75 day period was a substantial period of time for each side to prepare detailed cross-examination on all, and any, points that they considered important, prior to the resumption of the trial and the commencement of the expert evidence on 4 June 2019.

741.

The amount of the costs burden explained in [737] above has been incurred on the basis of the number of trial days made available. Obviously, further time for crossexamination would have led to even higher cost figures. There was a sufficient and proportionate amount of time at trial made available for the expert evidence to be properly challenged and tested, and there was more than sufficient time available for preparation of the time-limited cross-examination. I have set this out in detail in order to make it clear that the time limited nature of the Horizon trial, and the number of days allotted for cross-examination, has in my judgment had no effect on my findings on the Horizon Issues, or restricted either party’s opportunity to put its own case and test the case of the other party.

742.

The other feature of time-limited cross-examination, given the depth of experience of counsel, is that I permitted considerable latitude to the cross-examiners. If counsel wanted to put the same point to an expert again and again; or spend an hour on one specific sentence or detail, when the same point could have been accomplished in five minutes; or devote significant time to high-level issues and avoid descending to specifics; then that was a judgment call made by that counsel. With more junior counsel, in another case, the court might discourage excessive time on less important or agreed points, to encourage progress; but here both sides had highly experienced leading counsel. I therefore intervened rarely. The amount of time made available in the trial timetable for each side to cross-examine the other’s expert was set down in advance, and if one side wished to devote (say) half a day to having an expert state, and then restate, an agreed position on a particular point, then that was very much a matter for them. Such an approach would lead to that period not being available to allocate to other subjects, but in my judgment this case in particular benefitted from allowing the parties to conduct their cross-examination, more or less, in the direction they wished.

743.

I did, from time to time, seek either to have clarified what particular Horizon Issue was the subject of the cross-examination, or to reference it to particular other subjects.

However, this did not occur very often, and the parties were permitted to use their allocated time as they wished.

744.

Each side, both in their cross-examination and closing submissions, criticised the other side’s expert for not being sufficiently independent. This was done in different terms and in different ways, but the upshot of it was broadly the same. It was that their expert’s evidence should be preferred to that of the opposing side, for reasons of methodology and approach, as well as for reasons of detail.

745.

The court, when faced with opposing experts, might dismiss the views of one expert entirely on the basis that they had not complied with their duties to the court. However, it is not necessary to do that, in order to prefer the views of one expert over another. There may, depending upon the subject matter, be room for opposing expert opinion, and it is simply a case that the court accepts one opinion for a variety of reasons. Also, in some cases, it might be that one expert’s view is accepted on some points, and the opposing expert’s views on others.

746.

It is also not necessary to resolve every single point of detail between experts to arrive at the answers to the Horizon Issues. However, given these criticisms of the experts have been made, it is necessary to address them. This is because it is only fair both to the experts, and to the parties, to decide whether each expert has been sufficiently impartial in the exercise upon which they have been engaged such that the court can reliably base its answers to the Horizon Issues upon the conclusions reached by the experts. Impartiality and independence of experts is also of considerable, if not central, importance to the administration of justice generally.

747.

The case that is most often stated in the context of the importance of independence on the part of experts is National Justice Compania Naviera SA v Prudential

Assurance Co Ltd [1993] 2 Lloyd’s Rep 68 (Comm Ct), but it is almost always referred to by the name of the ship involved in the case, The Ikarian Reefer, which ran aground off Sierra Leone and then caught fire, becoming a total loss. Lord Woolf MR, the author of the Civil Procedure Rules, made clear in Stevens v Gullis [2000] 1 All ER 527, 533 that CPR Part 35.1 and its accompanying practice direction “did no more than reflect the position as it had been well enunciated in the authorities prior to the CPR coming into force”. The Ikarian Reefer remains central to the way that experts should perform their role, including the separate duty owed to the court.

748. In Imperial Chemicals Ltd v Merit Merrell Technology Ltd [2018] EWHC 1577

(TCC), a more recent case in which a number of experts all called for one party were found to be partisan, at [237] the following was stated:

“The principles that govern expert evidence must be carefully adhered to, both by the experts themselves, and the legal advisers who instruct them. If experts are unaware of these principles, they must have them explained to them by their instructing solicitors. This applies regardless of the amounts at stake in any particular case, and is a foundation stone of expert evidence. There is a lengthy practice direction to CPR Part 35, Practice Direction 35. Every expert should read it. In order to emphasise this point to experts in future cases, the following points ought to be borne in mind. These do not dilute, or change, the approach in The Ikarian Reefer. They are examples of the application of those principles in practice.

1.

Experts of like discipline should have access to the same material. No party should provide its own independent expert with material which is not made available to his or her opposite number.

2.

Where there is an issue, or are issues, of fact which are relevant to the opinion of an independent expert on any particular matter upon which they will be giving their opinion, it is not the place of an independent expert to identify which version of the facts they prefer. That is a matter for the court.

3….. 4…..

5.

Where late material emerges close to a trial, and if any expert considers that is going to lead to further analysis, consideration or testing, notice of this should be given to that expert's opposite number as soon as possible. Save in exceptional circumstances where it is unavoidable, no expert should produce a further report actually during a trial that takes the opposing party completely by surprise.

6.

No expert should allow the necessary adherence to the principles in The Ikarian Reefer to be loosened.

It is to be hoped that expert evidence such as that called by ICI in this case, and also in Bank of Ireland v Watts Group plc, does not become part of a worrying trend in this respect. There are some jurisdictions where partisan expert evidence is the norm. For the avoidance of any doubt, this jurisdiction is not one of them. Not only experts, but the legal advisers who instruct them, should take very careful note of the principles which govern expert evidence.”

749.

The Ikarian Reefer, and the requirements of CPR Part 35, are the touchstones of expert evidence in this jurisdiction. It is of the utmost importance that they are adhered to both by experts themselves, and also by those who instruct such experts.

Mr Coyne

750.

Mr Coyne is a Partner of IT Group UK Ltd, and has 30 years of experience in IT. He has designed, programmed and supported Electronic Point of Sale (or EPOS) systems for use in payment handling, stock control and distribution in addition to full business accounting systems. He has also assisted public sector councils with revenue and benefits processing systems, investigated failures within stock market trading systems, European gambling and gaming networks and dealt with fraud in retail banking, point of sale, cash in transit and electronic funds transfer systems.

751.

He is a member of the Academy of Experts, the British Computer Society and the Society of Computers and Law. He has an Expert Witness Diploma from Cardiff University Law School, achieving this in 2008. He has been instructed as an expert witness in both civil and criminal cases over the last 19 years, including by both UK and International firms of solicitors, in connection with technology project fault analysis, software development and digital forensic investigations. He has appeared before different tribunals both in the UK as well as overseas. He served two detailed reports with numerous appendices. He also signed four Joint Statements with Dr Worden.

752.

Both the experts agreed in the 3rd Joint Statement that Horizon as it is now is relatively robust. The passages in that Joint Statement are that:

“in line with most other large-scale computer systems, Horizon's robustness has generally improved. From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events. This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.”

753.

I consider that this is an important passage of the experts’ agreement, and indeed robustness is an important aspect of the Horizon Issues. It makes clear that, in part, it is the discovery by the Post Office of bugs, errors and defects, the application of fixes (which means changes to the software by means of a software fix) and the improvement of monitoring that has led to the increase in robustness that has been achieved in the system as it is at 2019. This inevitably means that in earlier times, Horizon was less robust, or not as robust as it is now. This applies both to Legacy Horizon and also to Horizon Online. There are undoubtedly more modern or recent bugs in Horizon, as can be seen from the Technical Appendix. Drop and Go, for example, was not even a product until about 2013, and the relevant bug in the Bug Table which is bug 28 occurred in July 2017. That caused an impact to the branch account in question of £100; modest or tiny in comparison to the total Drop and Go business for that year, or even that month, but still £100 which appeared as a deficit in the branch accounts, and still £100 which the SPM had to deal with. Until it was discovered the bug could not have been fixed. It would not have been present prior to 2013 (because Drop and Go did not exist) and any fix cannot have occurred prior to July 2017, if not a month or two later. The agreement that Horizon is relatively robust as at 2018 and 2019 is something that is doubtless of great importance to the Post Office, as it affects how the Horizon system as it is at the date of this judgment is both used by the Post Office, and perceived by the SPMs who use it every day. The Horizon Issues however deal with Horizon from the year 2000 onwards.

754.

The other aspect of this part of the experts’ agreements that bears consideration is that the Post Office have to know about bugs, defects and errors in order that they can agree with Fujitsu that fixes have to be written and applied. This makes it particularly important, in my judgment, that Fujitsu were open and transparent with the Post Office about the presence of software bugs, errors and defects. Given Horizon was being used by the Post Office as the accounting procedure between it and its SPMs, there is every reason that Fujitsu would (or should) have to make the Post Office aware of bugs, errors and defects.

755.

A point that was put to Mr Coyne (it was taken from the Post Office’s pleadings, but it is a theme that runs through both its submissions and also the evidence of Dr Worden) is that there are both what are called technical controls, and also operational procedures and practices conducted both by the Post Office and by SPMs, to increase the reliability of data. The passage of the Defence that was put was as follows:

“Q. So robustness -- we can move on to paragraph 54 on page {C3/3/23}:

" ... in addition to the technical controls referred to above, there are several operational procedures and practices conducted by Post Office and subpostmasters that serve to increase the reliability of the data stored in the central data centre as an accurate record of the transactions entered on branch terminals.

Then there is a list of those. So you will see that what's being said is that the robustness of the system is said to be affected both by technical measures and by human measures. Do you see that?

A.

Yes, I can see that. That is what is said, yes.”

756.

In my judgment it is correct to describe the second part of these, as the Post Office did in its pleading, as “human measures”. They rely upon humans actually performing or being aware of certain occurrences. Two of these rely upon SPMs, namely those at paragraph 54(3) and 55(4) of the Defence. The second of these is that “the accounting and record-keeping obligations placed on Subpostmasters reduce the risk of any errors going undetected.” In my judgment, for SPMs to be an effective – or indeed any – part of a “human measure” and to assist or contribute to the process of the Post Office discovering bugs, errors and defects in live use (so that, where necessary, software fixes can be initiated, developed and applied) the Post Office ought sensibly to be transparent with SPMs too.

757.

It does however need to be recognised that human measures are not part of the Horizon system. They are not included in the definition of the Horizon system that forms part of the Horizon Issues themselves.

758.

It was also, in substance, put to Mr Coyne that Horizon Issue 1 should have been addressed in the way that Dr Worden had approached it, by effectively looking at actual (rather than potential) impact, although it was not put in exactly those terms.

“Q. Given it was admitted that there have been occasions when these things have happened, to answer the question: is it possible or likely that these things have happened by saying "yes" isn't a very helpful answer, is it?

A.

If the answer was just simply "yes" and with no further context then I agree that that would ...

Q. The critical issue that's raised by the four issues that I have read to you is the word "extent" and the word "likelihood" and the word "risk", isn't it?

A. They certainly are critical elements that would need to be reviewed, yes.

Q. What you were being asked to do by those issues is to give your opinion to the court on the extent to which it was and is likely that bugs caused shortfalls in any particular set of accounts?

A. Well, across the accounts of subpostmasters, yes.

Q. The extent to which various measures in place reduce the risk faced by any given subpostmaster when doing a transaction or when producing a particular set of accounts, yes? A. Yes.

Q. That's the critical question on which the court needs your assistance, would you agree?

A. It is one of the critical questions, yes.

Q. Could you identify a question which you say is more critical than that in the context of these proceedings?

A. I haven't taken a view on -- I have taken each of the Horizon Issues that's put to me with equal priority. I haven't set out to decide a prioritisation for them.” (emphasis added)

759.

This approach by the Post Office was, in my judgment, an attempt effectively to redraft Horizon Issue 1 in terms more favourable to its case. I have already identified above that the Horizon issues were drafted and agreed by the parties and approved by the court. Horizon Issue 1 plainly states “to what extent was it possible or likely for bugs, errors or defects of the nature alleged….to have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to” SPMs’ accounts. The approach taken by the Post Office (and also Dr Worden) was entirely to ignore the word “potential”.

760.

I also consider that it is a matter for the court to decide which issues are more important than others. An expert might decide a particular issue is important in that it represents a starting point for their opinion, which will then inform or guide their views on other issues. But it is not for an expert to decide one issue in a list is more important to the court than others, and concentrate upon that to the potential exclusion of consideration of some or any of the others. Mr Coyne was therefore correct to treat all of the Horizon Issues with equal priority.

761.

Mr Coyne had not performed a similar statistical analysis to Dr Worden seeking to demonstrate either the likelihood, or unlikelihood, of a bug impacting the claimants’ branch accounts. In my judgment such an exercise was not necessary in order properly to give expert evidence to the court on the Horizon Issues. Dr Worden’s evidence, and his alternative approach, was properly put to Mr Coyne. Mr Coyne accepted that he could have approached his evidence in that way, but explained why he had not. He described the approach adopted by Dr Worden as “a fundamentally flawed approach”.

“Q: You could, using your skills as an IT risk analyst, you could have, by reference to the KELs and the PEAKs that you had identified, formed an assessment as to the likely financial impact in each of the instances that you had identified?

A.

Yes, I could. If I could please answer that by way of illustration to the problem with that approach. Q. Please do.

A. We will often see a bug, error or defect with a very wide range of impacts and the impact is typically whatever the counter was doing at that point in time. So if the counter was doing a foreign currency transaction for just £50 and something goes wrong, the discrepancy may be £50.

By knowing that there is a bug, error or defect in the Horizon system that leads to problems with foreign currencies, you can't then say it is only a £50 defect because that isn't incorrect. If another person was to be subject to that defect and they were doing a £10,000 foreign currency transaction, they would likely have that same level of defect.

Now that's an illustration to say you can't value bugs and their potential impact by looking at what has happened historically. You can't value it in that way.

Q. But what you can do is form an estimate having regard to the totality of the PEAKs that you have seen, can't you?

A. But it is a fundamentally flawed approach. If you have seen three branches that have had an impact because they were doing three foreign currency transactions, a £20, a £50 and a £100, but then there is a fourth person that believes that they have been subjected to that but that isn't recorded, you can't simply look at the three where it has been recorded and say the fourth couldn't possibly have occurred because we know there is a defect there.

Q. I'm not suggesting that you could arrive at a certain conclusion of an absolute cast iron number, but I repeat my question. Can't you form an estimate having regard to the totality of the PEAKs that you have seen?

A. Yes, but your estimate would have to be based on the three people where it has been recorded to have occurred, so you would say £20, £30 and £50, and the best you could possibly do is come up with an average of that, and you would say that the value of that defect is whatever that is.

Q. There is another way that you could do it, isn't there, which is that you could look at the three bugs, the receipts and payments mismatch, the suspense account bug and Callendar Square, the ones that have been thoroughly investigated, and you could form inferences from the scale of those bugs, yes? Would that be reasonable? A. For those types of bugs, potentially yes.

Q. Those are quite large bugs, aren't they? They are not small bugs in the scheme of things. That's why they were identified in the letter, because these were major bugs of which even Post Office was aware?

A. Yes. And carrying on from the one we were not told about until later, the Dalmellington one, there were some which were only pounds, just a few pounds, and I think there was at least one that was £25,000.

Q. But just fixing our attention for the moment on those three bugs, would you accept that the evidence indicates that altogether the total financial impact of those three bugs is no greater than £100,000. Do you agree with that?

A. No, I haven't done that so I would not know whether that was right or close to right”.

762.

The documentary evidence showed that Dalmellington had over 100 branches with some being impacted more than once. Using the Post Office’s definition of “lasting impact”, which means the branch accounts were given TCs to correct the effect that the bug had upon them, 114 of these impacts were made good. Of the other four, two turned out not be the Dalmellington bug and the other two had only very small financial discrepancies.

763.

However, this bug, which had taken 5 years to be discovered, was put to Mr Coyne as a good example of how countermeasures had to be considered.

“Q. In fact, Dalmellington is quite a good example of how countermeasures need to be brought into account and how it is important to look at financial impact to make an assessment of the extent question that's raised in Horizon Issue 1, isn't it? Because it is no good saying 118 branches were affected and some of them were affected by large amounts, when in fact we all know that even before the bug was detected the branches involved were in fact made whole anyway, yes?

A.

It does suggest that, but it also suggests that there were deficiencies with the countermeasures because I believe it was five years after the first occurrence of the bug that the defect was finally discovered.

Q. Because, as you have already agreed, to an outside observer it looks exactly like a human error because it was a human error. The bug wasn't causing losses, it was causing errors to be made and those errors were picked up, would you agree with that?

A. No, it wasn't a human error, it was a defect of the system. It made it look like a human error.”

(emphasis added)

764.

The Dalmellington bug was not accepted as a bug in the Bug Table at the time of the 3rd Joint Statement where it is Bug 4, but is now accepted as a bug by the Post Office but said to have “transient impact”. I find that it is a bug, and it cannot properly be described – as the Post Office did during the above cross examination– as human error. As Mr Coyne pointed out in his answers, it was a defect of the system. It also did take 5 years for it to be discovered. One of the branches affected by it had its branch accounts affected to the sum of approximately £25,000. The fact that a TC was issued does not mean that it did not have an impact upon branch accounts. That impact may have been later corrected by means of the issuing of a TC, but that cannot mask the fact that the branch account was affected by that amount for the period of time until that correction was made.

765.

Mr Coyne accepted that Dr Worden’s arithmetic was correct, and also accepted that the sum Dr Worden had calculated in respect of Dalmellington (£100,000) was “quite small compared to the £19 million that’s claimed by the claimants in this case. It is less than 1%”. As Mr Coyne accepted, on the “pure numbers”, that was correct.

766.

Mr Coyne did not accept Dr Worden’s basic logic as a starting point, nor did he accept the logic that there was “no evidence that” the claimants’ branches were “more or less likely to be hit by bugs than your average Post Office branch”. I deal with that point in further detail below at [821] and [822] below, but this amounts to an assumption by Dr Worden that a group of SPMs who specifically allege they have experienced the effects of bugs are to be treated, in statistical terms, as though they are a random group of SPMs of the same sample size drawn from the wider population of all SPMs. They plainly are not a randomly drawn sample of nearly 600 SPMs. They are a very specific group (or sample) of those who say their branch accounts have been impacted by, or have experienced, such incidents. In statistical terms, the correct term for the group is that they have a bias – they all allege that they have experienced the effects of bugs, errors and defects.

767.

An illuminating series of questions and answers highlighted the entirely different approach of the two experts.

“Q. Are you aware of some special factor applying to the claimant branches which marks them out as very different from the rest of the branch network making their susceptibility to bugs very different from the wide range of branches in the rest of the network?

A.

I'm not aware of it, no, but it would require -- in order to look at that it would require a detailed understanding of the particular processes of the branches. And I think that's illustrated by what we know of Dalmellington, that impacted -- I think one branch was impacted five times, where there was only a handful of branches -- I

think, was it 88 that was impacted in total?

Q. Mr Coyne, it is quite important not to get confused between particular instances of things happening and the law of averages. What I'm asking you about is what's likely to happen over a large number of cases, do you understand the difference? So of course if people are all walking down the streets, some of them are going to get hit by lightning. Everybody won't get hit by lightning in the same way. It will be a very small number of people who will be hit by lightning and they'll usually be in particular situations when it happens. But one can make a generalisation about the likelihood of people being hit by lightning, do you understand, because of the law of averages?

A. I do understand, but in that illustration it would be very unusual for one person to be hit by lightning five times.”

768.

This then continued with some discussion of instances where some people may have been hit by lightning more than once.

769.

There are some important points that arise out of this approach. Firstly, the terms of the contracts between the Post Office and an SPM does not invoke the law of averages, either as it was explained in this cross-examination, or at all. The terms of the contracts, whether SPMC or NTC, have a very precise accounting process which is run by Horizon. Secondly, the existence of a bug such as Dalmellington, which was present in the Legacy Horizon system for 5 years prior to discovery, is of direct relevance to the Horizon Issues. It does not support the proposition that a SPM with an outreach branch in the 5 years before it was discovered was more likely (or equally likely) to be hit by lightning as having their branch accounts impacted by the Dalmellington bug. Thirdly, even if the “law of averages” approach were the correct one to adopt in terms of individual SPM complaints that their branch accounts had been impacted by a bug, error or defect in Horizon (and to be clear, I find that it is not) it is an exercise that could only sensibly even be attempted by firstly using the total number of bugs present in the system (which would require at least this judgment, if not more investigation too following the judgment), a definitive total for the number of instances of each bug occurring (again, which would require another judgment), a definitive total for the full amount of losses caused by each incident of each bug (information again not yet available, if ever) and some sort of probability calculation taking account of all those matters. That calculation would only, in any event, lead to a figure for the probability chances of any SPM in the period having their branch accounts impacted by a bug. So many of the essential ingredients or pieces of data for such a calculation are not known, that to try to attempt it in the way put by the Post Office, is to start from the end and work backwards, and work backwards in a way wholly lacking in mathematical precision. The “law of averages” approach cannot be used, in my judgment, in the way contended for by the Post Office.

770.

So far as Dr Worden’s “scaling factor” is concerned, and his approach to the claimants’ branches, Mr Coyne was entirely reasonable, in my judgment, in how he would consider hypothetical situations put to him and also explain why they were not valid.

“Q. At this point in the equation what Dr Worden is trying to do is to work out how likely it is branches are going to be affected? A. Yes.

Q. He reasons that branches with more transactions, doing more transactions, are statistically over a large number of occasions more likely to be hit by a bug than a branch doing fewer transactions and would you agree the principle underlying that observation?

A.

I think that's probably a reasonable principle. If we look at the defects we found though, there is probably other factors that could be brought into that.

Q. But you are not aware of any, you are not in a position to suggest a single factor which you have any evidence to think actually applies in relation to the claimants as compared with the rest of the branch network?

A. I don't know the make up of the claimant, but one example might be, do they have an outreach branch?

Q. You are aware, aren't you, how small the number of outreach branches there is, how insignificant that is in the context of the Post Office network, aren't you? A. Well, it wouldn't be insignificant to the people that are impacted by the defect and I don't know whether any of the -- because I haven't studied the claimants, I don't know whether any of the claimants have that or not.

Q. What you are doing is you are speculating that some of the claimants might have outreach branches?

A. I'm not. I'm illustrating the potential problems with the approach that you are taking just simply using the unit of number of transactions.”

(emphasis added)

771.

Similar points can be made, in my judgment, to each of the bugs in the Bug Table, and not simply Dalmellington and outreach branches. Some affect foreign currency transactions. Bug 9 in the Bug Table, Reversals, only affected the value of transactions that were sought to be reversed by a SPM, and were impacted by the bug actually introduced into Horizon by version S30 which was released by Fujitsu. That this occurred at all was due to a failure in regression testing by Fujitsu prior to release of S30. This would, if the bug hit that branch, show the value of a reversal twice in branch accounts. The impact on the branch accounts would be the value of the reversal; if it were a large figure, the impact would be large, and vice versa. Something of that nature cannot sensibly be dealt with by the law of averages. Concurrent log ins, in the early days of Legacy Horizon, and numbered bug 18 in the Bug Table, would only occur if a user was logged in at the same time to two terminals. Again, that cannot sensibly be dealt with by use of the law of averages.

772.

As Mr Coyne put it when asked about Dalmellington above and reminded that there are not many outreach branches in the Post Office network and how insignificant this was, “well, it wouldn't be insignificant to the people that are impacted by the defect”. This answer, which in my judgment is an entirely accurate one in the context of this group litigation, is one that could apply to each of the instances or incidents of each of the bugs, errors and defects impacting branch accounts that have now come to light in the Horizon Issues trial. Individually and when considered across the whole of the 3 million sets of branch accounts in the period from 2000 onwards, the amounts are individually small amounts (although sometimes they are several thousand pounds). The same point can be reached by simply taking the sum at stake in the group litigation in terms of accounting losses for all the claimants (as opposed to other heads of claim not quantified) which is £18.7 million, and comparing it to the total amount of business transacted in the Post Office branch network as a whole across the entire UK from the year 2000 to date. No one has calculated that latter figure so far as I know (and if they have, they have not brought it to my attention) but it will be an enormous sum. The fact that £18.7 million is but a small proportion of that enormous

figure for total transacted business over 18 years is of no assistance whatsoever, and neither is the law of averages approach.

773.

Finally on this, in one of the answers to Mr Coyne’s requests for information (on another point, discrepancies from external data sources), the solicitors acting for the Post Office pointed out in the document dated 4 June 2018, the number of transactions that Horizon deals with each day. This was done in order to give a sense of scale to the request posed by Mr Coyne. The exact answer provided was “To give this a sense of scale, it is noted that Horizon processes around 6,000,000 transactions per day.” The reason for identifying this is that this figure for transactions per day, together with the agreed figure for the number of monthly branch accounts over the period agreed by the experts, namely 3 million, gives a sense of scale of the Post Office’s business conducted by Horizon and the total number of transactions over the period. Given there are fewer than 600 claimants, the number of impacts on branch accounts complained of by the claimants is a very small fraction of this (whatever its precise total). The law of averages approach does not assist the Post Office.

774.

So far as countermeasures are concerned, Mr Coyne stated that “typically if you were going to design a countermeasure you would want to have it finding the defect straightaway. If it was only going to find it sometime later it would be of limited value.” Not only do I agree with that in general IT terms, but also in terms of the Horizon system and its accounting function between the Post Office and each SPM’s branch trading period specifically. A “countermeasure” in this instance would be something that would correct an impact to the branch accounts. If that is only going to occur, as Mr Coyne put it “sometime later” then it would only be of limited utility, if any, depending upon the period of time this takes. Further, the Post Office used the term “countermeasure” as including “human measures and controls”, as made clear to Mr Coyne during his cross-examination:

“I'm using it as a term to refer to measures and controls in existence, both technical and human measures and controls, which are designed or have the effect of identifying bugs in Horizon or identifying the consequences of bugs in Horizon so that they can be looked at and fixed.”

775.

This means therefore that the Post Office was relying upon matters outside the definition of the Horizon System at the beginning of the Horizon Issues, which stated “ “The Horizon System” shall for the purposes of this list of issues mean the Horizon computer system hardware and software, communications equipment in branch and central data centres where records of transactions made in branch were processed, as defined in GPOC, at §16 and as admitted by Post Office in its Defence at §37.”

776.

As Mr Coyne explained, countermeasures in the system are addressed at the design stage:

“Typically countermeasures are a design aspiration. When you sit down to design a computer system and what controls are in place, then you would consider what controls are required, where to position them. Should it be the case that the controls fail do you have a human aspect doing a secondary check? It could go all the way to employing auditors to re-check figures. You could go on ad infinitum with –

Q. So you accept, then, that these controls or countermeasures, whatever you would like to call them, some of them have the effect of stopping the bug in its track. Well, some of them have the effect of stopping a bug arising in the first place?

A.

The best way of dealing with defects is to ensure that they don't arise in the first place ……..so what you would do is throughout your testing regimes you would flush out all of the defects at that point in time, and you would put in controls to ensure that if a defect does trigger, there is a way that you are alerted to it and it is fixed as soon as possible.

Q. Yes. So one of the objectives is to have a testing regime which picks up as many bugs as possible? A. Indeed.

Q. But I think you accept that it is impossible to have a testing regime that picks up all of them?

A. It is impossible to have a testing regime that picks up all of them and that is why you have a process of categorisation and you would allow a product to go live if -- one of the terms that's used -- all severity 1 and severity 2 defects have been dealt with. Severity 3 might be largely cosmetic and you would allow a system to go live with cosmetic defects.

Q. That's not quite what I'm asking you. You accept there are going to be bugs which arise in the operation of any IT system which would simply evade any test that any human being can devise?

A. There's always going to be bugs that are unknown to be within the system at the point that you go live and they are only discovered sometimes weeks afterwards, sometimes years afterwards.”

777.

Both defensive programming and impact management are part of, or are descriptive terms concerning, countermeasures. In my judgment they are both more to be described as “technical controls” (to use the language of the Post Office’s pleading) rather than “human measures”. These matters were all effectively part of the 3rd Joint Statement. So far as countermeasures within the Horizon System itself are concerned, these plainly affect the robustness of the system. The 3rd Joint Statement agreed that (using different numbering):

1.

The effectiveness of various countermeasures changed throughout the life of Horizon;

2.

Countermeasures are basic elements of practical IT system design;

3.

Countermeasures work by limiting the impact of Horizon bugs/error and defects on branch accounts. Countermeasures do not always eliminate the effects of adverse events (they are not perfect) but they are often effective in the area where they are deployed; that is why they have become basic elements of practical IT systems design;

4.

It is difficult to measure the extent of the robustness of Horizon, apart from how it might limit the extent of impact on branch accounts, as in Issue 1.

(emphasis added)

This shows that IT countermeasures are designed into systems. That is entirely to be expected. However, it also means that the robustness of a system cannot be measured by other measures (such as TCs, or a human complaints department as another example) taken to correct the effect of the IT system. A ready way of illustrating this is as follows. Consider a hypothetical IT accounting system that has 10 countermeasures designed into it, as a matter of IT design. It also has a human operated complaints department. All 10 of the countermeasures fail so a user has to contact the complaints department, which upholds the complaint and issues a correction, that undoes the impact within the accounts the IT system has allowed to occur. The efficiency of that department, and the accuracy with which it deals with such matters, cannot be used to assess in IT terms the robustness of the IT system itself. It is separate.

778.

The Post Office also has an inconsistent approach to TCs. On the one hand they are relied upon (through Dr Worden) as being an important countermeasure – even though they are not part of the Horizon system as defined – yet on the other hand questions about them were not dealt with as though they were relevant to the Horizon Issues. For example, one of the requests raised by Mr Coyne in respect of a particular PEAK was as follows:

“Regarding POL-0032919.pdf, the GoldenGate replication between Oracle 10g and 11g being aborted and resulting in a number of branches reporting cash declaration and stock reporting discrepancies, were any transaction corrections sent to the 247 affected branches as a result of the discrepancies and which branches were affected by the incident?”

779.

The answer that was given by the Post Office’s solicitors was as follows:

“The relevance of this request is not understood. Transaction Corrections are issued by Post Office to correct transient discrepancies in branch accounts in order to restore the correct position.”

780.

This answer to a request by the claimants’ IT expert therefore – correctly in my judgment – identifies that TCs are not part of the Horizon system. That is entirely at odds with them being part of a countermeasure within the Horizon system. This is effectively accepted in the Post Office’s opening submissions which expressly state at paragraph 114 that the “Post Office’s reconciliation processes are outside the scope of the Horizon Issues”. TCs are part of the Post Office’s reconciliation processes, and I find that they are outside the scope of the Horizon Issues. Dr Worden therefore cannot sensibly take them into account as a Horizon countermeasure. The Post Office cannot have it both ways.

781.

One of the matters that Mr Coyne relied upon was what he described as “the sheer volume” of KELs and reconciliation reports (a statement, it should be noted, made prior to discovery of the 5,000 KELs disclosed by the Post Office in October 2019). His evidence was that this confirmed “the wide-ranging extent of the impact of such bugs/errors/defects. This evidence demonstrates that such bugs/errors/defects would undermine the reliability of the Horizon system to accurately process and record transactions”. He accepted that there was the need for reconciliation reports, and that this was part of the way that a system would check whether (say) the banking records at the Post Office transactions matched the corresponding records at (say) the Bank of Ireland for the same transactions. In my judgment KELs are indeed part of the records (or contemporaneous records) that can be considered in order to arrive at an overall conclusion for the existence and potential impact of bugs errors and defects, but the

PEAKs need to be consulted too. This is because the PEAKs are the documents that often have greater information within them, added to which one single KEL could relate to a couple of short PEAKs, or alternatively to many a great many far lengthier PEAKs, and it is only from the PEAKs themselves that one can gain a better view of the impact of that particular bug. Indeed, some of the KELs simply have a start and end date, with very little information on the number of impacts during the intervening period. It is clear to me that if there is more than one PEAK referring to the same KEL, one would only realise that there had been a re-occurrence of the same defect if one considered the PEAKs; the KEL would not show this. Simply looking at the KEL could give the erroneous impression that such a defect had only occurred once. It is also telling, in my judgment, that some PEAKs that are clearly related to errors, for example the PEAK associated with the issue that the subject that “SSC Database users do not have correct permissions”, namely PC0208119 dated 10 March 2012, expressly states in one of the 25 October 2011 entries the following “Have relevant KELs been created or updated? No. None believed to be required”. This clearly demonstrates that Fujitsu would not always even create a KEL for known errors (as SSC users having the incorrect permissions, all having APPSUP when they were not supposed to have this, is clearly a known error). SSC users were (as set out in the PEAK) only supposed to be given “the optional role 'APPSUP' temporarily (by Security Ops authorisation) if required to make emergency amendments in BRDB Live”. I find that the KELs alone do not constitute a record of all bugs, errors and defects.

782.

Reconciliation reports are only generated when the information the Post Office has about a transaction or series of transactions is different from the information held by the client. These are called exceptions. Mr Coyne was asked why he considered the volume of these in the history of Horizon was confirmation of his conclusion relating to the wide ranging extent of bugs, errors and defects having impact on branch accounts.

“Q. But I don't understand, Mr Coyne, why you think that the existence of reconciliation reports confirms the wide-ranging extent of the impact of bugs in Horizon which are having an impact on branch accounts?

A.

I have worked and designed banking systems, stock broking systems. I have never seen the need for tens of thousands of transactions per week to have a human intervention. That suggests that something is going wrong. It is working outside of process on a larger scale than I would have expected.”

(emphasis added)

783.

The origin of the number of reconciliation reports is as follows. Mr Coyne had asked, in requests for information prior to the trial, what was the purpose of setting an NB102 exception to F99 by Fujitsu. The Post Office in the answer explained that this explanation would have to come from Fujitsu, and the explanation was given:

“When there is an incident involving a reconciliation exception in Network Banking which has been fully processed, then the transaction needs to be set to F99 to indicate that processing is complete.

Therefore, this is done for any transaction that appears in a reconciliation report, once the resolution is complete.

The reconciliation process document is SVM/SDM/PRO/0012 RECONCILIATION AND INCIDENT MANAGEMENT PROCESS and describes how reconciliation is carried out.

SVM/SDM/SD/0020 END TO END RECONCILIATION REPORTING is the specifications of the reports that are produced for the reconciliation service.

The CCD that describes the contracted service:

SVM/SDM/SD/0015 RECONCILIATION SERVICE: SERVICE DESCRIPTION.”

784.

The next request was “How often has [setting an NB102 exception to F99] occurred?”

785.

Initially the Post Office would not answer this, stating inter alia that it was very unlikely that the information would have been pooled or collated. When Mr Coyne persisted, and pointed out it would be very different if the answer was “it was an isolated incident in 2003” or “it was 10,000 transactions each day for the last ten years”, an answer was provided by the Post Office through its solicitors. I shall reproduce it in full:

“Fujitsu currently "F99" 10,000+ transactions per week across all NB102 associated reports (DCP and NBS)”.

786.

This was correctly understood by Mr de Garr Robinson for the Post Office who (understandably) corrected Mr Coyne when he initially said he thought it was hundreds of thousands per week.

“…I asked for the number of reconciliation problems that required manual intervention, and the answer I was given was that it was hundreds of thousands per week, I think was the –

Q. 10,000 a week that have to be F99ed, do you remember that?

A. Yes.”

787.

So far as the next request, which was associated with this, is concerned, that paints the same picture of the Horizon system in my judgment. The request was:

“What is the cause of an ‘Uncleared Transaction Corruptions’ and how often do

[Uncleared Transaction Corruptions] occur?”

The answer was, again from Fujitsu:

“The DCP report NB102 section 3 (Uncleared Corruptions) usually contains 100s if not 1000s of transactions per week (Kiosk transactions only).

This is where there is a discrepancy on banking and payment card transactions. To determine the exact cause of any individual Uncleared Transaction Corruption, that particular occurrence would have to be analysed. However, the most usual cause is for a banking request or authorisation failing to reach its destination, which is something that can have a variety of causes. An example may be that a bank may authorise a transaction, but that confirmation is not delivered to the counter so the SPM believes that the transaction in fact failed due to some issue.”

788.

This is a great number of transactions per week that require manual intervention – over 10,000, on the numbers provided by Fujitsu. I consider that Mr Coyne is correct when he says “this suggests that something is going wrong”. I do not accept that on a properly functioning and robust system there should be such a high number as that every week. I accept Mr Coyne’s evidence in his answer at [782] above.

789.

Mr Coyne also considered, in respect of the control environment, that the Ernst & Young management letter of 2011 had recommended changes to strengthen the change management process. E&Y had noted that the Post Office was not usually involved in testing fixes or maintenance changes to the in-scope applications (which includes Horizon Online) and they were unable to identify an internal control with the third-party service provider to authorise fixes and maintenance changes prior to development for applications. The risks were outlined by E&Y in the following terms:

“There is an increased risk that unauthorised and inappropriate changes are deployed if they are not adequately authorised, tested and approved prior to migration to the production environment”

Mr Coyne also drew attention to a PEAK some years earlier than this (in 2003) that provided evidence that Fujitsu did not even always follow the correct processes that were in place in any event. In this instance a branch terminal was replaced, which resulted in a number of bills which branch customers had paid (in terms of making payment at the branch Post Office) not in fact being paid (to the ultimate payee, the Post Office clients that had issued the bills). The PEAK stated that “something went wrong and the marooned transaction did not find their way back to the office”, something which had occurred because the mirror disc in the branch terminal was replaced (when it should not have been). The PEAK also stated:

“I believe that we shall have to confess to POCL that we have not followed the correct procedure and we should advise that POCL make manual payments…..”

POCL means the Post Office, under its earlier name of Post Office Counters Ltd.

790.

Mr Coyne was of the opinion that the specific recommendations in the 2011 E&Y letter directly impacted upon Dr Worden’s view of countermeasures. He also drew attention to the fact that the non-acceptance of these recommendations by the Post Office was an opportunity to impose or improve countermeasures in Horizon which the Post Office chose not to take. The four areas, and the countermeasures they would impact, were:

“1. Improvements to outsourcing application management (Quality and Change

Control; managing non-functional requirements; testing good practice);

2.

Improvement to segregation of duties within the “manage change” process (Quality and Change Control, Security);

3.

Strengthen the change management process (Quality and Change Control);

4.

Strengthen the review of privileged access (Security).”

791.

Mr Coyne also relied upon the fact that although both Legacy Horizon and Horizon Online did contain a number of measures and controls, these mechanisms had been shown to have failed, and this was agreed by the experts. The minutes of the Post Office’s own Risk Compliance Committee meeting of 18 September 2013 showed that not all of the risks identified in the then-recent E&Y audit had been addressed. All four pages of these minutes were heavily redacted, but one surviving entry shows that risk not addressed was “relating to the communication by Fujitsu of changes made to the Horizon system, was still outstanding. It was identified that it would cost over a £1m to implement the mitigation being suggested by the audit and that this was not proportionate to the risk being managed.” Mr Coyne accepted in his crossexamination that this did not relate to a bug being fixed, and I accept that evidence. In my judgment it does go to the robustness of Horizon.

792.

There was one area which was concentrated upon towards the end of his evidence, and this was an entry in the 2nd Joint Statement. This was dated 25 February 2019, and under Horizon Issue 1, under the topic “the extent bugs were possible or likely” the experts had agreed the following entries relevant to this:

“1.1 We agree that bugs were possible, and the table above displays a number which in the opinion of either or both experts appeared to have impacted branch accounts.

1.2 Referring to the table of bugs above, the experts agree that the bugs in rows 1, 2, 3, 10, 13, 14, 18, 23, 24, 25, 27 and 28 may have had financial impact on branch accounts. Other rows, the impact is not agreed between the experts.”

And later on in the 2nd Joint Statement:

“1.15 The number of distinct bugs, for which the experts have seen strong evidence of the bug causing a lasting discrepancy in branch accounts, is between 12 and 29.”

793.

Just before the short adjournment on the final day of his cross-examination, Mr Coyne was asked about the figures in that range. He was also asked to “forget Horizon Issue 4”. Given that Horizon Issue 4 asked “to what extent has there been potential for errors in data recorded within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data in Horizon?”, being asked to ignore it somewhat narrowed down the utility of the exercise, but it is without doubt the case that Mr Coyne, when asked for a number for the upper range in the Joint Statement, stated it should not be 29, it should be a different figure. He initially stated that it should be 13. The court then rose for the short adjournment. As soon as court resumed one hour later, he immediately sought to correct the number 13 that he had given. He said the number should be 21. A complication in this cross-examination exercise was that initially Mr Coyne was using what he called his bug table in his 2nd report (which had been dated 1 February 2019, and amended on 6 March 2019) and the cross-examination was being done by reference to something also referred to throughout the trial as “the Bug Table”, which was the one in the 2nd Joint Statement. The former in the report had a column headed “evidence of branch impact” with the entry “yes” or “no”; the entry in

the Joint Statement about which Mr Coyne was being asked was item 1.15 and is as set out above. The questioning was done solely by reference to Horizon Issue 1. Table 1 in Mr Coyne’s 2nd report made it clear that Phantom Transactions, Reconciliation Issues, Concurrent log-ins, Post & Go/TA discrepancies in POLSAP, Recovery Failures and Transaction Correction Issues were all listed under Horizon Issue 4, and not Horizon Issue 1. An associated difficulty was the distinction that was being made by the Post Office between lasting and transient effect.

794.

The correct figure being supported by Mr Coyne in his evidence was explored extensively in cross-examination. Mr Coyne did explain this as follows (in part only):

“Q. Are you suggesting that you have only ever suggested 21 and no one was suggesting anything above 21?

A. No. There are 29 bugs, errors and defects in the list and in the table starting at number 1, but some of those go to other Horizon Issues apart from Horizon Issue 1, and they do say that clearly in the heading.”

795.

There is no doubt that Mr Coyne made matters a little hard for himself during this passage of questioning, and the Post Office seized upon his change of number (which was, after all, an agreed number in an agreed statement by both experts) and made much of it:

“Q. Mr Coyne, everybody on this side of the court, and I mean everybody on this side of the court, believed that your view was that the evidence showed that there were 29 bugs with lasting financial impact. You are now saying that we made a mistake, are you?

A. Yes.

Q. Could I ask you to explain where you made it clear what you were actually saying?

A. Well, as set out in my second report, there is a table there that quite clearly says whether there is evidence of branch impact or not. And there is also a list of which Horizon Issue it specifically relates to. The only addition to that is when you come to the very last column, there's other bugs, errors and defects that are re-introduced and they are only covered in the joint statements.”

796.

Mr Coyne also said:

“A. Throughout the report I have set out next to every single defect whether it is relating to the test of Horizon Issue 1 or a later Horizon Issue, and I have also set out whether I believe there was impact on branch accounts all the way through report number 2.

Q. Let's not talk about report number 2 because the statement -- the most up to date statement of the position, I think you agreed with me at the beginning of today, the culmination of all your work, including work you did after your second report, is the bug table in joint statement 2?

A. Yes, but when you put that to me I said that that can't be read in isolation, it must be read with the second report.”

797.

So far as the number is concerned, Mr Coyne settled on 22 different bugs, his explanation of his earlier different numbers was as follows:

“Q. And I would like to ask you why is it you thought there were 21 bugs at 2 o'clock, whereas you thought -- was it 14 bugs at 1 o'clock? 13 bugs at 1 o'clock.

A. Because I was just counting down the table at number 1 in my report. In the pressure of the situation I just tallied that column down, I didn't have enough time to consider it fully.

Q. To go back to a question I asked you earlier, where do you make this clear in the joint statement?

A. In the heading next to the -- in the bug, error or defect.”

798.

I accept that explanation. Although Mr Coyne was criticised for certain entries in the 2nd Joint Statement, including the headings, and it was put that the “joint statement was seriously misleading” I do not consider this a valid criticism of him. It was a statement the text of which had been agreed by both experts, and it appeared in an experts’ agreed joint statement. There is no doubt that the upper figure of 29 would most likely have come from Mr Coyne, with the lower figure of 12 having come from Dr Worden (and the claimants’ legal team interpreted the figure of 29 in the same way as the Post Office, as it appeared in their written opening) but there is a limit to how much can be put on this point by the Post Office. Joint Statements are (or should be) reached by the parties’ independent experts between themselves.

799.

It is the experts’ responsibility to agree the content of their Joint Statements, and this is part of their duty to the court as independent experts. It is the basis on which the court gives permission for expert evidence. No authority is required for these propositions as this is wholly conventional, but to the extent that authority may assist,

Males J (as he then was) fully explained this in [3] of Mayr and others v CMS Cameron McKenna Nabarro Olswang LLP [2018] EWHC 3669 (Comm). The agreed text of the agreed parts of a joint statement therefore cannot be approached as though it were a unilateral passage in only one of the expert’s reports. It is an agreed passage. I do not consider that Mr Coyne’s changes of the number he was asked about demonstrates any unreliability on his part, or discredits him or his evidence.

800.

I consider Mr Coyne to have been a helpful and constructive witness, and I find the suggestions made to him that he was biased to the claimants and not independent are criticisms that are not justified. He, and his small number of assistants, had done a great amount of investigation into the very numerous PEAKs, and the smaller number of KELs, and he had embarked upon a careful and sensible exercise necessary for him to reach conclusions on the Horizon Issues, as drafted and agreed by the parties and as approved by the court.

801.

There were some fundamental methodological differences between him and Dr Worden, but the agreements that the two of them reached were detailed notwithstanding this. I found the contents of all of those agreements of great assistance.

Dr Worden

802.

Dr Worden was educated at the Université de Grenoble, Cambridge University and the California Institute of Technology, better known as Cal Tech. He has a Ph.D in Theoretical Particle Physics. He is currently a consultant with a practice called Charteris Associates, having started there in 1997, but he has been a consultant since August 2010. Starting with the first entry on his CV in terms of his work experience, between 1975 and 1986 he worked on the development of knowledge-based systems and developed Logica’s proprietary relational database management system, and between 1986 and 1990 he was the Managing Director of Logical Cambridge. This is an advanced technology subsidiary of Logica. Between 1991 and 1995 he acted as an independent reviewer of many of Logica’s largest projects, the key projects that constituted a large part of Logica’s commercial risks, and also a large part of its commercial reputation. Between 1995 and 1997 he was a Business and Technical Consultant in Logica, moving to Charteris in July 1997. He has given evidence in the High Court before on several occasions, including in what was at that time the biggest IT dispute tried, namely BSkyB Ltd v HP Enterprise Services UK Ltd (formerly EDS

Ltd) [2010] EWHC 86 (TCC). This case is notable for a range of reasons, not least that the judgment (excluding quantum) runs to some 2350 paragraphs; and also that the educational qualifications considered in the judgment included those of one main factual witness who turned out to have fabricated the establishment from where he claimed to have been awarded a MBA. This was a point demonstrated by counsel enrolling his own dog (“Lulu”) in the same establishment, Lulu gaining an MBA with higher marks than the witness. The hearing ran for 109 hearing days and Dr Worden was cross-examined in that case for more than 35 hours, which is a very considerable period of time for anyone to be in the witness box. In his CV he identifies that the majority of his opinions referred to in the judgment of Ramsey J were accepted by him. At [277] in that judgment he was described as “more of a generalist in the field of IT with some experience dealing with CRM systems but with good general knowledge of the estimating, systems integration and the types of problems which can arise on projects such as” the project in that case.

803.

He is a highly qualified person, both in IT generally and specifically, and in statistics. He has earlier experience with relational database management systems when he was at Logica in the late 1970s and early 1980s, and from the dates it is possible to tell that he has been involved in the IT field for many decades, and (in comparison with many IT personnel today) from its infancy. He has also, whilst at Charteris, developed and deployed software for data integration and migration. He described himself as having done databases “since the year dot” at one point. He describes himself in his CV as having strong practical experience of information technology and its business applications. He has several years’ line management experience, has successfully managed large IT projects, and acts as a trouble-shooter of major integration projects and consultant in information strategy. He has particular expertise in XML, in largescale information architectures and tools, data integration and data migration, scientific data processing, and in system performance. He explained that as he is trained as a scientist and engineer, he has used mathematics all his life and part of that mathematics is probability theory and statistics. Given the subject of his Ph.D, it will be no surprise that he uses mathematics extensively. I doubt that there are any theoretical particle physicists who are not extremely skilled and expert at mathematics at a very high level indeed. He therefore has great experience in statistics, and at one point in his cross-examination was described as a statistician, something with which he did not disagree. He does not however, as he put it, have any formal qualification in statistics.

804.

He is also a highly experienced expert witness, as demonstrated not simply from the BSkyB case I have referred to above but many others, and he has given expert evidence in different cases in the Technology and Construction Court, Chancery Division and Queen’s Bench Division too. He served two detailed reports with numerous appendices (adding schedules of corrections in the usual way in supplementary evidence in chief), and he also took part in the joint experts’ exercises and signed the four Joint Statements with Mr Coyne. He also served a third report directly upon the court, but no application to adduce this as part of his evidence in chief was made by the Post Office and it did not form part of his evidence.

805.

In the instant case, he was cross-examined for three days. One of his corrections in chief was in respect of paragraph 761 of his 1st report where he had stated the following in respect of his summary of what was called a statistical analysis of the likelihood of the bugs that had been found by the experts being the cause of the claimants’ shortfalls and discrepancies.

“759. In section 8.5, 'Scaling of Financial Impact of Bugs', I gave what I think is the simplest analysis of why, in quantitative terms, bugs in Horizon cannot have accounted for a large part of the Claimants' shortfalls. I summarise it here.

760.

Because Post Office has had an average of 13,560 branches over the lifetime of Horizon, the total number of monthly branch accounts has been about 3 million.

761.

Therefore, if a bug like the Suspense Account bug has occurred 16 times in the lifetime of Horizon, the chance of it having occurred in any given branch in any given month is about 16 in 3 million. Because the Claimants tended to have smaller branches than the average, doing fewer monthly transactions (by a factor 0.37), the chances of the bug occurring in a Claimant's branch would be about 2 in 10 million.

762.

I have considered a bug similar to the suspense account bug, which occurred about 10 times, and had a mean financial impact of about £1000 per occurrence. How many similar bugs would be needed, to give a one in ten chance of one such bug occurring, with an impact of £1000, on a particular Claimant's branch in a particular month?

763.

The answer, given by elementary arithmetic which I describe in section 8.5, is that there would need to be 50,000 of these distinct bugs. If there were fewer than 50,000 similar bugs, if any Claimant were to assert that in a given month a shortfall of £1000 in his accounts was caused by bugs in Horizon, then the chances of his assertion being correct are less than one in ten.

764.

So the Claimants cannot credibly assert that their shortfalls were caused by bugs in Horizon, unless there were something of the order of 50,000 such bugs.

765.

Only three such bugs have been found. My own search of KELs has found only 8 other possible bugs. Mr Coyne's examination of over 5000 KELs has found no other bugs which definitely caused shortfalls.

766.

Thus the Claimants' case requires 50,000 bugs in Horizon - but only a handful have been found by the experts. Neither expert can quantitatively support the Claimants' case.”

(emphasis added)

806.

His correction was that in respect of paragraph 761 the sentence that reads "the chances of the bug occurring in a Claimant's branch would be about 2 in 10 million" should be "the chances of the bug occurring in a Claimant's branch would be about 2 in a million". He said that this was a typographical error. When he was asked if he had spotted any other errors, he said:

“A. No, I didn't because there were quite a few statements of this "in a million" nature in my report. I made careful checks of the main thread that led to my key result in section 8.7, and that was my focus. And similarly on the transaction corrections I had made checks to that spreadsheet and, you know, I included the spreadsheets with the reports so that you can see how the arithmetic is done from the spreadsheets.

Q. And some of it you point out is basic arithmetic?

A. I think all of it is basic arithmetic.”

807.

He said, in relation to a question as to whether Mr Emery, his assistant, had “checked the maths” that “No, he didn't actually. The maths is there in the spreadsheets for anybody to check. It is not advanced statistics, it is basic multiplication and division, and it is laid out in the spreadsheets.”

(emphasis added)

808.

His methodology in terms of his statistical analysis was explained by him in section

8.5 of his 1st report. This stated:

“617. I shall assess the financial impact of bugs in Horizon over three different scopes:

617.1.

(a) Across all Post Office branches, during the lifetime of Horizon.

617.2.

(b) Across all Claimant branches, while they held them.

617.3.

(c) On a single Claimant branch in a single month.

618.

It is possible to relate the financial impacts on these scopes, by numerical scaling factors. I calculate those scaling factors in this sub-section.

619.

Over the period 2000 – 2018, (i.e. 19 years) the Post Office network has consisted of more than 11,000 branches. The mean number of branches in all years over the period has been about 13,560. This figure is derived from the spreadsheet referred to at paragraph 178 of Ms Van Den Bogerd's Second Witness Statement, assuming that the spreadsheet is accepted. If this evidence is accepted, the number of 'branch months' (a single branch, trading for a single month) has been 13,560 * 12 * 19 = 3,091,680. This is the number of monthly branch accounts that have been produced.

620.

This means that for a typical Post Office branch, the scaling factor between scope (a) and scope (c) for the impact of bugs in Horizon is a factor of 3 million.

621.

For Claimants' branches, rather than typical branches, the scaling factor of approximately 3 million may need to be adjusted for two possible effects:

621.1. It might be asserted that Claimants' branches are more or less likely than other branches to be hit by bugs in Horizon, because of some special property of Claimants. 621.2. Claimants' branches may, on average, be smaller or larger than typical branches across Post Office network. If they are smaller, they handle fewer transactions in a month, and so are less prone to Horizon bugs in those transactions.

622.

It seems implausible to me that there is some special factor about Claimants' branches, which makes them much more prone to bugs in Horizon - bugs which one would expect to strike any branch at random. Nevertheless, I have considered the possibility carefully in Appendix F. I have shown there that there is no significant difference between Claimants' branches and other branches, in proneness to bugs in Horizon.

623.

It appears, from the spreadsheet attached at paragraph 179 of Ms Van Den Bogerd's Second Witness Statement, that the Claimants' branches are, in terms of customer transactions carried out per day, smaller than the average across the whole Post Office branch network.

624.

If this spreadsheet is accepted, it implies the following about Claimants' branches:

624.1.

From summing rows of the spreadsheet, the 561 Claimants' branches carried out 558,000 customer transactions per week in 2007.

624.2.

This is 558,000/6 = 93,000 transactions per day, assuming a Post Office branch is open for 6 days a week.

624.3.

Across 561 Claimant branches, this is an average of 93,000/561 = 165 customer transactions per branch per day.

625.

For comparison with this figure, I need to estimate the average size of branches across Post Office Network. I have done this using two pieces of evidence. Ms Van Den Bogerd's Second Witness Statement says that, across the whole Post Office network, there are approximately 48 million customer transactions per week, or 8 million per day in 2017 (assuming again that branches are open for 6 days in the week).

[626. I have omitted the table of EPOSS Transaction Rates, which are then summarised in 627]

627.

The best estimate I can make of the average daily volume in 2003 from this table is to divide the peak month by 26 working days. This gives an estimate of approximately 4 million transactions per day in 2003, compared with 8 million transaction per day in 2017.

628.

I therefore estimate that the average volume of transactions over the period 20002018 has been approximately 6 million transactions per day, mid-way between the 2003 figure and the 2017 figure (assuming Mrs Van Den Bogerd's evidence is accepted. I note that Mr Coyne quotes the same figure from her).

629.

To estimate how much smaller the typical Claimant's branch is than the average Post Office branch:

629.1.

The average transaction rate for all Post Office branches is 6,000,000/13,650 = 439 customer transactions per branch per day.

629.2.

So, in terms of customer transactions per day, the typical Claimant branch was smaller than the average Post Office branch by a factor 165/439 = 0.37.

630.

So, Claimants' branches, being generally smaller than Post Office average, have fewer transactions per month and so are less likely to be hit by a Horizon bug in a given month. (The calculation of the factor 0.37 contains some minor approximations that can be improved with further effort. I intend to do so in my supplemental report.) The factor 0.37 increases the scaling factor above, between scopes (a) (see paragraph 617.1) and (c) (617.3) from about 3 million to about 8 million.

631.

I illustrate what the factor of 8 million means using a hypothetical example of a bug which has occurred 16 times over the lifetime of Horizon, with mean financial impact on these occasions of £1000. Call this Bug A. The financial impact of Bug A is similar to that of the Suspense Account bug.

632.

If I selected a Claimant's branch and a month at random, then the chances of Bug A occurring at that branch in that month are only 16 in 8 million, or 2 in a million - an extremely small probability.

633.

Different types of bug occur independently of one another, so their probabilities of occurring are additive. If there were a second bug similar to the hypothetical bug above - call it Bug B - then the chances of either Bug A or Bug B happening to one branch on one month are twice the previous figure - 32 parts in 8 million.

634.

If there were 100 similar bugs - called Bug A, Bug B, Bug C,... Bug Zz - the chances of any one of them happening to one branch on one month are still only 100 * 16 parts in 8 million, or one part in 5,000. This is still a very small probability.

635.

It then follows that in order for one occurrence of a bug, of similar financial impact to the Suspense Account bug, to have even a one-in-ten chance of occurring to one branch on one month, there would need to be 50,000 such distinct bugs - because 50,000 * 16 / 8,000,000 = 800,000/8,000,000 = 1/10. There would have to be a Bug A, Bug B, Bug C, and so on, in a list with 50,000 distinct bugs.

636.

Even if there were some 'super-bug' - with financial impact ten times larger than the suspense account bug - there would have to be approximately 5,000 such superbugs to give a one in ten chance of affecting a Claimant's branch accounts in a given month. There is no evidence for even one such super-bug - let alone for 5,000 different ones.”

[637 and 638 then provided a further table and calculation that I omit]

639.

The Claimants have never asserted that there are as many as 50,000 distinct bugs in Horizon, with each bug on the same scale of financial impact as the Suspense Account bug. Their case rests on two or three known bugs of this scale, and on the unproven assertion that there may be others.

640.

If any Claimant were to assert that, for instance, a deficit of £1000 had occurred in his branch in a particular month, caused by a bug similar to the Suspense Account bug, the chances of that assertion being correct are extremely small, because Horizon bugs strike so rarely - unless Horizon contained of the order of 50,000 distinct bugs of that kind (and even then, the chances are only one in ten).

641.

Mr Coyne has examined more than 5,000 KELs, and not found definite evidence for even one bug with impact similar to the Suspense Account bug - let alone 50,000 of them.

642.

The implication of this result - the very small probability of any error in one months' accounts from a bug in Horizon - is that the accounts for any branch on any month are overwhelmingly likely to be correct (apart from effects such as delayed TCs, which are corrected after a variable delay).

643.

In my experience, no commercial IT system could ever go live with as many as 50,000 serious bugs - and certainly could not have the good in-service record over 18 years that Horizon has had.”

(emphasis added)

809.

I have reproduced extensive passages of Dr Worden’s explanation of his own methodology in order to allow his explanation to be available to any reader of this judgment. I consider this a wholly flawed methodology, and obviously so, and I reject it.

810.

There is another aspect to this which needs addressing at this point. It needs some context in order properly to be understood. Sums recovered from SPMs for discrepancies in branch accounts would be put into something called the suspense account. The claimants pleaded in paragraph 38 of the Generic Particulars of Claim the following:

“38. The Defendant operated one or more suspense accounts in which it held unattributed surpluses including those generated from branch accounts. After a period of 3 years, such unattributed surpluses were credited to the Defendant's profits and reflected in its profit and loss accounts.

39. The Defendant thereby stood to benefit and/or did benefit from apparent shortfalls wrongly attributed to the Claimants which did not represent real losses to the Defendant.”

811.

This was pleaded to by the Post Office in the Defence as follows:

“73. As to paragraph 38:

(1)

Post Office does not have a clear understanding of what the Claimants mean by the term "suspense account" or by the phrase "unattributed surpluses including those generated from branch accounts".

(2)

In relation to its dealing with its clients, Post Office is almost always able to reconcile its figures to those of its clients. However, as is inevitable given the nature and scale of those dealings, occasionally Post Office is not able to do so. For example:

(a)

A client, such as a bank, makes a payment to Post Office in respect of a particular transaction undertaken in a branch but the payment exceeds the amount that Post Office considers to be due.

(b)

Post Office seeks but does not reach agreement with the client as to the amount due and the client does not accept repayment of what Post Office considers to be the overpayment. In parallel, Post Office also seeks to determine whether the overpayment should be credited to a branch or to any other relevant part of its business.

(c)

Where the proper beneficiary of a credit cannot be determined, Post Office temporarily records the overpayment in its accounts. If the overpayment is not resolved within 3 years, the overpayment may be credited to Post Office's profit and loss account.

(3) The existence of processes for recording and resolving credits of the type described above are an ordinary business practice.”

812.

This was then pleaded to by the claimants in their Reply and Defence to Counterclaim:

“29. The Defendant's professed lack of clear understanding of the term 'suspense account' at paragraph 73(1), is noted at paragraph 2.1 above, and its response to the Claimant's RFI is similarly evasive.

30. The effect of the Defendant's example pleaded at paragraph 73(2), is that the Defendant admits the possibility that:

30.1.

a sum of money may be received which the Defendant believes to be an 'overpayment' from a third party client (e.g. a bank) in respect of a transaction undertaken in a branch: paragraph 73(a);

30.2.

the Defendant would seek to determine whether the overpayment should be credited to a branch, or to any other relevant part of its business: paragraph 73(b);

30.3 where the Defendant does not identify ‘the proper beneficiary’ of the credit, Post Office ‘temporarily’ records the overpayment in its accounts and credits that to its profits, if not resolved within 3 years.

31. This logic would apply where the Defendant’s internal records were wrong due to a Horizon error or in the case of any failure to identify the relevant branch or Subpostmaster”.

813.

I find that there is no sensible basis for the professed lack of understanding by the Post Office in its Defence of the term “suspense account”. That term is widely used by witnesses for both sides in the evidence, and I will give two simple examples showing its use. One of the Post Office’s own witnesses, Mr Johnson used it in his evidence, in a passage quoted at [279] above. Further, in an internal document which was prepared after the Callendar Square bug was discovered, it was widely used. An extract from that document appears at [404] above. Not only is the term used in four separate places in that document, there is even reference to the Suspense Account Team. The notion that the Post Office did not understand the term used in the Particulars of Claim is wholly fanciful. Further, there is an agreed bug (number 3 in the Bug Table) which the experts call the Suspense Account bug.

814.

Putting to one side the fact that the Post Office professed not to know what the term suspense account meant, notwithstanding its use in internal documents, the Defence makes it clear that unattributed sums within it, the correct destination for which the Post Office could not work out, were credited to its own profit and loss account. I accept the logic explained in the Reply in respect of the effect of this.

815.

Dr Worden had wholly failed to consider this issue. The point was put to him, not by reference to the pleadings, but by reference to another document which he admitted he had read, namely a report by Second Sight. That report had stated what is basically the same point but in slightly different terms, namely as follows:

“2.11 We note that Post Office's control and reconciliation procedures rely on correct information being supplied by third party clients. It follows that, if incorrect information is provided by any client company, this can give rise to a loss being charged to a branch. We also note that, for most of the past five years, substantial credits have been made to Post Office’s Profit and Loss Account as a result of unreconciled balances held by Post Office in its Suspense Account.”

816.

The question and answer are as follows, having read out the passage:

“Now I'm not asking you about that as evidence of itself, I'm just asking were you aware that that was a concern that Second Sight had raised about substantial credits being made to Post Office's profit and loss account as a result of unreconciled balances held by Post Office in its suspense accounts?

A. I read the Second Sight reports and the Post Office's reply.

Q. But did you spot that?

A. I didn't spot that, no.”

817.

This point was put in the context of evidence Dr Worden had given, namely the effect of small bugs or what he called “micro bugs” being unlikely to have an effect.

“Q. Okay. Well, what about -- because elsewhere in your report, I don't want to stray too much at the moment, but you also mention micro bugs, small bugs, being unlikely

because you would expect to see large sums of money somewhere -- A. Money that disappears somewhere has to pop up --

Q. It's got to come back somewhere else, it's got to pop up somewhere else.”

818.

The above passage was then put to him. Dr Worden’s failure to spot what is, in my judgment, a point directly relevant to the likelihood of Horizon bugs, errors and defects having a potential impact upon branch accounts is a notable one. Whether the point is taken from the pleadings, or from the Second Sight report (which Dr Worden accepted he had read, together with the Post Office’s reply to that document) does not much matter, in my judgment. I consider that this one omission alone - which I consider to be a serious omission on his part – is sufficiently fundamental as to undermine his evidence on this issue on its own.

819.

The following points are relevant in respect of Dr Worden’s statistical exercise:

1.

Statistics is the branch of mathematics that concerns the collection, presentation and analysis of data. There was no permission granted by the court for expert statistical evidence in this case. The permission that was granted was for expert IT evidence.

2.

Had either party sought permission to rely upon evidence from an expert statistician, I would not have granted it. Statistical analysis does not assist in resolution of the Horizon Issues, which required (in my judgment) detailed technical analysis and consideration by IT experts of the detailed occurrences of what could, potentially, be bugs, errors and defects. This was so that their expert opinion could be brought to bear, firstly, upon the important issue of whether there were any, and also if there were, their extent and the degree to which they potentially could impact branch accounts. This both experts did in any event, and that evidence – based on contemporaneous records and evidence about their occurrence - is of far greater assistance than this wholly theoretical argument about why Horizon bugs cannot have affected the claimants’ branch accounts.

3.

Dr Worden uses statistics, as he uses mathematics, as part of his professional life. Any particle physicist would, and does, use mathematics every day. Dr Worden accepted that this was not advanced statistics, and he was right to do so. It is a combination of simple assumption together with basic multiplication and division. The assumptions are not, in my judgment, valid.

4.

None of the Horizon Issues required the experts to perform a mathematical or statistical analysis of the likelihood of the claimants’ alleged shortfalls and discrepancies being caused by bugs, errors and defects in Horizon. Horizon Issue 3 – using as it does the term “extremely unlikely to be the cause of shortfalls in branches” – does make this exercise by Dr Worden relevant if that was how he chose to address that issue. That does not however make it either correct, or of any assistance.

5.

The accepted correction to the figure of “2 in 10 million” – which he accepted before his cross-examination started should be “2 in a million” – shows, in my judgment, how very broad brush this exercise was. As he put it in his oral evidence, “…there were quite a few statements of this "in a million" nature in my report”.

6.

It is plain that Dr Worden’s own evidence accepts that there are two fundamental assumptions that are central to this exercise. The first is that any bugs in Horizon strike branches randomly (which is spelt out in his report). The second is that the claimants’ branches are a random sample. This can be seen, implicitly, from his statement at paragraph 621.1 that “it might be asserted that Claimants' branches are more or less likely than other branches to be hit by bugs in Horizon, because of some special property of Claimants” and also at paragraph 632 “if I selected a Claimant's branch and a month at random….” Neither of these assumptions are valid, and the reasons that they are not valid are explained in this judgment.

7.

His paragraph 636 makes it clear how fundamentally unsound this exercise it. He stated “Even if there were some 'super-bug' - with financial impact ten times larger than the suspense account bug - there would have to be approximately 5,000 such super-bugs to give a one in ten chance of affecting a Claimant's branch accounts in a given month.” However, the evidence shows, quite specifically, that even for bugs such as Dalmellington and Callendar Square, SPMs’ branch accounts were specifically affected, and some by quite significant amounts. This therefore demonstrates that some SPMs as individuals could, and did, have their branch accounts affected by bugs. If the evidence does not match a theory (which here it plainly does not) then that must mean that the theory is flawed.

8.

Of his four different ways of addressing financial impact, set out in his paragraph 614, the second assesses the net impact of all bugs referred to in KELs. However, KELs do not invariably recite known financial impact. In some, known financial impact is recited, but a conclusion in many KELs is that user error cannot be ruled out and therefore a conclusion is reached that a bug is not responsible. I consider that Fujitsu would regularly record user error simply as a “catch all” and, effectively, require someone to rule it out to Fujitsu’s satisfaction, and unless this was done Fujitsu would remain wedded to that as an explanation. Given the technical investigations were being done by Fujitsu itself, Fujitsu rarely (if ever) ruled our user error by the SPM. This was wholly in Fujitsu’s interests.

9.

Dr Worden’s reasoning is entirely circular. By layering assumption upon assumption (and in places routinely misdescribing these assumptions as estimates) he felt able to conclude that there would have to be as many as 50,000 serious bugs for the claimants to be correct (as set out in his paragraph 643) and that “no commercial IT system could ever go live with as many as 50,000 serious bugs - and certainly could not have the good in-service record over 18 years that Horizon has had”. The whole rationale of this litigation is to work out, more or less, exactly what the “inservice record” of Horizon in fact is. I do not consider reliance on an assertion that it has had “a good in-service record over 18 years” can be used to underpin an exercise, the whole purpose of which is to consider this very issue. I reject his exercise that led to his figure of 50,000 bugs – it is wholly flawed. I also reject the circularity of his reasoning, which lacks logic.

10.

Dr Worden also implicitly assumes that all known bugs have been both reported and discovered, as he uses this (the total number of occurrences of the Suspense Account bug) in his table at paragraph 637. It is his label F and the source is “evidence on the Suspense Account bug”. Mr Coyne estimated that there may be as many as 40, based on an extrapolation of the number of KELs and PEAKs inspected.

11.

Dr Worden also misidentified the approach of the claimants, or certainly the approach of Mr Coyne. He stated “Their [ie the claimants’] case rests on two or three known bugs of this scale, and on the unproven assertion that there may be others.” That is not correct. The approach of Mr Coyne has been to consider PEAKs and KELs; see what is reported contemporaneously and what the conclusions were (or appeared to be) within Fujitsu; and consider what that contemporaneous account shows. Mr Coyne then uses that to draw conclusions. A great many of the bugs in the Bug Table use contemporaneous entries by Fujitsu which records bugs, errors and defects, as part of the consideration of whether there were such bugs, errors and defects. This is not mere assertion.

12.

However and in any event, by the end of the Horizon Issues trial, the Post Office admitted the existence of far more bugs than the “two or three known bugs” that were referred to by Dr Worden in the preceding sub-paragraph.

13.

Finally, I consider that Dr Worden’s statistical exercise is nothing more than a superficial short cut which avoids a detailed technical analysis of the history of the Horizon system in the way undertaken by Mr Coyne. In this case, on these issues, with all the detailed evidence available, it can be shown to be an unjustified way to resolve any of the Horizon Issues.

820.

Dr Worden also assumed that bugs in Horizon strike branches randomly. This too is not borne out by the evidence. For example, the Dalmellington bug (bug 4 in the Bug Table) appears to have affected only outreach branches. Relatively few of the Post Office’s branches are outreach branches. Bug 14 in the Bug Table, which Mr Coyne said was a bug but which Dr Worden described as “appears to be a system error with impact upon branch accounts” is entitled in the bug table “bureau discrepancies” and relates to foreign exchange transactions. Dr Worden’s exercise takes no account of what is, effectively, an admitted bug, and one that has an effect only upon foreign exchange transactions. If a branch conducted very few, if any, of these, it would not be affected by this bug. The chances of a branch being impacted by the bureau discrepancies bug depends upon whether it carries out foreign exchange transactions, and if so how many. That is not a bug striking all branches randomly. Indeed, rather tellingly for Dr Worden’s exercise, he assumes that any such bug would strike randomly across the whole branch network and have an equivalent random effect upon the branch accounts of branches that had performed foreign exchange transactions, and those that had not. This is plainly invalid.

821.

Dr Worden also assumed that the claimants are a random sample of SPMs. That this is one of his assumptions can be identified in the following way. His analysis of likelihood of any bug impacting any SPM’s branch accounts would apply equally to any group of SPMs chosen at random from those SPMs with smaller branches. The only adjustment or account taken of the claimant SPMs’ particular characteristics is that they are, generally, from smaller branches, and for this Dr Worden applies what he calls a scaling factor. However, the claimants are not a random sample of SPMs, nor are they even a random sample of SPMs from smaller branches. As a sample, they

have already been filtered or selected in that these particular SPMs already complain of bugs, defects and errors in Horizon having affected their branch accounts. This means that they are not a random sample. The way this would be expressed in statistical terms is that the claimant SPMs do not accurately represent the population of SPMs as a whole (or even the population of SPMs who had smaller branches). The claimants are essentially self-selected, from those who believe they have experienced shortfalls and discrepancies in their accounts from the impact of bugs, errors and defects, and who have been prepared to join the litigation. The group has a bias, in statistical terms. They plainly cannot be treated, in statistical terms, as though they are a random group of 587 SPMs.

822.

The adjustment made by him (or scaling factor, which was refined from that in his 1st report) to take account of the particular characteristics of the claimants does not validate what is, in my judgment, an entirely invalid base assumption. He attempts to deal with this by considering whether there is any “special property” of the claimants’ branches that makes them more susceptible to bugs, and concluded (at paragraph 622 of his report) that there is not. This is to approach the matter somewhat backwards, in my judgment. The experts are agreed that there were 3 million sets of branch accounts in the life of Horizon. The number of sets of branch accounts of which complaint is made in this litigation is a small (even perhaps tiny) proportion of that 3 million. However, that basic arithmetical fact does not mean that bugs, errors and defects cannot on the balance of probabilities be responsible for shortfalls and discrepancies.

823.

The exercise also wholly ignores what branch accounts are, how they are supposed to be created from the full list of transactions carried out in that particular branch in that particular period, and how precise they are required to be. The analysis of the “lasting and transient impact” approach taken by Dr Worden has already identified that this ignores the effect on branch accounts of the impact of a bug if no transaction correction has been issued before the end of the next trading period.

824.

There is also another fundamental problem with Dr Worden’s exercise, which is that it entirely ignores the specific purpose of Horizon, at least so far as the contractual relationship between the Post Office and SPMs is concerned. The Horizon system is, whether under Legacy Horizon or Horizon Online, the method by which the Branch Trading Statement or BTS is produced for an individual SPM’s branch Post Office. The BTS is the accounting mechanism under either the SPMC or NTC (the two different types of contracts used by the Post Office for SPMs during the period from 2000 onwards, explained in detail in Judgment (No.3) on the Common Issues). It was the common intention of the parties that the Horizon system would be used for the creation and recording of accurate information regarding all transactions performed in the branch which would be used for the production of the Branch Trading Statement for that branch. This means that at the end of each branch trading period, the BTS will (or should) provide a statement of the branch activity, which includes whether the accounts are in balance (which is where the cash and stock in the branch matches the Horizon system’s records of what should be there) or whether there is a discrepancy. If there is a discrepancy and it is a shortfall, the contractual terms of the SPMC and the NTC require the SPM to “make good” the shortfall, which means pay the money to the Post Office. The other alternative for an SPM is to “settle centrally”, which means seek time to pay. If the Post Office agrees, the shortfall can be paid over 12 months by deduction from the SPM’s remuneration. This purpose of the Horizon

system is entirely ignored in Dr Worden’s statistical exercise across the whole branch network.

825.

Consider a hypothetical bug, bug X. Also consider that bug X impacts upon branch accounts in a single branch upon a single occasion leading to a shortfall in the branch for that branch trading period. That shortfall would be treated as a debt by the Post Office and the SPM either has to pay, seek time to pay, or may even end up in a dispute with the Post Office and/or be suspended as a SPM and not pay. Analysis and resolution of the correct and true situation of the branch accounts between the Post Office and the SPM for the trading period in question does not depend upon whether, in all the other millions of branch accounts, there was no such incidence of bug X. The correct analytical approach in my judgment is to consider the branch activity for that branch for that period; consider the evidence both for and against (1) the existence of bug X and (2) the likely cause of the discrepancy, bearing in mind both the burden and standard of proof; make findings on the cause of the discrepancy; and then apply those findings. Expert IT evidence of most assistance in that exercise would be whether or not bug X exists or existed, and what were its effects. It is of no assistance to have an exercise that in effect says the statistical likelihood of any bug having an impact upon the branch accounts of that branch in that period is very low.

826.

The section 8 analysis is, in my judgment, so riddled with plainly insupportable assumptions as to make it of no evidential value. It is the mathematical or arithmetic equivalent of stating that, given there are 3 million sets of branch accounts, and given there are so many sets of branch accounts of which no complaint is made, the Horizon system is mostly right, most of the time. It is a little more sophisticated than that, but not by very much.

827.

The way that this can be further tested against the evidence is as follows. Mr Godeseth accepted that the Dalmellington bug had probably been present in the system from 2010 until it was discovered in 2015. Dr Worden’s analysis would lead to the conclusion that it was overwhelmingly unlikely that it had affected branch accounts during that period. But all the evidence is that this bug did affect branch accounts. It is correct that the Post Office relies upon the correction of known and identified discrepancies by means of the later issuing of TCs – a human measure which in my judgment is outside the definition of Horizon - but even putting that to one side, during this 5 year period branch accounts were actually affected. The existence of the bug was not known about for 5 years before it was even discovered. This round of the group litigation is to make findings on the Horizon system and whether there are any, or many, or no, other such bugs, errors and defects in the Horizon system similar to this one, or with similar detrimental effects. The conclusion of Dr Worden’s section 8 exercise is that it was overwhelmingly unlikely that any of the claimants’ branch accounts were affected by any Horizon bugs. This must include a conclusion that the Dalmellington bug cannot have affected any of the claimants’ accounts. This means that he must assume that Fujitsu were 100% successful in tracking down all incidents of the Dalmellington bug, and that any claimant SPMs who report symptoms the same as the Dalmellington bug and who had an outreach branch are either wrong, or making it up. These assumptions are ones that cannot sensibly be made, in my judgment.

828.

The whole section 8 exercise effectively is entirely theoretical, and is made in a vacuum that must ignore real world evidence to the contrary from one of the Post Office’s own witnesses, Mr Godeseth, as set out at [445] to [450] above. Even if I am wrong about all of the above, and even if a statistical analysis of this nature is valid, then in my judgment this would not assist the Post Office in its case on the Horizon Issues in any event. There are agreed to have been over 3 million sets of branch accounts over the life of Horizon, and approximately 6 million transactions every day across the whole Post Office branch network. There are fewer than 600 individual claimants, with the total value of accounting discrepancies over a 17 year period, amounting to only £18 million. As a proportion of the value of the Post Office network business over that period, and as a proportion of the total number of transactions, the number of transactions which underpin the claimants’ case is exceedingly small in percentage terms.

829.

Dr Worden’s probability calculation regarding two bugs, Bug A and Bug B, striking independently, was expressed as follows:

“632. If I selected a Claimant's branch and a month at random, then the chances of Bug A occurring at that branch in that month are only 16 in 8 million, or 2 in a million - an extremely small probability.

633. Different types of bug occur independently of one another, so their probabilities of occurring are additive. If there were a second bug similar to the hypothetical bug above - call it Bug B - then the chances of either Bug A or Bug B happening to one branch on one month are twice the previous figure - 32 parts in 8 million.” (emphasis added)

830.

He described the probability of these two unconnected events both occurring, each with a probability of 16 in 8 million (or 2:106), as being additive and he added 16 in 8 million, to 16 in 8 million, to arrive at their sum, 32 in 8 million. In terms of basic probability theory, the chances of two independent events occurring is usually seen as the product of the two probabilities, not their addition. This was recognised, as a SPM would not need both bugs to strike in order to experience an impact on branch accounts, only one would be sufficient. The principle of additivity is a fundamental principle that if the event whose probability is sought can be represented as the union of several other events that have no outcomes in common, then the probability of the union is the sum of the probabilities of the individual events making up the union. However, regardless of this, the point was neither put nor explored in any detail, and the outcome in any event depends upon how the events, and the union, are defined. The situation is further complicated by the fact that depending upon the nature of how “the bug” was defined, to impact upon branch accounts, this could be either Bug A or Bug B (in Dr Worden’s example) independently, or in conjunction. The PEAKs show that sometimes branch accounts would be impacted by a combination of rare circumstances. Such bugs may, but would not necessarily, have to occur at the same time. If independently able to impact branch accounts, they would simply have to occur at any time during the 4 week branch trading period. I refer to this matter for three reasons. Firstly, to demonstrate that probability theory only takes one so far in this case, and that is not very far; secondly, to explain that depending upon how one defines “event” and the intersection (or joint probability) of both or either occurrence, one can arrive at different results; and thirdly, to demonstrate that there were numerous flaws and/or difficulties with Dr Worden’s approach, not all of which were put, but which in my judgment undermine the validity of his exercise. This point about the probability calculation of two bugs striking a branch is not, however, not the main reason (or indeed, any reason) why I have rejected his analysis. The probability calculation for two bugs occurring in one month is not a central element of the statistical exercise he performed; only one bug would be needed in any one branch trading period for it to affect those monthly branch accounts. I have explained this in order to illustrate that it depends upon how one defines the impact of a bug, and chooses to assess its probability, that one can end up with an answer that rather suits the way the question is initially posed.

831.

Dr Worden accepted so far as his evidence generally was concerned, that he had assumed that the Fujitsu evidence was true, and in particular that of Mr Jenkins (whom he had met, although Mr Jenkins was not listed as a source of information in his 1st expert report, and from whom no witness statement was available) and that “generally, if findings -- if things happen in oral evidence which go against witness statements then I would need to come back and say what's the impact.”

832.

This therefore means that his evidence is of limited assistance in any event. I have found that the evidence of Mr Godeseth was somewhat different, at the end of his cross-examination, than how it had been presented in his three witness statements. Litigation cannot go on endlessly. For the Horizon Issues trial, evidence of fact and expert evidence were both ordered, and findings can only be made by the court in its judgments that resolve the disputes of fact. Disputes of fact in the evidence for the Horizon Issues will be resolved in the Horizon Issues judgment, namely this one. This is entirely conventional. If an expert chooses to prepare his or her evidence on the basis that one side’s evidence is true – which Dr Worden did in his reports, in respect of the party instructing him, namely the Post Office – they are not entitled to serve further expert evidence and wholly amend their opinions if that evidence is not accepted by the court. Dr Worden must have known this. There was going to be no opportunity for him “to come back and say what’s the impact”.

833.

This was put to him in his cross-examination.

“Q. The content of this report in this respect, we will come to other examples later, focuses on what the consequences would be if the defendant's factual evidence is accepted?

A. Generally, if findings -- if things happen in oral evidence which go against witness statements then I would need to come back and say what's the impact.

Q. Yes. You understand, don't you, that the moment at which everyone finds out whether factual evidence is accepted or not is when the judgment is handed down? A. Absolutely, yes.

Q. So you are proposing to revise your opinions after that, are you?

A. No. That's a good point. What I was saying there is if it becomes evident, all I can do is make assumptions. The court will find findings and all I can do is make assumptions and drive them through my opinions and try and assist the court that way. So all of this is assumptions I have made based on the evidence I have seen, and the court may find differently.

Q. Can I pause there. You were assuming there that Mr Godeseth's version was true?

A. I was assuming mainly Mr Jenkins' written analysis which Mr Godeseth's evidence confirmed.”

(emphasis added)

834.

The “evidence I have seen” to which Dr Worden referred was the Post Office’s evidence, and what Mr Jenkins had said (which was not evidence in any event). He did not take any account of the claimants’ evidence. This was notwithstanding that in his expert’s declaration he had expressly stated “I have not assumed that any particular version of events is true…..” when, indeed, he plainly had. It was also based on Mr Godeseth’s written evidence, not on his oral evidence, which was markedly different.

835.

He also said further on this same subject:

“A. Well, my analysis of the claimants' evidence is mainly contained in my supplemental report, and I explained there that individual claimants' evidence, particularly individual subpostmaster evidence, I did not feel able to make strong use of that and I gave the reasons in my supplemental report. So my opinions have little dependence on that. And the core of my opinions, the numerical estimates I make, those estimates have been designed -- or the process and the method has been designed so that if -- so that to make my assumptions evident where those assumptions come in, and if the court finds something different from my assumptions the court can go to the spreadsheets and re-do the method for itself.”

(emphasis added)

836.

This can therefore clearly be seen as his accepting the Post Office/Fujitsu evidence as true, particularly that of Mr Jenkins (not even a witness, and in respect of whom no witness statement was served); and also failing to base his opinions upon, or take account of, any of the individual evidence of claimant SPMs served by the claimants.

837.

Dr Worden is also under a misapprehension if he considers that the court’s function is to “re-do the method for itself” if his assumptions were not made out in the factual evidence. Firstly, he could have provided alternative expert evidence and opinion, were the written evidence of Mr Godeseth and the information coming from Mr Jenkins to be rejected. Secondly, he should have taken the claimants’ evidence into account, because there is no basis for an assumption that it was not relevant to the Horizon Issues which his expert evidence was the subject of. It is entirely normal for experts to provide alternatives of their opinion, depending upon which party’s evidence is accepted. Thirdly, the “method”, such as it is, demonstrated in the spreadsheets simply cannot be “re-done” to take account of findings that the assumptions were incorrect. The invalid and unjustified assumptions lead, in my judgment, to the inescapable conclusion that the whole methodology is flawed. No amount of “re-doing” of the calculations by the court can remedy that.

838.

Dr Worden also accepted that although he had missed some days of the trial, he was “pretty sure” that he had heard all of the Post Office’s evidence; he was then asked the following:

“Q. Have you had any changes of heart about anything in your report having heard it?

A. Changes of heart?

Q. Anything you want to change about what you said?

A. Well, had there been some fundamental change I would have felt obliged to communicate it to the court.

Q. Of course, because you take your duty very seriously?

A. Yes, I do. I can't think of any major change of heart, but there may well be things that you bring me to and I say there's an adjustment here. But to come back to the point about precision, I am conscious of how precise my numerical results have to be in order to be of assistance to the court.”

839.

This effectively must mean that Dr Worden did not consider that any of the changes, in the Fujitsu evidence in particular from the cross-examination, would impact upon his expert opinion. This is notwithstanding that he had assumed that the Post Office’s evidence would all be accepted, and anyone present for Mr Godeseth’s crossexamination can only have concluded that the factual position presented by Fujitsu in the written witness statements was very far from an accurate one. Indeed, the extent to which the cross-examination did not have any effect – either appreciable or otherwise – on Dr Worden’s expert opinions shows how very theoretical the whole exercise was.

840.

Finally on this point, due to the disruption caused by the recusal application, and the unavoidable re-arrangement of the trial timetable, the experts came to give their evidence in June 2019, and all of the factual witnesses who were cross-examined (with the sole exception of Mr Parker) had completed their evidence by 21 March 2019, many weeks earlier. Even Mr Parker had completed his evidence by 11 April 2019. Dr Worden was called on 11 June 2019, two months after this. It is unusual for there to be such an interval in a trial, but circumstances in this case led to extra time being available. There was more than ample time for Dr Worden to have identified how, if at all, the Post Office’s evidence of fact that emerged earlier in the trial (in particular from Mr Godeseth, but also from Mr Parker too) impacted upon his opinions. It plainly did not.

841.

I will provide only one example of Dr Worden’s unsatisfactory approach to the evidence, although there were many. In terms of whether transactions could be inserted at the counter in Legacy Horizon, what is called “remote access”, Dr Worden had in his first report concluded that Mr Roll (a witness of fact) was factually wrong. He expressly stated that he, Dr Worden, had “established” that Mr Roll was wrong. He accepted that use of the word was wrong but what he had done in his 2nd report was equally telling, in my judgment. This was after those witnesses who were to be called for the Post Office, still employed at Fujtisu, had corrected (or partially corrected) their erroneous first witness statements, and accepted that remote access without the knowledge of the SPM was possible. He said (at paragraph 83) that he required:

“further factual information before I can comment on this evidence. Which 'specific person'? Under what circumstances? How frequently? Until I have that information, it remains possible in my view that any transaction which 'would appear to the subpostmaster as though it had been carried out through the counter in branch' might only be a transaction that he had given his consent for, as the 'specific person' - and which had in effect been made on his behalf.”

842.

Dr Worden also then stated:

“84. Therefore, Mr Roll's new evidence does not cause me to alter the opinion expressed at paragraph 1119 of my first report, when commenting on Mr Roll's first witness statement, that he could not alter branch accounts without the Subpostmaster knowing.”

843.

This approach by Dr Worden to the evidence for the claimants on the one hand (Mr Roll) and the Post Office on the other (Mr Parker and Mr Godeseth in their statements) was explored in cross-examination:

“Q. Yes, the one that we agreed was incorrect in approach and conclusion.

A. We agree now, but at the time I wrote the second report, on the basis of Mr Roll's evidence and Mr Parker's evidence, I didn't see reason to change that opinion.

Q. I would suggest to you that you were inexplicably reticent to accept something that was contrary to Post Office's interests.

A. No, I was reticent. Not inexplicable, I was reticent because I had not seen sufficient evidence to convince me that these things could be done without the knowledge of the subpostmaster.

Q. But Dr Worden, would you accept that the approach you have taken there contrasts very, very strikingly in how you approach Mr Roll's evidence with the approach you took at 1119 in your first report when you accepted, effectively, Mr Godeseth's evidence?

A. I have accepted that my approach at 1119, that use of the word "established" was wrong and my approach was wrong, and we have established that -- you know we have done that before the interval –

Q. My question is: do you accept the contrast is very striking?

A. I think the court will have to -- I accept that my attempt to make my position clear that I'm not trying to find findings of fact, I'm not trying to find one witness or the other, I accept that on this occasion I fell short of that.

Q. Do you mean only in your first report or also in your second report?

A. Not in the second report. I believe the second report was -- you know, I believe that the evidence from Mr Parker and Mr Roll, and if we look again at Mr Roll's paragraph 20, I believe that the question that I asked in paragraph 83 that I want further evidence, I believe that was a valid approach.

Q. When you say in paragraph 84 -- because we have dealt with what you accept about your first report at paragraph 1119 several times, haven't we? A. Yes.

Q. So in your second report, you say, notwithstanding what you have now accepted about 1119, in your second report that you were right not to change that view. A. I hadn't seen sufficient evidence to change that view.

Q. But do you say you were right not to change that view when you revisited the entire piece in your second –

A. At the time I believe it was right not to change that view because I hadn't seen evidence that convinced me that this change could be made without the knowledge of the subpostmaster. That's where I am.

Q. Dr Worden, do you accept that that betrays a complete failure to appreciate the need to consider the situation both on the basis of whether the claimants' evidence is right as well as on the basis that the defendant's evidence is right?

A. No, I don't accept that.”

(emphasis added)

844.

Dr Worden had a further witness statement from Mr Roll saying effectively that Fujitsu could perform, and had performed, remote access. Dr Worden wanted “further evidence” and also referred to needing “sufficient evidence”. He said he “hadn’t seen evidence that convinced” him. He accepted he was reticent to accept the claimants’ evidence. All of this plainly demonstrates that he was taking a partisan view on evidence of fact. I find that this approach to disputed evidence of fact is simply not the correct approach for any expert to adopt. By the time of Dr Worden’s 2nd report he had a responsive witness statement from Mr Roll saying that what Dr Worden had “established” could not be done (a conclusion that was in any event directly contrary to Mr Roll’s earlier witness statement), very specifically, could be done. Even then Dr Worden would not accept it. As he stated in his cross-examination, “I hadn’t seen evidence that convinced me” that this could be done. The evidence was, very plainly, the witness statement saying that it could be done, and he chose to disregard it. This is just a more sophisticated way of stating the witness statement was not true. He described this as his being “balanced and fair”. I disagree with that description: it was not balanced, and it was not fair. He also said that there were many, many occasions when he had “tried not only to be fair to the claimants but to be biased in my numerical estimates in favour of the claimants”. I disagree with that too. Firstly, it is no answer to a criticism that an expert has taken a partisan view of evidence of fact and ignored one side’s factual account, to say that this has been effectively cancelled out by numerical assumptions elsewhere in different parts of the report. Secondly, I do not accept that Dr Worden was performing numerical calculations assuming in the claimants’ favour in any event, when one assesses the Section 8 exercise as a whole. The assumptions he made were very much not in the claimants’ favour at all. Minor adjustments to numerical figures in an exercise based on assumptions specifically unfavourable to one group does not either make the result “biased” in favour of that group, or cancel out the unfavourable assumptions. Further, the court requires an independent approach from an expert at all stages of their analysis, not a raft of slanted analyses, either one way and/or the other, with the hope (or here, an assertion) that they will eventually cancel each other out. For the avoidance of doubt, here they do not.

845.

Another area of difference between Dr Worden and Mr Coyne is the issue of what Dr Worden called “countermeasures”. Mr Coyne described these as “self-defined countermeasures” and stated that many of them – Dr Worden had identified 18 different ones – were basic elements of practical system design, and that such design aspirations did not, in themselves, demonstrate that Horizon was particularly robust.

846.

The 18 different ones are as follows, together with the acronyms adopted by Dr Worden:

1.

Reliable and redundant hardware – RHW;

2.

Robust data communication and replication – ROC;

3.

Transactional Integrity and database recovery – TIN;

4.

Defensive programming – DEP;

5.

Generic, data driven software – DDS;

6.

Secure kernel hardware and software – SEK;

7.

Redundant data storage and computing, with cross-checks – RDS;

8.

Double entry accounting – DEA;

9.

Early detection of user errors – DUE;

10.

Later correction of user errors – UEC;

11.

Manual workarounds – WOR;

12.

Testing good practice – TGP;

13.

Manual Inspection of Data – MID;

14.

Bug finding and correction – BFC;

15.

Large scale IT architecture – ARC;

16.

Quality and change control – QCC;

17.

Managing non-functional requirements – NFR;

18.

Security – SEC.

847.

These acronyms are, for the most part, not used in the IT industry, and were adopted by Dr Worden in this case. Some of them are, such as NFR and SEC. I use them in this judgment for consistency, but their use should not be taken to be a suggestion or acceptance that the acronyms are industry norms; the majority are not. The actual term countermeasures suggests that these consist of measures present in the Horizon system which have the effect of countering something else (bugs, errors or defects, issues that would have an adverse impact on robustness) and thereby improving the robustness of the Horizon system as defined. The Horizon system had an agreed definition for the purpose of the list of Horizon Issues which was as follows: “the Horizon System” shall for the purposes of this list of issues mean the Horizon computer system hardware and software, communications equipment in branch and central data centres where records of transactions made in branch were processed”.

848.

However, it can be seen simply from the descriptions above, which are Dr Worden’s own descriptions, that not all of the 18 are even part of the Horizon system. Some are simply manual checks. Some genuinely are countermeasures, but Dr Worden groups both types together.

849.

This can be seen from the following. Transactional Integrity and database recovery or TIN is undoubtedly a countermeasure within the Horizon system. The explanation given by Dr Worden in his summary table for this is “database management systems provide many facilities so that numerous kinds of failure cannot leave the data in an inconsistent, unusable state, or lose any data that have been previously stored.” This means if there is a failure, which may happen in a variety of different ways, the data should not be left in an inconsistent or unstable state, or be lost. The database management system accomplishes this in a variety of ways. Indeed, Dr Worden himself, with his extensive experience of database management, has particular familiarity with this. He designed and developed one of the first relational database management systems; it is an essential requirement in such a system that data should not be capable of being left in an inconsistent or unstable state. The reliability of any database relies upon reliability of the data within that database.

850.

Transactional integrity is a fundamental facility built into all database management software and this is made clear in, amongst others, published material such as An Introduction to Database Systems 8th edition by CJ Pearson published in 2003 which was one of Dr Worden’s appendices. As stated at 15.1 of that edition:

“Recovery in a database system means, primarily, recovering the database itself: that is, restoring the database to a correct state after some failure has rendered the current state incorrect, or at least suspect. (We will elaborate on what we mean by "a correct state of the database" in the next section.) And the underlying principle on which such recovery is based is quite simple, and can be summed up in one word: redundancy. (Redundancy, that is, at the physical level; we do not want such redundancy to show through to the logical level, naturally, for reasons discussed in detail elsewhere in this book.) In other words, the way to make sure the database is recoverable is to make sure that any piece of information it contains can be reconstructed from some other information stored, redundantly, somewhere else in the system.

Before going any further, we should make it clear that the ideas of recovery (indeed, the ideas of transaction processing in general) are largely independent of whether the underlying system is relational or otherwise—though it is true that most of the theoretical work on transaction processing has historically been done, and continues to be done, in a relational context specifically.”

851.

This is a countermeasure. Transactional integrity is intended to give protection against a wide variety of conditions, such as hardware errors or communication failures, software errors (which can lead to failures and cancellations) and operations being cancelled when they have not been completed. Such cancellations can be made by what Dr Worden calls the operator (the SPM or their assistant) but cancellations can – and the factual evidence shows this occurred – be caused by outages, power cuts and other incidents. A lay person might say that a system has “crashed", or one of the terms used in the factual evidence in this trial was “system outage”.

852.

Other measures are not strictly speaking countermeasures, they are sensible design intentions (Mr Coyne used the term design aspirations). An example of this is defensive programming or DEP. Since programming is at the heart of any software – indeed, programming creates the software itself – the way that programs are originally written (what could be called the programming of the system) constitutes the very system itself. DEP was described by Dr Worden in the following way in his table summary: “software is divided into small self-contained modules, which do not assume that other modules are correct, but defend themselves by checking their inputs and raising alerts early.” A better description was given elsewhere in his report where he stated that DEP was “where small parts of a program are written to assume that other parts of the program may be in error and are written to always check their inputs for the presence of errors.” In other words, part of a program does not start from the point of assuming that all inputs are without error; they check for the presence of errors first (in a variety of ways) before moving to other aspects of the program.

853.

This could be described as a countermeasure, but a more accurate description of it is a design feature that ensures subsequent steps are taken without repeating or carrying through errors. It is a function of design.

854.

However, Dr Worden included within his 18 separate countermeasures matters that are not, in my judgment, countermeasures at all and include remedies. For instance, early detection of user errors, later correction of user errors, manual workarounds,

manual inspection of data and bug finding and correction are not all countermeasures built into the system. Just to use that latter item as an example, the many hundreds of PEAKs and associated KELs show that 3rd level of support in the SSC at Fujitsu would investigate certain instances when SPMs reported certain matters. Often this would take many months. Depending upon the outcome of the investigations, matters would sometimes be reported onwards to Development for a software fix. The investigation by the personnel at 3rd level support (of whom, almost invariably, there would be more than one) and also the software writer in the Development section, were done by personnel who are plainly not part of the Horizon system as designed. They are human beings who are undertaking certain tasks. In fact, a software fix designed to prevent a problem re-occurring would change the Horizon system when that fix was introduced in live, a term meaning added to the software within the system. It is, in my judgment, illogical to treat such a process in the same way as either defensive programming (DEP) or transactional integrity (TIN).

855.

This approach is at its most extreme in Dr Worden’s view of transaction corrections, which he considered to be a Horizon countermeasure. The issuing of transaction corrections are plainly a remedy, or something done by the Post Office outside of Horizon to correct issues that arise because of Horizon. As Dr Worden accepted, this is not even done in a consistent way, depending upon the client in question. He stated:

“There are many differences of detail in how reconciliation is carried out for different client organisations, or where it is carried out; sometimes the client organisation does it from a file sent to it by Post Office, and sometimes Post Office does it. However it is done, and wherever it is done, the result of reconciliation is always in principle the same.”

856.

The way that Dr Worden explained the process was as follows:

“285. When reconciliation finds a transaction for which Post Office's record and the client record do not match, it is passed to a department in Post Office accounts which handles reconciliation discrepancies. Each such discrepancy is, until it has been dealt with, an error in the accounts - and so it must be dealt with. The task of this department is to determine how each discrepancy arose, which therefore determines how it needs to be handled.

286. When the appropriate department in Post Office decides that responsibility for a discrepancy lies with the branch, a request for a TC is issued and is passed from POL FS to the BRDB in Horizon Online. At this point, there is no impact on branch accounts. The request is passed on from the BRDB to the branch Horizon system, so that it will show on the Subpostmaster's screen when he starts the Horizon system the next morning. At this point, the Subpostmaster may either accept the correction or may question it and ask for further investigation.”

(emphasis added)

857.

This description makes it clear that there is a substantial link, or series of links, in that chain that are entirely outside of Horizon. The discrepancy, which Dr Worden correctly identifies as “an error in the accounts”, is passed to a Post Office department. That department has to consider it, and determine how it arose. Then that department has to decide responsibility, potentially issue a request for a TC, and that

is then passed to the BRDB (which is part of Horizon) and the request goes to the branch. A substantial element of the whole process does not involve Horizon at all. To treat the issuing of TCs as a countermeasure in the way that Dr Worden does is not logical. Even the Post Office’s own witnesses of fact did not deal with the process of issuing transaction corrections, because the Post Office’s solicitor correctly took the view that transaction corrections were not part of the Horizon Issues and were outside the scope of the Horizon Issues trial.

858.

Further, the scope of the experts’ agreements in this respect under Horizon Issue 5 in the 3rd Joint Statement is relevant. That issue was “how, if at all, does the Horizon system itself compare transaction data recorded by Horizon against transaction data from sources outside of Horizon?” (emphasis added). Agreed entries include the following:

1. Under sub-topic Reconciliation:

“5.1 Reconciliation between transactions recorded on Horizon and transactions recorded by Post Office's clients is largely automated.

Detected discrepancies were subject to manual corrective fixes and/or the issue of Transaction Corrections/Error Notices to the Subpostmasters.” and:

“5.3 The adequacy of Post Office back office processes to prevent discrepancies in branch accounts can be measured by the quality of the TC process. This quality includes:

The processes of consideration of available data

The level of errors observed in the process

The level of complaints or disputes raised following a TC

The level of upheld complaints following a TC • The level of financial impact of erroneous TCs.”

3. Under sub-topic Third Party Data:

“5.4 Errors in third-party data have led to discrepancies in branch accounts, through erroneous TCs being issued on Subpostmasters.

5.5a PO does not control the level of errors made by its third-party client organisations (which may lead to errors in TCs), or the delays in their processes (which may lead to delays in TCs).

5.5b PO can and should ensure, by careful investigation of disputed TCs, that only a small proportion of errors by PO clients lead to losses for Subpostmasters, provided that the Subpostmasters are in good control of their branches and have the required information available.”

859.

This shows that not only do the experts treat TCs as being outside Horizon in the Joint Statement, and also agree that the SPMs must have available the required information, but also agree that the level of upheld complaints in TCs (and the value of erroneously issued ones) is an ingredient required in order to measure the quality of the process. I find that the level of upheld complaints in TCs, and the value of erroneously issued TCs, is necessary in order to assess the quality of the Post Office back office processes to deal with discrepancies. That data was simply not kept, according to the Post Office witnesses. Further, Mr Coyne’s analysis showed that 77% of disputed Santander TCS were upheld. Entry 5.6 in this Joint Statement, not agreed by Mr Coyne, was an entry by Dr Worden which stated:

“The figure quoted by Mr Coyne (77% of disputed Santander TCs upheld) illustrates that this process worked well.”

860.

This means that Dr Worden’s analysis of “working well” was 77% of TCs issued in respect of Santander transactions which were challenged by SPMs was upheld. It is obvious that if 77% upheld, this means that 23% of TCs were not upheld, in other words were TCs that should not have been issued. I do not consider that a ratio of approximately 1 in 4 TCs being incorrectly issued can be properly described as the process “working well”, particularly as each TC would have a direct impact upon branch accounts. In my judgment, for the process of issuing TCs to be “working well”, the level of accuracy required would be far higher than 77%, and in excess of 90% (if not in the high 90s as a percentage) at the very least. Nor do I consider that to be unrealistically or unachievably high, given TCs impact upon branch accounts, the accuracy of which is required to be very high, and expected by the Post Office to be such that shortfalls in branch accounts have to be paid by the SPMs.

861.

As put to Dr Worden in his cross-examination, if his approach to interventions by human beings were to be extended to its logical conclusion, the group litigation itself would be categorised as a Horizon countermeasure. In my judgment, that would be the logical consequence of his approach. Plainly this litigation cannot properly be seen as a countermeasure. His approach to countermeasures is not a sensible one; he mixed up both genuine countermeasures such as TIN; design functions such as DEP; and manual corrections as well, and treated them all in the same way. He was wrong to do so.

862.

It was also somewhat unhelpful for some of his countermeasures, such as later correction of user errors or UEC, to be so widely defined as to include both manual checks – including by the SPM themselves – and the Post Office and Fujitsu’s both automated and manual processes. Another criticism of Dr Worden’s approach to countermeasures relates to one which Dr Worden described in his paragraph 463 as being “one of the most important countermeasures in Horizon”, and that is manual inspection of data. I do not see how this can be a countermeasure in Horizon as defined at all. Dr Worden’s summary states that “Any large business IT system is used by many people, who view its outputs and check them against each other for consistency, and against their own knowledge of the business. Subpostmasters, watching their branch accounts, were a key component of this.” (emphasis added)

863.

The inherent illogicality in this is, in my judgment, obvious. SPMs, viewing their own branch accounts, which are created or are part of the Horizon system, cannot in my judgment be included as a countermeasure in the Horizon system (even if one ignores entirely the definition of the Horizon system in the Horizon Issues). The fact that a SPM can or may notice that (say) their accounts are (as the SPM sees it) wrong, or that there is discrepancy of some thousands of pounds (or more, or less) in relation to a transaction that they may not recognise, cannot sensibly be described as a countermeasure in Horizon in the way that Dr Worden does. As Mr Coyne explained, this “assumes that the person scrutinising the data is able to identify the correct source of data to be relied upon, in order to rule out what may or may not be erroneous in the first instance. Additionally, its utility as a countermeasure is heavily reliant upon the person scrutinising the data already knowing there is an issue that requires data inspection.” It also obviously relies upon a human element, which necessarily depends upon the skills or knowledge of the person. When this is put with the lack of notification to SPMs generally by Fujitsu and/or the Post Office of existing bugs (and what the effect of those bugs was), which was a feature of the evidence, it can be seen that there is no basis for including this in any list of such measures.

864.

The effect of some, but not all, of the bugs might be susceptible to being noticed in this way, for example if there were a “doubled up” transaction in the same financial figure at the same time – the SPM might remember that a customer performed a single transaction in the way that Mr Patny did with MoneyGram as set out at [135] above. But that would depend both upon the type of bug, the type of transaction, the memory of the SPM and also crucially the amount. Another bug, Dalmellington, had some noticeable effects as the evidence in this case shows. Other impacts of other bugs would not be noticed in the same way. This shows the random nature of what Dr Worden describes as a countermeasure. In my judgment, Dr Worden was quite wrong to include this as a countermeasure. I find that something that is a “manual corrective fix” cannot sensibly be described as a Horizon countermeasure.

865.

One of Dr Worden’s other countermeasures was “testing good practice”, what he termed TGP. The purpose of testing any system is not to prove that it is correct; rather it is to prove (or attempt to prove) that it is not, a point explained by Dr Worden. Examples of this are regression testing, user testing and testing of edge cases. Dr Worden relied upon what Mr Membury in particular had given evidence about in his statement. I have summarised Mr Membury’s evidence at [500] to [507] above, including his serious omissions regarding E&Y and their concerns. Dr Worden included what was essentially a caveat in his first report, namely at paragraph 315 where he said in his section dealing with development and testing of Horizon, specifically in relation to Mr Membury:

“His witness statement describes several different audit processes, designed to ensure by independent external review that quality is managed effectively in Horizon (Witness Statement of Mr William Membery, 28 September 2018). If these audits have taken place as described in his witness statement, and if the results are broadly as he describes them, then that would increase my confidence that the quality of Horizon has been effectively managed.”

866.

He also stated that “a large system such as Horizon can only be robust if sufficient attention has been paid to issues of quality in its development, testing and support.” I have already identified that the experts are agreed that Horizon as it is now is relatively robust. The documents also show that the approach of Fujitsu seems to be

an increase in focus upon such issues since about 2016 onwards. Testing and evaluation is in three stages (taking this from another of Dr Worden’s appendices):

1.

Unit (or component) testing takes place once the product is developed or procured and ensures that the product conforms to its requirements.

2.

Link (or integration) testing verifies that the product interworks with other major components. This level of testing is generally performed on a complete release.

3.

Acceptance testing proves to the Post Office's client that the development meets its functional requirements and to the Post Office that it may be brought into use without impacting on the existing solution or other applications.

867.

This is then followed by what is called release management, which may be either major or incrementally using interdependent components. This approach to testing is entirely standard in the IT industry. No sensible programmer – or company involved in a system such as Horizon - would introduce changes without testing their impact. However, there is a paucity of documentation that demonstrates the degree to which Fujitsu’s testing regime was successful. Dr Worden, in both his report and appendices, identifies in general terms what was done and why it was important; there is no document available to the experts that demonstrates (for example) Fujitsu’s testpass percentages, or how specific failures were managed and re-tested.

868.

I do not consider testing to be a countermeasure. I consider testing to be an essential part of the foundation of any system. I would go further and say that comprehensive testing is standard in the IT industry.

869.

Finally, so far as security or SEC is concerned, Dr Worden explained that “any system that could be easily subverted would not be robust. Horizon is secured mainly through ‘separation of duties’, user authentication, access control and audit.” The evidence of fact made it clear that the degree of access available to the whole of 3rd line support at SSC was far wider than intended, and was subject to criticism by E&Y. Further, the extent of the powerful APPSUP privileges was extraordinarily wide.

J. Conclusion on Expert Evidence

870.

I prefer the expert evidence and approach of Mr Coyne to that of Dr Worden. This is for the following reasons.

871.

Dr Worden took a partisan view of the evidence of fact, preferring that of the Post Office’s witnesses (by which I mean in this instance predominantly Fujitsu) to that adduced by the claimants. This was most obvious in respect of his treatment of Mr Roll, but also occurred in respect of the claimants’ evidence of individual experiences of events caused by Horizon. No independent expert should do this. One example of this was the important issue of whether Fujitsu could inject transactions under Legacy Horizon without the SPM in question being aware of this. Dr Worden stated that he had “established” that these would be visible to SPMs. Mr Roll’s evidence of fact was directly to the contrary. Dr Worden had dealt with this by stating “So, in my opinion, Mr Roll could not have made these changes to branch accounts “without the Subpostmaster knowing”. In other words, in this expert’s opinion, the evidence of a witness of fact (and a very important one at that) was simply not correct in fact.

872.

However, in this case, even the Fujitsu witnesses eventually accepted (but prior to the trial, in their supplementary witness statements) that this had been possible, and that Mr Roll had been correct. Dr Worden’ supplementary report dealt with this by stating that there was a lack of understanding or clarity on this point. He stated that “it seems to me that I require further factual information before I can comment on this evidence” and also that “Mr Rolls new evidence does not cause me to alter the opinion expressed at paragraph 1119 of my first report, when commenting on Mr Roll’s first witness statement, that he could not alter branch accounts without the Subpostmaster knowing”. His first opinion in paragraph 1119 was wrong, and in my judgment obviously wrong, and this should have been unequivocally corrected both in his 2nd report and also in his cross-examination. The extract of this in [843] above makes it clear that he did not accept that.

873.

The sequence is therefore as follows. Mr Roll gave written evidence he had done something, and Fujitsu gave written evidence this was not possible. Dr Worden “established” that Mr Roll could not have done what he said he had done, and plainly preferred the Fujitsu evidence. He therefore adopted a position of which party’s evidence of fact was to be preferred. More factual evidence was then served and Fujitsu witnesses accepted this could be done, in contradiction to their earlier evidence. Dr Worden declined to “comment on this evidence” and remained of the view that Mr Roll could not have done that which he said he had done – in other words, his factual evidence was incorrect.

874.

No independent expert giving evidence to the court should do this. It is an obvious preferring of the evidence of fact of the party instructing him, added, in this case, to a refusal or failure to accept further evidence of fact to the contrary which subsequently emerged. This is even though the issue of remote access was accepted by the Fujitsu witnesses in supplementary statements.

875.

Mr Coyne did not do this. Indeed, he was fair and balanced in identifying where there were such differences, and he was also clear that he did not prefer one side’s evidence of fact to the other side. This is the correct approach for an expert to adopt.

876.

I find Dr Worden’s approach to the factual evidence I have identified concerning Mr Roll a particularly egregious failure to maintain the necessary standard of impartiality. Injection of transactions remotely into Legacy Horizon was an important factual point in this litigation, and indeed in the whole history of the ongoing lengthy dispute between the Post Office and the claimant SPMs. Further, in his expert declaration, Dr Worden went wider than the standard wording and specifically referred to an earlier decision in another case. He stated at paragraph 1194 the following:

“I understand that my duty in providing this report and giving evidence is to help the Court on matters within my expertise, and this duty overrides any obligation to the party by whom I am engaged or the person who has paid or is liable to pay me. I confirm that I have complied and will continue to comply with this duty. I have not assumed that any particular version of events is true and I have had regard to the case of Imperial Chemicals Ltd v Merit Merrell Technology Ltd [2017] EWHC 1763

(TCC) in producing my report”.

877.

That particular case (of which there are a number of judgments, on liability and quantum) concerned an expert and his preference for the evidence of fact of the party who had instructed him. I had stated in that judgment at [74] that “An expert's role is not to decide issues of fact themselves, and choose what facts to believe and what not to believe”. Specific reference was made by Dr Worden to this judgment in his expert declaration. It might therefore be thought that the requirement upon him as an independent expert not to take a view on which factual evidence he preferred was, or should have been, at the forefront of his consideration. In any event, such an approach by the court to how experts should approach evidence of fact is not particularly ground-breaking. Independence requires an objective consideration by an expert of both sides of the factual evidence. It is the court’s role, not that of an expert, to decide which evidence of fact is accepted and which is not. Notwithstanding this, Dr Worden plainly accepted the initial factual evidence of Fujitsu and rejected the factual evidence of Mr Roll. Then when Mr Parker accepted Mr Roll’s account, Dr Worden still would not do so. He was quite wrong to do this on both occasions.

878.

Further guidance on the approach of an expert, if guidance be needed, is also provided in [237] of a later judgment in the same ICI v Merit series, namely [2018] EWHC 1577 (TCC) in which I stated:

“1. Experts of like discipline should have access to the same material. No party should provide its own independent expert with material which is not made available to his or her opposite number.

2. Where there is an issue, or are issues, of fact which are relevant to the opinion of an independent expert on any particular matter upon which they will be giving their opinion, it is not the place of an independent expert to identify which version of the facts they prefer.”

879.

Dr Worden went so far as to state, in respect of his consideration of the variations in robustness of Horizon over time, that “in my opinion, as expressed in section 8, the Claimants’ shortfalls are not caused by bugs in Horizon, or any lack of robustness in Horizon.” The question of whether the claimants’ shortfalls was caused by bugs was not one of the Horizon Issues as I have explained above, but in any event, Dr Worden’s conclusions that the specific losses were not caused by bugs/lack of robustness was only reached by means of his statistical approach in section 8 of his first report. This was so heavily based on assumptions that were either not valid, or conceptually flawed, that his conclusions were not valid either. I would go further – the whole process he adopted so far as his statistical exercise was concerned was both invalid and of no evidential value.

880.

He also relied – in my judgment heavily - upon information from Mr Jenkins. Mr Jenkins was not even identified as one of his sources of information in section 1.3 of his first report, headed “Sources of information”. This means that Dr Worden was given access to information that was not made available to his opposite number, contrary to the point at (1) above in [878] above. Although there were some references throughout the text of the report to Mr Jenkins, Dr Worden did not routinely identify where he had relied upon Mr Jenkins. He also provided a great deal more information about this contact with Mr Jenkins in his oral evidence than he did in his written reports. In his cross-examination, he identified, when asked about a passage, that he had obtained that information from Mr Jenkins, which plainly took place before the first report was served as he accepted on his first day in the witness box it was “a year ago”. This was not clear in the report itself. One example of this was paragraph 654.2, where the report was dealing with the effect of the Receipts/Payments Mismatch bug, and Dr Worden stated “Because the operation involved was apparently not a double-entry operation on the BRDB, the countermeasure of checking the double-entry constraint DEA did not catch it”. This information came from Mr Jenkins, but until Dr Worden was asked this in crossexamination, no reader would be able to tell this. The involvement of Mr Jenkins in this explanation in his report was simply hidden. Nowhere was there a note or summary of all the information that had been given to Dr Worden by Mr Jenkins. In this litigation in particular, and given the involvement of and information provided by Mr Jenkins, who knew so much about the Horizon system, such a note or summary was, in my judgment, essential. This was particularly important given there was no witness statement from Mr Jenkins. Dr Worden had been provided with, and had used, information from Mr Jenkins in addition to the witness statements served by the parties.

881.

Dr Worden also introduced the concept of “lasting” and “transient” effect of bugs or defects on branch accounts. There was no such distinction in the Horizon Issues themselves. A further difficulty with this approach is that Dr Worden confirmed that even an impact on branch accounts that was corrected several months later by a TC would, so far as he was concerned, be of “transient” impact. A different way of stating the same concept is that, so far as he was concerned, “transient” could be of quite long effect. Given TCs are corrections generated outwith the Horizon system and are effectively corrective actions issued by the Post Office itself, this moved his consideration away from the generic Horizon Issues into the realm of the efficacy of the Post Office business systems as a whole, not only including Horizon but their human processes too. That is not what the Horizon Issues were concerned with, and he frankly admitted in his cross-examination that had he not done this, his opinion evidence would have been rather different. Mr Coyne did not use lasting and transient effect, and correctly addressed the actual Horizon Issues themselves, as drafted and agreed by the parties and as approved by the court.

882.

Dr Worden also adopted this different approach to impact upon branch accounts. Dr Worden only considered an effect or impact on branch accounts where a discrepancy loss (or gain) was not rectified by a correction such as a Transaction Correction, as set out in the entry in the 2nd Joint Statement about the experts’ different approaches. This ignored the word “potential” in the Horizon Issues themselves. In my judgment, that was a wholly unhelpful approach, and also had the effect of minimising or reducing the number of bugs that have an impact upon branch accounts. This can be demonstrated by considering the following hypothetical example.

883.

Consider a situation where there is a software bug, which in this instance I shall call H. The effects of H are such that they cause the branch accounts of a particular branch Post Office to suffer an unexplained loss of £999.99. This occurs, and the SPM of that branch contacts the helpline and goes through to SSC, who create a PEAK and identify this and after a few weeks of investigation, work out that this is an unexplained software bug which they have not come across before. A KEL is therefore also created identifying H. Future incidents of H occurring would (or

should) lead to new PEAKs which will also (or should) link to the same KEL, and to the same bug. Fujitsu report to the Post Office about the existence of H, which they have now discovered, and understand is a software bug. The Post Office accept that the sum of £999.99 has arisen in the branch accounts as a result of H and issue a TC to the branch in that sum, to correct the unexplained loss in that sum in the branch accounts. If one were to use Dr Worden’s interpretation of impact upon branch accounts, by reason of the issue of the TC, H would not be a bug, error or defect with an impact upon branch accounts. Such an approach is, in my judgment, wholly artificial and incorrect. It also defies the proper meaning of the words.

884.

Mr Coyne also considered substantially more PEAKs and KELs than Dr Worden, by a factor of almost ten. Mr Coyne stated that he and his team had considered a broad total of about 7,500. Dr Worden had only considered a far smaller sample. Mr Coyne’s far more vigorous and thorough exercise relied far less on the process of extrapolation inherent in Dr Worden’s approach, who was inevitably basing his opinion on a far smaller sample than Mr Coyne. Given the breadth of the resources being deployed by the Post Office in conducting this case, I find it difficult to understand why Dr Worden restricted his consideration of PEAKs and KELs in this way. However, I consider that the conclusions of Mr Coyne, based on the far more substantial consideration of these important documents, are more reliable than those of Dr Worden.

885.

To be fair to Dr Worden, he entirely accepted that he and Mr Coyne had approached their evidence differently. He stated the following when explaining why he had not supported Mr Coyne’s information requests.

“A. Well, basically I think it became evident, and it is evident now, that two experts took very different approaches, and my approach was top down, understand the architecture and work down through things like KEL. So as far as I was concerned, I had plenty of information to go on. And Mr Coyne's information requests didn't strike me as things that, yes, I have got to really see that because I was -- had a different priority. I was trying to do a top down understanding of the architecture and top down look at robustness, and so on and so forth. And so I had plenty of documents to look at, basically. So I think it is the different approaches taken by the two experts that led to lack of overlap.”

886.

This continued:

“Q. Sorry, not having supported the very many requests he had to make?

A. Yes, I felt his interests were different from mine and I really had plenty to do.

Q. And you initially only looked at KELs, didn't you, you didn't actually look at PEAKs?

A. No, I looked at KELs and I looked at PEAKs where they were relevant. I felt that KELs were a more distilled form of information. I felt they were sufficient in many ways, especially when you go and look at the PEAKs.

Q. You looked at one or two PEAKs referred to in the KELs you looked at?

A. Yes. Look at some of them where you feel the KEL doesn't tell you enough.

Q. And Mr Coyne was looking at PEAKs and KELs?

A. I think in the first report he was looking mainly at KELs, like me, and in his second report he turned on PEAKs.”

887.

The reason that PEAKs are of great assistance is that there is a great amount of further information contained in them than in the KELs, and they also give a far more complete picture of the number of different occurrences of a bug. Both types of documents are however useful. I do not know why Dr Worden should have seen Mr Coyne as having “different interests” to him.

888.

Dr Worden also entirely seemed either to misunderstand (or restrict his consideration of) Horizon Issue 1, which concerned bugs, errors and defects having the potential to cause apparent/alleged discrepancies or shortfalls in branch accounts or transactions, or undermine the reliability of Horizon accurately to process and to record transactions. When challenged about his approach to this, including the absence of the word “cause” and the use of the word “potential” in the issue itself, he stated the following:

“A. What's the distinction between having the potential and actually doing? I mean, if it doesn't actually do, then in some sense it didn't have the potential.”

889.

This is not the correct, or conventional, use of the word “potential”. It also looks at the matter in reverse. In the context of the Horizon system, whether bugs, errors or defects in that system have, or had, the potential to impact branch accounts or transactions, should focus one’s attention upon the possible effect of those bugs, errors or defects. If, of course, a particular bug actually has had the effect of in fact impacting branch accounts, then by definition it must also have had the potential to do so. Actual effect is a sub-set of potential effect. It means impact upon branch accounts must have actually happened. However, potential effect does not require that impact to have actually occurred. The two terms are not synonymous. Dr Worden therefore moved away from the actual issues before the court, and approached a great deal of his evidence from the point of rather different issues, which he seemed to have narrowed down himself. I consider such narrowing of the issues to be unwarranted.

890.

In their opening submissions, the claimants summarised this different approach as follows:

“We respectfully will invite the court to adopt an approach more consistent with Mr Coyne's approach which is to look at what actually happened with particular examples and trace them through to a reasoned and careful conclusion and then from the ground up, as it were, draw inferences upwards rather than from an overarching hypothesis downwards.”

891.

The “overarching hypothesis” was that of Dr Worden. Because he had taken such an overall view, and an overall view on the most recent performance and functionality of the Horizon System (which is now HNG-A as of February 2017, and is no longer HNG-X), the degree of assistance his evidence provided to the Horizon Issues across the whole life of the system, both Legacy Horizon and Horizon Online, was more limited than that of Mr Coyne.

892.

In this group litigation, when the first individual claims are being tried out, further analysis will be required of the actual branch accounts of individual claimants. That stage of the litigation will deal with actual branch losses, if any, that are said by the claimants to be due to the actual impact of the bugs, errors and defects which exist. It is at that stage that findings will be made about actual impact upon the branch accounts of the different claimants. Some of the bugs and defects are agreed by Dr Worden and Mr Coyne, and those that are not agreed are resolved by me in the Technical Appendix. It is against the background of those findings that individual claims will be resolved. The purpose of the Horizon Issues trial was to resolve the generic issues, as the parties were poles apart about even the existence of such bugs, errors and defects, and whether they had the potential to act upon branch accounts and transactions in the way alleged by the claimants. Dr Worden moved away from consideration of those specifically drafted and agreed Horizon Issues and answered rather different ones.

893.

I also consider that Dr Worden, in his approach to countermeasures, insufficiently differentiated between measures which were part of the Horizon system and those that were not. He also included items in his countermeasures that plainly ought not to have been included as such, for example those the result of human intervention or involvement, and manual inspection of data by SPMs.

894.

In my judgment, Mr Coyne emerged more or less unscathed from his four days of cross-examination by leading counsel for the Post Office. That is not to criticise the cross-examination itself, which was very thorough and careful. It is just that what the Post Office set out to demonstrate, including lack of impartiality on Mr Coyne’s part (if not positive bias), lack of reliability in his conclusions, and the other detailed criticisms of him, simply was not achieved. He was described as “evasive” in some of the Post Office’s closing submissions, a term which I find is somewhat wide of the mark in describing how he gave evidence in the case before me, and how he answered questions. He was also criticised in the same submissions as adopting “an approach [that] was simply to look for bugs without giving a proper sense of scale, context or even balance”.

895.

In the Post Office’s closing submissions, there is a section headed “Judicial criticisms of Mr Coyne’s approach” where reliance is placed on a judgment in Scotland, called CGI IT UK Ltd v Agilisys [2018] CSOH 112 by Lord Bannatyne. In that case, Mr Coyne had appeared as an expert and he was criticised by the judge for an unbalanced view, amongst other things. The judgment was handed down on 4 December 2018. The fact that he had been judicially criticised in that case was not specifically put to Mr Coyne. However, the substantive criticisms of him, that he gave unbalanced and selective evidence, and the other criticisms, were put to him, and he did not accept them. The point that he had been previously criticised in a judgment ought specifically to have been put by reference to that case, so that he could answer this point. However, in any event I have read (as invited in the Post Office’s submissions) the entire Scottish judgment, in which the evidence of Dr Hunt, Mr Coyne’s opposite number, was preferred to his evidence. I have done this with particular care, and I have also considered all the criticisms made of him in that judgment, as well as the ones put to him in his cross-examination before me. The Scottish judgment provides a full explanation of the shortcomings of his approach in that case. I have also taken this prior judicial criticism into account in weighing up his expert evidence in this trial.

896.

The Post Office, amongst other things stated that “it is submitted that the examples set out above [ie in the Lord Bannatyne judgment] show the same tendency to reverse

and switch in response to difficult points.” In other words, it is submitted that Mr Coyne adopted the same approach in the Horizon Issues trial before me, as he had in the earlier case in 2018. I have considered the criticisms of Mr Coyne with particular care in the light of this previous judicial criticism, even though I do not have the benefit of Mr Coyne’s explanation to the specific points arising out of that case. I do not find his evidence in the Horizon Issues trial suffers from the same deficiencies as Lord Bannatyne found in that earlier case. In particular, one of those criticisms in CGI, that he had failed to consider material documents and matters, simply does not remotely apply here. He and his team had done an extraordinarily thorough job of examining many thousands of PEAKs and KELs, and many more than Dr Worden did. I do not find that Mr Coyne reversed his evidence, or switched, or that any of the other criticisms of him are made out. It may be that he has changed his approach as a result of his experience in that case or judgment, or there may be other reasons for it. However, I do not find the criticisms made of him to be made out in this case.

897.

In my judgment, given the Horizon Issues, which expressly include “bugs, errors or defects”, the approach of an IT expert to start by looking in detail at the large volume of the Fujitsu PEAKs and KELs for bugs is not only entirely sensible, it is verging on essential. I reject the criticism that Mr Coyne’s evidence was given without having any sense of scale, context or balance. I conclude that Mr Coyne was impartial in his evidence in the Horizon Issues trial, and was giving the court evidence that was entirely uninfluenced by the exigencies of litigation or the interests of the party who called him. I find that he was independent, and that his conclusions in this case on the Horizon Issues are reliable.

898.

There were some subjects upon which he became a little flustered in his crossexamination, for example on the final afternoon when he was taken to task for the entry in the 2nd Joint Experts’ Statement on an agreed passage where he and Dr Worden had agreed that “the number of distinct bugs, for which the experts have seen strong evidence of the bug causing a lasting discrepancy in branch accounts, is between 12 and 29”. That latter number had come from Mr Coyne and he wished to reduce the number, initially to 13 but then upon reflection to 21. Mr de Garr Robinson made a great deal of this, as one would expect of a skilled cross-examiner, but in reality it is a simple point, as I have explained above. It did show a change of position because the number of bugs changed as I have explained. However, it is easy to overestimate the degree to which making an error on figures in such circumstances does, or does not, represent a lack of reliability on the part of an expert who has performed a substantial body of work. It must be remembered that this was an agreed entry in an agreed Joint Statement that was 44 pages in length (the experts agreed four separate statements, running to 100 pages in total including appendix). I deal with the detailed analysis of each alleged bug in more detail in the Technical Appendix, but the fact that the number did not remain an immutable “29” in all circumstances is not, in my judgment, a point against Mr Coyne. His attempts to ensure the final number was correct was not as smooth as he might have liked, and it gave Mr de Garr Robinson an opportunity to deploy his forensic skill, but in reality it did not impact upon Mr Coyne’s reliability.

899.

Mr Coyne was widely attacked for a lack of partiality, and indeed, for positive bias in favour of the claimants. I do not find those complaints or criticisms to be well founded. They are not based on any real substantive instances that can be shown in

any specific respect to represent Mr Coyne demonstrating any lack of impartiality. They are not made out. The points made are, in my judgment, insufficient grounds to dent his credibility as an independent expert whose evidence can safely be relied upon in resolving the Horizon Issues. He had not been previously instructed to undertake work for either party, and he had no previous involvement in the history of the wider dispute between the Post Office and SPMs concerning the Horizon system and its reliability. I found his evidence before me to be neutral and independent, and not influenced by the fact that he was instructed for the claimants.

900.

Dr Worden’s decision, very late in the day, to prepare his 3rd Report, appears – by reason of the dates - to have coincided with the dismissal of the recusal application. This matter was not explored at all in cross-examination and so it is not necessary to consider the timing of Dr Worden’s realisation that he could perform a new exercise. Given that his oral evidence to me was that he was advised to serve that 3rd Report directly upon the court, and effectively was left to draft the covering email to the court himself, I am reluctant to criticise him for this unusual approach unduly. This is therefore not something that I weigh in the scales to any appreciable degree in deciding whether his opinion evidence and conclusions are more, or less, reliable than those of Mr Coyne.

901.

I do not wish it to be considered that I am being universally critical of Dr Worden in all respects. He had done a substantial amount of detailed analysis in his two reports, and the overall knowledge that the court had, at the conclusion of the crossexamination of both experts, was very much greater than it had been prior to the commencement of the Horizon Issues trial. Dr Worden discovered some bugs himself, bugs that neither the Post Office and/or Fujitsu either knew about, or had admitted the existence of, prior to Dr Worden’s involvement in the litigation. He also took a sensible and considered view of some elements of the documentation, the importance or relevance of which had not been admitted by the Post Office earlier in the litigation, such as the KELs. Some of the matters that were agreed by Dr Worden with Mr Coyne in the 1st to 4th Experts’ Joint Statements saved a considerable amount of court time. His entries in particular in respect of the Bug Table, where he considered the detailed contents of a number of PEAKs and KELs, were useful as well.

902.

However, Dr Worden had become closely involved with the Post Office’s case in the litigation. In my judgment he took a partisan view of the evidence of fact of Mr Roll, which he should not have done, and he also seemed to be acutely aware of the Fujitsu position. For example, if there was a problem in Horizon that originated with Riposte, rather than part of the system written by Fujitsu, he was always enthusiastic to make clear this differentiation in Fujitsu’s favour. It was not relevant to the Horizon Issues whether bugs, errors and defects in Horizon were caused by Fujitsu or Riposte (provided by Escher, a separate company), or any other part of the system. These are all points that lead to the same conclusion, namely that in my judgment his expert evidence was not entirely independent of the Post Office’s and/or Fujitsu’s interests, and his conclusions were substantially less reliable than those of Mr Coyne. I do not know whether he had found himself subconsciously influenced by his discussion with Mr Jenkins or not, but regardless of the reason, his conclusions were not reliable and I reject them in favour of those of Mr Coyne.

903.

In all the major respects therefore, and taking account of all the points made both in cross-examination and submissions by both parties concerning their own, and the opposing side’s, IT expert, I prefer the evidence and conclusions of Mr Coyne to those of Dr Worden.

The number of established bugs, errors and defects in Horizon

904.

These are included in summary form in Appendix 2, together with whether they were present in Legacy Horizon or Horizon Online (or both) and their approximate years of operation. For a detailed analysis, the Technical Appendix to this judgment (which is Appendix 1) should be consulted.

K Audit Data

905.

The experts are agreed, in item 4 of the points of agreement referred to above in respect of Horizon Issue 3 as part of the 3rd Joint Statement, that the Post Office does not consult the audit data. The actual text of that entry is the “Post Office does not consult the full audit data (unfiltered ARQ Data) before deciding how to handle discrepancies and issuing Transaction Corrections.” Regardless of this agreement, it is obvious on the factual evidence that the Post Office does not do so. Indeed, throughout both the Common Issues trial and the Horizon Issues trial, it has been increasingly obvious that the Post Office uses sources other than the audit data when it is challenging SPMs about what they have actually done in their branch Post Offices. Various terms are used for the audit data. The 3rd Joint Statement uses “full audit data” and “unfiltered ARQ Data”. In places the term “core audit data” and “core audit store” was used, which refers to the same data and store. I shall refer to it as audit data.

906.

The issue of audit data arises in the following way. Audit data is a complete and accurate record of everything that has occurred, which in the context of Horizon means including a full record of keystrokes used by a SPM (or assistant) in the branch. This accurate record is kept in what is called the audit store. This is a secure place for the keeping of such data. It is vital to the proper operation of a system such as Horizon that such accurate audit data is kept.

907.

In his evidence Mr Coyne referred to the concept of “Write Once Read Many”, which has its own acronym, WORM. Dr Worden accepted that this was “pretty common” in the industry. It means, effectively, that audit data is only written once – indeed, it must be only written once. That is the whole point of audit data. However, it is written (or in other words, recorded) so that it can subsequently be read, and the second part of this concept, “Read Many”, means that it can be read many times. The audit data is there, in my judgment, so that it can be consulted when required. The experts’ agreement makes it clear that the Post Office does not consult it before deciding how to handle discrepancies and issue TCs.

908.

The Post Office did not accept that it should access the audit store as suggested by the claimants, or consult the audit data. The claimants’ case is that the audit data should be used in circumstances where there was a dispute between the Post Office and the SPMs about what had occurred in Horizon in a particular branch. Mr Coyne for the claimants criticised the Post Office in his 2nd expert report for not using audit or ARQ data, and relying instead on what was called management data, which was said (by the

Post Office) to be sufficient to demonstrate what had occurred. I will reproduce part of Mr Coyne’s cross-examination on this.

“Q. But did you see any documents of any sort indicating or referring to the stream of data flowing on a continual basis out of the audit store into Post Office's management systems?

A. No, but that's not how things would work. If Post Office wanted to get access to the data in the audit store they would go to a screen or go to an application on their computer and they would run the request for that data.

Q. Mr Coyne, I would like to suggest to you that it is completely unrealistic to think that a separate sealed core audit store of the sort we're talking about should be cracked open hundreds of times a day in preference to using management information systems which are designed for that precise purpose?

A. I think the word "sealed" is misleading and the concept of cracking something open to get access to it I think is misleading as well.

Things in an audit store are only -- can be written to and only written to once, and the term that's often used is write once read many, WORM. So the process is written to once, but people can read from that store on many occasions.

Q. But just to be absolutely clear, you had not and indeed you have not seen any documents suggesting that Post Office had the ability to gain access to the audit store on its own systems, had you? There was no design facility, there was no -- there were no lines of communication between the audit store and Post Office in any document you had ever seen, correct?

A. No, it looks as if the majority of the references to audit database access was from Fujitsu personnel.

Q. And one final thing. Would I be right in thinking that now that you understand how the audit store actually works and the costs and delays associated with extracting data on a large basis from the audit store, would you accept that it would be disproportionate to be using the audit store as a basis for making decisions on transaction corrections in every single case?

A. Yes, it would seem that it would be very expensive and very slow to access the audit store, and effectively for the number of transaction corrections you couldn't do that, and therefore you accept that you make decisions on the management information systems rather than the audit store.”

(emphasis added)

909.

The Post Office therefore relied upon the fact that the audit data was effectively kept by Fujitsu; that accessing it was akin to “cracking open” something that was a “separate sealed” store; and that it would be disproportionate to use it to make decisions on transaction corrections in every case.

910.

The first and third of those points are accepted; the second one Mr Coyne described as misleading. I agree with that response to all three of those points. I also find that the concept of “cracking open” the “sealed store” is rather off the point of the very purpose of audit data, and the uses to which it both can and should be put.

911.

The following points are, in my judgment, important on this issue.

1.

Audit data is necessary so that there is an accurate record of what has occurred on the system, including what a SPM has done in branch, in terms of the keystrokes actually used. The use of the word “audit” makes this clear, and also this is not a controversial point.

2.

The need to consult such an accurate record does not arise in every case where a transaction correction is considered or issued. The management data is sufficient for a large number of transaction corrections.

3.

However, where there is a dispute between the Post Office and a SPM about branch accounts, with the arguments on both sides about what or who is to blame, the audit data should be consulted. That is one of the purposes of having it in the first place. I can think of no sensible reason not to consult the audit data in such a scenario.

4.

Doing so is not equivalent to “cracking open” a “sealed store”. The audit store is sealed in the sense that data, once in the store, should not be capable of being changed, or written to again. However, reading that data is not “cracking open” the store. It is consulting or reading the audit data, one of the main (if not the only) purposes of having such audit data in the first place.

5.

The evidence in both this trial, and the Common Issues trial, where the Post Office cross-examined a number of SPMs on events in their branch accounts by using a variety of management information, other than the audit data, makes clear to me just how important it is to use the audit data, rather than other sources including management information. The management information is confusing, contradictory, has been shown to be wrong and requires numerous assumptions or a “take it from me” type of approach on the part of a questioner. It is rarely agreed what that management data shows. The audit data, by its very nature, will be far superior and the best evidence available of what has occurred on Horizon. It should be consulted in circumstances where there is a dispute between the Post Office and a SPM.

6.

Finally, even the Post Office’s own witnesses were not agreed on what the management data would, or would not, show. At one point Mrs Van Den Bogerd said that Credence (which is management data) would record keystrokes in branch. This was then contradicted by another Post Office witness who said Credence did not do this. Whether the lack of understanding on the part of Mrs Van Den Bogerd was widespread or not within the Post Office is unclear to me. It is however clear to me that Credence data does not in all circumstances record all keystrokes performed in branch by a SPM. The audit data does.

912.

Dr Worden explained audit data as being a central principle of the whole system in the following terms:

“It is a central principle of Horizon that the Core Audit Database acts as a secure 'gold standard' for branch accounts (countermeasure SEK) and that the audit record can only contain events which originated at the counter - either in customer transactions or monthly balancing.”

913.

That is an accurate description in my judgment. I can see no sensible or justifiable reason for the Post Office’s reluctance to consult the audit data in cases of serious

dispute with SPMs, in particular the types of dispute which form the subject of this case, and without doubt, any dispute that involves criminal proceedings against SPMs. I also consider the same point applies in relation to internal Post Office proceedings that lead to the suspension or termination of SPMs. Acceptance of that point does not mean that that the audit data has to be consulted for every transaction correction issued by the Post Office. As can be seen from the passage of cross-examination at [908] above, the Post Office effectively jumped to the issue of whether audit data should be consulted before issuing every single TC. Mr Coyne accepted it was not necessary to do that. That is not the same as accepting it should not be consulted in circumstances where there was a dispute between the Post Office and SPMs.

914.

Mr Coyne had identified issues with using Credence data. There was a one-hour difference in the time stamps used between Fujitsu and Credence, which can hardly have helped sensible investigations when SPMs raised queries, but there is more to this than that. The E&Y review in March 2011 identified various issues with Credence, including weak change controls within the back-end of the systems which allowed Logica developers (the third-party provider) to move their own uncontrolled changes into the production environment, which included both Credence software code and the data within Credence used for what was called “audit evidence” but which has to be differentiated from what I am referring to audit records in the audit store. There was a lack of further documentation to approve fixes and patches applied to Credence outside of the release process, which meant that linking changes to issue tickets to record the original request for the bug fix was not possible.

915.

A concern was also noted by E&Y in respect of front end change process weakness. The passage states:

“During our walkthrough of user administration of the front end of Credence we noted several users with administrator rights, including some generic users (this is noted below as a separate point). These users have the access rights to create and amend reports, including those which may be relied upon for audit evidence. These users can change report design, and processing without documented request, test or approval. When users have the rights to change reports that are used by the business for reconciliation, exception reporting or other processing, there is the risk that the reports are manipulated either intentionally or accidentally.”

916.

There is also at least one specific occasion, considered in the evidence during the trial, which shows that the Credence data does not show the correct position. This was put to Ms Van Den Bogerd. This was an occurrence at Lepton in October 2012. It led to something that was referred to throughout the trial as the Helen Rose Report or Rose Report, as the author of the report into it was called Helen Rose. I have dealt with it to a degree at [227] above. That report records that:

“A transaction took place at Lepton SPSO 191320 on the 04/10/2012 at 10:42 for a British Telecom bill payment for £76.09; this was paid for by a Lloyds TSB cash withdrawal for £80.00 and change give for £3.91. At 10:37 on the same day the British Telecom bill payment was reversed out to cash settlement.

The branch was issued with a Transaction Correction for £76.09, which they duly settled; however the postmaster denied reversing this transaction and involved a Forensic Accountant as he believed his reputation was in doubt.” (“duly settled” means the SPM paid the Post Office that sum).

The Credence data showed that the SPM had reversed the transaction. By consulting the audit data, Mr Jenkins discovered that he had not. This was expressly confirmed, both in the Rose Report and also by Ms Van Den Bogerd in her cross-examination.

917.

This shows that the management data is not entirely reliable. Had the SPM at Lepton simply paid the approximately £80 and not engaged the forensic accountant – in other words, had he not been prepared to pursue this matter – the Post Office would simply have received that sum from the SPM to which it was not entitled, and the Rose Report would not have emerged.

918.

The issue of remote access also further complicates (or rather, weakens) the reliability of management data. All of these concerns reinforce the view that I have in any event, that audit data (as in, that stored in the audit store) is what should be used in the circumstances that I have identified.

919.

One point that occupied the parties a great deal in the Horizon Issues trial, upon which there was some evidence, and upon which Mr Coyne formed a view based on his analysis of some contemporaneous documents, is that Fujitsu raised a charge to the Post Office for audit data requests (also called “ARQ requests”) above a certain number per year. The numbers that were discussed were in the amount of hundreds of pounds per request. The parties could not agree on what these charges were, although it was accepted that there were a number each year that were either free, or the costs of which were included in the contract sum (depending upon which way one looks at this matter). The charging structure will, in terms of the amount per request above a particular number, have changed over the years in any event.

920.

That charges are raised by Fujitsu to the Post Office is not an adequate answer, in my judgment, to the Post Office’s failure to consult or provide the audit data in cases such as those in [912] above. There are some contemporaneous references within Post Office documents suggesting this may have been a disincentive in some cases to raising ARQ requests of Fujitsu. Further, there are numerous references within Post Office documents (at a high level) of the very great cost to the Post Office of the Horizon system generally, that cost being paid to Fujitsu. Fujitsu were said in one document to see the contract with the Post Office as a “cash cow”. This entry came in a document (parts of which were redacted, which I record simply for completeness) dated 17 January 2017 which were the minutes of the Post Office Group Executive Meeting of that date. Ms Van Den Bogerd was “in attendance” for two items only, numbered 6 and 7, but not in the list of those “present” by which I assume she was not there for the whole meeting. The Post Office Chief Executive and Group Legal counsel were there, together with a number of others.

921.

There is only one agenda item which is not redacted, namely item 5, IT Strategy. Parts of the minutes relevant to the Horizon Issues are as follows:

“RISKS

Need to be clear for the Board what operational risks we are facing and how we are mitigating those risks, are there any choices we can make to accelerate the solution to get inside the risk appetite.

Already making these changes to the areas of risk, but will make this clearer. Need a strategic discussion at GE on choices in the IT strategy

Horizon - cannot accelerate work on the data centre refresh, getting under control. FJ running this well, risk review & declaration every month. RH will kick again but doing as much as we can.

Credence - Support and maintenance has moved to Accenture but on old hardware. Single platform if goes, goes completely. Driving pressure into Accenture to get more capability. Accenture picked up difficult pass and the additional costs at the moment. Starting to see light at the end of the tunnel. RH having difficult conversations, flag up any issues, assumptions and risks to GE. KPMG engaged to undertake a GAP analysis to produce an IT control framework.

What are the risks of the 'hop' to thin client not working? RH - need to prove thin client works and the 'hop' is a safe way to test we could revert to HNGA if the 'hop' fails. All nervous as we are dealing with legacy systems. More comfortable that we are not changing Back & Front office at the same time, but still using Horizon. SSKs have proven that thin client can work, it is the journey to implement that is vital.

Fujitsu - negotiations will be tough as POL do not hold the power. FJ see the contract as a cash cow, so need to persuade them that working with POL to migrate to cloud technology is to their benefit against a 'too good' contract. They may say make us whole but we cannot give additional work because of the procurement risk. Only choice if they will not work with us is to start building an alternative. Come back to

GE with an update on the FJ negotiations plan”

(emphasis added)

922.

This document shows that Fujitsu, at least so far as the Post Office are concerned, are seen by the Post Office as believing that the contract with the Post Office is “too good” and a “cash cow”, such that Fujitsu may not even “work with us” [ie the Post Office] in terms of improvement. Part of that very attractive commercial arrangement (attractive for Fujitsu) may be charges that Fujitsu is entitled contractually to raise for ARQ requests. Part of an attempt to reduce the expense of the contract by the Post Office may be a reluctance to raise such requests. Dr Worden described this document as “chiming” and said that “I have always had the impression from the governance structure and the documentation and so on that Fujitsu were not short of budget, really”. By that he meant Fujitsu were well-remunerated, and this remuneration was paid by the Post Office.

923.

Such private commercial arrangements between the Post Office and Fujitsu are a matter for the Post Office and Fujitsu. They do not, in any way, justify any failure to seek the audit data – the best evidence – in cases where SPMs are being suspended and/or having their appointments terminated, particularly in circumstances where there are so many bugs acknowledged as existing, and also at a time (much earlier

than this judgment in 2019) when Fujitsu knew there were bugs in Horizon such as Dalmellington and Callendar Square, and also given the Credence data has been shown to have been wrong on occasion. There is quite enough in issue in this litigation as it is, without widening it to include the way the Post Office deals with its commercial arrangements with Fujitsu.

924.

Finally on this point, I am surprised that the desirability - if not the actual and basic need - to consult the audit data is a controversial point. In my judgment it is not only good practice to consult the audit data, given the very purpose of audit data within a complex IT system such as this one, but it is also obvious common sense. There is little point in having audit data if it is not consulted in the circumstances that I have identified above. I appreciate that the audit data itself will not be immediately comprehensible to some SPMs. There may also be charges raised by Fujitsu in respect of such requests. However, neither of these are reasons for the failure to consult it. I also make no findings on whether Fujitsu are, or are not, entitled to raise charges and if so how much. Those latter two points were not fully argued before me, and Fujitsu were not represented.

L. Overall Conclusions

925.

My judgment concerning the two experts, which of their evidence I prefer and why, has been reached on free-standing grounds. Because of the different way that the experts went about their tasks, it was not possible to consider in detail and track in Dr Worden’s reports any analysis and possible explanation of single incidents, and attribute that either to a specific bug, error or defect, or something else which was none of these things. The approach Dr Worden had adopted for his written evidence did not permit that, as he was (in a sense) working from the other end (overall reliability) back towards the beginning (individual experiences), in contrast to the approach of Mr Coyne. Dr Worden himself described his approach as “top down”. Mr Coyne, on the other hand, using PEAKs and KELs in particular, identified a great many specific incidents that Fujitsu had recorded, which plainly demonstrated the existence of numerous bugs. Indeed, some of the terms used by Fujitsu made it crystal clear that these were known to be bugs. One notable example is the expression used by Anne Chambers – “this bug has been around for years” – in February 2006. Dr Worden did engage with the detail of PEAKs both in the joint expert meetings and also in his oral evidence, and the final result is that by the end of the trial, the summary at the beginning of Appendix 2 of the Post Office closing submissions, was forced to accept the existence of a great many bugs in Horizon. The extent and nature of their impact was not agreed, but the Post Office could no longer deny their existence.

926.

Those claimants who gave examples of specific incidents specified them as well as they could, and in my judgment truthfully and accurately. It was usually Ms Van Den Bogerd who, for the Pot Office, attempted to demonstrate that this was not what had occurred, often by means of assertions that if the SPMs had followed procedures properly, what they said happened, simply could not have happened. The Fujitsu witnesses gave some limited evidence of fact in relation to specific instances and some specific PEAKs and KELs. The Post Office’s counsel did their best to challenge the factual accounts that were given by the claimants. I have provided the relevant findings in respect of those matters above. I indicated at the Case Management Conference on 22 February 2018 that the Horizon Issues should be capable of resolution with limited, if any, evidence of fact. Both parties chose to submit factual witness statements and called factual evidence. The factual evidence submitted by both the parties was, in my judgment, limited as expected and required. With the exception of the matter to which I have already referred concerning the evidence of Mr McLachlan and Mr Henderson at the PTR, the extent of each party’s evidence of fact was not subject to challenge prior to commencement of the Horizon Issues trial.

927.

The factual evidence of specific instances was of assistance in coming to conclusions on the Horizon Issues. Indeed, I found some of the factual evidence to be of great assistance. That of Mr Roll and Mr Godeseth was extremely useful. The latter, one of the Post Office’s main witnesses and the Chief Architect of Horizon, was sufficiently damaging to the Post Office’s case on the Horizon Issues that they were, essentially, forced almost to disavow him, and the Post Office’s closing submissions were highly critical of the accuracy of his evidence. As a further example only, Mr Latif gave evidence that he had performed certain basic routine steps to transfer the sum of £2,000 between terminals, and (in accounting terms) that sum had – to use his word – “disappeared” from the Horizon system. The Post Office evidence challenging this was from Mrs Van Den Bogerd, and essentially amounted to an assertion that if certain steps had been followed by him, this should not, or could not, have happened. That would be all well and good in an ideal theoretical world that did not pay any attention to reality, but it entirely ignores Mr Latif’s primary evidence of fact. I accept that primary evidence from the claimants’ witnesses in this trial as I have explained in Part D. Mr Latif had been a trainer trusted by the Post Office to train other SPMs, as explained at [98] above. Other references in the documents show both Romec engineers, and even Post Office auditors, being recorded in PEAKs as witnessing events supporting the type of occurrences that underpin the claimants’ case. These were ignored by those at Fujitsu tasked with completing these PEAKs, and conclusions are shown in those PEAKs being drawn by Fujitsu that flew in the face of what had occurred. Mr Latif’s experience is extraordinarily similar to the Dalmellington bug, which is numbered 4 in the Bug Table. Mr Latif did not operate core and outreach branches, therefore his experience is slightly different to what happened at Dalmellington. However, if one is to take an objective and sensible view of the evidence of a previously trusted SPM, whom the Post Office itself has used to train other SPMs for some years, who stated that he did something correctly and X occurred, then meeting that with a bare assertion that X simply cannot have happened is not particularly sensible, nor is it persuasive.

928.

The approach by the Post Office to the evidence of someone such as Mr Latif demonstrates a simple institutional obstinacy or refusal to consider any possible alternatives to their view of Horizon, which was maintained regardless of the weight of factual evidence to the contrary. That approach by the Post Office was continued, even though now there is also considerable expert evidence to the contrary as well (and much of it agreed expert evidence on the existence of numerous bugs).

929.

This approach by the Post Office has amounted, in reality, to bare assertions and denials that ignore what has actually occurred, at least so far as the witnesses called before me in the Horizon Issues trial are concerned. It amounts to the 21st century equivalent of maintaining that the earth is flat.

930.

When real world examples such as Mr Latif’s are put together with the expert evidence that I have accepted – or even with Dr Worden’s lower figure for accepted bugs of 11 different ones – it can be seen that this institutional obstinacy by the Post Office amounts to little more than repeated assertions that the Horizon system (both Legacy and Online) cannot be to blame for the claimants’ experiences, coupled with (for some) challenges to the claimants’ witnesses because the Post Office simply cannot accept their factual accounts. The findings that I have made, on the evidence in the Horizon Issues trial, show that the reality is rather different, and the existence of the bugs, errors and defects that I have found to exist do have the effect explained by Mr Coyne.

931.

The approach that I have adopted to the evidence can be considered as follows. Firstly, to consider the factual evidence, the relevant contemporaneous documents, and applying the burden and standard of proof, reach evidential findings. I have considered the challenges to that evidence, and if the specific instances of factual witnesses explaining what they experienced could be explained (as the Post Office sought to do with some of the SPMs) by them simply not being reliable, remembering incorrectly, having been careless or even (for some) being criminal. I have also considered all of the Post Office’s factual evidence in this trial, the explanations of the Post Office’s witnesses, and reached my factual conclusions. I have considered the expert evidence on its merits, and then tested my conclusions against the evidence of fact that was adduced by both parties, and considered whether my judgment on the evidence of fact matched and supported, or contradicted, or in any way affected or influenced my conclusions on the expert evidence. In a sense this was to consider the factual evidence, and the expert evidence that I accepted and preferred, separately, and see if they both reached the same conclusion. Secondly, the evidence can also be approached in a different sequence. This is by firstly considering the specific factual evidence, and then considering the expert evidence, and only in the light of the conclusions on the factual evidence. Alternatively, one could simply decide the Horizon Issues on the expert evidence alone, although that would be far less preferable and would ignore entirely the important factual evidence of the Fujitsu witnesses, in particular Mr Godeseth. However, even if that third way were to be adopted, the same answers to the Horizon Issues would, in my judgment, be reached.

932.

Whichever way is adopted, it must be remembered that any claimant can only ever have personal experience of one small element of the Horizon system on a number of occasions. The overview and historical performance of Horizon, both Legacy Horizon and Horizon Online, has to come from Fujitsu witnesses and the two experts. Dr Worden and Mr Coyne had far more of an overview of the life of the system, as did Mr Godeseth and Mr Parker. Mr Roll had experience of it in its early years, but his evidence was what led to the correct picture on remote access finally emerging.

933.

I have done all of these exercises. Regardless of whichever approach is adopted, the same conclusions are reached. In this respect, all roads lead to Rome. In my judgment, the evidence of the factual witnesses accepted by me entirely supports and corroborates the conclusions reached by Mr Coyne. Indeed, I would go further than this, and state that Mr Godeseth’s evidence alone is enough to support and corroborate Mr Coyne’s conclusions. When that is put together with the evidence of Mr Roll, and the concessions that were obtained from the Fujitsu witnesses (in the circumstances of their performance as witnesses, to which I have already referred) it is clear to me that

the correct conclusions to be drawn on the Horizon Issues are those drawn by Mr Coyne, save and to the extent that I have modified them in any specific respect.

934.

I consider that the evidence of the Fujitsu witnesses in particular, both former and current, has been of considerable assistance to me in resolving the Horizon Issues. Mr Roll, Mr Godeseth, and even in his own way Mr Parker (though not in the way that Mr Parker himself would have intended) have all provided clear evidence of the problems with the Horizon system, the bugs, errors and defects within both Legacy Horizon and Horizon Online in its HNG-X form, the way that these problems were (or were not) dealt with, and the way that Fujitsu had powers which, until shortly before the trial started, Fujitsu sought to keep from the court, and may not even have fully disclosed to the Post Office. Because the extent of these powers was kept secret in this way, the Post Office finds itself now having made misleading public statements previously. If one looks back to an earlier case management hearing and reconsiders how Fujitsu, through the Post Office, sought to portray the contents and lack of importance and relevance of PEAKs and KELs, then it can be seen that there has been a pattern of considerable defensiveness over the Horizon System. There has certainly been a lack of transparency, and a lack of accuracy in description.

935.

I have explained above at [172] how the Post Office sought to draw a particularly glowing reference in Fujitsu’s favour from the evidence of Mr Roll in their closing submissions. The Post Office stated that from his evidence “….a clear picture emerged of Fujitsu as an organisation which was thorough, professional and conscientious and which took considerable care to ensure that matters were properly investigated and dealt with.” I have already stated that this is not a conclusion that can be drawn from the evidence of Mr Roll. Nor is it a conclusion that, in my judgment, can sensibly be drawn from the totality of the evidence taken together, both factual and expert, in the Horizon Issues trial. As an example, on too many occasions are there PEAKs that conclude at one point that user error cannot be to blame, only for a misleading closure code to be used for such a PEAK. Fujitsu personnel routinely refer in such documents to the known existence of bugs, without this (so far as the documents deployed in the trial are concerned) being communicated to the SPM in question in these terms. In places there is even debate at Fujitsu shown in the documents about whether the Post Office and/or SPMs should be told. I do not see how a thorough, professional and conscientious organisation can have produced for disclosure in this litigation so many thousands of KELs during 2019 itself, both during and even after the trial. I reject that description; it is an inaccurate description of Fujitsu and/or its investigative motivation.

936.

I consider, as explained in the Technical Appendix, that Legacy Horizon was not robust, and that although Horizon Online in its HNG-X form was better than Legacy Horizon (not least, I consider, because Riposte was no longer part of Horizon) its robustness was questionable, and did not justify the confidence placed in it by the Post Office in terms of its accuracy. HNG-A is a different matter, and the experts are agreed that it is far more robust than Horizon in earlier times. On the face of the relevant KEL, it is not possible to say whether the Drop and Go bug, number 28 in the Bug Table, which occurred in June 2017, was HNG-X or HNG-A. However, if the latter, it is one of only two such items in the Bug Table that relate to the existing version of the Horizon system.

937.

I turn therefore to the points identified by Males LJ in Simetra Global Assets Ltd and another v Ikon Finance Ltd and another [2019] EWCA Civ 1413 to which I have referred at [74] above. I have already dealt with the first point, succinctness (obviously lacking in a judgment of this length). The second point is a requirement to consider all the evidence. I have done this, both for and against each of the cases advanced by both the claimants and the Post Office. I have not recited every single disputed fact as to do so would lead to an even longer judgment, and substantially delay production of this judgment well into 2020, and which would be contrary to the overriding objective. The third point is to identify the issues, consider the evidence in respect of the issues, and give reasons. For this trial, the Horizon Issues were agreed by the parties and approved by the court; the review of the evidence has been longer than is ideal, but I have considered it all, whether recited in this judgment, or the Technical Appendix, or not. The fourth point is somewhat salutary, namely that fairness requires that a judge should deal with apparently compelling evidence, where it exists, which is contrary to the conclusion which he proposes to reach.

938.

In my judgment, there is no such evidence advanced by the Post Office. It is pointless to speculate on how such compelling evidence might have been presented, if there had been any. The evidence that has been advanced for the individual occurrences, for example by Mrs Van Den Bogerd, can be seen once tested in cross-examination to amount to unfounded assertion and, sometimes, an expressed desire, even in 2019, to go away and do more research. This is not apparently compelling evidence to the contrary to my conclusions. Additionally, the time for further research by the Post Office is long past. The Post Office’s approach to evidence, even despite their considerable resources which are being liberally deployed at considerable cost, amounts to attack and disparagement of the claimants individually and collectively, together with the wholly unsatisfactory evidence of Fujitsu personnel such as Mr Parker. The Post Office evidence also includes a very high level overview of Horizon by its expert which amounts to little more than a claim that it has worked quite or very well, most of the time. This is effectively what Dr Worden’s section 8 exercise consists of. Counsel for the Post Office at one point put to Mr Coyne that there was more likelihood of an SPM being hit by lightning than of having a bug, error or defect in Horizon cause a branch account loss in their branch. Mr Coyne did not agree with that, and neither do I. Indeed, such a point wholly ignores the enormous number of PEAKs that show that branch accounts were potentially impacted on a great many occasions.

939.

Also relevant are the other points which Males LJ went on to consider, namely the importance of contemporaneous documents. He stated the following on this subject at [48] and [49] of Simetra:

“[48] In this regard I would say something about the importance of contemporary documents as a means of getting at the truth, not only of what was going on, but also as to the motivation and state of mind of those concerned. That applies to documents passing between the parties, but with even greater force to a party's internal documents including emails and instant messaging. Those tend to be the documents where a witness's guard is down and their true thoughts are plain to see. Indeed, it has become a commonplace of judgments in commercial cases where there is often extensive disclosure to emphasise the importance of the contemporary documents. Although this cannot be regarded as a rule of law, those documents are generally regarded as far more reliable than the oral evidence of witnesses, still less their demeanour while giving evidence…..

[49] It is therefore particularly important that, in a case where there are contemporary documents which appear on their face to provide cogent evidence contrary to the conclusion which the judge proposes to reach, he should explain why they are not to be taken at face value or are outweighed by other compelling considerations. It is, however, striking that the judgment in this case contains virtually no analysis of the contemporary documents many of which appear to shed considerable light on the nature and purpose of the critical confirmations and the way in which they were understood.”

(emphasis added)

940.

I have already explained that the subject matter of the Simetra case is very different to this one, and I emphasise here that my analysis of the contemporaneous documents is in respect of the Horizon Issues in the context of this case, not the Simetra case which concerned very different allegations. Here, the categories of documents that are most illuminating in terms of specific incidents with Horizon over the years are the very numerous PEAKs and KELs. These emanate from, and are created within, Fujitsu. They are, in my judgment, a very good means of getting at the truth in this case. They show what was going on and the type of unexplained problems that numerous SPMs were experiencing in practice over the years, as they were reported to the SSC. They contain statements made when Fujitsu personnel’s “guard is down and their true thoughts are plain to see”. Some of them also record that Romec engineers, or the Post Office’s own auditors, have seen what has occurred and ruled out user error. Notwithstanding this, Fujitsu attribute user error to what has occurred.

941.

They also contain some very telling expressions, not in the context of a Fujitsu employee giving evidence as a Post Office witness in this trial, but at the time, with such unguarded comments as Anne Chambers in February 2006 stating that “this problem has been around for years and affects a number of sites most weeks” and “this appears to be a genuine loss” on another occasion (as examples only). These entries are at odds to the publicly stated position by Fujitsu both then and indeed later. In my judgment, the PEAKs and KELs deployed in the Horizon Issues trial, and there were very many, are consistent with my conclusions on the evidence and the answers that I have arrived at to the Horizon Issues. Obviously no account was taken in any of the evidence, and no account is taken by me in this judgment, of the 5,000 KELs that Fujitsu only provided and disclosed in September 2019, months after the trial had ended.

942.

The Post Office’s internal Horizon Online Induction Training slides and accompanying notes dated 7 December 2009 form a useful snapshot of how Legacy Horizon was viewed. The purpose of HNG-X was “to deliver a significant reduction in the total annual cost of ownership of Horizon, whilst ensuring the system remains fit for purpose in the 21st Century.” These materials make it clear that this programme had been identified in 2005 and was "to deliver a significant reduction in the total annual cost of ownership of Horizon". Legacy Horizon was seen within the Post Office as being very expensive, although its overall cost is not relevant to the Horizon Issues. The notes to these materials state:

“However, we need to continue to manage firmly any over-expectations of the frontline that Horizon Online will deliver improved functionality - they may see this as a missed opportunity so will not cure all the issues and problems that users have with Horizon although where practical, and at no extra cost, we have smoothed away a number of "rough edges".

(emphasis added)

There was no significant upgrade or improvement in functionality intended as part of HNG-X.

943.

Another type of document that is illuminating in terms of the Horizon Issues are internal Fujitsu briefing documents, and to a lesser extent also some internal Post Office emails, that show the same approach to, or view of, Horizon within those organisations as are drawn by my conclusions. One example relates to the so-called Ping fix. After this was brought in, the number and value of TCs fell dramatically so far as Camelot/the lottery was concerned. This appears at [427] to [428] of the judgment on the Common Issues, Judgment (No.3). In 2007 there had been the following value of TCs in respect of the lottery: approximately (to 1 decimal point, which represents £100 thousand) £22.8 million; in 2008 there were £12.5 million; in 2009 there were £12.0 million; in 2010 this became £11.3 million; and in 2011 £4.5 million. In 2012, the year the Ping fix was introduced, these fell below £1 million for that year.

944.

The Introduction to a Post Office document of 2011 entitled “Transaction Acknowledgements – End to End Incident Management and Data Flow” explains the following purpose to the Ping fix:

“Introduction

The PING Project was part of the Back Office Efficiency Programme within Post Office Ltd.

The objective of this project was to automate a process that converts Client transactional data files (e.g. Paystation), into a data feed to the branch, i.e. a Transaction Acknowledgement file.

The branch will then process the data electronically on Horizon Online and thus avoid non compliance issues and the resulting cost to the business to process queries as a result.

Please note - The term PING is an internal Post Office Ltd term and branches and helplines will know this new functionality as 'Transaction Acknowledgements'.” (emphasis added)

945.

The compliance issues were caused by the fact that the lottery terminals were on one computer system, namely that operated by Camelot, whereas Horizon was the system operated by the branch that was selling the Camelot products, and there was no electronic transfer of data in to Horizon in respect of Camelot sales. The “objective” in the Post Office document makes it clear that the Ping fix is a required improvement to the functionality, to overcome issues caused by having two different systems which

meant that there was (prior to the Ping fix) no automated process to convert the client transactional data files into the data feed necessary for Horizon for the branch accounts. This has nothing to do with correcting excessive carelessness or fault on the part of SPMs. It is, in my judgment, about remedying a deficiency in the functionality of Horizon. That document also made it clear, because there are express entries to this effect, that outages might mean that the system would not deal with the matters sufficiently or accurately. A template message for SPMs is even included in the document for when this occurs, although that template message would not inform SPMs that outages would lead to inaccuracies. It is plain that outages would indeed lead to this, as this is mentioned in many places in the document, and acknowledged by its authors. This information was not however, based on the templates, to be given to SPMs.

946.

A theme contained within some of the internal documents is an extreme sensitivity (seeming to verge, on occasion, to institutional paranoia) concerning any information that may throw doubt on the reputation of Horizon, or expose it to further scrutiny. One entry in a document that makes it clear that the Post Office itself had already recognised this is contained in a document authored by Mrs Van Den Bogerd, entitled “Extracts from Lessons Learned Log” and dated 11 November 2015. One entry under “issues identified” was as follows in respect of the Post Office’s behaviour up to that date:

" Failure to be open and honest when issues arise eg roll out of Horizon, HNGx migration issues/issues affecting few branches not seemingly publicised."

(emphasis added)

947.

The Post Office’s guard may have been down when that document was written; it predates issue of the first claim form by about 6 months. I consider that it is the type of document that Males LJ had in mind when he spoke of a witness’ guard being down, when their true thoughts are plain to see. It plainly records Mrs Van Den Bogerd’s view of the Post Office’s approach to Horizon when it was written in November 2015.

948.

In July 2016, following the disclosure of the Dalmellington bug, the following emails went to and from very senior people in the Post Office. On 1 July 2016 the Chief Executive, Paula Vennells stated to Alisdair Cameron and Rob Houghton, in the context of an article that had appeared on the internet about problems with Horizon:

“Subject: The Dalmellington Error in Horizon / problemswithpol

Dear both,

This needs looking into please. https://problemswithpol.wordpress.com/2015/11/10/the-error-in- horizon/?iframe=true&theme_preview=true

[This is a reference to an internet post about a conviction of an SPM]

Can I have a report that takes the points in order and explains them.

Tim McCormack is campaigning against PO and Horizon. I had another note from him this am which Tom will forward, so you are both in the loop. We must take him seriously and professionally.

This particular blog is independent of Sparrow but clearly related in that it appears to present similar challenges that were raised in the course of the scheme.

I'm most concerned that we/our suppliers appear to be very lax at handling £24k. And want to know we've rectified all the issues raised, if they happened as Tim explains. Thanks.

Paula”

949.

Mr Houghton of the Post Office sent it on to someone else on the same day, copied to Mrs Van Den Bogerd. This stated:

“I need an urgent review and mini <taskforce> on this one. It probably needs to link up heavily with Angela’s work as FSC are mentioned extensively - Angela cfi. I don't know how we respond to this but can we section a few inside people to get all over it and give me/Al/ Paula evidence and understanding.”

950.

However, the same recipients also received the following message from Mr Houghton later on the very same day:

“Can you stand down on this please? [A redacted section then follows] Any specific actions and I will revert.

My apologies.”

951.

The email chain is heavily redacted and therefore the reasoning behind this volte face within the Post Office is not shown, nor was it explained by any Post Office witness.

952.

Thus the entirely understandable initial reaction of the Chief Executive of the Post Office, that a point by point investigation was required in respect of specific criticisms of Horizon in respect of one particular SPM who blamed Horizon for shortfalls and discrepancies, appears to have been very swiftly shut down and not pursued due to a decision at what must have been very high levels within the Post Office. I can think of no justifiable reason why the Post Office, institutionally, would not want to address the Chief Executive’s points and investigate as she initially intended, and find out for itself the true situation of what had occurred.

953.

Both the recipients of the Chief Executive’s email of 1 July 2016 above are specifically named in the Technology Strategy Update Decision Paper of 30 January 2017 in which the following is stated specifically. Mr Houghton is the Author of the paper, and Mr Al Cameron is “the Sponsor”. Part of that document records:

“This document forms an update to the IT Strategy approved in July 2016 by the PO Board. In July we outlined that IT was not fit for purpose , expensive and difficult to change.”

The same document also states that "We need to quickly rationalise and resolve misaligned contracts enacted to support legacy IT, obsolescence, a lack of PO technical competence, particular focus on Fujitsu and Accenture” .

(emphasis added)

954.

This therefore means that the decision taken not to investigate further in July 2016 and to “stand down”, in itself a surprising decision for a reputable institution to take

given all the circumstances, was taken at broadly the same time as a conclusion was reached that the IT “was not fit for purpose”. The reference to the “lack of PO technical competence” is a concern, given [75] of the Technical Appendix which makes it clear that currently the Post Office can introduce new clients into Horizon without requiring Fujitsu input. The level of competence may of course have changed since 2016. The IT Strategy, which is referred to in this Update Decision Paper, and outlined that “the IT was not fit for purpose” and which was approved in July 2016 by the Post Office Board is, in my judgment, entirely consistent with my conclusions on the evidence. It is not, however, consistent with the Post Office’s response to the claimants’ pre-action letter, or the Defence in the group litigation, or the evidence adduced by the Post Office in the Horizon Issues trial. The Post Office’s solicitors’ response to the pre-action letter is dated 28 July 2016, very lengthy, and states that “the investigations to date have consistently pointed towards human error or dishonest conduct in branches as the most likely cause of shortfalls.”

955.

Nor is it consistent with the way the Post Office explained, in its oral opening submissions, that the Post Office had been investigating incidents earlier in terms of bugs. So far as Dalmellington is concerned, the oral submissions were as follows:

“….even the documents that your Lordship saw this morning with the Dalmellington bug, your Lordship will see the rigour that's applied. There's concern that postmasters aren't given advice that might be incorrect. The rigour associated with that process and the determination of Fujitsu and the other people involved, the other stakeholders, to get to the bottom of what happened is quite striking in my submission.”

956.

Dalmellington was investigated in 2015, and the outcome of that investigation is available to this court. The outcome of that investigation is far from favourable to the Post Office’s position on bugs, errors and defects, and the robustness of Horizon. The oral submission that the Dalmellington bug was investigated rigorously (which it was) does not easily sit with the documents showing that one year later the Post Office specifically decided not to investigate other incidents at all, and to “stand down” those tasked with investigating it. This is notwithstanding that the Chief Executive’s initial (and in my judgment entirely proper, if not the only proper) reaction was that those allegations needed to be looked at “seriously and professionally”, that an investigation was needed, and that an urgent review and mini task force was required. Nor does it sit easily with the way that other bugs that were present in both Legacy Horizon and Horizon Online were investigated by Fujitsu.

957.

Further, a conclusion in terms that the IT is “not fit for purpose” is not something that would have been reached lightly in an IT strategy document, or approved lightly by the Post Office Board. The Post Office Board is a serious level within the organisation. The Board are not likely to be involved in, nor to have brought to their attention, matters that are anything other than serious, considered and fully researched. This conclusion in this internal contemporaneous document has not driven my conclusions on the evidence in the slightest. Rather, it is the other way around. I have reached my conclusions on the evidence independent of contemporaneous references such as this one, but then tested my evidential conclusions against these to see if there is a contradiction. However, there is none, and they are entirely consistent.

958.

There are only very isolated Post Office documents that are inconsistent with my conclusions. I shall deal with only one. On 17 August 2015 the BBC Panorama programme broadcast a programme which contained allegations about Horizon. The contemporaneous response to that by the Post Office in its statement to SPMs started with the following passages, which I shall reproduce. The bold emphasis was present in the original:

“The Post Office wholly rejects extremely serious allegations repeated in BBC’s Panorama programme of 17 August 2015. The allegations are based on partial, selective and misleading information.

The Post Office does not prosecute people for making innocent mistakes and never has There is no evidence that faults with the computer system caused money to go missing at these Post Office branches There is evidence that user actions, including dishonest conduct, were responsible for missing money

959.

The first of those bullets point is for another day, and is not for this court in any event, for the reasons explained at [60] to [66] above. This judgment deals with the detailed background and technical evidence behind the second bullet point. The confidence expressed by the Post Office in 2015, in that second bullet point, in the computer system and its accuracy is not consistent with the evidence I have accepted, the expert evidence on the number of bugs in the Horizon system (both Legacy Horizon and HNG-X) and my findings. This Post Office document was however drafted for public consumption, and this must explain the inconsistency. To the extent that the second bullet point conflicts with my findings in this judgment, that public statement in 2015 by the Post Office is factually incorrect. This is not to make any findings on any individual claim, or on the cause of money said to be missing in the accounts of any particular branch Post Office or individual claimant SPM.

960.

In my judgment, a number of both the Post Office’s own internal documents, and Fujitsu ones too, namely those that were not drafted for public consumption, plainly support my conclusions on the evidence. Further, certain matters that have emerged in the Horizon Issues trial – such as discussions within Fujitsu itself as to whether the Post Office should be told certain detrimental information about the Horizon system, and the Post Office’s own decision at the highest level not to investigate certain matters as recently as 2016 – are of great concern. The Post Office has gone to great lengths over the years, and spent a great deal of time and a huge amount of public money, in defending the performance of Horizon. It is also the case that the Post Office must have been reliant on Fujitsu to a certain degree in terms of being provided with accurate information of a technical nature. That is not only obvious from the evidence, but has also been agreed by the experts in the 3rd Joint Statement. That accuracy from Fujitsu has not always been available, as demonstrated by this judgment.

961.

However, regardless of that – and this judgment does not deal with who, if anyone, at the Post Office knew precisely what about Horizon, and when, as these are not part of the Horizon Issues – in my judgment a number of the specific internal contemporaneous documents, rather than being contrary to my conclusions on the evidence, are entirely consistent with those conclusions. I do not know if the Post

Office Board, who were told in July 2016 that its own IT was not fit for purpose, will be particularly surprised at the findings on the expert evidence in the Bug Table, and the number of admitted bugs. It does not much matter, for the purposes of resolving the Horizon Issues, whether they are surprised or not.

962.

The experts’ agreements in particular have been of great assistance, but everything has been considered. This judgment, together with the Technical Appendix, is of substantial length, and to recite everything that has been deployed by both sides, in terms of evidence, submission and reference to all contemporaneous documents, is neither necessary nor desirable.

963.

It must also be remembered that Horizon as it is today, or at least in the last couple of years since it became HNG-A, is a very different system to earlier times. Legacy Horizon ceased to be used in 2010. Modern Horizon Online as it is today is not the same as the online system that was introduced in 2010; it is different and more robust, on the agreed expert evidence, to the system as it was even when the litigation started with the issue of the first claim form on 11 April 2016. It became HNG-A in 2017, rather than HNG-X, and runs on a completely different Windows platform. The dates of the bugs in the Bug Table are (between them collectively) from 2000 to 2018, although there are far fewer in recent years. Having said that, some – such as Bug 8, Recovery Issues, and Bug 28, Drop and Go, have been experienced up to 2018 and 2017 respectively. The latter is a reference data bug and hence easily fixed. The

Technical Appendix at [421] provides a summary of the different bugs from Legacy Horizon and Horizon Online. It can be seen that there were more in Legacy Horizon than HNG-X in any event, and far fewer in HNG-A. In any case, the cut-off date for claims to be issued in the group litigation is 24 November 2017, as extended by paragraph 15 of the 1st Case Management Order sealed on 27 October 2017.

964.

The Post Office has been very concerned as to the outcome of this litigation on its wider business. It described the group litigation as long ago as its Opening Submissions in November 2018 for the Common Issues trial as representing “an existential threat” to that business. I do not consider that these answers to the Horizon Issues represent a significant threat to the Post Office’s entire business. Findings in this judgment as to the performance and robustness of Legacy Horizon from 2000 to 2010, and then of Horizon Online (in both its forms, HNG-X and HNG-A) from 2010 to 2018 are not findings on the Horizon system as it exists at the date of this judgment. These findings cannot be routinely applied to the way that HNG-A operates as at December 2019. It is agreed by the experts that the Horizon System in its HNGA form is now relatively robust. This judgment is a historical analysis of the Horizon System as it relates to the period in question in the group litigation, not a judgment upon Horizon HNG-A as it is today.

M. Answers to the Horizon Issues

965.

This section is intended as a very brief summary of what has gone before, both in this judgment and also the Technical Appendix. It does not replace the previous 964 paragraphs or that appendix. The detailed passages themselves should be consulted for a more detailed consideration of the issues.

966.

I therefore turn to the answers to the Horizon Issues themselves. For convenience, I shall reproduce the specific issue above each answer, in order to avoid the reader having constantly to turn back some hundreds of pages to remind themselves. I shall also retain the headings and sub-headings. The wording of the Horizon Issues was agreed by the parties and approved by the court.

BUGS, ERRORS AND DEFECTS IN HORIZON

Accuracy and integrity of data

967.

Issue (1): To what extent was it possible or likely for bugs, errors or defects of the nature alleged at §§23 and 24 of the GPOC and referred to in §§49 to 56 of the Generic Defence to have the potential (a) to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions as alleged at §24.1 GPOC?

968.

Answer: It was possible for bugs, errors or defects of the nature alleged by the claimants to have the potential both (a) to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, and also (b) to undermine the reliability of Horizon accurately to process and to record transactions as alleged by the claimants.

969.

Further, all the evidence in the Horizon Issues trial shows not only was there the potential for this to occur, but it actually has happened, and on numerous occasions. This applies both to Legacy Horizon and also Horizon Online. It has happened under both the HNG-X and HNG-A iterations of the Online system, but far less frequently under the latter than the former. Indeed, there are only isolated instances of it happening in respect of HNG-A, which the experts agree is a better system than either of the other two iterations of Horizon.

970.

I accept the claimants’ submissions that, in terms of likelihood, there was a significant and material risk on occasion of branch accounts being affected in the way alleged by the claimants by bugs, errors and defects. This is amply demonstrated by Dr Worden’s evidence. He accepted that there was strong evidence of at least 12 bugs causing a lasting discrepancy in branch accounts. This conclusion was reached even within the context of his unjustified creation of both “transient” and lasting impact, which limited his consideration of this, and is an approach which I have rejected. This number of bugs was very much smaller than the number contended for by Mr Coyne, and the number which I have found in Part E of the Technical Appendix, but even on that smaller number, was itself significant. Dr Worden’s reliance upon TCs as a countermeasure led to a failure by him to follow the agreed wording of the Horizon Issues, as TCs are not part of the Horizon system. Given TCs are part of the way that the Post Office corrects the branch accounts (and branch accounting statements) produced by Horizon, the only way that a fully precise answer could be given for the number of times such impacts have occurred would be if the figures were available for the years of operation of Horizon, demonstrating how many TCs each year had to be issued in respect of the effects on branch accounts of such discrepancies. Figures

would also be required of the number of TCs that were upheld when disputed by SPMs.

971.

However, those figures are simply not available for the majority of the life of both Legacy Horizon and Horizon Online. Nor is there any data centrally recorded by Fujitsu of the number of bugs, errors and defects which were discovered by dissatisfied SPMs reporting incidents to Fujitsu, with percentage outcomes of what the investigations by the SSC led to, in terms of result. Such data as there is, namely Mr Parker’s summation of closure codes of PEAKs, is wholly unreliable and I have rejected it. Nor are the closure codes that were used by Fujitsu accurate in any event. The evidence of fact was that full and accurate records of this nature are simply not kept by Fujitsu. Had they been, both experts could (and probably would) have used such data as a starting point for their investigations. The sheer scale of the number of TCs issued by the Post Office each year – which is over one hundred thousand for many of the years that are the subject of the group litigation – supports my conclusion that there was a significant and material risk of inaccuracy in branch accounts as a result of bugs, errors and defects in the Horizon System (both Legacy Horizon and HNG-X).

972.

Issue (2): Did the Horizon IT system itself alert Subpostmasters of such bugs, errors or defects as described in (1) above and if so how?

973.

Answer: Although the experts were agreed that the extent to which any IT system can automatically alert its users to bugs within the system itself is necessarily limited, and although Horizon has automated checks which would detect certain bugs, they were also agreed that there are types of bugs which would not be detected by such checks. Indeed, the evidence showed that some bugs lay undiscovered in the Horizon system for years. This issue is very easy, therefore, to answer. The correct answer is very short. The answer to Issue 12 is “No, the Horizon system did not alert SPMs”. The second part of the issue does not therefore arise.

974.

Issue (3): To what extent and in what respects is the Horizon System “robust” and extremely unlikely to be the cause of shortfalls in branches?

975.

Answer: This issue or question is in two parts. I shall deal with them sequentially. The first is the extent and in what respects the Horizon system is “robust”. The experts are agreed that HNG-A is relatively robust, and I have found that the system as it is in 2019 is far more robust than it was prior to 2017. The Technical Appendix in general, and [444] of that in particular, should be consulted for a more complete answer on robustness. However, the Horizon Issues considered the whole life of the system from 2000 to date. In summary terms only, Legacy Horizon was not remotely robust. The number, extent and type of impact of the numerous bugs, errors and defects that I have found in Legacy Horizon makes this clear. Further, Dr Worden considered that TCs were an essential countermeasure. If TCs are excluded from consideration of the Horizon system, which they must be, then the answer becomes even more stark, even on the evidence of the Post Office’s own expert.

976.

HNG-X, the first iteration of Horizon Online, was slightly more robust than Legacy Horizon, but still had a significant number of bugs, errors and defects. This was particularly so in the years 2010 to 2015, but also after that. This is clearly demonstrated from the findings in the Technical Appendix. The functionality of

Legacy Horizon was not therefore materially improved when the system moved to the online version, although one of the major differences between Legacy Horizon and HNG-X was the former’s use of Riposte (which was not used at all in HNG-X). I find that the robustness of HNG-X was questionable and did not justify the confidence routinely stated by the Post Office (prior to February 2017) in its accuracy. HNG-A is far more robust than either of the previous two iterations of the system. I have taken into account, in reaching these conclusions, not only the evidence of bugs, errors and defects, but also the lack of records or logs at Fujitsu over time over the use and controls of the powerful access roles permitted to Fujitsu personnel. This deficiency in reporting and controls seems to have been corrected in about 2017. This adds to my conclusion (and that of the experts) that HNG-A is more robust than HNG-X. As the experts agreed, this increase in robustness has, in part, developed from Post Office discovering bugs, errors and defects in live use and then applying fixes and improving monitoring. This is important, as it means that the robustness today of HNG-A cannot be relied upon by the Post Office in justifying, or being equated to, robustness in HNG-X. The same point applies to HNG-X and Legacy Horizon.

977.

I accept Mr Coyne’s evidence that there are far more TCs than expected even now, compared with comparable systems in the banking and finance sectors. However, it may be that this is the only way in which it can now sensibly be operated. TCs are outside the scope of the definition of the Horizon system in any event. The experts are agreed that HNG-A is robust, and far better than it was in either its Legacy Horizon or HNG-X form.

978.

Turning to the second part of the issue, namely to what extent and in what respects is the Horizon system extremely unlikely to be the cause of shortfalls in branches. I accept that this presupposes a shortfall in a branch, and then requires consideration of whether Horizon is extremely unlikely to be cause of that shortfall. By shortfall, this issue means an unexplained discrepancy of the type explained by the claimants’ witnesses, and also recorded in numerous PEAKs. In my judgment, there is a material risk that such a shortfall in a branch’s accounts was caused by the Horizon system during the years when both Legacy Horizon and HNG-X were in use, which is 2000 to 2010 and 2010 to 2017 respectively. For each of those iterations of the system, the Horizon system was therefore not “extremely unlikely” to be the cause of shortfalls in branches, as contended for by the Post Office. I realise that is a double negative, but I consider wording the answers as closely as possible to the agreed issues is likely to be of greater assistance to the parties.

Controls and measures for preventing / fixing bugs and developing the system

979.

Issue (4): To what extent has there been potential for errors in data recorded within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data in Horizon?

980.

Answer: The experts were agreed that there are a number of actual reported errors in data recorded within Horizon arising from each and all of (a) data entry, (b) transfer or (c) processing of data. Therefore, not only does the potential for these errors exist, but this has in fact happened, and this has led to financial discrepancies in branch accounts. The experts were also agreed that bugs, errors and defects identified in relation to Horizon Issue 1 were often relevant to Horizon Issue 4 in that they are ultimately errors arising from the processing of data in Horizon.

981.

The “extent” to which this has occurred has not been fully assessed by the experts. However, Dr Worden considered this issue to be a subset of Horizon Issue 3, in respect of which I have effectively or substantially accepted the claimants’ case, at least so far as Legacy Horizon and HNG-X are concerned.

982.

The experts do not agree as to its ‘extent’ in terms of actual effect, although that is not what the issue asks. It was agreed by the experts that there was evidence within the PEAKs and KELs of bugs/errors/defects within Horizon arising from parts (a), (b) and (c) of this issue that occurred that caused financial discrepancies, as well as some that occurred without causing financial discrepancies. It was also agreed that reference data is critical to the operation of Horizon. I have explained in the Technical Appendix what reference data is, and the important part it plays both in the software and operation of the system. It is agreed by the experts that errors in reference data have led to discrepancies in branch accounts. It has also been agreed by the experts that errors in third-party data have led to discrepancies in branch accounts. The Post Office does not have control over errors in third-party data, but given these have an impact upon branch accounts, unless they are corrected, then those errors will also have an adverse effect upon a SPM’s branch accounts.

983.

I do not consider that the extent of the actual effect can be fully answered in precise mathematical terms on the evidence as it is. This is not least because in order to do so, the experts would have to consider the full data set of KELs, and several thousand KELs were produced by Fujitsu during, and well after, the trial. However, a precise mathematical answer is not required. The answer to the issue as posed is that there is a material risk for errors in data recorded within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data in Horizon in both the Legacy Horizon and HNG-X forms.

984.

Issue (5): How, if at all, does the Horizon system itself compare transaction data recorded by Horizon against transaction data from sources outside of Horizon?

985.

Answer: This is largely automated. It has not however been automated throughout the whole life of Horizon from 2000, as shown by the Ping fix in respect of the Lottery/Camelot. The experts’ agreement on reconciliation states inter alia that “Reconciliation between transactions recorded on Horizon and transactions recorded by Post Office's clients is largely automated. Detected discrepancies were subject to manual corrective fixes and/or the issue of Transaction Corrections/Error Notices to the” SPMs. What this means is that if the data in Horizon was wrong (due to bugs, errors or defects within Horizon) and did not reconcile with third-party data, then “manual corrective fixes” and the issuing of TCs would be undertaken, which would not be automatic.

986.

Some comparison of transaction data from some clients of the Post Office is done within the Horizon system and other client data is compared outside the Horizon system. Again, the evidence did not deal in a comprehensive way with each such method for every one of the Post Office’s many hundreds of clients. Nor do I consider that such evidence would be required properly to answer this issue.

987.

Issue (6): To what extent did measures and/or controls that existed in Horizon prevent, detect, identify, report or reduce to an extremely low level of risk the following:

a.

data entry errors;

b.

data packet or system level errors (including data processing, effecting, and recording the same);

c.

a failure to detect, correct and remedy software coding errors or bugs;

d.

errors in the transmission, replication and storage of transaction record data; and

e.

the data stored in the central data centre not being an accurate record of transactions entered on branch terminals?

988.

Answer: The experts were agreed that there were many measures and controls within Horizon that existed to prevent, detect, identify report or reduce the risk of varying errors. However, the experts were also agreed that whilst Horizon contains measures and controls for detecting system integrity concerns, these automatic mechanisms have been shown to have failed in the past. The experts did not agree as to the ‘extent’ of prevention, detection, identification, reporting or risk reduction that the automatic and manual control measures delivered. There were numerous specific matters explored in some detail in the trial that clearly demonstrated such failures, such as the Callendar Square bug, the Receipts and Payments Mismatch bug, gaps and duplicates in entries in the audit store, keying and mis-keying errors (which were referred to in different PEAKs) and fixes which introduced other issues. The doubling of the foreign currency discrepancy that was put to Mr Godeseth was a specifically stark example, even though Fujitsu – inexplicably in my judgment – chose to attribute this to user error.

989.

I do not consider that the measures and/or controls that existed in Legacy Horizon, or HNG-X prevented, detected, identified, reported or reduced to an extremely low level of risk any of the sub-items at (a) to (e). Indeed, I reject the Post Office’s case that there was such an extremely low level of risk. That is, in my judgment, a wholly complacent and unjustified position that has existed for many years, based on my findings in the Technical Appendix.

OPERATION OF HORIZON

Remote Access

990.

Issue (7): Were Post Office and/or Fujitsu able to access transaction data recorded by Horizon remotely (i.e. not from within a branch)?

991.

Answer: Yes. This answer is substantially agreed. Both the Post Office and Fujitsu could read data remotely. Fujitsu could access all transaction data recorded by

Horizon. The experts agreed that Fujitsu needed remote access for support purposes. Fujistu has direct access to the branch database. In Legacy Horizon, this was the messagestores held on the correspondence servers; in Horizon Online, it is the BRDB. In Horizon Online, Fujitsu can also download logs stored on the counters. The Post Office has access to transaction data which is copied to its Management Information System or MIS. In Horizon Online, the Post Office has read-only access to branch data on the BRDB which includes the levels of cash at a particular branch.

Availability of Information and Report Writing

992.

Issue (8): What transaction data and reporting functions were available through Horizon to Post Office for identifying the occurrence of alleged shortfalls and the causes of alleged shortfalls in branches, including whether they were caused by bugs, errors and/or defects in the Horizon system?

993.

Answer: This issue is substantially agreed. The Post Office had access to data and systems that were not available to SPMs. The role of the Transaction Processing System (or TPS) is to gather the transactions taking place in the branches, and to pass them on both to Horizon’s back-end systems such as APS and DRS. These would also be passed on to other IT systems in the Post Office, such as Credence and a succession of systems based on SAP, which have finally culminated in what is called POLSAP. The Post Office therefore has a range of Horizon reports and its own management information systems (MIS). Dr Worden stated that he expected the MIS to include analytical facilities which were not required by, and so were not available to, Subpostmasters. None of this was controversial by the end of the trial, or in issue between the experts.

994.

The Post Office would also be able to use standard database reporting tools to retrieve and analyse information about branch transactions. It could also obtain audit data as explained in Part K of this judgment. A request had to be made to Fujitsu for this audit data, however. The Post Office could not simply access such audit data itself without recourse to Fujitsu, who had a charging structure in place in respect of such requests.

995.

Importantly in my judgment, the only source of actual key stroke information (in the sense of what buttons had been pressed in the branch) was to be found within the audit data. Credence does not provide this. I also consider that although the Post Office did have access to “the causes of alleged shortfalls in branches, including whether they were caused by bugs, errors and/or defects in the Horizon system”, this had to be obtained through Fujitsu. This should be recorded for completeness. The evidence does not show the Post Office IT department either being capable of investigating directly itself, or if it was capable of doing so, actually undertaking this. All investigations recorded in PEAKs and KELs show that Fujitsu did this. The terms of the experts’ agreement in this respect was that the Post Office was “reliant upon Fujitsu for diagnosis of whether the occurrence of shortfalls was caused by bugs/errors or defects within the Horizon system.” I find that the Post Office plainly was reliant upon Fujitsu in this way.

996.

Issue (9): At all material times, what transaction data and reporting functions (if any) were available through Horizon to Subpostmasters for:

a.

identifying apparent or alleged discrepancies and shortfalls and/or the causes of the same; and

b.

accessing and identifying transactions recorded on Horizon?

997.

Answer: The answer to this Issue is also substantially agreed. The experts agreed that the causes of some types of apparent or alleged discrepancies and shortfalls may be identified from reports or transaction data available to SPMs. Other causes of apparent or alleged discrepancies and shortfalls may be more difficult, or impossible, to identify from reports or transaction data available to the SPMs, because of their

limited knowledge of the complex back-end systems. I would add that the subject matter of every PEAK referred to in the Bug Table, and the sort of problems that initiated the creation of each of those PEAKs, plainly could not be investigated by the SPM or SPMs in question in each case on the material that Horizon made available to those SPMs.

998.

In Judgment No.3 I had identified certain features or characteristics of both the SPMC and the NTC, the contracts under which SPMs were engaged (dependent upon the dates of their engagement). The experts were not aware of these contractual findings when they reached any of their expert agreements, as that judgment was not provided even in draft until just before the Horizon Issues trial started. It was handed down formally at the end of the first week of factual evidence. It is of interest, I consider, that the two experts considered at least the essence of one of the implied terms (although not in contractual terms) in their agreement that “identification of the causes of apparent or alleged discrepancies required the co-operation of PO staff and SPMs”. Co-operation is an important part of any investigation, as it is an important part of the contractual relationship between the Post Office and SPMs.

999.

SPMs did have access to many reports relating to the different aspects of running a branch (such as currency exchange, stock and so on). They also had access to transaction logs that would show individual transactions, and event logs, but these were only available for a few weeks (initially this was for 6 weeks or 42 days, then this changed to 60 days). It was possible for a SPM to print these out on paper till rolls. There was evidence in the Common Issues trial from both claimant SPMs and Post Office witnesses about how unwieldy and non-user friendly this was. Indeed, in that earlier trial one of the Post Office witnesses used the size of the Super Court in the Rolls Building (where both the Common Issues trial and the Horizon Issues trial took place) graphically to demonstrate how long such a till roll would be even just for a day in a busy branch.

1000.

Because the reports and data available to SPMs were so limited, their ability to investigate was itself similarly limited. The expert agreement to which I refer at [998] above makes it clear in IT terms (based on the transaction data and reporting functions available to SPMs) that SPMs simply could not identify apparent or alleged discrepancies and shortfalls, their causes, nor access or properly identify transactions recorded on Horizon, themselves. They required the co-operation of the Post Office.

Access to and/or Editing of Transactions and Branch Accounts

1001.

Issue (10): Whether the Defendant and/or Fujitsu have had the ability/facility to (i) insert, inject, edit or delete transaction data or data in branch accounts; (ii) implement fixes in Horizon that had the potential to affect transaction data or data in branch accounts; or (iii) rebuild branch transaction data:

a.

at all;

b.

without the knowledge of the Subpostmaster in question; and

c.

without the consent of the Subpostmaster in question.

1002.

Answer: I shall answer this by reference firstly to Fujitsu, and then to the Post Office. 1003. Fujitsu: The answer is “yes” to each of Issue 10(i), 10(ii) and 10(iii), and to each of (a), (b) and (c) too. There are nine different questions posed (each of (a), (b) and (c), posed after the answer to each of (i), (ii) and (iii) if that first answer is “yes”) and the answer is “yes” regarding Fujitsu to all nine permutations. The expressions used in the evidence that permitted them to do this included those in the SSC role utilising various tools; those with the APPSUP role; those exercising Privileged User rights; and those exercising Database Administration rights. Mr Godeseth also referred to the “guys in Ireland” who had access to a very “low level”, by which he meant very powerful access (as in, they could go to a very deep or low level within the system). This arose in re-examination and hence was not pursued. There is no way of knowing if these Fujitsu personnel, who had the role that is called “UNIX user”, were included within the descriptions I have already given. If not, then UNIX user should be added to that list.

1004.

The injecting of transactions into branch accounts by Fujitsu, and the visibility of these (as in whether the SPM could tell these had been done in this way, due to the use of a counter position greater than 32) was very controversial prior to the trial. By the time the Fujitsu witnesses were called, they had accepted that what Mr Roll had said was correct, and Mr Godeseth confirmed this would look as though the SPM had done it.

1005.

The Post Office: this could only be done by the Post Office utilising its Global User role. This permitted the Post Office to inject transactions without the knowledge or consent of the SPM. However, it could only be done by someone who was physically present in the branch, as that is the only way Global User could be used by the Post Office. The answer therefore for the Post Office is “yes” only in respect of that part (relating to injecting transactions) of (i)(a) (b) and (c). With that exception, the answer for the Post Office is “no” to the other eight permutations. The qualifying statement has to be made that whoever at the Post Office was exercising Global User rights would have to be physically present in the branch. This would therefore be likely to mean that a SPM or their assistant would at least know someone was in their branch doing something in terms of access to their branch accounts. A description of “Global User” is provided at [674] above. The Post Office’s contended for answer to this Issue was “No” to all nine permutations so far as the Post Office (rather than Fujitsu) is concerned. This is wrong, and ignores the existence of the Global User role.

1006.

The Balancing Transaction Tool, and how many times it was used, occupied a certain amount of time at the trial, as its existence had arisen when the experts were preparing their reports and had led to inter-solicitor correspondence. The majority of the factual evidence on this is dealt with in the section of this judgment dealing with Mr Godeseth’s evidence. This tool enabled Fujitsu specifically to insert a transaction into branch accounts. The one documented occasion of its use was done with the consent of the SPM in question. However, in my judgment, this issue was a sideshow. This is because the design intent in the Low Level Design document in relation to this tool had plainly been overtaken, over time, as the numerous other times it was used showed, based on the messages in fact inserted. Secondly, and more importantly, the very powerful roles identified at [1003] above enable those with such roles to do far more than someone simply using the Tool.

1007.

Issue (11): If they did, did the Horizon system have any permission controls upon the use of the above facility, and did the system maintain a log of such actions and such permission controls?

1008.

Answer: Given the answers to Issue 10 above, this issue does arise and requires answering. There were permission controls, but the roles were very wide and in my judgment they were not controlled. This included but is not limited to the lack of any proper logs. Although at an earlier stage in the trial, the Balancing Transaction tool was focused upon, it emerged that there were other roles and tools available to Fujitsu in addition to the Balancing Transaction tool which enabled Fujitsu to achieve the same result, namely injection and alteration of transaction data. Indeed, not only were these roles (such as APPSUP and the other roles identified in the answer to Issue 10 above) more powerful, but they were widely available and in my judgment effectively unaudited. The whole of SSC had the APPSUP role for many years, and internal Fujitsu documents recorded that they were not supposed to have that role. There were a large number of personnel within SSC. Whilst some Privileged User Logs do exist, these logs did not exist prior to 2009 at all. Prior to July 2015, they only recorded when a user logged on and logged off, with no recording of what they were doing. This is a plainly inadequate record. Even after July 2015, the experts agree that such logs are not a useful source of evidence about remote access, due to their content (more accurately, this should be their lack of content). These deficiencies in the control environment were identified as long ago as before the audit report by E&Y in 2011, such that the deficiencies were recorded as being in existence in 2011. Even so, this was not promptly corrected.

1009.

OCPs do evidence documentation of remote alteration of transactional and other branch data. However, on the evidence available at the trial, these OCPs constitute what could be described as a piecemeal and ad hoc process, and a not insubstantial percentage were issued retrospectively. Further, in many cases there is evidence that the person raising the OCP was the person who was to monitor it. In my judgment, and given the powerful nature of the roles, this method of control is inadequate.

1010.

The view of Dr Worden, the Post Office’s own IT expert, who as I have identified is a man of considerable experience in the IT world stretching over decades, is that the number of Privileged Users and SSC users who can create a Balancing Transaction seems high. The lack of control of these roles, the recording of access, the deficiencies in the logs, and the concerns expressed by E&Y, demonstrate in my judgment an inadequacy of controls. Roles were not properly controlled even after concerns were raised by E&Y in the 2011 Audit, particularly in respect of the APPSUP role.

1011.

The answer therefore is that there were permission controls; they were however inadequate. Logs were maintained but only after 2009. From 2009 to July 2015 they did not record actions, only whether a user had logged on and logged off. Even after July 2015, the experts agree that such logs are not a useful source of evidence about remote access, due to their content (more accurately, this should be their lack of content). In my judgment, this amounts to a deficiency in controls.

1012.

Issue (12): If the Defendant and/or Fujitsu did have such ability, how often was that used, if at all?

1013.

Answer: Due to the nature of Fujitsu’s record keeping, the experts could not provide any confident evidence on this subject of frequency. Mr Coyne established that the APPSUP role was used 2,175 times between 2009 and 2019. Dr Worden confirmed that this was consistent with his review of the Privileged User logs and he equated this to about one person using this role each working day during that period. The Fujitsu witnesses stated this would be in a “support role” but very little detail is available about what this was, or why such regular access of someone with such a powerful role was required. In any event, the Fujitsu witnesses of fact had not reviewed the Privilege User Logs in any detail.

1014.

Indeed, the lack of evidence provided in respect of this issue supports and corroborates my conclusions on Horizon Issue 11 which precedes it. The inability of the Post Office, and/or the experts for both parties, to be able to provide a clear and precise answer to what is, in essence, a very simple question (even though the actual answer might be a large number) directly arises from Fujitsu’s plainly inadequate records. On the evidence from the two experts, the answer should be in approximate terms only and would be about once a day over the whole life of the system, to the extent that any records are available.

1015.

Issue (13): To what extent did use of any such facility have the potential to affect the reliability of the Branches’ accounting positions?

1016.

Answer: The design abilities of the roles to which I have referred was very wide – as a single example only, the experts agreed that anyone with the APPSUP role could pretty much do whatever they wanted. The answer to this issue is therefore that these facilities had the potential to affect the reliability of a SPM’s branch accounts to a material extent. Further, the evidence shows clearly that there were instances when this in fact occurred, which goes wider than the issue posed (which asks about potential) but which I find is also relevant to the Horizon Issues as a whole.

Branch trading statements, making good and disputing shortfalls

1017.

Before I turn to Horizon Issues 14 and 15, there is a point that is relevant to the group litigation as a whole that can conveniently be made at this stage. Firstly, the answers to both of the remaining two Horizon Issues are largely agreed by the experts. This means, almost by definition, that the subject matter of those issues ought not to be controversial.

1018.

However, the fact that a substantial part of Issue 14, in particular, was so controversial for so long is, in my judgment, notable. The Post Office ought to have known how its own system works. It is agreed by the experts that an SPM cannot record a dispute on Horizon at all. Indeed, this point was so very clearly conceded by Mr Godeseth when he was asked about it in cross examination that it was remarkable. He explained that there was no “dispute” button on Horizon, and that this was something expressly and deliberately done by design. Yet there were two substantial points in issue in the Common Issues trial on this very same subject. One was the ability of a SPM to dispute the figures shown on Horizon. The second was whether, in law, a branch trading statement had the effect of what is called a settled account between agent and principal (which would mean that certain legal consequences would flow if it were). Judgment No.3 found in favour of the claimants, and against the Post Office, on both those points. It is therefore more than a considerable surprise to see that, as a matter of evidence in the later Horizon Issues trial, those points should never have been put in issue by the Post Office at all.

1019.

In complex litigation, it is important that points that are not, in reality, contentious, are not put in issue if there is no proper factual basis for doing so. If, as a matter of reality, any particular point is not controversial, putting it in issue will simply increase time and costs. A SPM cannot be criticised for “accepting” an entry in their accounts if they have no choice but to accept that entry. Nor can a branch trading statement be given the status of a freely agreed (and therefore settled) account if it is no such thing. I wish to remind the parties to this litigation that points that should not be in issue ought to be conceded at a sufficiently early stage, such that time and legal costs are not spent in argument on an erroneous factual basis.

1020.

Issue (14): How (if at all) does the Horizon system and its functionality:

a.

enable Subpostmasters to compare the stock and cash in branch against the stock and cash indicated on Horizon?

b.

enable or require Subpostmasters to decide how to deal with, dispute, accept or make good alleged discrepancy by (i) providing his or her own personal funds or (ii) settling centrally?

c.

record and reflect the consequence of raising a dispute on an alleged discrepancy, on Horizon Branch account data and, in particular:

(i)

does raising a dispute with the helpline cause a block to be placed on the value of an alleged shortfall; and

(ii)

is that recorded on the Horizon system as a debt due to Post Office?

d.

enable Subpostmasters to produce (i) Cash Account before 2005 and (ii) Branch Trading Statement after 2005?

e.

enable or require Subpostmasters to continue to trade if they did not complete a Branch Trading Statement; and if so, on what basis and with what consequences on the Horizon system?

1021.

Answer: This is substantially agreed by the experts. The Horizon system can compare cash and stock figures, but it has no facility to record a dispute. The comparison is done by the system comparing its electronic records of what cash and stock is held in a branch, with the corresponding figures inputted by the SPM at the end of a trading period.

1022.

The functionality of the system is as follows. The SPM inputs, at the end of each trading period, the relevant figures for the cash and stock that are physically present. The Horizon system identifies any discrepancy and the system invites the SPM to transfer the amount of that discrepancy into the local suspense account and continue “to roll over”, or to discontinue this operation. Rolling over is required before the next trading period can commence. For discrepancies of less than £150, the SPM has to “make good” that amount, by adding money to the till (if it is negative) or removing it (if positive). For negative sums in excess of £150, the SPM has the choice of making good (by adding the money) or “settling centrally” which basically just means seeking

time from the Post Office to pay. This is extensively explained in the Common Issues judgment, Judgment No.3. This is not done as part of the Horizon system. A slightly different process was available prior to 2005 which was called cash accounting, but the process was broadly the same, although “settling centrally” was not available under cash accounting. Making good would cause the actual cash position to increase by the amount put into the till from the SPM’s own resources. If, say, there was a negative discrepancy of £100 between the actual cash position (the cash physically present in the till), and the derived cash position (shown on Horizon) then making good would lead to the figures being the same. The meaning of “settling centrally” simply means paying the amount to the Post Office over time (or having it deducted by the Post Office from the remuneration paid to the SPM).

1023.

A SPM had to do this in order to “roll over” to the next accounting period. This means that a branch trading statement had to be completed if the SPM wished to open the branch for the next trading period. The branch trading statement had to be done on the figures for cash and stock shown on the Horizon system. There was extensive evidence of fact in both the Common Issues trial and the Horizon Issues trial that this had to be done. Unless it was, the branch could not trade in the next trading period.

1024.

So far as disputed amounts are concerned, an SPM cannot dispute a discrepancy or any figure on Horizon, or record on Horizon that they have raised a dispute. Horizon records figures even when they are not correct, and there is evidence of figures on Horizon being known to be incorrect by the SPM and the Post Office, or both. A dispute is normally raised by a SPM through contacting the helpline, which is a telephone service which was widely referred to in the Horizon Issues trial. There is a finding in Judgment No.3 that by contacting the helpline an SPM was registering a dispute with the Post Office. The helpline and its function are however outside the scope of the Horizon Issues, and the helpline does not form part of the Horizon system.

Transaction Corrections

1025.

Issue (15): How did Horizon process and/or record Transaction Corrections?

1026.

Answer: This too is substantially agreed in terms of technical content, and I have explained in the body of the accompanying judgment, that TCs are outside the definition of the Horizon system.

1027.

The creating of TCs is a manual process. Dr Worden relied upon TCs as underpinning his conclusion on robustness of the Horizon system. The creation of TCs is no part of the Horizon system. Without TCs, which are very numerous, the Post Office system of accounting with its SPMs would be even more problematic that it has been. There have been over 100,000 TCs issued each year since 2006. This is more than 2,000 per week.

1028.

The Post Office has a central accounting function that comes to a decision that an adjustment has to be made to a branch’s accounts, as the branch transaction data (which comes from Horizon) is not consistent with the data the Post Office has received from its clients or suppliers. This adjustment is the result of a human process and is made by way of the issuing of TCs to a particular branch. This is done without

the involvement of SPMs and the decision to do so is taken outside the Horizon system. In the early days of Horizon they were called Error Notices.

1029.

A TC is then sent by the Post Office to the SPM and the SPM processes such TCs by accepting them. By that acceptance, the correction that is the subject of the TC will enter the branch accounts on Horizon. Disputes to TCs are done by the SPM contacting the helpline and again this is outside the Horizon system, which does not record that dispute. If the dispute is upheld (as in, the original TC is accepted by the Post Office as having been wrongly issued) then another TC will be issued to correct it. That subsequent TC will correct the effect of the first TC in the branch accounts when it is accepted by the SPM. The issuing of the subsequent TC is also done outside the Horizon system.

1030.

The Post Office does not have comprehensive records of how many TCs that have been challenged by SPMs have been upheld. I have provided at [859] and [860] above an explanation of part of the 3rd experts’ joint statement in respect of TCs, and the percentage of TCs relating to Santander that the Post Office said were upheld. 77% of these were upheld; which plainly means that 23%, or 1 in 4, of those disputed were wrongly issued. I will also record here for completeness (although they are not TCs) that Transaction Acknowledgments or TAs are also used for making corrections to branch accounts. To use the Lottery and Camelot as an example, the issuing of TAs is now automated following the so-called Ping fix. However, a SPM has no option whether to accept TAs or not; they have to be accepted in branch. This was agreed by a number of the Post Office’s factual witnesses.

Case No: HQ16X01238, HQ17X02637 and HQ17X04248

Neutral Citation Number: [2019] EWHC 3408 (QB)

THE POST OFFICE GROUP LITIGATION

IN THE HIGH COURT OF JUSTICE

QUEENS BENCH DIVISION

Rolls Building Fetter Lane London, EC4A 1NL

Date: 16 December 2019

Before :

THE HONOURABLE MR JUSTICE FRASER

-

- - - - - - - - - - - - - - - - - - - -

Between :

Alan Bates and Others

Claimants

- and -

Post Office Limited

Defendant

-

- - - - - - - - - - - - - - - - - - - -

Technical Appendix to

Judgment (No.6) “Horizon Issues”

-

- - - - - - - - - - - - - - - - - - - -

1.

This is the Technical Appendix to Judgment (No.6) which deals with the Horizon Issues. The purpose in preparing this is to avoid what is already a long judgment being vastly increased in length by including technical IT details within it, which are not necessary for the vast majority of readers to understand.

2.

This appendix has the following structure:

A.

The History of the Horizon System

B.

Structure of Horizon generally

C.

Legacy Horizon and Horizon Online

D.

Submissions and Evidence

E.

Bugs, errors and defects

F.

Conclusions on Technical Issues

3. The contents of this appendix are predominantly drawn from the two expert reports, as well as the factual evidence, but also the parties’ extensive submissions and some of the contemporaneous documents deployed in the Horizon Issues trial. Given the size of the system, and its evolution into Horizon Online from the days of Legacy Horizon, it would have been entirely possible to draft an appendix that dwarfed the length of Judgment (No.6). In order to avoid that, as such a lengthy document would not be proportionate or reasonable, I have summarised certain aspects of the technical evidence and documentation where necessary. There may therefore be some passages in this appendix which, for example, to someone vastly experienced in writing Oracle code, may not read as being 100% technically accurate or which seem to be a superficial summary. However, in my judgment, such lack of full technical accuracy in description or terminology does not affect either the consideration of the evidence, or the conclusions, or my findings on the Horizon Issues.

A. The History of the Horizon System

4.

There was a great amount of agreement between the experts on both the history of Horizon and its architecture, both Legacy Horizon and Horizon Online.

5.

Originally, in the 1990s, there was an intended tri-partite project between the Post Office, the government department responsible for welfare benefits, and ICL. During evidence the second of those two parties was often referred to as the Department of Work and Pensions, or the Benefits Agency. In fact, prior to 2001 the relevant department was the Department of Social Security or DSS (which had been created in 1988 from its predecessor, the Department of Health and Social Security or DHSS). The Department of Work and Pensions was not created until 2001. The different terminology does not matter, other than for the sake of accuracy. At least part of the reason for the project was so that benefits claimants could be paid by using a card, rather than what used to be the case with paper books which a branch Post Office would stamp, remove and exchange for cash.

6.

The tri-partite project was called Pathway. ICL Pathway (the department within ICL formed for the purpose) was awarded a contract for an electronic way of paying benefits, the Benefits Payment Card, in May 1996, and there was a Horizon Pilot that was introduced in a small number of branches in 1996. This tri-partite scheme was abandoned in 1999. Pathway cited “greater than expected complexity” and “…major implications for the degree of difficulty of the project”, quotations included in Mr Coyne’s report, as ultimately leading to the failure of the project. In July 1999 Post Office Counters Ltd (or POCL, the previous name of what is now the Post Office Ltd) and Pathway agreed to utilise the project to automate branch Post Offices. This is what is now called the Horizon System or Legacy Horizon, and it was introduced (the technical term is “rolled out”) from late 1999 onwards.

7.

Anyone with a modern smart phone will know that modern IT technology is marching ahead at an increasingly ferocious pace, and particularly those adept at its use now (including the younger generations) may be surprised that so much of what is now dealt with electronically, used to be done using paper. Prior to the introduction of Horizon, the branch Post Office system used a system of paper-based accounts. A paper-based system was used by SPMs to account to the Post Office for the branch activity, although some SPMs used software packages to assist them in this. Although Horizon is used as an accounting system – it is one of its main uses – there is more to it than that. However, computerised accounting systems have been in use since the 1950s, and Dr Worden explained that accounting was one of the “most mature applications” for computers in business. This means that as a computing application, accounting has been such an application for a long time, even in the late 1990s.

8.

Dr Worden explained the origin of double-entry bookkeeping and the principles that underlie it, and explained the process of manual bookkeeping, even (in footnote form) as far back as the clay tablets of Ur, a Sumerian city-state in ancient Mesopotamia which dates from 3800BC. The cuneiform tablets of Ur, as he pointed out, were not double-entry bookkeeping, which was invented in the 14th century. They were singleentry; an entry would be recorded once. Double-entry bookkeeping means that discrepancies can be discovered much more speedily, and easily, than otherwise. I consider Dr Worden’s explanation of this to be sufficiently concise and helpful that I will reproduce parts of it from one of his appendices:

“12. Because any accounting system is intended to track external reality, and to give the most accurate possible picture of that reality, it is essential from time to time to check the picture of reality, held in the accounting system, against reality itself - the physical assets of the business, its money in cash or banks, its obligations and debts.

13.

However, checking against external reality is (or was) an expensive process. You cannot simply look at the books - you have to get up from your desk, go out into the warehouse and count stock, check your bank balance and count your cash, consult other people, and so on. Because checking against reality was an expensive process, it did not get done very frequently. If the check is only made occasionally, and mistakes are found, the interval of time in which one or more mistakes might have occurred is a long one. The error is more likely to have arisen from multiple mistakes. Looking for several mistakes together is much harder than looking for one mistake; there is no 'telltale number' to look for. The process of 'drilling down' to find the origin of a mistake is difficult and unguided, with few clues.

14.

Double entry bookkeeping changed this. If a bookkeeping mistake is made, that mistake will lead to a discrepancy against external reality, which will eventually be found in the external reality check. But any mistake is most likely to have occurred in one column of figures, without any balancing mistakes on the other columns. So, the mistake will immediately destroy the trial balance. Checking the trial balance is much easier and cheaper than checking against external reality. It can all be done by sitting down at a desk with the books and an abacus or calculator - so it can be done much more frequently. When a discrepancy is found, it is now much easier to drill down and find its cause. For instance, if each entry in the books is dated (as it will be), by just inspecting the books you can find the exact date and nature of the transaction which was not recorded as zero-sum, and which destroyed the balance.

15.

So double entry bookkeeping immediately reduces the cost of keeping accurate and trustworthy accounts. It was an early, and highly effective, form of error repellency - in a time when manual errors of bookkeeping were likely to be frequent. The error repellency guarded against a single point of failure (i.e. a mistake in a single column of figures) by making any such mistake rapidly and obviously visible, in the next trial balance.”

(emphasis added)

9.

A very simple example was given by Dr Worden in his Appendix B, which again I will reproduce. For those with no experience of the pre-decimal system the units of money might be a little vague, but the principle applies whether one expresses the money in pounds or shillings:

“8. Double entry bookkeeping can be understood by starting with the simple case of a trader on a farmer's market, who has some money, and who has some sheep. He can keep a list of his money, and a separate list of his sheep - complete with their names if he wants to. He can track changes to those lists, with entries like '23rd July: sold one sheep - Dolly'. From the changes he can work out his current position, in money or in sheep: 'now I have 17 sheep left'. This is single-entry accounting.

9. Double-entry bookkeeping starts when his list of sheep contains two types of information - the sheep he has, and the price he paid for them; and he also keeps a separate list of his money. He tracks the changes to these lists in a series of dated transactions: '25th July: bought one sheep for 5 shillings'. With that transaction, the sum of his money list goes down by 5 shillings, and the money total of his sheep list goes up by 5 shillings. So, he makes two entries - in those two lists - with money value -5 shillings and +5 shillings, and the total money value of the two lists does not change. Because of the self-consistency of mathematics, however he chooses to do those two sums, the sum of the two sums should not change from day to day. This is his double entry 'trial balance'. If the number does change from any day to the next, he knows he has made a mistake - and he can start to track it down.”

10.

This simple example can be applied to any stock held in any business, in this litigation branch Post Offices, and the sum of money which is used to purchase stock in a branch by a customer. To return to the well-worn example of the book of 1st class stamps used earlier in the litigation, if a SPM sells one book of these to a customer for cash, the cash position of the branch improves by the cost of that book of stamps, paid to the branch by the customer (as of today, £8.40 for a book of 12 1st class stamps) and the stock of books of stamps held in the branch reduces by one. This is a simple example because the stamps are part of the branch’s stock. There are more complex transactions in a branch that do not involve stock actually held within the branch; this

will usually be because the branch offers services and products on behalf of the Post Office’s numerous clients, that are not stock in the traditional sense physically held within that branch.

11.

Dr Worden also explained the advantages of double-entry accounting, which include reduction of fraud and theft. Notwithstanding the differences to which accounting systems are put (or to put it another way, how the information held within such systems is used), such as management accounting (for those responsible for making management decisions about the business in question) or financial accounting (for example, for departments who have to declare profits and account for taxation) both sets of information are subject to audit, which is a word which means review, examination, inspection and appraisal. In lay terms, it is the checking of the information held within the accounts.

12.

The word “audit” is used in different ways, and for different things, in this group litigation. The Post Office employs a number of people called auditors to check on the accounts of branch Post Offices. When Mr Bates, for example, started as a SPM, the Closing Day Audit was conducted on the branch (run by his predecessor) as part of what occurred on what was called Branch Handover Day. This is done so that a SPM on their first day starts with particular stock and cash, identified by the Closing Day Audit, and their branch activity uses that as a starting point. Some SPMs may be suspended, or have their engagements suspended, as a result of what the Post Office (through its auditors) uncovers at other audits, which may be performed at any time (which is entirely understandable, and is included in the terms of the SPMs’ contracts). These audits are done to check the state of the branch accounts, and the cash and stock held in the branch.

13.

The Post Office itself is also audited by its auditors (such as Ernst & Young or E&Y), and the Horizon System is itself audited in the sense of checks being performed that it complies with various standards. Audit is also used as part of the expression “audit data”, further explained in Part K of Judgment (No.6) itself.

14.

The financial accounting functionality of the Post Office was provided by something called SAP, which is an industry leading resource planning and accounting system. From 2004 onwards this was by something called POLFS, and in 2010, with the introduction of Horizon Online, this became POLSAP. POLSAP was created by merging two other systems, POLFS and SAP ADS, the ADS in that latter acronym standing for Administrative Data Services. Horizon provided mainly two things; management accounting functionality, as well as Electronic Point of Sale functions for the branches (also called EPOS). There is, in Horizon Online, overlap between Horizon and POLSAP. Dr Worden stated:

“I understand that the financial accounting functionality for the Post Office was provided by SAP over most of the disputed period (from 2004 by POLFS, a SAP application, which in 2010 merged with SAP ADS to form POLSAP), whereas Horizon provided mainly management accounting functionality, as well as Point of Sale functions for the branches. This dual nature required close integration between Horizon and the SAP systems, so that pictures of financial reality given by the two systems were at all times mutually consistent. There is a large overlap between the information stored in the two systems, which will be described below.”

(emphasis added)

15.

Some computers provide a highly complex computation function. Dr Worden stated, and I agree with him, that the functions of an accounting system include the input, secure storage and output of many types of information, in large volumes (many transactions per day), with rather little computation. By far the most important type of computation that occurs in financial accounting is straightforward summation of numerical quantities such as money and stock, subject to the rules of double entry bookkeeping. For management accounting, some other kinds of computation are needed, such as for forecasting purposes. However, these are not usually computationally demanding or complex. This is not disparaging of those who designed the system, but given the function of Horizon so far as branch Post Offices are concerned, Horizon does not possess (nor does it need to possess) a highly complex computational function in the sense that very difficult and demanding complex mathematical equations have to be processed at high speed, for example. The computers at NASA in the United States that are responsible for space exploration, and control of satellites, will be dealing with far more complex computation in terms of mathematical function; however, rarely would there be 12,000 moon shots underway at once. Horizon deals with a huge number of transactions per day (about 6 million) across many thousands of branches, but the nature of the calculations is essentially arithmetical. This is not to understate the complexity of Horizon both in terms of the scale of the system, the number of products or the number of different clients with whom the Post Office (and hence SPMs in branches, on behalf of the Post Office) deal. Fujitsu, in a document of 14 January 2004, described Horizon as 'Europe's largest non-military IT contract', so Horizon is at the high end of complexity amongst IT systems. Dr Worden stated that it represents “many thousands of man-years effort in development and testing, and its documentation alone more than 100,000 documents”. It can therefore clearly be seen to be undoubtedly highly complex.

16.

However, Dr Worden also said this:

“On the other hand, compared with the IT estates of various large organisations I have worked for (such as the NHS, Barclays Bank, UBS or RBS), the Horizon system is probably no more complex, and in some ways less complex. The banks' IT estates, like Horizon, have a complex corporate back-end and an extensive branch office network. They were developed over a longer time period (30-40 years) using development team sizes similar to or larger than Horizon, often merging together or integrating the IT systems of previously independent organisations. This gave them a degree of legacy complexity, and design compromise, and corporate amnesia, not found in Horizon. There are parts of these IT estates which 'nobody dares touch'.”

17.

This means that simply because Horizon evolved as it has is not an answer to the more controversial of the Horizon Issues. Its origins as a system to accomplish something else, the payment of benefits, its incremental addition of products and clients, and its age, are not of themselves indicative of any particular level of performance.

B. Structure of Horizon Generally

18.

Although computerised accounting systems had been used widely for some decades since about the 1960s, in the 1980s there was what was called a “major step change in their architecture”, when relational databases were invented/developed and used. A relational database is a digital database based on the relational model of data, and was first proposed in 1970 by EF Codd. Mr Codd was an English computer scientist and invented the relational model for database management. This model uses a structure and language consistent with what is called first-order logic, although it is also called other things (such as quantificational logic) and this is used both in mathematics, computer science and also linguistics. Inter alia it allows the use of sentences that contain variables. In the relationship model, all data is represented in terms of tuples, grouped into relations. In relational database theory, a relation is a set of tuples (d1, d2, ..., dn), where each element dj is a member of Dj, which is called a data domain. Each element is termed an attribute value, and an attribute is a name paired with a domain (which is also called a data type, or simply a type). In mathematics a tuple is a finite ordered list or sequence of elements.

19.

A database organised in terms of the relational model is called a relational database. A software system used to manage relational databases is called a relational database management system or RDBMS. Many such systems use Structured Query Language or SQL to query and maintain the database. A famous and widely used RDBMS is Oracle RDBMS, which is sometimes called Oracle Database or even simply Oracle. This is a proprietary multi-model database management system produced and marketed by the Oracle Corporation, a US technology corporation based in California. It was first released in 1979.

20.

The system software therefore used for a relational database is built and supported not by the accounting system developer, but by the supplier of system software, which will be a major company such as Microsoft or Oracle. The accounting software makes calls to the RDBMS to store and retrieve the information. These work (in outline terms) by storing structured information. Unstructured information can be stored in computer systems generally, but this requires very advanced computer programs to understand it. More simple programs, such as accounting systems, store and use structured information. Structured information is (for example) something contained in a spreadsheet, made up of information in different cells in rows and columns. Adding up the value of a list of transactions can be accomplished simply by adding up the figures in a particular column that relates to this information for a number of transactions.

21.

The main use of a relational database is securely to store large volumes of structured information. The way it does so can be understood as having large numbers (tens, hundreds or even more) of different spreadsheets (which are called tables) and which are linked to one another. Two different tables in a database are linked to one another (in a 'relation') when they both have one or more columns with the same meaning and share values in those columns.

22.

Dr Worden explained:

“44. A typical relational database may have tens or hundreds of separate tables; each table may have tens or occasionally hundreds of columns; and a table can have any number of rows, up to millions if necessary.

45.

One relational database can act as a 'server' to one or more application programs, which are performing actions directly visible to their users. The application programs make calls (requests) to the relational database management system, which alters or retrieves the data to fulfil the requests. The core service provided by the relational database is to securely store and retrieve this information, for the application programs, or for others who retrieve information from the database more directly. Both the phrases 'securely store' and 'retrieve' have a lot packed into them.

46.

The phrase 'securely store' implies several guarantees: that once an item of information has been stored in the database, it will not be lost or unintentionally changed; and that it is a part of a consistent collection of information, whose integrity will not be compromised in any way.”

23.

The integrity of information is central to relational databases. Indeed, I consider that the integrity of information could be said, in a way, to be central to the concept of the operation of Horizon, if not accounting altogether.

24.

Dr Worden explained the concept of integrity, and integrity constraints, in the following way:

“47. The idea of integrity of information is central to relational databases. When the structure (tables and columns) of a relational database is first defined (in a relational schema) that schema defines various types of integrity constraint on the data, for instance that:

47.1.

certain columns in tables are mandatory, and will always be given values.

47.2.

There can never be a record in some table (call it table A) unless there is a corresponding record in some other table B. For instance, there can never be a record of an invoice to a customer, unless a record exists for the same customer. The invoice table and the customer table both have a column 'customer id'; for every invoice record, there must be a customer record with the same customer id.

48. The database management system guarantees that these integrity constraints will be true for all time. If any application tries to make a change to the database which would violate an integrity constraint, that change is rejected by the DBMS, and no change is made at all. One change to the database may involve changes to several tables at once, so that after all the changes are made, the integrity constraints are still true; but part way through the changes, the constraints are temporarily untrue. One such package of changes, involving changes to one or more tables, is called a database 'transaction'. The DBMS guarantees that:

48.1.

After any completed transaction, the database will still obey all its integrity constraints.

48.2.

After any completed transaction, the changes made to the database can never be lost - even in the event of hardware failures; there are robust ways to recover the information.

48.3.

If a transaction would violate an integrity constraint, it is not allowed by the DBMS; no changes will be made to any table, so the database will still be in a consistent state, as if the change had never been requested.

48.4.

When one application is making multiple changes to the database in a transaction, so that the database is temporarily in an inconsistent state (when some but not all of the changes have been made), those changes are never visible to other applications or users before they have all been completed. Other applications and users can only ever see a consistent state of the database (either before all the changes, or after all changes), in which all the integrity constraints are true.

49. This set of guarantees, given by the DBMS, is called 'transactional integrity'. It has been built into the fabric of all relational databases and has been relied upon by thousands of applications which use relational databases, since the 1980s. Builders of applications can have a very high degree of confidence that their DBMS will maintain transactional integrity. Builders of accounting systems have relied on that guarantee.”

25.

Relational databases support the SQL language, and small numbers of records can be retrieved from within application programs from a limited number of linked tables, and the records are filtered based on data values. Dr Worden considered that the possibility of there being a bug in Oracle to be “extremely remote”, and he ignored it. As he put it:

“51. SQL can be used directly by end users, without the use of any application program, to selectively retrieve and display records. However, it is more common for the users to use a general report writing tool, which not only retrieves the required records, but formats the information in easy-to-read reports - with appropriate column headers, formatting of fields, grouping of records, computations of sums of groups of records, and so on. Nearly all the reports needed from an accounting system are created using these report writing products, which can be rapidly configured to produce any new kind of report, either one-off or regularly as required.

52. All these capabilities of a relational database have been present in relational database products such as Oracle since the 1980s and have been tested by possibly millions of different applications which use those capabilities and rely on them. So, when I am discussing the issue of bugs in an application such as Horizon, the possibility that these bugs arise from bugs in the underlying DBMS - particularly when it is the world market leader Oracle - is extremely remote, and I shall ignore it.”

26.

This is a similar approach to that adopted by Mr Godeseth when he was giving his evidence about PEAK 0195561, when a problem was reported to the SSC on 4 March 2010. This was where a SPM had tried, on 2 March 2010, to transfer out £4,000 (referred to in the PEAK as 4,000 pds, which means either pounds (plural) or pounds sterling) from an individual stock unit into the shared main stock unit when the system crashed. Mr Godeseth had been looking for a bug at the time of a “red alert” during the early days of Horizon Online, and his evidence about this is included at

[341] to [349] of Judgment (No.6). He referred to one possibility being a bug within

Oracle, which is dealt with at [350] of that judgment, which he said “would be huge” and which he also said “was not the bug that I was looking at for the red alert”.

27.

For the avoidance of doubt, when the experts reach agreements, or where I make any findings, about the presence of bugs, errors and defects in the Horizon System (either Legacy or Online) either in this appendix, or in Judgment No.6, these are not made concerning exactly where within the different programs or applications within the Horizon System such bugs, errors or defects are present, or by apportioning responsibility to any particular software provider. The Horizon System includes Oracle as the RDBMS, but there are a considerable number of other companies involved. Those companies are not represented and that is not what the Horizon Issues trial is about. The experts would sometimes refer to what were called “sub-contracting parties” but it was not clear to me whether they were sub-contracted either to Fujitsu or to the Post Office. Nor is it relevant to the Horizon Issues whether the fault is within a particular layer within the system. The relevant issue is whether there are bugs, errors or defects within Horizon.

28.

The software in Horizon is what is called data-driven. If the accounting application software was written in a way that depended directly on the accounting needs of a client (here, the Post Office, what Dr Worden called a “chart of accounts”), then each company would need different accounting software. This would be very uneconomical (in writing and testing all that software), and this is not the way in which software is developed. The chart of accounts is not 'hard coded' into the application software. The software is written to work with any valid chart of accounts; and the chart of accounts itself is treated as data and stored in the database. In this way, changes in the chart of accounts (which occur from time to time) do not require changes in the accounting software.

“59. This is one example of data-driven software - in which any requirement which might change frequently is encoded as data, rather than software code. The code is written and tested to work with all allowed values of the data, to meet a wide range of requirements by changing the data rather than the code. The modern practice of software engineering is to use data-driven software as far as possible. This is not always as far as one might like; some differences between companies and their requirements are best expressed as differences in code, rather than data”.

29.

Accordingly, the accounting application or applications are built as layered software architecture. This means that the accounting application is built “on top of” the relational database. There will usually be a minimum of three layers in a typical system of this type. Layers are also sometimes called tiers. These are the user interface layer, which presents information to users, and accepts their inputs. This is the layer that a SPM would both observe, and which would react to the actions of the SPM (or their assistants) in the branch such as pressing buttons at the terminal for various transactions. The second layer is the business logic layer, responsible for carrying out business processes, and supporting users in doing so. The third layer is the data layer, responsible for storing and retrieving data. More complex architecture has more than these three layers, and there may even be layers within layers. In the case of Horizon, the user interface also includes a Point of Sale interface which is used in the branches. This interface can be thought of as part of the Horizon Point of Sale application, rather than the Horizon accounting application.

30.

Almost all complex IT applications are designed in this way, in order to isolate different kinds of complexity within different layers, and to reduce the possibility of unwanted interactions between functions in different layers. A typical 'client server' layering structure would include at the very least the three layers identified above (a user interface layer, a business logic layer, and a data layer) but the layering in Legacy Horizon is more complex than this.

31.

The main purpose of defining an architecture in layers is to separate the functionality into parts in the different layers, with well-defined and simple interfaces between the layers. This not only makes each layer easier to design, build and test; but also, if there are errors that are not found in testing, it should be easier to understand and isolate the cause of the errors by inspecting the exchanges between layers. Dr Worden considered layered architecture to be an important countermeasure. It could also be seen as an important part of designing such a system. It is part of modern practice.

Legacy Horizon

32.

The architecture of Legacy Horizon up to 2002 is described in a lengthy document called Technical Environment Description dated 22 October 2002. This document states: 'The system architecture adopted to meet these requirements is not based on conventional client-server models. Nor does it conform to traditional central-system models. It adopts an entirely original and highly innovative four-tier model that effectively merges the qualities of central systems and client server systems'. The diagram that accompanies this in the document actually has five layers rather than four. If two boxes in the diagram which are entitled 'Agents' and 'Correspondence' are counted as one 'Agents' layer, which is what Dr Worden did, there are four layers:

1.

Counter.

2.

Agent.

3.

Host.

4.

External Interface.

33.

The counter layer consists of all hardware and software in the branch. It includes all hardware and software required to support the counter activities required for all products and customer services offered in the branch. This was largely built based on a commercial product, Riposte from Escher Group, a separate company from Fujitsu.

34.

Riposte provided much of the Graphical User Interface (which is the basis of all user input and output at the counter) and provided a mechanism for secure distribution of messages between the branches and the two back-office data centres, which are also referred to in some places (including Dr Worden’s report) as campuses. These were located at Bootle and Wigan. The message distribution passed through the Wide Area Network in the diagram.

35.

The Correspondence Servers handled communication over the network.

36.

The function of the Agent layer was to provide two-way translation of data between the formats used in the counter layer and the network (these formats were described by Attribute Grammars) and the formats used in the Host layer. The agent layer is also responsible for extracting the Audit of all data passing through the Correspondence Servers.

37.

An Attribute Grammar is a description of a message structure which is set out in the same pattern as a tree. Dr Worden used the phrase “a tree-like message structure in terms of its parts and their sub-parts”. In more recent IT systems, tree-like messages are usually sent in XML (Extensible Message Language), with their structure defined in a notation called XML Schema. This is used in parts of Horizon Online. An XML document is a string of what are called characters. These characters are divided into markup and content. What is called the processor analyses the markup and passes structured information to an application. Generally, strings that constitute markup

begin with the character < and end with a >, or they begin with the character & and end with a ; (ie a semi-colon). Strings of characters that are not markup are content.

38.

Because Legacy Horizon was developed before the use of XML became widespread, Attribute Grammars fulfilled this function in Legacy Horizon. This is probably the way that Riposte worked. In Legacy Horizon, Riposte provided the facility for reliable replication of data between the branches and the back-office or data centres/campuses. This means that if certain types of data were created at the branches, Riposte should have ensured (the term used by Dr Worden was “guaranteed”, but I consider that overstates it) that the same data would be available at the data centres/campuses. Dr Worden accepted that “if the underlying network was unreliable, it might take some time for Riposte to deliver this guarantee”. He did not say how long that might be and he was not asked about it.

39.

One of the bugs in the bug table was also called the “Riposte Lock” problem. This will be further addressed in Part E of this appendix. It was a problem with Riposte that was known about at Fujitsu.

40.

The bulk of the back-office functionality was provided in the Host layer. Host applications were typically batch systems, processing data in large batches on a daily basis. A complex daily batch schedule was used to control the sequence and timing of these batch processes, using the Maestro scheduling product (not to be confused with the Maestro software, made freely available by NASA to allow users to look at photographs and the daily progress of the Mars exploration rovers). It was the Host layer (and for most purposes, only the Host layer) which communicated with the IT systems of Post Office client organisations, through the External Interface Gateways. Dr Worden described the three layers of architecture which resided at Wigan and Bootle as “the back-end architecture”, and these were the Agent layer (which included the Correspondence layer), the Host layer, and the External Interface layer.

41.

The following are shown in the above diagram as elements, and some description of them is required.

42.

RDMS is the Reference Data Management System. Reference Data is described further at [52] below. This management system is a dedicated IT application which is needed to manage the reference data, and to distribute it appropriately to branches.

43.

The Data Warehouse consists of one or more databases, whose structure is designed to support flexible and open-ended querying and reporting by the Post Office. Different kinds of information which pass through the host systems are siphoned off into the data warehouse and stored there. This storage is within data structures designed for reporting (which is done by means of querying those structures). Functionality which depends on a data warehouse includes the Management Information System or MIS and other applications such as those which look for unanticipated trends and correlations in data.

44.

MIS is the component built on the data warehouse to provide the Post Office with relevant information. The data warehouse and the MIS are important parts of the Horizon System. Dr Worden stated that “in cases of human error in business processes, operational errors in managing Post Office business on Horizon, or software errors in Horizon, some resulting discrepancy or aberration will be rapidly visible through the MIS. MIS facilities were also used by Fujitsu staff. Many pairs of eyes are inspecting the outputs of the MIS, in hundreds of different reports or spreadsheets. One purpose of this is to ensure the rapid detection and correction of many types of errors. These include software errors”. By “pairs of eyes” Dr Worden was referring to human intervention or observation.

45.

The Transaction Processing System or TPS has as its purpose the 'harvesting' of all types of transaction taking place in the branches, and to pass them on to other IT systems in the Post Office - initially to TIP, and later to POLFS when the system was changed.

46.

Transaction Information Processing or TIP was, as at 2003 (the date of the diagram above) the gateway to all other Post Office data processing, including accounting. After 2004, Post Office accounts were held on an SAP system, which was called POLFS; so TPS passed data to POLFS, rather than to TIP.

47.

The black circles in the diagram denote points, as Dr Worden expresses it, 'at which ownership of data conceptually changes and hence at which audit information is generated'.

48.

Dr Worden quotes the 2002 Technical Environment Description document, what he refers to as the “the 2003 design document” to explain the Host at paragraph 165 of his report. ‘One essential task that can only be carried out at the Host layer is reconciliation. The Host is the only system component that can detect discrepancies between the transactions carried out at the Counter (and hence reported back to Post Office Ltd via TPS), and those that were authorised or expected. It should be in a position to send reconciliation reports back to its Client. These enable the discrepancy with the TPS records to be identified and resolved.’.

49.

He considered that “this reconciliation, carried out in the Host layer, is an essential element within [Legacy] Horizon for detecting and correcting errors made at the counter (robustness measure UEC)”.

50.

Reconciliation and TCs have the effect of detecting and correcting the effects of many possible software errors, as well as human errors. Dr Worden was of the view that if there were any such software errors, these would probably occur with such high frequency, and occur uniformly across all branches, as giving rise to so many TCs, that the Post Office would soon suspect a software error (for instance, seeing the effect repeatedly in some MIS report) and require Fujitsu to correct it. However, this is supposition in my judgment. This is because it depends upon a software error occurring very frequently, and being noticed rapidly. Some accepted errors (for example Callendar Square) lay in the system for 5 years. The evidence of fact advanced by the Post Office itself, which includes that in respect of the admitted or acknowledged bugs, makes it obvious that this supposition is not made out. Further, the detail of the numerous PEAKs show that it is a combination of factors, including the speed at which a SPM performs some tasks, that sometimes led to unexpected events occurring. If something happened at a particular point – one point put by the Post Office to Mr Tank was that an outage occurred at a very precise moment – then an adverse impact would or may occur. That precise combination of circumstances may not occur with the frequency that Dr Worden stated. His supposition was that a bug would have a wide impact across many branches with a high frequency, which would mean that the Post Office would suddenly realise its existence. That is not a valid supposition or assumption given the evidence in this trial.

51.

The Horizon System has certain core components. They include both Reference Data and Transaction Data. The difference between these is as follows.

52.

Reference Data is effectively data about products and operational elements and is a critical element of the Horizon system. Reference Data interfaces with a wide range of Horizon components. Without Reference Data, Horizon would not fully function, nor could Subpostmasters operate their branches. It informs the operation of the Point of Sale system at the counter. This is now managed and operated by ATOS. Many of the business applications in the branches are driven by reference data, and this approach has advantages over hard-coding the different business applications, as that would require changes to the code if the applications were to be changed. It is much more flexible, to manage changes over time and across branches to use the reference data approach. This is because when changes are made, the reference data is changed, but not the code.

53.

The integrity of Reference Data is critical for the correct operation of a variety of systems within the Horizon architecture. The Post Office’s Reference Data Management Centre (RDMC) supports the loading, storage and release of Reference Data within the Horizon system. The Reference Data Distribution Service (RDDS) distributes Reference Data to Post Office branches and other data centre systems. The Post Office Reference Data Team deals with the delivery of Reference Data and verification of operational business change through Reference Data. Horizon counter Reference Data was distributed by the Riposte messaging facility in Legacy Horizon, although this is not used in Horizon Online.

54.

Mr Coyne considers that there were insufficient appropriate change control processes prior to 2017, based on a document he has seen which is dated July 2017 which stated “… we have now aligned that all Reference Data changes go through the appropriate change process”. Mr Coyne takes this to mean that prior to this date, all Reference

Data was not going through the appropriate change process, or that change process

did not exist. In my judgment that is probably reading too much into that particular entry; however and in any event, change control processes require two things in order to work effectively. Firstly, they must exist. Secondly, they must be applied.

55.

Some aspects of how business applications were built on Riposte were explained in the expert evidence, without going into enormous layers of explanation of technical complexity that were unnecessary. These included the following: Zero-sum baskets for customers meant that the overall or net impact of all the services provided for one customer was a sum of money which the customer was required to settle. It was required that the cash or other money produced by the customer should exactly match the cost of services provided; therefore, the whole basket of services and customer settlement had to be what is called “zero-sum”, before the basket could be recorded in the branch (and then, through Riposte replication, later recorded in the back-office systems). The impact of every business application would be fed into the relevant systems at the Post Office, typically overnight. These systems were operated by double entry bookkeeping. The only way to put postings into the accounting system was by double entry, which in turn could only be done for zero-sum baskets. Dr Worden considered this a robustness countermeasure.

56.

Any other actions performed in the branches which had an impact upon the Post Office’s accounts (which would include stock management, cash drawer management, balancing and reconciliation) also had to be zero-sum. These could only be carried out in the branch in packages of updates which were zero-sum, when summed across different Post Office account codes. This was another instance of a robustness countermeasure.

57.

All branch applications had to have what was called transactional integrity. This means that all customer business applications, balancing and reconciliation, cash management and stock management were built so that any zero-sum package of updates from those applications would either succeed completely, or would fail completely. In the latter case, they were supposed to have no impact on branch accounts. This transactional integrity was given the acronym TIN, and was supposed to be enforced by the Riposte infrastructure. It should have been impossible in the event of a failure (such as hardware failure) for a part-completed set of updates to be recorded in the branch and then replicated to the back-office systems. This was necessary to prevent the accounting system from being subjected to non-zero sum updates, which would violate its double entry basis and cause later failures of its trial balances.

58.

The only exception to this principle was that of so-called 'recoverable transactions' - where some irreversible interaction with a Post Office client system took place part way through a transaction - so it could not be undone in the case of a later failure. In these cases, the user on the counter would be guided through a short set of recovery steps, to produce a consistent zero-sum result which reflected what had happened. As a comment made here by me for convenience, it is the case that some of the evidence of fact from SPMs which I have accepted about transactions that gave rise to difficulty in branches occurred when there were outages. It will be seen from consideration of the bugs in the bug table that these were not the only instances when bugs had some impact on branch accounts. However, whether there was a hardware failure or outage, the system was built so that partly completed transactions were not supposed to be capable of entering the accounting system. Dr Worden opined that it

was, as he put it, “of course, possible for the user to make some mistake in these steps, which may have been unfamiliar. In these cases, the mistake would often be detected later by a reconciliation process, which would typically lead to a TC. This robustness measure was a correction of user errors (UEC).” However, that statement in my judgment jumps straight to a conclusion that user error would be to blame for failures in recoverable transactions, and this would “typically lead” to the Post Office issuing a TC. In my judgment, the correct approach is to consider bugs, errors or defects within the Horizon System and whether these would lead to failures in recoverable transactions. This is the correct approach to answering the Horizon Issues.

59.

Many applications were not coded individually but were coded as generic applications which could be configured to run different specific applications by altering reference data, as has already been explained. This is called “data-driven software”. These applications were referred to in Legacy Horizon as 'soft-centred' applications. They have the attraction of adaptability in that the data can be changed without new code having to be written. There were however still some hard-coded applications in Legacy Horizon in any event. Errors could often be corrected more quickly if applications are driven by reference data in this way. Reference data is much more concise and understandable than code, so it is much easier to create it or detect errors in it. Dr Worden considered that errors in the underlying generic code would affect a set of specific applications, and so should be easy to detect. However, this can be seen not to be a statement of universal application. The Callendar Square bug was present for 5 years before it was properly discovered (although its effects were experienced, to use Anne Chambers terms, “most weeks”).

60.

Reference data could be as simple as lists of available products and their prices (which clearly might change frequently), or might be more complex - for instance, to describe the sequence of steps needed to handle some business transaction.

61.

The other important component that can be usefully addressed following Reference Data is Transaction Data. There are four sources of transactions that make up transaction data:

1.

Those manually entered by a user (which will be a SPM or their assistant) in branch at the counter;

2.

Transaction Corrections (TCs) which are produced by the Post Office to be accepted by a user in branch to correct discrepancies in the accounts;

3.

Transaction Acknowledgements (TAs) which are non-counter transactions and typically initiate from somewhere else. This is from another area outside the Horizon System so far as the Lottery is concerned, because the information comes from Camelot relating to the lottery sales a branch has performed. These transactions are typically relayed to Post Office/Fujitsu and need “accepting” into Horizon before forming part of the branch’s transaction data. This is done by means of TAs sent to each branch. The SPM does not have the option to reject them. TAs did not exist prior to the Ping fix; and

4.

Fujitsu inserted transactions. These are injected into branch accounts by Fujitsu. They may be performed in order to ‘balance’ a discrepancy. These do NOT require acceptance by SPMs in the same manner as TCs and TAs.

Horizon Counter

62.

The user in a Post Office branch, the SPM or assistant, will deal with customers and for each transaction press a number of different icons in order to compile that customer’s “basket”. As has been seen, different products from different clients can be provided and sold to the customer in the same basket. These may be paying bills, prepayment for services, acquisition of licences (such as a TV licence), and money and insurance management.

63.

The SPM is not concerned that (for example) some of the money collected from a particular customer may need to go to DVLA (for car tax) or to a utility company (for a gas bill). It is the Post Office that accounts to these clients, and it does so centrally and by means of a back-office activity.

64.

As well as these counter activities, Horizon also needs to support the periodic process of balancing and rollover for each branch. Every branch operates in Trading Periods, which are either four or five weeks (according to a timetable published periodically by the Post Office). At the start of each Trading Period the branch is supposed to be 'in balance'. This means that the physical stock and cash in the branch agrees with the data on stock and cash held in Horizon. Then, during each Trading Period, Horizon records all customer transactions made at the branch, so it records the changes in cash and stock. It also records any replenishments or remittances of cash or stock in the branch. Horizon will (or should) record all changes in cash and stock held at the branch during the Trading Period, and can compute, from the starting amounts and the changes, the expected amounts of cash and stock at the end of the period. This leads to something called the Branch Trading Statement.

65.

At the end of each Trading Period, the SPM counts the physical cash and stock in the branch and compares it with Horizon's data for the same items. This is called 'balancing'. If the numbers are all equal, the branch is in balance and is permitted to 'roll over' to the next period. If the two sets of numbers are not equal, there is a discrepancy between what Horizon considers should be present in the branch (for example the actual cash) and what is physically present (the actual cash held at the branch). This also applies to some stock such as stamps. These discrepancies are at the heart of this group litigation, as the reason for the discrepancies are so controversial. As part of “rolling over”, a SPM with a shortfall or discrepancy has to “make good” or “settle centrally” at the time, as part of bringing that Trading Period to an end. Unless they are prepared to do that, the next branch Trading Period cannot commence. Each of those two acts, making good or settling centrally, involved payment by the SPM to the Post Office; the only difference is the former means payment immediately, and the latter means seeking time to pay.

66.

Horizon also supports the activities of replenishing stock such as stamps, and of replenishing or remitting cash. Accepting a delivery of cash from the Post Office into the branch is called, by factual witnesses on both sides in the litigation, “remming in” that cash. This is probably shorthand for remitting or remittance, but all involved in the case call it “remming”.

67.

Dr Worden used DVLA as an example of how the back-office works. He stated:

“Across the UK in any day, Post Office accepts a large amount of money from customers paying their road fund tax. All this money needs to be paid to DVLA. Therefore, Post Office has a back-office activity - carried out centrally –of summing all these amounts of money and paying DVLA. DVLA knows how much money it expects to receive in this way and checks the amount it expects against the amount calculated by Post Office. This cross-check is an example of reconciliation and supporting it and reflecting its outcomes are central to Horizon.”

68.

The counter included both hardware and software. Dr Worden described the hardware in the branches as “not always reliable” but considered there were sufficient measures built into Legacy Horizon to ensure that hardware failures and communication failures could not adversely affect branch accounts.

69.

Because of the difference in how data was and is stored in Legacy Horizon and Horizon Online respectively, availability of the network was required for the latter but not the former. In Legacy Horizon, sufficient data was held persistently in the branches, such that a branch could continue to trade, and could support most business applications, even if the wide-area network was unavailable. Whenever the network became available again, Riposte data replication should have ensured that the required data became available to the back-office systems. The only applications which could not run in this way were those that required some immediate validation from a client organisation - for instance, withdrawing cash from a bank account. Horizon Online this was no longer the case as the persistent data was all stored remotely in the BRDB. This meant that without a working network, a branch could no longer trade. The network infrastructure (which is another way of saying the reliability of the connections to the branch) by 2010 had made this viable.

70.

The role of the Agent layer was to manage communications and translate data between the representation used in the branches and the network on Riposte, and the representations used in the Host layer. It is the Host layer that takes data from external clients. This is explained in the main design document for Legacy Horizon:

‘The systems at the Host Layer can provide permanent storage for information if required by the application’s business rules. The Host systems can accept data from external Clients, and translate a file-based view of this information into discrete transactions or “messages”. These are then passed to the Counters via the Agent and Correspondence Layers. Similarly, messages received from the Counters are translated back into a file-based view for transmission to the external Clients.’

71.

Each of the different business applications in Legacy Horizon was typically tied only to the different Post Office client in respect of whose business it was concerned. For example, Camelot does not need to communicate with the DVLA. Transactions in respect of them may be transacted in the same basket, but this facility is separate to the applications that deal with Post Office/Camelot business and Post Office/DVLA. Accordingly, each different business application in Legacy Horizon can, in Dr Worden’s words “be regarded as a vertical 'slice' though the diagram and is largely independent of the other slices”. He categorised this as “another example of robustness through architecture” but again, it could be seen as a design element.

72.

Visual Basic, C and C++ programming languages were used in the Legacy Horizon system along with Oracle development tools. Riposte, the product developed and marketed by a company called Escher (and which featured in many PEAKs) was used in Legacy Horizon too.

73.

The Horizon Online counter and BAL (Branch Access Layer) were developed in Java, while the agents and host modules remained in C and C++. C is a general purpose code, an imperative procedural language which was invented by the late Dennis Ritchie. He was also one of the co-creators of UNIX with his colleague Ken Thompson. Neither of these gentlemen are household names in society generally in the same way as, for example, Bill Gates of Microsoft or Steve Jobs of Apple, but are very well known in the computer software world. Both C and UNIX are extremely important software technologies which are the heart of a great many, if not almost all, software products used today. Together Mr Ritchie and Mr Thompson won the Turing Award in 1983, a prize with similar status to a Nobel Prize, which is awarded by the Association for Computing Machinery. All of Java, C and C++ are well known languages that are in widespread use.

74.

Horizon Online was not a completely new system; it built upon, and continued to use, certain elements of Legacy Horizon. The move away from Riposte and Visual Basic to Java brought in more modern development tools and enabled techniques such as object orientation to be used. Object orientation means that modules must be selfcontained, and will communicate via pre-defined and documented interfaces. These interfaces are not dependent on the application’s physical implementation. Third party and commodity products (such as operating systems and device drivers) are simply bought in by Fujitsu. Mr Coyne stated that “Horizon is therefore ultimately a composition of many sub-contracted components”, which I consider to be a fair description.

75.

Applications are developed by suppliers (who supply generic software or products), third parties who supply applications that meet Horizon requirements, Fujitsu developers who develop software to meet specific goals, or the Post Office. If the latter, this is done by configuring standard components. As at the date of the expert reports for the Horizon Issues trial, the architecture is data driven such that the Post Office could introduce new clients without reference to Fujitsu. All of the different applications must however integrate with the other existing applications, as well as any other new applications that are added. The complexity of the system will therefore have increased, year on year, as further products and applications are added. Once an application has been developed, it must be tested and evaluated, and then the process of integration takes place. It will then be released (which means introduced into live) and maintained thereafter.

76.

The Horizon system includes an audit database. This is an accurate record of any activity which can affect the branch accounts. These records are available to be consulted in the event of any discrepancy arising anywhere in Horizon (for instance, due to a bug in some other Horizon application, or some operational error in running a batch process, or a dispute about what data was entered at the counter) but the parties could not agree on whether and when this should be done. The audit database is a robustness measure of a secure kernel (SEK) which also involves redundant data storage (RDS). In Legacy Horizon, all data travelled from the counter through the software application at the branch, through Riposte data replication to the two campuses, then through the Audit Agent to the Audit Store. In Horizon Online, all data travelled from the counter through the software application at the branch, through communications hardware and software to the Branch Access Layer (BAL), into the BRDB and then nightly to the audit store.

77.

The Post Office currently has several hundred client organisations, which shows the diversity of services and products available in a branch. This also makes it obvious that, for most of these clients, the service provided through Post Office will be different in nature from the service provided by the Post Office to its other clients, so some unique software functionality must be provided within Horizon both in the branch (for the counter) and also the back office to support the activities for each particular client. This is a part of what makes Horizon such a large and complex system.

78.

Checks are built into accounting systems. These can be by way of checks on data entry, double entry bookkeeping checks in two or adjacent layers, consistency checks and defensive programming.

79.

As I have already identified in Judgment (No.6), Dr Worden grouped countermeasures including within the Horizon System with other characteristics that are not properly described as that, together with those (such as manual checking by SPMs) that are not properly described in that way.

80.

In a Post Office internal presentation document dated 7 December 2009, the following passages appears in relation to “Horizon - Current State”. I consider it an apt description of Legacy Horizon:

“CURRENT STATE

Before discussing the future development of Horizon, lets remind ourselves of the 'system' we have today

OVERVIEW

Horizon is 13 years old and the design criteria/technology was different and the business needs were different - the demand for on-line working was low, the telecoms were expensive for on-line. Horizon was built to support off-line trading.

Very high security (e.g .. user ID/ access/ screens encryption) this was a requirement from Gov because Horizon was built to support benefits payments.

Horizon is the 2nd most secure system in Europe. The MOD being #1 !

Horizon is a costly system. For example Horizon is a bespoke system that uses a different encryption. Link for instance is unable to decrypt the encryption we have, so we have to decrypt before sending!

Horizon is also a system that is wrapped up in 'barbed wire' - making changes difficult and costly - test everything!

As time has passed and more product have been developed Horizon has evolved - from a technical perspective essentially we have bolted things on the side - we undo the barbed wire stick a bit in then wrap everything back up.

Design was optimised at the time to minimise costs (esp. network) - offline working.” (emphasis added)

81.

The following headline points emerge from this:

1.

Horizon was originally designed to support benefits payments to welfare claimants.

2.

Horizon was already old in 2009. Although there were some major changes made when it became Horizon Online from Legacy Horizon, Horizon Online still used major components of the existing system. It is now somewhat older than it was in 2009.

3.

It was originally built for off-line trading.

4.

It has a relational database at its heart, but very numerous other applications that have been bolted on to it over the years as the Post Office’s business has evolved.

5.

There are a very wide number of other computing companies involved in the evolution of this system, not only in terms of software. Oracle, Escher Group, ICL/Fujitsu, ATOS, Computacenter and many more. It is a bespoke system that uses different encryption to other systems, such as Link.

6.

The complexity of the different interfaces, as a result, is very high.

82.

There have also been a total of some 19,842 release notes (in relation to software changes) in the life of Horizon. This is consistent with each of these notes being a change to the Horizon system.

83.

Fujitsu is the entity that was responsible for Legacy Horizon (putting to one side any contractual nuances or differences between the Post Office and Fujitsu that may have existed, which are no part of this litigation) and it remains the main partner with the Post Office for Horizon Online. However, other suppliers are also now involved. Atos provides first line support via the Service Desk; Computacenter supplies the hardware, and Verizon is responsible for networking (what might also be called the connections between the different parts of the system). The main element within Fujitsu that was referred to in the evidence was SSC, in particular 3rd line support, and “Development”. SSC also serves other Fujitsu customers.

84.

The Post Office Account – headed by Mr Parker – is responsible for developing and maintaining Horizon, deploying it to branches, managing the operation of the system and supporting users. The organisation of the Post Office Account is shown by the following graphic:

85.

After the project to introduce the Payments Benefit Card in 1999 (the tri-partite project to which I have already referred) was abandoned, the Post Office and ICL moved to automate post offices and this was the introduction of what is now called Legacy Horizon. In August 2000 there was the Release of what is called the Horizon Core System, or Core System Release. This introduced Automated Payment Smart cards and APS/TPS reconciliation into branches. Not all branches received installation of Horizon in the same month; that would, I imagine, be impossible in practical terms given the number of branches in the Post Office network. Some dates are included in Judgment No.3 from the Common Issues trial, in relation to those claimants called as witnesses in that other trial. Horizon was installed in Mr Bates’ branch in October 2000; Mrs Stubbs said it was installed in her branch “in the course of 2001” but it must have been earlier than that, as letters in relation to her shortfalls both to, and from, Mr Manning are dated 1 and 2 November 2000. The precise date of introduction of Horizon at each of the branches in the Post Office network is not relevant.

86.

An upgrade of Horizon was performed in February 2001, which is called Maintenance Release M1. The prime purposes of the upgrade were the enhancement of the CSR+ Applications (APS, LFS, EPOSS, EPS, OBCS), enabling of the AP client variable day file transmission, enhancement to Reference Data products and minor changes to TIPAIS Pathway generated CPs to improve operability of the system.

87.

A summary of some of the Releases appears in the chronology milestones for Legacy Horizon below at [96] in the list of chronology milestones.

88.

Horizon included what is called an Electronic Point of Sale System or EPOSS, which refers to the part of Horizon within the branch. The parties referred to this as activity at the counter, or simply “counter”. This term comes from the counter in the branch where a customer is served. In addition to this part of the system, there was another element which the parties (for shorthand purposes in a sense) referred to as the “back

office” function. This refers to the way in which the Post Office reconciles its transactions with its own clients.

89.

EPOSS must allow the counter staff to record that some goods have been provided to a customer, compute the price of those goods, and allow the customer to pay the money required for all their purchased goods, for instance by cash or a credit card. If a customer wants to carry out two or more different activities in one visit to the counter - for instance, to settle a bill and to buy some stamps - Horizon does not require the customer to settle the amount in two separate pieces. Horizon has the concept of a customer carrying out a 'basket' of activities and settling the total amount due for the basket in several ways - by one credit card transaction, by a cheque, by cash, or by a mixture of these.

90.

At the stage when Horizon was introduced to branches – and indeed, this was a fundamental distinguishing feature of Legacy Horizon, compared to Horizon Online – the data was held in branches on terminals. Escher Group, a separate company, provided the Riposte Message Server (which was responsible for storing all the data in the Post Office branches and replicating it to the Data Centres). Terminals had two discs within them, one of which was termed a “mirror disc”, which retained all the data of what occurred on that terminal. The branch would have a hard disc, which would store the data for the branch and this would also be replicated on other counter positions (if a multi-counter branch) or, in the instance of a single counter branch, stored to additional external storage on the counter. It would then be passed on from the counter to the Horizon data centre where it is stored in the CS messagestore.

91.

In the summer of 2000, a ‘proof of concept’ was undertaken to investigate the integration of internet technologies within the then-current Horizon System (which was being/about to be rolled out) to support the delivery of banking transactions. This was called the Network Banking Solution. A primary facet of the Network Banking Solution was the delivery of the banking transactions within the already established Escher WebRiposte environment. The full installation and integration of this was the task of Fujitsu (at the time ICL Pathway). WebRiposte was a message passing technology from Escher Group that extended the functionality of Riposte Message.

92.

IBM were selected for the supply of the Network Banking Engine which was designed to handle the interface between the Horizon System and the agreed Financial Institutions; these are also called Post Office’s clients, although in some documents they are referred to as ‘External Clients’. This was described by Mr Coyne in the following way: “This communication was initially based upon Escher’s Riposte messaging system. This was later supported by the WebRiposte system to accommodate Network Banking changes and the Network Banking Engine supplied by IBM to deal with the increase and change in type of business transacted in the branches. This was subsequently migrated to Horizon Online. Horizon is therefore ultimately a composition of many sub-contracted components.”

93.

It can therefore be seen that Horizon Online, although it includes some major differences (for example where the data is stored; this is no longer in the branches as with Legacy Horizon) built upon and adapted the existing functionality of Legacy Horizon. It was not an entirely new system.

94.

Mr Coyne opines that the introduction of network banking (as well as developments in the EPOSS function) brought with it a more complex enhanced architecture within which further systems to ensure transaction integrity and reconciliation could be imposed. However, he also considered that Horizon “originally stemmed from an inherited system and architecture with an initial, fundamentally different design requirement” (namely the Pathway Benefits Payment Card project). Regardless of that, there is no doubt that Horizon is a highly complex and multi-faceted system, because it has so many different elements and interacts with so many other systems. It has been added to multiple times over the years, as clients are added, new products are introduced, and indeed other sub-contracted parties and software products have become involved. I do not consider that resolution of the Horizon Issues requires an analysis of the reason or reasons why, in historical terms, Horizon may have developed (or had inherent within it) any bugs, errors or defects.

95.

The Horizon System that was initially installed in the branches in 2000 onwards is Legacy Horizon. ICL Pathway was (at least partly) owned by Fujitsu and I use the term Fujitsu to refer to the computer specialists with whom Post Office had contractual relations in respect of Horizon. I therefore make no distinction between the years when ICL Pathway or Fujitsu was or were contractually responsible for Legacy Horizon, or indeed with any of the other computer specialist companies such as Escher Group who provided Riposte. Legacy Horizon involved the installation and use of Horizon terminals within branches, which were used by the staff there (including the SPM) to transact all of the branch’s Post Office business with the branch’s customers. If a customer in a branch wanted to purchase something from the associated retail outlet, they would transact that business at a separate till, not at the Horizon terminal. The National Lottery is an exception to this, as certainly now, it spans both parts of the business of a branch.

96.

Mr Coyne recited some “chronology milestones” for Legacy Horizon, which he explained were not exhaustive but which he considered were milestone dates. Dr Worden also drew attention to changes during the period 2000 to 2010 which he described as significant changes.

97.

This chronology included the following:

1.

Pathway (later Fujitsu Services) awarded contract for Benefits Payment Card May 1996

2.

Horizon Pilot 1996

3.

Pathway cited “greater than expected complexity” and “…major implications for the degree of difficulty of the project” which ultimately lead to failure of the project.

4.

Post Office Counters Ltd and Pathway sign agreement to utilise the project to automate Post Offices July 1999

5.

Horizon rollout 1999 – 2002

6.

Core System Release - This included the introduction of Automated Payment Smart cards and APS/TPS reconciliation. August 2000

7.

Maintenance Release M1 - Prime purposes of the upgrade were the enhancement of the CSR+ Applications (APS, LFS, EPOSS, EPS, OBCS), enabling of the AP client variable day file transmission, enhancement to Reference Data products and minor changes to TIPAIS Pathway generated CPs to improve operability of the system February 2001

8.

S04 Release Additional functionality on the Horizon Pilot outlets to permit the printing of forms approx. July 2001

9.

S06 Release Day D rectification measures - this included a new automatically generated broadcast message to detail when counters at an outlet were offline. This was to be implemented in a staged manner and included a receipts and payments fix June 2001

10.

S10 Release Data centre and counter upgrade introduced unattended reboot facility at the counter September 2001

11.

B11 Release of the first network banking release, changed the version of Tivoli used by the whole estate in approx. December 2001

12.

S11 Release January 2002

13.

B12 Release June 2002

14.

S20 Release September 2002

15.

B13 Release in approx. September 2002

16.

In 2003 there were different changes introduced. Network Banking was introduced at this time, as were the Data Reconciliation Services (DRS) and debit card processing. There were also the S30 and S50 releases which immediately follow this entry.

17.

S30 Mails Application /Escher Mails 3.3 package (1 Feb 2003)

19.

S50 Release October 2003

20.

S60 Release Approx. February 2004

21.

S52 Release March 2004

22.

S70 Release October 2004

23.

S75 Release (containing changes to support the changeover to use of NBX banking agents in approx. October 2004

24.

IMPACT Programme

25.

POLFS (a SAP-IS Retail System) was implemented in 2004

26.

S80 Release Jan 2004 - Aug 2005

27.

BI3 S80 T&T Harvester Agent accommodation was introduced. Mr Coyne thought that this may have been approximately Nov 2004.

28.

POLSAP rationalisation (rationalisation of disparate systems SAPADS and POLFS) between 2007-2009

28.

Pension & Allowance Order Books were replaced by the Post Office Card Account in 2005. The Post Office had always had business relationships with banks such as Girobank.

29.

S90 ‘Bureau Plastic’ accommodation introduced in January 2006

30.

S92 Release March 2006

31.

T10 Release August 2006

32.

T40 Release January 2007

32. AP/ADC was introduced in around 2007/2008.

In around 2010, POLFS and SAP ADS were merged to make POLSAP.

98.

Horizon was migrated to Horizon Online in 2010. Horizon Online is also called HNGX, but now it is on a different platform since its move to Windows 10 and it is now called HNG-A. The letters “NG” stand for “New Generation”. In 2010 there was no sudden change in the range of business applications supported by Old Horizon in the branches. This range of applications has increased continually over the lifetime of Legacy Horizon and Horizon Online. The main reason or rationale for the change to online was to exploit advances in the underlying communication technology, and improvements in its reliability. This meant that it had become possible to store all persistent data at a centre rather than in the branch (on the counters), with the consequence that a branch could only operate when communications were available, but the risk of failed communications was significantly lower in 2010 than it had been 10 to 12 years before. Dr Worden considered that this change mirrored the wider changes across the IT industry, where increased reliability of communications infrastructure meant that applications could be dealt with in this way.

99.

The centralised storage of transaction data allowed the architecture to become simpler in some respects, with simpler management of the branches in the event of hardware failures or replacements and other events. This is because in those cases branch data would not be lost. There was also no dependence on Riposte data replication, which meant that Riposte could be removed entirely. Riposte is therefore not used in Horizon Online. One feature of Horizon which emerges from all the evidence provided is that it had a difficult interface with the systems of the Post Office’s clients, such as Camelot. It is not necessary to go into the technicalities of why that was, and was recognised by the Post Office (in this instance by Ms Van Den Bogerd herself) so far as Camelot was concerned by the introduction of the Ping fix in 2012.

Horizon Online

100.

A document entitled Counter Business Architecture document of 4 August 2017 stated that “The objective of the HNG-X programme is to develop a system with structural and operational characteristics that substantially reduce ongoing support and maintenance costs with respect to the current Horizon system” and also “The overall requirement is that the business capabilities offered by the current system (Horizon) are preserved in the new system (HNG-X). However, a limited number of business capabilities will be revised based on a joint optimisation of business requirements and system properties.”

101.

The fundamental change was that in Horizon Online, no transaction data was held in any persistent form in the branches. The same document explained that storage of transactional data within counters in the branches led to the need for security mechanisms that affected “both the structural complexity and operational performance of the Counter Business Application.”

102.

In Horizon Online, before completion of a basket of customer services, that basket was transmitted and had to be acknowledged by the BRDB and the basket could not complete successfully at the counter until that had happened. This is why Horizon Online could not operate in the branches without working communications. The main difference at the “back-end” was the existence of the BRDB, which was the main persistent store of all transactions for all branches. Many business applications in the back-end were unchanged (and were referred to as 'legacy'), except for the need for them to interface with the BRDB rather than with the previous Agent layer. Other copies of transaction data continued to be stored in those applications.

103.

The previous branch architecture had been based on Riposte, which provided functionality on different levels. These included user interfaces, some business applications, and message storage and replication. Because in Horizon Online, Riposte was completely removed, all elements of the branch software were replaced. A great deal of the Legacy Horizon branch code had been written in Visual Basic, for Horizon Online nearly all the branch software was written in Java. Java is a newer language, and supported more modern programs (or what Dr Worden called modern programming paradigms). This allowed a more modern software architecture in the branch, which did not have to be fitted around the existing architecture of Riposte.

104.

The new architecture is shown in the following diagram:

105.

The top 'Presentation' layer is the one responsible for displaying information to the user and for accepting the inputs. The next 'Interaction' layer provides the foundation or building blocks for this interaction, such as menus. The effect of these two layers was to provide a user interface similar in style to that which had been provided by Riposte in Old Horizon. This therefore meant that the experience of the person, either SPM or assistant, using Horizon would be similar to what it had been in Legacy Horizon, but the system would be using Java technology, rather than Riposte as before.

106.

The 'Business' layer provides the functionality of the different business applications, which is done in what is called an object-oriented fashion. This means that there are several general-purpose software objects (i.e. modular blocks of software) with names such as customer, basket and payment, which represent the behaviour that is required of those in the real world, but is done in a way that can be easily reused in many different business applications. There are therefore certain core design elements used for many different applications.

107.

Business process objects and business data objects are more specialised to support the many business applications. Business process objects support the sequence of steps which make up a business process, and the business data objects hold the necessary data, which is presented at the counter or stored. As has already been noted, this is generally not done by writing completely different software for each business application, and a great many applications are driven by reference data. This data defines the sequence of steps in completing each type of service for a customer. This reference data-driven style of software is common modern practice. The theory is that

new applications can be supported just by adding new reference data, rather than by writing new software. There are different capabilities in the business layer which it is not necessary to list.

108.

The Services layer provides a set of software objects which provide services in support of many business applications. Most of these services deal with organising information and sending it for storage in the BRDB. The facilities for ensuring that each basket is zero-sum before it is sent, and transactional integrity are provided generically in the services layer. These functions do not therefore need to be coded individually in the business objects, and do not have to be built individually into any new business application. The facilities for zero-sum baskets, which is one of the ways of complying with the requirement for double entry bookkeeping, and transactional integrity are provided generically in the services layer.

109.

The Process Engine is a key component of the Services layer, and provides a simplified way for the counter to provide services which involve a sequence of steps. The sequences of steps need not be defined in Java code but are defined in a specialised Process Definition Language (PDL), which is executed by the Process Engine. PDL was developed for Horizon Online by Fujitsu. The use of PDL means that complex sequences of steps are simpler to define and test. This is an example of generic data-driven software.

110.

Some data are stored persistently on the branch hardware (as shown by the discshaped boxes in the diagram) however, these data do not include customer transaction information. These include business process definitions (definitions of sequences of steps in a process), other reference data, data defining reports that can be output in a branch, and other information required to support operations. The reference data is refreshed daily from the data centre. There are services which provide these data to the other layers in forms that are convenient for them to use.

111.

The Services layer of the branch architecture is the only layer which communicates with the data centre, through the communications subsystem. The purpose of this layered approach is to provide each kind of functionality (such as reliable and robust communication with the data centre) in one layer only, and not have to reinvent it for many different business applications. In effect, the services layer in Horizon Online now provides many of the services which were formerly provided by Riposte.

112.

The Services layer also provides interfaces for online services, where it is necessary to contact another non-Post Office IT system in order to provide the service to a branch customer. These online services include banking, the use of credit/debit cards, mobile phone top-ups and other services.

113.

Both the Branch Access Layer (BAL) and the BRDB are completely new elements of the Horizon Online back-end. The principal function of the BAL is to exchange messages with the counter software in the branches. It also has a role in checking that the information is conformant, logging of all exchanges, and recovery from many kinds of error conditions. Because it has to handle more than 25 million transactions per day, the BAL has many design features to ensure high performance (principally by distributing the load in parallel across many machines), as well as robustness - for instance, through reliable and redundant hardware.

114.

The branch database or BRDB is a large, high-performance Oracle database whose main function is to store all customer transactions which originate in any branch. It also has features to ensure high performance and robustness. These are, for example, through the use of transactional integrity and recovery.

115.

There are other notable milestones in my judgment of the Horizon System which are not included in the list above at [96] include the following:

1.

Migration to Horizon Online in 2010.

2.

The discovery of what has become called the Callendar Square bug. This was experienced and reported in that branch in 2005. Mr Godeseth’s evidence was that it had probably been in the Horizon System since 2000. Ms Chambers’ entry in one particular PEAK in February 2006 was that “this problem has been around for years and affects a number of sites most weeks.” The correct place to put it in any chronology, on the basis of the evidence currently before the court, is 2000.

3.

The introduction of what was called the Ping fix, which was in relation to the accounting with Camelot for the National Lottery, which was initiated in 2011 and occurred in 2012. Much evidence was given by Ms Van Den Bogerd, both in the Common Issues trial and the Horizon Issues trial, about the improvement in accuracy of accounting regarding Lottery sales since the Ping fix was introduced.

4.

In June 2014 ATOS started to provide 1st line support. The management and operations with regard to Reference Data has been outsourced from within Post Office control to ATOS since the same date.

5.

The move from HNG-X to HNG-A (which uses Windows 10 rather than Windows NT4). This change to Windows 10 occurred in February 2017 and the roll out was administered by Computacentre.

116.

Having provided what is essentially an overview of the Horizon System, both Legacy and Online, it is convenient to turn to the main battleground between the parties concerning the subject matter of the Horizon Issues trial. This is the presence of bugs, errors and defects within both Legacy Horizon, and also Horizon Online. This can be done most conveniently by retaining the numbering of the different alleged bugs from what was called “The Bug Table” in the 2nd Experts’ Joint Statement.

D. Submissions and Evidence

117.

There were two features of the Horizon Issues trial which were somewhat unusual. This was a lack of detailed evidence adduced by the Post Office in relation to the specific factual matters recorded in PEAKs and KELs in particular, and the way that this situation was sought to be remedied. This second element was by way of points explained or put “on instruction”, and also in submissions. This is referred to in [69] to [71] of the judgment that accompanies this appendix. It is an obvious point that submissions are not evidence, and vice versa.

118.

The first of these, the putting of points “on instruction”, in this case related predominantly to the way that certain matters were put to Mr Coyne in crossexamination to correct evidence of fact that had emerged during the crossexamination of Mr Godeseth. The second arose in the context of the Post Office’s Appendix 2 to its written Closing Submissions. The need for both, or either, arose in my judgment because of the way Dr Worden approached his task in comparison with Mr Coyne. Because he had done what he himself described as a “high level” analysis of Horizon, his written reports did not deal with the detailed contents of so many PEAKs and KELs in the same way that Mr Coyne did. Therefore, when the Post Office came to challenge Mr Coyne’s conclusions in closing submissions, they could not simply meet his evidence with opposing expert evidence of the same type.

119.

Lengthy written opening and closing submissions were provided by both parties, as is usually the case in modern and complicated cases. The Post Office’s Closing Submissions in particular were very lengthy, and included appendices, one of which, Appendix 2 at Section K2, itself ran to some 137 pages. This document attempted to collate material relevant to the Bug Table, in a logical form, dealing with each bug in numerical order. It identified key documents, expert evidence (usually from Mr Coyne), entries in joint statements by the experts, and other sections including “relevant discussions during trial.”

120.

During the oral closing submissions, the claimants identified certain instances in which positive evidential assertions were made in Appendix 2 that were not, it was said, contained in the evidence. Particularly given Appendix 2 was so large, it was simply not possible to resolve these objections during that oral hearing. I therefore directed that the claimants serve a schedule identifying all such passages to which objection was taken, so that a response could be provided by the Post Office. This led to a “clarification” schedule served by the Post Office, which stated in respect of each challenged passage the following:

Whether the passage was based on evidence in the trial, and if so the reference; any relevant documentary evidence; if documentary, whether that document was deployed in the trial and if so when; and/or whether the Post Office relied upon what Mr De Garr Robinson submitted would be a suitable response in some cases, namely common sense.

121.

That clarification document was itself rather lengthy. It contained a further 36 pages, with 175 different entries. Of those entries, 115 were to documents, and in only 28 cases had those documents been referred to at the trial. Further, some of the submissions went rather further that the documents supporting them justified. For example, on 8 Recovery Issues in the Bug Table (which was not accepted as a bug) the Post Office has submitted the following in relation to a necessary software fix:

“It went to the Model Office on 1 June 2010 and Fujitsu have advised that it would have been rolled out to the rest of the HNG-X pilot by mid-June.”

The supporting reference was to a document, PEAK PC0199000. However, the submission “Fujitsu have advised that it would have been rolled out….” reads as though Fujitsu have provided evidence, behind the scenes, and not called at the trial, about what would have or did happen. Further, the supporting entry in the PEAK relied on for the entries for 1 June 2010 and throughout that month are as follows:

“Date:01-Jun-2010 14:05:40 User:John Budworth

Pre-Requisite Reference Data and Data Centre BAL upgrades have been applied to live.

Please one-shot to the distribute and commit to Model Office branch 699010 by 15:30 today.

Please distribute tonight to the 50 branches in Pilot 50 RNT9566.xls attached as evidence.

Date:04-Jun-2010 17:20:16 User:John Budworth

Committed to Pilot 50 RNT9566.xls by MSS on Thursday June 3rd as verbally requested by RM.

Date:14-Jun-2010 11:57:50 User:John Budworth

Please distribute and commit to the balance of the HNGX estate

Date:14-Jun-2010 11:58:11 User:John Budworth

Evidence File Updated - RNT9566.”

122.

This entry reinforces my view that the submission goes a little wider than the document’s contents suggests. However, this is a rare example, and in any event I am entitled to draw an inference (that is, a common sense conclusion) that if the PEAK requests distribution to the entire estate, that did occur. For the most part this type of example is rare, and it is not usually the case that the submissions go further than the underlying documents suggest. I would like to record that I found Appendix 2 very useful; it must have taken an enormous time to produce. There are some omissions within it; for example, Dr Worden’s cross-examination is not always referred to. However, the task of producing this appendix to the judgment has been rather easier as a result of it.

123.

In order to use it however, it was not necessary to make a ruling on each and every one of the 115 different passages challenged by the claimants and the responses to those challenges by the Post Office. I have only referred to the actual material necessary in order for me to come to findings on the Bug Table, although I have considered the full content of Appendix 2 and its accompanying clarification document. I have not given documents that were not referred to at all in the trial the same weight as those that were, and I have not given those that were, the same weight as the actual evidence that was deployed in the trial. I have however considered them all.

E. Bugs, errors and defects and their symptoms

124.

The first three of the Horizon Issues have the heading “accuracy and integrity of data”. Given the purpose of the Horizon System, both Legacy and Online, and the fact that it was the accounting mechanism by which branch accounts were produced in terms of the Branch Trading Statement, the presence of bugs, errors and defects with the potential to cause apparent or alleged discrepancies or shortfalls in branch accounts was an extremely important element of the Horizon Issues trial. This is plainly included within Horizon Issue 1.

125.

The degree to which the Branch Trading Statement accounts did or did not, in law, have the effect of settled accounts between principal and agent was addressed in

Judgment (No.3) on the Common Issues. The majority of the Horizon Issues – and certainly the first three - were all aimed at the same principle, namely (in summary) that of computing and accounting accuracy of the Horizon System in both Legacy and Online form. It was doubtless for this reason that so much of the cross-examination of the two experts dealt with the presence of bugs, errors and defects.

126.

In my judgment, the most convenient approach is to consider each of the alleged bugs, errors and defects in the Experts’ so-called Bug Table in order to come to a judgment on the existence, or otherwise, of each. Horizon Issue 1 requires this, and in addition, Horizon Issue 3 should be answered in the context of whatever the answers are to the different component elements of Horizon Issue 1. This also means Horizon Issue 3 can be considered in the knowledge of how many, if any, of the alleged bugs errors and defects in fact existed in Horizon over time, and what their potential effect was. By the end of the Horizon Issues trial, in reality there was less in dispute between the parties on the different bugs in the Bug Table than appeared at the beginning of that process. This is not to say that there were many concessions, certainly in terms of the existence of bugs, although there were some. However, the expert evidence of both Dr Worden and Mr Coyne, although they differed in their conclusions, was not so far apart on the factual existence of certain bugs, errors and defects. Given the use during the trial of so many PEAKs and KELs, which contained contemporaneous records at Fujitsu of the effect upon SPMs of different features within Horizon, this is not so surprising. Some overarching themes remained, for example the difference between transient and lasting impact of bugs. These are addressed in the body of the judgment itself.

127.

Turning to the 29 different entries in the Bug Table in turn, and in summary terms only (as to reproduce references to all of the documentation, evidence and submissions on each would lead to a very lengthy analysis of each such bug, which in my judgment is disproportionate and not necessary), my findings on these are as follows. Because I am using the same numbering as in the bug table, the following bugs are not dealt with in this appendix in chronological order, but in the order chosen by the experts in their joint statement.

1. Receipts and Payments Mis-match bug.

128.

This bug occurred in 2010. It is a bug which appeared in Horizon Online. It was agreed in the Bug Table as an “acknowledged bug” which had an impact on branch accounts. Dr Worden added in his comments that “Therefore, the extent of this bug is well established, in the GJ analysis.” GJ is Gareth Jenkins. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. In the summary in Appendix 2, the Post Office stated that “In the event, however, this bug resulted in transient impact only.” I have dealt with the concept of transient impact in the main judgment on the Horizon Issues.

129.

The effect of this bug was identified upon the accounts of approximately 60 branches. Dr Worden and Mr Coyne, in the agreed entry in the 2nd Joint Statement, had stated that “the number of distinct bugs, for which the experts have seen strong evidence of the bug causing a lasting discrepancy in branch accounts, is between 12 and 29.” (emphasis added)

130.

Dr Worden had therefore seen “strong evidence of lasting impact” in respect of this bug. This was one of his 12. The same joint statement said it had affected “approximately 60 branches”. Mr Coyne in his report used the more precise number of 62. Of these 62, two branches were affected twice.

131.

The bug related to the process of moving discrepancies into the local suspense account. The majority of incidents are recorded as occurring between August and October 2010. The bug was documented in a report from Mr Gareth Jenkins dated 29 September 2010 where it was stated:

“This has the following consequences: There will be a receipts and payment mismatch corresponding to the value of discrepancies that were lost. Note that if the user doesn't check their final balance report carefully they may be unaware of the issue since there is no explicit message when a receipts and payment mismatch is found on the final balance (the user is only prompted when one is just detected during a trial balance)” (emphasis added)

132.

This issue is reported as causing discrepancies that disappeared at the counter or terminal when the branches followed certain steps, but which persisted or remained within the back-end branch account. It is therefore something which is contrary to the principle of double entry bookkeeping, and should plainly not have occurred. The issue occurred when a branch cancelled the completion of the trading period and then, within the same session, rolled into a new balance or trading period. Because the discrepancy disappeared at the counter, the SPM would not know that the discrepancy existed.

133.

The lack of an explicit message to the user, in my judgment, simply compounds the problem. This bug was acknowledged by the Post Office in its letter of response to the letter of claim sent in relation to the litigation. That letter of response was dated 28 July 2016. I am unaware of any acceptance by the Post Office of this bug prior to that date, but in any event that is not relevant to the substantive issue of answering the Horizon Issues.

134.

The Post Office in its Closing Submissions stated that this bug only occurred if there was “an unusual sequence of events”, namely as follows. If a branch had an unresolved discrepancy, when balancing the stock unit, and if the SPM pressed the Preview or Print button to produce the Trial Balance Report, this would cause the counter to return to the rollover screen. Having checked the Trial Balance Report, if the SPM then pressed the rollover button, Horizon would then ask the SPM whether they would like to transfer the discrepancy to the local suspense account or cancel the rollover.

135.

If they chose to cancel the rollover, this would cause the system to return to the rollover screen, allowing the SPM various choices, including cancelling the rollover process or the choice of again pressing the Preview or Print button to produce the Trial Balance Report. If the SPM then proceeded to rollover to a new Trading Period in the same session the fault would occur. This was a bug in the code that was triggered by the ‘cancel’ button being pressed and this incorrectly caused the discrepancy to be cleared so far as the user in the branch was concerned.

136.

I do not accept that this was a particularly unusual sequence. If there was a discrepancy, I do not see why it is suggested it would be unusual for a SPM to decide to cancel the rollover. This is, in my judgment, something which could be expected to

happen sometimes, and plainly the absence of any message to the user could potentially contribute to this.

137.

The Post Office also, in the main body of its submissions, stated that Fujitsu monitors for receipts and payment mismatches, and investigates such occurrences. However, that does not mean that this bug does not exist; rather to the contrary. The fact that Fujitsu was supposed to watch for it, and then investigate it, does not affect its existence. Further, the submissions also relied upon the fact that instances of this bug “were collected through [Fujitsu] reports without the need for an SPM to report an issue.” In my judgment this overstates the factual position in the Post Office’s favour. PEAK PC0204263 dated 16 September 2010 shows that this PEAK was raised by a “customer call” on 13 September 2010, which means a SPM, who had a discrepancy of £109. This was given call priority B by Fujitsu. The SPM did report the discrepancy; they could not report a payments and receipts mismatch because they would not have known either about the existence of the bug, or its effects.

138.

That this was a problem that required a code fix is shown by the following entry in the PEAK dated 24 September 2010.

“HNGX CODE FIX

-------------

FIX DESCRIPTION

Described Above

PROPOSED BRANCH

TBD

COUNTER JAVA FILES CHANGED

None.

COUNTER PDL FILES CHANGED

RolloverStockUnitBLO.pdl updated COUNTER REFDATA FILES CHANGED

None.

SHARED CODE FILES CHANGED

None.

BAL JAVA CODE FILES CHANGED

None.

SQL FILES CHANGED

None.

OTHER FILES CHANGED

None.

APPROPRIATE CODE COMMENTS

YES

DEPENDENCIES

None.

RELATED PROBLEMS

None

UNIT TESTING EVIDENCE screenshots attached.

REGRESSION TEST CLASS NA.

BACKWARDS COMPATIBILITY

its a reference data change hence backward compatibility test is required on LSt

DEVELOPMENT DOCUMENTATION

None.

REQUIREMENTS DOCUMENTATION

None.

HELP

None.”

139.

The Counter PDL files were changed, and as this was a reference data change, certain checks were needed which is called a backwards compatibility test. This is to ensure that the fix itself does not cause other problems. This led, eventually on 5 October 2010, to the entry in the PEAK of “Product Error Fixed” after a patch had been released. A patch is a change, or set of changes, to a programme or its supporting data, and it is something that changes the operation of the software. The entry of 14 October 2010 states:

“Ths Reference Data hot fix for this PEAK has completed LST testing and has been passed to the Reference Data Team for live deployment.

Note this hot fix will only work when a counter receives the 02.12 counter upgrade that is currently in live pilot. (Overnight package COUNTER_X0212 56_1 or immediate COUNTER_APP 56_1).

[End of Response]

Response code to call type L as Category 71 -- Final -- Fix Released to Call Logger Routing to Call Logger following Final Progress update.

Service Response was delivered to Consumer.”

140.

I find that this is a bug with potential lasting impact, and indeed it did cause actual impact. The fact that the Post Office, once it was discovered that it was a bug, corrected the accounts affected by issuing TCs (and in one instance Fujitsu injected the necessary correction into the branch accounts) does not reduce its importance. A reference data fix was necessary to correct it. As explained above, reference data was part of the architecture in Horizon, and in my judgment is part of the software. The fact that aspects of the system such as the SQL files or the BAL Java code was not changed does not mean that this was not a software defect. Even though the underlying code was not re-written or corrected does not minimise the impact of this

bug which required a change to the reference data as shown in the PEAK above, and other of the contemporaneous Fujitsu documents.

2. Callendar Square/Falkirk bug.

141.

It is agreed that this bug occurred between the years of 2000 and 2006, although there is an issue about when it stopped which I deal with at [149] below. It is a bug which was present in Legacy Horizon. It too was agreed by the Post Office in the letter of response in the litigation of 28 July 2016. It was also agreed in the Bug Table as an “acknowledged bug” which had an impact on branch accounts. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had “transient impact”. Dr Worden added different comments, including that “the bug arose from a fault in the underlying Riposte software, so it is not surprising that it took Fujitsu some time to understand it, or that they had to rely on the suppliers to fix it. It does not show poor system design or support by Fujitsu”. That latter sentence is in my judgment a little surprising. It is not relevant to the Horizon Issues whether fault or defects can be laid at the door of Fujitsu, Escher Group (who supplied Riposte) or any of the other specialist companies who provided software for different parts of Horizon. Dr Worden seemed to me to be very defensive on Fujitsu’s behalf. The Horizon Issues are about the operability and functionality of Horizon, and there are no findings made in the judgment or this Technical Appendix regarding which specialist company is to blame for the presence of any bugs, errors or defects.

142.

Riposte software was part of Legacy Horizon, although it is not used in Horizon Online, as one of the main changes between the two different types of system (Legacy, and Online) was specifically the way data was stored and messages sent. Legacy Horizon used Riposte and it was a central feature of that system. Riposte is not used in Horizon Online. The fact that Riposte is a product designed and supplied by Escher, and not Fujitsu, is not relevant to the dispute between the Post Office and the claimants. In any event, and as accepted and recited in paragraph 2 of the Post Office’s Appendix 2 to its closing submissions dealing with this bug, “Mr Coyne asserts that Bug 2: Callendar Square is a bug with lasting financial impact and in JS2, Dr Worden appears to agree that there is strong evidence of this….”

143.

In my judgment Dr Worden does more than “appear to agree”. Given the entry in the experts’ joint statement, he does in fact agree. This bug occurred when a SPM was transferring cash between different “stock units”, which means positions.

144.

One of the Post Office’s submissions on this accepted bug merits quotation in full. The Post Office in its Appendix 2 stated:

“Fujitsu was aware of the wider Ripsote issue for quite some time. Its various manifestations were being identified and resolved long before the Callendar Square PEAK was raised.”

145.

This is undoubtedly correct; Fujitsu was aware of the bug “for quite some time”. I do not consider that this assists the Post Office’s position on the Horizon Issues. The knowledge Fujitsu had about issues with Riposte was not shared with SPMs, so far as I can tell. This is also the bug that a Fujitsu employee, Anne Chambers, stated in an email in February 2006:

“Haven't looked at the recent evidence, but I know in the past this site had hit this Riposte lock problem 2 or 3 times within a few weeks. This problem has been around for years and affects a number of sites most weeks, and finally Escher say they have done something about it.”

146.

The Post Office stated in its submissions that “Fujitsu’s automatic reports identified instances of the Callendar Square bug without the need for each SPM to call in with a report”. That submission ignores the numerous examples in the numerous PEAKs where SPMs did in fact call in with a report of a discrepancy. The Post Office also, entirely incorrectly in my judgment, complains in its Closing Submissions that it did not have the time to cross-examine Mr Coyne fully on this bug. It submitted the following:

“Mr Coyne notes from the documents that the Callendar Square bug was “operating and resident in the system for years without any comprehensive linkage being observed by Fujitsu”. However, this provides only an incomplete picture, as Post Office would have demonstrated if it had had the time it would have needed to cross examine him on this bug (that time was taken by Mr Coyne’s extraordinary changes in evidence on the afternoon of Day 17, the final afternoon of his cross examination).”

147.

I reject this submission for two reasons. Firstly, it was a matter for the Post Office’s legal team how long it spent on each subject with Mr Coyne, who was crossexamined for four days. It might have been sensible to have started with the number of bugs for which Mr Coyne was contending, and then cross-examined on each or some in detail after that. The Post Office chose to leave the total number of bugs until the last day; there is nothing wrong with that, but it is not correct to state that there was insufficient time to cross-examine on this bug (or any bug) and try to blame Mr Coyne for it. Secondly, the submission entirely ignores the evidence of Mr Godeseth, called by the Post Office, who expressly accepted that the bug had probably been present since 2000.

148.

The Post Office also submitted that “a Peak would be raised and the SPM made good in the ordinary course” if this bug affected branch accounts. There are two points that this submission fails to address. Firstly, the Horizon Issues concern bugs, errors and defects with the potential to affect branch accounts. The Callendar Square bug is clearly one of these. Secondly, the “ordinary course” in this instance seems to be that not one of the SPMs across the Post Office branch network were told about the existence or effect of the Riposte lock problem, which Fujitsu recorded (by Anne Chambers in February 2006) had “been around for years”. In my judgment, the ordinary course ought to have been providing SPMs with notification of the existence of this bug and its effects. SPMs and the Post Office have agreements in place (at the time of this bug the SPMC) that govern their legal relationship. Judgment No.3 makes clear what the scope of the parties’ legal obligations are.

149.

One specific factual point of disagreement between the parties was the period of time when the effects of this bug were no longer experienced. The Post Office maintains this came to an end in 2006, and the claimants submitted that it ran until Riposte was no longer used, namely 2010 when Horizon Online came into being. One passage of re-examination of Dr Worden is pertinent in this respect:

“Q. You say another symptom of the Riposte problem. Perhaps you could explain a little bit what you mean by that?

A. Well I did explain and I will explain again. That there is a problem in Riposte which leads to a failure to replicate -- failures to release a lock I think -- and then on certain occasions that leads to a short-term failure to replicate. On other occasions there is a so-called event storm during which there is a longer term failure to replicate and during these failures to replicate all sorts of different things may happen, for instance they double transfer into a stock (inaudible), a precise Callendar Square phenomenon; whereas other things such as system freezes, I can’t remember all the other details, but there are many other symptoms of then underlying Riposte problem and they have been noted over this whole period.

(emphasis added)

150.

By “this whole period” Dr Worden was, in my judgment, clearly referring to the period 2000 to 2010. Whether the short-term failure to replicate, and the “event storm” with its longer term failure to replicate, are seen as one bug in Riposte (which is the way the experts approached it) or two sub-elements of one bug, which it could be (given one is a short term failure, and the other a long term failure, both being failures to replicate) does not for these purposes matter. In my judgment, the period when the effects of this occurred are 2000 to 2010. Dr Worden’s evidence does not support the Post Office’s position that this ended in 2006, and is indeed to the contrary.

151.

I find that this is a bug with potential lasting impact, and indeed it did cause actual impact.

3. Suspense Account bug.

152.

This is a bug in Horizon Online and its identified years of effect are 2010 to 2013. It was also agreed in the Bug Table as an “acknowledged bug”. This bug was the third of the existing bugs acknowledged by the Post Office in its Letter of Response on 28 July 2016.

153.

It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. Dr Worden’s comments do not include that it had an agreed impact upon branch accounts. This is because of the concept he introduced (expanded in his reports) of “transient effect”. What this meant was that if there was an impact on branch accounts by something within Horizon, but that was then corrected by a TC, he concluded it had a “transient effect” upon branch accounts and dealt with it differently. I address this concept in the substantive judgment under “lasting and transient effect”. Other comments of his in relation to this acknowledged bug are:

“It was a transient effect arising not from a fault in the software, but from a change in database archiving policy in 2010. The delay in correcting it arose from a failure of communication between PO and Fujitsu. Because the bug would only manifest itself annually for any affected branch, the effects of this delay were not widespread.”

Peak PC0223870 shows that Fujitsu were able to identify the branches affected, even when Subpostmasters did not report it. There is evidence that the branches were compensated, as I would expect from the normal error correction processes.”

154.

The Post Office accepts in its Closing Submissions that Fujitsu identified affected branches by use of “historical data”, in other words, it had to research it as an investigation was required. Mr Gareth Jenkins in 2013 prepared a note entitled “Local Suspense Problem” which identified that, at that stage, 14 branches had been affected. He stated:

“The root cause of the problem was that under some specific, rare circumstances some temporary data used in calculating the Local Suspense was not deleted when it should have been, and so was erroneously re-used a year later. When the SPMR was asked to clear Local Suspense the actual (ie incorrect) amount was recorded in the Audit Trail. This means that there was no corruption in the audit trail and it accurately reflects the transactions that occurred in the Branch.

If the BTS from the previous period was taken to provide a set of Opening Balances and all transactions that were logged to the audit trail during the period were taken as adjustments, then this would show the correct value that should be in the Local Suspense account.”

155.

He also stated, in a passage that is of interest in terms of the dispute between the parties about the use of Audit Data or management data such as Credence (the Post Office maintains that the latter are sufficient, the claimants insist the former are the relevant accurate records) the following:

“As well as passing these Local Suspense transactions to the normal accounting tables that are used to update POL SAP and Credence, they are also written to a table in the Branch Database that is used to support the printing of the Branch Trading Statement (BTS) after that Branch has been fully Balanced.”

156.

This demonstrates, in my judgment, that in this instance Credence would have been wrong. Further detail of the problem explained by Mr Jenkins in the same report was:

“In April 2011 a problem was found with the archiving strategy related to Stock Units that have been deleted in a Branch. A consequence of this is that some changes were made to the archiving strategy on 3rd July 2011. An unintended consequence of this change was that any Branch that deleted a Stock Unit at the end of 2010 which had local suspense transaction in that Stock Unit before it was deleted were left in the table used for constructing the BTS. This meant that as Trading Periods cycle around each year, these BTS records became visible in 2011 when the same Trading Period was reached.

The effect of these old records was that after the BTS was produced an incorrect figure was generated for the Opening Balance of the Local Suspense Account for the following period. This amount corresponded to the value of the historical record.

These orphaned records were created between 16th November 2010 and 9th December 2010.”

157.

This meant that the figures for the preceding year were effectively recycled as correct figures for the subsequent period. The Note prepared by Mr Jenkins makes it clear that, despite how the Post Office sought to present this in their submissions, this problem persisted.

158.

The document states: “This problem was not reported to Fujitsu in 2011/12 and only affected a small number of Branches and only for a single Trading Period. However the two branches with the largest discrepancies did report the issue to Post Office Ltd who could see the impact of the problem in their back end system and wrote off the loss or gain for the branch but did not ask Fujitsu to investigate further. At the same Trading Period in 2012/13, the problem re-occurred and this time one of the affected Branches reported the problem to Fujitsu on 25th February 2013 (Peak 223870) resulting in a detailed analysis of this issue and finding the orphaned BTS records. The root cause was determined by 28th February 2013 and a preliminary report was sent to Post Office Ltd. A further update was sent on 14th March 2013 with a full analysis of the issue and all the affected branches.”

(emphasis added)

159.

It is somewhat disingenuous, in my judgment, for Mr Jenkins in this note effectively to blame others (which appear to include SPMs and the Post Office), rather than Fujitsu, for this problem not becoming known about until some years later. The trial did not explore communications between the Post Office and Fujitsu on this subject in any great detail, but SPMs are not left with a choice of whether to report something to the Post Office, or whether to report it to Fujitsu. One branch had a loss of approximately £9,800 (the exact sum was £9,799.88); some were £161 or less, and another had a gain of £3,100.

160.

Mr Coyne’s view is that this bug could have affected branches prior to Fujitsu’s investigations. He also suggests that it is unlikely that Post Office and/or Fujitsu have captured its full effects. My findings on that are as follows. Fujitsu appear to have conducted a thorough investigation in 2014, and there are spreadsheets showing the branches identified as having been affected. A code fix was produced in 2013, and this was first piloted to 50 branches on 21 October 2013 and deployed to all branches on 29 October 2013. The bug could not have arisen prior to 2010 in any event as that was the date that Horizon Online was initiated.

161.

I do not know if any of the claimants in this litigation have losses which, when their claims are fully tried, they claim have arisen in the relevant period and as a result of this type of incident, but whose branches do not appear in the Fujitsu list. If so, that will have to be addressed at future trials. However, there is no evidence before the court currently of branches other than those included in the spreadsheet having been affected in the relevant period.

162.

I find that this is a bug with potential lasting impact, and indeed it did cause actual impact.

4. Dalmellington bug/Branch Outreach Issue.

163.

This is a Horizon Online bug. It was not expressly acknowledged as a bug in the Bug Table, but is effectively accepted by the Post Office as a bug, as it appears under the heading in its closing submissions as one of “the following bugs had transient impact.” It is therefore accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but this is subject to the point regarding “transient impact”. For some time during the trial the Post Office would refer to this as “an issue”, and avoid use of the word “bug”. I find that it is a bug – that much does not seem to be controversial, as paragraph 2 of the detailed part of Appendix 2 relating to this bug states the following:

“This is a bug which Mr Coyne states has lasting impact on branch accounts. Post Office submits that there was no lasting impact on branch accounts.”

164.

It is therefore the nature of its impact on branch accounts that is now in issue. Its effects were experienced between 2010 and 2015, although it is named after the branch where its occurrence led to its discovery as a bug in 2015. Fujitsu performed an investigation as a result of the impact of this bug in 2015 at the Dalmellington branch. The investigation shows the following:

Feb 2010 to Jan 2011 65 incidents

2011

6 incidents

2012

9 incidents

2013

7 incidents

2014

9 incidents

2015

16 incidents

165.

The same document showed fixes applied in April 2010, January 2011 and January 2016. It also showed only one call to SSC at Fujitsu in 2015, and none for any of the years 2010 to 2014 when the figures above show there were 93 occurrences of this bug.

166.

It is what is called a “cash remming error” and happened in certain circumstances. It was explained both by Mr Coyne, and further by Mr Godeseth in his evidence at [428] onwards, identified in the judgment itself. Basically, places where a full time branch Post Office is not justified, such as remote areas, may have such services on (say) half a day per week, in a mobile van or a village hall. These are called “outreach” branches, and they are connected to (or part of) a core branch, run by the same SPM. This means that the SPM is in charge of both the core branch and the outreach branch. The Dalmellington bug affected such branches.

167.

A PEAK was created on 13 October 2015 in respect of Dalmellington, namely PEAK PC0246949. The SPM in that branch had attempted to transfer cash to its outreach branches. The SPM sought to transfer £8,000 and ended up with a £32,000 transaction. This problem involved issues relating to two different scripts, the Post Log On script and the Forced Log Out script. The latter did not close down the former correctly (or in software terms, did not lead to the former script correctly closing down the operation) which meant that the script was left on the stack of incomplete processes. If the SPM then logged back into the counter to perform a transfer of cash

from their core branch to the outreach branch, the “pouch delivery” screen would still show, with “enter” being enabled. If the SPM were then to press “enter”, then the value of the transaction would repeat. As expressed in a confidential briefing given to the Post Office by Fujitsu:

“It relies on a mechanism which checks the stack of incomplete processes to see if it is complete. Due to the fact that the stack is not empty (following the first problem) it thinks it has not finished and as a result attempts to repeat the last part of the script, which in this case is to record the remittance transactions and print the receipts”

168.

This would result in the amount of cash (in the case of Dalmellington, £32,000 as enter was pressed a further three times) being ‘Remmed Out’ of the core branch. Because each “pouch” which holds cash has its own ID, Fujitsu researched the number of times that duplicate pouch IDs had occurred in the BLE files.

169.

There are two points upon which the Post Office relies in respect of this bug. The first point is that the Fujitsu investigation in December 2015 showed that 112 potential occurrences of the Dalmellington issue had been identified in the previous 5 years. Of these 112 potential occurrences, 108 items were corrected at the time, either by Post Office issuing a Transaction Correction or the SPM reversing the duplicate Rem In. This left 4 remaining potential occurrences, for which Fujitsu had not yet established the outcome. However, the claimants rely upon two points of their own in this respect. One is that this showed that even though the bug had occurred multiple times and affected branch accounts, neither Fujitsu or the Post Office even knew it was a bug prior to late 2015. In my judgment, that is a valid point for the claimants to make. The second feature is that TCs are, as has been identified in the Horizon Issues judgment itself, outside of the Horizon System.

170.

A third feature is that the Post Office did not disclose the existence of this bug more widely.

171.

The Post Office also submits that “once the full extent of the Dalmellington issue was understood, Fujitsu produced a code fix without delay and it was rolled out in January 2016.” However, it cannot seriously be suggested that correction of the bug was “without delay” when the full history of the PEAKs are considered, with incidents going back to 2010. By the time this bug was remedied in early 2016, the mediation scheme had been brought to an end some time before (this occurred in March 2015) but some of the branches affected by the Dalmellington bug were within the scheme, according to the Fujitsu investigation. However, none of the periods when the bug hit matched what were recorded in the investigation as “Mediation dates”. Not only that, but for the period up to the commencement of the mediation scheme, the Post Office was meeting a case brought by the claimants that there were widespread bugs within Horizon which impacted the branch accounts. Dalmellington shows that there was such a bug, which lay undiscovered (although its effects were apparent) until 2015.

172.

The Post Office also submitted the following:

“As put to Mr Coyne by Mr de Garr Robinson, from an outsider’s perspective, you would not be able to tell the difference between someone remming in twice by accident, due to human error and someone remming in twice because of an error on screen.”

173.

I am not sure why this submission seeks, as it does, to deploy that submission as a point in the Post Office’s favour on the Horizon Issues. It is rather a point in the claimants’ favour, as it means that there was no visible sign that the bug had done what it had done. It could be misinterpreted as human error; its effects looked exactly the same, and Mr Godeseth in his evidence accepted this. It is also not a point in the Post Office’s favour that any known instances of this bug were corrected by means of TCs, or “by the SPM himself reversing the REM in some way”, which is how it was put in cross-examination.

174.

Two of the four unknown outcomes were for £25,000, and £2,500. The Post Office, in its closing submissions, explained why neither of these were in fact incidents of the Dalmellington bug because they were not outreach services, and also although bar codes for cash pouches were supposed to be unique, there was a small batch of them that were not. The documents from which these submissions were drawn show that the two branches with these discrepancies were those with FAD codes 209311 and 157242. However, the documents also report that both these branches had duplicate IDs for remming in of cash, that there had been “no contact from the branches raising issues for the period in question (February and March 2013)” and come to the following conclusion (although I consider it is worth noting that the SPMs in each of these branches do not appear to have been asked, nor does the ARQ data appear to have been used):

“Post Office concludes the issues at the branches have arisen as a result of remittances pouches received at the branch entered manually which had the same barcode id. Thus creating duplicate entries which Fujitsu highlighted as part of the BLE files checks. However, in these instances from the available evidence Post Office concludes that the correct amount of pouches were delivered, accepted and entered on Horizon. This is supported by the fact that there has been no negative impact in the branch accounts and no record of an issue raised by the branches with Post Office.”

And “Moreover, these two branches have resolved in branch the issue encountered without referral to Post Office via NBSC and FSC or Fujitsu.”

175.

I find that this bug was present in Horizon Online from the date that Online was brought in, namely the year 2010. In my judgment, this is absolutely clear, and does not appear to be in dispute given the dates in the Fujitsu investigation.

176.

This bug would, in my judgment, have had an impact upon branch accounts – it is the extent of that impact that is in issue, whether it was lasting or transient. Dr Worden relied upon what he described as “a well-tested process of reconciliation and TCs to detect and correct errors in cash remming (used 20,000 times per year)” and said it was “straightforward for Horizon to detect any discrepancy between a “rem out” and the corresponding “rem in” (a mismatch arising either from a miscount, or a multiple count of a pouch) and then a TC can be issued.” He also added that “this process catches and corrects remming errors, whatever their cause - including if they arise from, or are provoked by, software faults.” He therefore implicitly accepted that the Dalmellington bug was a software fault, although he did not say so in terms. I find that it was plainly was. I also reject his evidence that it was “straightforward for Horizon to detect any discrepancy”; the evidence about this bug shows that Horizon did not detect it in these many cases.

177.

Dr Worden was also reliant upon the process of TCs to correct it. Further, the bug was there for 5 years and was not discovered, although its effects doubtless were experienced during that period.

178.

It is correct to record for completeness that all known impacts of this bug appear, on the face of the evidence before the court at the Horizon Issues trial, to have been corrected by the Post Office, with the exception of the two branches identified which were not outreach branches. The documents show that branch 209311 had no rem issue on, or about, the date of the duplicate ID incident of £2,500 (which was 1 March 2013) and the SPM had not raised any issue with the Relationship Manager Mr Andy Winn. There was no record of any TC having been issued for £2,500 and an audit in August 2013 had shown a shortfall at that branch of £350.55.

179.

The other branch, 157242, was operated by a SPM who transferred to a “Main agreement” in May 2014. Such an agreement did not feature in the Common Issues trial but I assume it means it is now a Main Post Office and not a branch. Cash rem in issues were shown at the branch in February 2013 but none of the same type as the £25,000 duplicate rem in, and none on the exact date in February which was 18 February 2013. Although two of these were for very substantial amounts, none were for £25,000 and Mr Winn again said that no issue had been raised. I recite the findings in respect of both these branches for completeness.

180.

I find that this was a software bug that impacted upon branch accounts. In my judgment, it had lasting impact upon branch accounts. The fact that the occurrences of it (over 100 in a 5 year period) were corrected by means of TCs does not mean that it ought not properly to be characterised as a bug causing lasting impact to branch accounts. However, there were no accounting shortfalls ultimately experienced by SPMs whose branches suffered from the effects of the Dalmellington bug as these were corrected by means of TCs. SPMs whose branches suffered the effect of this bug were not therefore required to pay the sums the subject of the shortfalls to the Post Office on the evidence before the court.

5. Remming In bug.

181.

This is a Horizon Online bug. It was present for about five months in 2010 between March and August. It was not expressly acknowledged as a bug in the Bug Table. However, Dr Worden accepted it was a bug or defect in his cross examination on Day 20 and it is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is again said to be one that had only transient impact. The years it was said to have been present in terms of effect was “March – August 2010 and recorded as fixed approx. 2011”. Mr Coyne said there were 14 branches affected. Dr Worden’s entry in the Joint Statement relied upon TCs, and stated “As for the Dalmellington bug, above – PO had a robust process for detecting and correcting remming errors, whatever their origin. So, there were no lasting effects on branch accounts.” In paragraph 2 of the detailed part of Appendix 2 relating to this bug, the Post Office submitted that “any discrepancy would be transient as instances of this bug are caught by automatic reporting.”

(emphasis added)

182.

The Post Office relies upon automatic reporting to catch incidents of this bug, and also maintains that Mr Coyne has conflated two issues, one relating to PEAK PC0203085 (which it terms “Issue 1”) and the other related to PEAK PC0195380 and other PEAKs associated with KEL acha4221Q (which it refers to “Issue 2”). Both of these relate to the remming in of cash, and both result in a cash pouch being recorded twice in error. However, they are said to be different issues because the sequence of steps taken by SPMs to trigger them are significantly different.

183.

The process of “remming in” works as follows. SPMs receive pouches of cash which are sent to the branch from the Post Office cash centre. A certain amount of cash is held in each branch for the purposes of transacting Post Office business. Each pouch has a unique barcode that needs to be scanned by the branch when it is received. This automatically looks up the contents of the pouch so that the branch can confirm that the physical contents of the pouch match up to the record on Horizon. It is for this reason that each code is supposed to be unique.

184.

The way the system is supposed to work in theory is best taken from the Post Office’s closing submissions: “If there is a difference between what the system says is in a pouch of cash and the amount of cash actually in the pouch, the Subpostmaster should raise the issue with NBSC in order to get a Transaction Correction. A remming error leads to a mismatch between the amounts of cash remmed out to one place and the amounts remmed in from another. Remming errors are a violation of Data Entry Accounting and are picked up by Horizon.”

185.

That latter sentence was said by the claimants, after the written submissions were received, not to be present in any of the evidence used in the Horizon Issues trial. This point is dealt with in general terms at [120] above. In response to that challenge, the Post Office clarified its submissions and relied upon Dr Worden’s report, which was in general terms in respect of the general principle of double entry accounting; the HNG-X architecture document, which again demonstrated the theory or principle of the system; a PEAK document; and Mr Coyne’s cross-examination. It is necessary to consider those therefore, in order to see if the submission is substantiated.

186.

The principles of what is supposed to happen within Horizon, and the architecture, is not in dispute. Mr Coyne’s cross-examination passage that is relied upon does not support the general submission made to the PEAKs the subject of this bug, as it was as follows. I will quote the passage relied upon and the passages either side for context:

“Q. So in forming a judgment on robustness you have first of all to see -- let's take a bug. A bug happens and the first question you ask is: did that bug or could that bug have had an impact on branch accounts? A. Yes.

Q. Then you form a judgment on whether and to what extent the countermeasures in place in the Horizon system would have enabled that impact to be identified and fixed, yes?

A. No, I think you would look at whether it did -- whether any countermeasure or control did prevent it from having a consequential impact, not whether it should have. Q. Well, whether it would have?

A. Or whether it would have.

Q. You don't consider whether it would have, you consider whether it did?

A. Well, if it did there would be evidence that it did. It would be documented.

Q. But in some cases it will be blindingly obvious, won't it, Mr Coyne? Let me give you an example, remming. A Post Office example. Remming in and remming out. Money is sent from Chesterfield to a branch. Chesterfield has a record of the money it sends over. The branch receives the money and types in -- or usually it is a barcode actually, but it records on the Horizon system how much money the branch has received. You are aware, aren't you, that there is an automatic system that checks Chesterfield's figures against the branch's figures? A. Yes.

Q. Are you suggesting that every time over the last 20 years there has been a discrepancy between Chesterfield's figures and the branch's figures, are you suggesting that it is necessary for you to see the evidence of what happened next, of whether a transaction correction was sent and what the evidence was in relation to that branch? Are you really suggesting that's necessary?

A. No, because in a typical scenario where the systems operate as they should, as they are designed, then the positioning of the countermeasures that you have put in place would pick up on that so that would work absolutely fine. In the scenario where the system doesn't operate as expected there is a bug, error or defect or communication fault, then a different set of scenarios will likely be encountered and it is understanding then what happens that is important.

Q. Let me get this straight. You are suggesting that there are cases when you can take it as read, the situation is such that you can take it as read that a problem will be identified and it will be fixed. But you are suggesting there are other situations where you can't take it as read where specific evidence is needed, is that right? A. Yes.

Q. Okay. But nonetheless, although you say that is necessary, it is your considered opinion that the Horizon system that you now see is robust, yes?

A. Relatively robust, yes.”

187.

This evidence does not support the submission that remming errors are picked up by Horizon. It is necessary therefore to look at the actual PEAKs, to see what they show. The one associated with what the Post Office called Issue 1, PC0203085, is dated 22 August 2010 and is headed “pouch remmed in on two counters at same time”. The first entry under impact statement is:

“The same pouch can be remmed in to the system more than once, resulting in a shortage at the branch which POL have to rectify by issuing a Transaction Correction.

1.

call has to be processed, and corrective action taken, by SSC, MSU and POL

2.

visible to POL and the branch when it happens

3.

very rare”

188.

In my judgment, that entry alone is evidence of a bug. It shows a pouch can be remmed in more than once – admittedly rarely – and that a TC is necessary to correct this. However, in order to be clear, the PEAK goes on for some pages – 13 in total – and includes contradictory entries within it from Fujitsu such as the following two. One is on 17 August 2010 from Anne Chambers:

“A cash pouch was remmed in twice at branch 126109:

Pouch barcode 399347067204

2p coin £60

50p coin £250

5p coin £100

Session 1-350379 16/09/2010 10:08

Session 2-195226 16/09/2010 10:08

The PM cannot reverse the transaction since rem reversal isn't allowed.

This is NOT another example of the duplicate rem problem that we have seen in the past, where use of the Prev key accepted the same pouch twice. In this case the pouch was processed on both counters...

09:05 c2 get pouch status, retrieve pouch details

09:06 c1 get pouch status, retrieve pouch details

09:08 c2 settle pouch delivery

09:08 c1 settle pouch delivery

There were some printer problems on counter 2 which probably explain why this was done.”

189.

The other contradictory entry within the same PEAK is the next day, 18 August 2010 and states, again from Anne Chambers

“I checked whether there were any exceptions in the BAL OSR logs for any of the messages, there was nothing.

Gareth Jenkins thinks that it should not be possible to complete the rem in on both counters. Please investigate.”

190.

Under Root cause and solution, the following entry is made on 19 August 2010:

“When an auto remin pouch id is settled successfully, the system updates the

COUNTER_READ_TIMESTAMP in LFS_RDC_HEADER to a not null value for that pouch id.

The race condition for auto remin pouch delivery is handled at SettlePouchDeliveryServiceSettlementProcessor.processPouch().

This method checks during settlement whether the

COUNTER_READ_TIMESTAMP in LFS_RDC_HEADER is null or not null value.

If null, the pouch id is good and settlement completes successfully.

If not null, the pouch id is already processed and error is thrown.

The query that gets the COUNTER_READ_TIMESTAMP from

LFS_RDC_HEADER is 'SettlePouchDeliveryPreCheck'.

In this query, the input parameter for pouch id is defined incorrectly.

It is given "pouchBarcode[String]", but in dyno the pouch id is "pouchId".

This is the root cause why the query always returns null although the COUNTER_READ_TIMESTAMP is not null.

The solution is to define the pouch id input param to "pouchId[String]" in 'SettlePouchDeliveryPreCheck'.

191.

A fix is then identified, stated to be a low risk and low complex issue, which also states:

“LIST OF LIKELY DELIVERABLES:

Updated sql of SettlePouchDeliveryPreCheck”.

192.

In my judgment, this PEAK is evidence of a bug and a fix is required to remedy it. It also shows that remming in errors are not always picked up by Horizon.

193.

The second issue is related to what are different PEAKs, namely PC0195380 and other PEAKs associated with KEL acha4221Q. I will not deal with them all, but these relate to what the Post Office terms Issue 2. This KEL was raised by Anne Chambers on 2 March 2010, updated on 3 May 2011, and the keywords are remittance, remittence, remin, pouch and delivery. It can be seen that the following is included in the KEL:

“Symptoms

The clerk went into the Delivery menu and scanned two pouches (one of currency and one of coins). The second pouch was recorded twice on the system, resulting in a loss of £80.

Two Rem In slips relating to the second pouch were output, both identical, as well as one for the first barcode.

In the most recent instance, the same pouch was remmed in on two different counters at about the same time.

Problem

This was caused by using the Prev key during / just after the pouch barcode scans. Now fixed - details in PC0195380.

In PC0203085, the same pouch was processed on both counters...

09:05 c2 get pouch status, retrieve pouch details

09:06 c1 get pouch status, retrieve pouch details

09:08 c2 settle pouch delivery

09:08 c1 settle pouch delivery

PC0203085 - fix applied Jan 2011.

Solution - ATOS

A transaction log search will prove that the Rem In has been duplicated.

Send call to SSC, quoting this KEL.

SSC:

Known problems have been fixed, so needs fresh investigation if it happens again.

POL may need to issue a TC to undo the effects of the extra rem in (in the meantime the branch will report a shortage), so MSU need to inform POL via BIMS.”

194.

Given that the PEAK on Issue 1 is dated August 2010, and the KEL itself was raised in March 2010, the KEL must have been updated with the entry relating to that PEAK and that the fix was applied in 2011. Those entries cannot have been included in the KEL originally, as they post date its creation in March 2010. This means therefore that someone at Fujitsu obviously concluded that the later PEAK in time (on Issue 1) was relevant to the same KEL (on Issue 2) as it was added to the KEL. I agree with whoever at Fujitsu linked these PEAKs to the same KEL. I consider that the KEL must refer to broadly the same issue, namely the remming in of pouches that were counted twice by the Horizon System. A fix was in any event issued for the issue identified in the KEL. However, it is correct and I accept the Post Office’s submissions that upon closer analysis it can be seen that there are two separate, but very similar in effect and related, issues here, both of which have the same end result, namely the duplication of remming in of cash pouches.

195.

Finally, in the Post Office’s clarification or response to the challenge brought by the claimants to the submissions on this bug, (which complained that there was no evidence to support the submission that remming in issues would be automatically “picked up” by Horizon), the Post Office referenced another document at {F/624/1}. It stated that the document was PC0197838, but in fact the reference is BE/0197828, as it is a BIMS Incident Report and not a PEAK, and is dated 16 April 2010. The reference number is the same but the initials are BE and not PC because it is a different type of document. However and in any event, the document relied upon by the Post Office identifies an exception type as “Receipts and Payments do not balance (post migration)” but it does deal with the same issue as this one, as it states within the body of the document that “The PM has used the Previous key whilst scanning in pouches, which has caused a duplicated rem” and also:

“The two identical transactions each comprise of:

Prod ID = 6287 (Rem In Cash from AD); Qty = 1; Amount = £1680 The Post office Counter log shows this being made up of:

£100 of 20p coins

£80 of 2p coins

£500 of 50p coins

£100 of 5p coins

£500 of £1 coins

Action for Post Office Ltd:

Post Office Ltd will need to contact the PM to confirm if the PM has two identical Rem In slips for Session 489165, both showing Pouch ID 399345656721.

If this is the case, then the Rem In has been duplicated and this duplication needs to be reconciled, possibly by using a Transaction Correction.” (emphasis added)

196.

I accept that the issues on this bug can be split into Issue 1 and Issue 2 in the way that the Post Office suggest. If I am wrong about that, then the PEAKs I have identified above, including the BIMS Incident Report, are evidence of and relate to the same bug. Given the finding of two issues, then Issue 1 and Issue 2 could either relate to different iterations of one bug, or to two different bugs. These bugs or this bug (depending upon which semantic analysis of Issue 1/Issue 2 is correct) both impact branch accounts. Issue 2 related to the pilot phase of Horizon Online. However, neither of these issues are caught by automatic reporting. The Post Office accepts that Issue 1 was reported to it by the SPM affected. The fact that the BIMS report requires the Post Office to “contact the PM to confirm if the PM has two identical Rem in slips” for the session identified makes it clear that this also applies to Issue 2 as well. There is therefore, in my judgment, no practical difference whether one approaches this as two different issues with the same effect, or one.

197.

On either analysis, it is a software bug and I find that these or this impacted upon branch accounts. In my judgment, it had lasting impact upon branch accounts. The fact that the occurrences of it were corrected by means of TCs does not mean that it ought not properly to be characterised as a bug causing lasting impact.

6. Remming Out bug.

198.

This arises under Legacy Horizon. The Post Office’s submissions effectively accept that there was a bug under this heading, because in the detailed entry in its Appendix 2 it recites the following:

“Mr Coyne states that Bug 6: Remming Out is a bug with lasting financial impact. It comprises two separate issues, only one of which was a bug. Any discrepancy caused by either issue would be transient as instances of both issues were caught by automatic reporting.”

(emphasis added)

199.

It was split into two in the Bug Table, 6(i) which was identified as KEL acha508S and 6(ii) which was identified as KEL GMaxwell3853P. The former was identified as February/April 2007, recorded as fixed approx. 2007, with Mr Coyne identifying 57 branches affected; and the latter was in May 2005 with one branch being affected. They were both remming out issues, hence the grouping by the experts in the bug table. It was not acknowledged as a bug in the bug table. However, it is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact.

200.

Again, for both of these iterations, 6(i) and 6(ii), Dr Worden used the same wording as he had in the entry for bug 5, the Remming In bug, namely “As for the Dalmellington bug, above – PO had a robust process for detecting and correcting remming errors, whatever their origin. So, there were no lasting effects on branch accounts” (emphasis added). It was also said by the Post Office in its submissions that Mr Coyne had conflated two unrelated issues under the heading “Remming Out”. Given this was an entry in the 2nd Joint Statement agreed by the experts, it is a little unfair to state that Mr Coyne had done this, as the entry was obviously agreed by Dr Worden. However, it was split into 6(i) and 6(ii) and I will consider each separately.

201.

Issue 6(i) arises as follows. What is called “a remming error” leads to a mismatch between the amounts of cash remmed out to one place and the amounts remmed in from another. The Post Office has submitted that remming errors are a clear violation of Data Entry Accounting and are picked up by Horizon. The two different issues are as follows.

202.

As the obverse of the coin of remming in, SPMs rem out pouches of cash to be returned to the Post Office Cash Centre. A single pouch may contain multiple bags of coins or cash and each bag can only hold one denomination, and there is a limit on how much cash can be placed into a pouch. The cash can be remmed out before it is physically collected. When remmed out the cash appears in a different line in the branch accounts. On collection, the collection team scan a barcode on the pouch and the cash is removed from the “cash in pouches” line of the accounts.

203.

When remming cash out, branches should have made one entry for each denomination and value and, if there were multiple bags for a particular denomination, the quantity of bags should have been specified in that single entry (e.g. 2 x £500 of £2 coins). However, if the SPMs had made multiple entries for each denomination and value (e.g. one entry for 1 x £500 of £2 coins and then a second entry for 1 x £500 of £2 coins), Horizon would only record the first bag as having left the branch’s cash holdings, but all of the bags would show on the “cash in pouches” line. This would create a discrepancy in the branch accounts, because all of the cash would have been collected.

204.

This issue occurred as a result of Release T30 INC1 (which is referred to in the Fujitsu Service Review Book of February 2007 covering that month) and the Post Office’s submissions accept this. A further entry for Monday 12 February 2007 in the same document states:

“The issue was found to be when Post Masters would REM out 2 identical items instead of using the quantity button. As this was an error in the code, the counter 36_6 s/w was regress overnight. After the regression of the s/w overnight, it was then identified that 49 branches had an issue with the REM out process causing a cash imbalance at the counter.”

205.

I consider this to be a bug. Fujitsu created KEL acha508S to advise branches to correct the problem manually and ran automated reports to spot any further occurrences. Fujitsu then made changes to Release T30 INC1 and rolled it back out to fix the issue. However, that the bug was fixed does not, in my judgment, mean it ought to be ignored. Paragraph 22 of the Appendix 2 submissions on this by the Post Office submits “in any event, it was picked up automatically”. This is not correct. An entry in the February Service Review Book on page 11 of 33 of that document shows that there were 526 calls to the Horizon Service Desk as a result of this specific issue, of what were said to be a high number of software calls in that month of 3792. That does not support a submission that it was picked up automatically.

206.

So far as Issue 6(ii) is concerned, the following evidence from Dr Worden – which is entirely ignored in the Post Office’s submissions in Appendix 2 – is relevant.

“Q. So we are looking at 6(ii).

A. Yes. So again my standard wording comes in.

Q. That's it. You will see there:

"'Remming out' (ii) Bug (not acknowledged)."

This isn't one that you had in either of your tables in your appendix originally. A. No.

Q. Did you have a look at the KEL in that case?

A. I believe I did, yes.

Q. Can we look at it now, please. This is at {F/276/1}.

A. Rem out ... Yes.

Q. This is the GMaxwell3853P KEL. If we look at the bottom where it says "Solution - Atos".

A. Bug in the code, yes, right.

Q. "Development have identified a possible bug in the counter code. However, due to the frequency of the problem & the risks involved in making the necessary changes it

has been decided that a code change will not be made." A. Yes.

Q. You hadn't noticed that, had you?

A. I believe I had, but again the point about permanent effects, lasting effects, on branch accounts is not altered by that.

Q. And the point about this is there was a lasting bug?

A. There may have been a lasting bug, but they took a decision based on its frequency and the difficulty of making the change, perhaps the risk of making the change, that it was not worth addressing. That actually the remming process would pick it up and its frequency -- they took a decision, I don't know what the details were of their trade off. They took a decision that fixing the code was not worthwhile. Q. I'm just asking you: it is a lasting bug?

A. It is a lasting bug, yes.

Q. Let's go to number 7 now, if we may, please.”

207.

There was then an interjection by counsel for the Post Office who drew attention to possible misunderstanding in terminology given the use of the phrase “lasting bug”. Dr Worden then added the following, when asked by the claimants’ counsel whether he had understood what was meant by the question:

“A. I did. And I'll clarify what I mean: it's a bug that was not fixed, we cannot see was fixed, but it was a bug without lasting effects in my opinion.”

208.

In my judgment, it was clearly accepted by Dr Worden as a bug, and the entry in the KEL itself also says there was “a possible bug in the counter code” but a decision was taken not to change the code because there was only one instance of this occurring.

209.

I find that 6(ii) also relates to a bug and reject the Post Office’s submission that “it wasn’t a bug”.

210.

The claimants’ case on the Remming Out bug in the Bug Table at item 6 is therefore made out. Both of these – issue 6(i) and 6(ii) – are, in my judgment, software bugs. I find that these had potential lasting impact upon branch accounts.

7. Local Suspense Account issue, not the same as 3. Suspense Account bug.

211.

This was reported in 2010 and recorded as fixed in September 2010. It arose in the early days of Horizon Online, during the pilot scheme. It was not acknowledged in the

Bug Table, but is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, although it is one of those bugs said to have had transient impact. Mr Parker had identified 33 branches affected. Mr Coyne recorded what he said were four associated KELs, namely acha5259Q (for which there were 6 PEAKs) (this KEL is mistakenly recorded twice in column 3 of the Bug Table, with acha5838T only mentioned in the text in that column); cardc2043L (10 PEAKs); PorterS199P (3 PEAKs); and acha5838T (which states there are “two different but similar problems” and appears in the text, but not in the list of KELs at the end of the text by Mr Coyne). Dr Worden’s comments in the table were that:

“The KEL acha5259Q implies that PO and Fujitsu were able to identify all occurrences of the problem, without being notified by any Subpostmaster. I would therefore expect them to have corrected any impact on branch accounts as part of normal error correction processes.

I would not expect evidence of all corrections to accounts to have survived to the present day. PEAKS and KELs are not used to record corrections of financial impact.”

212.

He also relied upon a statement by Mr Parker that there was “Temporary financial impact which would have been cancelled out in the following period by a corresponding discrepancy”. To use the Post Office’s own submissions it was “an intermittent system issue which temporarily prevented branches from rolling over into the next Trading Period”.

213.

The statement of Mr Parker introduces another concept, similar if not identical to Dr Worden’s “transient impact”, and that is “temporary financial impact”. Dr Worden is correct that PEAKs and KELs do not record corrections of financial impact; they do however record financial impact, because when an SPM reaches SSC (which raises the PEAK) they often start by recording “SPM says he/she has a problem in that……” and financial impact is often then recorded. The Post Office in paragraph 2 of the detailed part of Appendix 2 dealing with this bug states that “any discrepancy would not be lasting”. This does therefore accept, albeit implicitly, that it would have an impact upon branch accounts. It is also submitted that it was a “teething problem” from the early days of the Horizon Online pilot scheme.

214.

The cross-examination of Dr Worden on this bug was as follows, and I find to be highly relevant.

“Q. Then you say:

“I would not expect evidence of all corrections to accounts to have survived to the present day. PEAKs and KELs are not used to record corrections of financial impact." A. Yes.

Q. Fujitsu, in fairness, analysed the KEL, you say here, and said:

"... 'Temporary financial impact which would have been cancelled out in the following period by a corresponding discrepancy.'" Do you see that? A. Yes.

Q. Now, did you look at the KEL?

A. Well, I infer from that Fujitsu did it, but I had done it previously.

Q. Let's go, please, to {F/637/1} which is the KEL acha5259Q.

A. Right. Sorry, can I remind myself of the ..."... local suspense cleared ... balance report ... cash ..."

Q. Do you see if we look at the problem --

A. Yes.

Q. -- the summary was that the PM was forced to clear local suspense several times resulting in the cash figure on the balance report changing and possibly a discrepancy in the new trading period. A. Yes.

Q. So that certainly had the potential to cause an impact on branch accounts at the time, subject to later correction? A. Yes.

Q. And probably did, yes?

A. I think it did, yes.”

215.

It is in my judgment self-evident, that if the branch accounts had to be corrected later, in a new trading period, that the accounts for the period when the effects of the bug were experienced were wrong. This was a bug. It was a bug that had a lasting impact upon branch accounts as I have defined that phrase. The fact that the discrepancy was later corrected, in the words of Mr Parker a discrepancy “which would have been cancelled out in the following period by a corresponding discrepancy” makes this entirely clear.

8. Recovery Issues.

216.

These are Horizon Online issues. The Post Office does not accept these are bugs at all. These are not agreed as bugs by the experts, and there are four different types included in the table, with years of effect from 2010 to 2018. Mr Coyne’s entry stated that “The text within the PEAKS and KELs suggests that in each case a branch account discrepancy would be evident and would require correction by the Post Office.” Dr Worden stated that:

“The KELs and PEAKs cited by Mr Coyne are not indicative of errors in Horizon. They provide guidance on how to correct discrepancies caused by human errors or other errors in transaction recovery ('recoverable transactions')

Because there were many such errors, there were many calls to the help desk and many PEAKs and KELs. Normally, correction of errors involved back office reconciliation and issuing TCs. This was accurate and effective; I have derived an upper limit of £2 per branch per month on the mean impact of erroneous TCs.

One important KEL acha959T was guidance to the back office MSU, not for Subpostmasters”.

217.

There was a minor typographic error in the submissions in paragraph 4 of the detailed part of Appendix 2 which referred to bug 9. However, the four different types of recovery issues are addressed in the subsequent paragraphs and the conclusion paragraphs deal with bug 8. This was later corrected in a sheet of corrections, which clarified that paragraph 4 of the detailed part of Appendix 2 should have stated "Mr Coyne states that Bug 8: Recovery Issues is a bug with lasting financial impact. Post Office submits that it is not a bug at all".

218.

Recovery deals with the situation where there has been an interruption, which means when a basket of transactions is interrupted during the course of dealing with a customer. This could be for several reasons including a power failure, system crash or communication failure between the branch and data centre. This is a risk which has to be dealt with as it cannot sensibly be eliminated in a system such as Horizon. Horizon runs various automated reports each day to look for failed recovery events. Mr Coyne dealt with two different documents in his report, PC0197769 and KEL acha959T. The former was created on 15 April 2010 during the pilot phase of Horizon Online. The root cause of the issue was identified on 26 April 2010, work on a fix was started and this was released in early June as shown in Release PEAK PC0199000 (and then estate-wide to pilot branches) by mid-June 2010. It did however undoubtedly evidence a bug. The reference to the “959T” KEL below is the KEL whose full title is acha959T which was raised by Anne Chambers.

219.

KELacha959T is also the very same KEL referred to in PEAK PC0214226 dated 14 December 2011, headed “Failed Recovery Transaction(s) 12 Dec 11” which I have considered at [110] and [111] of the substantive judgment, as it related to Mr Tank’s specific experience at his branch regarding the £195 transaction. I have already dealt with one entry in that PEAK in that part of the judgment. I have accepted Mr Tank’s evidence on this incident. That PEAK was closed with the categorisation “Final – Advice after Investigation” and “Root Cause – General in Procedure”. The entry at 10:40:51 on 14 December 2011, also from Mr Bragg, shows the transaction details which plainly identify an authorised transaction, a failure of the basket settlement, the failure of repeated retries by the system and the failed recovery. It was this analysis that preceded Mr Bragg’s conclusion that “the customer’s account should be correct but the branch will have a shortage (for a withdrawal) because the session hasn’t been recorded”. This is plainly a fault in Horizon, and is directly contrary both to the Post Office’s case on failed recoveries in the Bug Table, and also its case on Mr Tank’s evidence.

220.

Dr Worden’s cross-examination on the issue of recoveries, by reference to entry 8 in the Bug Table, was also highly relevant:

“Q. Let's move if we may now to number 8 and we find that -- just to do it the same way again, if we go back to the joint statement 2 table and we look at {D1/2/7}. A. Row 8.

Q. It is row 8. The recovery issues are identified there.

A. Mm.

Q. Mr Coyne has made references to the text in certain PEAKs.

A. Yes.

Q. And you have given your opinion there, you say they are not indicative of errors in Horizon, they provide guidance on how to correct discrepancies caused by human errors or other errors. A. Yes.

Q. You say: "Because there were many such errors, there were many calls to the help desk and many PEAKs and KELs." Then it is basically your same theory, that: “Normally, correction of errors involved back office reconciliation and issuing TCs." And that's accurate.

A. It is a different theory really. Recoverable transactions are a big subject and they are complicated because the point at which a recoverable transaction can go wrong is very variable through the sequence, and therefore the number of recovery actions, the type of recovery actions is complicated. Horizon was designed so that with the assistance of the postmaster most of these could be recovered, but there are things called failed recoveries, which again were part of the design of Horizon, and those were the failed recoveries but particularly the ones where Fujitsu had to get involved. But failed recoveries means the recovery process had failed, that's the way Horizon was designed because these things are so complicated you can't handle them all automatically. So it is a big subject but there is plenty of useful evidence about it. Q. You say in your table at {D3/2/124}.

A. This is the 959T KEL, is it?

Q. Yes. You say:

"This is another complex KEL with strong overlap with previous KEL." A. Yes. This KEL is cited in a large number of PEAKs.

Q. And in fact when we go back to the table at {D1/2/7}, Mr Coyne has actually quoted from one of the PEAKs we see there, the first one he quotes from. Do you see "PC0198352"? A. Yes.

Q. And he has quoted: "This problem caused a loss at the branch for which they should not be liable."

Pausing there. This did have the potential to cause the type of discrepancy which you have given evidence about being later corrected, didn't it? A. The word "problem" doesn't imply a bug in Horizon.

Q. You didn't read these PEAKs yourself, did you?

A. I have read a fair sample of recovery PEAKs. Maybe not these ones, but I have read a fair sample.

Q. If we go over the page {D1/2/8}, do you see Mr Coyne points out there that this

KEL is referred to by 2,473 PEAKs --

A. Yes, that's --

Q. -- from 2010 to 2018.

MR JUSTICE FRASER: Where are we looking?

MR GREEN: At the top of that page under the "Coyne Opinion as to branch account impact", my Lord.

MR JUSTICE FRASER: Yes.

A. That is correct.

Q. Just pausing there, Dr Worden. If something is handled by NBSC and corrected straightaway, it doesn't make it necessarily up the line to SSC, does it?

A. No. I mean there are various levels of recoverable transactions. There is a recoverable transaction that the subpostmaster can follow the instructions and can fix it himself, there is the ones where he needs help from NBSC, there's ones where he needs help from further down, and there are failed recoveries where PEAKs are involved. All of those things happen.

Q. In the case of a failed recovery, it can be sorted by NBSC in some way or by a transaction correction without referring to SSC?

A. That's a good question. I think the failed recovery port usually involves not just NBSC, it involves MSU, and there is quite a complicated process of guiding the transaction through various states until it gets to the F99 state.

Q. Can I just invite you to look at what Mr Coyne said --

A. Yes.

Q. -- having actually tried to identify through the available disclosure how many PEAKs were referenced and set them out. He says it is referred to by 2,473 other PEAKs from over that range:

" ... each of these may (but may not) indicate a similar issue with the Horizon recovery process and potentially creating a discrepancy within branch accounts." If we accept his definition of discrepancy and not yours, yes?

A. This is all about temporary discrepancies in branch accounts.

Q. Park the temporary discrepancy point about which we have heard a lot.

A. Yes.

Q. What he has said there is correct in the number of PEAKs as far as you know?

A. It is correct, yes.

Q. And it is scrupulously fair in how he has described it?

A. Well, "issue with the Horizon recovery process", I believe these issues are how Horizon was designed.

Q. But what he's said there, he has carefully put "issue", leaving open the availability of an argument between experts about what the nature of the issue is, but what he has scrupulously done there is perfectly fair, isn't it?

A. I think so. I mean the difference between the experts is that, to summarise something Mr Coyne said in his oral evidence, he believes that these recovery processes can be automatic. Now, I don't believe that. I believe recovery of recoverable transactions is very complicated and needs human intervention and things like failed recoveries are inevitable.”

(emphasis added)

221.

I prefer the evidence of Mr Coyne that recovery processes can be automatic in systems, and I also accept that “failed recoveries” as they were defined above are inevitable. The issue here is whether the Horizon System process for dealing with the recovery processes when they had in fact failed was working as it should, or defectively. The fact that there are so many PEAKs identified in the KEL, and the contents of those PEAKs clearly demonstrates, in my judgment clearly, that it was not. I reject Dr Worden’s evidence that human intervention should be required or expected for some, most or all failed recoveries. I also reject his evidence about the word in the PEAK “problem”. He said “The word "problem" doesn't imply a bug in Horizon.” In my judgment, on this subject, it clearly does.

222.

The Post Office’s own submissions set out how trading periods were affected. I shall reproduce them:

“10. The issue is described in KEL acha5650L. It involved recovered transactions being written to a different trading period than the original transaction. In summary:

i)

if a transaction failed in one trading period and the recovered transaction went into the next trading period, there would be a loss in the first trading period 5 and a corresponding gain in the next trading period such that there would be no overall loss in the branch; and

ii)

if the recovered transaction was written to an earlier trading period, there would be a loss in the current trading period but no corresponding gain because the previous trading period would have already been rolled over again. There would be a net loss in that scenario.”

(emphasis added)

223.

That demonstrates that branch accounts would be affected. In the first possibility, the accounts would be wrong for one trading period (until the so-called “corresponding gain” occurred in the next period) and in the second, it is accepted there would be a net loss.

224.

In my judgment, the issuing of a later TC is neither here nor there in this scenario. Where there has been a failed recovery, this is now flagged automatically by Horizon to SSC at Fujitsu for investigation. So far as the modern or more recent way that the system deals with failed recoveries, it is plainly the case that Horizon now automatically flags these. The fact that it did not do so previously, in my judgment, and the existence of all these PEAKs on this subject, clearly demonstrate a bug, error or defect, and that conclusion is not avoided by the fact that a later TC was issued.

225.

On 16 January 2017, the automatic monitoring service spotted a failed recovery and PEAK PC0256502 was raised by Fujitsu. A BIMS was issued to Post Office to correct the discrepancy. On 17 January 2017, the SPM in question contacted the helpline regarding the same issue. A second PEAK PC0256566 was raised. As quoted by Mr Coyne in his report, this PEAK clearly records that this issue was already identified by Fujitsu and resolved under the earlier PEAK. Hence the later PEAK in time was closed. However, although this example required corrective action by the Post Office (not by Horizon) it is not evidence of a bug. The former PEAK namely PC0197769 did evidence a bug. I also find that PEAK PC0214226 is evidence of a bug.

226.

Mr Parker’s evidence also makes it clear that this bug had an impact upon branch accounts. Mr Tank’s evidence demonstrates the same, albeit for a different date and time. This impact would persist until it was later corrected.

9. Reversals.

227.

This is a Legacy Horizon bug and occurred in 2003. It was not acknowledged as a bug in the Bug Table but is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2. It is however said only to have had transient impact.

228.

Dr Worden’s entry in the 2nd Joint Statement in relation to this stated “Transaction reversals are a complex area which, like recoverable transactions, are less familiar to Subpostmasters and are more prone to human error. They lead to many calls to the help line and to many KELs and PEAKs - not necessarily related to any fault in Horizon.” This can be seen as an example of Dr Worden attributing fault to the SPMs rather than accepting the presence of a bug, but given it is now accepted as a bug it is not necessary to consider that any further. Mr Coyne’s entry in the 2nd Joint Statement stated “In April 2003 due to a failure in regression testing, Horizon version S30 was released by Fujitsu and this introduced a bug where the value of transactions reversed by Subpostmasters was shown twice in the amount of the reversal in branch accounts.”

229.

This was clearly a bug. A Fujitsu document, PEAK PC0089918, identifies an issue in which attempted reversals of remming in transactions resulted in the magnitude of the transaction doubling, rather than the transaction being reversed. Initially, this is

blamed upon the SPM in question and the Post Office in its Appendix 2 continues this, submitting that “The Subpostmaster (branch FAD003227) was trading in Stock Unit Y and remmed in £13,910 of cash. He continued to trade in that stock unit in error, instead of moving to Stock Unit I. He therefore attempted to reverse all of the transactions he had made in Y (including the rem in). Instead of Y showing a balance of zero, the rem in had doubled to show a discrepancy of £27,820. Having spoken to NBSC, the Subpostmaster attempted further reversals but these failed and produced an error message indicating that the initial reversal had been completed successfully.”

230.

However, it was clearly caused by a bug. This is effectively accepted later in the submissions which state, correctly in my judgment, that “the issue was caused by a software error, which had been introduced as part of implementing the fix for PC0083954 in Legacy Horizon, and which incorrectly calculated the reversal value and quantity (essentially, the wrong mathematical sign was applied when reversing RIAD transactions (+ instead of -))”.

231.

If a minus arithmetical function is required, namely subtraction, but by software fault this actually occurs as an addition, then this is clearly a bug. It is certainly an error or defect in the system. This was then compounded by PC0083954, a fix which introduced a code change to ensure that a partial settlement of cash always tried to reduce the stack value. However, that fix did not work when reversing a rem in and so the solution was to pass the mode to the function also.

232.

A fix was required, and because of the potential impact on live branches, the development of a fix is said to have been “fast-tracked”. The Post Office submissions stated that “within 15 days of the issue first being raised, Fujitsu had implemented a fix to 2,178 branches. It is not known precisely when the fix reached all branches in the estate but Fujitsu believe it is likely to have been before 2 June 2003….” That again comes perilously close to introducing evidence by the back door through submissions.

233.

This was clearly a bug with potential lasting impact to branch accounts. The fact that the impacts of its occurrence were corrected by means of TCs does not affect its existence. It occurred for a short period in 2003. There is no evidence before the court of branches being affected by this outside that period, or of branch accounts affected by the bug that were not later corrected.

10. Data Tree Build Failure discrepancies.

234.

This is a Legacy Horizon bug. Its identified effect was during 1999 and 2000, so the very early days of Horizon. However, KELs relating to the PEAKs in those dates were created in July 2005 and updated in November 2007. A data tree is an accounting node hierarchy, or a cash account node hierarchy.

235.

It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. The PEAK dealing with this reads “Data trees have been failing to build fully, and the system has not been detecting this.” Data trees were part of Legacy Horizon, and were used to build a summary (or picture, the word used by the Post Office in its Appendix 2) of the accounts. The building of data trees is a software function, and given “the system” is

Horizon and is supposed to detect failures of this nature, it is difficult to see how it

could ever have been in issue that this was a bug. Mr Coyne’s entry in the Bug Table recites that “Dugannon branch suffered a £43,000 discrepancy but the cause was not immediately known. £52,814.29 at the Yate Sodbury Branch. £9,368.40 at the Appleby Westmoreland branch.” These are sizeable sums and arose in the branch accounts. That this is a bug was de facto accepted by Dr Worden in the Bug Table due to the text of his entry which states:

There was a bug which has potential impact on branch accounts, early in the lifetime of Horizon. Soon after it arose, the error was trapped and detected by DEP and was then soon fixed.

The fault was easily noticeable at branches before the error trapping which was soon introduced and would be even more noticeable after that. Only three branches appear to have been affected, as described by Mr Coyne.

Because it was so noticeable at the branch, and the Peak is concerned with a software error rather than any other cause, I would expect any discrepancies in branch accounts to have been corrected.”

(emphasis added)

236.

This is therefore undoubtedly a bug, but findings in respect of this therefore depend upon the nature of its impact on branch accounts. Dr Worden in his cross-examination changed the text of one of his entries in one of the appendices to his report, where he had said “This is further evidence of the failed recovery report doing its job - alerting [Fujitsu] to failed recoveries, so they can investigate them and make any necessary corrections to accounts.” He said this later part should be changed: “Perhaps I should have said have "guided the transactions to the correct state". He said it was caused by a complex grey communications failure, by which he meant an intermittent one, although he was not really sure where he had obtained the word “grey” from, he thought perhaps from a PEAK or KEL.

237.

The first PEAK PC0033128 related to an issue in November 1999 at Dungannon. The SPM reported a £43,000 discrepancy after balancing stock units and doing an office snapshot. An office snapshot is a Horizon report that showed the current accounting position of the branch, including any cash discrepancy. To produce the office snapshot, Horizon scanned the messagestore for the necessary information (eg. initial cash and stock levels, all cash and stock transactions, plus other service transactions). It then compiled that information together in order to produce the office snapshot. This is the “data tree”. On occasion Horizon would fail to read the necessary transaction data for various reasons so the report would be incomplete. This failure at Dungannon was caused by a missing payments node.

238.

Two other branches were affected by the same issue, Yate Sodbury and Appleby-inWestmorland. The snapshot at the former was incorrect showing a shortfall of £52,814.29 and Appleby’s snapshot showed a shortfall of £9,368.40. MSU were involved in the investigation of this and the Post Office submits that “it is therefore likely that the issues at Yate Sodbury and Appleby were resolved via a BIMS report and that the Subpostmasters were held harmless. However, due to the age of [this issue], comprehensive records are not available and therefore Post Office is not in a position to provide detailed commentary.” Two points need to be made. The shortfall was in a snapshot and not in a Branch Trading Statement; and secondly, there is no evidence of how it was corrected if at all. There are references in the PEAK in respect of Dungannon that shows the SPM and the Post Office agreed to remove the discrepancy from the account. For present purposes, although all these shortfalls are substantial, there is no evidence that the Post Office required payment of these sums from SPMs.

239.

In my judgment, the Post Office correctly identifies branch impact in the following passage in its submissions:

240.

“Of itself, Issue 1 [ie the issue above] would not affect branch accounts; there was no issue with the underlying transaction data and, if the office snapshot was re-run, it would very likely provide the correct information, because the data reading issue was temporary. However, if the branch ran the office snapshot, got an inaccurate report and then rolled over (making good any discrepancies in the process), then the shortfall would have an impact on branch accounts.”

241.

There is an associated issue in respect of data trees, namely PEAK PC0121925. As part of the changes made to move branch accounting from weekly cash accounts to monthly branch trading statements (a programme called IMPACT), changes to the data server were made to reduce the number of times that the messagestore was scanned to pick up transactions during balancing. A Riposte mechanism known as “Notifications” was used to add new transactions to the existing totals as further transactions were generated during the balancing process (rather than rebuilding the data server tree of transactions from scratch). One of the test branches experienced an issue, where the test branch had a gain of £45.05 following a cash declaration and rolling into branch trading. Initially, Escher were unable to replicate this scenario and so no further action could be taken. Subsequently, as demonstrated in a later PEAK in 2005, PC0123319 Fujitsu was able to replicate the scenario and implement a fix on 5 September 2005. The fix was implemented before any branches had switched to use the new branch trading code, meaning that the issue in PC0121925 could not have impacted any live branches. This is, however, a bug, error or defect, but not in the Horizon System as it is used in actual branches.

242.

Entry 10 in the Bug Table therefore does relate to a bug in Legacy Horizon which had an impact upon branch accounts. As the Post Office submits, in this instance the impact of that bug was corrected.

243.

Two PEAKs relate to a second element of this, one in February 2006, PC0132133 and another in March 2007, PC0144386. The Post Office accepts these PEAKs are essentially the same (one is said by the Post Office to be a manifestation of the other) and the former is said to “relate to an issue in which the notification mechanism referred to in PC0121925 was accidentally switched off”. There is no explanation of how or why this switching off occurred. The KEL that goes with these states in its title:

“Title: Multiple cash declarations may cause incorrect figures in Discrepancy, Variance and Balance Reports

Summary: Intermittent misleading figures in Discrepancy, Variance and Balance Reports”

244.

In the text in the KEL, certain passages make it clear that this is a software issue that persisted into 2007.

“Solution - ATOS

<b>Helpdesk:</b><br/>The Declare Cash problem clears itself overnight. If the PM logs a call on the day he is having problems, ask him to try the following workaround:<br/><br/>1. The clerk should log out of the affected counter.<br/>2. Another clerk attached to a different (individual, not shared) stock unit should log into the <b>same</b> counter, declare cash for his own stock unit, then logout.<br/>3. The first clerk can now login to the same counter and declare cash again. The variance should be correctly recalculated.

Alternatively log on to a different counter and do the cash declaration there. If the workaround is not successful or the problem does not clear itself overnight, send a call to SSC, otherwise no call is needed. November 2007: a fix is currently being piloted and is likely to be sent to the whole estate in January

(COUNTER_EPOSS 39_3 or later). If this problem is reported after

COUNTER_EPOSS 39_3 has been applied, send call to SSC.”

(emphasis added)

245.

A fix is only required if there is a bug, error or defect in the software.

246.

The Post Office submitted that there was “no long term impact upon branch accounts”. This was therefore a slightly different approach to “lasting impact” upon branch accounts. “Long term” is not defined. If the impact of this bug occurred prior to a BTS then correction by means of human intervention and TC would be required.

247.

I therefore conclude that this is a bug with the potential to cause lasting impact to branch accounts. The fact that the impacts of its occurrence were corrected by means of TCs does not affect its existence. The November 2007 entry in the KEL shows that a fix was required in late 2007 intended to be released in January (which must mean January 2008). Accordingly, it cannot be considered to have been limited in its effect only to the early days of Legacy Horizon and obviously persisted for some years.

11. Girobank discrepancies.

248.

This was a Legacy Horizon issue that occurred between May and September 2000. Mr Coyne considered this was a bug. Dr Worden did not and stated that “the first fault concerns reports. A fault in a report is not a discrepancy in branch accounts, and only causes one if it causes a person to make a mistake.” This is now essentially accepted as a bug by the Post Office, but it is submitted it had no branch impact. It is included in paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact.” The detailed part of Appendix 2 dealing with this submits that there is no evidence of any financial impact upon branch accounts, let alone a lasting impact. This was in the early days of Legacy Horizon. There are said by the Post Office to be six distinct issues arising under this heading.

249.

I do not know if what are called “giros” are still in use at all in 2019, but they look similar to cheques and used to be widely used, for benefits payments amongst other things. In basic terms, the giro (which is paper, and was sometimes included in a book of giros) would be exchanged for cash. A customer would present a giro for (say) £50

to the branch Post Office. The SPM or their assistant would take the giro, giving the customer the £50 in cash from the till, entering that transaction on Horizon. This would reduce the cash holding in the branch by £50. The giro would be sent to Girobank at the end of the day and that would be considered by Girobank together with the entries that would have come via Horizon. Girobank would then pay the Post Office the sum of £50 in respect of that particular transaction.

250.

If there was a difference between the value of the giros entered on Horizon, and the physical giros received by Girobank, then Girobank would not pay the Post Office the full amount. In other words, there could be a mismatch between Girobank’s view of the amount of money paid out in exchange for giros (based on the physical giros it received), and the Post Office’s view (and hence the amount of money paid out of the branch by the SPM to customers presenting giros).

251.

The summaries of the different issues taken from the PEAKs show the following problems:

1.

Girobank taking the view that there was a discrepancy between giros received from (say) Branch X, and cash paid out by Branch X for those giros. The explanation for why this occurred was the “cut off” time for post being collected from Branch X (which would include the giros to go to Girobank) being earlier than the end of the trading day. Branch X might pay out (say) £85 to a customer at 4.45pm, but the post (including all giro vouchers then available at the branch) might have left at 3.15pm or 4.30pm. This would lead to the difference explained at [250] above. It would also lead to an error notice being issued by Girobank. The Post Office’s submissions state that “in this particular case, the only possible impact would be if the branch had accepted the error notice received because of the reporting issue”. In other words, the only way the system would work correctly would be if the SPM did not accept the error notice. The Post Office have to rely upon a SPM not accepting an error notice issued to him or her to avoid any impact on branch accounts. That is not a software bug, it is true; it is however an error or defect in the system. The error notices are issued through or as part of the Horizon System. This type of incident is referred to in PC0044232, which concerned a £505.72 discrepancy. This was a known issue dealt with by KEL MWright531p. This KEL is now said by Fujitsu to have been deleted and irretrievable due to its age.

2.

A second issue was identified by Fujitsu in the course of investigation of the incident in PC0044232. This was that the same £81 giro deposit had been included on two consecutive daily reports. Part of the Post Office’s submissions on this are as follows: “This is because the transaction was entered onto Horizon in a precise (and very small) window of time between two system calls being undertaken, resulting in a duplication. The overall branch position would still have been correct, but the daily reports to Girobank may have been wrong. If they were (i.e. if the same transaction was included on two consecutive daily reports), it is expected that this would have been spotted and a TC would not have been issued to the branch.” This is, in my judgment, evidence of a bug, error or defect in the system, as the duplication referred to should not have occurred. There is no evidential basis for any “expectation” that if the reports to Girobank had been wrong, it would “have been spotted” and led to the issuing of a TC. No such TC has been produced.

252.

Some other issues relating to Girobank were identified by the Post Office separately, as issues 3, 4, 5 and 6 under the same heading of entry 8 in the Bug Table. Issue 3 applied to two PEAKs, both from 2000. They were PC0052575 (in which the SPM reported discrepancies of £20 and £628.25) and the issue was diagnosed as arising out of the use of a shared stock unit. The Post Office submissions were “There is a window of time between a user printing and cutting-off a report. If another user was to perform a transaction during that window, that transaction may not show on the report. The issue was already due to be fixed in a future release”. Mr Coyne accepted in cross-examination that these were indications of a discrepancy being identified, but these were not of themselves evidence of a bug in Horizon. His overall evidence on this was contained in his final answer which was “It created a financial discrepancy within the Horizon system which could then ultimately have an impact on branch accounts.” I accept that this issue caused a financial discrepancy. Given the issue was “due to be fixed” in what is described as a “future release” the issue arose from the operation of the system and is therefore, in my judgment, correctly described as a defect.

253.

Issue 4 was identified as follows in the Post Office’s detailed submissions on Entry 8 in Appendix 2:

“26. Issue 4 applies to PC0068633 (referred to at para 3.124). This Peak was raised in 2001 and so comprehensive records are not available.

27.

In that Peak, the Subpostmaster reported that his cash account showed two giro deposits of £1,503 but that his reports showed only one. The Subpostmaster received an error notice from Girobank which cleared the error, but he raised the issue because he believed that an error in Horizon was duplicating the transaction.

28.

The issue was caused by a cut-off being performed on one counter despite an attempt to print a transaction failing on another counter. This resulted in the cut-off report including the transaction that had failed to print.

29.

A fix was actioned by Fujitsu on 18 December 2001.

30.

Issue 4 only occurred in a very specific set of circumstances and would have had no direct financial impact on accounts; it merely had the effect that a transaction was missing from the reports.”

(emphasis added)

254.

In my judgment, the content of those submissions clearly demonstrate that this was a bug, error or defect which had an effect upon the report available to the SPM. This is made very clear in the PEAK itself, which states “I have duplicated this bug. In fact it occurs in all reports that use dataserver (i.e. the majority). I shall now check to see whether or not the problem still occurs at S10.” (emphasis added) The author of that PEAK was a Fujitsu employee and expressly referred to a bug. It did not have an effect upon the branch accounts of the SPM in question but it is undoubtedly evidence of a bug.

255.

Issue 5 applies to PC0073855 and PC0075312 from 2002. Again, there was not a great amount of material available due to the age of these PEAKs. A summary is as follows.

256.

In PC0073855, a SPM reported that the office snapshot figures were double the figures on the balance snapshot (with a £6.76 discrepancy). Fujitsu were unable to replicate the issue and were therefore unable to issue a specific fix. However, a new version of the component was released with extra tracing code so that if the issue reoccurred, Fujitsu could gather more evidence. The SPM would have had an inaccurate report but the correct data was still available and this would not have affected the branch when balancing. In PC0075312, another SPM raised an issue with printing her giro deposits. The issue was identified as being caused by the same root issue as PC0073855 which was already with the development team. These did not impact branch accounts. However – and I consider this to be important – they constitute evidence of a bug. The KEL with which they are associated is KEL AChambers4410R, the same KEL as the PEAK in Issue 4. The only reason the Fujitsu “development team” would be involved would be to develop a fix. A fix is only required if changes are required to the software, which means there is a bug.

257.

Finally, Issue 6 applies to PC0076065, which was that two giro deposits were reported by a SPM not to be showing on the previous or that day’s reports. The SPM identified the amounts of the deposits (£11 and £24) which were said to be missing, but Fujitsu discovered that SPM had produced two reports, and on this occasion it was the SPM who was in error. One report did not have the deposits on it, but the next day’s report had both. I find that this is not a bug and is an example of user error (that is, and to be clear, SPM error).

258.

There is an associated issue with giros which is identified in another PEAK, which is not included in the six issues. I shall reproduce the Post Office’s submissions on this

PEAK:

“The issue in PC0050418 was thought to be the same issue [as in [251](1) above] – the branch’s largest discrepancy was £323.32. However, due to the length of time it took for the issue to reach SSC, the branch’s messagestore had been archived – the Subpostmaster raised the call on 29 June and the call was sent to PINICL on 17 July. The Subpostmaster was told that they would need to provide further information (such as the Transaction ID) to allow Fujitsu to investigate. The Subpostmaster does not appear to have pursued this. The Peak notes that Girobank were going to send an error notice, but due to the age of this Peak the relevant records are not available and Post Office is not in a position to provide detailed commentary.” (emphasis added)

259.

In my judgment, that is prima facie evidence of a shortfall in the sum of £323.32 being caused to that branch’s accounts as a result of this defect or error, notwithstanding that the SPM reported it. The SPM was told further evidence was needed in order to investigate. It is a self-contained example that in my judgment supports the claimants’ case on the existence of a bug, error or defect affecting Girobank issues. The fact that the SPM for whatever reason did not pursue it does not affect that.

260.

Although the very numerous PEAKs identified in respect of this bug in the Bug Table share some common features, as can be seen above there are different issues in respect of many of the PEAKs. This was counted as a single bug in the Bug Table, even though my detailed findings above show that there were more than one bug, error and defect that related to Girobank. I will however for consistency count it as a

single bug in the overall total, as that is how the experts treated the total number of bugs in the Bug Table.

12. Counter-replacement issues.

261.

This also is a Legacy Horizon issue. The counter is part of the hardware of Horizon.

262.

There were two KELs associated with this dealt within the bug table. The first was created in 2000 and last updated in July 2007, and refers to 88 PEAKs. The second was created in 2002 and notes occurrences running up to 2009. It is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Mr Coyne considered that when replacing a counter within a branch, the process could result in “the total loss of a transaction”. Dr Worden stated that the cause, recorded in the first KEL (which was created in December 2000 and last updated in July 2007) was that Riposte was coming online from the Recovery mode too early, and causing messages to be overwritten.

263.

In theory, when a counter was replaced, it builds its messagestore by replicating with its neighbours in “recovery mode”. The neighbours it has depends on the office size (which would affect the number of other counters) and node number. For a single counter office, the neighbours are the correspondence server in the datacentre and the mirror disk (the second hard drive in the same counter). For a multi-counter office, the neighbours are the correspondence server and all other nodes at the office, or all the other nodes in the office (known as slaves) depending upon the node number of the counter being replaced.

264.

A replacement counter is supposed to come out of recovery mode when it believes it has successfully replicated all relevant messages from its neighbours. The Post Office submissions state that “In this case, the replacement counter came out of recovery mode early, before it had replicated all messages from its neighbour. The replacement counter started writing messages from the point at which it believed it had replicated all relevant messages from its neighbour. This meant that it used message IDs that had been used for messages that had not been replicated from its neighbour and this prevented the “missing” messages from being replicated later on (because that would have created duplicate message IDs). The missing message was therefore “overwritten” by the replacement counter.”

265.

The Post Office also stated that “the issue arose in cases of counter replacements where the new counter was not connected to all of its configured neighbours while rebuilding. This may have been because the branch infrastructure was not complete (eg not all neighbouring counters are online, multiple swaps or a counter increase/decrease occurring) or the engineer did not connect the system properly.” Regardless of the reason, the failure to replicate messages is a failure in the internal processes of Horizon to cope with the hardware replacement of the counter, and in my judgment is a bug, error or defect.

266.

PEAK PC0052823 gives a technical explanation for this, part of which is:

“It would appear that after the recovery from the squirrel completed, the message processor came out of recovery mode after synchronising up to message number 660 for node id 1. Suspect this was by replication with the Correspondence server.

Counter then wrote a Riposte on-line message as 661 for node 1 before 1 second later attempting to synchronise to the F: drive mirror message store. At this point a red event regarding 'self originated message' was generated, the server switched back to recovery mode and the remaining messages from the F: drive mirror message store above 661 were synchronised.”

It also notes that “Gareth Jenkins viewed this error on rig with Mike Berrisford.”

267.

The nature of the correction in the KEL was stated as being “to find the overwritten transactions for reconciliation we need to look at the Ripostemirror messagestore' followed by detailed instructions”. Riposte, as has been explained, was part of Legacy

Horizon and was a product provided by Escher, but it is plainly part of Horizon. Dr Worden also stated that “the incident arose from a hardware replacement (probably from a hardware fault) not from a fault in Horizon. It is a different kind of recovery issue.” In my judgment the hardware is part of Horizon.

268.

There was a short term and long term fix. The former dealt with the actual branch by the SSC inspecting the Riposte mirror messagestore, and retrieving the specific messages which had been overwritten. Information of the overwritten messages was passed to MSU who created a BIMS report for the Post Office. The Post Office submits that “an error notice would have been issued to hold the branch harmless thereafter”; no such error notice has been produced but I consider that the KEL and PEAK together suggest that such a notice was produced in order to correct the impact upon branch accounts.

269.

The long-term fix for the issue was an actual fix, which is detailed in PC0052823. This involved enforcing a minimum number of local neighbours for replication and then to slowly lower this number over time. A further change was also made to stop Riposte writing messages as it came online. This further change was a change to how Riposte operated. This therefore means that the fix changed the software of the system.

270.

The dates of other PEAKs shows that this persisted.

271.

PC0071836 is accepted by the Post Office in Appendix 2 as an example of the same issue as KEL JBallantyne5328R; this is dated November 2001. This SPM had a receipts and payments mismatch of £3.27 as a result of three overwritten messages

following a replacement of the branch’s single counter. The same fix was applied following KEL JBallantyne5328R. PC0133822 is not the same issue as JBallantyne5328R but is accepted as being related. This PEAK is dated 24 March 2006. The branch had two counters removed, leaving it as a single counter branch. However, the counter did not have a mirror disk; the mirror disk was a second hard disk within a single counter that provided a replication neighbour for the main hard disk messagestore. This meant that the branch would have no replication of data if it was not connected to the datacentre and six messages on the counter had not been replicated to the data centre. The messages were extracted and sent to MSU for a BIMS report to be raised. The Post Office submits that “An error notice would have followed the BIMS report and so there would have been no lasting impact on branch accounts”. However, that effectively accepts that absent the error notice, the branch accounts would have been affected. PC0153851 is not the same issue as JBallantyne5328R but it does involve a receipts and payments mismatch; it is dated 7 February 2008. Riposte failed to index four messages resulting in some items being missing from the receipts side of the balance report. The PEAK itself notes that the branch did not experience a discrepancy as a result because this was a reporting issue only; indexes are not used when replicating data and so cash/stock were unaffected. This does however show that the problem with Riposte persisted.

272.

I find that this was a bug, error or defect with potential lasting impact to branch accounts. The fact that it originated with the necessity to replace a counter (which is plainly hardware, as Dr Worden notes) does not matter because the hardware is part of the Horizon System. Nor does the fact that the impacts of its occurrence were corrected by means of error notices/TCs affect the existence of the fault. A change, or changes, had to be made to Riposte to stop this occurring, and there were still later manifestations of it occurring.

13. Withdrawn stock discrepancies.

273.

This is a Horizon Online issue. The Post Office does not accept these as a bug. Mr Coyne maintains that it is, and in the bug table he extracted part of a PEAK that stated these “Can cause confusion and unexpected (though hopefully temporary) discrepancies at branches by allowing them to declare stock which has already been withdrawn.” Dr Worden stated that “some impact on branch accounts cannot be ruled out, although it is small”. The Post Office’s detailed submissions in Appendix 2 of its Closing Submissions concentrated on the fact that when the Post Office withdrew stock – which of course is part of the way that the Post Office manages its business, adding and removing types of products from time to time – this removal of products is done by means of an update to reference data, “and not a change to the core code in the system.” This again is concentrating on the code, rather than the way the system operates.

274.

There were two steps that led to this happening. The Post Office submissions on this rather gloss over the entries in the PEAK, so I will reproduce the actual entries of 19 January 2011 in PEAK PC0207834. Although that is the date of PEAK, the text makes clear that the event occurred in November 2010. The context is that the Post Office withdrew the £5 saving stamp from its business, but obviously some branches retained (or had at the time they were withdrawn) such stamps in their stock:

“This office physically held 137 £5 PO saving stamps and did not rem them out before the date the rem out icon disappeared. The office physically returned the stamps to Transaction Processing as advised and the office then did a Trading Period balance on 17/11/2010 and showed this value as a loss. The spmr put the cash into the till at this point to make good the loss. Transaction Processing then issued a transaction correction to the branch on 19/11/2010 for the value of £685 to make good the loss. The office brought this to account and when they did the next Trading Period balance on 15/12/2010 it showed a gain of £537.49 which was settled centrally. The spmr says that they did have a loss showing again in the balance for the value of the PO Saving stamps, £685, again, even though this stock line was not showing in either adjust or declare stock. After this Trading Period balance the spmr took the £685 back out of the till and when they balanced again on 12/01/2011 it again showed a loss of 137 £5 PO Saving stamps and the nett discrepancy in the branch was £1370, which again was settled centrally. The office declares stock when they do the balance but insist there is no line showing in either adjust stock or declare stock for the PO Saving stamps. The spmr is convinced the £1370 loss, which is twice the £685, is due to the fact these stamps are still on the system somehow and keep giving the office the losses. The office has faxed a copy of their stock holdings for the branch at the moment and it clearly shows “Saving Stamps £5 137”. This stock line should have been removed when the stock item was made obsolete and removed from Horizon.

The two user names who do the balance are SKI001 and PCA001.

I have spoken to Phil Herrett in Transaction Processing who has confirmed he is aware of about 8 offices with similar issues with the stamps still showing on the stock. He gave me FAD 0640123 as an example.

Can this be investigated to see why the stock is still showing in the office, and if this keeps giving the office a loss of £685 every time they do a Trading Period balance.” (emphasis added)

275.

The Post Office submissions state that “However, the Subpostmaster in this case returned the stamps (with a total value of £685) to Post Office without first remming them out. This aspect of PC0207834 was pure user error.” However, that is not what the first entry in the PEAK states. It states that the SPM could not rem them out because the rem out icon had disappeared for this stock. That is undoubtedly an error by the SPM. However, the fact that the withdrawn stock remained on the stock holdings, which are part of the branch accounts, even after the stock has been returned and a TC issued to correct the discrepancy for their value, cannot be laid at the door of the SPM. Further, other offices (“about 8” of them) were also experiencing the same issue. This is an error or defect with Horizon in my judgment. Part of the submissions by the Post Office states as follows:

“8. The branch declared a £685 shortfall. The lack of a rem out meant that Horizon thought that the branch was still holding the stamps and therefore when the Subpostmaster either declared or adjusted the stock of stamps to zero without a corresponding gain in cash, a shortfall was generated. The Subpostmaster elected to make good the shortfall and a credit transaction correction was subsequently issued for £685 to rectify the issue.

9. However, a bug in Horizon caused the £685 of stamps to be subsequently reintroduced into the branch’s accounts on two occasions. By this point, Horizon was showing that the branch was holding £1,370 of the stamps. The Subpostmaster noticed this, adjusted or declared the stock as zero, and reported the issue.” (emphasis added)

276.

The Post Office therefore accept in those submissions that the subsequent effect upon the accounts was a bug in Horizon. I do not consider it much matters whether this is characterised as a bug, or an error, or a defect in Horizon. It plainly was a bug that had the potential to cause lasting impact to branch accounts (and in this case actually did so prior to the issue of the TC).

14. Bureau discrepancies.

277.

These relate to foreign exchange, hence the name. It must be differentiated from alleged bug at entry 23 in the Bug Table (also concerning foreign currency, but entitled Bureau de Change). This is a Horizon Online issue and arose in 2017.

278.

Mr Coyne considered it to be a bug, and Dr Worden effectively agreed in that his entry stated “This appears to be a system error with impact on branch accounts. Although it is possible that a subsequent discrepancy between branch accounting and POLSAP would reveal the problem, leading to a correction (e.g. see Peak PC0265443, and Mr Coyne's para 3.146), I cannot be certain of this.” It is now accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. The detailed part of Appendix 2 states that Mr Coyne has drawn together two distinct issues. Paragraph 4 of the detailed submissions also states the following:

“Bug 14: Bureau Discrepancies is a bug with the potential for lasting financial impact. There are two distinct issues which fall under this heading. With regards to the first issue the branch was made good and a fix was implemented. The second issue was not a bug in Horizon nor an issue which could have impacted branch accounts; it created what was essentially a cash flow problem for the branch.”

279.

So far as the first issue of the two identified by the Post Office is concerned, given a software fix was required, it is undoubtedly a bug, and it is correct to record (as the Post Office do) that there was the potential for lasting financial impact, as without that, there would be no need for the “branch to be made good” because that means the Post Office had to correct the branch account discrepancy caused by the bug. In a sense that is stating the obvious. Given those submissions, I will deal with the first issue very briefly.

280.

This occurred as follows. In August 2017 an SPM tried to pre-order £1,000.07 in Indonesian rupiah and £204.59 in Singaporean dollars for a single customer. The rupiah order was created, but there was a network timeout at the point when the SPM tried to perform the dollar order. When the system re-connected, a warning message suggested that the second order may not have been placed, but the basket and transaction log were showing both orders. The SPM attempted to cancel the whole order, but the cancellation only worked for the rupiah order, leaving the dollar order of £204.59 in the branch accounts. This is because as recently as August 2017 multiple currency orders were processed as multiple transactions. The rupiah order was added first at the counter and then sent to the BAL, and this caused the order ID PBX1048411 to be created. The network timeout occurred at this point – such a timeout has nothing to do with the SPM. At that time that the dollar order had been added to the counter stack but it had not yet reached the BAL. This meant that it did not become associated with order PBX1048411 and so, when the SPM attempted to cancel the whole transaction (PBX1048411), the dollar order was not cancelled. This then created a shortfall as Horizon will have expected the SPM to have taken a payment from the customer of £204.59.

281.

This issue could only occur because of the very specific circumstances of PC0261541 including the occurrence of a network timeout at a very precise moment in the transaction. The Post Office’s detailed submissions state the following:

“The PEAK notes that the issue was referred to Post Office in order for them to “decide what reconciliation or transaction correction is required to balance” at the branch. Post Office have confirmed that a transaction correction was issued to the branch on 2 November 2017 and accepted by the branch on 7 November 2017 – this is not apparent from the PEAK itself.”

282.

This means that the accounts of that branch were incorrect, and in the Post Office’s favour, for the period mid-August to mid-November 2017. A change to the software was made in November 2017 via a change to the AP-ADC script which had the effect of simplifying multiple currency orders to be processed as a single transaction.

283.

The second issue is even more marked, in my judgment, in terms of arriving at the correct answer to Horizon Issue 1. PC0265443 is dated 29 December 2017 but relates to a call on 17 December 2017. The Post Office’s internal cash management team’s system (which is referred to as POLSAP) indicated that the branch in Burgess Hill was holding €4,500 and US$1,000 more than the branch was declaring it held. This meant that, from the cash management team’s perspective, the branch held sufficient cash to conduct transactions, and so were reluctant to provide more cash to the branch. This resulted in a cash flow issue for the branch who were sometimes unable to serve customers as a result. This PEAK is 57 pages long; this is many pages longer than most PEAKs, which are usually 10 pages or less. It evidences a very long investigation.

284.

This investigation is best summarised by quoting the Post Office’s own submissions under the heading “Resolution”:

“13. The Peak is long and complex and it demonstrates that the issue was extensively investigated by Post Office, ATOS, Accenture and Fujitsu, since the issue was thought to be occurring somewhere between the branch, Horizon and POLSAP. The conclusion of Fujitsu and Accenture’s investigations (over approximately six months) was that the issue was not caused by any issue within their respective systems. Further, Post Office sent a trainer to the branch to verify the cash position and found no discrepancies; the amount of cash physically held in branch was the same as the amount showing on Horizon (but not POLSAP).

14. The root cause of the issue does not appear to have been determined. As it did not appear to be a case of user error, it appears that Post Office wrote off the discrepancy in POLSAP at no cost to the Subpostmaster to bring that system’s figures into line with those in Horizon.”

(emphasis added)

285.

This shows that POLSAP was not in line with Horizon; that no problem could be found by Fujitsu with its system, nor by Accenture; but that the amount of cash (in terms of foreign currency) in the branch was being correctly declared by the branch.

286.

Not only do I find that this is evidence that supports the existence of this bug in the Bug Table, which potentially causes lasting impact to branch accounts, but it is also consistent with my finding on a separate issue, namely the need to use ARQ data in circumstances where there are serious disputes between the Post Office and SPMs, rather than Post Office management data.

15. Phantom Transactions.

287.

These arise under Legacy Horizon in terms of the dates. There are three different issues grouped under this heading. The Post Office does not accept these as a bug. Mr Coyne in his cross-examination stated that this was referable to Horizon Issue 4, rather than Horizon Issue 1. He also accepted that some of the issues were not bugs in Horizon. Dr Worden relied upon the fact that what he called “the master PEAK” was closed as “no fault in product”; however, the evidence of fact on this is dealt with in the substantive judgment where I consider the evidence of Mrs Van Den Bogerd when the PEAK was put to her.

288.

Dr Worden stated that “There is no evidence for bugs in Horizon with impact on branch accounts.” That entry was obviously before the cross-examination of Mrs Van Den Bogerd, which in my judgment provided greater factual information. I do not consider reliance can be placed upon the Fujitsu conclusion in the PEAK.

289.

PEAK 0065021 is dated 17 April 2001. I consider it to be an important PEAK. It relates to multiple branches. It concerns phantom transactions. It identifies dissatisfaction from more than one SPM as to how phantom transactions are being investigated and resolved, or more accurately, how they are not being. It shows calls being “closed” – ie brought to an end, without the permission of the SPM, even though that should not be done. It also shows at least one SPM threatening not to comply with their contractual obligations due to lack of confidence in the system, and also threats of legal action. Further, in one branch, where items were the subject of phantom transactions (according to the SPM) ROMEC, the Royal Mail’s own engineers, attended that branch to fit suppressors and other equipment in an effort to rectify this.

290.

The PEAK plainly records the involvement of ROMEC, the Royal Mail’s own engineering personnel, as follows. “ ROMEC have been to site and state that they have actually seen the phantom transactions , so it is not just the PM's word now.” (emphasis added). I consider that this is significant. Mrs Van Den Bogerd agreed that this was “independent site visit corroboration of the problem by Royal Mail’s own engineers at the branch”, and she also agreed that this was “clearly not user error any more”. This entry in the PEAK is, in my judgment, important corroboration from those with experience of Horizon (the Royal Mail’s own engineers) who state they have seen the phantom transactions. The conclusion reached by Fujitsu and recorded in the PEAK was as follows:

"Phantom transactions have not been proven in circumstances which preclude user error. In all cases where these have occurred a user error related cause can be attributed to the phenomenon." The PEAK also concludes “No fault in product".

291.

I reject both of those conclusions. In my judgment, they are both self-serving and are contradicted by the factual matters reported within the PEAK by Fujitsu, which include that these have been witnessed by ROMEC engineers.

292.

A Fujitsu entry from 19 June 2001 in PEAK PC0065021 states:

I now have pressing evidence to suggest that unwanted peripheral input is occurring, the likely source being the screen. This has been seen at Old Iselworth (OI) and Wawne (W) with OI being the best site; when the PM has been asked to leave the screen on overnight I have observed system activity corresponding to screen presses happening with no corresponding evidence of either routine system activity or human interference, the way forward now is to correlate this with the Microtouch supplied monitoring software and to this ends Wendy is arranging for installation of Kit at OI on Friday, we can then, provided the PM agrees, leave screens on over the weekend and record what happens.

Once these results have been analysed I feel sure that we will be in a position to move forwards at OI. All other cases should be considered on their individual merits but you must appreciate that this is a fairly intensive analytical activity and I cannot hope to provide answers on all cases in the short term.”

293.

I consider this to be evidence of a bug in Horizon. Mr Coyne notes that there is no specific branch account discrepancies noted in this respect; however, given Fujitsu concluded in November 2001 the following “Phantom Txns have not been proven in circumstances which preclude user error. In all cases where these have occurred a user error related cause can be attributed to the phenomenon. I am therefore closing this call as no fault in product” then the lack of noted discrepancies for this is not surprising. Nor, in my judgment, is it necessary for there to be specific identified discrepancies given the word “potential” in Horizon Issue 1.

294.

Both the ROMEC engineers who had observed the phantom transactions, and indeed Mr Carroll in his entry of 19 June 2001, considered based on the evidence they had at that time that there were so-called phantom transactions occurring. Mr Carroll’s entry in particular that states “when the PM has been asked to leave the screen on overnight I have observed system activity corresponding to screen presses happening with no corresponding evidence of either routine system activity or human interference” can only sensibly be explained by a bug. The Post Office submitted that “For the reasons set out above [in its submissions], there is no indication that phantom transactions had a cause other than user error, as indicated in the PEAKs.” However, that ignores the entries in what was referred to as the master PEAK that indicate precisely the opposite.

295.

I consider that an inference, that is to say a common sense conclusion, can be drawn from all the evidence on these matters that there was such a bug in 2001 that caused phantom transactions. This had the potential to cause impact to branch accounts.

16. Reconciliation issues.

296.

This heading in the Bug Table grouped both Legacy Horizon and Horizon Online issues with reconciliation together. The Post Office does not accept these as a bug. There are a number of different issues grouped together in this heading. The issue was that the SPM was shown a discrepancy on his or her screen. Mr Coyne accepted that the discrepancy would not be shown in the branch accounts; Dr Worden stated that “as it concerns an issue in reporting, the software fault (which was fixed after 5 months) had no direct impact on branch accounts. The only effect of an error in this report would be to mislead or confuse the Subpostmaster - probably leading him to check his figures more carefully and costing him some time.” (emphasis added)

297.

That entry by Dr Worden plainly accepts a software fault; the fault was in fact a miscounting of the number of files by the system. However, there are six different PEAKs grouped under this heading and the process of reconciliation, which is effectively the comparison by Horizon of two different sets of data, is part of its design function. The recovery messages were held in the branch messagestore. Mr Coyne in his cross-examination also stated that this was referable to Horizon Issue 4, rather than Horizon Issue 1. Three of the six different issues were missed off the copy of the submissions lodged for oral submissions in July 2019, but this was made clear by the omission of “Issue 4” to “Issue 6” and following submission of Mr Parsons’ 19th witness statement these were added.

298.

This is one area where the cross-examination of Mr Coyne in particular does lead to a conclusion that this is not a bug that causes potential lasting impact to branch accounts. The six matters identified by the Post Office in the detailed part of Appendix 2 all deal with different PEAKs.

299.

Peak PC0039832 is dated 3 March 2000 and is an example of an issue affecting a SPM’s Cash Account Period. Reconciliation discrepancies appeared but did not feature on the expected reports. The counter calculated information each day for the Cash Account in two independent ways to ensure that they matched and a report was generated if they did not match. In this case, the reconciliation software detected discrepancies relating to two low value transactions (£8.06 and £0.08, totalling £8.14). The PEAK suggests that “there was a bug in the reconciliation software, although the Peak is not fully conclusive” as the Post Office’s submissions recognise in this sentence that I have quoted. This would have generated a false reconciliation report but there would have been no financial impact on the branch. The reconciliation error was captured, as that is what triggered the investigation in the PEAK. The fix is documented in PC0047955 of 19 June 2000. It is evidence of a bug but given the nature of it, it would not have impacted upon branch accounts.

300.

PEAKs PC0075240, PC0075415 and PC0077508 are all from April 2002. These all relate to the same (or at least a markedly similar) issue where a branch counter total differed from the amount on the TPS host. There was no discrepancy in branch accounts. In all these cases, the counter calculated a value of 1p and the calculations carried out at the host gave a value of £0.0099. As the first three digits were 000 the system took this to be 0p and so reported a false discrepancy. There was however no financial discrepancy. A fix was however required, which demonstrates it was a bug, error or defect, but it was not one with the potential to affect branch accounts.

301.

PEAK PC0049578 is from July 2000. It was raised in testing and fixed before the software went live. It was a bug, error or defect but the purpose of testing is to discover such things, and as such this had no potential to impact branch accounts. The problem was producing a report used to confirm that all data has been passed to TIP. The underlying data transferred to TIP was correct, however, the number of files transferred did not match. There was also a report to confirm what files were transferred, and that report was also incorrect in the count of files transmitted. If this had gone to live then the answer to this might be different, however regardless of that, it was fixed before release.

302.

The fourth issue relates to a PEAK dated May 2000. The submissions stated that “the

Peak describes an automatic detection of a receipts and payments mismatch of £4,464.46. It was automatically reported to Fujitsu and was investigated. The issue related to a hardware failure. It therefore did not relate to reconciliation issues arising as a result of a software fault.” It is however a defect or error with the Horizon System, which comprises both hardware and software.

303.

PEAK PC0236246 is dated 12 September 2014. It relates to a discrepancy of £110,706. The Fujitsu entry in the PEAK is somewhat illuminating:

“This problem is likely to occur whenever products are introduced with non-midnight times, a common occurrence when data is created (and subsequently corrected) in MDM. Based on current data, such products may be seen roughly once a month.

When it does occur, any transactions against the products are omitted from the CTS report, resulting in discrepancies in trading totals, and operations are required to investigate, usually requiring the assistance of development.

The CTS Report is used by POL to settle payments with their Clients. Incorrect values on the CTS report cause Post Office Clients to lose confidence in POL Accounting resulting in loss of face to our Customer. POL also therefore have loss of confidence in the precision of our IT Systems and this is an especially sensitive issue whilst POL are being investigated for accounting accuracy.

POL’'s lack of confidence in our IT Solution will not help our bid for the renewal of the Front Office contract.”

(emphasis added)

304.

CTS stands for Client Transaction Summary. In my judgment, the error shown in this PEAK is plainly a software fault that affects the accuracy of accounting at the Post Office. However, in this case, it has an effect upon the CTS Report, used to account between the Post Office and its clients, and not branch accounts. There was therefore, under this heading, a software fault, however it had no potential to impact upon branch accounts for the reasons I have identified above.

305.

The final of the six different issues also relates to the CTS Report, but one of the two PEAKs is that which I have already reproduced at [303] already. The other is PEAK PC0204872 and is dated 4 October 2010. The Post Office submissions state that “the issue relates to a difference between the CTS Report and the branch figure. The root cause identified was an issue with POLSAP, and not with Horizon”. The PEAK records that there was a problem with the CTS report for a branch concerning the Alliance & Leicester:

“The CTS report is received daily and is compared with the vendor (in this case A&L) reports. The figures for each day should match.

If the CTS report is larger than the vendor figure, the vendor account will be credited. The credit usually shows a couple of days later as a positive discrepancy.

The CTS report was showing as being larger than the vendor figures on the following dates, although there does not appear to have been any counter credit showing on the vendor figures following on from this:

7th May 2010 - CTS was greater than vendor figures by £84.86. POL have suggested that this may have been related to an event from 27th February for FAD 490519, although we can find no BIMS record of this from a Reconciliation perspective.

25th July 2010 - CTS was greater than the vendor figures by £3,260.00. No additional information is available.

27th August 2010 - CTS was greater than the vendor figures by £846.00. No additional information is available.”

306.

The next entry is:

“Following this additional information this would appear to be an issue with POLSAP.

The SSC do not support POLSAP, so we can take no action here.”

307.

That submission and the entry in the PEAK is of assistance in dealing with the ARQ data/management data dispute, but does not necessarily evidence a bug in Horizon potentially impacting branch accounts. It does not, however, for obvious reasons, take the matter very much further forwards.

308.

I find that these PEAKs, although demonstrating software faults, do not show that there is a bug with the potential to cause impact to branch accounts

17. Branch Customer discrepancies.

309.

This was a Legacy Horizon issue and is an entry only in respect of Horizon Issue 4 and recorded as such in the Bug Table. The Post Office does not accept these as a

bug. Mr Coyne accepted in his cross-examination that the entry in the PEAK did not suggest any impact upon branch accounts. The entry in the bug table originated in a single PEAK, although there were two PEAKs that relate to the same issue, which was a discrepancy between the financial records held by the Post Office (from Horizon) and a bank’s records, after a counter crashed whilst a transaction was being processed. The amount in question was £165.26. The date of the PEAK is 29 March 2008 and the incident occurred between 22 March 2008 and 26 March 2008 (the date of the card account).

310.

The SPM did not take the sum in question from the branch customer, but that person had their account debited. The first PEAK was raised by Fujitsu who picked up on this as part of automatic reporting (an NB102: Exception Summary report) and the SPM was advised how to go through recovery. Fujitsu had issued a BIMS Incident Report to the Post Office, and had confirmed that the recovery messages were in the branch messagestore (even though they do not appear to have been shown by Horizon to the SPM in the first place). However, the branch appeared in the NB102 report again, and upon investigation this was found to be because the SPM (quite properly, in my judgment) had not followed the recovery messages because they had not taken the sums from the customer.

311.

This is effectively accepted by the Post Office in its submissions which state: “Fujitsu issued a second BIMS Incident Report to Post Office which advised that while the recovery messages were received by the SPM, the SPM had declined them. This indicates that no money changed hands during the transaction (i.e. the SPM elected not to recover the transaction because no money had changed hands; if money had changed hands, the SPM should have elected to recover the transaction).”

312.

The Post Office also submits the following:

“While there was a possibility that the Financial Institution registered a withdrawal from the customer’s account (depending on how the Financial Institution’s IT system is configured), this would have caused a loss for the customer, not the branch (assuming that the customer received no cash from the branch). If the customer did receive cash from the branch, the resultant discrepancy would have been caused by the branch not following the recovery procedure correctly.”

313.

I find that a very surprising submission. The counter – part of Horizon – crashed during a transaction. That is not the fault of the customer, and it is not the fault of the SPM, nor is it the fault of the customer’s bank. I do not know if solace is taken by the Post Office that any loss caused to the branch customer can, apparently, be laid at the door of the customer’s branch, but I doubt solace can be properly taken. Nor do I see how the SPM can be criticised for “not following the recovery procedure correctly” as it is accepted that this should have occurred if the SPM had taken money from the customer, but if they had not, it should not have been followed. On either analysis, this is a fault with Horizon.

314.

However, it does not appear to have occurred more than this single occasion. I agree with Mr Coyne, and this is relevant to Horizon Issue 4 only. I consider that it is an error or defect in Horizon, but does not constitute one that had the potential to cause impact to branch accounts.

18. Concurrent logins.

315.

This is a Legacy Horizon bug. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. In paragraph 2 of the detailed part of Appendix 2 dealing with this bug, the following is stated by the Post Office: “Post Office accepts that Bug 18: Concurrent Logins had a potentially lasting financial impact. There is no evidence of any discrepancy in the PEAKs referred to by the experts.” The problem was in the early days of Legacy Horizon when it was possible for users to log in to two terminals at once. Dr Worden in the bug table made the following useful summary statement:

“discrepancies could occur - manifesting themselves as a receipts/payments mismatch. This had the potential to affect branch accounts. The mismatch would bring it to the attention of the Subpostmaster, who would require it to be investigated, except possibly in the case of small mismatches, which he might pass off as an error in the branch (e.g. of counting stock).”

316.

He also stated that “….Fujitsu believed it was a problem with the underlying Riposte software, and passed it to Escher. In September 2000, the problem was 'Now formally fixed in Build 223 update 19 which was released overnight.' However, the new release from Escher did not, as it was expected to, fix the problem. Escher denied that it was a bug in Riposte, but Fujitsu believed in July 2001 that 'This is clearly a bug in the Supplier's code'.

(emphasis added)

317.

Paragraph 2 of the detailed submissions by the Post Office in its Appendix 2 state:

“[The] Post Office accepts that Bug 18: Concurrent Logins had a potentially lasting financial impact. There is no evidence of any discrepancy in the PEAKs referred to by the experts.”

318.

Dr Worden considered that “these faults had the potential to produce discrepancies in branch accounts, of small amounts, for a short period of time”. How the bug is therefore characterised depends upon my findings in relation to lasting and transient impact, which is dealt with in the substantive judgment to which this is an appendix. I find that this was a bug, with the potential to cause lasting impact to branch accounts. Dr Worden attempts to play this down and so, in my judgment, does the Post Office. Fujitsu clearly recognised and recorded that in 2001. The fact that the bug was in the code supplied by Escher, as part of Riposte, and not that of Fujitsu is wholly irrelevant for the purposes of the Horizon Issues.

319.

One of the PEAKs related to the printing of a report whilst a counter crashed. The Gateway counter was then used, and because the first counter had crashed (what is called a “slave” counter) this was permitted because it could not communicate to the Gateway counter who was logged in to it, or what it was doing. This PEAK is PC0027581 and relates to an incident in July 1999. That PEAK runs to February 2002. It is notable, and continuing the theme of Fujitsu’s closure codes, that it is closed with the code “Administrative Response” even though the entry for 13 July 2001 states “This is clearly a bug in the Supplier’s Code and as you say, management pressure must be brought to bear as necessary to make the Supplier accept and respond to that fact.”

320.

The second PEAK is PC0051327 dated 2 August 2000. The entry which opened the PEAK is:

CALL PC0051327:Priority B:CallType L - Target 02/08/00 12:32:13

28/07/00 12:22 office 182432 reports for CAP17 a receipts total of £412224.58 and a payments total of £430724.58. the difference is £18500.00. this office earlier raised a query because a transfer for an amount of £9250.00 seemed to have gone missing. the amount of the transfer is exactly half the amount of the difference between the receipts & payments. a call raised earlier PC0050974 was closed in error.

28/07/00 12:27 GB082158

Information: passing to EDSC as requested.”

(emphasis added)

321.

A further entry is:

“PM advises that the problem started when she transferred from counter No8 (by mistake - should have been from counter No1) £5590 to counter No3 and £3660 to counter No4 - while No8 was in the process of rolling over! Consequently, the cash left counter No8 in CAP 18 and arrived in counters Nos3 and 4 in CAP 17. She eventually rolled the office, accepting the £9250 discrepancy and then transferred

£9250 from counter No1 to counter No8. How was she able to transfer out of counter No8 while it was rolling over?”

322.

Part of the Post Office submissions are:

“19. The net difference of these receipts and payments mismatches amounted to zero, meaning there was no overall impact on the branch. This would have been clear to Mr Coyne as the very nature of this issue is not that it causes money to be lost, but money to be accounted for in the incorrect time period. The PEAK shows that there are offsetting receipts and payments gains and losses and therefore it is possible to ascertain from the Peak whether the branch suffered a discrepancy.

20. The issue was fixed through a planned software roll out that changed the code in the area that caused the bug (release CI45) and the call was closed on 30 November 2000.”

(emphasis added)

That applies to the second PEAK but not to the one headed “Simultaneous Logon” which is PEAK PC0027581, which was closed in February 2002.

323.

I find that this was a bug in Horizon which plainly had the potential to cause impact to branch accounts. The fact that in this specific case the discrepancy was later corrected does not, in my judgment, alter that characterisation. Indeed, on the Post Office’s own case, the accounts were affected (albeit corrected). Further, the way that PEAK PC0027581 unfolded – a lengthy PEAK which runs from July 1999 to February 2002 – shows the unsatisfactory way some of these incidents were dealt with at Fujitsu. The entry of 9 November 1999 states “….at first glance it could have a fair bit of business impact”. Even as late at August 2001 and January 2002 this PEAK was still given the code “Incident under Investigation”. One wonders quite how long Fujitsu would need properly to investigate something identified as “a bug in the Supplier’s Code” in July 2001. The final entry contains the sentence “Mr Lui [who is the SPM] is no longer employed by the Post Office and has not been for some years.”

19. Post & Go/TA discrepancies in POLSAP.

324.

This is during Horizon Online and occurred in 2012. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had “transient” impact. The entry by Mr Coyne in the Bug Table relates to Horizon Issue 4 rather than 1. “Post & Go” are self-service terminals, that are no longer available in branch Post Offices, and are only available in Crown and WH Smith main branches. They are basically self-service kiosks, designed to avoid queuing. A customer can weigh a letter or parcel, the terminal will print the relevant stamps/labels, the customer deals with that themselves, and the item is then posted without the need for queuing. The Post Office submitted that this issue was irrelevant to the Horizon Issues trial as “this issue does not relate to branches that are the focus of this trial”.

325.

I do not consider that this is irrelevant as the Horizon Issues require consideration of bugs, errors and/or defects in the Horizon System.

326.

One of the PEAKs shows that Post and Go transactions were being accurately recorded by the relevant terminal and then transferred to Horizon. Horizon creates a

TA (or transaction acknowledgement) of all the transactions on the P&G machine and, when accepted by the branch, the value of these transactions are added to the branch accounts. The data is also passed from Horizon to various other systems, including POLSAP. This incident was caused due to an issue at the reconciliation step within POLSAP. Fujitsu transfers the relevant data to POLSAP via BLE files. When POLSAP was comparing the Horizon data to the Wincor data, discrepancies were found as data for two (of the total of six) terminals in the branch was not being sent to POLSAP. The Post Office submissions state “There were six P&G terminals at the branch. Data from four terminals only were being transferred to POLSAP because the two other terminals were not associated with conducting P&G transactions”.

327.

This is not an accurate summary of the PEAK. This states:

“An example the customer has provided shows amounts of 115.05, 46.88, 52.13 & 75.23 totalling 289.29 received on the file from Wincor and into POLSAP via BLE.

The same (contra) amounts are also showing as being received from the branch when the TA has been accepted and are closed items in the account (netted off to 0.00).

However, there is another amount of 289.29 which just has the date in the assignment field.”

328.

The Fujitsu response is:

“Postings on the TfS call refer to a similar previous incident (A1040049 => Peak PC021943293), which was resolved between POL and Wincor Nixdorf; no details of this resolution are available to us. This incident is a week old, but only came to SSC late last night… The trading-date in this call, 2012-08-09, is three weeks ago which too old for us to be able to see the incoming file from Wincor Nixdorf… There is no evidence of a fault in HNG-X, and without the incoming file from Wincor Nixdorf there is nothing further for us to investigate.

We can only suggest that POL do the same as they did with A1040049, and refer the matter to Wincor Nixdorf.”

329.

Anne Chambers records that:

“Branch 020511 has many entries in the Subfiles_on_hold report. This report should be monitored (by ?) to make sure problems are followed up - this should be resolved before closing this call.

Horizon is receiving PG data for 6 separate PG tills at the branch, but only 4 of them have associated stock units. This causes the entire subfile for the branch to be Held, and the transaction data is not being sent to POLSAP. However the TA data for the 4 tills which are properly associated IS being sent through, and I think this is probably the cause of the POLSAP anomalies.

The two unassociated tills are not doing any cash transactions - this is a known problem (see PC021870294), and means the PM isn't prompted to create an association. This may need fixing via MSC.”

330.

It was not that two of the six terminals were not doing Post & Go transactions, which is how the Post Office submissions seek to have the PEAK interpreted. It was that they were not going cash transactions. There are types of transaction associated with Post & Go that are not cash transactions. They plainly are all six of them Post & Go terminals, as the PEAK records. If they were not doing Post & Go transactions at all, there would be no problem because there would be no subfiles on hold, or held.

331.

As Mr Coyne said in his supplementary report “a bug fix to the Horizon system was identified by Fujitsu, scheduled for implementation 13 September 2012 after 1800hrs and the Branches stock was to be corrected at 1700hrs that same day.” On 17 September 2012 Anne Chambers herself reported in the PEAK that:

“Following a change made centrally to facilitate this, the stock unit associations for the two new Post and Go terminals have been created by the branch and all the held external data (43 different days) has now been processed and passed through to POLSAP… We strongly recommend that POL monitor the SubfilesOnHold report which is sent to them daily, so that any other external terminals with problems can be investigated quickly in case a similar correction is needed”.

332.

The “change made centrally” can only have been to the software. It is also the case that the underling bug, error or defect was having an effect for 43 days, which is in excess of one trading period which are four weeks long (and occasionally five weeks) but both those periods are shorter than 43 days. It is also the case that POLSAP was wrong for that period, which is a factor which impacts upon the dispute between the parties about the use of ARQ data/management data. The Post Office submits that it was “quickly resolved” but I reject that. 43 days, or over 7 weeks, is not resolving this bug (or its effects) quickly.

333.

I find that this is a bug with potential to cause impact to branch accounts. I do not know the date when the Post & Go service was withdrawn from branch post offices, or how many branches had it during the period when it was so available. The statements made in respect of this in the Post Office submissions are said, in the clarification table, to be based on “instructions from [the] Post Office”. The information is vague. I accept that this service is only now available in WH Smith and Crown branches, but other details cannot be specified because there are none available from any source.

20. Recovery Failures.

334.

The entry in the Bug Table identifies this as relating to Horizon Issue 4. These arise under Horizon Online and not Legacy Horizon. The Post Office does not accept these as a bug and states that there is no evidence of a bug in Horizon; it refers to the subject matter of the different PEAKs as “issues”. There are three different PEAKs relevant to this. Mr Coyne accepted in his cross-examination that this (or these) should be removed from the table that he used as the originator for his list of bugs (which was Mr Coyne’s Table 1). Dr Worden stated in the bug table that “there was some implication of hardware faults, with a replacement of a base unit, but the PEAK has no evidence of software faults in Horizon.” In one of the PEAKs, he concluded that there was no evidence of any fault in Horizon.

335.

The situation in terms of PEAK PC0220532 is unsatisfactory. The Post Office submit that this “relates to a cash discrepancy that was only raised by the Subpostmaster with Post Office and Fujitsu nine months after the issue occurred.”

336.

It is therefore clearly submitted that this took nine months for the SPM to report at all. This is based on the date of the PEAK (15 September 2012) and the date given in the very first entry that this happened on 6 January of the same year. However, that is simply incorrect on the face of the document. The date of the PEAK does not mean that the SPM took that period to report the matter to the Helpline or their manager, for example. Reporting in January by the SPM is in fact recorded in the PEAK itself, but not in the first entry. Further, that submission ignores these entries later in the PEAK for 6 September 2012 that refer to a TFS call being raised on 7 January 2012:

“However I have checked our call logging system which has provided some useful information. According to our archived TFS call login system, the Node: 5 did have ?Physical memory Dump? on 6/1/12 and a TFS call (5043049) was raised. The node was swapped on 7/1/12.”

On 13/1/12, PM raised another TFS call (5060273). PM reported that she had a loss of £300, she wanted to know if this was related to the Base unit fault. PM was referred to NBSC to investigate the discrepancy.

On 13/1/12 NBSC raised a TFS call (5060420) and asked for the discrepancy to be investigated.

On 13/1/12 @16:32pm, the frontline helpdesk contacted the PM and the following information was logged on the call.

?Spoke to the PM who states that she had the loss before the base unit was swapped out the loss was for 190.00 PM states that she rolled over her TP on the Wednesday which had no loss or gains. PM states that she then did a transaction log on the 5th Jan this shows a loss of 190.00 The base unit got replaced on Saturday 7th Jan.

PM states that she had a different error message on the system every day but did not record the error message PM states that she had physical memory dump message come up on the Friday 6th Jan and the base unit was then swapped on the 7th Jan?” (emphasis added)

337. Later entries in the PEAK state:

“Assuming the cash declaration on 04/01/12 was correct, NBSC will need to go through the transactions that were done in BB during this period (possibly by use of Transaction Log), to try to determine where the discrepancy came from. The fact that on 04/01/12 there was a negative cash discrepancy balanced out by a similar but positive stamps discrepancy, may be related to this issue.?

It is not clear what investigation/s were carried out by NBSC and no further TFS call was raised with regarding to this discrepancy.

If further investigation by Fujitsu is required, Post Office will have to request that the branch transaction data is retrieved from the audit server. If there is any possibility that this is required for litigation, it must come through the Security (ARQ) route.

Otherwise queries of this nature should be sent via Mark Wardle at POL, and should be routed to the reconciliation team in the first instance. Such requests may be chargeable.”

338.

This trial is not to determine individual discrepancies, although they are identified where relevant to the Horizon Issues. On the information before the court on this item, it is not possible to resolve this. This means that there is insufficient evidence to make a finding of a specific bug in this case, or on this PEAK.

339.

The second issue under this heading related to PEAK PC0241242 on 23 February 2015. This caused a discrepancy and the Post Office submits that “a Transaction Correction was done for the shortage”. The shortage was £70.66, and the claimants challenge the Post Office on the origin of this positive statement that this was done. In clarification, the Post Office relied upon Fujitsu stating in the PEAK that the Post Office needed to do this. The PEAK states on 5 March 2012 at 1546 hrs:

“This office was doing the following cash withdrawal txn for £70.66 on 23/2/15 @10:46.

The session also contained a non financial Health Lottery txn.

The settlement failed due to poor communication with the data centre. The disconnected session receipts were printed which advised PM to pay the money out. Spoke to PM who confirmed that the money exchanged hands.

When PM tried to log back the recovery kicked-in. However the Health Lottery APADC recovery script failed and left counter unusable.

Yesterday POL Branch Support team authorised us to remove/update the session.

Today I have carried out and completed the task. PM is now able to use the node again.

Reconciliation needed for the banking transaction:

The cash withdrawal txn was authorised and PM said they paid the money out.

This will leave this office £70.66 short (cash shortage) as the session not completed fully. POL need to do appropriate reconciliation; transaction correction.”

340.

However, a later correction in the PEAK for 5 March 2015 at 16.06 states “MSU please note that it was a CAPO cash withdrawal for £140 and NOT £70.66.” This is repeated at 16.12. The entry for 9 March 2015 states “I have also advised MSU to do the necessary reconciliation and raise BIMS with POL for the Cash Withdrawal txn for £140.”

341.

It is clear therefore that the root cause of the issue was the failure of the AP-ADC recovery script (Automated Payment Advanced Data Capture script, which is written by Atos). This therefore directly shows that there was an error or defect in Horizon, and this led to a potential impact upon branch accounts.

342.

For the purposes of the Horizon Issues trial, this is relevant to Horizon Issue 4, as explained by Mr Coyne.

343.

The third issue shown in another PEAK is PC0197643 dated 14 April 2010. This is a failure of automatic recovery. This will inevitably happen in any system, as accepted by Mr Coyne.

344.

The Post Office submitted in respect of all three of these PEAKs the following:

“For the reasons set out above, this is not a bug at all. There are three distinct issues under this heading. There is no evidence of a bug in Horizon and in any event, instances of any issue were caught by automatic reporting.”

345.

That is correct as far as it goes – the PEAKs are not evidence of bugs. The first two are however evidence of errors and defects in Horizon. They do however go to Horizon Issue 4, not Horizon Issue 1.

21. Transaction Correction Issues.

346.

This arises under Legacy Horizon. This is de facto accepted as a bug by the Post Office, but it is submitted it had no branch impact. Mr Coyne’s entry in the Bug Table was that “Transaction Correction bugs/errors and defects do not cause discrepancies with branch accounts but” and he then listed various consequences. In his report he referred to “technical flaws”. He also in his cross-examination accepted that he did not consider this to be a bug with lasting financial impact. Dr Worden, in respect of some PEAKs under this heading, accepted they were evidence of a bug, such as PEAK PC0129587 where he stated “In my opinion this bug would result in an inconvenience to the Subpostmaster (inability to rollover to the next TP) but would not result in inaccurate processing of any TC, or any impact on branch accounts” (emphasis added). The bug is included by the Post Office in the list of paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact.”

347.

The submissions by the Post Office is that there is “no evidence of any financial impact upon branch accounts”. It is therefore effectively accepted that it is, or there were, bugs responsible for these issues, but it is also effectively agreed by the parties that there would have been no financial impact upon branch accounts.

348.

Transaction Corrections were introduced in September 2005, replacing the previous method which were called Error Notices.

349.

Some of the PEAKs, such as PC0120459 dated 4 May 2005 and others from around that time, refer to issues with the TC button on the screen at that terminal in a branch not working as it should. The so-called “pick list” shown on the screen would freeze. These issues were logged by the team at Fujitsu called the Solution Validation and Integration team. This was part of testing. None of the documents referred to by the Post Office in its submissions to justify the submission “As such, these PEAKs all relate to issues raised by users during testing prior to the TC functionality going live in September 2005 (when it replaced error notices)” were referred to in the trial. I do however accept that they show that the TC process was tested prior to release to the whole branch network, and other PEAKs also show this. The Post Office also submits that this was not “a bug at all”. If that is right, then it must have been an error or

defect, but given that the documents show that it was fixed during the testing process, the precise terminology probably does not matter.

350.

Other PEAKs however relate to the period after September 2005 when the testing process was over. PEAKs PC0129587 and Peak PC0130056 refer to the Petersfield branch, and are dated 4 December 2005 and 14 December 2005. They both relate to the same issue and the latter is what is called a clone of the former. The first PEAK relates to a TC for a £9,000 debit and part of that PEAK states:

“01/12/05 16:01 Caller states that each time they try to do a transaction correction the counter is freezing up and going to system busy, has attempted on 3 different counters and the G/W was on system busy for about 2 hours, pm has had to reboot 3 times attempting to do this.

01/12/05 16:07 uk955556

Advice: Asked pm to provide details regarding transaction and fault.

01/12/05 16:08 uk955556

Information: pm has attempted to run this correction on 3 counters now, 1, 5 and 2 and each time the counter is freezing up and going to system busy, will not come back from this.

01/12/05 16:09 uk955556

Information: Transaction was for a premium bond on the AP system.

PM was working on stock unit - E at the time.

01/12/05 16:10 uk955556

Information: Transaction was carried out on 29th October at 12.23.

01/12/05 16:10 uk955556

Advice: Advised pm that i will have to look into this and get back to her.

01/12/05 16:10 uk955556

Information: pm is fine with this but states if it is not resolved by 14th December she will be unable to rollover.

01/12/05 16:32 uk955556

KEL Ref No.: checked for kel's but all kel's found on transaction corrections refer to specefic events which are not present in this case.”

The lack of previous KELs on this subject at that point means that this was a new issue which had not arisen before. It cannot therefore be put down to testing which was completed by then.

351.

A later entry on 6 December 2005 is:

“Several interesting points to note:

(a)

111111111 subscription group on core server contains 35 more msgs than on the counters: 3280 compared with 3315 msgs. The extra msgs are just Bureau: CurrencyRates and CSvcMargins msgs.

(b)

A couple of times after getting tired of waiting for the TC to process, the Clerk did a session transfer to another counter which, according to EP/SPG/001, should not be possible.

(c)

I have reproduced the problem by importing the mstore onto the SSC counters and setting a cut off date of 01/12/05 11:27:10 which was before the PM?s first attempt.

(d)

Peak PC0120459, raised on S80 E2E XI, reported the same symptoms and the root cause was found to be missing/incorrect ref data. This call was investigated by Mike Coon.”

(emphasis added) (question mark present in original as “PM?s”)

352.

“Ref data” means reference data, which is part of the software of Horizon. In my judgment this shows clearly that this was a bug. This was therefore known about by Fujitsu in 2005.

353.

I shall quote some paragraphs of the Post Office’s detailed submissions on the next PEAK:

“10. Peak PC0129587 relates to a report by a SPM on 1 December 2005 that each time she went to select a TC for a £9,000 debit, which arose from an incorrect entry for Premium Bonds, her screen froze and she was unable to accept it.

11.

Peak PC0129774 and Peak PC0130057 refer to the Bosham branch. These PEAKs relate to a report by a SPM on 6 December 2005 that each time she went to select a TC, a £22,500 debit for an incorrect entry for Premium Bonds, her screen froze and she was unable to accept it.

12.

Mr Coyne’s analysis proceeds on the basis that this issue was only experienced by these two branches. Mr Coyne fails to refer to KEL LKiang2837P [dated 5 January 2006] which demonstrates that: 50 other branches reported an issue with their screen freezing to Fujitsu during December 2005; 48 branches reported that this prevented them selecting an outstanding Camelot Lottery TC and rolling over into the next trading period; and 4 branches reported that this prevented them selecting an outstanding Premium Bond Sale TC and rolling over into the next trading period, including the two branches referred to in the 4 PEAKs mentioned by Mr Coyne.

13.

The diagnosis of the issue was that the text drafted into the TCs contained a string of 35 characters without a space. The code to put each TC on a screen was attempting to split the text at a space (as is normal for a word processor), but the code was unable to process a string of this length.

14.

The issue was resolved quickly. Having first been reported on 1 December 2005, a work-around was developed whereby the affected branches were instructed to roll over all stock units except one (the one relating to the unprocessed TC). This enabled them to roll over into the new trading period. A software fix was then released on 22 December 2005 which enabled the branches to select and process the outstanding TC, and roll over the last stock unit. To avoid the issue reoccurring in future, Post Office confirmed that strings of this length without spaces would no longer be included in the text of TCs.

15.

Although included in Mr Coyne’s analysis of branch affecting bugs, the impact of Issue 2 [ie this issue in this PEAK] was limited to the affected branches being unable to roll over to the new trading period for a short period of time. It had no financial impact on branch accounts.”

(emphasis added)

354.

The code explanation above was not contained in any evidence and was taken by the Post Office in its submissions from documents that were not even deployed in the trial. However, these show that it was clearly a bug, which Fujitsu knew about in late 2005 and which was fixed by means of a software fix at the very end of that year, with a KEL being produced in January 2006. TCs were introduced in September 2005.

355.

There was no impact on branch accounts, as accepted by Mr Coyne. This therefore goes to Horizon Issue 4 and not Horizon Issue 1.

356.

There is another PEAK from September 2010, PEAK PC0204350 which refers to Horizon Online and a report by an SPM on 14 September 2010 that he had suffered an £80 cash loss which he believed was due to a “system error”. The PEAK itself states at the beginning “Summary: can you please look into this problem. Our second line cannot find any user error.” The SPM referred the Fujitsu representative on the call to a TC report he had generated on 10 September 2010, which had allowed him to request a date range of 60 days, but which did not provide him with data on TCs which were more than 40 days old.

357.

The PEAK indicates that on 14 September 2010, Fujitsu requested detail of the specific transactions relating to the £80 loss from the SPM in order that it could carry out an investigation into the issue. No information was provided by the SPM. Given the absence of evidence, the matter was thereafter closed by Fujitsu as Category 95 “Final – Advice after investigation” and “no fault in product”. I find that this is a misleading description ascribed to close this PEAK. It also states “TC data is kept on the live database [BRDB] for 40 days so any txn query for a TC where it was put through the system more than 40 days previous WILL NOT show up in the branch. System is working as designed.” The first sentence of this is correct. Those timescales are indeed what occurred. However, the entry “system is working as designed” is at odds with the fact that second line support at Fujitsu had already said there was no user error. The penultimate entry in the PEAK also states “however, beyond that, I will have to request archived data from our Audit Team in order to confirm those TC txns in July 2010.” There is no evidence to suggest this was done; the matter was simply parked without any proper resolution. I find this to be evidence of a bug. Given that Mr Coyne stated in cross-examination that there was no evidence of impact on branch accounts (although the £80 referred to might suggest otherwise) I take this into account in relation to Horizon Issue 4. However, it plainly has the potential to cause impact to branch accounts and therefore should also be taken into account under Horizon Issue 1.

22. Bugs/errors/defects introduced by previously applied PEAK fixes.

358.

This also arises in Legacy Horizon. It is de facto accepted as a bug by the Post Office, but it is submitted that it had no branch impact. It is included in paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact”. It is also submitted by the Post Office that “a number of PEAKs arose in testing. The PEAKs that arose in the live environment do not indicate evidence of branch impact.” It is listed in the Bug Table as concerning Horizon Issue 4. Mr Coyne stated in the Joint Statement that “Branch accounts would be affected by this bug which would cause a discrepancy when handling cheques where the value of the cheque would be doubled” although he accepted that the SPM in question was processing a cheque in a different manner to that recommended. Dr Worden used the term “a fault” in his entries, maintained that there was no impact on branch accounts if the SPM followed correct procedures, and he also (in respect of another branch) relied upon the fact that the effect was an error of 2p only.

359.

The Post Office submits that there are three distinct issues and that they were all grouped under the same head. Some of the PEAKs are from 2000 and one is from 2004.

360.

PEAK PC0053160 refers to a report by a Fujitsu test team member on 29 August 2000 which concerned an issue with the Training Counter which froze when completing a transaction log report. Further testing by Fujitsu demonstrated that the issue could also be present in the live environment, and a software fix was deployed on 6 September 2000. Mr Coyne concluded that the fix implemented what are called regression bugs. There are references to this in the PEAK. A regression bug is something that prevents a function from operating as it should. Entries in the PEAK such as:

“Training Counter freezes using transaction log

Training Counter freezes when miskeying using transaction log. If you do the following:

perform sales transaction select Reports/Transaction Log select Value from .10p, Receipt select value to £100, Receipt

Continue (F16)

Complete (F16 from Print screen, instead of F4)

Reprint (F4 from Reports menu) you go back to the Transactions Log menu as expected but the counter freezes and you can't get out. Continue and all the other buttons do not work (except to flash when selected).

This manifests itself in training, if a delegate mis-hears or miskeys a keying sequence

in doing a transaction log report. Could be a live issue too.” And:

“This is NOT a Traing Mode issue, but an EPOSS issue. The problem is that if you Prev out of the Report/Reprints menu you are returned to a previous Transaction Log selection menu from which there is no escape.

Dean has not specified which build he was using, but I have reproduced this on both CI4L1 and CI4R streams.”

And

“F} Response:

The problem described at the start is fixed. However, application of the work packages to the training counter appears to have caused some regression, namely: If you perform a number of different types of transactions (e.g. passports, APS, BT bills) the daily reports are correctly populated, however no reports are listed when you press the Summaries icon (mandatory or otherwise), and the system does not require you to print and cut off the mandatory reports before rolling over the stock unit into the next CAP.

[END OF REFERENCE 21808091]

Responded to call type P as Category 42 -Product Error Diagnosed

The response was delivered on the system”

(emphasis added)

361.

These suggest that Mr Coyne is right – Fujitsu itself diagnosed it both as a regression issue and also diagnosed a product error. The Post Office’s written submissions stated that “further investigations by Fujitsu have concluded that rather than a regression, this was an error in the way that the test rig was setup; an approved combination of work packages was not being used” although this was challenged by the claimants as not being contained in any of the evidence.

362.

The answer to the challenge that was given was that this was taken from a document not deployed in the trial, and from common sense. The document was PEAK PC0053160. The entry expanding on common sense states “the developer is stating that the tester failed to re-test against an approved combination of work packages; hence the assertion by the tester that there was a code regression is incorrect”. It is also stated “work packages sometimes have interdependencies in order to provide a complete fix to any given issue.” I consider that the PEAK document was deployed in the trial in the sense that it is referenced by Dr Worden in his entry in the Bug Table, but in any event the entry relied upon is 22 September 2000 when Les Ong stated:

“It's not stated which build this is but, by adding two M1 work packages in isolation, I'm not surprised there's further problems. These two WPs contain EPOSSCore, EPOSSReportbroker, EPOSSStockunit and BESReports. Any of these may have dependencies on other WPs or Ref Data.

You need to stick with a CI4L1, CI4R or current M1 build. At least bugs in these are known and can be quoted.

[END OF REFERENCE 21817364]

Responded to call type P as Category 74 -Fixed at Future release”

363.

This is clearly a reference to a bug, and is also something that arose on a training rig, but it is in my judgment a regression issue. The tester thought that it was, and that would have been one of the things that the tester was looking out for.

364.

In any event, the next PEAK clearly was a regression issue. I will quote the Post Office’s submissions:

“8. Peak PC0098230 refers to an issue reported by a SPM on 13 January 2004 relating to a discrepancy with his cash account. Fujitsu called the SPM to obtain further details and ascertained that when the SPM was declaring his cheques, the value of cheques declared as stock doubled.

9. The issue was diagnosed as a code regression relating to the fix implemented in Peak PC0097081. A work-around was proposed two days later to the SPM, on 15 January 2004, that the SPM should follow a different procedure when declaring his cheques. A software fix was thereafter released for the code regression.”

365.

The PEAKs relating to the third issue were regression bugs but were discovered during testing in early 2000.

366.

Therefore, in my judgment, Mr Coyne is correct in that the PEAKs that were the subject of this heading in the Bug Table, were regression bugs. Given the lack of impact upon branch accounts, these relate to Horizon Issue 4. The one in 2004 is particularly notable, but given it was resolved so very quickly (a matter of days) there was in fact no impact on branch accounts, but there was potential impact only. I consider this bug to be relevant to Horizon Issue 4.

23. Bureau de change.

367.

These occurred in both Legacy Horizon and Horizon Online and arose in 2005, 2006 and 2010, and should be differentiated from bug 14, which arose in 2017, which was during Horizon Online and is referred to as Bureau Discrepancies. The Post Office does not accept this as a bug. The Post Office also states that Mr Coyne has identified three issues all with the same heading, and submits that they all relate to user error. Dr Worden’s entry in the Bug Table in relation to one of them states “Analysing the second KEL (2010) I noted: ‘Impact small until bug fixed - rounding errors 10-5 in exchange rates.’ The Post Office’s submissions that this is not a bug is a somewhat bold submission, given the KEL to which Dr Worden refers expressly states, in terms, that the problem is a different exchange rate appearing on the HNG-X rateboard and the Horizon rateboard, two different parts of the same system. It also ignores Dr Worden’s statement “until bug fixed” (emphasis added). HNG-X is Horizon Online, but this cannot applies to the occurrences in 2005 and 2006.

368.

KEL AgnihotriV917N dated 23 June 2010 states within the KEL itself:

“Problem – there is a bug in the code” (emphasis added).

The full entry reads as follows:

“Problem

There is a bug in the code.

Steps to reproduce:-

1.

In spotrates.xml make the value for any currency (say Egypt Pound EGP) as 10 and in margins.xml as 1.004 under BUY- EGP.

2.

Start the Counter

3.

Configure RatesBoard with this currency.

4.

Click on update.

5.

Under Buy Notes column the value displayed for EGP will be 10.1 instead of

10.100.

Solution - ATOS

This bug is harmless as the business rule is working as desired except for displaying effective rate.

It means in the background the value used for effective rate will be as calculated from the formula but when it comes to display, value will be stripped of trailing zeroes.

See also PC0200042 and KEL Agnihotriv245L.

Solution in progress.”

(emphasis added)

369.

There plainly is a bug – it is referred to twice in terms in the same KEL. The author of the KEL states that it “is harmless”. The other KEL referred to, KEL Agnihotriv245L, is by the same person. It refers to the fact that customers will (or may) be given the wrong rate to the one displayed in the branch. It states:

“There is a bug in the code.

Steps to reproduce:-

1.

In spotrates.xml make the value for any currency (say Egypt Pound EGP) as 8.3325 and in margins.xml as 11.2400 under BUY- EGP.

2.

Start the Counter

3.

Configure RatesBoard with this currency.

4.

Click on update.

5.

Under Buy Notes column the value displays for EGP will be 9.269 instead of

9.2691.

Although the system does use the correct values to calculate the rates and the issue here is with the display only. However, because the display is on the Customer facing rates board there is potential for annoying Customers. They may get a slightly different rate to that advertised on the board. And may challenge Post Masters on this issue.

As the issue relates to the 5th or 6th significant figure the impact will be fairly low - and in favour of the Customer 50% of the time e.g. if I buy £1000 of XYZ currency at a rate of 99.12 then I expect to get 99120.00 in XYZ (according to the board) but I will actually get 99115.90 in XYZ (by the Counter) - a difference of 4.10 XYZ or the equivalent of £0.04. Its effect may be magnified by high value transactions but generally, different rate bands apply in these cases - so the rate board display is inconsequential.”

Solution - ATOS

The solution is a code fix to change the code to treat values in the affected ranges correctly.

See also PC0200090 and KEL AgnihotriV917N

Solution in progress.”

370.

This plainly shows a complacent, if not lackadaisical, attitude to financial precision. The error may be in the customer’s favour ½ the time, but that means it will be in the Post Office’s favour the other ½ the time, and the customers will not be the same person on both occasions, and are highly likely to be entirely different people. The KEL also recognises that “its effect may be magnified by high value transactions”.

371.

The solution in one of the other KELs, raised by Anne Chambers namely AChambers2252R is:

“Solution - ATOS

If a PM reports a loss connected with a currency transaction that was reversed, it is possible that the reversal had not been carried out successfully.<br><br>Ask the PM to check the Reversal Receipt. If this shows<br><br>Cash FROM CUSTOMER 750.00<br>Cash TO CUSTOMER 750.00<br><br>they have reversed just the cash settlement part of the transaction, which has no overall effect. The currency and margin part of the transaction has not been reversed.<br><br>Do a transaction log search using the Session Id from the original receipt, or by date/time.<br><br>This should show 3 elements - for example<br> <br>2-29826-2 SC Euro 1- 720.00-<br>229826-2 SC Curr Sell Margin 1- 30.00-<br>2-29826-3 SC Cash 1

750.00<br><br>The element which should be reversed is 2-29826-2 (i.e. Euros and margin). As long as the PM has not yet rolled over the stock unit, they should be able to reverse this transaction now.<br> <br>If the stock unit has been rolled over, NBSC will have to advise on what can be done to resolve the system loss relating to this transaction.”

372.

They are therefore two different issues. The Post Office, again boldly, submitted at paragraph 2 on page 511 (which is part of its Appendix 2) in relation to this bug that “There is no evidence of a bug in Horizon. Each issue is an example of user error in Horizon.” I find those submissions to be verging on the extraordinary. The substantive judgment to which this is an appendix identifies what the Known Error Log is. The entries above make it clear that there is a bug – the very word chosen by the Fujitsu employee who wrote the two KELs is “bug”. To see this characterised in submission as there not being a bug, and being evidence of human error, is not only puzzling but flies in the face of the terms in the Fujitsu documents. I find that it is evidence of a bug. One of the PEAKs only of those listed in the Bug Table, namely PC0137437 is user error, a point identified by both Mr Coyne and Dr Worden. However, simply because one PEAK is this, does not mean that the summary submissions can be accurately stated as “There is no evidence of a bug in Horizon. Each issue is an example of user error in Horizon.” That is simply wrong in fact.

373.

Further, Mr Coyne identified in his evidence that there were 8 PEAKs associated with these KELs that fall in the date range of 2010 to 2018.

374.

There are some unsatisfactory aspects to the way the Post Office tries to explain its position on this bug. At paragraph 14 of its submissions, it submitted that the Post Office subsequently “reviewed ARQ data for the branch” in question on one of the PEAKs, and that “there is nothing to suggest that the branch’s remming in/out of foreign currency caused a loss in branch.” When challenged by the claimants that this was not in the evidence, the Post Office clarified by stating this came “on instructions from Post Office, which did not have an opportunity to adduce evidence on the point”. That is not correct. This submission was made in relation to PEAK PC0151787 which is the second entry in the final column in the Bug Table itself, headed “supporting evidence”. It is also referred to, correctly, in paragraph 4 of the Post Office’s submissions in the following terms “In JS2 Mr Coyne mentions three PEAKs under the heading of “Bureau de Change””. JS2 means the 2nd Experts’ Joint Statement, which was dated 25 February 2019, well before the start of the trial. There was such an opportunity, as supplementary questions in chief could have been put (or permission to do so asked) and it is wrong to suggest that the Post Office was denied the opportunity to meet this evidence. Dr Worden chose to adopt a different way of giving his expert evidence, namely very high level.

375.

In any event, there is nothing in my judgment to justify a finding of user error which is relied upon by the Post Office in relation to this PEAK. The first entry in the PEAK states, not user error, but the following: “PM states that that he has remmed out some foreign currencies, reversed them and re-remmed them but when he has come to do the stock balancing his main stock unit is £907.97. Richard at NBSC 2nd line has requested that I log a call and send to SMC.” Even Fujitsu categorise the call closure as “Defect cause updated to 42 -- Gen - Outside Pathway Control.” That is not consistent with user error.

376.

In any event, I find that there is such a bug and I take this into account in my overall conclusions on Horizon Issue 1 as well as Horizon Issue 4.

24. Wrong branch customer change displayed.

377.

This is a Legacy Horizon Issue and occurred in 2005. Mr Coyne concluded this was a bug and part of his entry in the 2nd Joint Statement was “the KEL explains that ‘the cash amount entered is multiplied by the Qty and hence the new stack total is wrong’”, and that this Horizon bug was due to incorrect reference data. The role that reference data plays in the operation of Horizon has been explained. Here, this led to an incorrect amount of change being displayed on the branch screen leading to the operator (which means SPM or their assistant) providing the branch customer with the wrong amount of money, thereby leaving a discrepancy in Branch Accounts. He also stated that “It is possible that the amount of change shown on screen is more than the actual money tendered by the customer.” Dr Worden’s entry was “When analysing this KEL I noted ‘Sounds like a genuine problem which may have led to giving the customer the wrong amount - i.e. not recoverable.’ It is now accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. It is also said that it is a reference data bug, and that therefore once discovered could be quickly fixed by changing the relevant reference data.

378.

The summary of the Post Office’s submissions are as follows:

“2. Bug 24: Wrong Branch Customer Change is a bug with the potential for lasting financial impact. This is a reference data bug. The experts have agreed that while reference data bugs may be a significant proportion of the bugs with financial impact, once discovered, they could be quickly fixed (by a change to the reference data) once the bug is correctly identified. This was the case with Bug 24. The issue would have visible to the SPM as the incorrect quantity would have displayed on the screen. Fujitsu identified the root cause and developed a fix within two weeks of the issue being reported by the SPM.

3.

This issue relates to quantities of stamps and postage labels (Smartpost Transactions) not correctly resetting to 1.

4.

When a quantity of greater than 1 was entered for a Smartpost Transaction, the quantity was not reset to 1 when the clerk moved on to the settlement screen. This could result in subsequent items in the session being multiplied by whatever quantity remained and could affect further items being sold or the amount being tendered towards settlement.

5.

Peak PC0128264 was opened on 4 November 2005 as a result of a SPM reporting the issue on 4 November 2005. The matter was passed to Fujitsu’s development team for a software fix on around 10 November 2005 and a fix was implemented on 18 November 2005. The Peak shows that Fujitsu suspected that the problem was introduced by changes to the Smartpost Transactions that had been implemented from 24 October 2005.

6.

On 6 December 2005, a further instance was reported (Peak PC0129791). The root cause was identified and it was found that this issue related to Peak PC0128264 which documented the fix that had been put in place. On 7 December 2005, Fujitsu found that the fix had not been applied to a group of branches and the reference change data fix was then implemented overnight to the remaining branches.”

379.

The KEL on this was again one authored by Anne Chambers. It is undoubtedly a bug. Not only was it a bug, but once it was discovered and a software fix introduced, it persisted because “the fix had not been applied to a group of branches”. No detail is given as to how large that “group of branches” was – the only description in the submissions is that it was “was an active group of branches, group 111111112”. Nor is there any explanation available in the evidence as to why the fix, which was known to be required to fix this bug, was not applied to every single branch, and why or how this “active group” of branches was omitted.

380.

I find that it is a bug with potential to impact upon branch accounts and I take its existence into account in my answers to both Horizon Issue 1 and Horizon Issue 4.

25. Lyca top up.

381.

This is a Horizon Online bug. Lyca is a type of mobile phone “top up” card which allows people to pre-pay for mobile phone services. This is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. It is also accepted as being a reference data bug, and paragraph 2 of the detailed part of

the submissions on this in Appendix 2 states “Lyca Top Up is a bug with the potential for lasting financial impact. This is also [a] reference data bug. As set out above, the experts have agreed that while reference data bugs may be a significant proportion of the bugs with financial impact, once discovered, they could be quickly fixed (by a change to the reference data) once the bug is correctly identified.” It is also submitted by the Post Office that it was identified through Fujitsu’s automatic reporting.

382.

The nature of the bug, which occurred in August 2010, is that a customer will pay the SPM a certain sum (for these purposes, an example of £20). The transaction is entered on the Horizon counter by the SPM and, once the transaction is processed by Horizon and authorised by E-Pay (the financial institution), a receipt is printed for the transaction which the SPM should provide to the customer. It is this receipt that contains the customer’s voucher to apply the top up to their mobile phone. Applying that voucher will give the customer £20 of credit to use on their phone.

383.

The problem was picked up by SPMs who reported it to the NSBC, and also by Fujitsu through its NB102 reporting. Depending upon what happened when the customer and the SPM did this transaction, would affect whether branch accounts were adversely affected. However, the Post Office accepts that “if the SPM recovered the transaction and incorrectly confirmed on Horizon that the top up had been successful, despite no top up receipt having been printed, the transaction would be recorded in the branch’s accounts, meaning the branch would likely have experienced a shortfall to the value of the top-up as no money would have been taken from the customer. If the SPM logged back into the counter and correctly confirmed that the transaction had not been successful, a zero value transaction would be recorded in the branch accounts and a reversal generated for the top up. This should have resulted in the affected branch accounts being correct, but due to the reference data issues the reversal being sent to E-Pay caused E-Pay to treat the top-up incorrectly as a successful transaction.”

384.

The PEAK shows that it was described as a “system error during transaction” and that it was resolved by means of a change to the RDT generator mechanism. One entry also states that “This problem could also show up as incorrect Welsh on receipts as special Welsh characters may also be incorrectly translated.” The corrective reference data was released to live to take effect overnight on 20 August 2010.

385.

I find that it was a bug and I take its existence into account in my answers to both Horizon Issue 1 and Horizon Issue 4.

26. TPSC 250 Report.

386.

This is a Legacy Horizon bug. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. The origin of the experts discovering this bug is a KEL raised by Anne Chambers in February 2005, and last updated in April 2008.

387.

There are approximately 24 different PEAKs dealing with this bug. Their date range is between 2005 and 2009, with the majority of the incidents occurring in 2005. It relates to the printing of labels for postage. The KEL was originated by Anne Chambers.

388.

The Post Office has identified, within its submissions which deal with all of the PEAKs, a number of different sub-issues. The summary paragraph states:

“1. The experts have drawn together five distinct issues under the heading TPSC 250 Report…..

2. This was a backend reporting problem and so the chances of branch impact are small. Of the five issues: issue 1 resulted in incorrectly flagged exceptions but no reconciliation was required; issue 2 resulted in no financial or operational impact on a branch accounts and was limited solely to the process of copying/ harvesting transactions to Post Office's back-end systems; issue 3 did not result in a mismatch between the files sent to Post Office and the branch data; issue 4 was flagged by automatic reporting and issue 5 could result in a receipts and payments mismatch thus needing reconciliation.”

389.

I do find it notable that although Dr Worden says the amounts involved are what he calls “small”, he accepts it is as a “back end reporting problem” – which is nothing to do with the SPM, by definition – and also the KEL states that “the accounting tree has not handled this properly when calculating the daily recon figures and it has resulted in a mismatch…” The mismatch in that case was 66p. The accounting tree is part of the Horizon system. The KEL also appears to be incomplete because only the first page is present. The section that follows the heading “Evidence” is entirely missing. There is also nothing in terms of the text available that demonstrates what occurred in 2008, even though it is plain from the page that is available that the KEL was updated then.

390.

Regardless of whether the impact was small or not – and I accept that for the most part, the amounts referred to in the PEAKs are in pence or a few pounds – this is a reflection of the fact that the cost of postage labels is relatively modest. One PEAK, namely PC0123056, gives a good outline in its entry for 12 July 2005 by Anne Chambers:

“Yes this is another instance of KEL AChambers253L - mails transaction total value £1.86, prepaid £4.26, so postage label for -£2.40 generated. This has upset the counter reconciliation figures.”

391.

The reason the counter reconciliation figures are “upset” is that the label was generated for a minus figure, namely “-£2.40”. Mr Coyne identifies that the values are less than £2, and he also opines that SPMs may simply write off such amounts.

392.

The “five distinct issues” identified by the Post Office in its submissions are grouped by the experts under the same single heading, because (I assume) they all deal with a similar scenario. I will count them as a single bug, even though manifestations of the early 2005 example were dealt with as follows, taken from the Post Office submissions: “A permanent fix involving a code change was released to all branches in 2005 as part of the S80 software upgrade.”

393.

The occurrences of bug(s) or issues within Legacy Horizon after the S80 software upgrade therefore were either new, or different. On the dates in the documents, problems with TPSC reporting plainly did persist after the software upgrade was introduced in 2005. The majority of these were picked up by Fujitsu reporting, but

that does not mean that they were not there. Fujitsu did not even mark these issues high priority, although the explanation given for that by the Post Office is that this was done because there was no impact upon branch accounts.

394.

The S80 software upgrade was accompanied by something called “S80 Release Note – Deferred PEAKs List – Counter.” This document is dated 13 October 2005 and is 32 pages long. It “details PEAKs that are outstanding at S80” and the approved form of that document is in the trial bundle. The Technical Design Authority for it was Gareth Jenkins. It includes analysis of the PEAKs that affect the counter only, and the document is an addendum to another document. It identifies 45 different PEAKs that affect the counter. This document shows that there were other issues, not simply the one relating to TPSC reporting, where a decision was taken not to deal with certain errors and/or coding bugs at that time. One example only, cash volume adjustment states “This is a code error but the problem has been in the system since before S80 and doesn’t appear to be causing any significant confusion. A KEL should be raised and a fix considered in a Future Release.”. Of the 45 PEAKs in this document, most are to be dealt with at a “future release”, one is accepted (which is cosmetic), one closed, and another is to be dealt with by a documentation update. It would therefore not be correct to assume that all known PEAKs were fixed by release S80.

395.

I find that it was a bug with the potential to impact upon branch accounts and I take its existence into account in my answers to both Horizon Issue 1 and Horizon Issue 4.

27. TPS.

396.

This again is a Legacy Horizon bug. Its years of effect are 2006 to 2010 with the majority diagnosed by Fujitsu in the PEAKs as “Development – Code”. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. One of the Fujitsu documents referenced in the 2nd Joint Statement states that the Transaction Repair Tool or TRT is being used “to repair 1 harvester exceptions for” a particular branch and “There is no correction to be performed and hence no call for confirmtiprepair - this is just an oddity performed by that very flaky mails code.” (emphasis added) The reference to “flaky mails code” makes it clear that it is the code that is the cause, which I consider means it is a bug. Mr Parker gave some evidence on this, and I have reviewed this too even though the Post Office submissions do not specifically identify his statement as a “key document” in the detailed submissions on Bug 27.

397.

There were 40 associated PEAKs in the Bug Table under this entry, and Mr Coyne observed that both the credit and debit sides of a transaction were doubled, so the net impact of the bug was zero, although he drew attention to the entry in two PEAKs that suggested that SSC requested confirmation of any gain or loss at the counter. Dr Worden believed it was a back-end reporting problem, although the chances of impact upon branch accounts were small.

398.

The issue is described as follows by the Post Office in its submissions:

“3. This is a reconciliation reporting issue that affected SmartPost transactions. The SmartPost application was supplied by Escher and was designed to help users to calculate the postage required for any item and print labels to attach to the relevant items.

4. Peak PC0141145 (Footnote: 1) related to a problem with SmartPost which wrote slightly corrupt transactions.” (emphasis added)

399.

That this bug was known about at Fujitsu at least, is very clear. There is a KEL dealing with it which dates from 2006, and runs to 2010 when Legacy Horizon stopped being used. It states in part:

“Symptoms

Branch shows that the TPS Total and Counter total values for the Number and Abs Quantity columns are the same, but there is a difference in the Absolute Value.

<br><br>Absolute Value for Counter Total is greater than the corresponding TPS Total by £14.8

Problem

Mails doubling the <Credit:> attribute. Search for <Mode:SC>><Credit:(value no decimal point)> NOTE: This will be the difference between the Absolute Value of Counter and Host figures on the TPSC250 report.<br>The column title in TPSC250 is "Absolute Value". This means that Debits will contribute in just the same fashion as credits so if the first search yields nothing useful then look for <Mode:SC>> <Debit:(value no decimal point)><br><br>28-Aug-2008 PC0164058 Here there were <b>two</b> messages where Credit was double SaleValue. The sum of these two messages came to the difference in Absolute Value in TPSC250. This branch also appeared in TPSC257.<br><br>Another occurence in

<b>PC097941.</b><br><br>The Smartpost messages are unusual in that the Credit / Debit attribute is near the beginning of the message instead of at the end. The SaleValue is correct but the DisplayValue and Credit are twice as much as expected.<br><br>This problem affects Bulk Mails (T&T) transactions. It seems likely that there is some way of backtracking through screens which causes this, but haven't managed to reproduce it so far.<br><br> Often this has no effect on balancing - the messages are as expected, except for the Credit/Debit attributes. These attributes are only used for the calculation of the EPOSSDailyRecon absolute values. The office should roll over successfully with no Receipts and Payments mismatch.<br><br>However occasionally there is a very similar problem with incorrect Credit / debit attributes in bulk mails smartpost messages where the session doesn't net to zero. This will additionally cause an entry on TPSC257 IncompleteSummaries, and also give a receipts and payments mismatch. <br><br>Another variation is where mails code omits the message for a transaction and then does write the balancing message. The effect is the branch appears in TPSC250 and TPSC257. <br> <br>See KEL MaxwellG460L for how to fix

TPS_POL_FS_Summaries_Incomp.<br><br>16-Jan-2007 PC0142604. <br>13-Dec2007 PC0152156<br>24-Jan-2008 PC0153333<br>05-Aug-2008 PC0162929<br>16Dec-2008 PC0171637”

Solution - ATOS

PC0141145 is with development, which has similar symptoms. Add further instances to that call.<br> <br>If the session nets to zero (add up all the SaleValues for the same SessionId) no reconciliation is needed. If it doesn't, a correction must be made to send the data to POLFS (see <a href=kel_view_kel.jsp? KELRef=MaxwellG460L>MaxwellG460L</a>) and the PM may need to be told about a possible receipts and payments mismatch, or at least watch out in case one is raised.”

(emphasis added)

400.

This shows the following. Fujitsu picked up this bug through automatic reporting. It also shows that KEL itself recognises that sometimes a receipts and payments mismatch may occur in branch accounts, and that if so “a correction must be made”. That correction would be made outside of the Horizon System, by means of a TC. That plainly, in my judgment, makes it clear that this bug has the potential to impact upon branch accounts.

401.

I find that it was a bug with the potential to impact upon branch accounts and I take its existence into account in my answers to both Horizon Issue 1 and Horizon Issue 4.

28. Drop and Go.

402.

This is something that occurred in the summer of 2017 and is a Horizon Online issue. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of

“bugs with lasting impact (although they were resolved)”. This actually occurred in

June 2017 (although the KEL is dated July 2017) and concerns a duplicate “Drop and Go” transaction for £100. This was a top up which had to be performed twice; the branch was debited with £200, but the customer credited with only £100 (which had been the amount “topped up” by the customer). Mr Coyne considered it a bug with impact upon branch accounts; Dr Worden stated in his entry “My analysis of this KEL was ‘Possible financial impact. Seems very visible on the counter. Script = reference data - therefore fixed easily”. Paragraph 2 of the detailed part of Appendix 2 of the Post Office’s submissions on this bug states “Peak PC0260269 relates to an issue involving a Drop and Go transaction (a £100 mobile phone top up) that timed out on Horizon” (emphasis added). This does not appear to be correct, and I do not believe that the transaction relates to a mobile phone. It is a postal-type service.

403.

However, regardless of the type of service it deals with (mobile phone card top up, which is what the Post Office says, or Drop and Go postal service, which is what I consider it deals with) is not of the highest relevance. It is a Post Office service for which customers in the branch pay money. Mr Parker gave some evidence on this, and I have reviewed this too even though the Post Office’s submissions do not specifically identify his statement as a “key document” in the detailed submissions on Bug 28.

404.

The Fujitsu evidence on this, as with all the other bugs which are now acknowledged to exist, is that there was no impact on branch accounts. Indeed, Mr Parker accepts in his table at item 28 that “this would have caused a loss in the branch accounts” although he goes on to downplay this by saying the SPM would have noticed and “it would have been resolved by a” TC. TCs are outside the Horizon System. In this case, there plainly was branch impact. The customer was credited with £100 (which means that is the amount the customer paid to the branch) but the branch was debited with £200. The fact that the branch was later corrected does not, in my judgment, matter

for the purpose of the Horizon Issues. The issue really is – was there a bug in Horizon that allowed this to happen, and/or was there a bug in Horizon that caused this to happen? Both ways of putting it adequately summarise the real issue.

405.

The answer to that is plainly, yes. Dr Worden states that the script that led to this occurring could be easily fixed. However, the fact that a fix is needed at all demonstrates that it is a bug and that it had the potential to cause lasting impact to branch accounts. It can plainly be seen from the following analysis of the KEL.

406.

The KEL on this, cardc235Q is dated 5 July 2017 but examining the log entries reproduced in the KEL itself it can be seen that the transaction(s) in question occurred on 20 June 2016. The KEL states (and clerk means the SPM or their assistant):

“Symptoms

The clerk initiated a Drop and Go transaction for £100 which failed due to timeouts, but then a success message was displayed. The clerk settled the transaction and the customer handed over £100. The customer checked the balance and stated that the top up had not gone through, so the clerk then performed another Drop&Go transaction which was successful. The customer has paid in £100 but the branch account has been debited by £200. Accenture verified that only the second Drop&Go top up was successful.”

407.

This shows that Accenture are involved in some way in investigating this, and have verified that only one of the top ups was successful. The word “verify” can only mean that they have established this to their (ie Accenture’s) satisfaction. The KEL also reproduces the log extracts from both the counter log and the message log, which I shall reproduce in full:

“2017-06-20 13:49:07,912 UTC Button: WS-F-Home-1-55 / Drop & Go...

2017-06-20 13:49:09,113 UTC Button:WS-F-PostalServicesDG-1-22/Balance and

Top Up

2017-06-20 13:49:11,046 UTC Swipe: [18954123=***]

2017-06-20 13:49:11,747 UTC Sending Request, service url= [ https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/

0 ] ...

2017-06-20 13:49:12,198 UTC Response Received, Status OK, service url= [ https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/

0 ] ...

2017-06-20 13:49:12,689 UTC MSG10800: Confirm Account Details

2017-06-20 13:49:21,071 UTC Button: 0 / Yes

2017-06-20 13:49:21,532 UTC MSG10800: Top Up Required?

2017-06-20 13:49:27,020 UTC Button: 0 / Yes

2017-06-20 13:49:31,627 UTC Button: enter / Enter

2017-06-20 13:49:34,101 UTC Button: 0 / Cash

2017-06-20 13:49:34,582 UTC MSG10802: Take Payment from Customer

2017-06-20 13:49:37,406 UTC Button: 0 / OK

2017-06-20 13:49:38,007 UTC Sending Request, service url= [ https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/

0 ] ...

2017-06-20 13:50:23,065 UTC IOException occurred while accessing service at URL:

https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/

0 ...

2017-06-20 13:50:28,473 UTC Sending Request, service url= [ https://vbal001:9000/recoveryService/delta/0 ] ...

2017-06-20 13:51:13,551 UTC IOException occurred while accessing service at URL: https://vbal001:9000/recoveryService/delta/0 ...

2017-06-20 13:51:14,332 UTC MSG10802: Top Up Timed Out

2017-06-20 13:51:18,769 UTC Button: 0 / OK

2017-06-20 13:51:19,760 UTC Sending Request, service url= [ https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/

0 ] ...

2017-06-20 13:52:04,848 UTC IOException occurred while accessing service at URL:

https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/

0 ...

2017-06-20 13:52:10,146 UTC MSG10802: Top Up Unsuccessful

2017-06-20 13:52:14,352 UTC Button: 0 / OK

2017-06-20 13:52:14,843 UTC SelectFunctionOptionsBLO.IOP39BLO.ADCScript- CDBalanceTopUp.AdvancedDataCaptureUIA / HSID: 0-@@- 2017-06-20 13:52:14,953 UTC MSG10802: Top Up Successful

2017-06-20 13:52:17,477 UTC Button: 0 / OK

2017-06-20 13:52:20,542 UTC Button: fastCash / Fast Cash 2017-06-20 13:52:21,023 UTC Sending Request, service url= [ https://vbal001:9000/BasketSettlementService/Settle/delta/2 ] ...

2017-06-20 13:52:22,124 UTC Response Received, Status OK, service url= [ https://vbal001:9000/BasketSettlementService/Settle/delta/2 ] ...” (Emphasis added)

408.

The reason for reproducing all these entries is they clearly show that the Top Up attempt timed out, that the Top Up was unsuccessful, and that the second attempt was successful. However, under “Problem”, Fujitsu recorded the following in the KEL:

“Problem

This may be an issue with script ADCScript-CDBalanceTopUp, or a user error.

Solution - ATOS

The Drop&Go scripts are supplied and maintained by ATOS. Therefore please route calls to ATOS. Solution - SMC

The Drop&Go scripts are supplied and maintained by ATOS. Therefore please route calls to ATOS.”

409.

The suggestion that there may be a user error is without any foundation, if the entries in the logs are carefully looked at. These entries clearly show both the time out and the first unsuccessful transaction. It can only be a script issue, something confirmed in

my judgment by the Solution both for ATOS and SMC being “The Drop&Go scripts are supplied and maintained by ATOS. Therefore please route calls to ATOS.”

410.

Dr Worden looked at this and in his Appendix 5 he also identified “possible financial impact” which means impact to branch accounts. There obviously would have been in this case. The SPM has taken £100 from the customer but the Horizon system has treated it as though his branch should be debited for £200 (2 x £100) even though one of the top ups was clearly not successful because it had timed out. The other point of note is that the KEL, the trial bundle reference of which is {F/1660}, is headed “HNG-X KEL cardc235Q”, although on the dates within it (namely June and July 2017) other evidence in the trial was that the system by then was not HNG-X. This is because it became HNG-A on February 2017 when the platform became Windows 10 rather than the older platform (and by 2017 unsupported) Windows NT4. It is not therefore clear on the face of the KEL whether this bug – which it clearly is, and which I find is a bug – is within either the HNG-X or HNG-A version of Horizon Online.

411.

I find that it is a bug and with the potential to impact upon branch accounts and I take its existence into account in my answers to both Horizon Issue 1 and Horizon Issue 4.

29. Network Banking Bug.

412.

This is a Legacy Horizon matter.

413.

This related to a KEL which was raised in 2004 and updated in 2005, with 12 associated PEAKs in the range 2004 and continuing on to 2010 when Legacy Horizon was discontinued. The Post Office does not accept this as a bug. Mr Coyne stated that “Horizon appears to mis-handle communications, leading to errors within network banking and in turn causing the potential for branch account discrepancies.” One of these was pension transactions being declined, yet the customer’s bank account being debited; when the customer complained, the SPM themselves refunded the £50 in question. Dr Worden considered it was “mainly about a communication problem from BT, outside Horizon” but also wanted to investigate further due to an entry that referred to a “CNIM own goal”. The Post Office maintained there were two separate issues and the £50 was paid out due to “user error”.

414.

Dr Worden’s evidence on this was not of enormous detail. The opportunity for an expert to go away and do further or any research after a trial is not of great assistance to the judge tasked with making findings on the evidence in fact available at a trial.

415.

Mr Coyne accepted that this “bug” did not have lasting financial impact on branch accounts. Post Office submits that there is no evidence of a bug in Horizon. The Post Office also submitted that “neither of the two PEAKs referred to [in respect of this bug] can properly be described as instances of a “Network Banking Bug”; both issues stem from intermittent communications failures emanating from outside Horizon (most likely from systems/kit operated by BT).”

416.

The PEAKs show that it was communication difficulties with the branch that probably led to these issues. There were two PEAKs. One related to an ISDN line possible failure or other communication defect. The Horizon Issues define the system as follows: “the Horizon System” shall for the purposes of this list of issues mean the Horizon computer system hardware and software, communications equipment in branch and central data centres where records of transactions made in branch were processed, as defined in GPOC, at §16 and as admitted by Post Office in its Defence at §37”. (emphasis added)

417.

ISDN lines are provided by BT and go between branches and what would be called “central data centres”. That link is not obviously included in the definition of the system I have reproduced above. Such lines are not used any more and have been replaced by ASDL lines.

418.

In any event, there is insufficient evidence in my judgment to conclude that there is a “network banking bug” as expressed as this entry in the Bug Table. I therefore find that there is no such network banking bug. I take that finding into account when considering Horizon Issue 4. Mr Coyne accepted that this was not a separate bug that went to Horizon Issue 1 in any

Type of Permissions

419.

There is another issue that is relevant to the potential impact to branch accounts, and that is the extent of permissions to Fujitsu personnel. Until 2012 all the personnel in SSC had far wider permissions than were necessary, and wider than they were supposed to have.

420.

The PEAK that is associated with this issue, namely PC0208119 dated 10 March 2012, states in the subject summary “SSC Database users do not have correct permissions”. The associated entries go from 1 February 2011 to 9 June 2015 (even though the date on the covering page, or page 1 of the PEAK, is 14 May 2012). It is necessary to go through all the way to the final page, page 8, to see the most recent date when the PEAK was closed. Under “Impact Statement” the following entry appears “3. The customer is not aware of this problem or change”. In my judgment this shows two things. Firstly, the Post Office had not been told by Fujitsu that SSC database users did not have the correct permissions. Secondly, the Post Office had not been told by Fujitsu that a change is proposed to change this. There is a fix extensively debated in the same PEAK and the entries strongly suggest that the fix (which is the change referred to in the Impact Statement) is being done without the customer (which must be the Post Office) knowing about it.

421.

The entry of 16 August 2011 in the same PEAK states:

“The optional role 'APPSUP' is extremely powerful. The original BRDB design was that 3rd line support should be given the 'SSC' role (which is select_any_table + select_catalogue) and only given the optional role 'APPSUP' temporarily (by Security Ops authorisation) if required to make emergency amendments in BRDB Live. Since then Host-Dev have delivered a series of auditable amendment tools for known SSC data amendment operations in Live, and these are assigned by role to individual SSC user accounts. As such SSC should not require the APPSUP role in BRDB, unless there is an unforeseen update required to Live.

Transferring to Steve Parker for review/assessment.”

422.

An entry of 30 September 2011 states, in terms:

It is a security breach if any user write access is not audited on Branch Database, hence the emergency MSC for any APPSUP role activity must have session logs attached under the MSC. Host-Dev previously provided scripts, such as the Transaction Correction Tool, are written to run under the SSC role and also write to the audit logs.

SSC users created on BRDB should only have the SSC role, and the user creation script should be amended by Host-Dev to reflect this. A separate script giving/revoking emergency MSC access via APPSUP can be delivered, logging this to the hostaudit directory. In parallel Host-Dev should investigate any Host-Dev delivered script to ensure they are all executable by the SSC role. SSC should investigate any of their own scripts to ensure they have sufficient permissions under the SSC role, taking into account they should primarily perform their work on BRSS.

Any day to day scripts should not access BRDB directly.”

(emphasis added)

The Technical Summary of the proposed fix in the entry of 25 October 2011 that deals with the “development impact of fix” identifies which “HNG-X platforms are impacted” and states:

“TECHNICAL SUMMARY:

This change is to change the script used by POA UNIX and POA ORACLE DBA when creating new SSC/Support users.

LIST OF KNOWN DIMENSIONS DESIGN PARTS AFFECTED BY THE CHANGE:

-

BRDB_SOFTWARE_INSTALLATION_POSTMIG08_0622_<REL>

-

BRDB_HNGX_POSTMIG08_CHANGES_0622_<REL>”

423. This shows that prior to this change to the script (which cannot have taken place prior to the date it was implemented, for obvious reasons) all the SSC users had the very powerful permission (also sometimes called privileges) of the APPSUP user. The experts were agreed that such users could, in terms, do whatever they wanted in terms of access to the system. That could obviously have an impact on branch accounts, depending upon what was done by any particular user on any particular occasion. SSC users should only have had the far more limited SSC role and Fujitsu itself were aware of this, and can only be the entity responsible for them having the incorrect wider role, as they were all Fujitsu employees. The section of the accompanying judgment of Mr Godeseth in the judgment that accompanies this also refers to evidence from My Coyne and Mr Parker on the same subject. Mr Parker challenged Mr Coyne’s figure but had no basis for doing so, as all he had was his impression, and he had specifically failed to do a proper investigation even though I find Fujitsu could have provided far more proper, cogent evidence of the number of occasions. I accept Mr Coyne’s evidence on this, and given both Dr Worden and Mr Godeseth accepted that the powerful APPSUP permission or privileges were more widely available, and

less controlled, than they ought to have been (even based on Fujitsu’s own internal controls) then this inevitably has a detrimental effect upon the robustness of Horizon.

F. Conclusions on Technical Issues

424.

Turning therefore to the summary of how many bugs were present in Horizon (either Legacy Horizon, or Horizon Online), the Post Office cross-examined Mr Coyne on the total number of bugs from the Bug Table of which he had seen evidence of lasting impact to branch accounts. Although the Bug Table was numbered 1 to 29 (suggesting there were a total of 29 separate ones) rather confusingly Bug 6 was split into two, 6(i) and 6(ii). The bugs which were accepted by the IT expert for the claimants as not providing evidence showing lasting impact on branch accounts were those in the Bug Table, other than those numbered 1 to 14 inclusive, 18, 22, 23, 24, 25, 26, 27 and 28. That is a total of 22 different bugs that did show such evidence, but only if Bug 6 is counted as one bug. If counted as two bugs, there would be 23. This is a convoluted way of saying that in the experts’ agreement, when they agreed they had seen a range of 12 to 29 bugs that showed evidence of lasting impact to branch accounts, that upper limit should be 23 not 29. Of the other bugs, for the most part Mr Coyne considered they were relevant to Horizon Issue 4 regarding robustness.

425.

However, the cross-examination which elicited this put the questions in terms of evidence of lasting impact on branch accounts. The distinction between transient and lasting impact was, as explained in the substantive judgment, a differentiation that the Post Office adopted (originally it seems in Dr Worden’s reports, but also in its evidence and the way the case was put) and is not included in the Horizon Issues. Of the 29 bugs in the Bug Table (30 if one counts number 6 as two separate ones) I have found that those numbered 16, 17, 20 and 29 are not bugs. The remainder plainly are, in my judgment, and that remainder, of which there are 25 (26 if one counts number 6 as two separate ones) are bugs with the potential to impact upon branch accounts. I

accept Mr Coyne’s evidence that there are 22 bugs in the Bug Table (again, the total is 23 if one counts number 6 as two) with evidence of actual lasting impact to branch accounts having occurred. In my judgment, that is a high number of different bugs present in Horizon.

426.

Further, in terms of my findings, 16, 17, 20 and 29 are as follows. Entry 16 shows software faults, although these do not have the potential to cause impact to branch accounts. Entry 17 is an error or defect, and has occurred only once. Entry 20 is not a bug but it is an error or defect as it consists of a hardware fault. Entry 29 is not a bug, and is a fault with the BT lines that connect the branch to the central data centre. It is really only therefore Entry 29 in which the Post Office has had any success in defeating the claimants’ case on the existence of bugs.

427.

Ultimately, the Post Office in Appendix 2 of its Closing Submissions stated that of the 29 bugs in the Bug Table, eight were not bugs at all. Those that it challenged as not being bugs at all were those numbered 8, 13, 15, 16, 17, 20, 23 and 29. This by definition means that 21 of the 29 were accepted by the Post Office as existing. This was a sensible approach; in my judgment the claimants’ evidence of the existence of those 21 as bugs was very strong. The dispute therefore moved to the effect of bugs, rather than whether they existed at all. Three were said to have had no branch impact; nine had or potentially had only transient impact; and nine were said cause or had the potential to cause lasting impact, but it was said this was resolved by the Post Office and Fujitsu.

428.

I consider, for the purposes of the agreed Horizon Issues (rather than the re-cast issues, with the Post Office introducing themes helpful to their case) that even those bugs with what were said to have had “transient” impact had the potential to cause impact to branch accounts, with the sole exception of Entry 29. I have found that Entry 15 was a bug; Entry 16 were software faults, albeit with no potential to cause impact to branch accounts; Entry 17 was an error or defect that only occurred once. Entry 20 was not a bug but was an error or defect. Therefore, even those that were not bugs were errors or defects.

429.

The three which the Post Office maintained had no branch impact were Entries 11, 21 and 22. I have found Entry 11 was a bug; I have treated it as one single bug, as that is how it was treated in the Bug Table (with numerous PEAKs against single entries). However, as explained above against that entry, the PEAKs show a variety of different software problems concerning Girobank. I have also found that Entries 21 and 22 are both bugs.

430.

Arriving at an overall final total of bugs therefore is not apt to be helpful, in my judgment. This is particularly so if one attempts, as the Post Office did, to downplay their effect by introducing new concepts such as “transient” impact. All that does is dilute, or seek to confuse, in terms of the potential to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions; and/or to undermine the reliability of Horizon accurately to process and to record transactions. That last sentence keeps, so far as possible, to the actual wording of Horizon Issue 1, which was agreed by the parties a long time before the Horizon Issues trial. On some of the entries in the Bug Table, such as Bug 11 concerning Girobank, I have termed it a single bug (as the experts did that) but the underlying PEAKs show multiple bugs under that single heading. It could quite easily be counted as six separate ones, a point supported by the Post Office’s own submissions identifying the six separate matters contained in the numerous PEAKs grouped by the experts under one single heading. If it is counted as six bugs, and if Bug 6(i) and 6(ii) are counted as two (which they ought to be, but which the experts grouped together in their range of “12 to 29”, (later corrected by Mr Coyne to an upper number of 23) then the total number of bugs of which I have found specific evidence of having the potential to cause impact to branch accounts is about 30.

431.

There is no specific number of bugs in the Bug Table which acts as a threshold below which Horizon suddenly becomes reliable, or above which it suddenly becomes unreliable. The whole of the expert evidence, including the analysis and agreements within the Bug Table, and the effect of the very numerous bugs, errors and defects which I have found to exist, must all be taken into account, as well as the ultimate total. That I have done in reaching my conclusions on the Horizon Issues.

432.

Finally on this subject, during the claimants’ oral closing submissions, there was an outline chronology narrative undertaken of a series of Post Office and Fujitsu documents running from May 2000 onwards, as the claimants’ leading counsel explained it, to “identify the problem that SPMs had with doubles”. I have referred to this in part at [62] to [64] of the substantive judgment. This “doubling” problem or issue was that sometimes there would be a discrepancy – in the example given in

PEAK 0043811 of May 2000, a sum which was expressed in the PEAK as being a “discrepancy in the cash account”, in that instance the sum being £6,343,07 – which then inexplicably doubled to £12,686.14. The chronology then went through different years, 2003, 2004, 2006 and into 2010. Similar documents showed a variety of issues all related to such doubling. The majority of the documents had all been put to different witnesses in the trial, and were referred to in the Bug Table. Finally the chronology ended with a heavily redacted internal Post Office document of 23 May 2018, called “Agenda – Operations Board” one entry of which stated (bureau means bureau de change, or foreign currency):

“SAP GUI Bureau Value Issues

There have been 3 reported instances this week where the sterling value of remittances of bureau to branches has doubled when the delivery has been booked into the branch, For example, Aylesbury GPO were sent a bureau rem with a total value of £6.219,81, however when they booked it into branch it populated as £12,439.62”

(emphasis added)

433. This narrative exercise drew some criticism from leading counsel for the Post Office. He said that the exercise in which the claimants’ counsel was engaged was contrary to the rules of commercial litigation, and the objection to this was put as follows:

“Another one of those rules is not to conflate a whole series of issues, all of which are very different, bureau rems in, we have got other kinds of rems, we have got transfers between stock units, all sorts of different issues which my learned friend picks up randomly over a period of 20 years and seeks to promote them to your Lordship as a single problem which Post Office has known about for all that time.

Your Lordship will have seen from the PEAKs that that's not the case. There were individual instances of individual problems, as far as I can tell, some of these documents I have not seen before of course, which itself is extraordinary given the nature of this trial and that we are at the end of it.

But my Lord what's happened is my learned friend is seeking to jumble things up and then create an impression.”

He invited the claimants’ counsel to stop. The claimants’ counsel countered this objection by explaining that he was demonstrating that “we have found a wide range of strands of causes of doubling issues in the accounts of SPMs. We don't want to conflate them. We want to identify that, not only are there cases where a SPM -- for example, the regional line manager puts it into suspense [account] and it doubles….”

434.

The reason for reproducing this is as follows. The Post Office seemed to draw comfort from the fact that there was no single bug or cause of what was, in my judgment, on the face of the PEAKs, quite inexplicable doubling of branch discrepancies, but rather the 18 year history of these arose from a series of different problems within Horizon. The Post Office also insisted, and seemed to wish to continue to insist, that these were “individual instances of individual problems”. That approach has gone on for many years, and one would have thought that by now the

Post Office would have realised that something of a pattern has emerged in terms of Horizon’s performance. The internal Post Office and Fujitsu documents (running through the whole life of Horizon in this example) all demonstrated, to my satisfaction, unreliability of certain branch account entries due to this doubling up of discrepancies. Even the 2018 Operations Board document showed that these were still occurring in mid-2018 – on three separate occasions in that week alone. It does not matter, in my judgment, to the claimants’ case, whether there was one single cause for these specific discrepancies, or a number of different ones, all of which had a similar effect. The claimants’ case was, in any event, that there were a number of different bugs, errors and defects in Horizon that all had this, or a similar, effect. Nor does it much matter for the purposes of the Horizon Issues whether there was one single cause, or a number of different ones over a number of years. My findings in this appendix show that there were a large number of bugs, errors and defects. There was a far larger number than were admitted by the Post Office at the beginning of the Horizon Issues trial, and a far larger number than ought to have been present in the system if Legacy Horizon and HNG-X were to be considered sufficiently robust such that they were extremely unlikely to be the cause of shortfalls in branches.

435.

The issue of the state of knowledge of the Post Office is for another subsequent trial. This trial deals with the answers to the Horizon Issues only, and all the other matters in the litigation remain for trial on future occasions.

436.

Going through the evidence, PEAKs and KELs, it can be seen that there is a slight majority of the bugs, errors and defects that I have found to exist present in Legacy Horizon, as compared to the number in Horizon Online. However, there are still a significant number in Horizon Online, although not quite as many as in Legacy Horizon. The former regarding Horizon Online are those numbered 1, 3, 4, 5, 7 (although in the pilot), 8, 13, 14, 19, 23, 25 and 28. Those in Legacy Horizon are 2, 6, 9, 10, 11, 12, 15, 17 (an error or defect, not a bug), 18, 20, 21, 22, 24, 26 and 27. It is also notable, in my judgment, that those in Horizon Online were in its earlier years. There is only one that dates from 2017 onwards, namely bug 14, Bureau Discrepancies, with some instances of bug 8, Recovery Issues running into 2018. It must however be remembered that this analysis is on the KELs made available to the experts. A very significant number of KELs were produced by Fujitsu towards the end of the trial which the experts did not have time to consider, and an even more significant number were produced in September 2019, well after the trial had ended. Neither expert has studied these.

437.

I do not know if what appears to be a marked improvement in the performance of Horizon since about 2016/2017 is because, since this litigation commenced, matters have changed at Fujitsu in relation to the Horizon System, including reporting and controls. There is no need to speculate. The experts are agreed that Horizon, as it is in 2019, is relatively robust.

438.

However, the Horizon Issues are not concerned with the Horizon system as it is at 2019. They are concerned with the whole life of the system, from 2000 onwards. Mr Coyne also gave evidence that, given he had not been able to consider all of the PEAKs and KELs that were disclosed prior to the trial, as a matter of extrapolation (on the number of KELs of which the claimants were aware at that time, which did not include the many thousands disclosed later in 2019), he would expect there to be a maximum of about 40 bugs in the Horizon System.

439.

I accept that Mr Coyne was doing his best to assist the court, but this evidence is rather broad brush in its approach. Given there have been, effectively, three iterations of the system, the number of bugs, errors and defects have to be considered against the iteration of the system in which they occurred. An obvious example of this is Legacy Horizon and Riposte. Riposte is no longer used and has not been since 2010. A simple extrapolation across all 19 years of the life of Horizon (ie, 2000 to 2019) would not lead to a sound conclusion. In any event, the number of bugs, errors and defects that have been found by me based on the Bug Table, without extrapolation but by the above analysis, shows that there is ample material, without the need for any extrapolation, to come to measured conclusions on robustness in any event.

440.

The Post Office contended, in its closing submissions, for answers (this was most stark in the issues that arose regarding robustness) that treated Horizon simply as a single system. As an example, the answer for which it contended on robustness was summarised as “the Horizon system is and was very robust (being more robust than most comparable systems).”

441.

I do not consider that it is justified to have one single or simplistic answer to the issue of robustness that governs the whole life of Horizon over its 19 years of operation. Given the expert evidence and agreements alone, it is not correct that any finding of the robustness of Horizon as at 2018 and 2019 equally applies to the system, say, in 2001 and 2002. I reject this as the correct approach.

442.

I consider that the correct way to approach this issue is to consider the following three iterations of the system:

1.

Legacy Horizon, 2000 to 2010;

2.

Horizon Online in its HNG-X form, which ran from the introduction of Horizon Online in 2010 to February 2017;

3.

Horizon Online in its HNG-A form, when it was changed to run on a different platform, namely Windows 10.

443.

It is correct to record that the experts did not approach the matter with a routine marked distinction between the two versions of Horizon Online, HNG-X and HNG-A. However, that distinction undoubtedly exists and Mr Godeseth explained the switch from NT4 to Windows 10 in his evidence. The internal Post Office IT risk management document from 2017 stated that “the HNG-X platform is end of life and is running on unsupported Windows software”, that it needed replacing, and also that the "Branch counter technology is aged and unreliable, with frequent hardware failures, resulting in branch disruptions." This plainly recognised that replacing the platform, which is what occurred when the system became HNG-A, would result in some improvements, and this appears to have happened, although the counter technology in the branch would not necessarily also be changed by a change from one Windows platform (NT4) to another. The risk management document envisaged the required replacement of both. Also, it has to be taken into account that the Horizon Online bugs analysed in Part E of this appendix are, for the most part, those in HNGX. I do not see why the HNG-A form, which both experts agree is relatively robust, should be considered in exactly the same light of the PEAKs that arose (and there are a great many) during the HNG-X years. Dr Worden really only concentrated on the HNG-A iteration of the system in any event, as he considered the operation and functionality of the system as at 2018 and 2019, and this means HNG-A. He readily accepted that his reports concentrated on Horizon as it was at the time of his investigation.

444.

In my judgment, the following conclusions on robustness can be drawn regarding the three different iterations of the Horizon system from 2000 to date:

1.

Legacy Horizon: This was not remotely robust. Indeed, the issue about its robustness (or more accurately, its lack of robustness) became increasingly obvious during the Horizon Issues trial. The fact that the Post Office’s final submissions were forced to concede the existence of so many bugs, with the battleground moving to the type of effect they had, rather than their existence, clearly demonstrates in my judgment that all the weight of evidence, both of fact and expert, was heavily against the proposition that Legacy Horizon was robust. It clearly was not.

2.

Horizon Online in its HNG-X form: This appears to be slightly more robust than Legacy Horizon, but still had a significant number of bugs, particularly in the years 2010 to 2015, but also after that and leading up to 2017. Its robustness was therefore, in my judgment, questionable and did not justify the confidence routinely stated by the Post Office (prior to February 2017) in its accuracy.

3.

Horizon Online in its HNG-A form: This is far more robust than either of the previous two iterations of the system. I accept Mr Coyne’s evidence that there are still far more TCs than expected arising during its use, compared with comparable systems in the banking and finance sectors. However, it may be that this is the only way in which it can sensibly still be operated. TCs are outside the scope of the definition of the Horizon system in any event. It is also an unavoidable conclusion that, notwithstanding the change of platform from Windows NT4 to Windows 10, it is still a Horizon system, which given my conclusions at (1) and (2) is hardly a glowing endorsement of its antecedents. However, the experts are agreed that HNG-A is robust, and better than it was in either its Legacy Horizon or HNG-X forms. Horizon is not free from bugs from any particular date. As shown by the Drop and Go bug, number 28 in the Bug Table, this occurred in June 2017 (the KEL is headed HNG-X as explained in [410] which is after the date HNG-A was introduced, so it may be a bug in HNG-A). Drop and Go appears to be a product introduced by the Post Office relatively recently, but I am unaware of the precise date. The experts should seek to agree whether Bug 28 is a HNG-X or HNG-A bug.

445.

It is also relevant to record that the Horizon system as defined in the Horizon Issues, and as pleaded in the Generic Particulars of Claim and Defence referred to in that definition, does not provide any particular end date. Therefore HNG-A is included within that definition. It is also, however, relevant to record that the effect of any bugs, errors and defects in HNG-A is likely to be limited in this group litigation, as there is a cut-off date for claimants to have joined the group litigation. This cut-off date is 24 November 2017, as extended in paragraph 15 of the 1st Case Management Order sealed on 27 October 2017. The final date by which claims had to have been entered on to the Group Register was 8 December 2017. November 2017 is only about nine months after HNG-A came into being; therefore the vast majority of the claims are likely to arise under Legacy Horizon and HNG-X, rather than HNG-A.

446.

However, that does not mean that HNG-A is free from faults. Paragraph 3.1 of the 3rd Joint Statement recorded that the experts agreed that “'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events.” Also, Mr Latif’s experience with the Lottery and the scratch card discrepancy occurred in January 2018. That means it arose under HNG-A. I have accepted his evidence on this, which means that HNG-A must still contain bugs, errors and defects that have the potential to impact branch accounts.

447.

I have also identified in the substantive judgment that the Post Office’s plans, if there are any, in terms of the evolution of its IT systems and replacement of HNG-A, are not of themselves to be taken as supporting the claimants’ case. I assume any plans for what was referred to as Thin Client are either developed or developing, but there is no need to speculate.

448.

I return therefore to the passage at [8] above which is reproduced from Dr Worden’s report where he was dealing with the double entry accounting principle. He stated that “any mistake is most likely to have occurred in one column of figures, without any balancing mistakes on the other columns. So, the mistake will immediately destroy the trial balance.” The consequence of the many bugs that I have found that exist in Horizon, in both its Legacy and Online form, have the effect of making the detection of errors so much harder than in conventional double entry bookkeeping. This is demonstrated by so many of the Fujitsu documents where a great amount of investigation was required by Fujitsu simply to work out what had actually happened, and why. On many investigations this took even Fujitsu an extraordinary length of time, in excess of a year, and sometimes even two years. Members of 3rd line support were involved, on multiple occasions, in this process. The experts are also agreed that SPMs only had a relatively limited amount of reporting available to them on Horizon, and that proper investigation required co-operation between the Post Office and the SPMs. It must also be understood that the trading periods for each Post Office branch were usually 4 weeks, and sometimes 5 weeks. At the end of each trading period a SPM had to complete a Branch Trading Statement and then “roll over” for the next period to start. If a bug in Horizon had an impact on branch accounts, then the Branch Trading Statement for that period would include the effect of that impact.

449.

One of the other functions of an accounting IT system was described by Dr Worden as the secure storage and output of many types of information. That secure storage (what was called in this litigation the audit store) is not effectively used if it is never, or rarely, accessed, particularly in circumstances where there is a dispute between both parties to the account. By “both parties” I refer to a SPM and the Post Office. This point is addressed in more detail in Part K of the substantive judgment.

450.

There is reference in some of the more recent documents to the involvement of Accenture. The analysis of the Drop and Go bug, number 28 in the Bug Table, shows that even when a conclusion was drawn by Accenture (in this instance relating to the failure of one of two £100 “top up” transactions), this did not prevent Fujitsu from identifying in the KEL the following: “Problem: this may be an issue with script ADCScript-CDBalanceTopUP, or a user error”, even though there was plainly no user error. User error means an error by the SPM. I do not know if this would be recorded

at Fujitsu in the same way if an incident similar to this were to occur today. However, it shows that for an occurrence even as relatively recently as the summer of 2017, Fujitsu failed to give sufficient – or indeed any - weight to the views of Accenture, who had “verified” that one of the top-ups had failed. Fujitsu still recorded potential user error in the KEL, a point which is plainly contradictory to the outcome of Accenture’s investigation in that instance.

451.

The expression put to Ms Van Den Bogerd in cross-examination was “user error bias”, an expression which she said she had not heard before. This term was explained as a tendency or bias regularly to blame the user of an IT system for something, and the point put to her was that this was “a common theme that runs through the Post Office’s approach” in this dispute. The exact passages of her evidence are as follows: “Q. UEB, have you ever heard of that? A. No.

Q. User error bias?

A. No, I can't say --

Q. It is where people in IT constantly blame the user when actually it is not their fault?

A. Okay, I haven't heard that before.

Q. You haven't heard that?

A. No.

Q. Well, let's look at this because I'm suggesting to you this is a common theme that runs through Post Office's approach when these issues are raised. Is that a fair suggestion?

A. I think what I have tried to explain is what the approach is, in 99 out of 100 that would be the case. In this situation it is slightly different.”

452.

I do not know from where Ms Van Den Bogerd obtained her 99% statistic. Given there are 6 million transactions per day, it may not be particularly helpful to the Post Office in this group litigation even if it were accurate. If it was meant as a 99% statement of certainty on her part of the strength of the Post Office’s case on the Horizon Issues, it is plainly very wrong, as demonstrated by my findings both in this appendix and the substantive judgment. As can be seen from the answers to the Horizon Issues in Part M of the judgment, the claimants have been largely successful in their case on the Horizon Issues. In my judgment – and regardless of whether Fujitsu and/or the Post Office was applying user error bias, or not – it is clear to me that Fujitsu was far too ready, even after investigations that clearly included the express discovery of bugs in code, to ascribe possible user error to the effect of bugs, errors and defects that caused impact to branch accounts. It is to be hoped that this is no longer the situation.

Case No: HQ16X01238, HQ17X02637 and HQ17X04248

Neutral Citation Number: [2019] EWHC 3408 (QB)

THE POST OFFICE GROUP LITIGATION

IN THE HIGH COURT OF JUSTICE

QUEENS BENCH DIVISION

Rolls Building Fetter Lane London, EC4A 1NL

Date: 16 December 2019

Before :

THE HONOURABLE MR JUSTICE FRASER

-

- - - - - - - - - - - - - - - - - - - -

Between :

Alan Bates and Others

Claimant

- and –

Post Office Limited

Defendant

-

- - - - - - - - - - - - - - - - - - - -

Appendix 2

Summary of Bugs, Errors, Defects

Judgment (No.6) “Horizon Issues”

-

- - - - - - - - - - - - - - - - - - - -

Summary of Bugs, Errors and Defects (Legacy Horizon or Online) and Years Experienced

1.

Receipts and Payments Mis-match bug. This is a bug present in Horizon Online This bug occurred in 2010.

The majority of incidents are recorded as occurring between August and October 2010.

2.

Callendar Square/Falkirk bug.

This is a bug present in Legacy Horizon.

It is agreed that this bug occurred between the years of 2000 and 2006, although there is an issue about when it stopped. In my judgment, the period when the effects of this occurred are 2000 to 2010.

3.

Suspense Account bug.

This is a bug present in Horizon Online.

Its identified years of effect are 2010 to 2013.

4.

Dalmellington bug/Branch Outreach Issue. This is a bug present in Horizon Online.

Its effects were experienced between 2010 and 2015.

5.

Remming In bug.

This is a bug present in Horizon Online.

It was present for about five months in 2010 between March and August.

6.

Remming Out bug

This is a bug present in Legacy Horizon.

It was split into two in the Bug Table, 6(i) and 6(ii).

Bug 6(i) was identified as February/April 2007, recorded as fixed approx. 2007. Bug 6(ii) was in May 2005.

7.

Local Suspense Account issue, not the same as 3. Suspense Account bug This is a bug present in Horizon Online.

This was reported in 2010 and recorded as fixed in September 2010.

8.

Recovery Issues.

These are bugs, errors and defects present in Horizon Online. With years of effect from 2010 to 2018.

9.

Reversals.

This is a bug present in Legacy Horizon.

Occurred for a short period in 2003.

10.

Data Tree Build Failure discrepancies.

This is a bug present in Legacy Horizon.

Its identified effect was during 1999 and 2000, running up to 2007.

11.

Girobank discrepancies.

This is a bug present in Legacy Horizon.

This is treated as a single bug by the experts in the Bug Table. The Technical Appendix finds there was more than one bug, error or defect under this heading. It occurred between May and September 2000.

12.

Counter-replacement issues.

This is a bug, error, or defect present in Legacy Horizon. It also included hardware issues. There were two KELs associated with this. The first was created in 2000 and last updated in July 2007. The second was created in 2002 and notes occurrences running up to 2009.

13.

Withdrawn stock discrepancies.

This is a bug present in Horizon Online.

Mentioned in 2010 and 2011.

14.

Bureau discrepancies.

This is a bug present in Horizon Online.

It arose in 2017.

15.

Phantom Transactions.

This is a bug present in Legacy Horizon. This is referable to Horizon issue 4 and not Horizon issue 1.

It arose in 2001.

16.

Reconciliation issues.

This is a bug present in both Legacy Horizon and Horizon Online.

This is not a bug with a potential to cause impact on branch accounts. The underlying PEAKs do however demonstrate software faults. Mentioned in 2000, 2002, 2010 and 2014.

17.

Branch Customer discrepancies.

This arises in Legacy Horizon. This is referable to Horizon issue 4 and not Horizon issue 1.

This is an error, or defect and does not have the potential to cause impact to branch accounts. Occurred in March 2008.

18.

Concurrent logins.

This is a bug present in Legacy Horizon.

This occurred in 1999 and 2000, and the PEAKS run to 2002.

19.

Post & Go/TA discrepancies in POLSAP.

This is a bug present in Horizon Online.

Occurred in 2012

20.

Recovery Failures.

This arises under Horizon Online. This is referable to Horizon issue 4 and not Horizon issue 1.

Two of the PEAKs show errors and defects, they do not show bugs. Mentioned in 2010, 2012 and 2015.

21.

Transaction Correction Issues.

This arises in Legacy Horizon. This is referable to Horizon issue 4 and not Horizon issue 1. Although the Technical Appendix finds it is evidence of a bug, there is no impact on branch accounts.

Occurred in 2005 to 2010.

22.

Bugs/errors/defects introduced by previously applied PEAK fixes.

This is a bug present in Legacy Horizon. There is no impact on branch accounts and this is therefore taken into account under Horizon issue 4 and Horizon issue 1. Some of the PEAKS are from 2000 and one is from 2004.

23.

Bureau de change.

This is a bug present in both Legacy Horizon and Horizon Online. Arose in 2005, 2006 and 2010 onwards.

24.

Wrong branch customer change displayed.

This is a bug present in Legacy Horizon.

Occurred in 2005.

25.

Lyca top up.

This is a bug present in Horizon Online.

Occurred in August 2010.

26.

TPSC 250 Report.

This is a bug present in Legacy Horizon.

The date range is between 2005 and 2009, with the majority of the incidents occurring in 2005

27.

TPS.

This is a bug present in Legacy Horizon.

Its years of effect are 2006 to 2010

28.

Drop and Go.

This is a bug present in Horizon Online.

Occurred in July 2017. The date of the KEL suggests HNG-A, but the KEL is expressly headed HNG-X.

29.

Network Banking Bug.

This arises under Legacy Horizon.

This did not have financial impact on branch accounts. It is an error, or defect but arises from communication failures from outside Horizon, probably from equipment or lines provided by BT.

KEL raised in 2004 and updated in 2005, with 12 associated PEAKs in the range 2004 and continuing on to 2010 when Legacy Horizon was discontinued.

Case No: HQ16X01238, HQ17X02637 and HQ17X04248

Neutral Citation Number: [2019] EWHC 3408 (QB)

THE POST OFFICE GROUP LITIGATION

IN THE HIGH COURT OF JUSTICE

QUEENS BENCH DIVISION

Rolls Building Fetter Lane London, EC4A 1NL

Date: 16 December 2019

Before :

THE HONOURABLE MR JUSTICE FRASER

-

- - - - - - - - - - - - - - - - - - - -

Between :

Alan Bates and Others

Claimant

- and –

Post Office Limited

Defendant

-

- - - - - - - - - - - - - - - - - - - -

Appendix 3

Glossary

Judgment (No.6) “Horizon Issues”

-

- - - - - - - - - - - - - - - - - - - -

Term

Meaning

AP

Automated Payment or Automated Debt Collection

APOP

Automated Payment Out-pay Database

APS

Automated Payment Service

BAL

Branch Access Layer

BMS

Business Management System

BP

Balancing Period

BRDB

Branch Database

BTS

Branch Trading Statement

CA

Cash Account

CAP

Cash Account Period

CMMI

Capability Maturity Model Integration

DBA

Database administrator

DBMS

Database Management System

DEA

Double entry accounting

DRS

Data Reconciliation Service

DVLA

Driver and Vehicle Licencing Agency

DWH

Data Warehouse

DWP

Department of Word & Pensions

EPOSS

Electronic Point of Sale Software

ERP

Enterprise Resource Planning

HLD

High Level Design

HNG

Horizon New Generation (or Online) implemented in 2010

HNG - A

Horizon Online since 2017

HNG – X

Horizon Online 2010 – 2017

HSD

Horizon Service Desk

IS

Infrastructure Services

JSN

Journal Sequence Number

KEL

Known Error Log

Legacy Horizon

Horizon 2000 – 2010

LFS

Logistics Feeder Service

MD5

Technique for creating a hash code for some piece of data, which would change if the data was altered in any way

MID

Manual Inspection of data

MIS

Management Information System

MSC

Managed Service Change

NBSC

Network Business Support Centre

Pathway

ICL division created for Horizon, which later became part of Fujitsu

PCI

Payment Card Industry

PDL

Process Definition Language (an example of reference data)

POA

Post Office Account

POL FS

Earlier name for POLSAP, from 2004 – 2010

POLSAP

PO’s central accounting system since 2010

POS

Point of Sale

PRINCE2

Project Management Methodology

PTR

Pre-Trial Review

RAC

Real Application Custer

RDDS

Reference Data Delivery Service

RDMC

Reference Data Management Centre

RDMS

Reference Data Management System

RDS

Redundant data storage and computing

remming

Remitting cash or stock

Riposte

Software product from Escher, used to build software for Post Offices

SAP

Industry leading resource planning and accounting system

SAP ADS

SAP applied to administrative data services

SEC

Security

SEK

Secure kernel hardware and software

SLA

Service Level Agreement

SMC

System Management Centre

SPM

Subpostmaster

SSC

Software Support Centre

SU

Stock Unit

TA

Transaction Acknowledgement

TC

Transaction Correction

TES

Transaction Enquiry Service

TIN

Transactional integrity and recovery

TIP

Transaction Information Processing

TP

Trading Period

TPS

Transaction Processing Service

VPN

Virtual Private Network

WSPOS

Web Services POS

XML

Extensible Message Language

Bates & Ors v the Post Office Ltd (No 6: Horizon Issues) (Rev 1)

[2019] EWHC 3408 (QB)

Download options

Download this judgment as a PDF (5.0 MB)

The original format of the judgment as handed down by the court, for printing and downloading.

Download this judgment as XML

The judgment in machine-readable LegalDocML format for developers, data scientists and researchers.