Case No: HQ16X01238, HQ17X02637 and HQ17X04248
Neutral Citation Number: [2019] EWHC 3408 (QB)
THE POST OFFICE GROUP LITIGATION
IN THE HIGH COURT OF JUSTICE
QUEENS BENCH DIVISION
Rolls Building Fetter Lane London, EC4A 1NL
Date: 16 December 2019
Before :
THE HONOURABLE MR JUSTICE FRASER
- - - - - - - - - - - - - - - - - - - - -
Between :
Alan Bates and Others
Claimants
- and -
Post Office Limited
Defendant
- - - - - - - - - - - - - - - - - - - -
Judgment (No.6) “Horizon Issues”
- - - - - - - - - - - - - - - - - - - -
Patrick Green QC, Kathleen Donnelly, Ognjen Miletic and Reanne MacKenzie for the Claimants (instructed by Freeths LLP)
Antony de Garr Robinson QC, Simon Henderson, Owain Draper and Rebecca Keating for the Defendant (instructed by Womble Bond Dickinson LLP)
Hearing dates: 11, 12, 13, 14, 18, 19, 20, 21 and 27 March;
3, 9 and 12 April; 4, 5, 6, 7, 11, 13 and 14 June 2019;
1 and 2 July 2019
Draft distributed to parties on 28 November 2019
- - - - - - - - - - - - - - - - - - - -
Mr Justice Fraser:
This judgment is part of the long-running legal dispute between the claimants, who are all either former or serving sub-post masters and sub-post mistresses (“SPMs”), and the Post Office. All the SPMs in the litigation had contracted with the Post Office at different times to run branch Post Offices in different locations across the country. The Post Office introduced a computer system called Horizon in 2000 across all its branches. That was changed in 2010 to an online version called Horizon Online or HNG-X, and the former version is now called Legacy Horizon. The claimants’ case is essentially that both Legacy Horizon and Horizon Online (which used many elements of the existing system) were or are unreliable, and this led to unexplained shortfalls and discrepancies in their branch accounts. The Post Office denies this, asserting that the systems were and are robust, and extremely unlikely to be the cause of such matters. There are numerous different causes of action brought against the Post Office, and the Post Office counterclaims against the claimants, including seeking damages for fraud. This is group litigation under CPR Part 19.
This judgment concerns the operation and functionality of the Horizon system itself. It follows the lengthy Common Issues judgment, which is Judgment (No.3). That judgment was extremely long, for two main reasons. It concerned six different claimants, each of whom had contracted with the Post Office at different times, and each of those six claimants’ cases concerned very different facts. Very few of the material facts relating to contract formation were agreed for any of these six, and therefore factual findings had to be made for each of them in this respect. Approximately 500 paragraphs were necessary to do this. Although first instance judgments must be reasoned so that parties understand the outcome (and so that any judgment can be subject to appropriate review), very lengthy judgments can be difficult to follow. If a judgment becomes too long, or too technical, it can be counterproductive to wider understanding on the part of those not immersed in the fine detail of every aspect of the case.
I have therefore included technical detail about the Horizon system, its operation, and some aspects of the technical evidence, in a Technical Appendix to this judgment. The contents of that appendix should not be seen as being of subordinate effect to the contents of the judgment itself; this is done simply for the convenience of readers. It is also intended that readers who do need to immerse themselves in technical matters to a very fine level of detail may not find it necessary to study the appendix, and reading the judgment alone may be sufficient. Notwithstanding that approach, however, this judgment too is extremely long. This is due to the complexity of the Horizon system, which even for computer systems is extraordinarily complicated for the reasons explained below; the period of time over which the complaints range (which starts with the introduction of Horizon in 2000 and runs to date); and due to the way that the litigation has unfolded. There are some matters in this judgment that go to issues affecting the group litigation going forwards, such as disclosure, which have increased its length. It is also the case that this litigation is being very strongly contested on both sides. I have endeavoured to provide a reasonable level of detail to explain my findings on the Horizon Issues to assist both sides as much as possible. Finally, the only findings that are made in this judgment are those necessary to come to conclusions on the Horizon Issues. All other matters in all the claims of the claimants and the counterclaims by the Post Office remain for decision in later judgments.
This judgment is in the following parts:
Paragraph no.
Introduction 5
The Horizon Issues 18
Features of this Group Litigation 57
Evidence of Fact: The Claimants 76 E. Evidence of Fact: The Post Office 202
Documents and Available Information 559
The Experts’ Agreements 654
The Period 21 March 2019 to 4 June 2019 706
The Expert Evidence 731
Conclusion on Expert Evidence 870
Audit Data 905
Overall Conclusions 925
Answers to the Horizon Issues 965
Appendices:
Technical Appendix
Summary of bugs, errors and defects
Glossary
A. Introduction
These proceedings are being conducted pursuant to a Group Litigation Order (“GLO”) made on 22 March 2017 by Senior Master Fontaine, and approved by the President of the Queen’s Bench Division. A more comprehensive introduction to the issues generally between the parties is contained at [2] to [43] of Judgment (No.3) “Common Issues”, which is at [2019] EWHC 606 (QB). Other judgments concerning procedural rather than substantive issues, are the first Judgment at [2017] EWHC 2844 (QB) and Judgment (No.2) “Strike Out” at [2018] EWHC 2698 (QB). During the Horizon Issues trial which is the subject of this judgment, the Post Office issued an application that I recuse myself as Managing Judge in this group litigation, and stop the Horizon Issues trial, so that it could be recommenced at some later date in the future (before a replacement Managing Judge). That application was refused and the judgment is at Judgment (No.4) “Recusal Application” at [2019] EWHC 871 (QB). Permission to appeal was refused by the single Lord Justice on 9 May 2019. There is also Judgment (No.5) “Common Issues Costs” at [2019] EWHC 1373 (QB) which made various orders in respect of the costs of the Common Issues trial, a hearing in respect of that having taken place on 23 May 2019.
This trial concerns what the parties referred to at the case management stage as the Horizon Issues. The intention behind the case management of this litigation was that the contractual issues (which were called the “Common Issues” and affected all the claimants) and the computer issues relating to the operation, functionality and reliability of the Horizon system (which were called the “Horizon Issues”, and also affected all the claimants) would be resolved first. The parties needed time to prepare for these two trials, which took place in late 2018 (for the Common Issues) and into
the spring of 2019 (for this one), in particular to perform disclosure, and prepare and serve evidence. For the Horizon Issues, this included expert evidence, for which I gave permission, from two IT experts, one for the claimants (Mr Coyne) and one for the Post Office (Dr Worden).
The recusal application caused significant delay and disruption to the Horizon Issues trial. Originally, that trial had been programmed to finish (including delivery of oral closing submissions) by 11 April 2019, a date which became unachievable as a result of that application. Following Judgment (No.4) the remaining witness of fact for the Post Office to be called in person, Mr Parker, a senior Fujitsu employee, was called on 11 April 2019.
The two IT experts could not be called until 4 June 2019 onwards. The two experts therefore had a period of some weeks, following on from the evidence of fact, before they were called to give their oral evidence. There was therefore an interval between 11 April 2019 and 4 June 2019 when the experts were called. Their evidence collectively took two court weeks.
The effect of this interruption upon the trial timetable meant that there was therefore time available for each of the experts to consider the full scope of the crossexamination of the factual witnesses that had taken place some weeks before, and reflect (with more time than is usual for experts whilst a trial is underway) upon whether that impacted upon their views. Experts are supposed to consider the factual evidence advanced by both sides in litigation neutrally, and if witnesses are crossexamined, then – at least potentially – other evidence that might emerge ought, if it is important, to be considered by those experts in arriving at their final opinions given in evidence. In a conventional trial, expert evidence will usually follow on almost immediately after the evidence of fact has been heard. This means that, in practical terms, most experts will only have a very short time in which to consider any potentially important factual developments in any case before they are themselves cross-examined. Here, there was no such restriction.
The Horizon Issues trial timetable was therefore different to a conventional trial in this respect, because the vast majority of the oral factual evidence, with the single exception of Mr Parker’s evidence, was completed by 21 March 2019. That is 75 days, or almost 11 weeks, before Mr Coyne was called by the claimants, and even longer before Dr Worden was called by the Post Office. The approach of the experts to the opportunity presented by that interval further to assist the court with their evidence is notable, in my judgment. I will deal with this further in Parts H and I of this judgment. For a more detailed description of the architecture of the Horizon system (both Legacy Horizon and Horizon Online) reference should be made to the Technical Appendix. For a general understanding of the issues in this litigation, an overview is sufficient.
Horizon is both an accounting system, and also supports a large number of what are called business applications. Some transactions that a customer might wish to carry out in their local Post Office branch are what are called retail activities, such as buying a Post Office product. One example of this is buying a book of stamps. However, the Post Office has a large number of business customers whose products are offered to the public through branch Post Offices. The Post Office calls these companies “clients”. For every kind of activity which a customer wishes to transact in their Post Office branch, Horizon needs to have the functionality to perform it. This functionality means supporting the counter activity of carrying out the transaction, which is another way of saying Horizon enables the SPM or their assistant to transact the business over the counter. That business may be a combination of Post Office retail activity, and purchasing services or products offered by the Post Office on behalf of its clients. However, in conjunction with the ability of a SPM to serve a customer (what is called counter or front office), there is the associated “back office activity” of settling with Post Office's 'client' organisation, who has provided some service to the customer - such as the DVLA, or a bank. Accounting is a theme or thread that runs through all of these business requirements of Horizon, but it is only a part of them. The number of services provided by Post Office branches is large and has increased steadily from 1998 to the present day. The number of clients has almost certainly increased, and the functionality of Horizon has expanded in line with the growth in service, both on the counter and in the back office.
For those activities where the Post Office branch is acting like a retail outlet (such as selling stamps), both the hardware and software is provided by Horizon to support this activity. This is the Electronic Point of Sale Software component of Horizon, that is referred to as “EPOSS”. EPOSS allows the SPM or assistant to record that some goods have been provided to a customer, compute the price of those goods, and allow the customer to pay the money required for all their purchased goods, using either cash or a credit/debit card. Often, a customer may wish to carry out two or more different activities in one visit to the counter - for instance, to settle a utilities bill and to buy some stamps. This can be done in the same activity and so Horizon has the concept of a customer carrying out a 'basket' of activities and settling the total amount due for the basket in several ways - by one single credit card transaction, by a cheque, or by cash.
Baskets of Post Office activities and non-Post Office activities are not supported by Horizon. Often, a local Post Office branch will be a retail outlet too, selling non-Post Office goods such as groceries, newspapers or even (as with Mr Bates at Craig-yDon) a haberdashery. If a customer wishes to buy a newspaper and some stamps, the newspaper is not sold by the Post Office; it is simply sold by the associated, though in accounting terms separate, retail outlet run by the SPM which uses the same premises. So, the customer has to settle in two parts. In some premises, a customer may queue up to purchase (say) greetings cards and pens (from the retail side of the branch Post Office). If they then wish to perform some activities with the Post Office, they may then need to queue up at a separate counter position to do that; that separate and second activity would be transacted through Horizon. In this respect, the National Lottery is an exception and spans the two businesses.
The Post Office’s clients include high street banks, Camelot, gas and electricity companies (for paying utility bills), DWP (for paying benefits and pensions) and DVLA (for paying road fund tax). Because of the different nature of the services provided through the Post Office for all of these many hundreds of client organisations, the services provided through the Post Office will be different from the service provided for other clients. There is therefore the need for some unique software functionality within Horizon. This must be provided both in the branch and the back office to support the activities for that client. This is a part of what makes Horizon such a large and complex system. The other reason that Horizon is so complex is because it has evolved over a long period. As will be seen below, the original concept and design for Horizon was for a joint Post Office/DWP project so that welfare benefits could be paid to benefit claimants in a certain way. That changed and the DWP withdrew from the project, but the need to offer other and an increasing number of products has led to many additions being added on to the system, increasing its complexity over time.
As well as the counter activities necessary to run a branch Post Office, Horizon also supports the periodic process of balancing and rollover for each branch; essential elements in the accounting to the Post Office performed by the SPM. Every branch operates in Trading Periods, which are either four or five weeks (according to a timetable published periodically by Post Office). At the start of each Trading Period the branch is supposed to be 'in balance'. This means that the physical stock and cash in the branch agrees with the data regarding the stock and cash held in Horizon. Then, during the Trading Period, Horizon records all customer transactions made at the branch, so it records the changes in cash and stock. It is also required to record any replenishments or remittances of cash or stock in the branch. Thus, Horizon records all changes in cash and stock held at the branch during the Trading Period, and should be able to compute, from the starting amounts and the changes, the expected amounts of cash and stock at the end of the period.
Without reciting the entire background to the group litigation between the Post Office and the claimants, it is the accounting and functional accuracy of Horizon that is at the heart of the current disputes, which have run for a great many years. The claimants maintain that the Horizon system in operation threw up numerous discrepancies and shortfalls in their branch accounts, for which the Post Office unfairly held them responsible. The Post Office dispute this, and maintain that these occurrences are explicable by carelessness, fault or criminality on the part of the claimants.
Horizon, whether in its first incarnation as Legacy Horizon until 2010, or what is called Horizon Online since then (originally HNG-X, now since 2017 HNG-A running on a different windows system), is at the centre of this group litigation. It should also be noted that it is not a system that is operated by the Post Office. It was originally designed, “rolled out” and operated by ICL, a company that was partly owned by Fujitsu, and which is now fully absorbed within Fujitsu, and has been for many years. There is a contract between Fujitsu and the Post Office in respect of Horizon, and perhaps in its current form it is rather different than the original contractual relations between those two entities. The contractual relationship between the Post Office and Fujitsu is not of direct relevance to the Horizon Issues, but arises tangentially only in respect of what is a sub-issue, namely the charging structure operated by Fujitsu for what are called ARQ requests for audit data and whether this inhibited the Post Office in this respect. The operation, functionality and accuracy of Horizon is an extremely thorny issue (or bundle of issues), although given the breadth of allegation and counter-allegation in the group litigation they are not the only issues. Providing the answers to the Horizon Issues will not lead to complete resolution of the litigation. It should however resolve one of the central issues about Horizon’s accuracy and functionality.
B. The Horizon Issues
18. The Horizon Issues are as follows. In the 3rd Case Management Order of 1 March 2018, leading counsel for the parties were ordered to meet and seek to agree the Horizon Issues to be tried in March 2019. This was done, the issues were agreed by the parties and approved by the court. I take the following from the Case Management
Order dated 23 March 2018, which was the next order made, and was a Consent
Order. The paragraph references to the pleadings were included in the list of Horizon Issues, which were themselves appended to the Order of 23 March 2018 itself. These include at the beginning an agreed definition of “the Horizon System”.
“AGREED LIST OF HORIZON ISSUES
definitions for the purpose of this list of issues
“the Horizon System” shall for the purposes of this list of issues mean the Horizon computer system hardware and software, communications equipment in branch and central data centres where records of transactions made in branch were processed, as defined in the Generic Particulars of Claim (“the GPOC”), at §16 and as admitted by Post Office in its Defence at §37.
BUGS, ERRORS AND DEFECTS IN HORIZON
Accuracy and integrity of data
To what extent was it possible or likely for bugs, errors or defects of the nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to 56 of the Generic Defence to have the potential (a) to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions as alleged at §24.1 GPOC?
Did the Horizon IT system itself alert Subpostmasters of such bugs, errors or defects as described in (1) above and if so how?
To what extent and in what respects is the Horizon System “robust” and extremely unlikely to be the cause of shortfalls in branches?
Controls and measures for preventing / fixing bugs and developing the system
To what extent has there been potential for errors in data recorded within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data in Horizon?
How, if at all, does the Horizon system itself compare transaction data recorded by Horizon against transaction data from sources outside of Horizon?
To what extent did measures and/or controls that existed on Horizon prevent, detect, identify, report or reduce an extremely low level of risk of the following:
data entry errors;
data packet or system level errors (including data processing, effecting, and recording the same);
a failure to detect, correct and remedy software coding errors or bugs;
errors in the transmission, replication and storage of transaction record data; and
the data stored in the central data centre not being an accurate record of transactions entered on branch terminals?
OPERATION OF HORIZON
Remote Access
(7) Were Post Office and/or Fujitsu able to access transaction data recorded by Horizon remotely (i.e. not from within a branch)?
Availability of Information and Report Writing
What transaction data and reporting functions were available through Horizon to Post Office for identifying the occurrence of alleged shortfalls and the causes of alleged shortfalls in branches, including whether they were caused by bugs, errors and/or defects in the Horizon system?
At all material times, what transaction data and reporting functions (if any) were available through Horizon to Subpostmasters for:
identifying apparent or alleged discrepancies and shortfalls and/or the causes of the same; and
accessing and identifying transactions recorded on Horizon?
Access to and/or Editing of Transactions and Branch Accounts
(10) Whether the Defendant and/or Fujitsu have had the ability/facility to (i) insert, inject, edit or delete transaction data or data in branch accounts; (ii) implement fixes in Horizon that had the potential to affect transaction data or data in branch accounts; or (iii) rebuild branch transaction data:
at all;
without the knowledge of the Subpostmasters in question; and
without the consent of the Subpostmaster in question.
If they did, did the Horizon system have any permission controls upon the use of the above facility, and did the system maintain a log of such actions and such permission controls?
If the Defendant and/or Fujitsu did have such ability, how often was that used, if at all?
To what extent did use of any such facility have the potential to affect the reliability of the Branches’ accounting positions?
Branch trading statements, making good and disputing shortfalls
(14) How (if at all) does the Horizon system and its functionality:
enable Subpostmasters to compare the stock and cash in branch against the stock and cash indicated on Horizon?
enable or require Subpostmasters to decide how to deal with, dispute, accept or make good alleged discrepancy by (i) providing his or her own personal funds or (ii) settling centrally?
record and reflect the consequence of raising a dispute on an alleged discrepancy, on Horizon Branch account data and, in particular:
does raising a dispute with the helpline cause a block to be placed on the value of an alleged shortfall; and
is that recorded on the Horizon system as a debt due to Post Office?
enable Subpostmasters to produce (i) Cash Account before 2005 and (ii) Branch Trading Statement after 2005?
enable or require Subpostmasters to continue to trade if they did not complete a Branch Trading Statement; and if so, on what basis and with what consequences on the Horizon system?
Transaction Corrections
(15) How did Horizon process and/or record Transaction Corrections?”
Given the nature of the proceedings, and the disputes about the Horizon system which were apparent on the face of the pleadings, I had indicated to the parties at an early stage that the generic disputes about the operation of the Horizon system, which would need expert evidence, would be resolved after the contractual issues (which became called the Common Issues, and led to Judgment (No.3)). The issues were not foisted on the parties by the court. The court approved the wording of the issues agreed by the parties during the case management stage of the litigation. There was no difficulty about this at the time.
However, the meaning of issues 1 and 3 in particular proved, at the Horizon Issues trial itself, to be controversial. This is regrettable. Nor was any controversy aired with the court prior to the actual trial. That too is regrettable. The court is well used to parties who disagree over the answers to certain issues in litigation generally. It is unsatisfactory when they also disagree about what any particular issue itself actually
means, or the question that is being posed (including being posed to experts) by an issue, particularly once that issue has already been finalised and ordered.
Here, the two main areas of dispute over the meaning of the Horizon Issues were as follows (there were others, but these are the most important ones). The claimants approached Issue 1 as it was worded, requiring consideration of whether bugs, errors or defects had the potential to cause discrepancies or shortfalls in SPMs’ branch accounts or transactions. In other words, this issue was not limited to consideration of whether bugs, errors or defects had in fact actually caused discrepancies or shortfalls. The Post Office, on the other hand, approached Issue 1 as requiring consideration of whether bugs, errors or defects had in fact actually caused discrepancies or shortfalls, and by reference to the claimants’ accounts specifically. This approach by the Post Office sought, in my judgment, to narrow the scope of the Horizon Issues. This was not raised at the case management stage.
The second main area of dispute concerned Issue 3. This includes consideration of the concept of “robustness”. This word has been at the heart of the Post Office’s defence of the Horizon system for many years. The Post Office has said publicly, and on many occasions, that no computer system is 100% accurate and/or perfect but Horizon is “robust”. This approach by the Post Office pre-dated the commencement of the group litigation and was at the heart of the Post Office’s response to the increasing criticisms from different quarters about Horizon. I deal with this important concept below in more detail at [36] below. What the Post Office effectively means by this (in outline terms only) is that the Horizon system can properly and safely be relied upon by the Post Office for the purposes for which it is designed and intended. The parties – and this is clear on the pleadings, as well as in the terms of Horizon Issue 3 – disagree over whether Horizon is robust. In the Reply, the claimants challenged this in the following terms:
“It is therefore denied that Horizon 'is robust and [...] is extremely unlikely to the cause of losses in branches' (paragraph 16). In fact, the relatively small chance of errors admitted by the Defendant, would be likely to produce the very picture reflected in the Claimants' case.”
Mr de Garr Robinson for the Post Office was somewhat critical at the trial of the drafting of both Issues 1 and 3. He was not involved in the drafting of the Horizon Issues, as the Post Office had used different leading counsel. However, that criticism overlooked that the Post Office had agreed to the wording of all the Horizon Issues through its other leading counsel, and the wording had also been approved by the court. Although there have been a total of three leading counsel altogether instructed for the Post Office thus far at first instance, with a fourth leading counsel used by the Post Office to try and appeal the Common Issues to the Court of Appeal, simply because different counsel are used does not mean that the issues ordered to be tried become different. The Horizon Issues themselves were appended to the Case Management Order itself.
Lists of issues, whether in group litigation or generally, are very important. I can do no better than quote Longmore LJ in Scicluna v Zippy Stitch Ltd [2018] EWCA Civ
1320, who said at [14] “ever since the Woolf reforms, parties in the High Court have been required to agree lists of issues formulating the points which need to be determined by the judge. That list of issues then constitutes the road map by which the judge is to navigate his or her way to a just determination of the case.” In this litigation, the Horizon Issues are not only important, but are vitally important. The whole of the proceedings concerns the operation and reliability of the Horizon system. It could quite easily be called the Horizon Post Office Case. Although there are a great number of different causes of action pleaded in both directions, both by the claimants and by the Post Office, against one another, the central core of the case is about Horizon. I consider that the wording of both Horizon Issues 1 and 3 – indeed, all the Horizon Issues – to be clear and had they not been, I would not have approved the agreed wording.
However, some explanation is necessary given the disagreement to which I have referred.
The phrase “bugs, errors or defects” is sufficiently wide to capture the many different faults or characteristics by which a computer system might not work correctly. The parties in this litigation, usually but not always for convenience, would often refer simply to “bugs”, and one part of the 2nd Experts’ Joint Statement became known as “the Bug Table”. Computer professionals will often refer simply to “code”, and a software bug can refer to errors within a system’s source code, but “software bugs” has become more of a general term and is not restricted, in my judgment, to meaning an error or defect specifically within source code, or even code in an operating system. Source code is not the only type of software used in a system, particularly in a complex system such as Horizon which uses numerous applications or programmes too. Reference data is part of the software of most modern systems, and this can be changed without the underlying code necessarily being changed. Indeed, that is one of the attractions of reference data. Software bug means something within a system that causes it to cause an incorrect or unexpected result. During Mr de Garr Robinson’s cross-examination of Mr Roll, he concentrated on “code” very specifically and carefully. There is more to the criticisms levelled at Horizon by the claimants than complaints merely about bugs within the Horizon source code.
The wording of Horizon Issue 1, “to what extent was it possible or likely for bugs, errors or defects…..” is therefore very wide. Although that wording was proposed and agreed by the parties, I considered at the case management stage when the Horizon Issues were ordered, and still consider now, that these words clearly cover the whole range of criticisms levelled at Horizon by the claimants, and the matters that could potentially be wrong with the Horizon system without restriction. Bugs, errors or defects is not a phrase restricted solely to something contained in the source code, or any code. It includes, for example, data errors, data packet errors, data corruption, duplication of entries, errors in reference data and/or the operation of the system, as well as a very wide type of different problems or defects within the system. “Bugs, errors or defects” is wide enough wording to include a wide range of alleged problems with the system.
I also consider that the words “possible or likely” are also wide enough to cover both ends of the spectrum of what the parties would have the court decide. In other words, it is a neutrally worded issue of wide effect which does not, by its phrasing, indicate any particular starting point or end point. “Possible” means something potentially could happen; or whether it could happen or not. “Likely” means that something is, on
the balance of probabilities, more likely to have happened than not. The issue uses both terms, and therefore poses a wide question so that the answer provided by the court will be of maximum utility in the group litigation.
Finally, the word “potential”, in the clause “have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon…..” clearly means, on its express terms, that this issue is dealing with possible or prospective effects, rather than whether bugs, errors or defects have in fact caused actual or alleged discrepancies or shortfalls. The way the wording of this issue was dealt with in the expert evidence will be dealt with below.
If there were any ambiguity in Issue 1, and I do not consider that there is, the issue specifically refers to certain passages in the pleadings. It would therefore be of assistance to consider the pleadings. The relevant passages from the Particulars of Claim as amended are as follows:
“23. However, the Claimants aver that there were a large number of software coding errors, bugs or defects which required fixes to be developed and implemented. There were also data or data packet errors. There was a frequent need for Fujitsu to rebuild branch transaction data from backups, giving rise to the further risk of error being introduced into the branch transaction records. The Claimants understand that Fujitsu maintained a 'Known Error Log' relating to some or all of these issues which was provided to the Defendant but which has not been disclosed.
24. Further, the Claimants aver and rely upon the following:
Insufficient error repellency in the system (including sufficient prevention, detection, identification and reporting of errors), both at the data entry level and at the data packet or system level (including data processing, effecting and reconciling transactions, and recording the same);
24.1A bugs and/or errors and/or defects in Horizon and any data or data packet errors had the potential to produce apparent shortfalls which did not represent a real loss to the Defendant;
Horizon is imperfect and has the potential for creating errors (as the Defendant has admitted in pre-action correspondence, in the Letter of Response, dated 28 July 2016, at paragraph 1.3);
bugs and/or errors have on some occasions produced discrepancies and/or apparent shortfalls (as the Defendant has admitted in pre-action correspondence, in the Letter of Response, Schedule 6) and such shortfalls may also have arisen from data or data packet errors; and, further
the Defendant sought and/or recovered such alleged shortfalls from Subpostmasters (as is presently understood to be admitted by the Defendant in the Letter of Response, Schedule 6, paragraphs 4.1 to 4.5).”
The passages in the Defence which are identified in the Issue are paragraphs 49 to 56. This is a lengthy series of paragraphs but as all are referred to expressly within Issue 1, I will reproduce them all. These state the following:
“Bugs, errors or defects in Horizon
49. As to paragraph 22:
If and to the extent that the Claimants wish to assert that any of the shortfalls for which they were held responsible were Horizon-generated shortfalls, it is for them to make that distinct allegation and seek to prove it. Post Office notes that they do not make the allegation in the GPoC. It further notes that, in paragraph 20 of their solicitors' letter to Post Office's solicitors dated 27 October 2016, the Claimants make it clear that they do not allege that there is a systematic flaw in Horizon or indeed any flaw which has caused any Claimant to be wrongly held responsible for any shortfall.
It is denied that Post Office has unreasonably or otherwise failed to provide "obviously relevant disclosure" in relation to bugs, errors or defects in Horizon. There has been no order or application for disclosure and, in the premises set out above, there appears to be no basis for providing such disclosure.
50. Paragraph 23 is embarrassing for its lack of particularity, in that (amongst other things) it does not identify the errors, bugs or defects on which the Claimants rely or how "large" their number was or the period in which they are said to have occurred and nor does it identify the transaction data that Fujitsu is alleged to have rebuilt, how "frequent" was the need to rebuild it or the extent of the ''risk of error" which is said to have been introduced. In the premises, Post Office cannot plead to the first three sentences of this paragraph. However:
All IT systems experience software coding errors or bugs which require fixes to be developed and implemented. As is noted in paragraph 53 and 54 below, there are robust measures in place in Horizon for their detection, correction and remediation.
All IT systems involving the transmission of data over the internet experience data or data packet errors during transmission and such systems routinely have protective measures in place to prevent such errors creating any difference between the data transmitted and the data received and retained by the recipient. Horizon has robust controls making it extremely unlikely that transaction data input in a branch would be corrupted when being transferred to, and stored in, Post Office's data centre in a manner that would not be detected and remedied.
Like all IT systems, Horizon has backups to guard against any loss of data due to local hardware failure. Where hardware fails, the data on that hardware is recovered from the backup. Post Office takes the term "rebuild" to refer to the situation before the introduction of Horizon Online where a new terminal was introduced to a branch and the data stored on the other branch terminals (or on a disc where it was a single counter branch) was restored to the new terminal. In this context, Post Office does not accept that there was a "frequent" need to "rebuild" data from back-ups.
It is admitted that Fujitsu maintain a "Known Error Log". This is not used by Post Office and nor is it in Post Office's control. To the best of Post Office's information and belief, the Known Error Log is a knowledge base document used by Fujitsu which explains how to deal with, or work around, minor issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide fixes have not been developed and implemented. It is not a record of software coding errors or bugs for which system-wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief, there is no issue in the Known Error Log that could affect the accuracy of a branch's accounts or the secure transmission and storage of transaction data.
In paragraph 24, the Claimants again bundle many ambiguous and/ or misleading allegations together. Post Office separates out and addresses those allegations in paragraphs 52 to 56 below.
As paragraph 24.1 does not explain what is meant by "error repellency", what sorts of errors are referred to, what is meant by "data entry level", what would constitute "sufficient" prevention, detection, identification or reporting of these errors, or in what respects the error repellency of Horizon was insufficient, Post Office cannot plead to this paragraph. However, the general thrust of paragraph 24.1 is denied and the robust controls, procedures and practices pleaded in paragraphs 53 and 54 below are noted.
As to paragraph 24.1A, it is a truism that errors or bugs in an IT system and data or data packet errors have the potential to create errors in the data held in that system. However, Horizon has at all material times included technical control measures to reduce to an extremely low level the risk of an error in the transmission, replication and storage of the transaction record data. These have varied from time to time and they currently include the following:
Horizon creates, transmits and stores trans action data in the form of "baskets". A basket is a complete transactional session between a customer and Post Office and may include one, several or many individual transactions taking place within the same session. Horizon will not accept a basket of transactions that does not net to zero (i.e. the value of any sales is set off by the value of any payment made or received). This reduces greatly the risk of any error in the data entered within any given basket.
If a basket of transactions fails properly to complete its transmission to the central database (because, for example, of a power loss), the system rejects any partial transmission and requests the full basket from the branch terminal. This reduces greatly the possibility of baskets of transactions failing to be recorded.
At the point of a basket being accepted by Horizon, it is assigned a unique sequential number (a "JSN") that allows it to be identified relative to the other baskets transmitted by that branch. This reduces greatly the risk of recording duplicate baskets or there being a missing basket.
Each basket is also given a digital signature, i.e. a unique code calculated by using industry standard cryptography. If the data in the basket were to change after the digital signature was generated, this would be apparent upon checking the digital signature.
Initial data integrity checks are undertaken when baskets are received at the Post Office data centre from a branch. Baskets are then copied from the central database to the Audit Store where a digital seal is then applied (the "Audit Store Seal"). If the baskets and/ or the data within the baskets were altered after the application of the Audit Store Seal, this would be apparent when the baskets are extracted from the Audit Store.
Horizon and the above controls are themselves subject to various audits and checks including audits carried out by third parties.
54. Further as to paragraph 24.1A, in addition to the technical controls referred to above, there are several operational procedures and practices conducted by Post Office and Subpostmasters that serve to increase the reliability of the data stored in the central data centre as an accurate record of the transactions entered on branch terminals. These currently include the following:
For many transaction types, Post Office compares its own transaction record against the corresponding records held by Post Office clients. If an error in Horizon were to result in the corruption of transaction data, this should be revealed by the comparison.
There are detailed procedures in place to address the risk of data loss resulting from interrupted sessions, power outages or telecommunications failures in branches. These are set out in the "Recovery - Horizon Online Quick Reference Guide" and Horizon guides the system user through the recovery process (which include completing any transactions that are cut short). These procedures should prevent any data errors arising from interrupted sessions, power outages and telecommunications failures.
The display of the transactions being effected on-screen at the branch terminal allows the user of the system to identify any inconsistency between the information shown on the screen and the transaction that the user has keyed into the system. If, for example, a hypothetical bug in the terminal were to cause a key-strike on number S to be recorded as an input of number 6, this would be detected rapidly by system users, given the large number of system users and the huge number of transactions effected on Horizon.
The accounting and record-keeping obligations placed on Subpostmasters reduce the risk of any errors going undetected. For example, there is an obligation for each branch to count and declare to Post Office the cash it holds on a daily basis, which increases the likelihood of promptly detecting any overstatement or understatement of the cash position on Horizon. If a Subpostmaster detects that an error has been made at an early stage, its cause is more likely to be identified.
Fujitsu operates industry standard processes for developing and updating Horizon and for investigating and resolving any identified potential system errors.
As to paragraph 24.2, Post Office admits that, like all other IT systems, Horizon is not a perfect system which has never had any errors or bugs. However, as indicated in paragraphs 53 and 54 above, it has robust systems in place to identify them, fix them and correct their consequences (if any).
As to paragraphs 24.3 and 24.4:
There have been occasions on which bugs or errors in Horizon have resulted in discrepancies and thus shortfalls or net gains in some branch accounts, as outlined in Schedule 6 of the Letter of Response. It is denied (if it be alleged) that such bugs or errors have affected any of the Claimants.
On each occasion, both the bugs or errors and the resulting discrepancies in the relevant branch accounts were corrected. Post Office took steps to ensure that it had identified all branches affected by the bugs or errors and that no Subpostmaster was ultimately held responsible for any resultant shortfalls.
Paragraphs 4.1 to 4.5 of Schedule 6 to the Letter of Response relate to the socalled Suspense Account Bug. Without prejudice to the burden of proof, none of the branches affected by the Suspense Account Bug are branches for which the Claimants were responsible.
None of the Subpostmasters whose branches were affected by the Suspense Account Bug were ultimately held responsible for the shortfalls that it generated. The Claimants are therefore wrong to understand Post Office as having admitted that it "recovered such alleged shortfalls from Subpostmasters". Where Subpostmasters in the affected branches had made good or settled centrally shortfalls that were later corrected, those Subpostmasters received a payment or credit in the amount of the shortfall.”
It can be seen that the word “robust” is used throughout these passages of the Defence, and “robust systems” in paragraph 55. Given “robustness” forms part of Issue 3 there is no need separately to consider it at this stage when analysing the differences between the parties about what Issue 1 means.
The passages in both parties’ pleadings which I have reproduced above show that it is the potential, in general terms, of the Horizon system to have the effect complained of that is the subject matter of Horizon Issue 1. In order to read Issue 1 in the manner contended for by the Post Office, one would have to delete entirely the phrase “to have the potential” from the issue. One would also have to substitute the term “Subpostmasters’ branch accounts or transactions” with “the claimants’ branch accounts or transactions”. That is not what Issue 1 states, it is not what the parties agreed the issue would be, and it is not the issue that the court approved and ordered. In my judgment, the approach of the Post Office to what Horizon Issue 1 actually means is too narrow.
In my judgment, the correct construction of Horizon Issue 1 is that contended for by the claimants. In other words, it involves a two-stage process. Firstly, consideration of whether there were, or are, bugs, errors or defects in the Horizon system as alleged by the claimants. Secondly, if the answer is that there were, or are, such bugs, errors or defects, the second stage is to consider whether these have (or did have previously) the potential to cause apparent or alleged discrepancies in SPMs’ branch accounts generally. The issue is not whether such bugs, errors or defects did in fact cause such discrepancies or shortfalls in the claimants’ accounts specifically. That separate or different issue – the effect upon claimants’ branch accounts - is a more claimant specific one. It will have to be determined at some stage, for any of the claimants whose individual claims come to trial in the future. It may require expert forensic accountancy evidence. It was not ordered to be dealt with in the Horizon Issues trial. The Horizon Issues were intended to be, and in my judgment on their wording are, generic issues relating to Horizon and its operation. However, if a bug, error or defect is shown to have had an actual impact on any SPM’s accounts, then by definition it has had the potential to have such an impact. Actual impact on branch accounts can therefore be of assistance in considering Issue 1 as ordered, namely including potential.
The passages in the pleadings quoted above do show that the Post Office had, in
Schedule 6 to its Letter of Response, accepted the existence of two bugs or errors in Horizon, one of which was called the Suspense Account Bug, but denied that the latter had affected any branch accounts of any of the claimants. The passages in paragraph 56 of the Defence that explained that no SPMs had been “ultimately held responsible for the shortfalls that it generated” reads a little differently with the hindsight provided from evidence in two lengthy trials. This is because the Post Office’s Defence effectively accepts that “Subpostmasters in the affected branches had made good or settled centrally” the sums in question – that is the shortfalls in their branch accounts that Horizon showed – but states that these were later corrected. Both the expressions “making good” and “settled centrally” means that the Post Office had originally held the SPM responsible for the losses, as the shortfalls formed part of their branch accounts. These terms mean that the SPMs had either paid the money (making good) or been given time to pay sums which the Post Office treated as a debt, even if they were disputed sums (settling centrally), meanings given to these terms that were confirmed by Ms Van Den Bogerd in her evidence for the Common Issues trial. However, the pleading means that the Post Office had then, on the Post Office’s case, subsequently corrected this impact on branch accounts which was why it pleaded no SPMs were “ultimately held responsible”. Therefore there might prove not to be as much between the parties on this point in reality, as the group litigation unfolds, as there appears to be on the pleaded case. The Suspense Account Bug undoubtedly had actual impact upon SPMs’ branch accounts.
The meaning of “robustness”
Turning to the disagreement about Issue 3, given the parties disagreed about whether the Horizon system is (or was) “robust”, it is a fairly elementary step to consider the meaning of that term, and how it is being used by the parties. Context is important so far as the meaning of the word “robust” is concerned. If someone is in robust health, it usually is taken to mean that they are healthy, or even very healthy. A robust exchange of views can be a polite way of referring to an argument. Given the importance of the concept to the Horizon system, its prominence in the Post Office’s defence of the system, and its express inclusion (admittedly in inverted commas) in the Horizon Issues, I asked each side in the litigation during oral closing submissions for a reference from their pleadings or submissions for the meaning which they ascribed to the word. I referred to this as their benchmark definition. Robustness was referred to by both sides in the litigation in numerous places, but not always in the same precise terms, and clarity is to be welcomed.
The claimants answered this by reference to the remainder of Issue 3, namely “extremely unlikely to be the cause of shortfalls in branches” and explained that the claimants had found the word robust “difficult to define” other than by reference to this. This would mean therefore that it had no separate independent meaning other than as a summary of the longer second part of Issue 3. In other words, a robust system would be one that is extremely unlikely to be the cause of shortfalls in branches. The claimants also implicitly, if not expressly, criticised use of the term both by the Post Office in its pleadings and written submissions as being more aligned to public relations than as a performance standard.
The Post Office asked for some time to provide the reference that I requested. Given the meaning of “robust” is so central in the Post Office’s defence of the Horizon system, I granted the Post Office the time that was requested.
The Post Office subsequently, after the trial ended, submitted a short document entitled “the Post Office’s case on the meaning of robustness”. This was not what was intended when I sought a reference from the Post Office to their definition, and the document submitted went rather further and made wider ranging submissions. The document did state, so far as the meaning of the word is concerned, the following: “In Post Office’s submission, the meaning of robustness is a matter for expert opinion. Robustness is a well-established concept in the IT industry and is the subject of academic study: see para. 361 of Post Office’s written closing”.
I do not consider that the meaning of words is a matter for expert opinion. The two experts in this case are IT experts, not experts in linguistics or the meaning of language. However, the meaning of robustness within the field of IT is, arguably, a matter upon which the experts’ opinions should be considered, not least because they were applying that term to their expert exercise. The Post Office also relied upon the 1st Experts’ Joint Statement which in respect of Issue 3 stated the following as agreed:
“There are different dimensions of robustness, such as robustness against hardware failure, software defects and user error. The robustness of the system also depends on the processes around it.
Robustness does not mean perfection; but that the consequences of imperfection must be managed appropriately. If the extent of imperfection is too high, this would be very difficult to do which would imply less robustness.
Horizon has evolved since its inception. Therefore, its robustness may have varied throughout its lifetime. The level of robustness may have increased or decreased as the system was changed.
The existence of branch shortfalls is agreed. The experts do not agree at this point as to whether this indicates any lack of robustness.”
In the areas of disagreement in this Joint Statement, each expert provided the following. Mr Coyne stated (inter alia):
“For the purposes of addressing the robustness of Horizon, I have applied the following definition of robustness:
‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling.’ ”
In consideration of the likelihood of Horizon to be the cause of shortfalls in branches, Horizon is not determined to be robust in this regard because:
it contained high levels of bugs, errors and defects as set out under Issue 1 above which created discrepancies in the branch accounts of Subpostmasters;
it suffered failures of internal mechanisms which were intended to ensure integrity of data;
the system did not enable such discrepancies to be detected, accurately identified and/or recorded either reliably, consistently or at all;
the system did not reliably identify ‘Mis-keying’, which is inevitable in any system with user input, and did not reliably have in place functionality to restrict users from progressing a mis-key;
it required numerous processes and workarounds to be in place to allow Fujitsu to modify data already recorded by Horizon, which would not be required in a “robust” system; and/or
there were weaknesses and risks of errors and other sources of unreliability within Horizon.”
(italics present in original)
Dr Worden stated in the same Joint Statement:
“The definition of 'robust' proposed above by Mr Coyne is not adequate, for reasons given below. The term 'robust' is not, as implied in para 3.1 of the outline, either illdefined or a piece of IT public relations. Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.
In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?
In my report I shall survey the evidence I have found that Fujitsu paid sufficient attention to the dimensions of robustness, and that they did so successfully. I shall also address evidence from Mr Coyne implying that Horizon fell short of its robustness objectives.
In my current preliminary opinion, Horizon is a highly robust system, and this has important implications for the other Horizon issues, notably issue 1.”
It can be seen therefore that Dr Worden in the Joint Statement did not agree Mr Coyne’s definition, and expressly said it was not adequate. In any event, the meaning of any word – even “robust”, or “robustness” – ought to be capable of description by the parties themselves. Although on its face it did not appear that Dr Worden agreed with Mr Coyne’s definition, a footnote in the Post Office first set of post-hearing submissions suggested that Dr Worden was not disagreeing with the first part of Mr Coyne’s text, in other words that part of the text that contained his definition of robustness (which was in italics in the 1st Joint Statement). Obviously if the parties (or their experts) could agree the definition to be applied so far as the Horizon System is concerned, that ought to be identified. I therefore asked the Post Office via email whether it agreed with the definition adopted by Mr Coyne, and if not, what its alternative definition was.
This led to a further document being received from the Post Office dated 18 July 2019. It referred to the passage in the 1st Joint Statement (which is quoted at [40] above) as “the agreed definition”. That rather overlooks that Mr Coyne identified the definition of robustness which he was applying, and Dr Worden expressly disagreed with this in the same Joint Statement under the heading “Areas of Disagreement”, and stated “the definition of ‘robust’ proposed above by Mr Coyne is not adequate, for the reasons given below”. It also overlooks that in the 3rd Joint Statement, paragraph 3.1 had an agreed entry which stated the following:
“Irrespective of how you define the detail of robustness, in line with most other largescale computer systems, Horizon's robustness has generally improved.
From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events.
This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.”
(emphasis added)
Later in the same document of 18 July 2019 the submission was made by the Post Office that “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. It was also submitted that “As Post Office understands it, this is what Mr Coyne meant when in his comments in [the 1st Joint Statement] he defined robustness as “the ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.
This was precisely the definition which Dr Worden, in his areas of disagreement on the 1st Joint Statement, described as “inadequate”. The end position therefore is as follows.
The claimants found “robustness” difficult to define in the abstract and tied it in with the other wording of Horizon Issue 3; a robust system would be “extremely unlikely to be the cause of shortfalls in branches”. That however is a consequence of how a robust system would operate, not a definition of what robustness means.
The Post Office defined it as follows: “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. The Post Office was also prepared to accept Mr Coyne’s italicised definition in the 1st Joint Statement, namely ‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.
Mr Coyne applied the definition he set out in italics in the 1st Joint Statement, quoted in the immediately preceding paragraph of this judgment and at [41] above.
Dr Worden’s definition was as follows:
“Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.
In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?”
The Post Office also submitted that Mr Coyne’s definition was not “materially different” to that of Dr Worden.
The Post Office made submissions in paragraph 3(b) of the written submissions dated 18 July 2019 on robustness that stated that Mr Coyne cannot have intended to exclude the effect of countermeasures when he considered the concept of robustness. I shall return to this topic when dealing with countermeasures. This is because some of the countermeasures considered by Dr Worden are not parts of the Horizon System at all, such as SPMs noticing adverse entries in their branch accounts, and the manual issuing of Transaction Corrections (TCs) by the Post Office (which both parties agree are outside of the Horizon System).
I do however accept the Post Office’s submissions that there is not a great or material difference in the definitions of robustness adopted by the parties’ experts. I do not accept the claimants’ submission that robustness is difficult to define. Dr Worden defined robustness by using what he termed as “the dimensions of robustness”. It is rather circular to describe the meaning of robustness as being “robust against” some particular risk. Although Mr Coyne provided his definition in the 1st Joint Statement, the statement by Dr Worden that this was “inadequate” may only have been aimed at the entirety of Mr Coyne’s entry in the areas of disagreement, as effectively accepted by the Post Office in their most recent written submissions on the subject. Whether that is an explanation of the lack of agreement in the Joint Statement, I also agree with the Post Office that Mr Coyne’s definition is not materially different to that used by Dr Worden.
Robustness is indeed an engineering concept. It means the ability of any system to withstand or overcome adverse conditions. A robust system is strong and effective in all or most conditions. The robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences; this is the same meaning as how robustness was described in the Post Office’s written submissions dated 18 July 19. Robustness does not mean perfection.
The exercise necessary above, to arrive at the definition of robustness in [54] above, is not judicial pedantry. Given the central importance of robustness to the disputes about the Horizon System, and the Horizon Issues, it is in my judgment essential. It is mildly surprising, given how central the assertion of robustness has been to the Post Office’s defence of the Horizon System, that Dr Worden’s interpretation of the term has been relied upon so heavily by the Post Office, given the term was used by the Post Office for some years prior to his involvement.
However, regardless of that passing observation, I find that both experts correctly understood what robustness in fact means, and applied the definition at [54] above in considering their expert evidence. I will return to the expert evidence in some detail later in the judgment, including in the Technical Appendix.
C. Features of this Group Litigation
There are many different ways of managing group litigation. The subject matter of such litigation is different from group to group, and what is appropriate in one set of proceedings will not necessarily be the best approach in another set. Group litigation may involve many claimants – even tens of thousands, in some cases – all with what is essentially the same type of claim, governed by a number of common issues that apply across all or most of the cases. Some group litigation may involve similar numbers of claimants, all with similar claims that are factually different. In this case, there are 584 claimants, which in the context of some group litigation, is not very many, although it is still in the many hundreds, and the period of time over which the relevant events are said to have taken place is about 16 or 17 years. There are different aspects to each individual case, but to deal with the litigation efficiently, cost-effectively and proportionately it is simply not feasible for the same judge to try all the claims, one after the other, in full on all of their respective merits. Such a process would take several years. It is not what group litigation is intended to achieve.
The concept of group litigation is that the Managing Judge, whoever he or she may be, with the assistance of the parties insofar as that may be available, selects the most suitable mechanism for that particular set of proceedings in order to achieve compliance with the overriding objective. That will – or should – lead to cost and time saving. That does however require a high degree of co-operation from the litigants. Here, my intention of holding a substantive trial each judicial term onwards to resolve as much of this group litigation as possible, as quickly as practicable, became simply unachievable as a result of the issuing of the recusal application by the Post Office, explained at [5] above. A further trial in the autumn of 2019 also became undesirable as a result of the parties wishing to have time to consider mediation, and to find out whether the Court of Appeal would grant the Post Office permission to appeal on the contractual issues in Judgment (No.3). Over the numerous hearings and two full substantive trials that I have conducted, I have gained the distinct impression that the Post Office is less committed to speedy resolution of the entire group litigation than are the claimants, but it is not possible to state with finality whether that is correct.
All litigation is important to those involved in it. In this group litigation, all the claimants are of the view summarised at [10] in Judgment (No.3) in terms of the behaviour of the Post Office. Recitation that the claimants are of that view does not mean that any similar views are held by the court, or that any decisions have yet been made by the court on that point one way or the other. The total sums claimed by way of liquidated sums (excluding heads of general damages not yet quantified) are approximately £18.7 million. The Post Office counterclaims for certain sums against the claimants and also alleges fraud. Reputations are plainly at stake on both sides. That is the case in many types of proceedings, particularly high-profile ones that attract press interest. However, that sum of money is not large, in the context of large scale and protracted litigation. The parties’ joint costs are approximately now £27 million, and because of the notification provisions in one of the earlier Case Management Orders I made, regular notifications of the total sum of costs are made to the court. The Post Office alone spent over one million pounds in little more than a month earlier this year; the notification letter of 13 May 2019 from its solicitors stated that its costs were in excess of £12,800,000; by the time of its letter of 25 June 2019, the Post Office’s costs were in excess of £13,900,000. Both this level, and rate, of expenditure is very high, even by the standards of commercial litigation between very high value blue chip companies.
I have already explained that the Post Office has now used four different leading counsel, and it also engaged a second commercial firm of solicitors to act for it on its appeal. The claimants are funded by means of litigation funding, explained in Judgment No.3 so far as it is relevant. The claimants are ordinary individuals who ran branch Post Offices, and the Post Office is publicly funded, so none of the parties are high value FTSE 100 companies. Not all of the Post Office’s costs will be recoverable costs regardless of the outcome, not least due to the existence of two Costs Management Orders which are in smaller amounts than the Post Office’s rapidly growing grand total, but the total figures still represent real expenditure of actual money.
These proceedings also have the additional feature of criminal convictions on the part of some SPMs. As explained at [10] in Judgment (No.3), there is a Criminal Cases Review Commission (“CCRC”) review underway in respect of the convictions of
some claimants. I have been told that these are subject to a stay pending judgment upon the Horizon Issues. I have explained this before in open court, but matters such as criminal convictions are no part of this group litigation. This court has no jurisdiction over such matters.
Although criminal convictions are no part of these proceedings, examples of alleged software bugs, errors and/or defects affecting branch accounts, which led to certain adverse consequences for SPMs, did form part of the subject matter of the Horizon Issues trial. Given the impact (actual or potential) upon branch accounts, with the potentially adverse implications for SPMs generally in the background, and given the way that the SPM witnesses in this trial were cross-examined (as with the Common Issues trial, some being accused expressly of criminal offences), criminal convictions were not part of the trial, but were part of the background. In the claimants’ oral closing submissions, Mr Green QC for the claimants identified a high level chronology of what he called “doubling up”, by which he meant bugs, errors and/or defects that led to certain entries in branch accounts being “doubled” incorrectly. He did so by specific reference to allegations by a particular SPM who had been convicted of a criminal offence.
Mr de Garr Robinson QC for the Post Office objected to this, complaining of what he said was jury advocacy. In the course of what became an increasingly vigorous exchange between counsel at the very end of the trial he stated, by reference to what he said were the rules of commercial litigation, that “one of those rules is that one doesn't say things incautiously that might have an impact on evaluations being done in another place in relation to different proceedings.”
Prior to considering the “rule” to which he was referring – it is not one of the Civil Procedure Rules, so far as I am aware – it was therefore necessary to identify exactly what “different proceedings” were being referred to in this disagreement between the parties. There are five elements to this matter.
Firstly, it had never been raised by any party at any stage during this group litigation that there were any criminal prosecutions currently underway such that it was necessary to consider any reporting restrictions for the group litigation. I expressly raised this with the Post Office on the occasion referred to at [62] and [63] above, who confirmed there were no such prosecutions underway. The “different proceedings” to which Mr de Garr Robinson referred were the proceedings already before the CCRC.
Secondly, the possibility of future (as opposed to current) criminal prosecutions, or the potentially criminal impact upon individual SPMs, did more than hover in the background to the Horizon Issues trial. Some claimants who gave evidence in this trial were expressly accused by the Post Office of criminal offences in crossexamination in this trial, something which had also occurred in the Common Issues trial. It should be clearly understood that any findings I make in respect of any witnesses do not determine with finality any issues to be tried in any particular trial to follow, including the trial(s) of their individual claims.
Thirdly, the rule to which leading counsel for the Post Office referred was not identified. Journalists must be alive to the risk of serious prejudice when criminal proceedings are live, as a result of the Contempt of Court Act 1981, and are thereby restricted in reporting matters generally in that respect. That statutory provision replaced what had been referred to as the sub judice rule, but this did not relate to statements made in evidence in open court. The principle of open justice is a most important one and there were no restrictions imposed on any reporting of the Horizon Issues trial (nor the Common Issues trial) at all, nor were any sought by the claimants or by the Post Office.
Fourthly, there was no specific restrictions imposed on the parties by the court in terms of the evidence they could adduce and what could be said by way of submissions (written or oral), due to potential impact upon any “different proceedings”. I have not heard any argument on the matter, but I do not consider it would be proper to do so in any event. It was also not argued before me that the CCRC proceedings, which are subject in any event to a stay pending at least part of the outcome of the group litigation, are in any way of such a character that reporting of the Horizon Issues trial ought to be restricted in any way, or such that they could be prejudiced by the group litigation. Indeed, the presence of a stay of those proceedings pending the outcome of some of the group litigation issues would suggest directly to the contrary. The objection therefore by the Post Office to the way in which the claimants sought to make closing submissions on the actual impact upon different SPMs of what were said to be bugs, errors and defects in the Horizon System was, so far as it could be understood, not a well-founded one.
Finally, the claimants by their counsel were trying to draw parallels between what had happened to one particular SPM regarding “doubling up”, and numerous entries in internal Fujitsu and Post Office documents prior to that of similar “doubling” occurrences being caused by bugs, errors or defects. Those parallels are obvious on the face of the documents. The degree to which the CCRC find such parallels of assistance, if at all, is a matter entirely for them. The Post Office submitted most strongly that these different references were not to one single bug, but were references to a number of different issues or bugs that had been experienced in Horizon over the years. Whether that makes the claimants’ point for them or not, it is important to remind all the parties that the issues in this litigation are not going to be decided with sympathy, or lack of it, coming into account in the analysis in any respect whatsoever.
Mr Green for the claimants submitted that the Post Office was “trying to distract from the stinging nature of the underlying documents by objecting” to the submissions and that the Post Office may be finding it “uncomfortable” to have its own documents, in a particular date range, highlighted by reference to particular criminal cases, prosecutions mounted by the Post Office, and the experiences of SPMs in those cases. Certainly the Post Office and Fujitsu have been exposed to a degree of scrutiny in this litigation which does not appear to have occurred before; however, there has not been litigation of this type on these issues before.
The claimants and the Post Office doubtless know this already – certainly their legal representatives will - but given the high profile nature of this dispute and the fact that this judgment may be read by those other than the parties themselves, I will make the position very clear:
The Senior Courts of England and Wales (who acquired this name by reason of the Constitutional Reform Act 2005) comprises the Court of Appeal, the High Court and the Crown Court.
The Court of Appeal has two divisions, Criminal and Civil. Appeals from the High Court are to the Civil Division. Appeals from the Crown Court are to the Court of Appeal Criminal Division, also known as the CACD.
The High Court is the highest court of first instance in civil cases.
The Crown Court is the highest court of first instance in criminal cases. Some criminal matters are dealt with by the High Court (by way of Divisional Court) but these are narrow in scope and do not arise in any respect concerning the group litigation. All appeals against both convictions and sentence from the Crown Court are dealt with by the CACD.
The Criminal Cases Review Commission, or CCRC, is a statutory body responsible for investigating alleged miscarriages of justice in England, Wales and Northern Ireland. It was established by section 8 of the Criminal Appeal Act 1995. It is an independent non-departmental government body, and has the power to refer cases to the CACD.
This group litigation is concerned only with the issues arising in the civil claims being brought against the Post Office by the claimants, and the Post Office’s counterclaims. It will result in a series of judgments on those issues which are public. What, if anything, the CCRC do in any respect following any of the judgments is entirely a matter for the CCRC and forms absolutely no part whatsoever of the group litigation.
This court has no jurisdiction in respect of any of the convictions of those SPMs who were successfully prosecuted by the Post Office. Although the presence of criminal convictions does have evidential effect in respect of individual claims by individual claimants who have been convicted of false accounting, these have not arisen in either of the two substantive trials held to date (Common Issues and Horizon Issues) nor will they arise in either of the next two (the principles governing Heads of Loss, and then some individual claims).
There has been no restriction imposed on any party, or any witness, in this group litigation by the court in terms of the evidence that could be adduced, or submissions that could be made, with one exception. Mr Henderson of Second Sight was subject to a restriction upon his evidence by reason of the terms of an agreement that he had entered into with the Post Office when Second Sight were engaged in what was originally intended to be consensual resolution of the claims. This was called the Second Sight Mediation Scheme. This feature of the evidence of Mr Henderson is a matter which is dealt with in more detail in Part D of this judgment. It was a restriction imposed upon him, or agreed, by the parties, and the court was not involved in that restriction in any way.
Turning to a different matter, I have already made certain criticisms of the Post Office in Judgment (No.3) in terms of how it had conducted itself in the litigation, and I had
also made criticisms of some of its senior witnesses. One of those witnesses, Ms Van Den Bogerd, was also called as a witness in the Horizon Issues trial. So far as I am concerned, she came to the court for the Horizon Issues trial with a clean slate in terms of whether her evidence on the Horizon Issues would be accepted or not. A different way of putting the same point is that simply because certain adverse findings had been made concerning the evidence she had given in the Common Issues trial, this did not mean that I adopted any particular starting position so far as her factual evidence was concerned for this trial. The court was entirely neutral in terms of starting position. Although it was certainly not a point that went to her credit that I had already made adverse findings of the accuracy of her evidence in the Common Issues trial, that did not mean that I started with any fixed view of the likely accuracy of her evidence in the Horizon Issues trial. Her evidence in the Horizon Issues trial is dealt with at [203] below.
Further, before turning to the detail of each side’s evidence, an approach was adopted by the Post Office on occasion of seeking to adduce what was (or should have been) in reality evidence of fact, but by way of submission, or points made “on instruction”. Sometimes, depending upon the nature of the subject matter, such an approach is understandable or unavoidable, and may be unobjectionable. It is not therefore sensible to state that this should never be done in any conceivable circumstance in any trial. However, on important points that have been dealt with by a particular witness in their evidence of fact, it is not a suitable device to adopt. This was particularly done in terms of the cross-examination of Mr Coyne concerning evidence already given by Mr Godeseth in his cross-examination, about alteration by Fujitsu of a particular branch account. I deal with that in detail at [376] to [379] below.
This was also done in Appendix 2 of the Post Office’s Closing Submissions, where (sometimes detailed) factual explanations were given in respect of bugs in the Bug Table. That appendix was compiled by different teams of solicitors and counsel, something explained by the Post Office when, some months after it was submitted, they discovered that three pages were missing and sought permission to serve them rather late. I granted permission for them to be added, as they had been prepared before the deadline for service and omitted due to an administrative oversight.
Submissions should not contain evidence, or positive evidential assertions, that are not present in the evidence served in the trial. This is a fundamental point. I provide some examples in the Technical Appendix by reference to specific entries for specific bugs. Blurring (or ignoring) the lines between submission and evidence is entirely unhelpful. Evidence is something that comes from a witness (lay or expert) and which the opposing side is entitled to test by way of cross-examination. It is not appropriate for detailed factual assertions to be made in closing submissions that are not directly referable to evidence in the case. There is no way such factual assertions can be tested; if they come in closing submissions, there is no way that the opposing party can deal with those assertions in their own evidence, or even put relevant points to witnesses for the other party in cross-examination.
Further, this is not a case that is being tried in a Specialist List, such as the Technology and Construction Court – it is a general Queen’s Bench Division case – but it readily could have been tried in such a list. It contains a great deal of technical subject matter, particularly in this trial dealing with the Horizon Issues. The two IT
experts have each given evidence in other computer litigation before. Such subject matter, and such expert evidence, is readily suited to analysis by the parties and precision, which is the usual approach of courts generally. In my judgment, this is particularly important in technical matters such as these. Bluster, unfounded assertion in cross-examination, detailed technical explanations of fact given “on instruction” and submission are rarely helpful.
Finally, there has been a vast amount of highly detailed material deployed by both sides, not simply evidence of fact, and expert evidence, but also reference to a great many documents. The use of an electronic bundle has made this easier to manage than otherwise, but written submissions alone were in excess of 1100 pages in total from both sides. It is neither possible nor desirable to recite in this judgment, or in the Technical Appendix, or resolve, every single disputed item, or every single disputed fact, no matter how minor. I only make findings in this judgment that are necessary to enable me to resolve the Horizon Issues themselves. Simply because I do not specifically refer to a particular submission or piece of evidence, it should not be thought that I have not had regard to it. I have considered all the material, evidence (both factual and expert), submissions, and passages in contemporaneous documents, multiple times. The experts’ agreements in particular have been of great assistance, but everything has been considered. This judgment, together with the Technical Appendix, will be of substantial length, and to recite everything would very probably make it of unmanageable proportions. It is in any event far longer than is ideal, but that is partly explained by the fact it deals with the life of a complex computer system that spans some 18 years.
In adopting this approach, I have borne very much in mind the overriding objective in the Civil Procedure Rules, the need for proportionality, but also the obvious need to provide a reasoned judgment. I have also taken specific account of the dicta of Males
LJ in Simetra Global Assets Ltd and another v Ikon Finance Ltd and another
[2019] EWCA Civ 1413. That case concerned foreign exchange trading, and claims for dishonest assistance and damages for deceit and conspiracy against a total of 12 different defendants, both personal and corporate. Although it concerns those matters, plainly very different to the Horizon Issues, it does state generally what ought to be included in a judgment. The requirement for a judge to give adequate reasons in a judgment is analysed at [37] to [46], with the expression “the building blocks of the reasoned judicial process” (used by Henry LJ in Glicksman v Redbridge Healthcare NHS Trust [2001] EWCA Civ 1097) approved at [42] by Males LJ. Four points are summarised at [46] of Simetra:
“Without attempting to be comprehensive or prescriptive, not least because it has been said many times that what is required will depend on the nature of the case and that no universal template is possible, I would make four points which appear from the authorities and which are particularly relevant in this case. First, succinctness is as desirable in a judgment as it is in counsel's submissions, but short judgments must be careful judgments. Second, it is not necessary to deal expressly with every point, but a judge must say enough to show that care has been taken and that the evidence as a
whole has been properly considered. Which points need to be dealt with and which can be omitted itself requires an exercise of judgment. Third, the best way to demonstrate the exercise of the necessary care is to make use of "the building blocks of the reasoned judicial process" by identifying the issues which need to be decided, marshalling (however briefly and without needing to recite every point) the evidence which bears on those issues, and giving reasons why the principally relevant evidence is either accepted or rejected as unreliable. Fourth, and in particular, fairness requires that a judge should deal with apparently compelling evidence, where it exists, which is contrary to the conclusion which he proposes to reach and explain why he does not accept it.”
75. I am acutely conscious that the first of those points, succinctness, is not likely to be achieved in this judgment. This is not only due to the nature of the subject matter, the Horizon Issues, but also the fact that both Legacy Horizon and Horizon Online are involved, and the system was brought in some time ago, namely the year 2000. Due to the nature of the claims brought (and the limitation issues certain to arise), the functionality of the Horizon system(s), robustness, and the other Horizon Issues had to be dealt with at an early stage in the group litigation. I will return to the other three of Males LJ’s four points after my review of the evidence, as well as his comments on the importance of contemporaneous documents at [48] and [49] of Simetra. I do this at [937] and following below.
D. Evidence of Fact: The Claimants
The claimants originally served statements from a greater number of witnesses of fact than were in fact called at the trial. This was because, just before the Pre-Trial Review on 14 February 2019, the Post Office objected to the evidence of two particular individuals. Very shortly before the Common Issues trial, the Post Office had issued an application to strike out a significant number of passages in the six Lead Claimants’ witness statements, and that application had been dismissed in Judgment (No.2) “Strike Out”. On this occasion, although no application to strike out was issued, the Post Office sought a ruling at the PTR in respect of the evidence of Mr McLachlan and Mr Henderson.
Mr McLachlan had been called as an expert witness in the criminal trial of Mrs Seema Misra, a SPM at West Byfleet in Surrey who was charged both with theft from her branch, and also false accounting. He was called for the defence. The sums in question were approximately £74,000. Mrs Misra pleaded not guilty, and her defence was that the Horizon system was to blame. She was convicted by a jury after a trial in late 2010 at Guildford Crown Court, and was sentenced to a term of imprisonment of 15 months. She was pregnant at the time of her conviction and imprisonment. Expert evidence was called by the Post Office at her trial from Mr Gareth Jenkins from Fujitsu. Mr Jenkins was not called as a witness by the Post Office in the Horizon Issues trial before me, but a large amount of the evidence from the Fujitsu witnesses was attributed to information directly given to them by Mr Jenkins. This was a controversial matter between the parties at the Horizon trial. The Post Office did not proffer an explanation for Mr Jenkins’ non-appearance as a witness; they were not obliged to do so. However, in their closing submissions and after all the evidence had been called by both parties, they did so. The explanation provided was by way of submission and not evidence. This is a matter with which I deal with further at [509] to [513] below. Prior to the PTR, the Post Office objected to Mr McLachlan’s evidence on the basis that it contained opinion evidence and the claimants did not have permission for it, which is something required under the CPR. This was resolved by the claimants deciding not to call Mr McLachlan, and explaining at the PTR itself
that they would not call him. On the face of it, the claimants were wise to concede this point as if his statement had included opinion evidence, the Post Office would have had a valid point of objection. I have not therefore considered the evidence of Mr McLachlan which forms no part of the case.
The other witness whose statement was subject to objection was Mr Henderson. Mr Henderson is a Director of a company called Second Sight Support Services Limited (“Second Sight”). Second Sight was appointed by the Post Office to conduct a review into problems with the Horizon system in July 2012. A number of reports were produced by Second Sight, and the Post Office responded to these. These reports were as follows:
Interim Report dated 8 July 2013;
Briefing Report Part One dated 25 July 2014;
Briefing Report Part Two version 1 dated 21 August 2014;
Briefing Report Part Two version 2 dated 9 April 2015.
The Post Office submitted at the PTR that I should give a ruling concerning those parts of Mr Henderson’s witness statement upon which the Post Office was required to cross-examine, and those parts upon which it was not. This was said to be justified by a concern the Post Office had, namely that if it were required to cross-examine upon the correctness or accuracy of the contents of the Second Sight reports themselves, the Horizon Issues evidence would not be capable of being completed within the time estimate for the trial, which at that stage was 16 days. Mr de Garr Robinson also expressed concern that any failure by the Post Office specifically to cross-examine upon any particular point in any of the accompanying Second Sight material, rather than the actual witness statement of Mr Henderson, would lead to the claimants submitting that such a point was not challenged, and that any such point would therefore be taken by the court as being formally agreed by the Post Office. I declined to give a ruling directing the Post Office upon those parts of Mr Henderson’s statement which should be subject to cross-examination. This was for three reasons.
Firstly, the answers to the different Horizon Issues would not be determined based on the accuracy or correctness of the contents of the Second Sight reports themselves. The court was to hear detailed expert evidence from two IT experts whose evidence would go directly to the Horizon Issues. Whether the Second Sight reports were, or were not, correct in their conclusions (which were generally critical of Horizon) would not form part of that process, and would not even qualify as a sideshow. Certainly their contents would not be determinative of the Horizon Issues in the group litigation. I have had no regard to the contents of the different Second Sight reports, nor to the Second Sight conclusions, in arriving at the answers to the Horizon Issues. I am aware that all of the Second Sight conclusions were challenged by the Post Office.
Secondly, it is not for the court in civil litigation to identify in advance to any litigant that it need not cross-examine upon particular evidence of fact of the opposing party. This is particularly so in a time-limited trial such as this one, but in my judgment is a general point of principle. The court was effectively being asked to direct the Post Office as to how it should conduct its case, and also consequentially, about how much of its time at the trial should be used for particular witnesses. This is not the function of the court. I was aware that Mr Henderson’s evidence was challenged by the Post
Office. It is for this reason that he was to be cross-examined. How the Post Office chose to do that, the trial time its advisers chose to allocate to that exercise, and which (if any) parts of the statement were to be challenged and how, were all matters for the Post Office to decide.
Thirdly, it would in any event be wholly unconventional in a time-limited trial dealing with the subject matter of this group litigation for the court to give any weight to any submissions by the claimants that particular points of criticism or detail in such detailed documentation as the Second Sight reports were essentially “accepted” by the Post Office because they had not been specifically challenged in cross-examination. There is never sufficient time, in any time limited trial, to cross-examine upon everything. A time limited trial in the 21st century is not conducted in the same manner as a trial would have been many years ago, particularly in a detailed technical dispute such as this one. There was no forensic route available to the claimants whereby a failure to cross-examine upon the Second Sight conclusions or reports, whether due to the length of the trial or otherwise, would or could lead to a conclusion by the court that some aspect of the Second Sight reports was agreed or admitted by the Post Office, when it plainly was not. The period in question in the group litigation spans some 18 years. If attention is to be paid to other court users, the overriding objective and the Civil Procedure Rules, it is simply not possible (nor desirable) for any trial judge to permit the parties to have unlimited time to cross examine upon everything. The quid pro quo of that is a party cannot be expected to cross examine upon everything. There should have been no concern on the part of the Post Office that the Second Sight report(s) would be taken as not challenged by the Post Office unless their contents were subject to cross examination. Given there appeared to be such a concern, I explained the position at the PTR. The claimants also expressly said at the PTR that no such point would be taken. That latter element was unnecessary, as whether the point was taken by the claimants or not, it would not have been accepted.
The position was therefore clarified or explained at the PTR and the Post Office, notwithstanding that no formal ruling was made on those specific parts of Mr Henderson’s statement that it had to challenge in cross-examination, appeared to be content. A transcript of this hearing is available. The contents of the Second Sight reports were not cross-examined upon when Mr Henderson gave evidence, but in my judgment there was no need for them to be. As it happened, Mr Henderson was not cross-examined for very long, but that does not matter. I have not taken any account of the contents of the Second Sight reports in deciding the Horizon Issues.
The claimants therefore called the following witnesses of fact. For the reasons that I explain below in relation to Mr Roll for the claimants, and Mr Godeseth for the Post Office, I consider these two witnesses in particular to have been of the greatest assistance in resolving the Horizon Issues. However, all of the factual witnesses contributed to my understanding of the system over the years, both generally and specifically. The specific experiences of the claimant witnesses who gave evidence about what occurred in their branches are, if I accept them, specific examples of the working of the Horizon system at the branch Post Office end in practice. I will deal with each of the witnesses in turn.
Mr Latif
Mr Adrees Latif was the SPM at Caddington Post Office in Caddington, Bedfordshire, since 2001 until late September 2018. His appointment with the Post Office therefore ended after the litigation commenced, and he was subjected to an audit in September 2018. He was cross-examined by video link from Islamabad, as it had been necessary for him to travel to Pakistan as a result of a family bereavement. He left the UK on 19 February 2019, and he travelled from Kashmir to Islamabad to give his evidence by video.
He gave evidence about two specific incidents. One occurred in July 2015 and related to the transfer of £2,000 from the AA stock unit to the stock unit designated SP1. Originally his statement had said it was the SJ1 stock unit but this was a typographical error which he readily accepted. He successfully transferred the £2,000 from AA, but when he went to the SP1 unit, the same sum had not transferred into that unit successfully. There was no explanation for this that he could come up with, including having checked his own CCTV, and he was sure he had carried out the transaction correctly – it was not an unusual transaction. He described the sum of £2,000 as having “simply disappeared from Horizon” and explained that this would lead to a shortfall in the branch account for that sum.
The second incident occurred in January 2018 in relation to Camelot and scratch cards. His heading for this he corrected to “Transaction Acknowledgement Issue”. He explained that certain information was sent by Camelot to Horizon twice, and the Post Office sent out a notice stating that due to this mistake by Camelot, a Transaction Correction or TC would be issued. The correction occurred and he accepted it, but the stock figure for scratch cards in his branch on Horizon remained unchanged. This therefore showed his branch as having £1,000 more in scratch cards than was actually present in the branch. This issue was still outstanding as at the date of his amended statement which was 1 March 2019, and remained outstanding as at the date of his cross-examination which was 12 March 2019. This is, obviously, a period well in excess of one year.
The Post Office’s case on the £2,000 was put methodically to Mr Latif. That regarding the stock transfer is best summed up in the evidence of Ms Van Den Bogerd, who stated that provided certain steps or action were carried out correctly, what Mr Latif had said happened simply would not occur. She said, inter alia “providing these two actions are completed, the stock unit from where the cash is transferred should not show a discrepancy" (emphasis added). As she put it, her “strong belief is that Mr Latif has recalled these events incorrectly”. She also said that “the records that Post Office has reviewed do not support what Mr Latif has said and I believe that he may have mis-recollected events from 3 years ago”. The Post Office’s case was essentially that Mr Latif had not done the steps correctly, because had he done so, what he said happened could not have happened.
Mr Latif’s response to that was both consistent, considered and credible, and was best summarised in one of his answers:
“A. I'm experienced -- I have been running a post office for 17 years, sir. I have also worked for the Post Office on training other offices how to run a post office. I was also involved in running and introducing the new Horizon software changes in 2006 onwards, where I went to several offices on behalf of the Post Office to give them training. So I'm an experienced, trained subpostmaster and I ran my business successfully for 17 years. So I may have been a bit brief in the statement but obviously I can run through those -- exactly those steps that we would take to make sure that there is no operator error on our behalf.”
The conflicting evidence on this particular point is a good illustration of the “poles apart” position to Horizon by the Post Office and the claimant SPMs in this litigation. Because of the Post Office’s position on Horizon, almost all and any of the criticisms or accounts of factual events which the claimants made, or make, about how this system worked in practice are attributed to fault or carelessness by the SPM or their assistants. Indeed, without fault or carelessness by an SPM, the Post Office simply cannot explain these occurrences. The Post Office’s position is therefore to challenge the factual account – which it is entitled to do – because if the factual account by an SPM is accepted as truthful and accurate, then the Post Office would have to accept that there must be a fault or faults within Horizon. Therefore, the Post Office cannot accept that the factual account is truthful and/or accurate. Thus the dispute goes around and around in endless circles. This litigation is aimed at breaking that deadlock.
It is also the case that Mr Latif had the following point positively put to him about why he had checked the CCTV that he had within his branch. “So you now say you looked at the CCTV because your colleagues were concerned that you hadn't done the transaction properly?” even though Mr Latif had said no such thing. Mr Latif was subject to a fairly robust attack, not only on his account, how it matched up with other records which the Post Office said contradicted it, what he and his assistants had or had not been doing, and indeed upon the full scope of his evidence and his credibility – as shown by the question I have reproduced. That question was framed as though even his own colleagues had concerns about what he had done. It was positively put to him by the Post Office that he had not even complained to the Post Office, although he provided the name of his Area Manager Mr Navjot Jando and said he had complained to him many times. The Post Office did not call Mr Jando to rebut this. One exchange will suffice as an example of the type of attack upon Mr Latif:
“Q. You don't say anywhere in your witness statement that the £2,000 physical cash also somehow disappeared, but that seems to be what you are now saying, is that right?
Well, the system gave a shortfall of £2,000 and that's been my statement all the way through, sir, so I don't know what you're trying to confuse me, but there's a shortfall of £2,000 in stock unit AA and there should not be a stock shortfall. The money is physically there.”
Mr Latif’s evidence had never been, so far as his witness statement and evidence orally before the court, that £2,000 in cash had physically disappeared. His evidence was that there was a shortfall of that amount shown in Horizon as a result of what he had done. There were some aspects of the cross-examination of Mr Latif which were simply unhelpful. Firstly, extensive spreadsheets were put to Mr Latif which he had not seen before. They are plainly not in chronological order, and had in any event been what was called “filtered” by the Post Office legal team; they were not agreed by the claimants, nor was the “filtering” process explained at any point. There is a limitation on the degree of assistance to be obtained by such an exercise. Further,
there was no agreed exercise by the experts whereby the two of them had gone through all the records directly relating to this specific instance and agreed what the records did, or did not, show. The Post Office’s case amounted, literally, to a pure challenge of fact that what Mr Latif said had occurred, simply did not occur in fact.
“Q. On the basis of that, Mr Latif, Post Office says there was no failed transfer such as that described in your witness statement and that you are simply wrong about that, it never happened.
So you are calling me a liar?
Q. Mr Latif, you may be mistaken or you may be lying. I put the question that it didn't happen.
A. Well, I state that they did.”
So far as the other issue experienced by Mr Latif in January 2018 was concerned, the Post Office accepted, as Ms Van Den Bogerd’s original written evidence (prior to later correction) put it, that its data showed “that the branch received two TAs (meaning Transaction Acknowledgement) on 18 January 2018. However, due to an error by Post Office, instead of increasing the scratch-card stock, the TAs decreased the stock. To be clear, this was a data entry error by Post Office and not an issue with Horizon. Horizon processed the TAs accurately. I note that the TAs were accepted by the branch, which could have been challenged at that point if the user had noticed that the TAs were not for a positive number, as they should have been.”
(emphasis added)
However, this is notable for the following reasons. The first was that the blame is shifted back to Mr Latif for not spotting the mistake and challenging the TA. His evidence was that it was not possible to challenge a TA. They simply have to be accepted. This was accepted by Ms Van Den Bogerd when she came to give evidence orally. Therefore the notable point made by Ms Van Den Bogerd, a director of the Post Office, in her written evidence, to shift the blame back onto Mr Latif, is simply wrong in fact. TAs cannot be challenged, they have to be accepted.
The second is that the Post Office evidence entirely omits any reference to the “memo view” – a type of communication used by the Post Office on Horizon which leads to a “pop up” on the terminal the next time each user logs on. There was simply no reference to this at all by Ms Van Den Bogerd, and Mr Latif said that this was sent out by the Post Office “to everyone” saying the error on the Lottery would be corrected by means of the issuing of a TC. Finally, Ms Van Den Bogerd corrected this passage in her evidence in chief by way of printed corrections prepared before she was called. She said that these corrections had been handed to her solicitors prior to the trial. If that is correct, it is surprising that Mr Latif was cross-examined on the basis of the original evidence in her statement, as that was not going to be her evidence in chief. However, regardless of that, she readily agreed that there is no choice regarding the acceptance of TAs – they simply have to be accepted by the SPM. In those circumstances, there is no explanation as to how the emphasised sentence in [93] above came to be in her statement at all, and it was plainly incorrect. Also, the call logs (about which Ms Van Den Bogerd was cross-examined) concerning this incident entirely support Mr Latif’s evidence.
Finally, although this is a minor point compared to the ones in the preceding paragraph, the tenor of her witness evidence that Horizon processed the TAs in this respect gave the impression that there were no problems with the Lottery and TAs so far as Horizon were concerned.
Not only did the steps taken by the Post Office – the TAs – not correct the issue that occurred in Mr Latif’s branch, but in cross-examination Mr Latif explained further:
“A. Can I just confirm, there was an audit done in September of this year, an audit by a Post Office trained auditor, and my stockholding was still showing negative. And a Jane Lawrence is the auditor and she has – still could not resolve this matter, so the problem hasn't gone away, the problem is still there. And there have been a number of calls to the helpline to resolve that negative stock and it hasn't worked. They haven't come back with a response.”
Mr Latif was very clear about this:
“A. I still state that an independent audit was done by the Post Office, a Mrs Jane Lawrence, in September 2018 the stockholding was still negative and as the branch was handed over to another subpostmaster that is going to be investigated, it's going to be investigated. If you are now coming back with this evidence, I still say that there is a problem somewhere and I don't know what's happened but we have still got a negative stock figure within our branch.”
He returned to this again, in the light of being pressed yet further on the point that it was human error at the branch that had caused the discrepancy or shortfall.
“So the fact remains there was an independent audit happened by your Post Office -- by Post Office's auditor, her recommendation was that they look into it as is (inaudible) the strategy is still there. And there were a number of calls to the helpline pleading with them to resolve this issue before the audit and there will be a complete trail of that, sir.”
He also referred to the what had happened as being “a glitch”.
I accept the evidence of Mr Latif, who struck me as a reliable and careful person, and who had personally been the one who had tried to perform the transfer from one stock unit to another. He had personally experienced what he explained to the court in this respect. I accept his direct evidence on this in preference to that of the Post Office, which effectively was from people who were not there, who maintained, more or less, that it simply could not have happened, and who had nothing to substantiate or corroborate the challenge made to Mr Latif’s primary evidence. I find as a fact that it did happen as Mr Latif explained. I find that Mr Latif performed the required steps correctly in respect of the stock transfer between units, as one would expect of someone who had 17 years of experience, and was sufficiently skilled at his role such that the Post Office had, prior to the litigation, been sufficiently satisfied of his competence that he was used by the Post Office as a trainer for training other SPMs. I also accept his evidence, which is direct primary evidence of the state of the accounts at this branch, that neither he nor the auditor for the Post Office whom he named, have resolved the issue concerning the Lottery, the incorrect TAs and the effect of the
TC, which remains an unexplained shortfall or discrepancy. Ms Van Den Bogerd’s evidence is a number of steps removed from the branch, and is little more sophisticated than assertions that there must have been other matters to blame, alternatively reliance upon records which did not, due either to their contents or to the way that they were deployed in cross-examination of Mr Latif, demonstrate the points that the Post Office maintained they demonstrated. I make further findings in respect of Ms Van Den Bogerd’s evidence in the section below dealing with the Defendant’s Evidence in Part E. Mr Latif’s branch was subject to an audit, and his appointment as a SPM ended then or shortly afterwards.
In due course Mr Latif will have his own individual claim, and the Post Office with have its individual counterclaim against him, tried. The only findings I am making in respect of his evidence are those necessary for me to resolve the Horizon Issues. All other issues remain to be tried in those later proceedings. Those later proceedings may explore in some detail not only the two specific matters in respect of which he gave evidence in the Horizon Issues trial, but any others which are relevant to both claim and counterclaim. They will do so by reference to other documents, as explained in the section of this judgment Part K, Audit Data. The degree to which the findings of fact that I make affect my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Tank
Mr Jayesh Tank was the SPM at Fleckney Post Office, in Fleckney Leicestershire, from 4 May 2006 to 15 March 2017 when it was closed as part of the Network Transformation Programme. He gave evidence regarding certain issues with Horizon that he had experienced, which included the effects of a complete power failure to the building, which occurred mid-transaction. This occurred whilst a customer was making a withdrawal in the branch from her Post Office card account. On his evidence, this led to a shortfall in his accounts of about £600. Mr Tank paid this sum to the Post Office, and the letter from the Post Office stating how it came to be owed described it as a “branch discrepancy”; it was in fact £660. He paid it by way of a deduction from his remuneration.
The second issue was one concerning mail labels, when Horizon would (as he put it) “jump ahead” to the end of the transaction and no label (which is what would or should be affixed to the packet to be posted) would be printed. This would cause a loss in the branch accounts.
Ms Van Den Bogerd again gave evidence about these specific instances. Her witness statement led Mr Tank to correct the date when he remembered the power failure occurring (he accepted it was 2014, not earlier in 2010 or 2011 as he had initially thought), and he corrected this in a supplemental statement. Her statement had helped him pin down the date. This incident became the subject of a PEAK at Fujitsu, although Mr Tank did not know this at the time. This supplementary statement also exhibited to it old “posts” he had made contemporaneously on an internet forum on the website Yahoo in respect of another loss, £195, which he had suffered in 2011.
Mr Tank had been interviewed by the Post Office investigators in 2015 and in the
Horizon Issues trial it became clear that certain matters extremely critical of his conduct were going to be put to him in cross-examination. He was therefore given the warning against self-incrimination under the Civil Evidence Act in the same way that the two claimants in the Common Issues trial, who were accused in cross-examination of criminal offences, were given it. Mr Tank answered all the questions that were put to him. Mr Tank had, eventually, used the icon on Horizon in his branch for “Official Postage” incorrectly, and his explanation for this was that this was deliberate and was done in order to get the attention of the Post Office. He said at his 2015 interview that all his attempts by way of phoning the helpline and contact with his area manager had simply not resolved the numerous complaints he had made. He frankly accepted that he had not sent the post by official means which he had entered on the Horizon system as official postage, doing what he did by way of protest. It was put to him in cross-examination that he had “helped himself” to official postage, which on the evidence available was a suggestion that did not seem to have any basis in fact, if he had been doing what he said he was doing. There was certainly no evidence before the court to substantiate this assertion.
In his interview on 5 November 2015 by the Post Office investigator Mr Bridges, it had been accepted by Mr Bridges (who interviewed him) that Mr Tank had reversed the entries for official postage in any event. This acceptance at the time by the Post Office investigator was directly contrary to the way that the points were put to Mr Tank in his cross-examination, which were that he had “helped himself” to postage and “improved his financial position” by acting as he did. Detailed findings on this will have to wait for the full trial of his claim and counterclaim, but it is notable, in my judgment, that the attack on the credibility of this witness in the Horizon Issues trial, and positive allegations to him of criminality, were not consistent with the contemporaneous record of the acceptance by Mr Bridges for the Post Office in the investigation interview in 2015. Mr Bridge clearly accepted that the entries had been reversed. Indeed, Mr Bridges introduced this subject in that interview as follows:
“Okay thank you. Let's move on to the postage claims for the moment. So in my letter I gave details of the claims and the reversals that you completed since 25 August. I think my first question would be that I know that you reversed them but why undertake them in the first place bearing in mind these are, in effect you are stating the transactions which took, well say the transactions have taken place which you have used official postage for which in effect did not take place.”
The questions in cross-examination to Mr Tank, which must have been put on instruction, did not take account of what the Post Office had accepted at that time in the interview. Another feature of the interview is that Mr Tank was told by the interviewer “there are no issues with Horizon”. That exchange is as follows:
“KB (Mr Bridges) I can confirm that there are no issues with Horizon and as my letter stated there is no issue with the production of labels.
Jay (Mr Tank) Then obviously you conducted your investigation into my concerns without asking me for my evidence. I've got the evidence here to show you. I'm happy for you to take copies but I'm going to hang on to it but how can you say something is fully investigated if you've not looked at all of the evidence.”
On the face of it, that is a valid point that Mr Tank made. Had there been an investigation, his evidence would plainly be something that ought to have been taken into account by any investigator.
The Post Office relied, again, upon evidence from Ms Van Den Bogerd. Mr Tank was cross-examined by reference to this and also by reference to a technical explanation concerning the fact, it was said in cross-examination, that the power outage must have occurred at a particular point in time. The cross-examination was as follows:
“MR HENDERSON: So what appears to have happened was a transaction from the Post Office card account was in the middle of being processed and so it was in the stack presumably.
Yes.
Q. But had not yet been posted to Horizon. So you hadn't cashed out on that transaction, you hadn't completed everything to do with that transaction?
A. On the stack – Q. It's on the stack.
A. It's on the stack, but the stack has a balance of zero, so to clear the stack you just press "enter" and it goes straight –
Q. But you hadn't got to the point of clearing the stack?
A. I'm not sure.
Q. Okay. My suggestion is that there was probably an outage at just the point where the money had been taken from Post Office card account but had not been processed onto Horizon. That's my suggestion to you. A. Okay.
Mr Justice Fraser: Well, is the witness going to be in a position to agree or disagree?
Mr Henderson: Well, he might be if he recalled.
Mr Justice Fraser: Do you recall that happening when there were outages? A. No. I cannot recall.
Mr Justice Fraser: Were you aware of when outages would occur like that? A. Not all the time.
Mr Justice Fraser: Do you want to put the question again?
Mr Henderson: Yes. What I'm suggesting is that the cause of this problem was that an outage occurred at a particular point in time. A. Yes.
Q. You were in the process of effecting a transaction from POCA?
A. Yes.
Q. It was in the stack and it had cleared from POCA?
A. Actually you mentioned the word "outage". I'm not -- was there a power outage?
Q. I'm not sure if it was a power outage, but I think it may have been a problem with the system.
A. Ah, okay.
Q. The system went down in some way.
A. So -- yes, because you said that if there's a power outage then there's evidence when you have to log back in, so did that happen on this occasion?
Q. Okay, I want to come to all this and I'm doing this clumsily. What I'm suggesting is that what may have happened -- and if you don't recall, you don't recall, but what may have happened is that the transaction was in the stack, the money had been taken from the Post Office card account and before you cleared the stack there was an outage . A. Possibly.
Q. Okay.”
(emphasis added)
I have reproduced this passage because it demonstrates the following. The Post Office put a possible explanation, based on technical grounds, to the witness who did not have either at the time, or at the trial, the technical expertise, or the relevant technical records, or the data, to enable him to agree or disagree. Even that explanation itself accepted “there may have been a problem with the system” and “the system went down in some way”. The witness’ answer, which in my judgment is the only sensible answer that could sensibly be given in these circumstances, was “possibly”.
This type of evidence obtained on cross-examination is of no assistance in resolving the Horizon Issues, other than an implicit acceptance by the Post Office (on this occasion) that outages could, potentially, in technically terms lead to what Mr Tank experienced. In any event, the Post Office’s explanation accepted that “there may have been a problem with the system” which is equally consistent with the claimants’ case, as it is with the Post Office’s case, if not more consistent with the claimants’ case than with that of the Post Office. Power outages do happen, and there is a process to be followed. Given Mr Tank was using Horizon Online, he was then in his crossexamination taken to Version 5 of a HOL quick reference guide. The version to which he was taken was not, however, the version available to him at the time, as that was a single A4 double-sided version in his branch, and the one put to him at the trial was longer than two pages and available online. The date attributed in the electronic trial bundle to Version 5, the version put to him in cross-examination, is 30 July 2015. His forum posts and his supplementary witness statement made it clear that the incident occurred in September 2014, a later date admittedly than given in his first statement, but still somewhat earlier than July 2015. This was however the version of the guide that Ms Van Den Bogerd relied upon in paragraph 78 of her 2nd witness statement to challenge Mr Tank’s account. I find the version that he would have had available at the time was not Version 5, and was the single A4 version that he had available to him in his branch in September 2014. However, and regardless of this, Mr Tank accepted that recovery receipts would or should be generated and set out the procedure. He also accepted that giving all the receipts to the customer was not the proper procedure.
In the PEAK dealing with the shortfall of his accounts of £195, which is PEAK PC0214226 and is headed “Failed Recovery Transaction(s)” Fujitsu recorded:
“Date:14-Dec-2011 10:43:53 User: Wayne Bragg
Summary:
The banking transaction had completed (A3 rec'd and authorised @ 13:33:37), including the receipt print (13:33:42), and money should have changed hands.
The basket settlement failed from 13:35 with 'No response received from data centre', and the two retries also failed, and the attempt CANCELLED at 13:37.
The Disconnected Session receipts show "Cash TO CUSTOMER 195.04" so the customer's account should be correct but the branch will have a shortage (for a withdrawal) because the session hasn't been recorded.”
(emphasis added)
In my judgment this entry clearly supports Mr Tank’s evidence about how Horizon operated on the occasion to which he referred. The heading of the PEAK was “failed recovery transactions”. The PEAK clearly states that the customer’s account would be correct – they were given the cash of £195.04 as they expected, but “the branch will have a shortage (for a withdrawal)” as the session was not recorded. A transaction correction was issued. There were related PEAKs, and also there was an entry “KEL acha959T may be relevant” on the PEAK, and Anne Chambers was referred to. The PEAK also recorded “Sending to SSC for investigation.” That KEL was raised by Anne Chambers on 28 February 2010, and the title was “HNGX banking reconciliation – state 4”. This is an important KEL, and I deal with its further detail in the Technical Appendix.
There was a dispute between Mr Tank and counsel for the Post Office about whether this was an example of “Horizon working as it is supposed to”, which Mr Tank did not accept, and also whether he would have been refunded the sums (which he accepted had occurred) had he not phoned in and reported the problem. It was put to him that he would have been refunded even had he not called in to report, although how this would have occurred given the way PEAKs are initiated was not explained to him. In any event, the PEAK demonstrated that what Mr Tank had said occurred, had indeed occurred. Transaction Corrections are issued outside the scope of the Horizon system.
The position regarding the procedure for spoiled labels was, again, a situation whereby the Post Office’s explanation was this simply could not have happened. Ms Van Den Bogerd’s evidence was that there was a procedure available that Mr Tank should have used, yet the document put to Mr Tank that was said to support this stated expressly in terms that “the label could only be spoiled if the label was on hand” and Mr Tank’s point was that no label was “on hand”, given the problem he had was that the label did not print at all. Using the procedure in the document therefore would not be possible, and indeed would be contrary to the Post Office instructions that it could only be used if the label was on hand. These instructions also required the SPM to write “spoilt” on the label and keep it with the Horizon receipt for two years. This could not be done if the label had never printed. The explanation by Ms Van Den Bogerd, put to Mr Tank, was also directly contrary to the contents of a letter dated 7 September 2015 from the Post Office to Mr Tank which stated:
“Your enquiry has been investigated and I can confirm there are no issues with Horizon online or the production of postage labels which would cause the situation you described. However if a user either pressed the yes or return key quickly before screen messages appear this can lead to a user confirming a postage print has happened when in rare circumstances it may have failed and put the cost of the failed print into the basket. In this circumstance Subpostmasters should contact Network Business Support Centre to arrange a credit for the spoiled postage.”
This means that the Post Office in that letter accepted that in some circumstances, depending upon how quickly a key was pressed, the printing of a postage label could fail and cause an impact to branch accounts. The SPM must make a telephone call to the helpline, to correct the fact that the branch account would show a charge for the postage even though no label had been printed and hence could not be provided to the customer. This letter suggests not that Mr Tank had not followed a particular procedure for dealing with spoiled (rather than non-existent labels) but that the system would, depending upon how quickly a user pressed a particular key, not print a label, even though the cost of the failed print would go in the basket. By going “in the basket” this means the branch accounts would include the cost of that label as a debit to those accounts. In other words, this letter supported Mr Tank’s direct evidence.
The Post Office in cross-examination put the following to Mr Tank:
“ Q: There were procedures built into Horizon to cater for the situation that you explained -- I have to say in the vaguest of terms, but as I understand what you are saying, there were procedures in place which ensured you could deal with the situation, weren't there? A. No.
Q. We will have to differ.”
I would express this rather differently. If the speed of pressing a key could lead to the cost of a printed label being added to the basket (and hence branch accounts) even though that label had not printed, it is difficult to see that there is a “procedure built into Horizon” as defined to cater for this. On the contrary, it appears from the Post Office’s own letter in 2015 that there was no such procedure. For the avoidance of doubt, I find that there was no such procedure built into Horizon. The evidence demonstrates to the contrary. Phoning the helpline and asking for a refund is not “a procedure built into Horizon”.
Further, it is difficult to see how the explanation proffered by Ms Van Den Bogerd regarding how to deal with a spoiled label could apply to Mr Tank for two reasons. Firstly, I accept that as a matter of common sense interpretation and language, a label cannot be “on hand” if it was never printed. Mr Tank did not consider he could properly use the procedure in the guide for the problems he experienced, and I agree with him. Secondly, the Post Office’s own explanation to him in 2015 was very different to what he was told in cross-examination in 2019 that he should have been doing. Even were I to assume (in the Post Office’s favour) that there must have been instructions from the Post Office somewhere regarding how a SPM should behave when a label failed entirely to print, as print failures (which would not produce a label “on hand”) were plainly not unheard of, it remains to be seen what those instructions were, and whether those instructions conflicted with what he was told in the letter in 2015. No such instructions have been produced by the Post Office in any event.
I consider Mr Tank to have been a credible witness. I find that Mr Tank’s experience in branch was not an example of Horizon working “as it was supposed to”. I find that the creation both of a PEAK, and reference to a KEL within that PEAK, to be consistent with my conclusion. The proposition that Horizon worked as it was supposed to is an obviously flawed one. Indeed, that proposition is also inconsistent with common sense. PEAKs and KELs are not created for situations where Horizon is working “as it is supposed to”. They are created to deal with errors. The acronym KEL is for Known Error Log. Nor are matters referred to the SSC for investigation when Horizon is working correctly. Whether what occurred afterwards was an example of Horizon working correctly depends on the categorisation of Transaction Corrections and whether they are part of the Horizon system or not. I explain later in this judgment that they are not, therefore the issuing of a TC cannot be Horizon working as it should. Mr Tank also gave evidence, which I accept – and this is made out in the contemporaneous documents – that he had some difficulty in advancing this matter through the correct channels. The helpline operator with whom he was dealing initially refused to “pass up” the matter to a more senior person. Whether this was an isolated instance of unhelpfulness in this single case, or a more generally obstructive approach across the helpline, will have to wait for future trials in this litigation, and I recite it for completeness only in respect of Mr Tank’s experience. It does not form part of my consideration of the Horizon Issues.
Mr Tank was accused of criminal offences and it was said that he had “helped himself” to official postage and had “taken official postage”. These accusations were not put to him at his interview in 2015 and indeed the text of that interview shows that the Post Office interviewer expressly accepted that he had reversed the transactions, which is not consistent with Mr Tank “helping himself”. There are 10 pages of detailed closing submissions by the Post Office about Mr Tank’s evidence, and although he is criticised for being vague, confused, imprecise and for not having prepared his evidence with care, the allegation of criminality is not raised. However, the point was positively put to him in open court, and Mr Tank is entitled to a finding on it. In my judgment, it would be quite wrong to leave such an accusation hanging in the air, unresolved in this way. I find that on the material before the court in this trial and on the evidence available and put to him, these serious accusations are not made out. That is not to say I have made binding findings on the full details of his claim which remains to be tried in due course. The Post Office submitted, both in respect of Mr Tank and other of the claimants’ witnesses, that they confined their evidence to what was described as “a small sub-set of his complaints, presumably because it was felt that this threw light on the Horizon issues – and Post Office’s evidence was similarly confined”.
However, there are two important points that must be borne in mind about that submission, both as it is made in respect of Mr Tank, and indeed the other witnesses called for the claimants. This has not been a trial of their individual claims. There may be other issues about Mr Tank’s branch accounts, raised both by him and by the Post Office, at the full trial of his claim and the Post Office’s counterclaim. That does not mean that this specific evidence, about specific incidents experienced on Horizon, should be discounted. Indeed, this specific evidence is not only directly relevant to the Horizon Issues, but is, in my judgment, very important evidence going to those issues. The second point is the parties’ agreed wording of the Horizon Issues does not allow for any wider issues going to Mr Tank’s branch accounts to be considered in this trial. All the other issues between Mr Tank and the Post Office remain to be tried in the later proceedings. The only point of difference between them on one of his two issues concerning Horizon was whether what occurred when a transaction correction was issued could properly be described as “Horizon working as it is supposed to”, and certainly the KEL referred to in the PEAK appeared during the Horizon Issues trial more than once. The creation of a PEAK, and the KEL to which reference was made
within it, in my judgment demonstrates that this is not Horizon working as it should, as does the issue of a TC. The wider consideration of these matters that are dealt with both by the experts and later in this judgment. I reject the suggestion that Horizon was working as it was supposed to on this occasion. So far as Mr Tank’s experience with the £195 shortfall is concerned, there was a Horizon Online failure; three identical receipts were printed, which should not have happened; the receipts all showed a disconnected session; and there was a loss in the branch accounts that evening for the amount paid out to the customer. This was even though Mr Tank had settled with the customer for the amount specified on the receipt. There was no record of the transaction at all on the transaction log that was produced for the period of the Horizon Online failure. None of these, in my judgment, are examples of “Horizon working as it was supposed to”.
The operation of the helpline is not part of the Horizon Issues, so it is not necessary to consider and make findings on what Mr Tank said the helpline told him, which he explained in his posts was “that the loss is mine unless I can sort out with customer directly.”
The later proceedings concerning Mr Tank’s claims (and any counterclaim) will probably hear further evidence which are relevant to both claim and counterclaim. They will do so by reference to other documents, as explained in the section of this judgment Part K, Audit Data. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Anup Patny
Mr Anup Patny became the SPM of Spencefield Post Office in Leicester in October 2014. He is currently suspended by the Post Office, having been suspended on 17 August 2016. His son, Mr Aakash Patny, also gave evidence. I shall refer to Mr Anup Patny as Mr Patny Senior for this reason. I shall also deal with his evidence, and that of his son, together for reasons that will become apparent.
There was what he described as a “major system outage” on 9 May 2016 in his branch, which was a single counter branch run in conjunction with his retail business in the same premises. This was just a few months before he was suspended. This outage led to the closure of his branch and he believed that this had affected the whole network. At the end of the next trading period, which was on 11 May 2016, there was a shortfall of almost £17,000, predominantly made up of a shortage of 16,000 coins of £1 denomination. This was a very large number and he said that he knew he would never have such a large amount of £1 coins in his branch. He was also the person in the branch who had dealt with the cash delivery that took place on the relevant morning, which is called “remming in” the cash. This means accepting cash into the branch from the cash delivery that Post Office makes to the branch, and entering the cash received into Horizon. His son, who also gave evidence, followed this matter up with the Post Office. On 19 May 2016 the Post Office contacted him about a discrepancy in respect of stamps, which so far as he was concerned related to the £16,000 regarding coins.
The Post Office accepted that there had been an outage but disputed that this either did, or could have, resulted in the discrepancy which Mr Patny said occurred at his
branch. Ms Van Den Bogerd gave evidence about this, and the Post Office’s position is best summarised in the explanation put to Mr Patny Senior in his crossexamination:
“Q. Well, what I'm suggesting is that the most natural explanation for this, whatever adjustments were made, is that at some point on 11 May someone hadn't counted a big pile of £10 notes. They had been put in a safe and forgotten about, which is understandable, and that they were found the next day, or located, and there was an accurate cash declaration on 12 May. A. I don't know about that, sir.
Q. It's perfectly plausible, isn't it?
I can't say anything to that.”
This suggestion by the Post Office is, in my judgment, somewhat fanciful. A “big pile of £10 notes” is not the sort of thing that ordinarily gets “put in a safe and forgotten about”, and then suddenly “found”. It was also put to the witness that “things were pretty chaotic in your branch when it comes to these sorts of things” and “the cash declarations look like they are all over the place”. Mr Patny Senior did not agree with this characterisation of the way his branch was run. He certainly did not agree with the “big pile of £10 notes” theory the Post Office put forward. His evidence was also that it was his son who had contacted the helpline, and who had made the adjustments to Horizon that he was advised to do by the helpline. Initially the Post Office sought to challenge this account through cross-examination of Mr Patny Senior only. Counsel for the Post Office did not initially intend to put questions on this to his son, even though his son was plainly the correct witness to answer the detailed questions, as he was the person who said he had done certain things, and he was about to be called as a witness. The reluctance of the Post Office to put questions to the person who was plainly the relevant person to ask about this was said to be because “the records are the records” and the Post Office did not intend to repeat cross-examination that had already been done. I was not prepared to permit this. I required questions about what Mr Patny had, or had not, done, to be put to him directly, and not have the crossexamination conducted by proxy through his father. Mr Patny Senior’s son had given a witness statement in respect of these matters, and was plainly the correct witness to whom the questions should be put. He was the next witness listed on the trial timetable, and was going to be called next for the claimants. The notion that questions challenging what he had done, or not done, should more properly be directed to his father is contrary to how cross-examination should be conducted.
Mr Aakash Patny
Mr Aakash Patny is the son of Mr Patny Senior, and worked in the branch and retail premises with him, as did Mrs Patny, the wife of Mr Patny Senior and mother of Mr Patny. He is not a claimant. He was given the warning against self-incrimination under the Civil Evidence Act. He did however answer all the questions put to him.
Mr Patny assisted his father, particularly with balancing. He would usually arrive at lunch time, and so was not present when the national outage occurred on the morning of 9 May 2016, which had been resolved by the time he arrived that date. He gave evidence concerning the same amount of money, which both he and his father considered a shortfall, and which the Post Office had claimed was due to it and was a
branch discrepancy for which the branch was responsible. It had been settled centrally but had not been paid to the Post Office.
So far as 11 May 2016 was concerned, when Mr Aakash Patny had arrived at the branch he became aware of the shortfall of over £17,000, which was a “complete surprise” to him and which he considered could not be explained. He had contacted the helpline, and had followed the steps he was advised to take, which he thought had resolved the issue. These steps were things he did on the Horizon terminal, and sequences of entries he was talked through over the phone. The documents available supported his evidence that he had made such a call, although some of the entries did not match what he said he had told them. There was no record of what he was told to do, nor was there any record put to him showing what key strokes had been inputted at the branch.
The Post Office’s case on the shortfall was that the outage on 9 May 2016 did not lead to any discrepancy. It was not put to Mr Patny that he had not made key strokes as advised over the telephone, or that he made different keystrokes to those advised. All that was put, somewhat unclearly, was that “the problem” was resolved, cash had been physically “found” that had been missed at the time of the previous cash declaration, and also that a figure showed on a cash declaration of “plus £17,000 odd” had the effect of “cancelling out” the previous discrepancy and showed that “there was no longer a cash problem”. Mr Patny denied any money had been found that had been lost; denied that if a discrepancy had been cancelled out the figure would have been plus £17,000, rather than zero; and said that although he thought following the key strokes he was told to input had resolved the issue, it had not. He was told to “readjust” the cash stock figure, something he had never done before, and relied upon the instructions he was given by the helpline. The Post Office’s cross-examination was done by reference to the cash management report, not the audit data. I will return to audit data later in this judgment in Part K, but the audit data would have showed what keys were pressed, in what sequence and when. Such data would have been of great assistance, but was not produced by the Post Office.
On 19 May 2016 the branch was contacted by the Post Office, in respect of an overdeclaration of stamps. The amount of this was approximately the same value as the amount of the discrepancy of cash had been a few days earlier. He had done the declaration for stamps. A person called Debra Lambley phoned the branch. He was struck by the co-incidence in the amount of the stamp discrepancy, and told Ms Lambley the stamp declaration was impossible due to the number of stamps this represented. To assist a reader of this judgment on that latter point, but something that was not explored in the evidence, to obtain a broad idea of how many stamps that would represent, the following points are relevant. On 28 March 2016 the Royal Mail announced an increase to take effect that week in the price of stamps. 1st class stamps became 64p each, and 2nd class stamps became 55p each. This shortfall would therefore have represented 25,000 1st class stamps, or over 29,000 2nd class stamps. It goes without saying that this is a great deal of stamps.
He was instructed to re-adjust the stamp stock figure, followed her instructions as to how to do this, and the system then showed the £16,000 shortage. It was by now past 7.00pm, he phoned the helpline, they contradicted Ms Lambley and he then had to re-
declare the following day, being unable to do so that day as it was after the 7.00pm cut off.
The Post Office’s case on this, as it was put to Mr Patny, was firstly that he had made a mistake in declaring the stamps. He denied this. The circularity of the Post Office’s case can be seen from a lengthy passage of evidence, which I will not reproduce, from page 26 on that day’s transcript to page 35. Mr Patny explained, basically, that the helpline gave instructions to resolve the cash discrepancy; this resulted in the figure for stamps being wrong by the same amount; Ms Lambley was not prepared to have this, and following her instructions this resolved the stamps situation, but the cash discrepancy reappeared. Then the sequence happened a second time. Mr Patny was prepared to accept he may have made a mistake once, but denied he could have made exactly the same mistake the same way a second time.
It was also then positively put to him that one way of disguising a shortfall in cash would be to over-declare the number of stamps. A positive allegation of dishonesty was put to him, which he rejected.
The premise behind this suggestion of dishonesty was that it was said by the Post Office that if the stamps figure went up, the cash figure would automatically go down by the same amount. He did not agree with this. It was put as a positive statement in the following terms:
“Q. I suggest finally, Mr Patny, it is rather surprising, given that you do all the balancing, or did all the balancing in the branch, that you don't understand that the effect of increasing the declaration of stamps is to have a corresponding effect on the declaration of cash?
I wasn't aware of that, no.”
There was no direct evidence to support the proposition that by increasing a declaration of stamps, it would have a corresponding effect in the opposite direction upon the declaration of cash held in the branch. In any case, in the Common Issues trial, there was evidence that making a cash declaration required specifying how many notes of each denomination was physically held in the branch. An SPM had to declare both the cash and the stock that was held in the branch. It is difficult to see how changing the number of stamps held in a branch would, or could, affect the physical counting and declaration of the amount of cash held in a branch by way of specific denominations. However, and in any event, the allegation of dishonesty against Mr Patny was a line of cross-examination that was not supported by evidence from Ms Van Den Bogerd, who was expressly asked about it, and who accepted that she had not made an allegation of dishonesty in her witness statement and said she did not offer an opinion on whether Mr Patny was dishonest or not. That means that there was no evidence from any witness called for the Post Office to support the allegation of dishonesty.
The second problem in relation to which Mr Patny gave evidence was a MoneyGram transaction in February 2016. This related to a failed payment by card by a customer, who tried to send £3,100 by MoneyGram but whose card was declined twice. However, the branch accounts still showed two debits in that amount in the branch accounts that led to a loss showing at the branch of £6,200. A TC was issued but only for one of these, which led to a £3,100 loss. Mr Patny accepted that he had only cancelled the transaction at the time, and not reversed it as well (which was required) until later, in the evening, when advised to do so by the helpline. As he put it, the amount had “doubled up”, in that the MoneyGram transaction, which was cancelled and reversed later on (which was for £3,100) had led to a loss showing of £6,200. If that is correct, then it is indeed double the value of the transaction. Ms Van Den Bogerd said that this was “not supported by the data” and also relied upon the fact that the cash shortfall on the day was greater than £6,200, namely about £500 greater than the cash shortfall which Mr Patny said had been caused by MoneyGram. She could not proffer an alternative explanation, however, and in her witness statement said:
“On 23 February, the branch declared cash holdings of £25,803.87, a net downward movement of £8,601.59. The net value of transactions during this period resulted in a £1,806.71 decrease in cash. The Moneygram transaction described above would account for a further £3,100, bringing the total explainable cash movement to a £4,906.71 decrease. However, this leaves £3,694.88 of cash movement unaccounted for. I cannot say for certain what caused this additional loss of cash but there is nothing in the accounts that suggest a problem with Horizon. It appears more likely to me to be a problem with cash handling in the branch or a user error when making cash declarations.”
(emphasis added)
This statement does not answer the evidence of Mr Patny on the point, and the more that counsel for the Post Office tried to press the point(s) in cross-examination, the more stark it became that this evidence by the Post Office was no answer. Firstly, it assumes that the MoneyGram transaction was only responsible for £3,100, not the figure of £6,200 which Mr Patny stated. It assumes that £3,694.88 does not include the lower figure of £3,100, which in my judgment, on the documents deployed, it plainly does. Secondly, Ms Van Den Bogerd cannot say what caused the additional loss of cash. Thirdly, it states that that there is “nothing in the accounts that suggest a problem with Horizon” (emphasis added). That is rather stating the obvious. If all the problems complained of by the claimants with Horizon could be seen in the branch accounts, this litigation would not be occurring. Fourthly, she retreats to criticism of the way cash is handled in the branch, or other mistakes – “user error”. It is a perfect summary of the Post Office’s case, but it does not provide an answer. Other than vague “user error”, there is no explanation provided. I find that there is nothing in any of the documents to suggest user error.
Additionally, Mr Patny explained that he had fully explained all of this to the area manager Mr Irwin, who agreed with him that the MoneyGram transaction had appeared to “double up”. Further disclosure by the Post Office when the trial of Mr Patny Senior’s claim comes to be tried, together with evidence from Mr Irwin if he is called, will provide further information. Finally, the exercise undertaken by the Post Office in cross-examination failed to take account of the fact that one had to consider both the cash figure, and the shortfall, from day to day to make sure that one is comparing apples with apples. I consider the exercise undertaken in crossexamination to be flawed.
I cannot resolve this purely on the factual material put before the court by the Post Office. The audit data showing the actual key strokes inputted will be required in order to do that, together with findings on the expert evidence and my findings on the existence of bugs, defects and errors within Horizon. However, I do find that Mr Patny came across as a careful person, and as a credible witness, and I accept that he took the steps he did as explained in his evidence. He accepted that he had not reversed the transaction until later, rather than immediately as he ought to have done. The resolution of the movement of the cash position, and what if any explanation is available, simply cannot be done with finality at this stage on the basis of the factual evidence before the court alone. The Post Office had, according to Mr Patny, failed to provide the Credence data which would make it clear what had happened and how. The helpline logs show that Mr Patny had chased this on numerous occasions, and also the entries suggested that a Credence report was being prepared. The audit data showing what key strokes he had undertaken and when, was not produced by the Post Office.
There was no evidence before the court to support the allegation of dishonesty that was expressly made against him, and it was not supported by Ms Van Den Bogerd. On the material before the court in this trial, I find the allegation of dishonesty against him not to be made out and I reject it. That is not to say that the subsequent trial dealing with Mr Patny Senior’s claims will automatically accept all the evidence given by his son, if he calls him as a witness. All future issues in individual claims will be dealt with, and resolved, on the evidence led by the parties in those subsequent trials. I am only deciding matters in this judgment that are necessary to resolve the Horizon Issues. However, in circumstances where positive allegations of criminality are expressly put to a witness in a trial such as this, it is only fair to deal with those allegations. They cannot as a matter of fairness be left hanging in mid-air, particularly where there is no evidence provided in support from the party making the allegation in any event.
The later proceedings concerning Mr Patny Senior’s claims will explore in some detail not only the specific matters in respect of which he and his son gave evidence in the Horizon Issues trial, but any others which are relevant to both his claim and counterclaim. They will do so by reference to the evidence advanced in that later trial and the other documents, as explained in the section of this judgment Part K, Audit Data. The degree to which the evidence of Mr Patny Senior and his son affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mrs Burke
Mrs Angela Burke had worked for the Post Office in various roles since she was 16. Most recently, she worked in the Newport Post Office in Brough where her husband was the SPM. She was a Branch Assistant and had worked as an assistant for about 15 years, and had also previously been a SPM herself. She and her husband had an associated greeting cards and stationery business, and the branch was closed in October 2017 as part of Network Transformation. The card business has also closed.
Her evidence related to events of 9 May 2016 when the national outage occurred, the same date as the occasion about which both the Mr Patnys had given evidence. She gave evidence about the impact upon the branch business, the way she was serving customers and how Horizon was being very slow that day, with a sand timer appearing on the screen for a very long time. She served one customer, who was making a cash withdrawal, as she obtained the relevant messages and approvals on the screen. However, after they had left, a receipt printed saying “Recovery failed” and the withdrawal of £150 was not shown. She then later studied the transaction log and this latter transaction did not appear.
She realised what was happening and decided to close her branch that morning. She sat down and worked through all the transactions in her branch, using what was available to her. She was given an explanation by the helpline which she did not accept, as she knew and could remember that the payment to the customer had been authorised, which is what led to her handing over that sum. She also was engaged in a contemporaneous exercise whilst it was very fresh in her memory. She said there was no alert from Horizon, and “there was no means through the Horizon system for the discrepancy to be identified or for its cause to be established in my situation”. The transcript of this call was available – she and her husband had obtained it using a subject access request from the Post Office.
Mrs Burke then went to extraordinary lengths. She also proved herself very tenacious, as many people may well have simply given up on the sum of £150. She identified the customer, and she tracked him down. She went to his house and explained what had occurred. He happened still to have the receipt from the transaction at her Post Office. It entirely matches her account. She went with the customer to the customer’s bank, which was the TSB in Goole. She explained with the customer to the bank cashier what had happened, and the cashier printed out the bank statement and showed that the sum had been withdrawn from the customer’s bank account. The customer permitted Mrs Burke to have this.
She and her husband then pursued this through the helpline and their call was escalated. They did receive a TC for £150, but actually that referred to an amount in respect of Lloyds Bank and not to TSB. This was pointed out to the Post Office who claimed there was no code on Horizon for the TSB. If that is correct, I find it highly surprising that there should be no separate code for the TSB. The bank currently known as Lloyds used to be known as Lloyds TSB – there are cases on bailii that use that name in relation to other litigation, for example – but the TSB was split from Lloyds TSB, which then became simply Lloyds Bank. This split occurred some years before the events of which Mrs Burke gave evidence. TSB and Lloyds Bank had been separate for some time by May 2016.
Mrs Burke’s evidence was explored in very considerable detail in cross-examination, including the fact that she chose to keep the stack open in order to serve more than one customer at once, how she had served them and the other details of what had occurred. She was challenged on what was said to be a “relatively small” point, of whether she would have received a TC from the Post Office without having done all the work that she had done, in tracking down the customer and going to his bank. I reject that categorisation of the point by the Post Office. It is not a small point. It is, in the context of the overall litigation, a major point. It is difficult to see, given the evidence collected by Mrs Burke which was provided to the Post Office, how they could have failed to issue her with a TC. Whether this TC would have issued absent the work that she did to demonstrate the fault and its effect is not a “small point”.
There are really two points that are relevant. The first is a broader one: is the process for issuing TCs part of the Horizon system? The second is a case specific one: would, on the balance of probabilities, the Post Office have issued a TC in her case without the evidence that Mrs Burke herself collected and provided to the Post Office?
So far as the first point is concerned, the process for issuing TCs is not part of the Horizon system. That is addressed elsewhere in the judgment. So far as the second point is concerned, the documents put to Mrs Burke demonstrate that Fujitsu had worked out what had occurred; that the internal document attached to an internal email of 12 May 2016 did not accept that Newport branch had in fact paid the money to the customer, even though Mrs Burke had told them this on 9 May; and that by 16 May 2016 the decision had been taken that the branch would receive a TC anyway. Her response to the “small point” was “possibly, yes”.
The conclusion that I draw from this is that the Post Office required something more from Mrs Burke than her word that the money had been paid out by her branch to a customer – the internal documents refer to “not knowing” if the money had been paid, even though she had already told the Post Office this. I find that the fact that this is a Fujitsu document rather than a Post Office document does not matter for this purpose. That something more that was required, in her case, was the evidence she had personally compiled that was of great weight, and was accepted by the Post Office, as the TC was issued. Whether evidence of lesser weight would have been accepted by the Post Office is not possible to say, and is entirely hypothetical. Fujitsu had, though, identified this particular problem experienced at the Newport branch by the national outage. The information of what occurred, and the problem, was not visible to the SPM or their assistants in the branch.
Counsel for the Post Office’s “relatively small point” put to Mrs Burke, that the Post Office would have issued her with a TC anyway, is in fact not entirely consistent with the Post Office’s own evidence for this trial. Mrs Van Den Bogerd stated in paragraph 110 of her witness statement that “Once Post Office was presented with evidence that the customer had received the cash and the customer's bank had recorded the withdrawal, a transaction correction was issued to bring the branch.” She makes it clear, in my judgment, that it was the work that Mrs Burke did personally that had led to the issuing of the TC. She also makes it clear that she believes that what happened occurred as a result of what Mrs Burke did:
“Following Mrs Burke's investigation, Post Office generated a transaction correction for the £150 withdrawal that had not been recovered at the time. If Mrs Burke had not done this herself, there is a process built into Horizon for flagging non-recovered transactions which would have prompted an investigation and I'm sure would have led to the same outcome.”
She explains that had this not been done, more would have been required from the Post Office – namely an investigation – before the Post Office would have reached the decision to refund the £150, although she does say that she is sure the same outcome would have been reached. This is rather different to the way the case was put to Mrs Burke, which did not include the requirement for any such further investigation by the
Post Office being required. The impression sought to be given was that on the Fujitsu documents already available, a TC would have been issued anyway to Mrs Burke a few days later. I do not accept that. Something more at the Post Office end would have been plainly required, as Ms Van Den Bogerd explains. Mrs Burke was the person who had in fact provided proof of payment out, and so no investigation was undertaken.
It is in my judgment likely, on the basis of all the evidence in the litigation to date, that the sum of £150 would have showed up as a shortfall at the end of the next branch trading period, as any further investigation was not likely to have been completed by the end of that period. However, if Mrs Burke had not acted as she had, and if an investigation had been done by the Post Office, if that had been resolved in her favour, and if that had led to a TC being issued, then the shortfall in the branch accounts would have been corrected in a later branch trading period, namely the period during which the TC was issued. No loss flowed to Mrs Burke in respect of this incident, as a result of her own investigation. In any event, this is not a point of difference between the parties that it is necessary to resolve in order to resolve the Horizon Issues. There was plainly a potential impact to her branch accounts in any event, and that potential impact was caused by the Horizon system.
Mr Roll
Mr Richard Roll had worked at Fujitsu between 2001 and 2004. This was during the days of Legacy Horizon and he had no experience of Horizon Online. He provided two witness statements, the first of which was dated 11 July 2016. This was obviously very much in the early days of the group litigation. The second statement addressed certain factual matters in Dr Worden’s first expert report. I consider Mr Roll to be an important factual witness in this group litigation. He has no personal interest in the litigation and is not a claimant. He has never worked for the Post Office, although whilst at Fujitsu working on Legacy Horizon he obviously had a great amount of involvement in the Horizon system. However, given the expert evidence and particularly the degree of agreement between Mr Coyne and Dr Worden on the number of accepted bugs in Horizon, which even on Dr Worden’s position was 11 different ones, in addition to Mr Roll’s evidence there was ample other evidence in relation to my findings on the Horizon Issues. Mr Roll’s career has started in the Royal Air Force, which he joined in 1976 and left in 1989. Whilst in the RAF, which he had joined as an avionics engineer, he worked on mainframe computer systems and was selected for a software development team working on aircraft control and attack systems. After he left the RAF, he worked in various roles in development and support, and joined Fujitsu in January 2001. There, he worked in the Software Support Centre, or SSC, in Fujitsu at Bracknell in 3rd line support.
After he left Fujitsu he changed career direction entirely and attended the University of Southampton, from where he obtained a BSc in Podiatry. He worked in the NHS until 2011 and then went into private practice, where he remains, now with his own clinic. The claimants described Mr Roll as a “whistleblower”, effectively making public allegations about how Fujitsu dealt with Legacy Horizon and the access which was available to Fujitsu to SPMs’ branch accounts. This was access of a far wider nature than admitted to by the Post Office or Fujitsu publicly.
It was Mr Roll’s evidence that was substantially behind the eventual acceptance by Fujitsu witnesses of the ability to obtain remote access without SPM knowledge or permission, and the injection of messages into the counter. In particular Mr Parker had, in his first statement, described that Fujitsu lacked this “power” and had said that Mr Roll’s evidence was inaccurate and misleading. Mr Parker later corrected his 1st statement, and I deal with his explanation about this change – and what he said he meant in his 1st statement – in my summary of Mr Parker’s evidence below at [464] to [498].
Mr Roll described himself as an IT specialist, a description with which I agree, and one which is demonstrably correct given the number of years he performed his role at Fujitsu in the SSC in 3rd line support. Problems or issues would only reach 3rd line support if 1st or 2nd line support did not resolve them. His IT experience includes, in my judgment, a considerable degree of expertise in software. Indeed, although I am critical of Fujitsu elsewhere in this judgment, that criticism does not extend to any suggestion that they employed personnel within SSC who were not experienced and sufficiently expert in software matters to perform that role. Had Mr Roll not been sufficiently qualified to work in 3rd line support, he would not have been there for the period that he was.
The Post Office in cross-examination attacked both his recollection, and seemed also to challenge his expertise. This the Post Office is entitled to do. Given he was 3rd line support, in my judgment he obviously had a high degree of expertise. There was a debate at one stage about whether he was “elite” or “super elite”, and he put himself in the former category. The Post Office also concentrated on a narrower band, or subset, of the issues regarding which he gave evidence, focussing more on potential problems with code and Mr Roll’s experience of those, rather than the wider issues generally in respect of Horizon of which he gave evidence. He had made clear very early in his cross-examination that, so far as he was concerned, data corruption was an issue in Legacy Horizon as well as software issues which were related to the code.
Mr Roll had left Fujitsu in 2004 and there will inevitably be difficulties in recollection by any witness of detailed points after such a passage of time. He was willing to indicate when he could not remember something, and would agree with points put to him by the Post Office when relevant. These were often on the basis of hypotheses which he was asked to accept, often in relation to a spreadsheet which Mr Parker had prepared showing how relatively few software issues Mr Roll was said to have worked upon. Mr Roll also, again inevitably, was not submerged in the fine details of the subject matter of the litigation, in the same way as some of the Fujitsu witnesses such as, for example, Mr Parker himself, who is the Head of Post Office Application Support and the SSC, and who has worked on the Post Office account at Fujitsu since 1997, before Horizon was even introduced. Given Mr Roll left Fujitsu in 2004, it is 15 years since he worked on Horizon.
His two statements were fairly short (4 and 8 pages respectively) and given at a high level. One example can be provided from his cross-examination. He was of the view that problems referred to 3rd line support, which by definition would mean that 1st and 2nd line support had not been able to deal with them, would not include problems caused by mistakes by SPMs. Mr Parker disagreed with this and this was put to Mr Roll.
“Q: Then the next unnumbered paragraph, Mr Parker says:
"If NBSC were unable to identify the cause of a discrepancy they would often fall back on a default statement along the lines of 'this looks like a software issue' so that the SSC would investigate it. However, Mr Roll's statement that 'if an error was referred to us then it was extremely unlikely to be due to a mistake made by a postmaster' is not correct. The vast majority of discrepancies investigated by the SSC as pseudo 'software issues' were (and are) not caused by software issues."
I would like to suggest to you, Mr Roll, that that's true, isn't it?
The way that I remember it, it was issues to do with the software that were causing the problems. Whether that was the programme that had been written or data corruption, that's what I remember as our problems being.
Q. You remember there being problems with data corruption?
A. Yes.
Q. I'm not asking you about data corruption, Mr Roll, I'm asking you about software issues. The claim I would like to put to you again is the last sentence of that paragraph:
"The vast majority of discrepancies investigated by the SSC as pseudo 'software issues' were (and are) not caused by software issues." Are you in a position to agree to that?
A. From my recollection I would disagree with that, but it was a long time ago. Q. I'm grateful.”
This passage of cross-examination, and there are a number of other similar examples, shows the following:
Mr Roll’s conclusion was a general one, with which Mr Parker disagreed. Assertion and counter-assertion are not, of themselves, of assistance to resolving the Horizon Issues. His conclusion, as with that of Mr Parker, is simply their point of view.
If a problem made it to 3rd line support, then by definition it had not been capable of being resolved by 1st and 2nd line support.
Fujitsu would, even at the 3rd line support level, attribute some problems which they could not understand or resolve to SPM error, and Mr Parker’s evidence is consistent with this approach by Fujitsu. This is made clear in the text of PEAKs put to Mr Parker. There is a very stark example in one of the PEAKs where the Post Office’s own auditors have specifically ruled out user error, yet Anne Chambers chose to close the PEAK with the defect cause being user error.
The Post Office would try to keep the evidence in cross-examination of Mr Roll within very narrow bounds. This passage shows that Mr Roll expressly gave data corruption as one of the problems of which he had experience. Horizon Issue 1 included the expression “bugs, errors and defects”. In my judgment data corruption is included within that phrase. Mr Roll was kept very close in his cross-examination to evidence concerning code alone. Bugs, errors and defects are far wider than simply code. The Horizon Issues were not specifically restricted to software issues as narrowly re-defined by the Post Office, nor are they restricted to code.
Mr Roll would sensibly concede that the events of which he gave evidence were a long time ago.
As shown by the answer “the way that I remember it, it was issues to do with the software that were causing the problems”, even at this remove of time, Mr Roll had a specific recollection that there were “issues with the software” that were “causing the problems”. This is as long ago as 2001 to 2004.
Some of the cross-examination was of no assistance in resolving the Horizon Issues, for example an exercise with a spreadsheet which Mr Parker had been involved in preparing, showing the percentage of calls to 3rd line support and the broad categories into which they had been placed (by Mr Parker, as it turned out, and inaccurately at that). Mr Roll was in no position to give any helpful evidence in respect of this, and the categories were in any event misleading as headline descriptive terms. A witness who had left Fujitsu’s employment some 15 years ago would not, sensibly, be in a position to agree or disagree with such detailed points or collation of data in such a detailed spreadsheet in any way that would provide the court with useful evidence, particularly when it emerged (as it later did when Mr Parker was cross-examined) how misleading the Fujitsu headings were.
Mr Roll had looked at Mr Parker’s statements, although he said only briefly, and he did not know how many statements Mr Parker had made. He could not remember some of the details, such as the detailed descriptions of 1st and 2nd line support at SSC, and was frank about this. He could not recall the split between 1st line support by Fujitsu and the Post Office’s own 1st line help desk called the National Business Support Centre (or NBSC). He acknowledged that he worked in 3rd line support, rather than “third/fourth line support” as he had put it in his witness statement. He did not however over-state his abilities; for example he said some members of 3rd line support were “super elite”, but he would not count himself as falling within that description. Part of his cross-examination which I consider fairly encompasses his approach to giving evidence is the following, where parts of Mr Parker’s witness statements were being put to him.
“Q. And if we can pick it up at paragraph 28 of Mr Parker's witness statement, you worked in third line for between -- well:
"Between 1 January 2001 and 31 December 2014 ...” and those are the years that you worked for Fujitsu "... the SSC received a total of 27,005 calls, meaning that on average 563 calls per month were dealt with over this 4-year period."
And he refers to a spreadsheet setting that out and he then analyses the data in that spreadsheet. Would you accept that that is a fair reflection of the amount of calls coming in, the amount of incidents coming in to SSC when you were there, third line?
I can't really remember. I know there were periods when it was very busy and periods when it wasn't so busy. Sometimes we had three or four jobs on the go at once, other times we were given other work to do from the manager. Q. And paragraph 29, Mr Parker says:
"Transferred calls (ie those not resolved by the SSC) are of interest." He says:
"A very small proportion of calls transferred to 4th line support would have concerned software errors requiring resolution ..." Stopping there, Mr Roll, that's true, isn't it? A. Yes.
Q. So he then says:
"... it would be interesting [therefore] to know the number of calls transferred to fourth line."
Would you agree to that? It would give some indication of the extent to which incidents coming into the SSC properly, genuinely represented software areas that required fixing? That would be a useful way of -- a touchstone of trying to work out -?
A. Yes.
Q. Thank you. He then says that, unfortunately:
"... while the SSC have records of the volume of transferred calls, we do not retain records of where they are transferred to and it is not the case that all of these would have been transferred to 4th line support. For example incidents would often arrive at SSC from internal teams for routing back to help desks." Do you remember that?
A. I don't remember that particular ...
Q. But would it be right to say that of the calls coming into third line support, a significant proportion would go -- of calls that would then be transferred out, would go to places other than fourth line support? Would that be fair? Does that accord with your recollection?
A. The way I recollect it is that calls would come in and we would work on them, either fix them, in which case they would go back to the originator, or we would pass them on.
Q. To other people?
A. To other people.
Q. Depending on the nature of the problem?
A. Yes.
Q. So if there was an infrastructure problem you would probably pass the call on to the infrastructure team, it wouldn't go to fourth line support, would it? A. Probably.
Q. And there are a number of other teams that calls coming into third line would be passed to who would not be fourth line support?
A. Probably. I can't remember the full details.”
This can be seen all to be at a very high level. Mr Roll is not likely to have known the total number of calls that came into SSC during a 4 year (or indeed any) period; the average number of calls per month; or of how these were dealt with. He would also not be likely to be able to recall these details, even had he known them at the time. The analysis which was put to him would have been “useful to know” – the percentage, or even total number, of calls transferred from 3rd line support to 4th line support – would not have been something he would have known at the time either. Interestingly, not only would such information have been something that Fujitsu could have readily recorded at the time, as both 3rd and 4th line support at SSC are both Fujitsu functions, and not only was this plainly accepted as something that would be of assistance, but Fujitsu “did not retain records of where they are transferred to”. Thus the absence of a useful record in this respect is a result of how Fujitsu retained its records.
Mr Roll was then taken to an analysis of calls coming in, and calls going out, which had been done by Mr Parker. This related to some 3,764 calls. Mr Roll was never going to be in a position to dispute that figure; to dispute or agree the percentages given for how the calls were dealt with; or provide any meaningful evidence about Mr Parker’s analysis at all. It was presented to him for the first time in the witness box. Sometimes unreasonable or unhelpful witnesses take issue with matters of which they have no knowledge; Mr Roll did not do so. If he did not know, or was not in a position to answer substantively, he would simply say so.
Another passage of his evidence was as follows, when a passage of Mr Parker’s evidence was put to him that stated "From the SSC, only a tiny proportion of incidents were escalated to the 4th line support team. It follows that only a tiny fraction of incidents raised actually needed to be looked at by the only team who might potentially effect changes in software." He was asked if he accepted the conclusion:
“A. When you take it as an average then yes I suppose so, but the system evolved from -- I started there in 2000 I think it was and left in August 2004. The nature of the work changed over time and the ones that stick in my mind are the ones where there was sort of the fire-fighting efforts where it was -- there were difficult periods where there were software issues, so ...
Q. Yes, Mr Roll. We could by all means go back to the graphs because you will have seen from the live PEAKs into and out of the SSC it actually gave monthly figures, but what I'm suggesting to you, Mr Roll, is that -- although you may well have been busy on all sorts of things, the fact of the matter is that software problems requiring a software fix represented a tiny fraction of the work that was handled by the SSC third line support.
Yes.”
This theme of the way that the Post Office put its case carried on into the evidence of Dr Worden and his Section 8 statistical exercise, which I address below. The expressions “tiny proportion”, “tiny fraction” and other terms used sought to demonstrate, basically, that Horizon worked very well most, if not all, of the time. Given the use to which the Horizon system was put by the Post Office, namely monthly balances which governed the branch accounts of SPMs, with shortfalls and discrepancies being “made good” – which means paid – by the SPMs in question, this approach can be seen to have limited utility. Firstly, these are subjective terms. Secondly, given the PEAKs show that a SPM may report a single unexplained discrepancy for (say) £25,000, or even £1,000 or £500, the fact that this may be only a tiny fraction of the number of calls the SSC worked on that month might come as cold comfort to that specific SPM. The number of calls, and how they were dealt with, is of some relevance to robustness, but such high level and subjective points were not likely to be of enormous assistance, one way or the other, in challenging Mr Roll’s evidence.
It was put to Mr Roll that his role, which he described as “product specialist”, was “the junior level of people working” in SSC, which he disputed. He explained that most people in SSC were “product specialists”. 25 of the 30 or so total number of people were, according to Mr Roll, all product specialists.
“Q. -- the SSC? Of that 30 people how many people were at your level?
When I worked there I -- it was two or three people were senior levels, I think Mr Parker was one, a couple of others, then there was Mike Peach and I believe the rest of us -- 25 or so.
Q. The figures I would like to ask suggest to you, Mr Roll, are about 25 people had the junior level and there were about five people who were true specialists who I think you fairly described as specialist earlier on in your evidence. Would you accept that that might be the case?
A. Junior level seems to -- it doesn't fully explain the complexity of the system or the knowledge of the system required, but yes, I suppose ...
Q. Well, let's agree on ordinary -- I'm not seeking to cast any imputations and it is right that you should – words do have implications. So the ordinary level was your level – A. Base level.
Q. -- and there was this perhaps five or so people above -- the senior people above you and what did they do? Did they do the more challenging work?”
An interesting post script to this line of questioning is that later in the trial it emerged that all of the SSC personnel had the very powerful APPSUP privileges at this time. It is highly unlikely that this was given to those at the “junior level”.
The Post Office attempted at the trial to dilute Mr Roll’s experience and cast him as someone who was not sufficiently skilful to understand, or give evidence about, the matters he explained in his witness statement. It is clear to me, and this was effectively confirmed by the documents put to Mr Parker later in the trial, that everyone at 3rd level support in SSC was a specialist. The notion that these 30 people, in a department separate from the normal Fujitsu work areas, to which access was security restricted, were all at “the junior level” or “ordinary” with 5 people managing them who were “the true specialists” is simply misleading. There were a handful of people at the senior level, and the rest of 3rd line support were product specialists. A Fujitsu internal document, put to Mr Parker by the claimants (but not to Mr Roll by the Post Office) explained it rather more comprehensively. This is from a 2011 document but there was no evidence to suggest that there was a wholesale reorganisation of SSC 3rd and 4th line support between Mr Roll’s time there and this document.
“3rd line support
3rd line support groups within RMGA include:
SSC – 3rd line support for RGMA written application code.
MSS – 3rd line support for software distribution and event management
3rd line support staff apply analytical skills to the symptoms and evidence gathered by 1st and 2nd line and undertake in-depth investigation into incidents. They have detailed knowledge of the system based on documentation and source code inspection.
Trained on operating systems, COTS packages that underlie the application and the coding languages used within the application. They are also expected to self train by examination of support guides, design documentation written for the components of the end user application. They will also have access to development and package management tools to allow the production of specialised diagnostic code, scripts and support tools.
It is incumbent upon the 3rd line support unit to produce a work around and on 4th line to produce the final code solution to any software problem. This does not preclude the production of a workaround by other units or negate the requirement for 4th line to provide assistance in the generation of a workaround.
The SSC are responsible for the implementation of any workarounds that require data changes to the live system. They are the only unit with authorisation and sufficient physical security controls to perform this function.”
(emphasis added)
This gives a more comprehensive picture of the type of expertise required to be in 3rd line support, which is where Mr Roll worked whilst he was at Fujitsu. This is of more assistance than the terms deployed, such as “the five true specialists” (for the senior managers) and “the junior level” for Mr Roll. It shows that 3rd level support have a high level of necessary expertise. They are the only unit with the necessary authorisation and security controls to be permitted to implement workarounds that require data changes to the live system. This is not the role of junior personnel.
This area of questioning also focused on code, and introduced another concept, the “core specialist team”.
“Q. I'm really trying to grope towards what the five true specialists, what kind of work they were doing. They were doing the more challenging kind of work, weren't they?
A. I find it difficult to answer that from what I remember of the way we worked. There were some areas where some of the senior people where I would perhaps have been more -- had more experience because of previous work and previous programming that I would have been better off and had more knowledge than they would have done, but in other areas then they would be far superior to me.
Q. Well, let me suggest one area where they would have been far superior to you, Mr Roll, or at least it was perceived within the organisation -- I'm not trying to have a debate with you about your own perceptions of yourself, that would be completely unfair, but the perception at the SSC was that were there software errors or potential software errors that required large amounts of code to be examined, the people who would generally do that examination would be those five people, people like Mr Parker. Would you accept that?
A. Generally I suppose, yes, although –
Q. And it would be relatively rare for someone outside that core specialist team to be doing that kind of work -- not impossible but relatively rare, yes?
A. Several of us looked at the code on occasions. I was a C programmer in that and other languages before I moved to Fujitsu so there were areas of knowledge we had from previous areas, but I suppose most of it would then be passed on to those people.
Q. I see. So would you accept from me then that generally it would be them that would look at lots of code but occasionally there might be occasions when someone else would look at lots of code?
A. From a PinICL perspective yes, but some of us looked at code more often, just out of interest.”
This series of questions introduced another subjective variable, “large amounts of code”. Mr Roll, in his answer above, explained that he had been a C programmer before he moved to Fujitsu. C is a general purpose programming language which is widely used. Mr Roll, as someone who had been a programmer both in C and other languages before he went to Fujitsu, had both written code himself (as a C programmer), and I find would have been very familiar with code in its different guises. Indeed, given the Fujitsu documentation describing the expertise of 3rd line support in [168] above, which stated 3rd line support would have “detailed knowledge of the system based on documentation and source code inspection”, it is clear to me, and I find, that Mr Roll was amply experienced to give evidence to this court on the matters that he did concerning the Horizon system (which was at that point Legacy Horizon), the problems that were experienced with it, and what occurred at SSC. Source code is a term used to describe a version of software, or a description, as it is originally written. There are different definitions of source code in different computer authorities (such as the Linux Information Project) but the precise definition is not important for the purposes of the Horizon Issues trial. The Horizon Issues do not concentrate solely on problems either with code, or source code. The phrase is “bugs, errors and defects” and the Horizon System is defined in the Horizon Issues.
What is important, in my judgment, is that Mr Roll used to programme in C programme language and other languages before he went to Fujitsu; he was required to understand and have “detailed knowledge of the system based on documentation and source code inspection” by virtue of being a member of 3rd line support at SSC; all the evidence demonstrates that he did his job effectively and competently; and he did not leave Fujitsu under any sort of a cloud. It was effectively accepted by the Post Office that he had a responsible position and a responsible role, and was well trained. The Post Office also sought to draw the following conclusion from Mr Roll’s evidence in favour of Fujitsu: namely that “….a clear picture emerged of Fujitsu as an organisation which was thorough, professional and conscientious and which took considerable care to ensure that matters were properly investigated and dealt with.” I do not accept that this picture emerged from Mr Roll’s evidence, but there was a considerable amount of other evidence in respect of Fujitsu in the trial other than that of Mr Roll. I will return to this claim by the Post Office at the end of my review of all the evidence, factual and expert, which is in Part L “Overall Conclusions”. That section of the judgment begins at [925] below.
Mr Roll’s evidence was that errors made by SPMs were relatively easy to pick up at 1st and 2nd line support level; that most errors he dealt with were coding errors or data corruption; that issues were identified that required software “fixes” to be written by Development; and – importantly in my judgment - that the type of issues that were routinely encountered at SSC could and did cause financial discrepancies to branch accounts. He also stated that:
“If we were unable to find the cause of the discrepancy then this was reported up the chain and it was assumed that the postmaster was to blame.”
He said that even if software fixes were developed the problem would sometimes reappear several weeks later. He also stated that remote access to branch systems was possible; the ability was extensive; that this was done without the SPMs being aware; that data and transaction information could be changed by Fujitsu; and that sometimes SSC would log into a branch system whilst it was switched on but not in use.
The final paragraph of his 1st statement said:
“In summary, the issues with coding in the Horizon system were extensive. Furthermore, the coding issues impacted on transaction data and caused financial discrepancies on the Horizon system at Branch level. It was those issues that I, and other colleagues at Fujitsu, were routinely working on daily. Furthermore, remote access to the Horizon system at Branch level was extensive, as was the ability to change data and change transaction information, even while the postmaster was working, without the postmaster being aware of this.”
He had also said that “during the course of resolving the software issues, we would frequently access a Post Office counter IT system remotely”.
His use of “frequently” and “routine” are, in my judgment, subjective, and as explained above in terms of “tiny fraction”, subjective terms are not entirely helpful. What he meant by this is it was not unusual for this to occur. It is difficult to judge, at the remove of 15 years from when Mr Roll left Fujitsu, just how often something that he remembers as frequently or routine in fact occurred. He sensibly accepted that the more regular or mundane matters in which one is involved tend not to be remembered, with rather more unusual events sticking in the memory. In terms therefore of how often this occurred whilst he was there, it is not possible to come to a concluded view. However, I consider it important that the occasions to which Mr Roll refers were not isolated incidents on his evidence, nor were they unusual, and that financial discrepancies on the Horizon system at branch level was something which he recalls working on daily. It is also important that he gave evidence that if Fujitsu could not track down the cause then it was assumed the SPM was responsible, in other words user error would be used as a default setting for investigations. This matches the evidence of an enormous number of PEAKs in the Technical Appendix.
He also gave evidence about hardware failures, and gave specific responsive evidence in respect of Dr Worden’s expert report.
I accept Mr Roll’s evidence, which is supported by the contemporaneous documents. The Post Office in its closing submissions accepted that he was a careful and precise witness, and also that “he was at pains to assist the court and to give accurate answers”. It also, however, submitted that his evidence “was, unsurprisingly, hazy in many respects”.
Notwithstanding the limitations on his evidence due to the passage of time, I found Mr Roll to be a reliable and helpful witness. I do not consider it was hazy in any important respects. I also found his evidence to be very important. The Post Office set out in cross-examination to demonstrate that he did not have the expertise that he claimed to have; that he was at too low a level in 3rd line support to have been involved in the matters which he described, certainly so far as software issues were concerned; and that Mr Parker’s evidence in particular about what he had been doing should be preferred. On those two former points the Post Office failed, in my judgment. The success or failure of the latter point also, but not exclusively, depended upon the evidence of Mr Parker. After Mr Parker was cross-examined, it was clear to me that Mr Parker’s exercises that were put to Mr Roll could not be relied upon to demonstrate what they sought to demonstrate. I prefer and accept the evidence of Mr Roll to that of Mr Parker and the other Fujitsu witnesses (with the exception of Mr Godeseth) by some considerable margin. I find that during the years when Mr Roll worked at Fujitsu, 2001 to December 2004, SSC were not infrequently involved in attempting to remedy unexplained shortfalls and discrepancies in branch accounts reported by SPMs. If they were unable to find the cause of the discrepancy then the assumption would be made that it must be the SPM to blame. This is clearly shown in my analysis in the Technical Appendix particularly on the different entries in the Bug Table, and my findings on the number of bugs present in the system.
There is one area of Mr Roll’s evidence, again a wholly subjective one, to which I do not attribute any weight, not that I disbelieve him. This was concerning the pressure of work within the SSC and the fact that members of the SSC were under pressure to “close calls”, which basically means record them as having been completed. The relevant passages are as follows:
“15. During my time at Fujitsu I know that there were budget pressures and redundancies which impacted system development and testing. The test team felt they were under enormous pressure to complete the testing within certain timescales which negatively affected the test regime. Meanwhile, the development team had to balance time spent on fixes, with time spent on developing new features for Legacy Horizon and time spent developing a new system which I believe later became Horizon Online.
16. In my first statement, I refer to the pressure that the SSC team and Fujitsu were under generally due to an awareness of the financial penalties imposed by the service level agreements between Post Office and Fujitsu (paragraph 12 of my first statement). I believe that although individual penalties were quite modest, when applied across multiple counters/post offices the cumulative figures involved were very high, potentially amounting to tens of millions or more. I disagree with Stephen Parker’s statement that these potential financial penalties were not a factor for the SSC (paragraph 43 of Stephen Parker’s witness statement) as we were aware of them and often commented on them, e.g. “That’s saved Fujitsu another £25 million”.”
I do not attach much weight to this as pressure in the workplace is such a subjective matter. Some members of SSC may have felt under great pressure; others less so. This is a subjective account of what it was to work at SSC at that time from the point of view of Mr Roll. The type of statement to which Mr Roll refers in terms of millions of pounds saved does not advance matters one way or the other. Given the type of problems that 3rd level support was involved in attempting to resolve, there would have been an inevitable pressure involved in attempting to resolve such issues speedily. Mr Roll chose to leave Fujitsu, and therefore he left those pressures behind. Putting it at its mildest, it would be extraordinarily disappointing if PEAKs were closed, attributing fault to a SPM, simply because that was the easiest and quickest way for a SSC product specialist to keep on top of their workload. Without separate cross-examination of each product specialist who chose to close PEAKs where prior entries suggested user error had been ruled out, it is not possible to determine the degree to which this was their motivation in any individual case. However, it is not relevant. It is no part of the Horizon Issues to determine why, or whether, individual personnel at SSC acted in a particular way – budgetary concerns for their employer; exercises of judgement; even laziness; or any of the other many possibilities. Mr Parker gave evidence that “the possibility of financial penalties was never a factor for the SSC”, and Mr Roll said that it was. Mr Parker’s evidence on this suffers from a wholesale attribution of this lack of motive to every member of SSC, as indeed does Mr Roll’s, in the other direction. Everyone is different, and there will have been a
range of different reasons operating on each member of SSC each day. Any sensible business will, in any event, always have at least part of its attention on financial performance and this is understandable. The Horizon Issues trial is not an inquiry into how Fujitsu manages its personnel, or its business. I do question the allocation of Category A, B or C to some of the PEAKs that were used in the trial, but this is something that was pursued (though not very far) with Mr Parker and I deal with it there.
The way that Dr Worden was simply not prepared to accept Mr Roll’s evidence at face value, and set out to disprove his factual evidence, something which in my judgment an independent expert should never do, is dealt with in the section of this judgment where I deal with Dr Worden’s evidence. The degree to which the evidence of Mr Roll, and his experience of Legacy Horizon between 2001 and 2004, affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
The fact that I find Mr Roll reliable does not mean that I automatically accept all his conclusions. I have given a specific example at [181] about financial pressures. However, Mr Roll’s evidence was supportive of the claimants’ case and it was necessary for the Post Office to render his evidence unreliable, insofar as they could, in order to damage the claimants’ case and bolster their own. In my judgment, this the Post Office failed to do. Mr Roll’s evidence was also what directly led to the true picture emerging of remote access. However, of itself this does not provide an entire answer to the Horizon Issues.
Mr Henderson
As explained in [78] and [79] above, Mr Henderson is a director of a company called Second Sight which was involved in a mediation scheme, until the Post Office withdrew from it. Initially Second Sight was engaged to perform an investigation, but after its interim report its scope of involvement changed. It ran from 2012 to 2015, when Second Sight’s appointment was terminated. The investigation and subsequent mediation scheme were set up in the light of disquiet about the complaints of various SPMs, the operation of Horizon, and concerns that were publicly discussed, not least by Members of Parliament. All of the documents and emails which Second Sight had at that time were returned to the Post Office. Mr Henderson estimates these as 16,500 emails and 18,500 other documents, which he said were sensibly organised in 1,700 folders. They were in good order.
The documents in that set that were provided by the Post Office to the claimants for this litigation were not, according to Mr Henderson, provided in the same way. He criticised them for having no folder structure, having had metadata removed, and lacking the original date and time stamps which were present on the original documents with which Second Sight had been provided by the Post Office. Some important documents, such as the Fujitsu XML transaction reports, were described by the Post Office as “unreadable”. He had however reviewed some, but not all, of the XML data when inspection was provided of some of the unreadable documents. He did not believe that the documents provided in the Horizon litigation were the complete set of data provided to Second Sight, which he had returned. I accept his evidence on this. He was clear in his recollection about how the documents were organised, and approximately how many there were and their type. I do not know why the claimants were not given, in disclosure, the same documents, prepared and collated in the same way, as the Post Office themselves had received them from Second Sight.
He described Mr Jenkins, whom he had met, as the Fujitsu lead engineer on the Post Office contract. He had met him in September 2012. He explained in his witness statement that he had been given sample XML data by Mr Jenkins, and had used this to see a level of detail that neither SPMs nor, interestingly, the Post Office, had available to them to view. He had also been told at the same meeting that remote access was “occasionally used” by Fujitsu. He asked for complete email records for 2008 for the 7 employees known to be working at Bracknell at the time, so he could satisfy himself as to whether such remote access was done without the SPM’s permission or knowledge. He was not given these, and the partial records he was given were inconclusive. His reverse engineering of the XML data he was given gave him grounds for concern but Second Sight’s engagement was terminated before he could reach any concluded view.
He had also identified what he considered to be particular difficulties that Horizon had with foreign currency – it is (or was) a single currency system and only used pounds sterling – and also the National Lottery. In view of the evidence of Ms Van Den Bogerd in the Common Issues trial and the Ping fix, that latter point may not be particularly controversial in any event as this litigation progresses. Foreign exchange is the subject matter of bugs 14 and 23 in the bug table. Bug 14 is now accepted by the Post Office in Appendix 2 of its Closing Submissions as one of a number of “bugs with lasting impact (although they were resolved)”. I find in the Technical Appendix that Bug 23 is a bug. The contents of the Technical Appendix are therefore consistent with Mr Henderson’s views on this.
A working group was involved in the Second Sight scheme, and the progress of cases all the way through the mediation scheme was slower than the working group wanted. Second Sight’s appointment was terminated before Mr Henderson could reach finalised conclusions on most of the issues that he said had been uncovered. I have already stated the approach that I adopt in relation to the technical nature of the Horizon Issues, and how the evidence of the experts in IT who gave evidence before me effectively takes precedence over observations by other witnesses on technical issues, which must include Mr Henderson.
Mr Henderson explained in his oral evidence that he considered his evidence to the court not to be entirely unrestricted. He considered it was subject to a restriction that had been imposed upon him. There was a letter of engagement in this respect, which
was dated 1 July 2014 and accompanied by a document which was signed by someone called Chris Aujard for the Post Office. The letter of engagement set out several pages of terms governing Second Sight’s engagement. Clauses 2.1 onwards set out what Second Sight were to do:
“2.1 The Scheme has been set up to try to achieve the mutual and final resolution of a Subpostmaster's concerns about Horizon and any associated issues.
Second Sight Support Services Limited ("Second Sight" or "you") has agreed to be a member of the "Working Group" whose role it is to oversee the Scheme and to assist in investigating individual Subpostmaster complaints.
This letter and its schedules will form the basis of the terms of Second Sight's engagement by Post Office Limited ("Post Office") to provide Services to the Working Group in relation to the Scheme.”
Clause 6.2 stated:
“6.2 Second Sight will not, and will ensure that the SS Directors and any SS Personnel will not, act, directly or indirectly, in any capacity (whether for any former or current Subpostmaster or a competitor of Post Office or otherwise) against Post Office or any of its officers, directors or employees save to the extent a) that it is expressly agreed in writing by Post Office that the work proposed to be undertaken will not have a material adverse effect on Post Office's commercial or financial interests or reputation, or b) as required by applicable law or by the mandatory rules or requirements of any regulatory authority, government department or agency to which Second Sight is subject or c) as required by an order of a court of competent jurisdiction.”
Clause 6.3 imposed this for a period of 15 months.
The shorter one page document which Mr Henderson and his co-director were asked to sign and return contained different terms. The letter of engagement set out the terms upon which Second Sight would act in the mediation scheme. The shorter document stated the following:
“I will not act, directly or indirectly, in any capacity (whether for any former or current Subpostmaster or a competitor of Post Office or otherwise) against Post Office or any of its officers, directors or employees save to the extent a) that it is expressly agreed, in writing by Post Office that the work proposed to be undertaken will not have a material adverse effect on Post Office's commercial or financial interests or reputation, or b) as required by applicable law or by the mandatory rules or requirements of any regulatory authority, government department or agency to which Second Sight is subject or c) as required by an order of a court of competent jurisdiction.”
The period of that was stated to be 15 months, corrected by hand from the period initially typed in of 12 months. This obviously therefore, as signed, represented if not the same then almost identical terms as those contained in paragraph 6.2 of the letter of engagement.
It is a curiosity that the detailed terms in the letter of engagement were produced by the Post Office – as Mr Henderson explained – about half way through the process. It is also the case that the services to be provided by Second Sight were to be provided to the Working Group, which also included members of the Justice for Subpostmasters Alliance or JFSA, although the services were to be paid for by the Post Office. I have outlined these terms above to make it clear, on a transparent basis, what restrictions were imposed upon Second Sight. The Post Office’s wish to avoid any “material adverse effect on Post Office's commercial or financial interests or reputation” is expressed in very wide terms. Its concern about its reputation is similar to a provision included in its funding agreement with the National Federation of Subpostmasters which is referred to in Judgment (No.3).
The restriction to which Mr Henderson referred in his oral evidence, which he considered limited his evidence, was an agreement between the Post Office and the claimants about what Mr Henderson could state in evidence in this litigation, which he referred to as the “protocol agreement”. The Post Office was therefore in the position that it could, to a certain extent, control the scope and extent of Mr Henderson’s evidence. He also thought that there was a restriction on the length of the witness statement he could provide. For clarity, there was no such restriction imposed by the court.
The Post Office, in its closing submissions, submitted the following:
“Mr Henderson was asked whether confidentiality restrictions had caused him any inhibition in answering the questions put to him in cross-examination. He said that he had the issue in the back of his mind and that he had tried to make sure that his answers did not infringe the protocol. It is understandable that Mr Henderson would wish to be careful, but the idea that he was restricted in answering Post Office’s questions is bizarre.”
The submissions then went on to deal with the questions put to Mr Henderson, and how there was no restriction that could have impacted upon his answers.
The actual question posed to Mr Henderson was not as set out in the above extract of the submissions at [195] above, it was rather wider. It was posed by me and it was at the end of his cross-examination. It was in the following terms:
“I just want to be clear: is it your evidence therefore that because of that protocol agreement your evidence of fact to this court is narrower in scope than it would be absent the protocol agreement?”
His answer was “yes, it is”. He was then asked by counsel for the Post Office whether it had inhibited him in answering questions, and he said it did not. I do not consider that the closing submissions are an accurate summary of both questions put to him about this, or the restrictions he considers had an impact upon his evidence. The restriction was wider than impacting upon the questions he was asked, and his answer showed that his evidence as a whole had been affected. In any event, those questions were posed to him by the Post Office, as he was called as a witness for the claimants.
The mediation scheme was brought to an end by way of notice given by the Post Office to Second Sight on 10 March 2015. The cross-examination on this was as follows:
“Q:…. If we could just look at it very briefly, the termination letter to which you have just referred is at {F/13/24.1}. That's the termination letter I think to which you at least indirectly referred. A. That's correct.
Q. If we could look then to {F/1324.2}. This is a much longer letter from Post Office that came to you on the same day. Do you recall this longer letter? If I maybe tell you broadly what it is to do with. It's a letter in which Post Office sets out a plan for how Second Sight could finish its outstanding work. Do you remember that letter? A. Yes, I do.
Q. And the second paragraph makes clear that Second Sight was expected to continue working during the notice period and that even beyond the notice period there would be a proposed future role for Second Sight. That's what this document dealt with, isn't it? It's a fairly long document.
A. It is, but there's another document of that date or very close to that date which you haven't mentioned which was the press release from Post Office announcing the winding up with immediate effect of the mediation scheme itself. That I understand was the primary decision and our termination was a consequence of that, not a separate issue.
Q. Well, I think when you say mediation scheme it's fair to say, isn't it, that this winding up process, as you describe it, did not put an end to mediations; in fact –
A. It put an end to the mediation working group with, as I understand it, no consultation and the announcement was made I think the day immediately before the next planned meeting of the working group, so it was a considerable sort of shock to everybody.”
The following points can be made in respect of the evidence from Mr Henderson:
Second Sight was engaged to act in a mediation process. Mediation is consensual dispute resolution, and such processes require a high degree of confidentiality within them if they are to be effective.
The termination of that process unilaterally by the Post Office, with or without consultation, is, so far as this group litigation is concerned, something that may or may not in the future call for further consideration in terms of costs. It is not at this stage relevant to the answers to the Horizon Issues.
Reasonable and conventional terms of confidentiality were to be expected in any terms of engagement agreed between the Post Office and Second Sight, given the scheme was a mediation scheme.
Terms seeking to protect the “Post Office's commercial or financial interests or reputation”, which were included, whether justified or not in all the circumstances of the role being performed by Second Sight, are not relevant to the answers to the Horizon Issues.
5. In Farm Assist Ltd (in liquidation) v Secretary of State for the Environment,
Food and Rural Affairs [2009] EWHC 1102 (TCC) a mediator, who had been issued with a witness summons compelling her attendance to give evidence, failed in her bid to have the summons set aside. Ramsey J gave a judgment refusing to do so, having considered the position on confidentiality, without prejudice privilege and other privilege. He stated, in his conclusions at [44]:
“The court will generally uphold that confidentiality [of mediation proceedings] but where it is necessary in the interests of justice for evidence to be given of confidential matters, the Courts will order or permit that evidence to be given or produced.” The point has not, however, been argued in this case in any respect, and it may never arise.
In this case the restriction has been imposed by the Post Office and agreed by the claimants. It is regrettable, in my judgment, that any witness of fact feels their evidence to be restricted by any existing agreement with a party to that litigation. Apart from anything else, it is something of a contradiction for a witness, who swears or affirms that their evidence is “the truth, the whole truth and nothing but the truth” to then add that there is such a restriction. This appears to contradict the requirement to tell the whole truth. However, the court has never been asked to become involved in resolving any dispute between the parties in this respect.
I do not consider that any such restriction – the scope of which I am in any case unaware – will have had any effect upon my consideration of the correct answers to the Horizon Issues, or the answers themselves.
Mr Henderson seemed to me to be a careful and honest witness. Although his involvement in the Second Sight scheme may have led to his having particular personal views about Horizon, he was measured about how he expressed these. His views about concerns over issues in particular affecting the National Lottery and foreign currency exchange are, to a certain extent, corroborated both by the Ping fix and the position of the Post Office on bug 14 in the bug table. However, whether they are corroborated or not, it is not necessary to come to a concluded view on Mr Henderson’s evidence about these, as it is the two IT experts whose evidence will be of most assistance in this respect.
The degree to which the evidence of fact by the claimants affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
E. Evidence of Fact: The Post Office
The Post Office called the following witnesses. For the reasons explained further below, I consider Mrs Van Den Bogerd and Mr Godeseth to be the most important witnesses called by the Post Office.
Mrs Van Den Bogerd
Mrs Van Den Bogerd gave evidence in the Common Issues trial, and indeed her witness statement for the Horizon Issues trial was served during that trial in November 2018, and shortly before she was called as a witness. She had become the Business Improvement Director of the Post Office shortly before the Common Issues trial. She is a senior director at the Post Office and is, so far, the most senior member of Post Office personnel to have given evidence in the group litigation. I had made certain adverse findings about her evidence in Judgment (No.3) on the Common Issues, which was sent out to the parties in draft form under the usual embargo shortly before the Horizon Issues trial started, and was formally handed down at the end of the first week in the that trial, just after the claimants’ evidence of fact had been completed.
I have already explained that Ms Van Den Bogerd’s evidence was considered completely afresh in this trial, and that simply because I had made the findings that I had about her previously, did not mean that I would adopt the same conclusions in this trial. Her written evidence for the Horizon Issues trial was still substantially of the same tenor in relation to individual SPMs, in terms of widespread attribution of fault to SPMs as a default setting, but I did detect a change of approach in Ms Van Den Bogerd in this trial, compared to the Common Issues trial.
Originally, for example, in her statement Mrs Van Den Bogerd had specifically attributed Mr Patny’s loss on MoneyGram to user error, based on what she said the data showed. This passage, in paragraph 72, was corrected by her on a sheet of corrections of a number of the Post Office’s witnesses, and in that correction (there were a number of others) she explained that Mr Patny had cancelled the transaction but had not reversed it. She therefore accepted that the transaction had been cancelled.
She generally demonstrated in her cross-examination a more realistic approach to the accuracy of her evidence than she originally demonstrated in the Common Issues trial. By the time she was cross-examined she would have had the benefit of reading Judgment (No.3), and that might explain her new approach.
She had also amended, in her sheet of corrections, certain statements that had been included in her Horizon Issues witness statement that were simply not sustainable on the facts. One example of this was in relation to Mrs Stubbs (a claimant and witness in the Common Issues trial, though not in the Horizon Issues trial) whom Mrs Van Den Bogerd had said “chose to settle centrally” items that were, in fact, obviously and plainly disputed by Mrs Stubbs; and in respect of which no SPM had any real “choice”. Their choice, such as it was, was either paying immediately or settling centrally, which meant not paying immediately, but seeking time to pay. Mrs Van Den Bogerd’s explanation for this was that she had not known these sums were disputed by Mrs Stubbs. Quite how that could be, given the extended saga in relation
to these sums, the involvement of Mrs Stubbs’ MP on her behalf (Sir John Redwood, a former Cabinet Minister), the Post Office’s promises both to Mrs Stubbs and her MP of an investigation (the results of which, if one was ever done, have still not emerged, so far as I know), and indeed Mrs Stubbs’ own evidence in the Common
Issues trial, is entirely unclear.
She also explained that in relation to Mrs Burke “I have looked at other evidence in relation to Mrs Burke and what was very clear to me is that Mrs Burke had done absolutely nothing wrong in that situation.” This was not at all how Mrs Burke had been cross-examined. Mrs Van Den Bogerd said that the corrections she had made to her statement had been communicated to the Post Office’s solicitors before the Horizon Issues trial had started. If that is true, I do not see how counsel for the Post Office would have cross-examined on the basis of her un-corrected statement.
She gave evidence about out of hours transactions and so-called phantom sales, the latter of which she explained (in her written evidence) as follows. “I am informed by Post Office’s solicitors that in the course of investigating this matter, Fujitsu have advised that ‘phantom sales’ were reported in around 2000 which appeared to be caused by hardware issues”. There is a master PEAK in relation to this from 2001, and even though Mrs Van Den Bogerd was very closely involved in the issues on Horizon, she had not known about this until some time later. Indeed, she could not remember even the approximate year when she had become aware of it. She did not even recall, in the witness box, having seen the master PEAK before.
I am most surprised that Mrs Van Den Bogerd could not remember seeing this PEAK before she was shown it in cross-examination. It is a very important PEAK. It is PEAK number PC0065021, dated 17 April 2001. The reason it is important is as follows. It relates to multiple branches. It concerns phantom transactions. It identifies dissatisfaction from more than one SPM as to how phantom transactions are being investigated and resolved, or more accurately, how they are not being. It shows calls being “closed” – ie brought to an end - without the permission of the SPM, even though that should not be done. It also shows at least one SPM threatening not to comply with their contractual obligations due to lack of confidence in the system, and also threats of legal action. Further, in one branch, where items had been the subject of phantom transactions (according to the SPM) ROMEC, the Royal Mail’s own engineers, attended that branch to fit suppressors and other equipment in an effort to rectify this.
The PEAK plainly records the involvement of ROMEC, the Royal Mail’s own engineering personnel, as follows. “ROMEC have been to site and state that they have actually seen the phantom transactions, so it is not just the PM's word now.” (emphasis added). The significance of this entry is obvious, and notable. Mrs Van Den Bogerd agreed that this was “independent site visit corroboration of the problem by Royal Mail’s own engineers at the branch”, and she also agreed that this was “clearly not user error any more”. I do not understand how the master PEAK containing such important information could not have been at the forefront of Mrs Van Den Bogerd’s mind. It is, in my judgment, important corroboration from those with experience of Horizon (the Royal Mail’s own engineers) who stated they had actually seen the phantom transactions.
However, the conclusion reached by Fujitsu and recorded in the PEAK was as follows:
"Phantom transactions have not been proven in circumstances which preclude user error. In all cases where these have occurred a user error related cause can be attributed to the phenomenon"
The PEAK also concludes “No fault in product".
This conclusion by Fujitsu is only not made out on the factual evidence, including the contemporaneous entries in the PEAK itself, but it is, in my judgment, simply and entirely unsupportable. It wholly ignores the independent support of the ROMEC engineers, who have reported that “they have actually seen the phantom transactions” and it arrives at a conclusion that, in my judgment, entirely contradicts the evidence available to Fujitsu at the time, and indeed contradicts common sense. Given the entry that “it is not just the PM’s word now”, this conclusion ignores two entirely different sources of actual evidence. One, what the SPM reported. Two, what the ROMEC engineers visiting the branch actually saw.
Another PEAK Mrs Van Den Bogerd was asked about demonstrated the lack of accuracy in Fujitsu’s characterisation of the type of problems that made their way to the SSC (and which formed the basis of the Post Office’s cross-examination of Mr Roll based on the Fujitsu/Parker spreadsheet). This was PEAK PC0208335 of 14 February 2011, also headed “Phantom Stock Declaration”. This related to withdrawn stock discrepancies, and was summarised as "Branches will be forced to declare stock when they don't want to. Apparent reappearance of withdrawn stock may cause spurious discrepancies." It was recorded in the PEAK that this could affect 10 branches per week over the next few months. The PEAK showed that a SPM was told to declare the correct stock, which that SPM disagreed with, and which Mrs Van Den Bogerd agreed would mean the SPM entering an account into the Horizon system with which the SPM disagreed. Mrs Van Den Bogerd had not seen this PEAK either, before she was asked about it in the witness box. She agreed that this PEAK appeared to be a software problem, and also that it was in a sense easier for the SPM because the phantom transaction related to stock that was not even any longer in use. The SPM in question simply could not have dealt in that stock.
In any event, I found this PEAK of great assistance, not only due to the content I have summarised above. A fix was developed by Fujitsu but Anne Chambers, a Fujitsu employee who had also given evidence in legal proceedings in court before (in at least one case, the Castleton case) had entered the following in the PEAK which stated that “this fix and the MSC already applied doesn't remove all old declarations”. Further, Fujitsu chose to categorise this PEAK as “Administrative Response”. That term does not begin, in my judgment, properly to describe or summarise the problems to which the body of the PEAK referred. I find that this PEAK clearly related to a software problem, regardless of the misleading way that Fujitsu chose to categorise it as “Administrative Response”. I consider that anyone reading this PEAK at the time could only sensibly conclude that this was a software problem.
Mrs Van Den Bogerd was taken through a number of examples of real-world situations, recorded in a variety of contemporaneous documents, where a wide variety
of SPMs reported a very wide range of problems. In one an internal Post Office email reported that a SPM had “found sensitive issue with Horizon when the system put a phantom cheque on the cheque line in July 2013. Claims to have evidence to support his claim. Although he himself did not suffer a loss, thinks that Horizon is flawed. Did not ask to be contacted about this. Just wanted to say that he had this information and threatened to go to MP as a result.”
The question was posed internally at the Post Office:
"Given the current media and in particular the BBC's attention on Horizon, do you think it is worthwhile looking into this 'alleged flaw' with Horizon that this SPMR has highlighted to pre-empt any enquiries from his MP?"
The ultimate response from Andrew Winn of the Post Office was that the claim could not be investigated without further details and Fujitsu involvement, that Mr Winn did not understand the purpose of the call by the SPM, and also stated:
“My instinct is that we have enough on with people asking us to look at things.”
Mrs Van Den Bogerd agreed that this was an inadequate response. She said it would have been very easy for Mr Winn to have contacted the branch and obtained further details, and he should have done so. In my judgment, the stance taken by the Post Office at the time in 2013 demonstrates the most dreadful complacency, and total lack of interest in investigating these serious issues, bordering on fearfulness of what might be found if they were properly investigated. This SPM, whose branch was known to the Post Office, should obviously have been asked for further details (if further details were required for an investigation), and the Post Office and/or Fujitsu should plainly have investigated the matter as a matter of some importance. By 2013 Horizon was an extraordinarily controversial subject; there can simply be no sensible excuse for the Post Office’s failure to try and understand this particular subject. This is particularly reprehensible given that an internal Post Office document in August 2013 showed that Mr Winn’s involvement in this was because his area of responsibility was as follows: “also responsible for resolving specific branch accounting issues.” It was his specific job to resolve specific branch accounting issues, yet he decided at the time that “we have enough on”. I agree with Mrs Van Den Bogerd that this is inadequate – that is putting it at its most favourable for the Post Office. Somewhat stronger terms are also justified.
Mrs Van Den Bogerd had only learned of the ability of Fujitsu in terms of remote access, namely the insertion of transactions at the counter under Legacy Horizon “within the last year or so”, which given she is central to the Post Office’s position in the group litigation shows, in my judgment, a remarkable situation. It is not necessary, in order to resolve the Horizon Issues, to go further into why she has found this out so belatedly, or why that might be.
Mrs Van Den Bogerd would accept changes to her evidence in cross-examination where these were justified. One example was in relation to Mr Latif and corrections issued in respect of the lottery. Her written statement had said:
“However, due to an error by Post Office, instead of increasing the scratch card stock, the TAs decreased the stock. To be clear, this was a data entry error by Post Office and not an issue with Horizon.”
Actually, given the process of issuing TAs for the lottery was entirely automated, and done in conjunction with the Camelot system and by Horizon, she accepted that the passage should be corrected. The cross-examination was as follows:
“Q: So would it be fair to correct that part of your statement – A. It would be actually, yes.
Q. -- to say "To be clear, this is an issue with Horizon and not a data entry error by Post Office"; is that a fair correction to make?
I have made a mistake in that the way I have worded that, absolutely, yes. So, yes, it is fair.
Q. Would you agree with the formulation I have given, or would you prefer something slightly different? What would you want the court to note as your evidence?
A. So the automated -- my understanding is the transaction -- the TA is the information that comes from Camelot to us and then it is passed through into Horizon, so in that respect Horizon just conveys it, is my understanding, and the information that's come from Camelot in that respect would be incorrect.
Q. So the point is that either way, it is not a manual data entry by Post Office?
A. No, it's not.
Q. Whatever it is, it's definitely not that, you agree with that?
A. Yes, absolutely not. Agree, yes.
Q. What it could be is some problem with the information somewhere between the terminal in the branch – A. The lottery terminal, yes.
Q. The lottery terminal in the branch and the matters showing up on the face of the Horizon terminal in the branch? A. Yes, I agree that.
Q. Somewhere there?
A. Yes.
Q. So it's definitely not a user error, is it?
A. No, that's not user error.
Q. Right. And on the face of it, it at least suggests some doubt as to the robustness and integrity of the Camelot data coming through in that automated system, doesn't it?
A. I would say yes.”
It can therefore be seen that her written evidence – “To be clear, this was a data entry error by Post Office and not an issue with Horizon” – could more accurately be stated, after she answered these questions, as “to be clear, this was not a data entry error by Post Office and was an issue with Horizon”. There is a world of difference between those two statements, which are, in my judgment, poles apart both in their content and effect. Indeed, the accurate statement is the exact opposite of how it had been put in her witness statement.
She also accepted that whereas her written evidence relied, so far as Mr Latif’s experience was concerned, upon the fact that TAs had been “accepted by the branch,
which could have been challenged at that point”, in fact a branch had no choice but to accept TAs. They simply had to be accepted by the SPM, and could not be challenged. She also accepted that there were problems with TAs not just for the Lottery, but for Post and Go, and also Paystation, including in Crown Offices (which were not branch Post Offices but which also used Horizon).
She had also said that Mr Coyne had relied upon extracts from the Rose Report which were “taken out of context and mistakenly claim[s] that the relevant reversal was issued in error by Horizon and not the” SPM. She then went on to explain why this was not correct. The Rose Report is explained in more detail both below and at [940] onwards where I consider contemporaneous documents. However, in her crossexamination, she accepted that this part of her statement was not correct. The Rose Report, in fact and in express terms, stated that the reversal was done by the system, something Mrs Van Den Bogerd accepted; she said that it was “done by the system, absolutely yes”, that it was “definitely generated by the system” and there was “no question” it was generated by the system. She stated she was not “making herself clear” in her witness statement.
Mrs Van Den Bogerd was re-examined on this specific point and the question and answer are as follows:
“Q: The extracts taken from the report by Helen Rose referred to by Mr Coyne are taken out of context and mistakenly claim that the relevant reversal was issued in error by Horizon not the subpostmaster."
I just want to give you an opportunity -- Mr Green was pressing you to accept that actually what you had said in that sentence was wrong. I would like to give you an opportunity to explain to the court what you actually meant.
What I meant was that the actual reversal was part of that recovery and it had actually taken place as it should have taken place, which is what I meant in that. So it wasn't a failed reversal because it actually had happened as it should have happened, but I accepted in there that the -- it wasn't obvious to the postmaster at the time that what had happened -- that he hadn't -- because it didn't show that he had actually -- it showed that he had done it and he knew he hadn't done what we referred to earlier was an explicit reversal. That's what I meant in that.”
I do not accept Mrs Van Den Bogerd’s characterisation of the evidence she had originally given in her witness statement. She had clearly stated that Mr Coyne, having taken passages out of context, “mistakenly claim[ed] that the relevant reversal was issued in error by Horizon, not the Subpostmaster”. In fact she accepted that the reversal had been generated by the system, which in my judgment it plainly had. Not making oneself clear is a curious way of describing that her own statement had said the exact opposite of the factual situation. In my judgment her witness statement had, on the face of the statement, stated the exact opposite on this part of the Rose Report.
In brief terms, the Rose Report dealt with a situation whereby the data being used made it appear that a reversal had been done by the SPM, when in fact it had been done by the Horizon system. Mrs Van Den Bogerd was asked about one particular feature of the Rose Report (named after its author, Helen Rose), which dealt with the fact that no separate code was used such that this could be identified. This arose as a result of an occurrence at Lepton when the SPM engaged a forensic accountant as although he paid the shortfall shown on Horizon (about £80) he was adamant that he had not done the reversal that the system showed he had done. In other words, a reversal could appear in the Credence data as though it had been done by the SPM, yet in fact it had been the system (and not the SPM) that had done the reversal. I consider this to be important, given that the Credence data was used by the Post Office in any investigations. An extract from the report was put to her in her crossexamination (because she had given the evidence above about Mr Coyne, the Rose Report) together with a quotation from an email exchange with Mr Jenkins:
"Q:
“Just one question from my part - if the reversal is system created but shows as an existing reversal, could this not be reflected with a different code ie SR (system reversed) to clear up any initial challenges. My feelings at the moment are not questioning what Horizon does as I
fully believe that it is working as it should, it is just that I don't think that some of the system based correction and adjustment transactions are clear to us on either Credence or ARQ logs."
That's what she is saying, yes? A. Yes.
Q. And that was a fair observation, wasn't it, by her?
A. That was a fair observation, yes.
Q. About the shortcomings of Credence and ARQ logs, yes?
A. Yes.
Q. And Gareth Jenkins' answer:
"I understand your concerns. It would be relatively simple to add an extra column into the existing ARQ report spreadsheet, that would make it clear whether the reversal basket was generated by recovery or not. I think this would address your concern. I'm not sure what the formal process is for changing the report layout. Penny, can you advise as to the process: is this done through a CR?"
Do you know what a CR is? A. Change request.
Q. Change request, okay. Then at the bottom:
" I do believe that the system has behaved as it should and I do not see this scenario occurring regularly and creating large losses. However, my concerns are that we cannot clearly see what has happened on the data available to us and this in itself may be misinterpreted when giving evidence and using the same data for prosecutions.
My recommendation is that a change request is submitted so that all system created reversals are clearly identifiable on both Fujitsu and Credence."
Do you know if that change request was acted on after that?
A. I don't believe it has been acted on.
Q. You don't believe it has been?
A. I don't believe so.”
(emphasis added)
Another express passage or conclusion in the report was also put to her:
Q. "The reversal was due to recovery (Counter Mode Id= 8 118) so this was not an explicit reversal by the clerk. This scenario is fairly rare so it is certainly quite easy for the clerk to have made a mistake and either he or the customer could be in
pocket/out-of-pocket (depending on exactly what happened!)." Then this:
"The system is behaving as it should."
So there were issues, weren't there, where a problem could arise for a subpostmaster by design of the system; were you aware of that? There was a whole category of PEAK codes for faults which are agreed between Fujitsu and Post Office to just stay like that as part of the design?
A. Okay.”
These two questions were then put individually:
“Let's take it in stages. You can see there it says "The system is behaving as it should"? A. Yes.
Q. That would be problematic for a subpostmaster, wouldn't it?
A. The way Gareth [Jenkins] describes it here would be, yes, because what he says is that it would have printed the session receipt but it doesn't seem as if it did, when actually disconnection transaction receipts were actually printed in this example and a recovery receipt was printed. But that's not referred to in here.
Q. Well, let's just have a look at that. Let me just ask you my second question and then we will go on to probe that with more care. The second point is are you aware of a closure code for Fujitsu for PEAKs which refers to faults which are known in the Horizon system but agreed between Post Office and Fujitsu to stay there? A. I'm not aware of a closure code.
Q. You didn't know about that?
A. No, I don't know about that.”
The following important points arise in respect of this:
Mr Jenkins thought that “the system had behaved as it should”. Given Mrs Van Den Bogerd accepted that this situation could lead either to an SPM, or a customer, either being in pocket or out of pocket, I disagree that an accounting system should work in that way. This is not only an optimistic description, it is in my judgment entirely wrong.
The conclusion was that the scenario might not occur regularly and create “large losses”. The size of the potential losses was plainly relevant to Fujitsu. Given how the Post Office used the Horizon system vis à vis its SPMs, sums that were not “large” to Fujitsu does not mean that smaller losses would not still be appreciable sums to individual SPMs, who would have to pay for them if there were branch shortfalls. There is also no way of discerning what Mr Jenkins meant by “regularly” in terms of the risk of re-occurrence. Both regularity, and whether potential losses were large, are subjective. The Lepton incident showed that the Post Office required the SPM initially to pay the shortfall, which he did.
The Credence data was inadequate to show what had actually happened. Indeed, not only was this data inadequate, it made it look as though the SPM had done something that he or she had not done, and which the system had done.
It was expressly accepted that more was required in terms of the required accuracy for “giving evidence and using the same data for prosecutions”. It was concluded that system created reversals had to be identifiable, which they were not as at the date of the Rose Report and these emails.
A change required a formal Change Request, and that was neither initiated nor acted upon.
Finally, and in my judgment also importantly, any risks that Fujitsu and/or the Post Office “cannot clearly see what has happened on the data available to” them and “this in itself may be misinterpreted when giving evidence and using the same data for prosecutions” is a serious matter. I do not understand how a report containing such a reference to such a serious matter could be mis-summarised by Mrs Van Den Bogerd in her witness statement dealing with Mr Coyne’s analysis of this.
A large number of documents, including operational documents, were put to her, many of which it appeared she had not seen before. For example, she was shown documents in respect of MoneyGram dealing with a software release in January 2016 which showed, to use the IT speak of the document, “multiple instances of system latency” – or, to use more normal language, problems in the operation of the system. These persisted through 2016 until nearly the end of that year. The internal summary of the problem was:
"For the last several months Post Office has experienced a live operational issue with MoneyGram transactions across the branch network . In the event of a transaction timing out at the counter, a system error message is displayed to the user ... and the transaction is aborted. This leaves no record of the transaction at the counter and the transactions and funds may or may not have been committed in the MoneyGram domain. This causes significant issues for Post Office and MoneyGram and for customers."
(emphasis added)
Mrs Van Den Bogerd said that she had not seen the internal document setting all this out until she gave evidence in the witness box. One of Mr Patny’s issues in his evidence was about MoneyGram.
She had also been involved in a meeting on 17 May 2012 with James Arbuthnot MP, Oliver Letwin MP, and others from the Post Office, including the Chairman and the Chief Executive of the Post Office. The documents, including the briefing pack, demonstrate that the MPs were told that the Post Office wished to be “open and transparent”. This was not accurate. The MPs were told that the Horizon system had undergone an upgrade in 2010 and had the full support of the NFSP. In reality, its functionality for SPMs had not been upgraded, and the NFSP had privately expressed its concern to the Post Office about Horizon. There is nothing open and transparent about telling these MPs information to the contrary. James, now Lord, Arbuthnot had been the Minister for Defence Procurement under the Conservative government of Sir
John Major. Oliver, now Sir Oliver, Letwin was Minister of State for Government Policy under the Coalition government, which had come to power on 11 May 2010. These gentlemen had become involved in this matter on behalf of their constituents. They were entitled to expect accurate information from the Post Office. They did not receive it.
A large number of different issues with both Horizon Legacy and Horizon Online were also put to Mrs Van Den Bogerd, only some of which she knew about. She has been closely involved with the defence of Horizon for some years. To be fair to Mrs Van Den Bogerd, she has also been responsible for some changes over time which have made Horizon easier to use, more reliable and (to use the wording of Horizon Issue 4), more robust. Horizon as at the date the experts were looking at it is a more robust system than it was, say, 5 or even 3 years ago. A great deal of this is likely to be due to the efforts of Mrs Van Den Bogerd.
Equally, however, the contents of some of the internal Post Office documents that were put to her were, in my judgment, very damaging to the Post Office’s case on the Horizon Issues. In an “Extracts from Lessons Learned Log” document of 11 November 2015, which was heavily redacted (including the redaction of single words), one entry under “issues identified” was as follows regarding the Post Office:
" Failure to be open and honest when issues arise eg roll out of Horizon, HNGx migration issues/issues affecting few branches not seemingly publicised."
(emphasis added)
This frank expression in a document authored by Mrs Van Den Bogerd herself as recently as 2015 is consistent with the general tenor of the claimants’ case and their criticism of the Post Office. This entry in this contemporaneous internal Post Office document is, in my judgment, inconsistent with the picture which the Post Office continually seeks to portray, namely that it wishes to be open and transparent about issues with Horizon, and is as interested in getting to the bottom of any problems with Horizon as anyone else. It is an internal recognition of a “failure to be open and honest when issues arise.”
There were two periods that she was asked about which clearly showed the problems experienced by Horizon almost from the outset. These were the introduction of Legacy Horizon, in late 1999 and 2000 (the national roll-out); and the migration to Horizon Online in 2010 (including the pilot scheme in 2009). Mrs Van Den Bogerd gave some evidence in her statements about both of these, in terms complimentary of Horizon, although she was far more involved in the latter than the former. At the time of the introduction of Legacy Horizon she was only responsible for a smaller number of branches in her then role.
In PEAK PC0033128 an entry in November 1999 shows that the branch at Dungannon experienced a discrepancy of £43,000. The PEAK states:
“PM - Dugannon PO £43k discrepancy
Outlet has a discrepancy of £43,000 after balancing SUs and doing office snapshot.
Phil Turnock POCL BSM has advised outlet on this week’s balance. Steve Warwick development is investigating why this mis-balance occurred.
Immediate impact of this week’s balance has been addressed but POCL are concerned that the cause is still unknown and this will affect this and other outlets.”
After some investigation the following entry appears:
"I have talked with development ref this problem. It is seen as a one-off. No fault can be found and developments do not expect to be able to find a fault with the evidence available. There is no additional information available as evidence. I suggest this call be placed on monitor for 1 month."
The approach to this discrepancy mirrors so much of the case. A discrepancy occurs; Fujitsu cannot find a fault; and say they do not expect to be able be find one “with the evidence available.”
By February 2000 Fujitsu stated “counter 5 looks suspect” and continued:
“Further examination of the event logs for these two counters indicate that counter 5 looks suspect (C drive nearly full and big gap of no messages).
Calls from PO into HSH for period between 30-Oct and 10-Nov indicate a reboot (counter not specified, but would tie in with counter 5 event log) on Saturday 30-Oct1999.
The evidence in the message store was that messages continued to be written to the message store but that all the 'Payment' transactions which should have been recorded in the rollover trailer messages failed to appear (although others did, such as the Rem OUT and Transfer OUT totals).
This indicates that the problem was not one of running out of Disk space but of failing either to retrieve, or write out, transaction totals for one particular node in the node hierarchy.
Given that there were known problems with corrupted Persistent Object indexes at about this time, it is possible that an update to an EPOSSNodes object failed to be registered correctly at the outlet, causing the node accumulation to fail.
It was decided to prove this out by deleting the 'Payments' node in the node Hierarchy and then running the SU balance, to attempt to identify the root cause of the problem. Call passed to testing to be scheduled.
Update 18th Feb 2000
The test was carried out on 16th February as follows: delete the Payments EPOSSNodes object before producing a SU balance, on a version of the current live system (CI2_2R).
When trying to print the Payments part of the SU balance, the missing node is detected by the system and an error tablet with message "A system error has occurred whilst printing. Please ring the helpdesk. Error at 67640." is generated. So the balance could not be finished.
This type of error trapping error trapping was introduced at the end of last year when resolving AI298 issues and we are investigating if the outlet did not have such error handling when the problem occurred.
Certainly, with the current system, a missing Payments node now would not go undetected.
The problem is currently back with development for further investigation.”
On 17 March 2000 there is another entry that says there has been another incident “of a very similar nature” at another office. This can therefore be seen to be something that was not, as recorded initially, a “one-off”, even if that were acceptable given the discrepancy of £43,000. Mrs Van Den Bogerd agreed that this showed that the underlying information was not checked until February 2000, even though the discrepancy occurred in November 1999. She was asked about a later entry in April 2000 that referred to another occurrence, this time in Appleby for £9,000 approximately. The cause for this was recorded in July 2000 as “data trees failing to build” and that entry states “there have been a number of calls relating to this kind of issue. A fix has been put in at C14 which will prevent this happening”. Mrs Van Den Bogerd accepted that this was an example of an error occurring, and the system itself failing to spot that the error had occurred.
She was shown PEAK PC0027887 which was dated July 1999 and related to “receipts and payments misbalance”. This showed an escalating misbalance that increased, in my judgment dramatically, in steps up to over one million pounds:
“On week 9 receipts and payments misbalance of £1337.05, week 10 misbalance of £24000, week 11 misbalance of £12000, week 12 misbalance of £1051111.48 and week 13 misbalance of £17426.05, she has a difference on week 11 of balance due to post office and balance brought fwd on week 12 of £1082544.32
Overall these weeks net out a difference of £27343.84. she needs business support (reconciliation) to look into this.”
Mrs Van Den Bogerd agreed that £1.05 million was a large amount for any branch and could not be a “real amount of cash” that “the SPM has put in her pocket”. A later entry in the same PEAK of 27 July 1999 was:
“Balance brought forward was multiplied twice due the known software error. The initial balance brought forward for this CAP was £1196622.72. This was multiplied twice to give a total BBF of £2279189.04.
The discrepancy was therefore £1082540.28. This was due a known software error which has no been resolved.”
She did not know if “no resolved” meant “now resolved” or “not resolved”. That PEAK was not closed until 31 August 2000, and was categorised as “Administrative Response”. One of the latest entries in this PEAK, which was not put to Mrs Van Den Bogerd but in my judgment is relevant to Fujitsu’s approach generally, in particular to categorisation of PEAKs (as this had already stated there was a “known software error”), was dated 31 August 2000 (one year after the original issue occurred) and it is most convenient to reproduce here in this judgment. This stated:
“I have now taken over analysing this problem from Steve Warwick who requested additional information on 13/10/1999 17:49:21. I have looked at the information and I am not convinced it can possibly be complete.
For example he said he wanted information from 20th May until 26th May. I opened the spreadsheet mc20may.xls and observed it only had 6 counter 32 transactions in - is this correct?
In any case my team is not used to dealing with data in this form (spreadsheets). We have developed techniques to look at problems using a full message store plus audit and event logs from the failing counter. Even if all the relevant spreadsheets could be obtained I do not think it would be worthwhile me getting to grips with this new method of analysing problems.
I see this is a very old problem (21/07/1999) and there have been many software updates since then.
May I suggest we discontinue investigation of this particular problem but that if a similar problem occurs again you send full message store plus audit and event logs from the failing counter.”
And
“Closing call on basis of insufficient evidence. As this is such an old call I have not contacted the call originator. I suggest that this call remains closed!
INSUFFICIENT EVIDENCE - ADVICE GIVEN”
(block capitals present in original)
To close such a PEAK as “Administrative Response” is, in my judgment, remarkable.
The number of different incidents recorded in the monthly incident review increased in November and December 2000, which is shortly after (for example) the date when Horizon was installed in Mr Bates’ branch office at Craig-y-Don, as set out at [105] of Judgment (No.3). By then the Horizon System had been rolled out nationally. As the ICL Management Summary that was put to Mrs Van Den Bogerd states:
“The most frequently occurring incidents in November were both types of Receipts and Payments Incidents (Migration and Post Migration), with 31 incidents per category. The Migration incidents have remained at the same level, whereby the Post Migration occurrences have increased. This was followed by 17 Transactions Polled by TIP but not by HAPS, these were due to delayed transactions as reported on APSS 2133c. These transactions are added back into normal processing.”
Mrs Van Den Bogerd was not aware of these problems going on at the time; she was a Regional Network Manager and did not have a national role then. No background information was provided to her then about problems that branches other than her own were experiencing, which were being passed back to ICL at the time. She therefore had no knowledge of the type of problem experienced by Mr Bates in June 2001, which was recorded in a Post Office audit document of his branch that was put to her which stated “A correct assessment of cash holdings could not be made because the Horizon system intermittently adds the previous days cash holdings to the daily declaration.”
By 2010 and the introduction of Horizon Online, she did have a national role and broader responsibility. She agreed that Horizon Online did not upgrade functionality, and documents were put to her that showed, as Mr Green put it “a litany of issues” with Horizon. The pilot scheme went sufficiently badly that Fujitsu put it on “red alert” and the high volume pilot was suspended in April 2010. Mrs Van Den Bogerd
had not even mentioned this in her witness statements, and said she did not know this had occurred. The NFSP are recorded as “raised concerns but remain supportive”. All migrations were suspended as from 25 March 2010. There were database errors recorded in a Post Office document at F/614 called “Horizon Online Programme Update” which required an Oracle patch set to be supplied by Oracle, and on one occasion for a limited period 50% of MoneyGram transactions failed.
A Network Functional Report of May 2010 was put to her. Over 5 pages of the 6 pages are redacted. Mrs Van Den Bogerd could not recall if she had seen this document, and said that given the extent of redactions, it would be hard for her to recognise it, an observation with which I agree. She said that she was not aware of there being issues with Horizon over Easter 2010, which the document recorded.
Redactions had been incorrectly applied by the Post Office and/or its solicitors to some of the relevant documents. This can be seen from the following example. I asked leading counsel for the Post Office to review certain redactions and having done so, an unredacted version was disclosed of a document dated 25 June 2014 entitled “Branch Support Programme”. This had been co-authored by Mrs Van Den Bogerd. The purpose of the paper was to “Update the Post Office Executive Committee on the progress of the Branch Support Programme.” Each one of the 6 key performance indicators against which the programme would be tracked had initially been redacted by the Post Office, and following the review by counsel as requested by the court they were unredacted. They were as follows:
“* Reduction of operating costs by £3m per annum
Reduction in net agent debt by £1m
Reduction in subpostmaster suspensions as a result of audit shortages to a level of 60 per year
Reduction of calls into NBSC by 25%
Reduction of audit losses of £10k of over by 50%
Satisfaction with on-line training models of 95%”
Apart from the first bullet point, three of the other five are consistent with an aim by the Post Office to make improvements to reduce the amount of debt incurred by, and suspensions, of SPMs, and reduce the audit losses. The other two are related to training. This is not consistent with a view that the debt/suspensions/audit losses are incurred by carelessness on the part of SPMs or criminal activity. It is also hard to see how it could be justified that these had been redacted originally.
So far as her evidence about Mr Latif was concerned, after Mrs Van Den Bogerd had been taken through the features of his experience with the failed stock transfer and the similarities with accepted bugs in Horizon, namely the printing out of two receipts very close to one another (in his case, 4 seconds apart), and asked if this worried her, she said she “would want to go back and have a look at” it. In my judgment, part of the way through the Horizon Issues trial in 2019 is rather too late for that. I conclude
that there was no real answer available to Mrs Van Den Bogerd to the points put to her concerning Mr Latif’s experience. Had there been, I have no doubt she would have provided that answer in evidence.
Mrs Van Den Bogerd was in the witness box for in excess of one day, the longest period of any of the witnesses of fact for either the claimants or the Post Office. Her cross examination led to a far greater understanding of the Horizon Issues on the part of the court, although her written evidence was, as originally drafted, extraordinarily one-sided. She minimised any reference to problems or issues with Horizon, and reverted to potential user error whenever possible as a potential explanation, an approach which she explained in her written statement as providing “plausible” explanations. Her witness statement also stated, in terms, the exact opposite of what the reality of the situation was, and I have given examples at [221], [223] and [226] above. Witness statements are supposed to be factually accurate, and care must be taken in future rounds of this group litigation that they are drafted in accordance with the rules. Making statements that are the exact opposite of the facts is never helpful, to put it at its mildest. It is also the opposite of what witness statements are supposed to be.
However, during her cross-examination Mrs Van Den Bogerd readily conceded a number of points which were put to her. Her accuracy in her oral evidence was also more satisfactory than it had been during the Common Issues trial. She accepted her written evidence suggesting a causative user error by Mr Tank was not correct, for example. I do not consider that her written evidence had provided plausible explanations. It provided explanations that the Post Office wished to advance, as these explanations would, if accepted, provide a defence to the claimants’ factual case. These explanations were not based on the facts.
There were no evident attempts on this occasion to mislead me in her oral evidence, or avoid uncomfortable points through a claim of ignorance (with the possible exception of the PEAK referred to at [210] above) or of not having seen a document before. When she told me that she had not seen a particular document before, I accept that. However, she also told me that she had been assisted in preparing her evidence by a team of ten people, and the Post Office had devoted “high level resources” to her evidence, as one would expect. It is therefore very surprising that she had not seen many of the important documents that Mr Green put to her in cross-examination. Some, such as the master PEAK on phantom sales, one of the recurring issues on Horizon, would have been the obvious place for anyone, still less a team of ten, to start when considering preparation of evidence for a witness statement. She explained that there was some pressure of time in terms of how long was available to her to prepare the statement, and this was further explained in supplementary reexamination, but I do not accept insufficient time as a valid explanation for her lack of knowledge on such important points. For example, she told me that at the time of preparing her witness statement, she had not even heard of the Callendar Square bug, one of two bugs that the Post Office accepted some time ago had been present in the Horizon system. This is an extraordinary gap in her knowledge. She did not know that there was a KEL dealing with failed recoveries, originally raised by Anne Chambers in Fujitsu as long ago as 2010, which is called KEL acha959T and which was updated most recently in 2017. This described failed recoveries, and seemed on its face to accept that these would recur, and was very close to the experience of both Mr Tank
and also Mrs Burke. I do not see how Mrs Van Den Bogerd (assisted by her team of ten, and with the benefit of the Post Office’s considerable resources) could seek to give accurate evidence in the Horizon Issues trial without referring to this KEL, still less without even knowing about it. I am also somewhat disappointed – putting it at its very best for the Post Office – that a team of ten could have assisted Mrs Van Den Bogerd in preparing a witness statement that was so inaccurate on such important points as I have identified above.
Although my findings on her evidence in the Common Issues trial cannot be ignored, I am of the view that her approach in the Horizon Issues trial to answering questions was far more constructive and aligned to what is expected of any witness giving evidence in court, particularly a senior witness of an organisation such as the Post Office. I do however consider that this litigation, and indeed her cross-examination, is a very expensive way for a senior director at the Post Office to become educated about the myriad issues contained in the documents that were put to her. Either the team of ten people assisting her with her evidence had the aim of producing entirely one-sided evidence in chief, or they were unaware of all the documents relied upon by the claimants. Either alternative is highly regrettable.
Finally, and this is a point in Mrs Van Den Bogerd’s favour, there is no doubt that the Post Office is not as sufficiently close to the detail of what has occurred over the years on Legacy Horizon and Horizon Online, as Fujitsu. As will be seen from my analysis of the Fujitsu evidence of fact, I have certain views about the lack of accuracy on the part of Fujitsu witnesses in their evidence. If that lack of accuracy has also been included in reporting to the Post Office by Fujitsu, then that goes some way to explaining the Post Office’s lack of grasp of so much material that is consistent with the claimants’ case. As at late 2019, the date of this judgment, the Post Office also has the added benefit of the views of both the IT experts in the litigation, their four joint statements, the agreed number of bugs in the bug table (12, if one takes the number accepted by Dr Worden), the total number of bugs Mr Coyne says he has discovered (which so far as Horizon Issue 1 is concerned became 21), and collectively my findings on the Horizon Issues. These are not restricted solely to the number of bugs which I have found to exist which relate to Horizon Issue 1, but also those that relate to Horizon Issue 4. There are multiple bugs, errors and defects in both Legacy Horizon and Horizon Online in its HNG-X form.
The degree to which the evidence of Mrs Van Den Bogerd sits with my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Ms Phillips
Ms Phillips is a Team Leader at the Post Office for Agent Accounting and Santander Banking. She was originally engaged by the Post Office in 1999 through an employment agency, and has worked in a number of roles since then. In her current role, she oversees the processes of recovering losses from SPMs that they have declared in their branches. The department used to have “debt” rather than “accounting” in the name, but its function remained the same.
Ms Philips confirmed what had been in dispute for so long during the Common Issues trial, namely that SPMs had no option but to accept the figures provided to them, even
though they may have been “settled centrally”. This is notwithstanding that her terminology in her witness statement was that SPMs “chose to accept”. This terminology was used throughout the witness statements for the Post Office, and it has already been seen that Mrs Van Den Bogerd accepted TAs had to be accepted, and there was no choice in this respect, as Mr Latif had explained. SPMs clearly did not have a choice, and the wording “chose to accept” is not an accurate description. It obviously states or suggests that SPMs had a choice, when no such choice was available to them. The only choice they had was whether to pay the amount immediately, or settle centrally which means obtaining time to pay. It had been necessary for me to make findings on what “settling centrally” meant in the Common Issues trial, and the Lead Claimants in that trial had been directly challenged on this very point in their cross-examination.
There is no reference in the letters sent to the SPMs in question relating to any ability to dispute the sums which the Post Office treated as debts. Ms Phillips’ evidence went to the way that the Post Office seek to collect sums from SPMs who have settled centrally, even though they may have earlier raised a dispute (or sought to raise a dispute) with the helpline. Sometimes amounts above £5,000 would be checked by reference to the helpline call logs, but the amount was not a set threshold and it depended upon how many sums had been settled centrally as to whether this happened. This checking did not therefore always occur, even above that figure.
Her evidence demonstrated that there was no dispute function within Horizon. Essentially, her task was debt collecting, as the Post Office saw amounts “settled centrally” as debts. Her concern was to obtain payment, or agreements to pay, and if not, then to escalate the matter to a SPM’s contract adviser. Usually, she said, the first or even second letter would lead to payment by an SPM. Neither of those standard letters referred to disputes. Mr Godeseth confirmed that the lack of any dispute button or function on Horizon was, as he put it, “by design”. This shows that the Post Office had been advancing a case, at least for a substantial part of the Common Issues trial, which was directly contrary to the evidence of its own witnesses of fact in the later Horizon Issues trial. I find this difficult to understand or explain. However, Ms Phillips’ role was fairly clear and she explained the process of branch dispute forms which would be used when a SPM raised a dispute directly with her team. This elementary system seems sensible, but it was only introduced in early 2018, and it would lead to a seven day period after which (if the form was not returned) the “block” placed on the amount would be lifted. She estimates that it would be used at least 5 times a month.
Ms Phillips gave useful evidence on the system used by her team, and I found her credible. The evidence did however go more to the Post Office’s accounting functions, and how it pursued sums that it considered to be due from SPMs. The reforms introduced in 2018 in terms of the branch dispute form are part of that process, but are not part of the Horizon system.
She had also provided some evidence to Mr Smith (who was also called as a Post Office witness) in terms of the number of Santander TCs that were issued, which had not been described correctly in his witness statement and which had been misunderstood by both the experts. This was clarified, and also she explained that of the number of TCs issued in one year in her particular area, 2016-2017, which was
3,968, the Post Office could not say how many were disputed because that data was not kept. This would appear to be a fairly elementary piece of information for the Post Office to monitor or record. The failure to do so is surprising.
Ms Phillips’ evidence in her witness statement did not directly impact upon the Horizon Issues to any appreciable degree. Her evidence went to wider issues in the group litigation. Her figures for Santander TCs, which had been given in evidence (for some reason) by Mr Smith and not by her, were considered by the experts.
Mrs Mather
Mrs Tracy Mather is a Team Leader of the Finance Service Centre or FSC. She started working for the Post Office in 1987 and has been in the FSC throughout, becoming a team leader in 1999. She has managed teams in Cheques, Postal Order, Pay-out and MoneyGram. Her witness statement was served in response to Mr Coyne’s 1st Expert Report, and explained why the Post Office used Credence to investigate discrepancies. She explained the difference between Credence, which records the transactional data, and POLSAP, which is what is called a “back end” accounting system. She explained the latter is on a higher level. Credence is used by her team to investigate differences between what a branch says has happened to a transaction through Horizon, and as she put it, what a different source of information might say. Credence was adopted in 2009.
She confirmed that Credence does not record key strokes, it records “transactional data as in sales and non-sales”. Mrs Van Den Bogerd had stated that Credence records actual key strokes by stating in her written evidence that Credence “records all key stroke activity performed in that branch by the user ID, date and time.” This was not correct, and was corrected by Mrs Mather in chief. Her team does not have access to the ARQ data, which has to be obtained from Fujitsu. This is also called audit data. Credence is not used for MoneyGram. I accept her evidence on this point and find that Credence does not record all key stroke activity.
Doubts expressed in the Ernst & Young Management Letter for the year ended 27 March 2011 about the integrity of data in Credence were put to her, but she was unaware of any such doubts having been expressed and did not know about them. She had not been in a team that had requested audit data, and said that would be the fraud or security team who would do this. Passages from internal documents suggesting the audit data came at a high cost to the Post Office were not something of which she seemed to be aware. She started using Credence in 2016.
She gave written evidence about what was called the Rose Report, although she did not actually know about it herself, and she also was unaware of anyone being deterred from making ARQ requests due to the cost to the Post Office (charged by Fujitsu), which was a point Mr Coyne had raised. Given her team did not make ARQ requests, her evidence on this was unsurprising. She was aware that a certain amount of requests were allowed, and above that amount a charge was raised by Fujitsu; however she did not know what the charge was.
There was one passage of her witness statement which went plainly beyond any direct evidence she could usefully give. It was in paragraph 20 and gave vague evidence
about her understanding of some confirmation given by Fujitsu to the Post Office’s solicitors about the contractual limit on ARQ requests, when or if that had been exceeded, a “new commercial term” between Fujitsu and the Post Office about what was agreed and the cost of these. Her conclusion was “any requests above the agreed limit are chargeable but I understand that the terms depend on the details of the requirement.” She was not cross examined about this, and given her answers in cross examination about her understanding of ARQ requests (which was minimal) and her experience of having made such requests (which was never), that is understandable. However, in terms of providing guidance to the parties on the contents of witness statements, to assist the efficient conduct of these proceedings going forwards, witnesses should not have paragraphs of this nature in their witness statements about matters in which they have had no involvement at all. Mrs Mather – and there is no criticism of her in this – has simply never been involved either in making ARQ requests, or in the contractual arrangements between Fujitsu and the Post Office, or the “new commercial term” between those two entities in relation to the extra data required for the litigation. She stated that she had “no idea” about the charges that would be raised.
I found Mrs Mather to be a credible and helpful witness. Her evidence was very useful regarding Credence, and otherwise of less than central relevance to the Horizon Issues. The degree to which the evidence of fact supports my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Smith
Mr Paul Smith is the Operations Support Manager at the FSC and has been since November 2016, having started at the Post Office in 1996. Originally he started in pensions and then moved to human resources and payroll, until in 1999 he moved to the Network Business Support Centre helpline, or NBSC. In 2000 he moved to work as a Problem Manager, dealing with problems that came through the helpline. He led the team that updated the knowledge base for the helpline, and was Lottery Manager in 2004. After performing different roles he became a Change Analyst in 2011, and then the Santander manager. He returned to Lottery within the Post Office in 2013 and worked there as a Team Leader. His existing role has a number of different activities including handling specific product based problems. His evidence dealt with volume of TCs and a suggestion made in Mr Coyne’s report that TCs for a branch in Potters Bar were initially made in error.
One passage in his statement is of particular importance in resolving the Horizon
Issues. It states:
“TCs are issued by FSC. I understand from Post Office’s solicitors that the processes by which FSC determines whether a TC is required is outside the scope of the Horizon Issues trial.”
The reason that this is important is as follows. Dr Worden relies upon TCs as one of his “countermeasures” that goes to the robustness of the Horizon system. Mr Coyne does not, and the claimants submit that the whole process of TCs is entirely outside the scope of the Horizon system. This evidence from one of the Post Office’s own witnesses makes it clear that the claimants’ point of view is shared (or at least was
when Mr Smith’s statement was prepared) by the Post Office’s solicitors, and the decisions regarding whether TCs are issued are outside the scope of the Horizon Issues. This evidence by Mr Smith supports the claimants’ position on this, and also supports the approach of the expert evidence of Mr Coyne, and is contrary to the position adopted by Dr Worden.
A case management system was introduced by the Post Office in September 2018 to record each individual challenge to a TC. This is called the Dynamics System, and is still in what is called “roll out”. Individual challenges to TCs were not recorded prior to that. Further information about how this system was working was obtained in cross-examination, and leads me to the conclusion that the way that TCs are recorded now is far improved from what it was before the Dynamics System was introduced. That does not however take one very far. Mr Smith knew of reports that were available from that system but he had not produced any. Those reports would be the best evidence of the type of points Mr Smith was seeking to make for the most recent periods at least, but given he had not used them, it is not possible to speculate on what they might show.
Mr Smith had obtained almost all the data contained in his statement from others, and some was plainly incorrect, and other data so vague as to be wholly unhelpful. For example, for Santander, originally he had provided a table with a column stating that 2,890 disputes were received from branches in the 2016/2017 financial year. In fact, that column should have been disputes received by Santander from the Post Office. This was corrected in his second witness statement, but Mr Smith was unnecessarily combative, in my judgment, about the need for the correction, maintaining that “the facts and figures were correct” in the first statement when they plainly were not. The figures were correct as figures but the descriptions of what the figures were, were plainly wrong.
Even after the correction, the various figures extracted by him from information provided by others was not particularly useful. For example, the figures for the numbers of TCs disputed, and for compensating TCs, for BOI Retracts and the Lottery, were both estimated at 1,500 and 500, for both line entries. This estimate was plainly of the most vague type, and even Mr Smith could not comment on the reliability of either of those estimated figures for either line entry, although he maintained the figure for issued TCs would be reliable. This evidence from him, was of the most general type, and was not of great, or indeed any, assistance. For example, in respect of MoneyGram his evidence was that one or two disputes a month were received but they were rarely if ever accepted. Given he had no idea at all of the overall total, this evidence does not assist me in resolving the Horizon Issues one way or the other.
His explanation for the TCs at Potters Bar was that a cheque that was deposited for £90,000 was mis-keyed, and entered for £900,000. Accordingly TCs were required to correct this at the branch, and the need for two was because the payment was made by cheque.
I found Mr Smith’s evidence, with three exceptions, to be of no assistance whatsoever. The two exceptions are his evidence I have summarised in [269] which is relevant to the scope of the Horizon Issues and Dr Worden’s countermeasures, and his explanation about Potters Bar and the two TCs. I accept his explanation about the Potters Bar TCs. Both of those exceptions are relevant to the Horizon Issues. The third exception is his evidence about the Dynamics System. This goes to the robustness of Horizon as of 2019 and is therefore relevant. However, the absence of such a system also goes to robustness prior to its introduction in 2018, and is also therefore relevant. However, his wider evidence is of limited relevance at best. His knowledge of the figures contained in his statement, and the vagueness of his evidence generally, were such that the utility of the exercise which he presented in his statement is de minimis. In any event, the substance of that evidence went to an attempt to demonstrate that some TCs are challenged by branches, and fewer challenges are upheld than are made. The figures vary inevitably from year to year, and this case is not about general statistical trends in any event. In some areas of business TCs have fallen to a very low level, for example the DVLA, which Mr Smith explained was only 4 in the last year, none of which have been upheld. However, the records that were put to him show that there were 2,717 in the period 2010/2011. All this evidence does is show that far fewer TCs are now issued by the Post Office in relation to the DVLA business, which as Mr Smith explained, has fallen considerably in any event. None of these points are of particular relevance to resolving the Horizon Issues. The experts are agreed that Horizon as it is in mid-2019 is more robust than it was in the past. One would expect, therefore, fewer TCs to be issued now.
Mr Johnson
Mr David Johnson is a Training and Audit Advisor at the Post Office. He started his career at the Post Office in 1984 as a counter clerk in a Crown Branch, and worked his way up until he was the Branch Manager at the Barry Crown Office. He was in that post when Horizon Online was introduced in 2010. He moved to his current role in 2012, and his role used to be called Field Support Adviser. Two thirds of his time is spent training SPMs; the other third is spent auditing them. He had also been involved in training some of the legal representatives for both sides involved in the group litigation, and the experts, on the Horizon system. As well as his first witness statement, he had provided a supplementary one dealing with some points in Mr Henderson’s witness statement which was served for the claimants.
He was cross-examined about screen layout and design, the risk of mis-keying (which did not seem to concern him, although there were internal Post Office documents showing it was a known problem) and also the reports and functions available to SPMs to investigate shortfalls and discrepancies, rather than those which the Post Office itself used such as Credence. He agreed that the ways available to SPMs to investigate in branch were “not the most user-friendly way of investigating”.
He agreed with a statement in a 2016 internal Post Office document entitled “Network Development Programme Operation Simplification” that stated:
"There are a number of branch operations processes, especially around branch accounting and reconciliation, which operate using legacy processes. They are unnecessarily complex and detract Post Office branch resources from serving customers. Stock Unit Management and accountability is very poorly controlled and is operated on very complex business rules. The lack of accountability and visibility of cash and stock transfers between Stock Units can lead to errors, rework and provides opportunities for fraud."
(emphasis added)
He also stated that the following entry was “not completely unreasonable” although he said that his experience of suspense accounts in branches was that they worked and did what they were supposed to do:
"Similarly, Suspense Accounting is based upon legacy cash accounting practices, illdefined and out of date processes. Inefficiencies lead to poor utilisation of resources both in Post Office branches and Support Services."
Mr Johnson is a highly experienced person, as one would expect given that the Post Office chose him to train the legal teams and experts on Horizon. All his experience, so far as using Horizon is concerned, is in Crown Offices, which are not the same as branch Post Offices, in that the Post Office operates Crown Offices itself. This means that a SPM in a branch Post Office is personally responsible for any losses in that branch; whereas a Crown Office manager is an employee of the Post Office. This therefore gives him a somewhat different angle of approach to the use of Horizon, but I accept him as an accurate and helpful witness.
His evidence was very useful in terms of how Horizon works. The Horizon Issues are not about the actual design of the Horizon terminals in branch, and whether (say) this could be improved by having larger buttons or the screen layout adapted differently. In some of Mr Johnson’s evidence, the claimants’ counsel strayed into the areas of potential improvements to functionality in branch and design. These are not part of the Horizon Issues. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Dunks
Mr Andy Dunks is at IT Security Analyst at Fujitsu Services Ltd, and was the first Fujitsu witness to be called for the Post Office. He has been employed at Fujitsu on the Post Office account since 2002. He has working knowledge of the Horizon System. His evidence went to audit data extraction and the integrity of the data during that process. His evidence related to the process as it was when he made his statement, although given he had been involved for some 17 years his experience also related to Legacy Horizon and his evidence about data extraction covered that period too. As emerged in his cross-examination, he had also given a witness statement in the criminal trial of Mrs Seema Misra.
He agreed with the definition of data integrity put to him, which is that data integrity is the overall completeness, accuracy and consistency of that data which you can measure by comparing between sources. He said the 12 controls that he had listed in his statement explaining these he had compiled as a list from his recollection, and that he had not worked off a document to come up with the list in his statement.
Mr Dunks gave every indication, in the first part of his cross-examination, of being helpful and frank, and explained the process of data extraction and answered questions put to him openly. However, this encouraging start came to an end when he was asked about two curiously worded paragraphs in his statement. These stated as follows:
“8. There is no reason to believe that the information in this statement is inaccurate because of the improper use of the system. To the best of my knowledge and belief at all material times the system was operating properly, or if not, any respect in which it was not operating properly, or was out of operation was not such as to effect the information held within it.
9. Any records to which I refer in my statement form part of the records relating to the business of Fujitsu Services Limited. These were compiled during the ordinary course of business from information supplied by persons who have, or may reasonably be supposed to have, personal knowledge of the matter dealt with in the information supplied, but are unlikely to have any recollection of the information or cannot be traced. As part of my duties, I have access to these records.”
Before I come to his evidence about this, it is obvious that the wording of paragraph 8 is almost that of a legal disclaimer (or a legally worded claim of accuracy, to be more precise), rather than a witness’ actual evidence. It would be very curious for a witness of fact to decide to put such a formally (and rather clumsily) worded paragraph in their witness statement. However, he was asked about these passages. He was asked what he meant by “the system”. The transcript records the following:
“Q: What is the "system" there? Is that the system of the process of extracting audit data, or is it something else?
(Pause).
Good question. There's no -- I'm not sure what I was meaning by that, "There is no reason to believe ..."”
He was asked if, in relation to the part which states “the system was working properly or, if not, any respect in which it was not operating properly, or was out of operation” the following:
“Q: Again, pausing there, it is slightly confusing. Are you aware or not aware of any instances where that system was not operating properly?
No, not really, no.”
He was asked if that part of the statement had been written for him by someone else, and he said he could not remember. He was also asked if it was a “Fujitsu party line” to put this in statements when it comes to extracting data and he said no, and that he was not aware of a Fujitsu party line. He accepted that gaps and duplicates in audit data were problematic, and he explained that they had to be investigated by audit support. It was clear to me that Mr Dunks considered audit data that had gaps and duplicates should not, in that form, be used. He said he would contact audit support if these occurred.
However, some of Mr Dunks’ evidence, in particular what he had to say about paragraph 8 of his own statement, proved in my judgment to be somewhat misleading.
He was shown a witness statement from Mr Gareth Jenkins from the Misra criminal trial in 2010. He said he had not seen this statement before. The statement put to him is some 8 years older, approximately, than his own statement. Mr Jenkins’ statement stated at the end:
“There is no reason to believe that the information in this statement is inaccurate because of the improper use of the computer. To the best of my knowledge and belief at all material times the computer was operating properly, or if not, any respect in which it was not operating properly, or was out of operation was not such as to effect the information held on it. I hold a responsible position in relation to the working of the computer.”
The similarities between this passage, and the one in paragraph 8 of Mr Dunks’ witness statement almost 9 years later, are somewhat striking. Given he had not seen the statement before, there was no obvious explanation. His answer was as follows:
“Q. And you're not sure why it would largely replicate your paragraph 8?
No, I mean, we do have a standard witness statement that we produce for ARQs. When we supply ARQs we are sometimes asked for a witness statement to go through the process and verify as far as I'm aware that the data I supplied is accurate. Now, we use that quite a lot and it may actually be in that statement.
Q. I see. So when I asked you earlier whether it was something of a Fujitsu party line you said you didn't think it was, but actually it looks as though it is on the basis of what you have just told me and on this document as well.
A. It could be. It may be part of our standard witness statement that we supply.”
Mr Jenkins had given evidence in his witness statement from 2010 about duplicate entries in the audit data that had had to be removed. Mr Dunks was then taken to the witness statement of another Fujitsu witness in the Misra trial, called Penelope Thomas. She had given evidence about data extraction and listed a number of controls, which were remarkably similar to the ones that Mr Dunks gave evidence about (although Ms Thomas had omitted one). This is hardly likely to be a coincidence, if Mr Dunks really had compiled his list of controls from memory or experience. She also had the identical passage in her witness statement as Mr Dunks’ paragraph 8.
Mr Dunks also said in respect of this, which in my judgment painted a rather different picture to that portrayed in his witness statement, the following:
“A. The statement again is to verify the integrity of the data once extracted and given. We don't control what's in the data. Our process is about extracting it and securely passing it over to the Post Office. It's not our concern of what's in the data. Q. It's not your concern what's in the data?
No, it's what we're -- we're process-driven to extract certain types of data for certain requests.
Q. And so whether or not that might match another record or not, or replicate or duplicate or have gaps, that's not part of your remit, that's not really part of your concern?
A. No, it's not.
Q. I see. So reading that and reading the paragraph 8 as it was in your witness statement, it doesn't give much comfort to somebody that's then trying to rely on ARQ data as being a gold standard to compare and investigate anomalies, does it? A. Possibly not.”
The reference to “gold standard” was the expression that had been used by Dr Worden in his report to describe the quality of the audit data that was held by Fujitsu. It should be a very great concern to anyone tasked with extracting audit data, should that data prove to have gaps and duplicates in it. This is because gaps and duplicates in the data affects the accuracy of that data. I consider that such a cavalier approach to whether audit data has gaps and duplicates, as evidenced by Mr Dunks’ saying that such things were not part of his concern, to be entirely contradictory to a statement verifying the accuracy of such audit data.
My findings in relation to this are as follows:
Mr Dunks expressly sought to mislead me by stating that there was no “Fujitsu party line” when it came to the contents of drafting witness statements about audit records for legal proceedings. There plainly is; it was used in the Fujitsu statements in 2010 and it was used by him in his statement for the Horizon Issues trial.
The passage included in paragraph 8 of his statement is plainly a standard form of words, and reads as though it is a legally drafted assertion of accuracy. The witness’ assertion that the system was at all material times working properly, or if it wasn’t, this did not affect accuracy, does not sit consistently with Mr Dunk’s acceptance of gaps and duplicates in audit data.
That Mr Dunks did not really know very much about paragraph 8 of his own witness statement was confirmed by the way he was puzzled when he was asked about it.
However, Mr Dunks’ acceptance of standard wording provided by others (probably within Fujitsu) is less important than the fact that he originally tried to mislead the court.
I do not accept that he identified the 12 controls in his statement from his recollection. He must have been working off another document, which he was not prepared to identify. I draw this conclusion because of the marked similarity with the controls included in Mrs Thomas’ statement from 2010; the wording of the controls themselves; and the fact he was prepared to mislead me about the paragraph 8 point.
In re-examination he asserted that the statements in his paragraph 8 were true, and that he would not have signed his statement had this not been the case. I find that he cannot have known whether those statements were true or not. There is no evidence of any steps he took to check whether or not they were true or not. I find they were simply standard sentences supplied to him, and I find that he signed his statement without any independent knowledge of whether they were true or not.
I find that passage in his witness statement simply to be meaningless assertion.
He said that he was unaware of Post Office internal concerns that there was “a lack of system audit trail” in a document of 22 October 2016. I find his lack of knowledge about this surprising, given his role.
I found Mr Dunks very unsatisfactory as a witness. He was both plainly aware of the Fujitsu “party line”, or corporate position, regarding the words asserting accuracy of audit data, and he was very anxious to keep to it, whilst initially denying that there was one. He sought to mislead me about both his paragraph 8 wording, and the way he had compiled his list of controls. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Godeseth
Mr Torstein Godeseth is employed by Fujitsu and is the Chief Architect on the Post Office account. He had provided three witness statements, as had Mr Parker. After graduating from Oxford with a degree in physics in 1974, he worked for Rolls Royce as a combustion engineer and then joined the Royal Navy in 1977 as an Instructor Officer. Whilst in the Royal Navy, he started his career in IT working in systems programming. It will be remembered that in the 1970s computing and IT was a relatively young field, and Mr Godeseth has therefore been involved in that industry since its relative infancy. He left the Royal Navy in 1981 and worked in systems programming and technical support for the IT at Forward Trust Ltd. He joined the Post Office IT department in November 1987. Initially this was to work on a project to introduce technology into Post Office branches.
He was working with the Post Office as a technical advisor when the Post Office and Benefits Agency procured the Horizon system originally. He remained involved in different roles, including being outsourced from the Royal Mail IT department to Xansa in 2003, and contracted to the Post Office to act as the technical adviser interfacing with Fujitsu amongst others. Although an independent contractor, he worked between 2005 and 2010 with the Post Office as the technical advisor on IT projects including the change from Legacy Horizon to Horizon Online. He joined Fujitsu as a full employee in 2010. He described himself in cross-examination as the designer of Horizon.
He had a greater knowledge of Horizon Online than of Legacy Horizon, although given his involvement in Horizon from its early days he had significant knowledge about both. Horizon was initially, in the late 1990s, called the Pathway project and was a joint effort between the Post Office and the Benefits Agency, including a way to computerise the payment of benefits to those entitled to them, through payments to benefits claimants which were to be made through the Post Office by means of a swipe card method. This project ran from approximately 1996 to 1999 and was to be provided by ICL (which Fujitsu partly owned and later fully acquired). At some point during the development of this project the Benefits Agency withdrew, and what had been the Pathway project, essentially a tri-partite venture (the parties being ICL, the Post Office and the Benefits Agency) became Horizon, now called Legacy Horizon.
Although this litigation is not about the way that the Horizon system was designed, it is about its functionality and robustness in use, Mr Godeseth said that it was a specific Post Office decision not to have any dispute button/function for SPMs built into the Horizon system.
He said:
“I think the basic argument was that disputes -- we wanted the flow of data through the system as quickly as possible because that keeps our books tidy and it was an inference that there was always the -- you had to press a button to take things through, but then you would pick up the phone to NBSC and say that wasn't right.” (emphasis added)
I accept this factual evidence. It is, however, directly contrary to a major part of the Post Office’s case in the Common Issues trial, namely that by reason of an SPM accepting items at the end of a branch trading period, that gave the branch trading statement the effect in law of being what is called a “settled account”. I rejected that argument by the Post Office then, unaware when I did so of later evidence of fact that would emerge of the specific decision taken at the time by the Post Office that expressly did not include any feature within the Horizon system for a SPM to dispute items with which they disagreed. My finding in the Common Issues trial is consistent with Mr Godeseth’s evidence on this point.
I deal with Mr Godeseth’s evidence within this judgment in more detail than any of the other witnesses of fact from either the claimants or the Post Office. This is because his evidence was considerably more detailed, and of more direct assistance in resolving the Horizon Issues, than any other witness of fact. He has been involved in Horizon in both its iterations for 20 years and has a vast amount of knowledge of its operation. I found the majority of his evidence reliable. I provide my conclusions regarding him as a witness at the end of this section below at [453] to [463].
In Legacy Horizon, a messaging system was responsible for storing all data in Post Office branches and replicating it to data centres. This was called Riposte. Mr Godeseth decribed himself as being “on the other side of the fence” during this period and he had consulted Mr Jenkins for the section of his evidence dealing with when Riposte was in use. However, Mr Godeseth did describe himself as “having a pretty good knowledge of Riposte”.
When it was brought in, Horizon Online was aimed, not at improving functionality of Horizon, but at reducing cost, and re-used (the term put to Mr Godeseth was “recycled”) many application components of Legacy Horizon. The data was no longer to be held at the branches (on the counters), it was to be held centrally in the branch database, or BDRB. Accordingly, harvesters were required to extract transactions from the BDRB. The branch database receives information from different sources. Data in Horizon Online is obtained from different sources.
As an example with the lottery, if a person buys some lottery tickets in their branch Post Office, the information about a particular transaction starts at the lottery terminal in a branch; goes directly from there to Camelot; from Camelot it goes to Credence; and then from Credence it goes to the branch database, which is effectively that information arriving in Horizon. It is then transmitted to the terminals (in the branch)
and the SPM would see a series of TAs on the terminal in the branch, which that SPM would have to accept; these TAs would relate to the lottery tickets sold at the beginning of this short history. The transaction of the purchase of the lottery tickets by this route enters the branch accounts when the TAs are accepted, and the data is captured by the audit system. This is how lottery transactions work after what was called the Ping fix. Prior to that it was somewhat less streamlined, although given the process I have described is the post-Ping fix streamlined version, it is obvious that pre-Ping fix, the route for the data was even more convoluted.
Other data for other transactions for other products goes into the branch accounts from what is called the basket. That basket is compiled by, as an example, a customer at the counter purchasing various Post Office and other client products; the SPM or assistant serving that customer and pressing an icon on the screen (for example for stamps, and other purchases); or for other products, from a PIN pad, scanning a barcode or weigh scales. The cost of a book of stamps is taken from the reference data table and the terminal shows the SPM or assistant how much money the total number of transactions are worth and what the cost to the customer is to be. It is basically the shopping basket which is put together by the SPM (or assistant) serving the customer and adding, by means of pressing buttons on the screen, different items to the basket as the customer goes through the items or products they wish to buy. At the end of this hypothetical transaction, if the cost of the basket is (say) £21.98, the customer will pay that amount (say by cash) which will be taken by the SPM and put into the cash amount held in the branch.
In Legacy Horizon, the data was held in the branch on the counters in a message store. The message system called Riposte was how the replicating of the data to data centres occurred. The data was then stored on the correspondence server message store. This was explained by Mr Godeseth in the following way.
All counter data was held in a bespoke message store (which was part of the Riposte product supplied by another company called Escher Inc.). This data was replicated within each branch to all counter positions and from each branch to the data centres where it was held in the correspondence server message stores. Similarly, any data inserted into the message store at the data centre (for example reference data or authorisations for banking transactions) would be replicated back to the branch counters. Selected data was then extracted from the correspondence servers to update Post Office's back end systems.
Users with sufficient access permissions could inject additional messages (i.e. data) at the correspondence server. Any additional messages injected at the correspondence server by users with sufficient access permissions included information including the identity of the user. That information would not be visible in the standard audit extracts, but it would be visible in a detailed examination of the raw audit data. An SPM did not have access to audit extracts.
There was some disagreement between Mr Godeseth and internal Post Office documents about whether the audit data from Riposte was held in Riposte attributable language (his view) or whether an audit conversion tool was required to convert existing audit data from Riposte to another readable/searchable format. This ultimately may not matter, and it may be that the author of the internal document suggesting otherwise did not understand that Riposte attributable language was a readable/searchable format. This difference in view does not matter for the purpose of resolving the Horizon Issues. The Riposte software was provided by another entity called Escher. There was no evidence about the contractual relationship between Fujitsu and Escher, and that does not matter for the purpose of resolving the Horizon Issues.
Mr Godeseth had studied a number of problems in Horizon, and knew there were problems with Riposte, having seen numerous PEAKs referring to this. He had also been responsible for some testing (for example potential failures of controls, such as a basket not balancing to zero). He was not aware of items with the same Journal Sequence Number (or JSN) having been committed to the BRDB, and agreed that this should not be possible. This is because the whole way the BRDB works (or one of the principles of it, at least) is that each item committed there should be given a unique JSN. It should not be possible for two different items to be given the same JSN.
Mr Godeseth explained that “the legacy version of Horizon was far more susceptible to communication glitches” and agreed that there were two improvements made when Horizon Online was introduced in that respect. These were that the susceptibility to communication glitches was reduced; and also the quality of the communications infrastructure was improved. It should also be added that because the storage of data was moved to somewhere not in branch (somewhere which had not existed in Legacy Horizon, namely the BRDB), and because Horizon Online was an online system, the communications infrastructure had to change substantially in any event.
One internal document from as long ago as 2008 stated: “In addition, in common with many elderly systems that have been subjected to a succession of major changes, it [ie Legacy Horizon] has become increasingly difficult to make those changes, and expensive to operate." This was put to Mr Godeseth:
“Q: Now, that's a fair description of how the system originally was perhaps designed jointly with the DSS at the beginning and launched and then over the years, between then and 2008, there have been lots of sort of bolt-ons and additional things that have been changed on the system, haven't there?
Yes, I think that's fair. The major one probably in my mind would be banking.”
The reference to the DSS meant the Benefits Agency. In other places this was referred to as DWP (or Department of Work and Pensions) but all three references – DSS, Benefits Agency, DWP – meant the department involved in paying welfare benefits. Different documents and references would use different acronyms to refer to the same entity.
He explained that a small group of Fujitsu users from the SSC (about 30 users) had the ability to inject additional transactions into a branch's accounts in Horizon Online, using a designed piece of functionality called a Balancing Transaction. This was approximately the whole number of personnel who worked in SSC.
He had given information in his first witness statement, namely that in Legacy Horizon, any transactions injected by SSC would have used the computer server address as the counter position which would be a number greater than 32, so it would
be clear that a transaction had been injected in this way by someone other than the SPM. This is important because it would be consistent with the case originally advanced by the Post Office that any such injections would be entirely visible as having been done externally (ie, not within the branch) due to the counter number used.
However, this important information was simply incorrect, and was corrected both by Mr Godeseth and Mr Parker in subsequent statements before they were called, and as a direct result of Mr Roll’s evidence. The information that was incorrect, and therefore had to be corrected, had come directly from Mr Jenkins. This shows that Mr Jenkins did, in at least one very important respect, give Mr Godeseth directly incorrect information about the visibility of injected transactions, which not only could have an effect on branch accounts, and whether this would show (or rather, not show) that the impact on those accounts had come from injections made outside the branch. Mr Godeseth only found out the true position when Mr Parker was preparing his subsequent witness statement in the weeks prior to the commencement of the Horizon Issues trial, in other words in 2019. He had not known that before. His explanation about this was as follows.
“Q. You were finding out a detail that you didn't know before in quite a controversial area, weren't you?
It was clearly an area that was going to be of interest because of the fact that we were inserting transactions into Riposte. It was an operational necessity and it was done in a controlled way. I had believed that the way that transactions were being injected would give them a counter position greater than 32 because the correspondence servers basically had nodes or addresses which were above 32, there was a special address for the gateway server, there was a special address for the extra disc in a single position branch and I had basically expected messages to be introduced using a different counter position and having read a whole number of PEAKs, I can quite clearly see that the standard practice in Fujitsu was to label something which was being inserted into Riposte so as to make it as clear as possible that it was not being done -- it was being done as something out of the ordinary, it was being inserted because of a problem.
So we had techniques for doing that. You could put in an attribute because this wouldn't be visible to a subpostmaster, I fully understand that, but it would be visible in the audit trail, when you ever come back to pull out the audit trail, you could put in an attribute to say "This was done under PEAK 75". Subpostmasters would never see that. They would not see it in their account in their branches and I'm fully aware that that is the case. It was a better audit than Mr Roll was alluding to when he said that it was left in a PINICL, because that would have been an audit which is separate from the actual data that we would be looking at should we ever need to pull stuff out of the audit trail and the intention was always to make it as clear as possible that this had been done under exceptional circumstances.
The techniques used to make it as visible to the subpostmaster as possible would be to put in references which referred to a counter that didn't exist in the branch, such as -- I saw a technique described in a number of cases which said put in a -- you know, if you are correcting something for counter 1, call it counter 11; if you're correcting something for counter 2, call it counter 12. These things would have been visible to a subpostmaster and the reason that you had to do it that way was to make sure that these transactions also got picked up and dealt with because these were legitimate counter numbers.
If I start to put in data with a number which is not a legitimate counter number then it's going to be ignored by systems further down the track. Q. So which were legitimate counter numbers?
A. Up to 32.”
(emphasis added)
He also said in relation to Riposte that “Riposte was responsible for actually wrapping the message that we were looking to insert at the counter, and in doing that, Riposte will tell you the counter ID, or technically it was a stream, it would pick up the user ID, it would pick up the time, so this was effectively the envelope which wrapped the payload that we were looking to inject. If there was no user logged on at the counter then Riposte would introduce a blank user ID and that would be picked up in later processing.” This therefore meant that if, say, a SPM (or an assistant) was logged on at the time that the message was inserted at the counter, it would appear next to that SPM’s user ID (or that of the assistant). It would look as though the SPM had been responsible.
This was made crystal clear in different passages of cross-examination:
“Q: You realised for the first time that it was possible to inject or insert a transaction with a counter position less than 32 when Mr Parker was preparing his second statement?
Correct.
Q. And you knew that that was a contentious issue in this litigation, yes? A. Yes.”
And further:
“Q. Now, it would be possible, would it not, to use a counter number of 1, or 2, or 3? A. It would.
Q. And if that counter number was a counter number actually in use by the SPM, it would appear to the SPM, from the records they could see, that it was a transaction which had been done in their branch, by them or their assistants?
Yes.”
(emphasis added)
I consider this to be extremely important evidence, both in resolving the Horizon Issues, and indeed in the whole group litigation. Its import is obvious. It means that Fujitsu could remotely insert a transaction into the accounts of a branch using a counter number which was the same as a counter number actually in use by the SPM (or an assistant). This would appear to the SPM from the records that they could see (and anyone else looking at those records) as though the inserted transaction had been performed in the branch itself. This information was only disclosed by Fujitsu (and therefore the Post Office) in this group litigation in January and February 2019. Even Mr Godeseth, a very senior person in Fujitsu so far as Horizon is concerned, said that he did not know this before.
Mr Godeseth was also shown a Fujitsu document which was "Fujitsu restricted" and "Copyright Fujitsu Limited 2017". The title of the document was "Post Office Account – Customer service problem management procedure" and its purpose is described as "To describe and document the customer service problem management process." He was not sure he had seen the document before, and he was certainly not familiar with its contents. Both experts had seen the document and proceeded on the basis that what it recorded had in fact been adopted. “The customer” for the purpose of the document was the Post Office.
He sensibly accepted that to have a robust system it was important to make informed assessments of where problems lie, based on the relevant information that was available, and also that it was important to capture and track that information in a way that could be readily analysed. Indeed, Mr Godeseth’s role included awareness that changes being made to the system were implemented without prejudicing the continued operation of the system. The document itself expressly defined “problem” as the unknown underlying root cause of one or more incidents. That, to me, seems a sensible definition. An unknown root cause of even one incident is a problem in a system such as this one. This is accepted by the document’s definition of “problem”.
The document identified the metrics that were to be used within Fujitsu to measure this. Mr Godeseth agreed that they were professional and sensible metrics. They are stated as follows:
“The following metrics, to be reported monthly, will be used to measure effectiveness of the process and drive performance of the process and overall service in general:
resolved
-active actions and trend analysis
arising from proactive actions
CP) activities
services/major releases.”
However, when the claimants had sought to obtain reports that would be expected to exist, which included or demonstrated the reporting against these bullet points which
Fujitsu’s own document stated were required, and which were supposed to be prepared every month, Fujitsu stated (through the Post Office’s solicitors) that “Fujitsu believes that it does not record problems in such a way that would allow this to be determined without retrospectively carrying out detailed analyses” and that it would require “a disproportionate effort and cost” to provide these. Mr Godeseth also gave written evidence that the reporting system identified in Fujitsu’s own document had not been implemented and that the records did not even exist. The passage in his statement said:
"I have spoken to my colleague Steve Bansal, Fujitsu's senior service delivery manager, who has informed me that the Post Office account customer service problem management procedure document was introduced by Saheed Salawu, Fujitsu's former Horizon lead service delivery manager and that Saheed Salawu left the Fujitsu Post Office account in around February 2013, before the new procedure had been implemented. I understand from Steve that Saheed Salawu's replacement did not wish to implement the changes and therefore the records referred to by Mr Coyne in paragraphs 5.157 to 5.159 of his report do not exist, as we continued to follow the previous existing reporting methodology.”
(emphasis added)
Mr Godeseth did not know who Saheed Salawu’s replacement was, although he thought it might be a Mr Wicks. He did not know what the “previous existing reporting methodology” to which he referred in his own witness statement in fact was. He could not explain why the report which was said to have been abandoned, or at least not pursued after February 2013 when Saheed Salawu had left Fujitsu, was issued for approval in July 2014, over one year later. Indeed, he could not really explain why the records sought by the claimants did not exist at all. He thought the reporting might be done at meetings with ATOS, but he could not remember going to one. In fact, his evidence on this was so very vague as to be somewhat alarming, given his role within Fujitsu. He did not know if there were any records of the particular type apart from between the years 2014 to 2017, and he only knew that because Mr Bansal had told him.
In another document, he was shown an entry which recorded that First Rate, a joint venture between the Post Office and the Bank of Ireland dealing with foreign exchange (also called bureau de change or bureau services), had identified an anomaly over the way that Horizon reversed transactions that were recorded and polled through to them. This was fixed in something called counter release 9, but although he said this happened “on his watch” this had not been brought to his attention.
In the 2015 Problem Management Problem Review, the Dallmellington bug (which is an acknowledged and agreed software bug) was referred to, but without identifying how many branches were affected, for how long they were affected, or the amounts by which their branch accounts were impacted. He accepted that it was fair to describe the report as “not a particularly rigorous or robust treatment of recording the problem, its extent and duration and effect.”
The following conclusions can be drawn from this evidence by Mr Godeseth. Fujitsu should, had it been interested in providing a robust system to the Post Office, been collating and reporting problems, which should obviously have included software
bugs, errors and defects, against the metrics contained in the Customer service problem management procedure document, or very similar ones. On the Fujitsu evidence before the court, it did not do so. The discrepancies in the dates of distribution and issue of the document entitled "Post Office Account – Customer service problem management procedure" and its "Copyright Fujitsu Limited 2017" do not tally with the account in Mr Godeseth’s witness statement that this process was abandoned or not pursued when Mr Salawu left Fujitsu in 2013. The dates just do not match up. The document was issued for approval one year after that gentleman left Fujitsu. Even the evidence that it was not pursued or implemented after Mr Salawu left was second hand from Mr Bansal, the source of Mr Godeseth’s knowledge.
Had Fujitsu done what its own document, copyrighted in 2017, states was required, the experts in this litigation could, and probably would, have used those reports. Those reports would have included comprehensive records of precisely the sort of matters both experts had to investigate in order to provide their evidence to the court on the Horizon Issues. It is, in my judgment, a stark deficiency in the robustness both of Legacy Horizon and Horizon Online that such records were not kept by Fujitsu. This deficiency must inevitably also have an adverse impact upon the quality of information that the Post Office itself would have had.
In the Post Office’s closing submissions, an explanation was given in submission in relation to the explanation given by Mr Godeseth in his witness statement which I have set out in [325] above about why the records did not exist, that explanation being that the procedure was not implemented. The Post Office submitted that it was only section 1.4 of the document that was not implemented, and not the whole document. This was in paragraph 147 of its written closing submissions, which dealt with what the Post Office submitted were “errors” made by Mr Godeseth which it submitted (and which the Post Office accepted) were “significant” and which it also stated had taken the Post Office “by surprise”. It sought to correct these errors, by way of submission rather than evidence, and one of those corrections was in relation to the implementation of the document in question. The claimants objected to this, and said the explanation that was provided was nowhere in the evidence, and the Post Office should not be permitted to give evidence by way of submissions.
Leading counsel for the Post Office was asked about this specific part of the submissions at the end of oral closing submissions and said that the explanation in paragraph 147.4 of the closing was “my instructions but, my Lord, it is based upon a previous version of the document and then an amended version of the document. And I will undertake to give your Lordship the two references.” In an email of 10 July 2019, which provided some references, further explanation was given which were said to support the submission that it was only section 1.4 of the document that was not implemented. The Post Office relied upon the following four points in support of that submission:
The passage in Mr Godeseth’s witness statement I have reproduced at [325] above.
That there were different versions of the document that were disclosed, but only one had been put in the trial bundle. The first version containing section 1.4 was dated 6 June 2010 and was authored by Mr Salawu.
This indicated therefore that the change was introduced by him. Most of the procedure document had been in existence for a number of years before that.
Earlier versions had been disclosed, none of which contained section 1.4, and none included section 1.4 and none were distributed to or even mentioned Mr Salawu.
The following points can be made about this.
Witness statements are expected to be factually correct.
Each party in this litigation who has submitted witness statements had ample opportunity to ensure those statements contain accurate evidence before they were served. Further, each witness is asked, at the beginning of their evidence when they confirm the contents of their statements in their evidence in chief. Apart from qualifications that were sometimes made, in this case (and in the great majority) the statements are usually stated to be true to the best of that witness’ knowledge and belief.
If a witness’ written evidence is shown to be incorrect (as Mr Godeseth’s, along with other of the Fujitsu witnesses, was) then the appropriate way to correct this is either in re-examination or by way of supplementary evidence. Here, Mr Godeseth was asked about this document and gave evidence in respect thereof on 20 March 2019. Mr Parker, also a senior witness from Fujitsu, was not called to give evidence until 11 April 2019. That was, again, ample time for this self-contained point to have been dealt with by supplementary evidence from Mr Parker, if that was thought by the Post Office to be important. In any event, there is no reason for it having been incorrect in the first place. Whether a procedure was, or was not, adopted by Fujitsu internally, is not something that ought to be capable of misinterpretation in any event, let alone on a subject as important as recording problems in the Horizon system.
I do not accept a matter such as this is suitable to be corrected by submission “on instruction” in any event.
However, even if it were, I am afraid that my view of information emanating from Fujitsu is, based on numerous other instances of such information being wrong in the actual evidence, to be somewhat sceptical of its reliability. It would also be potentially unfair to the claimants to allow this type of “correction” as they would be deprived of the opportunity of testing the new explanation. Absent some corroboration of this type of information coming from Fujitsu that supported it, I would not accept it in this case on this issue.
Further, and even if I were to take a contrary view of the points at 1 to 5 above, and accept that the Post Office should be given the opportunity of correcting Mr Godeseth’s evidence in this way, the explanation given by way of closing submissions does not assist the Post Office, for the reasons explained in the following paragraphs, [334] and [335] below.
This is for the following reasons. Mr Coyne’s point, to which Mr Godeseth was responding, was that the Fujitsu document identified metrics and KPIs to measure/control and reduce the risk of failure to detect, correct and remedy Horizon
errors and bugs. He was basing this on section 1.4 of the document. His conclusion was that “From the above, it is my opinion that Post Office should be aware of all recorded bugs/errors/defects in addition to those previously acknowledged by them, from the process metrics compiled above.” These records were sought by the claimants and the Post Office explained that Fujitsu’s position was that the records did not exist. They were not therefore provided to the experts. Mr Godeseth’s crossexamination showed that the Fujitsu explanation was wrong (and the Post Office’s closing submissions accepted it was wrong). The records however were not produced.
Finally on this, after receiving the email of 10 July 2019 to which I refer at [332] above, I asked in an email of 12 July 2019 for hard copies of the documents referred to in the relevant paragraph of that email (and hence relied upon by the Post Office in this respect) to be delivered to me. This was done, and a file served on the court containing four versions of the Post Office’s Customer Service Problem Management Procedure. They are version 0.1 dated 13 November 2007; version 2.0 dated 22 April 2008; version 2.1 dated 6 June 2010; and the version put to Mr Godeseth which was in the trial bundle, dated 5 September 2017. Given the additional versions produced all pre-date Mr Salawu leaving Fujitsu in 2013, this means that the points I have identified at [329] and [330] are still valid ones, and are still in the claimants’ favour.
Even on Dr Worden’s evidence and his views on the lower number of bugs in the Bug Table (he accepts that there are 11, although Mr Coyne’s figure is somewhat higher), there are plainly more bugs in Horizon than Fujitsu itself was aware of. Over the years of both Legacy Horizon and Horizon Online, the total number of software bugs, defects and errors in Horizon considered by the experts is far greater than the number to which Fujitsu have admitted. This is shown in the appendix to this judgment summarising the number of bugs, errors and defects, and their years of operation. The total number of software bugs, defects and errors in both versions of Horizon is very important information. Why Fujitsu chose not to collate and report these in the manner that even a Fujitsu internal document stated would (or should) be done, is wholly unclear. A comprehensive record, whether using the metrics identified by Mr Salawu or other similar ones, would surely only assist both Fujitsu and the Post Office in its administration of Horizon. Given that such a comprehensive record has not been produced to the experts for their consideration, notwithstanding the express request made on behalf of Mr Coyne, the court is entitled to conclude that no such comprehensive record exists.
Mr Godeseth was also asked about the BRDB transaction correction tool. This applies to Horizon Online. The original design document for this stated that it was to allow the SSC to correct transactions by inserting balancing records to transactional/accounting or stock tables directly into the BRDB system, and also to audit the changes made. It was not designed to delete or update the records. However, in cross-examination the claimants’ leading counsel analysed, and put to him, some careful points on how this tool had been used in practice.
The tool works by means of an SQL insert into a variety of tables in the BRDB. SQL is an abbreviation for Structure Query Language, and is a domain-specific language used in programming and designed for managing data. It is very widely used. Although Mr Godeseth would refer simply to “Oracle” as the programme, Oracle Database or Oracle is a database management system which is produced and marketed
by the US corporation which is also called Oracle. Although that is the name of the company, and SQL was developed (or invented) before the Oracle Corporation was founded, in this trial the terminology sometimes became a little shorthand on the part of those IT professionals very experienced in such matters. The Oracle Corporation is very well known, not only in the computing field, but through its co-founder Mr Larry Ellison who, amongst other achievements, has backed and competed in various America’s Cup races, including winning the 33rd America’s Cup in 2010 in the trimaran USA 17 with Oracle BMW Racing. Mr Godeseth, and the experts, are all highly experienced in the field and shorthand references to well-known languages or applications such as SQL and Oracle are to be expected. This use of shorthand did not lead to any lack of clarity in the evidence, and I refer to it only for completeness.
Information provided by the Post Office’s solicitors prior to the trial stated that the transaction correction tool had been used 2,297 times. However, all save one of these usages were said to be “Type 1” which were for the most part used to unlock stock units and which had no impact on branch accounts. There was one admitted instance of what was called “Type 2”, which was where a Balancing Transaction was inserted where it changed transaction data in the main transactional tables. This will have affected branch accounts. Mr Godeseth had not been involved in drafting that information supplied by the Post Office’s solicitors, but he knew about it and it matched his view of the intended use of the transaction correction tool. The 10 database objects, or tables, and sequences into which the tool was permitted to write are identified in a table at 2.4.1 of the low level design document for the tool. They are identified in terms of their object names, and were the BRDB operational exceptions table; the system parameters table; the FAD hash outlet mapping table; the process audit table; the process audit sequence; the transaction correction tool journal table; the FAD hash current instance table; the transaction correction tool control table; the branch information table; and finally the branch operators (or operational) exception sequence.
The design document made clear that:
“The following transaction tables have been granted INSERT privileges to OPS$SUPPORTTOOLUSER.
The transaction correction statement is only allowed to insert into these tables.”
These 9 tables were then listed. The process would use a "Transaction file containing an SQL INSERT statement that creates the required balancing transaction.", which is an Oracle command, and the SQL INSERT statement would provide a missing half of a transaction, where only one half was present. As it was put to Mr Godeseth, “the "SQL INSERT" statement effectively goes in and puts in the missing other side of that transaction?”, a point with which he agreed. The SQL INSERT command would put that missing other side into the records.
The SSC members would log into their own UNIX user, and then change directory and place their transaction file into the sub-directory. The document then records “They [ie the SSC member using the tool] will then invoke BRDBX015 manually.
The shell script module will be owned by the UNIX user 'supporttooluser'."
However, the explanation that was provided by the Post Office’s solicitors, of which Mr Godeseth said he was fully aware, about the use of the tool was as follows:
"Each document is associated with a single SQL statement which made a database correction. There are two different types of correction shown in the files - the SQL statements for each are of the form:
"1. Update OPS$BRDB.brdb_rx_recovery_transactionsSET_ settlement_complete _time stamp = ..." And then the "INSERT INTO" command.
The explanation of Type 1 was "Type 1 reflects the action taken to reset the recovery flag on a transaction. This will have no effect on branch accounts….”. It was also stated expressly that "Type 2 reflects the action taken to insert a Balancing Transaction where it changes transaction data in the main transactional tables. This will affect branch accounts."
Mr Godeseth expressly accepted, however, that this showed that the command used for what was being called Type 1 was not an insert command, it was an update command. Mr Godeseth described himself as not expert on Oracle, but he had a working knowledge of it, and he is plainly more experienced with using Oracle than most people. I find his experience in Oracle more than sufficient for him to be able to answer questions on this subject. Mr Godeseth’s evidence shows that Type 1 was an update – something which the low level design document expressly said the tool would not be used for. The design document did not contain the necessary database object fields table for performing the unlocking function, which the Post Office’s solicitors said represented all of the occasions, save one, when the tool had been used. This was all put carefully and clearly to Mr Godeseth in his cross-examination, and he agreed with all the points put.
This was further confirmed – the mandatory use of insert, and the fact that an update command was not included in the design intent of the tool – elsewhere in the design document where it stated under method “The module will read the contents of the input transaction file, which will be in the form of an SQL insert statement. Only a single insert statement is allowed and (after an optional introductory comment) it must start with the 'insert into' clause." The insert would be in the SQL script, and Mr Godeseth expressly confirmed this. This is another example of Fujitsu failing to observe its own design intent for the use of a tool that can have an impact upon branch accounts, and in my judgment is a factor that goes to the robustness of Horizon as well as the accuracy of data. As was put to Mr Godeseth, the use of the tool had gone “way beyond” the intent in the design document, and he agreed. He said that “there is tooling which is based on this” – by which he meant based on the transaction correction tool - “which has two aspects to it, certainly, so I think I’m agreeing with you”. It is agreed that there was only one occasion when what was called Type 2 was used. This evidence goes to the wider point that I have explained, namely Fujitsu failing to observe its own design intent.
Mr Godeseth was taken, very carefully, through a specific use of the transaction correction tool in 2010. In PEAK 0195561, a problem was reported to the SSC on 4 March 2010 where a SPM had tried, on 2 March 2010, to transfer out £4,000 (referred to in the PEAK as 4,000 pds, which means either pounds (plural) or pounds sterling)
from an individual stock unit into the shared main stock unit when the system crashed. The SPM was then issued with 2 x £4,000 receipts. These two receipts had the same session number. The PEAK, as one would expect, records various matters in note form and also uses informal shorthand. However, the main thrust is that when the SPM did the cash declaration, although the main stock unit (into which the £4,000 was being transferred) “was fine”, the unit from which the cash was taken “was out by 4000 pounds (a loss of 4000 pds)”. This is very similar to what Mr Latif said had happened to him, although the transfer in July 2015 to which he referred was £2,000. The PEAK related to Horizon Online and was the admitted occasion when the Balancing Transaction tool had been used.
Mr Godeseth had obtained the information in his witness statement dealing with this from Mr Parker. He also said he thought it was puzzling, and he was not sure that the PEAK “would be a totally accurate reflection of what happened” and was “an interpretation put together by a developer who is investigating the problems.” He suggested the original log files would be better (by which he meant better sources of information) and still exist, but they were not before the court. These log files would be held by Fujitsu, so any failure to have them before the court was no fault of the parties in this litigation, and that can only be laid at the door of Fujitsu. If a senior Fujitsu witness thought they would be better records, they should have been accessed and evidence about them could have been given directly. Certainly Fujitsu witnesses could have given evidence that relied upon them, had they so wished.
The PEAK records the following:
“After discussion with Gareth Jenkins, the suggested correction is to negate the duplicate transfer out by writing 2 lines to the BRDB_RX_REP_SESSION and BRDB_RX_EPOSS_TRANSACTIONS tables, with:
Product 1, Quantity 1, Amount 4000.00, Counter mode id 7 (TI)
Product 6276, Quantity -1, Amount -4000.00, Counter mode id 7 (TI)
This should be done using the Transaction Correction tool. An OCP approved by POL will be needed.”
This concerns Horizon Online, due to the references to OCP and OSR, and shows extensive efforts being made at Fujitsu to try to understand why this has occurred, and general doubt that the efforts to reproduce the fault in testing has worked. Some at Fujitsu wanted the priority to be downgraded; other entries resisted this, shown by some of the entries such as:
“What is missing from this Peak is an explanation of the events in terms of the requests, how they were ordered and when any was committed. Only then can we qualify the priority. The assumption is that we have a fix. The facts are –
1: A settlement request to timed.
2: A retry of request timeout occurred.
3: According to the DB entries both later succeeded.
Now unlike other reconciliation Peaks this stands out…..
We can’t reduce the priority unless we understand what is going on.”
The PEAK is lengthy and demonstrates a degree of frustration within Fujitsu at their own failure to get to the root cause. One entry on 24 March 2010 states inter alia “…..this shouldn’t have happened”. One might observe that this is stating the obvious. A later one on 25 March 2010 states “advised on the latest update in the call she states she was going to check if it was the same thing that happened before. [The Post Office] are want to know why this has happened? - why does it keep happening? - can you advise on this.”
(emphasis added)
One of the reasons or concerns expressed in the PEAK is that JSN entries – which should be unique and in respect of which no duplicates should be permitted in the journal table - appear to have occurred. Mr Godeseth confirmed that this should not have happened. Duplicate entries should not be permitted in the database. The documents showed that this did occur. Mr Godeseth said that the “JSN is part of the primary key into the message table, I cannot see how an Oracle database would allow that to happen” and “that would have to be a bug in Oracle, and it was certainly not the bug that I was looking – it was not the bug that I was looking at for the red alert”. He also said: “If this in fact happened as written down here [ie in the PEAK] then it would have had to have been a bug in Oracle and I certainly don't remember any such bug.”
The PEAK also shows computer script being used, both in an attempt to correct, and understand and analyse, what occurred. On 23 April 2010 an entry states:
“I think we have done as much as we can on this one. In conclusion although we haven’t been able to totally explain the behaviour, the risk of this type of PEAK occurring again has been minimised in live due to a change of behaviour in the BAL with respect to transactions…..This may now be marked as a duplicate of PC0194893”.
Rather concerningly – but entirely consistently with what I consider to be Fujitsu’s general approach to closure codes - the following entry appears that same day in another entry stating that the “Defect cause” is “updated to 40 – General – User”.
However, later the same day the following entry is made:
“I am sending this call back with Response Rejected.
Closing a call as 'Duplicate Call' results in a black mark against me. It basically means that I should not have sent the call over since the same problem has already been sent over in a previous call.
PC0195561 (duplicate transfer of 4000.00 cash) may have been caused by the same underlying fault as PC0194893 (banking reconciliation), however I could not have been reasonably expected to link the 2 calls and take the decision that it was not necessary to send PC0195561 over for further investigation.
Please close this call with category 'Advice After Investigation'
[End of Response]
Response code to call type L as Category 52 -- Pending -- Response Rejected
Response was delivered to Consumer.”
Eventually KEL cardc262s was updated with the information in the PEAK on 4 May 2010 and the call was closed. One of the final entries states that day “Category 60 – Final – S/W Fix Released to Call Logger”. “S/W fix” means software fix. The call is closed the same day. This is 2 months after the incident. The PEAK also referred to other PEAKs. Mr Godeseth was asked about those too. PEAK 0195962 records that the transaction tool has “been used in live”. That led to an Operational Change Process or OCP 25882 and stated:
“Due to a system fault, the branch did a Transfer Out of £4000 and a corresponding Transfer In of £8000
Justification: Correct a loss of £4000 at the branch due to a system fault
When: Planned for 10/03/2010 16:00 with a duration of 30 minutes
Extra detail: The Transfer In details were incorrectly doubled up when they were written to the BRDB. This needs to be corrected using the Transaction Correction tool.”
(emphasis added)
The actual printed address of the OCP document is as follows:
http://deathstar/SSC2/SSC_OCP/viewocp.jsp?OCPRef=25882&SID=f630192836137
This may show that the folder in which the document was kept was named by someone with an interest in a particular series of well-known films, or it may not, and this does not matter. It does, however, show that it was plainly generated or kept by the SSC.
Mr Godeseth agreed that the OCP showed at least one use of the transaction correction tool for one balancing transaction. Part of Mr Godeseth’s explanation for what had occurred, even though there had been two duplicate JSN entries, was as follows:
“In this situation the symptoms, as I'm reading them, are that because there was a bug in Horizon and this is in the new system, it was pretty early days of the Horizon Online, because there was a problem, we had something coming through which got through the journal filter but then failed at the branch database and so therefore, as far as the branch database is concerned, it has not happened.
MR JUSTICE FRASER: I understand that. I think what Mr Green is putting to you is that this shouldn't have got past the journal filter.
A. It's a bug, so certainly the way that the system should have worked, a JSN -- the same JSN coming up would be just a simple repeat of the message for -- because there was some sort of glitch.”
(emphasis added)
His final evidence in cross-examination on this was the following:
“A. Somebody may have been trying to look for a duplicate JSN entry. I can't really comment on what the guy was doing at the time who was trying to investigate. I am simply asserting that barring an Oracle bug, which would have been huge, you cannot have two entries with the same JSN.”
In my judgment, the Horizon Issues are sufficiently wide and costly to resolve concerning the Horizon System, both Legacy and Online, without widening them to include some attempt at including specific bugs within Oracle too. The Oracle Corporation is not even a party, and has had no opportunity to provide evidence or submissions. I simply recount Mr Godeseth’s evidence as he attempted to explain the PEAK. He did, as can be seen, refer to potential bugs in Oracle as an explanation for something that he believed to be puzzling. He accepted it was a bug and/or “a glitch”. In any event, other than Mr Godeseth’s evidence expanded in re-examination that he was involved in 2010 when establishing himself in Fujitsu (and in this he referred to an Oracle bug, which involved nodes in the database going down) there is no evidence before me that there were any bugs in Oracle in respect of this doubling up, and I reject the suggestion that this can be laid at the door of Oracle. What this PEAK shows, in my judgment, is that Fujitsu itself, with a great deal of time to prepare for the Horizon Issues trial, simply could not explain then, and cannot explain now, what caused this event to occur. That is relevant to my consideration of the Horizon Issues, and in my judgment shows that there was in that respect a bug, error or defect within Horizon. It also shows that there were those in Fujitsu who wished to close the call with the error put down to the user, as shown by the “Defect cause” being “updated to 40 – General – User”. There is nothing in any of the material on this incident to suggest it was in any way caused by the user.
He was taken to another PEAK, PC0175821 which was dated 19 February 2009. This is during the period of Legacy Horizon.
PEAK 01320275 was in relation to an incident on 21 December 2005 which showed a SPM, at a five counter site with seven stock units, had rolled over a particular stock unit called BB. He rolled it over in what was called “an effectively empty state” but having declared the correct amount of cash, and also adjusting the stock levels to the correct volumes, his branch account showed “a gain of approximately £18,000”. This was an obvious impact upon his branch account, and it was investigated. One entry in the PEAK stated:
“We are unable to correct the system figures safely. We can however provide accurate figures for what should have been in the Final Balance for BB, to enable POL [ie the Post Office] to make the correction perhaps by using a Transaction Correction.
POL need to make a decision on whether they are able to correct the problem in this way, however we do not see any other alternative. Corrective action should be taken before 11th January when the branch is due to roll into TP10.
The cause of the problem is unknown and is under investigation.”
(emphasis added)
This accepts in terms that there is a problem, and accepts that its cause is unknown. The suggestion by Fujitsu later in the PEAK on 3 January 2006 was to generate a negative figure effectively by changing the relevant data in the messagestore to try and cancel out the £18,000 which the SPM had showing as a positive figure in his branch accounts. It is obvious to me that this would only remedy the effect of the problem, rather than identify what had caused the discrepancy in the branch accounts in the first place. However, the entry is illuminating in any event:
“If we get to the problem before the office is rolled we are able to change objects in the messagestore to reset the stockunit back to the CAP (TP) rollover trailer. The PM can then rollover. PM should get a large shortage which cancels out the large gain.
We don't want to be having to do this as making manual changes to the messagestore is open to error and each time we have to seek authorisation from POL to make the changes.
If we get to the problem after the office is rolled (as in this call) then we are unable to correct the system figures safely. Its not been decided how we get the PM sorted out.
All in all, we want this fixed asap.”
(emphasis added)
Mr Godeseth accepted that generating the artificial negative figure in the message store could be done, but said he was “not involved interfacing with Horizon” at the time and was not at the operational level shown in the PEAK. He was not involved in obtaining authority from the Post Office for such matters. In my judgment, the way that Fujitsu anticipated resolving this, by artificially seeking to engineer “a large shortage which cancels out the large gain” is not exactly grasping the nettle. It would have the effect of disguising why the large gain had occurred in the first place, in other words a bug, error or defect.
Another PEAK 0152014 dated 10 December 2007 showed a problem that was described in the PEAK as “POLFS Incomplete Summaries Report” which related to a foreign currency transaction. At this time, £484 sterling could buy $1,000 (the PEAK did not specify US or Canadian, but the associated OCP identified US$ specifically) and the PEAK recorded that:
“This is due to a single SC line written for $1000 (£484) with no settlement in the middle of two RISP transactions.
On call PC0151718 the harvester exception was corrected and now the transaction for the day don't zero, hence this issue with the incomplete summaries report.
Am currently retrieving the messagestore for this branch, we will then be inserting a new message on the counter to remove the effects of this. OCP 17510 has been raised.”
Again, this was being done to correct the effect of what had occurred, rather than discovering what had led to the issue in the first place. In two places in the PEAK, the following entry appeared:
“**Again, this may also have caused a receipts and payments error, can EDSC please confirm whether this is a gain or loss at the counter and the amount.**”
Mr Godeseth had been asked to look at this PEAK in advance (his evidence went over two days and this was agreed by the parties), and therefore he had been given a chance to familiarise himself with its contents, and those of an associated KEL. The PEAK stated "Worth noting that the branch did not have any issues with the mismatched transactions because this was fixed before they did the roll. The branch is not aware of this and it's best that the branch is not advised." (emphasis added)
When it was put to Mr Godeseth that this showed that on not all occasions were SPMs advised of impacts on their branch accounts of particular problems, he said that was “a fair inference”. An inference is a common sense conclusion, and I agree with Mr Godeseth that it is indeed fair to draw that inference. He did however then add “I think I would say that Post Office were well aware of this and I would argue that it's a Post Office decision whether or not to tell a subpostmaster.” This shows that Mr Godeseth was not immune from arguing the case or making suggestions not supported by the evidence. Nothing was produced by either side to show that the SPM in this case was told. Certainly the person at Fujitsu who wrote “….it’s best that the branch is not advised” cannot have shared Mr Godeseth’s view at the time. Mr Godeseth did however also say, in my judgment again importantly, that “in the background to this there was a dialogue with [the] Post Office” and he agreed with counsel who put to him that the “Post Office would have been aware of what was being done by Fujitsu”.
The associated OCP however, whose subject was “write corrective bureau message for FAD 183227” – bureau meaning foreign currency – gave a great amount more detail. It stated:
“A single SC message 183227-7-1101211 was written in error on 26th November at 12:43:17, selling 1000 US dollars, with no corresponding settlement line. To remove the effects of this message at both the branch and on POLFS, we will insert a new message to negate the effects of the original message.
Justification: If the change is not made in the counter messagestore (before the stock unit is balanced on Wednesday), the branch will have an unexpected gain of £484 (or thereabouts - depends on exchange rate), and a receipts and payments mismatch. This gain would have to be resolved at the branch. There would also be an inconsistency between the branch and POLFS to be resolved. By correcting the problem locally, the branch may not be aware of the problem, and there will be no inconsistency between the branch and POLFS.”
And:
“The message will include a comment to show it has been inserted to resolve this problem (this will not be visible to the branch).
This change will first be applied to a copy of the messagestore within the SSC environment, and the stock unit then rolled over to make sure there are no unexpected consequences.
Neither the new nor the old message will be included in data sent to POLFS.
Gary Blackburn (POL) is already aware of this issue.”
(emphasis added)
This shows that the Post Office did know, as Mr Godeseth had said. It also shows that the branch would not be able to see what was done. It also showed that the intention was to reverse, by means of a specific insert into the messagestore by Fujitsu, an entry or data in order to correct a discrepancy in the branch accounts caused by an error or problem within Horizon Legacy. This was approved by the Post Office as shown by a later entry. The OCP had been raised by Anne Chambers. The “extra detail” – effectively what the message needed to contain or achieve – was:
“Extra detail: The original message had ProductNo:5129, Qty:1, SaleValue:484, PQty:1000. The new message will have Qty:-1, SaleValue:-484, PQty:-1000, with other attributes (including exchange rate) as before.”
The specific message that was inserted into POL FS, as shown in the OCR, was as follows:
“Extra detail: This OCR is being raised so that EDSC is authorised to amend the txn details for branch 183227 and insert these into tps_pol_fs_summaries_incomp table on the host.
Comments
Andy Keil (POA SSC Support) wrote at 12/12/2007 15:07: Updated POLFS feed for branch 183227 product 5129 mode SC with SaleValue=1014.73 and PQty=2080”
Mr Godeseth agreed that this showed “just over $2,000 being inserted in the Post Office system”.
However, the PEAK clearly demonstrated that the fix or corrective action that had been applied had simply made matters worse. This is because it stated for 14
December 2007, two days after the message had been inserted by Andy Keil of Fujitsu, the following:
“The counter problem which caused the first issue has been corrected by inserting a message into the messagestore, for equal but opposite values/quantities, as agreed with POL (OCP 17510).
As a result of this corrective action, the net effect on POLFS is zero, and POLFS figures are in line with the branch. POLMIS received both the original message and the corrective message.
Once the problem was corrected, there should have been no impact on the branch. However it has been noted that the stock unit BDC had a loss of $1000, which was generated after the correction was made. We have already notified Gary Blackburn at POL (email attached). This appears to be a genuine loss at the branch, not a consequence of the problem or correction.”
(emphasis added)
372. On 17 December Anne Chambers also stated in the PEAK:
“Summary for development:
A single SC line was written for $1000 (£484), with no settlement, in the middle of two RISP transactions.
The line was missing some AdditionalData so it wasn't harvested properly, but the main problem was the lack of settlement. POL authorised us to insert an equal but opposite message, to prevent a discrepancy (in theory anyway) and to avoid problems on POLFS. Please note that this is exceptional and must not be seen as a convenient avoidance in place of a fix.
Subsequent investigations showed that a second packed pouch barcode had been scanned, before the receipt for the first had finished processing. See the audit log / messages on counter 7 26th Nov at 12:43 / 12:44.
PC0147357 is already with development for what I think is a similar problem - Mode reverts to what it was prior to RISP.
Please also see update on that call.
Routing to EPOSS-Dev via QFP.
[End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under
Investigation.”
(emphasis added)
The PEAK was finally closed on 17 April 2008, the same day that the “defect cause” was “updated to 14: Development – Code”.
There are two important points to make in relation to the entry in this PEAK which I have quoted at [371] above. Firstly, the discrepancy in the branch accounts, which
was, after the “corrective action”, in deficit, was broadly in the amount of the difference between the two corrections that were made, namely the one by the OCP and the OCR. These corrections should, in my judgment, have been in the same amount, as one relates to “front end” (or the branch messagestore) and the other “back end” of the accounts so far as Horizon is concerned. The statement in the PEAK that “this appears to be a genuine loss at the branch, not a consequence of the problem or correction” is simply insupportable. That is a statement made by Anne Chambers, which in my judgment flies in the face of the documents. I am supported in my conclusion by two things. Firstly, Mr Godeseth’s evidence, when it was put to him that this was a fair possibility when he said “having read this PEAK in more detail overnight then yes, clearly that is what appears to have been the case.” I accept that evidence. It is supported by the documents and by the text of the two different corrections. Secondly, plain and obvious common sense.
The second important point is that the PEAK makes it clear this, or something very similar, has happened before. It states “PC0147357 is already with development for what I think is a similar problem - Mode reverts to what it was prior to RISP.” The reference to “development” is the department within Fujitsu where software is written to correct issues with the system. This is supported by the closure of the PEAK and defect cause stating “Development – Code”.
There is an associated issue that arises on this PEAK, the number of which I will again recite for clarity, PEAK PC0152014. I will deal with this here, rather than in the section dealing with Mr Coyne’s evidence or within the Technical Appendix (although the latter place could be a more suitable home for it). This is because it goes to the weight to be given to Mr Godeseth’s evidence.
A detailed explanation was put to Mr Coyne on this very matter, which occurred on the third day of his evidence on 6 June 2019 and went from page 103 of the transcript, ran until page 115 when the OCP was then put (which went to page 120) and then the OCR which was put between page 120 and 126. Leading counsel for the Post Office stated “I can tell you on instruction what that figure means”; “So on instructions I can tell you than the sale value of £1,0114.73 and the quantity of 2,080……” and “I appreciate that I’m telling you this on instructions because I have to say nobody knew that this suggestion was going to be made until it was put to Mr Godeseth….” It was made clear that there was what was being put to Mr Coyne as a detailed explanation of what had occurred, was “on instruction”, and it was nowhere in the evidence.
Mr Godeseth was cross-examined on 20 and 21 March 2019. When Mr Coyne was cross examined, about 2 ½ months later, a great deal was put to him “on instruction”, as I have identified. It was a detailed factual explanation that was not included in any evidence at all. Firstly, the OCR that went with it showed that “This OCR is being raised so that EDSC is authorised to amend the txn details for branch 183227 and insert these into tps_pol_fs_summaries_incomp table on the host.” This was required precisely because Mr Godeseth was correct, in my judgment, in his understanding of what had occurred. I accept his evidence on this. Secondly, the explanation put to Mr Coyne “on instruction” was (broadly) that there were two problems. A careful attempt was made to explain to Mr Coyne how (basically) Mr Godeseth had misunderstood, and to downplay the effect of what had occurred.
I do not accept that this subject was capable of being corrected – even if that is what the documents showed, which in my judgment they plainly did not – in the way adopted by the Post Office by attempting to give evidence by way of submissions from counsel “on instruction”. This deprived the claimants of any ability to challenge the explanation, if explanation it was, by cross-examination. Further and in any event, there was ample time for the Post Office to have provided a short witness statement from a Fujitsu witness, such as Mr Parker (who was not called until 11 April 2019) dealing with this, if that was the evidential explanation. If that had been done, permission for that statement would have been required, but given the matter arose from something Mr Godeseth was asked about, that application could have been dealt with at the time and if this genuinely was evidence that the Post Office could not have known in advance would be required, it would have been allowed in. Even supplementary questions in chief could have been used as a method of obtaining actual evidence on this point. It simply is not procedurally acceptable, or fair, for evidence of this nature to be given by way of submission “on instruction”. However, and in any case, I have considered the Post Office’s case on this notwithstanding these points, and I find that the documents do not substantiate the explanation given on Day 16 to Mr Coyne when he was being cross-examined.
The explanation did not make sense on the face of the documents, and they did not support the points that were being put “on instruction”. Firstly, as Mr Coyne pointed out, the PEAK identified that a message was to be inserted into the messagestore. None of the documents put to Mr Coyne “on instruction” identified what that message was. This point was made by Mr Coyne during the cross-examination:
“Q. Yes. So what they are talking about here, what this change here is, is a change being made by using the TIP repair tool into the TPS, correct?
A. Right, so this doesn't relate to the creation of the message then.
Q. It doesn't relate to the branch accounts, Mr Coyne, does it? This is an OCR which involves an exercise -- well, the use of the TIP repair tool to change data that is in the TPS system, yes?
A. Yes, but the PEAK refers to the insertion of a message into the messagestore.”
The critical point, that the explanation skipped over, was what the message that was inserted in fact was, in actual terms. Had that message been produced, it could have been looked at, and deciphered. Understanding such a message is not an insuperable task beyond mortals. The second point is that the PEAK refers to a KEL, namely KEL obengc3120K. At the beginning of the trial, I asked for hard copies of all PEAKs and KELs put to witnesses to be prepared in hard copy too, so that I had a working file that I could use during the trial for PEAKs and KELs already referred to (in addition to the three OPUS trial bundle screens that I had to work from). This was done. That KEL is at tab 21 of volume 2 of the file entitled “PEAKs and KELs referred to in Days 5-8 and 12”. The KEL states the following, and I will reproduce the text in full, including the relevant table references.
“KEL obengc3120K
Title:
Not harvested: Could not update database: Updating table TMS_RX_BDC_ TRANSACTIONS, ORA-02290: check constraint
(OPS$TPS.BT23B_BUREAU_REGION_CHK) violated.
Summary:
Harvesterd did not harvest message: Could not update database: Updating table
TMS_RX_BDC_ TRANSACTIONS, ORA-02290: check constraint (OPS$TPS.BT23B_BUREAU_REGION_CHK) violated
Symptoms
The Harvester tried to insert into the BdC table TMS_RX_BDC_TRANSACTIONS a message with missing ‘Blackbox’ data. This caused Oracle [ORA-02290] to detect a check constraint (OPS$TPS.BT23B_BUREAU_REGION_CHK) violation, hence the appearance in the TPSC254.<br><br>Harvester exceptions do not normally cause entries on TPSC250, but this one seems to. <br><br>TPSC257 displays the contents of the TPS Incomplete Summary table where the summarisation program detects that the total harvested transactions for the day don’t NET to zero.
Problem
Due to some data in the ‘Blackbox’ or AdditionalData of the messagestore not being populated, the bureau product was missing key data including<br>BUREAU_REGION<br>MARGIN<br>MARGIN_PRODUCT<br>EF FECTIVE_EXCHANGE_RATE<br>and so when the Harvester attempted to insert the message into the BdC table TMS_RX_BDC_TRANSACTIONS, however Oracle [ORA-02290] detected a check constraint
(OPS$TPS.BT23B_BUREAU_REGION_CHK) being violated.
Solution – ATOS
Warning: Check the AMOUNT (or sum of AMOUNTs) for the branch on the TPSC254 report.<br><br>If the total AMOUNT is NOT ZERO, and the branch is NOT on TPSC257 (same day), <br>or the total AMOUNT IS zero and the branch IS on TPSC257 (same day) then there is probably a problem with the messages written on the counter – a bureau SC message with no corresponding settlement. Use KEL <ahref=kel._view_kel.jsp?KELRef=acha3159Q>acha3159Q</a>.<br><br>Check the messages anyway, just to be on the safe side. If the session was settled properly….<br><br>MSU must raise an OCR so that SSC can use the TIP Repair Tool to populate the missing columns, using values from another transaction for the same currency, same branch, same day if possible.<br><br>NOTE: By repairing the txn, the TPS_POL_FS_SUMMARIES_INCOMP will be corrected automatically after that txn has been successfully harvested. <br><br>*** In the past, we have just repaired these, on the assumption that they are genuine SC transactions which have been corrupted. However I think it may not be a genuine sale, but is related to an attempted pouch reversal (this is certainly the case for the more extreme instances where the settlement is missing). If the SaleValue and the PQty have opposite signs, and there is no reversal (mode ER) for the transaction, this may have caused a loss or gain at the branch which they can’t resolve themselves. Let Anne know if there is another occurrence like this.
(emphasis added)
This KEL shows that there was a potential impact on branch accounts – it states “this may have caused a loss or gain at the branch which they can’t resolve themselves”, which is very clear. It shows that the explanation put to Mr Coyne “on instruction” is, in my judgment, completely wrong. It also shows that it has happened before, as “in the past we have just repaired these”. I would also note for completeness that this KEL is not dealt with at all by Mr Parker in the table accompanying his 1st witness statement. Two KELs from Ms Obeng are, namely CObeng1123 at number 24 in the table at E2/11/40; and obengc5933K at number 31 in the table at E2/11/43. However, this one is not.
I accept Mr Godeseth’s evidence on this point, which is wholly consistent with the documents, in particular the PEAK and the KEL. The text of those documents shows that a message was intended to be written into the messagestore; the actual message used is nowhere available; the effects of all this clearly being an impact on branch accounts.
A further PEAK was PC0175821, which was dated 22 February 2009. This also related to foreign currency transactions. This PEAK related to two problems, or a problem with two elements to it. It is at F/485/1 in the trial bundle, and it clearly and expressly refers to the same KEL, obengc3120K.
The two aspects were explained by Mr Godeseth as a change using the transaction repair tool to “get the feed into POLFS correct” and also “a change to the messagestore to get the branch aligned.” The repair “had to go two ways” because the branch and the POLFS had to be aligned. If they were not, there would be a discrepancy in branch accounts. This showed the two problems, or two sides to the same problem. It was stated as follows in the PEAK:
“The first is where all five SC transactions missing core data as described in the above-mentioned KEL." This means obengc3120K, referred to at the beginning of the PEAK.
And the “Second is absence of equal but opposite 2 (ie settlement) lines. See PC0152014 for a similar problem and how problem was resolved."
“There are two sides to the problem relating to these txns. The first is where all five SC txns missing core data as described in the above mentioned KEL. Second is absence of equal but opposite [i.e. settlement] lines. See PC0152014 for a similar problem and how problem was resolved.
For the first problem, I have used the TRT [the transaction repair tool] to insert the missing data i.e. Region, Margin, Margin Product and EffectiveExRate.”
The reference to “txns” means transactions. PC0152014 is the very PEAK referred to in [376] above.
Mr Godeseth agreed with both of those points in the PEAK as they were put to him by the claimants. Basically, half the data was missing. The transaction repair tool or TRT was used for the first problem.
Numerous other PEAKs and some KELs were put to him with very specific details. Both Gareth Jenkins and Anne Chambers were involved in raising, or investigating, a great many of them and they are referred to in the text of both classes of documents. In all of them, Mr Godeseth’s answers to questions were, in my judgment, consistent with the entries within the documents, and also consistent with the claimants’ case on the Horizon Issues. In particular, those in the PEAKs dealing with foreign currency which I have explained above was clear, and settlement lines were missing, which Fujitsu sought to correct by injecting lines into both the messagestore and into POLFS.
Finally, a series of questions were put to him demonstrating what are called APPSUP permissions. This is a very powerful permission (which really means level of access) and PEAK PC0208119, which runs from February 2011 onwards into 2012 and beyond (the final entry being in 2015), made the following relevant comment:
“As per the previous PEAK comments, the role 'APPSUP' is extremely powerful and should only be used under extreme circumstances and under MSC supervision. As such the Branch Database design was that 3rd line support users should be given the
'SSC' role, which is effectively read access, ie. 'select_any_table + select_catalogue'. SSC team members should only have to [access] BRSS for normal support investigations, unless the information has not replicated in time. SSC should only given the optional role 'APPSUP' temporarily (by Security Ops authorisation/emergency MSC) if required to make emergency data amendments in BRDB Live.
It is a security breach if any user write access is not audited on Branch Database, hence the emergency MSC for any APPSUP role activity must have session logs attached under the MSC. Host-Dev previously provided scripts, such as the Transaction Correction Tool, are written to run under the SSC role and also write to the audit logs.
SSC users created on BRDB should only have the SSC role, and the user creation script should be amended by Host-Dev to reflect this. A separate script giving/revoking emergency MSC access via APPSUP can be delivered, logging this to the hostaudit directory. In parallel Host-Dev should investigate any Host-Dev delivered script to ensure they are all executable by the SSC role. SSC should investigate any of their own scripts to ensure they have sufficient permissions under the SSC role, taking into account they should primarily perform their work on BRSS.
Any day to day scripts should not access BRDB directly. Any scripts needing more than the SSC role should be questioned, except those that would run under MSC APPSUP. Once the investigation is complete, all BRDB SSC users with APPSUP should have the role removed by ISD (via MSC) and ensure they do have the SSC role.
If anyone is in disagreement with the above course of action, then I'll set up a meeting with yourselves and Security when I'm down in BRA01 next week.”
(emphasis added)
This entry was from Andy Beardmore, Senior Software and Solution Design Architect Application Services. The experts are agreed that the APPSUP role would, effectively, permit anyone who had that permission to do almost anything on Horizon. It was available to 3rd line support at SSC, the level at which Mr Roll was employed by Fujitsu. This PEAK further substantiates the evidence of Mr Roll and is consistent with it. APPSUP was described by Mr Parker as “the more technically correct name for a type of privileged access to the BRDB”. It is a very powerful permission.
A later entry in the same PEAK states:
“The Business Impact has been updated:
Cost: There is currently no "cost" to this issue. The users affected have more access than is required.
Perceived Impact: The customer is not aware of this problem or change.
Scope: No actual impact/incidents of problems relating to this issue have been experienced yet (and not expected)”.
(emphasis added)
The customer was the Post Office. The “issue” referred to was Fujitsu SSC users having had far greater access, namely the powerful APPSUP permissions, on more than the intended basis, which was supposed to be in extreme circumstances only and temporarily. This is shown in an entry for 1 February 2011 by Mark Wright who stated “….SSC users have the APPSUP role” and “When we created SSC users for BDB/BRS etc. we used “appsup” as that is what SSC have always been and what they migrated as on Horizon databases.” (emphasis added)
It is clear that all the members SSC had always APPSUP, but were not supposed to have this powerful role all the time.
Anne Chambers said, in an entry on 1 February 2011:
“Unfortunately development write their scripts explicitly to use SSC. So I think we're stuck with it unless they deliver new scripts (which would not be a popular or quick option).
When we go off piste we use appsup. Can we have both??”
(emphasis added)
Mr Godeseth said that by “off piste” he was very confident that she meant “having to fix a problem that was not catered for by a script that is available to these people” by which he meant SSC. It was clear that SSC were using APPSUP and wished to continue to do so. It was equally clear that all the members of SSC were not supposed to have it.
The Post Office’s auditors for the year ended 27 March 2011 were Ernst & Young (“E&Y”) a very well-known global firm. In the Management Letter for that year, E&Y had identified the following:
“The main area we would encourage management focus on in the current year is improving the IT governance and control environment.
Within the IT environment our audit work has again identified weaknesses mainly relating to the control environment operated by POL’s third party IT suppliers.
Our key recommendations can be summarised into the following four areas:
(emphasis added)
The “third party IT suppliers” referred to were Fujitsu. Whether it was the work performed by E&Y that had led to the decision at Fujitsu that the APPSUP permissions were not being sufficiently restricted or controlled, or something else, does not much matter. A company such as E&Y would not lightly refer to “again identified weaknesses” in the Management Letter for a particular year unless that conclusion had been reached after a high degree of professional and in depth investigation, of a type that even a lengthy adversarial trial such as this one cannot hope to replicate.
Indeed, the Post Office’s solicitors confirmed in a letter in this litigation dated 11 February 2019 that Fujitsu SSC users who were identified as PERSON-POA UNIX (also sometimes referred to as POA Unix users) had a range of privileges and were granted the UNXADM role. This may be a descriptor for Unix (hence UNX) Administrator (hence ADM) but this does not matter. The UNXADM role contains, to quote from the letter of 11 February 2019 itself, the “DBA role” which is stated to be:
“This is an Oracle supplied role for use by database administrators (DBAs). Lots of privileges are granted to this role so users have the ability to update/delete/insert into any of the Branch database tables.”
(emphasis added)
POA Unix users also are:
“Granted the ability to execute the following executable functions:
(a) OPS$SUPPORTTOOLUSER.PKG BRDB TXN_CORRECTION —framework to
allow the user to insert fully audited balancing records into a BRDB transaction table (made against node ID 99).”
Both of these are powerful roles. Mr Godeseth was asked specifically about the DBA role explained in [395] above. His specific answers merit quotation verbatim:
“Q. You would agree that those people have the role which allows them privileges to update, delete, or insert into branch database tables whether they are using the correction tool or not?
A. Those people could log on to the database and do an awful lot of damage.
Q. And the only audit of that that we have prior to 2015 was log on and log off; that's correct, isn't it? A. Correct.”
(emphasis added)
Mr Coyne had given evidence that he had identified 2,175 occasions when it had been used. Mr Parker in his 3rd witness statement had said that figure was wrong, but also said:
“I have not examined the privileged user logs, but based on my experience my expectation is that these uses of APPSUP, or at least the vast majority, are for support work that does not involve changes to transaction data. I cannot recall any cases in which it has been used to change transaction data, but I cannot state unequivocally that there are no circumstances in which it has ever happened.”
Mr Godeseth said in his written evidence that “Privileged users can, theoretically, inject, edit or delete transaction data in Horizon Online. As far as I am aware, this has never happened.” His evidence above about prior to 2015 makes it clear that he could not work out the number of times this may have happened in any event. Therefore the fact that he was not aware it had ever happened is neither here nor there, and does not assist the Post Office.
In his re-examination Mr Godeseth was asked about an entry in an OCP in relation to a fix that stated UNIX user brdb. He said he did not know the detail, it meant logging on to the database, and the “UNIX user” referred to “would be one of the guys in Ireland”. These were not members of the SSC. When asked to clarify “who are the ‘guys in Ireland’ exactly?” he said “the people who support the hardware, so UNIX is an operating system so they work at a pretty low level on the systems” and by “low level” he meant “they have pretty powerful user rights but they are very much driven by process as to how they use them”. He also said there was complete and utter control on the process they go through before they did any of this type of activity, which in that re-examination example was "Logon to BRDB Node 1 as UNIX user
'brdb'."
He was also cross-examined about some of the bugs, which appear in the Bug Table considered and partly agreed by the experts. The main admitted bug by the Post Office (in distinction to other bugs admitted by its expert, Dr Worden) is called the Callendar Square bug. Mr Godeseth gave evidence in his witness statement about this, partly in response to a statement that had been given by Mr McLachlan, who was not
in the event called by the claimants. However, Mr Godeseth’s evidence about Callendar Square on this was still called by the Post Office. It is a Legacy Horizon issue. Mr Godeseth had no first hand knowledge of this but had reviewed documents and spoken to Mr Jenkins.
One of the KELs to which he was taken, with the reference JSimpkins338Q, related to problems with Riposte and what was sometimes called the Riposte lock or unlock problem. This was when an unexpected error occurred in Riposte while attempting to insert a message. The KEL shows it ran from 2002, with other events occurring in 2003, 2004 and 2005. The entry for September 2005 expressly states that “this problem is occurring every week, in one case at the same site on 2 consecutive weeks”. A PEAK was sent to development. Under “solution” in the KEL the following entry appears:
“SMC: If the event is seen at a multi-counter office during the working hours of the PO, or up to 18:00 on a weekday (in case they are balancing out-of-hours), RING THE OFFICE AND GET THEM TO REBOOT the eventing counter. If they are in the process of balancing, it is strongly advised that they reboot before continuing with balancing as they are at risk of producing an incorrect balance. Warn the PM that if transactions appear to be missing, they should not be re-entered - they will become visible after the counter has been rebooted.
If a reboot/Cleardesk does not resolve this problem, send the call over for further investigation - SSC can rebuild the messagestore on the affected counter.”
(Block capitals in original; emphasis by underlining added)
The risk of a branch producing an incorrect balance as a result of this problem with Riposte is clearly recorded. An incorrect balance is another way of referring to discrepancies in branch accounts, a central issue in this litigation.
The Callendar Square issue or consequences (now admitted by the Post Office to be a bug) were discovered in 2005 at the Callendar Square branch (hence the name). It is also sometimes called the Falkirk bug, but I shall refer to it as Callendar Square. It undoubtedly caused discrepancies in the accounts. The SPM in question had support from their Area Manager and the documents demonstrate the following, from the Area Intervention Manager Visit Log:
“This office had severe problems balancing on Wk 25, resulting in a shortage of £6,414.46. After checking various reports I am satisfied that the error is made up of:
£3,489.69 – Transfers
£2,870 – Giro Deposits
£54.52 – unidentified (however due to all the coming and going with re- keying entries, then this could come back as an error).
The Spmr claims that there was a Horizon software problem on 14.09.05 from 15.30 onwards. This was picked up when a member of staff noticed that a transaction, which had been taken by another member of staff, had not been entered onto the system, so therefore she put the transaction through. She checked at the time with her colleague who said that she thought she had put it through already however she accepted that she could have made a mistake. Following on from that, it was picked up that other giro business deposits that had been entered had not come up on the system, so they were re-keyed.
There was also a problem with transfers from one stock to another, in that they had doubled up. The Spmr made several telephone calls to the NBSC, telling them about his problems and he was advised to carry on with balancing and produce his Cash Account. Whilst doing this a warning came up, however the NBSC told the staff to continue to roll over. The result was that the office balanced £6,414.46 short.
The Spmr spoke to the Horizon Support Centre (ref E0509150123) who investigated and agreed that there had been a navigation problem that had now been rectified. They told the Spmr that they would report to NBSC that they had identified and rectified the problem and that the amount could be held in the suspense account. However, as part of the shortage relates to transfers, and no error notice will be issued, then the Suspense Account Team are not prepared to authorise the entry.
I telephoned The Suspense Account Team (Ann Wilde), who told me that checks could be made with Girobank after next Wednesday, and if that shows that duplications have been made, then they will authorise the amount to be moved to the suspense account, until the office receives an error notice. However, Ann stands by what she said about the transfer problems, and that they would not move this amount to The Suspense Account.
I went back to the Horizon Support Centre and spoke to a supervisor (Ken). He said that the call had now been closed as the problem had been rectified. I asked what was to happen about the resulting shortage and he referred me back to NBSC, who they said would do various checks. I then contacted NBSC, spoke to Rob Hughes and told him the story – he said he would put a call through to Service Support. No follow up was received from Service Support regarding this call.”
(emphasis added)
The important points from this are as follows. This problem turned out to be an admitted bug. At the time, the SPM noticed the problem – and could only identify it as a “problem with Horizon” and reported it. The branch accounts were plainly affected by a sizeable amount, over £6,400. Horizon Support Centre, after investigation, agreed that there was a “navigation problem” – this is a problem with the Horizon system, and was caused by a bug. The Suspense Account Team would not authorise the shortfall in the branch accounts being moved to the Suspense Account; in other words, they were not prepared not to hold the SPM responsible. Mr Godeseth also accepted that “the genesis of it appears to go back to February 2003 and similar lock agent problems back in November 2000”, a point that he accepted as “that’s fair”. The history of the problem was therefore somewhat longer than something that just occurred in 2005 for the first time. During that five year period, branch accounts were usually being produced every four weeks.
Mr Godeseth admitted that it was “a horrible position” for the SPM to have been in. He said “Service Support” was not Fujitsu, it was the Post Office. The story at [404] and [405] read like a mini-summary of many of the factual issues in the litigation generally, with the exception that this is an admitted bug by the Post Office.
The subsequent documents in this respect show that the matter went on for some considerable time. In November 2005 the following is stated:
“1/ This problem is the root cause of the reconciliation error closed in PC0126042
2/ Presumably the root cause is deemed to be software not hardware
3/ The Postmaster has a workaround in place which is not to duplicate transactions (e.g. Transfer In) just because the original attempts were successful but not showing on all nodes
4/ POA CS MSU have a workaround in place which is that if 3/ above is not followed & PC0126042 reoccurs, a BIMS will be issued advising POL to issue a Transaction Correction
5/ There is no SLT for software fixes as they are delivered based upon the priority or severity of the issue and could remain open until both businesses decide a fix is necessary or the work around is adequate.”
This shows that the entry in the PEAK identifies this as a software problem; that is effectively now accepted, but this entry from 2005 did not lead to any public acceptance of a bug in Legacy Horizon either then, shortly afterwards, or for some years after that.
The acronym SLT means Service Level Target. What this entry means is that this problem had occurred before, and was known to have occurred before; there was something called “a workaround” in place, but that required the SPM physically deciding not to duplicate transactions if the original transaction did not show up as successful on all the nodes; if that were not done, TCs would be required; both Fujitsu and the Post Office had to decide a software fix was required in order for this to be corrected by fixing the software. No such decision had been taken at this point.
At Callendar Square itself, in January 2006 the SPM phoned one of his superiors. The record of that call in the log states the following:
“Issue Raised: PM WANTS TO SPEAK TO SANDRA MCKAY URGENTLY REGARDING SERIOUS PROBLEMS WITH HORIZON IT IS REPEAT PERFORMANCE AS TO WHAT WAS HAPPENING BEFORE AND HSH
FAILING TO PICK IT UP
Response by SPM: Telephoned the office and Allan said that he was having problems again with transfers. He has contacted the Horizon helpdesk who have subsequently come back to him to say that there is no system problem and that he should contact
NBSC. He did this and from what I can understand the NBSC have told him that he is
trying to balance on two different terminals. Allan disputes this and is adamant that there is a system error.”
(Block capitals in original; emphasis by underlining added)
The statement that “there is no system problem” which is told to the SPM evidently conflicts with what the PEAK at [407] had recorded, that “presumably the root cause” was the software.
Emails in February 2006 to and from Anne Chambers stated expressly the following:
“…. I notice that in the early guise of this problem in the call it states the PM as Female:
Wed 12 October 2005 17:39 by UK956078 / HSH1 Saved: Wed 12 October 2005 17:39 Pm was trying to transfer £2490 from node 7 onto node 2. She states that she has accepted the transfer on node 2 but the system is not showing this. On node 7 it is showing pending transfer but it is not showing on node 2. It appears on her transfer sheet as completed.
At the bottom of this email re a magical £43k appearing and disappearing the PM is Male He reports:
You may recall that in September the above office had major problems with their Horizon system relating to transfers between stock units.
The Spmr has reported that he is again experiencing problems with transfers, (05.01.06) which resulted in a loss of around £43k which has subsequently rectified itself. I know that the Spmr has reported this to Horizon Support, who have come back to him stating that they cannot find any problem.
Clearly the Spmr is concerned as we have just spent a number of months trying to sort out the first instance and he doesn't want a repeat performance. He is convinced that there is something wrong with his Horizon kit. I would be grateful if you could investigate this and give him any support that you can. I'm due to visit the office tomorrow to have a look at his paperwork and discuss the situation with him.
So apologies for the long windedness but I have been given this by Liz as a problem so:
Is there a problem at this branch? is it Horizon kit or is there an issue with staff there?
If there is an issue is this S90 release the cure? how confident are you/we it will fix the problem?
S90 counter release due week 4th March. Getting Sarah to check if this site is in the pilot 24th or just part of the general release 4th March.
Appreciate your comments please.”
(emphasis added)
412. In the answer to this question posed in February 2006, Anne Chambers stated the following:
“Haven't looked at the recent evidence, but I know in the past this site had hit this Riposte lock problem 2 or 3 times within a few weeks. This problem has been around for years and affects a number of sites most weeks, and finally Escher say they have done something about it. I am interested in whether they really have fixed it which it why I left the call open - to remind me to check over the whole estate once S90 is live - call me cynical but I do not just accept a 3rd party's word that they have fixed something!
What I never got to the bottom of, having usually had more pressing things to do, was why this outlet was particularly prone to the problem. Possibly because they follow some particular procedure/sequence which makes it more likely to happen? This could still be worth investigating, especially if they have continuing problems, but I don't think it is worthwhile until we know the S90 position.
Please note that KELs tell SMC that they must contact sites and warn them of balancing problems if they notice the event storms caused by the held lock, and advise them to reboot the affected counter before continuing with the balance. Unfortunately in practice it seems to take SMC several hours to notice these storms by which time the damage may have been done.”
(emphasis added)
This shows the following important points. At least Anne Chambers in early 2006, and all those with whom she was corresponding, knew that this problem – now admitted to be a software bug – had been around “for years”. Horizon Support were telling the SPM, whose branch accounts were affected by discrepancies, that “they cannot find any problem”. The SMC – the part within Fujitsu responsible for providing corrective action for the “event storms” – would not always notice these had occurred in time and by then “the damage may have been done”. I find that by “the damage” this can only mean impact upon branch accounts.
Mr Godeseth gave evidence about S90, the software release intended to remedy this, which was distributed in March 2006. This was 6 years after Legacy Horizon became in use. His evidence was that this release was one year before what he called “the relevant period” for Ms Misra’s branch, one of the SPMs convicted of criminal offences, which he was asked about. He also gave evidence that the Callendar Square bug affected 30 branches, and had caused mismatches in the branch accounts at 19 (corrected in supplementary evidence in chief from 20) of them. He said in his witness statement that this information had come from Matthew Lenton, Fujitsu’s Post Office Account Document Manager and that “Fujitsu has established this for the purposes of this statement using the event logs described above.” He accepted that this part of his witness statement was not correct. The figures that he had provided had not come from an exercise specifically for the purposes of his evidence, or for the Horizon Issues trial. It had actually come from “the spreadsheet [which] is something that was done at the time, as I understand it by Anne Chambers, to actually investigate what had happened when we had had these locks.”
This was an important correction by Mr Godeseth, for reasons that I will now explain.
It is a matter of public record that Anne Chambers gave evidence before the High Court in the case of Post Office Ltd v Castleton [2007] EWHC 5 (QB) before HHJ Havery QC. Her evidence in that case is summarised at [23] of that judgment. Callendar Square is referred to, but as a single branch affected. The reason that I have referred to this case is because it was specifically referred to by Mr Jenkins in an email of 8 March 2010 which was referred to by Mr Godeseth in paragraph 14 of his witness statement. Mr Godeseth said that he agreed with that email, which stated:
“I’ve been asked about the issue at Callender Square, Falkirk (I’m not quite sure about the spelling) that came up at the Castleton Trial.
I thought I’d better keep you in the loop on this.
I’ve now dug back into the archives to provide the following summary:
The problem occurred when transferring Cash or Stock between Stock Units. Note that West Byfleet does operate multiple Stock Units so the issue could have occurred.
It manifests itself by the Receiving Stock Unit not being able to “see” the Transfer made by the “sending” Stock Unit and is compounded by attempting to make a further transfer. Note that such transactions usually reappear the next day.
It is clearly visible to the User as a “Receipts and Payments mismatch” at the time that one of the Stock Units is Balanced. This usually results in the Branch raising a call. There are no such calls in Andy Dunks’ Witness Statement which summarises the calls raised by West Byfleet. Also this can be checked on any Balance Reports or Branch Trading Statements that are available from the Branch which should show that Receipts and Payments do match and that the Trading Position is zero.
The problem is also visible when looking at system events associated with the Branch. The System events from 30/06/2005 to 31/12/2009 for West Byfleet have been checked and no such events have been found.
The problem was fixed in the S90 Release which went live in March 2006 and so would not have been relevant at the time of the detailed Transaction Logs obtained for West Byfleet between December 2006 and December 2007.
Therefore I can conclude that the problems identified in Calendar Square, Falkirk are not relevant to West Byfleet.”
This email does not provide anywhere near the same degree of information about the Callendar Square bug as was available at the time of the Horizon Issues trial, or as recorded in the PEAKs above. There has obviously been further investigation at Fujitsu into this specific issue since then, as made clear by Mr Godeseth’s evidence about the information coming from Mr Lenton and Ms Anne Chambers. The email does not use the term “software bug”. It does not refer to the fact that it is accepted by
Fujitsu, following an investigation, that the admitted bug had affected at least 30 branches. Nor does it refer to the fact that the Callendar Square bug caused mismatches at 19 of those 30 branches.
The spreadsheet to which Mr Godeseth referred as having been “prepared at the time” by Anne Chambers was also put to him. It included textual notes and a list of call references and branches affected by approximate date, which were all in 2004 and 2005. The notes were:
“NB many other branches had multiple events, preventing replication, but these are the majority of those which came to Peak, having either reported a problem or it caused a reconciliation report entry.
From Sept 2005, cash accounts were replaced by branch trading statements and the TPSC256 report was no longer populated. I can't remember how we then knew about receipts and payments mismatches and if we would have picked up on further issues.”
(emphasis added)
These notes make clear that there were other branches potentially affected, and the list of 30 branches compiled in the spreadsheet were “the majority”. It does not state that it is an exhaustive list and the notes suggest it is not. The summary for each branch listed gives a variety of consequences or effects. They include:
Receipts and Payments Mismatch.
Receipts and Payments Error.
Host Generated Cash Account Line.
Unable to roll Stock Units.
Reconciliation Report.
Slow running counter.
Pension and Allowance Report printouts.
Screen freezes.
Problems with declarations.
Unable to create AA.
Loss due to double transfer.
Transfers not showing on nodes.
CAC Lines not matching.
The spreadsheet was disclosed to the claimants on 27 February 2019, or very shortly before the Horizon Issues trial, and the properties shown on the electronic document identify Anne Chambers as the author, with a date of 22 December 2015.
Mr Godeseth also, entirely accurately in my judgment, accepted that the Callendar Square bug had been present for a considerable amount of time prior to 2005. He said “I would agree that the underlying bug had been there for a considerable time, probably since the Horizon went in”. This is made clear by, amongst other documents, Anne Chambers’ entry in February 2006 stating that the problem had “been around for years”. In another of the many PEAKs that dealt with this issue, the entry for 22 September 2005 states “Note - a few of these errors seem to occur every week at different sites.”
Mr Godeseth was taken to the contents of two letters from the Post Office’s solicitors to the claimants’ solicitors. The first was dated 28 July 2016. This was the Post Office’s solicitors’ substantive response to the claimants’ letter of claim, is very lengthy (as was the letter of claim itself) and ran to 99 pages. The second letter was dated 11 January 2019. He agreed that certain important elements of the paragraphs in those letters explaining Callendar Square were incorrect. They were:
That the bug was only discovered in 2005.
That the bug had only affected one branch. The second letter stated in very clear terms that “The Falkirk/Callendar Square issue was only known to have affected that one branch.”
Not only were both those statements in the Post Office’s solicitors’ correspondence completely wrong, but the numerous documents at [398] to [421] above demonstrate that there were a great number of references, including in both KELs and PEAKs (documents of a type that had been at one stage claimed to be wholly irrelevant to the issues in this litigation) to the direct contrary.
Mr Godeseth’s witness statement about this, prior to his cross-examination, presented a very different picture to the one that eventually emerged. Indeed, I would go further, and I find that his witness statements omitted some very important headline points in respect of the Callendar Square bug, presented a chronology very different to the real one, and had the effect (whether intended or accidental) of giving a misleading impression of the Callendar Square bug and its impact.
I find that the headline points omitted by Mr Godeseth are as follows:
The Callendar Square bug had existed “probably” (Mr Godeseth’s word in crossexamination) since the inception of Horizon in 2000. There were numerous incidents of it occurring prior to 2005.
By February 2006, Anne Chambers and others at Fujitsu knew that this bug “had been around for years”.
It had an impact “most weeks”. Preparing a software fix was not seen as a priority.
The bug had directly impacted the branch accounts of at least 19 branches, and possibly more, as the Anne Chambers’ spreadsheet listed the 30 branches as “these are the majority”.
There were numerous effects of this bug, and at least 13 different ones identified in the spreadsheet.
Even in 2006, Horizon Support was telling at least one SPM that they “could not find any problem” with Horizon when the bug’s impacts were reported to it, and this was recorded in documents linked by Fujitsu directly with the Callendar Square bug.
Fujitsu had not established the total number of branches affected by consulting the events logs for the purposes of preparing Mr Godeseth’s witness statement, or otherwise. To represent the spreadsheet prepared by Anne Chambers as being the totality of branches affected by the bug is to ignore the text of the notes.
Finally on this point, the Post Office’s solicitors’ letters were obviously not factually inaccurate. Nor can the Post Office’s own legal team have known about the Anne Chambers’ spreadsheet until very early 2019, otherwise disclosure of such an important document would surely not have been given only on 27 February 2019, only seven working days before the Horizon Issues trial began. I do not know why the Anne Chambers’ spreadsheet of 22 December 2015 was only disclosed to the claimants a mere 7 working days before the start of the Horizon Issues trial, and it is not necessary to speculate.
Although the Callendar Square bug in Legacy Horizon could, amongst other effects, cause a receipts and payments mismatch, there is another bug in Horizon Online that also had the same effect. This was called the Receipts and Payments Mismatch bug and is the first entry in the Bug Table, Bug 1.
Mr Godeseth gave evidence about the Receipts and Payments Mismatch bug. This was a problem at the cache. His information came both from Mr Jenkins and his own investigations. He did not know about it at the time in 2010, and he did not in his witness statement refer to what is in my judgment a very important document concerning this bug, namely the Receipts/Payments Mismatch issue notes dated in the trial bundle index 17 October 2012, although it is likely to be a 2010 document as an associated document from Mr Jenkins (trial bundle reference F/1000/1 and F/1001/1) is dated 29 September 2010. I will reproduce certain parts of the issue notes, as I consider them to be highly relevant to the Horizon Issues. There were 10 attendees, including Mr Jenkins. The document was marked Commercial in Confidence and is, according to the trial bundle index, a memo of a meeting. Another document with the same date is headed “Correcting Accounts for “lost” Discrepancies” and states “any branch encountering the problem will have corrupted accounts”. The issues notes document stated:
“What is the issue?
Discrepancies showing at the Horizon counter disappear when the branch follows certain process steps, but will still show within the back end branch account. This is currently impacting circa 40 Branches since migration onto Horizon Online, with an overall cash value of circa £20k loss. This issue will only occur if a branch cancels the completion of the trading period, but within the same session continues to roll into a new balance period.
At this time we have not communicated with branches affected and we do not believe they are exploiting this bug intentionally. (emphasis added)
The problem occurs as part of the process when moving discrepancies on the Horizon System into Local Suspense.
When Discrepancies are found during Stock Unit rollover into a new Transaction Period, then the User is asked if the discrepancy should be moved to Local Suspense. If the branch presses cancel at this point the Discrepancy is zeroed on the Horizon System.
Note at this point nothing into feeds POLSAP and Credence, so in effect the POLSAP and Credence shows the discrepancy whereas the Horizon system in the branch doesn't. So the branch will then believe they have balanced.
If at the next screen the rollover is completely cancelled, then no harm is done. However if the Rollover is re-attempted at this point, the rollover will continue without any discrepancy meaning Horizon doesn't match POLSAP or Credence.
This has the following consequences:
There will be a Receipts and Payment mismatch corresponding to the value of Discrepancies that were "lost"
Note the Branch will not get a prompt from the system to say there is Receipts and Payment mismatch, therefore the branch will believe they have balanced correctly.
When the Branch begins the new Branch Trading period the discrepancies will show at Zero, however the Receipts and Payment mismatch will carry over into the next period.
Impact
The branch has appeared to have balanced, whereas in fact they could have a loss or a gain.
Our accounting systems will be out of sync with what is recorded at the branch
If widely known could cause a loss of confident in the Horizon System by branches • Potential impact upon ongoing legal cases where branches are disputing the integrity of Horizon Data
It could provide branches ammunition to blame Horizon for future discrepancies.
Identifying the issue and forward resolution
The Receipts and Payment mismatch will result in an error code being generated which will allow Fujitsu to isolate branches affected this by this problem, although this is not seen by the branches. We have asked Fujitsu why it has taken so long to react to and escalate an issue which began in May. They will provide feedback in due course.
Fujitsu are writing a code fix which stop the discrepancy disappearing from Horizon in the future. They are aiming to deliver this into test week commencing 4th October. With live proving at the model office week commencing 11th October. With full roll out to the network completed by the 21st of October. We have explored moving this forward and this is the earliest it can be released into live.
The code fix will on stop the issue occurring in the future, but it will not fix any current mismatch at branch.
Proposal for affected Branches
There are three potential solutions to apply to the impacted branches, the groups recommendation is that solution two should be progressed.
SOLUTION ONE - Alter the Horizon Branch figure at the counter to show the discrepancy. Fujitsu would have to manually write an entry value to the local branch account.
IMPACT - When the branch comes to complete next Trading Period they would have a discrepancy, which they would have to bring to account.
RISK- This has significant data integrity concerns and could lead to questions of "tampering" with the branch system and could generate questions around how the discrepancy was caused. This solution could have moral implications of Post Office changing branch data without informing the branch.
SOLUTION TWO - P&BA will journal values from the discrepancy account into the Customer Account and recover/refund via normal processes. This will need to be supported by an approved POL communication. Unlike the branch "POLSAP" remains in balance albeit with an account (discrepancies) that should be cleared. IMPACT - Post Office will be required to explain the reason for a debt recovery/ refund even though there is no discrepancy at the branch.
RISK - Could potentially highlight to branches that Horizon can lose data.
SOLUTION THREE - It is decided not to correct the data in the branches (ie Post
Office would prefer to write off the "lost"
IMPACT - Post office must absorb circa £20K loss
RISK - Huge moral implications to the integrity of the business, as there are agents that were potentially due a cash gain on their system.”
(bold present in original)
I find this to be a most disturbing document in the context of this group litigation. It is a 2010 document and issues between the Post Office and many SPMs concerning the accuracy of Horizon had, for Legacy Horizon, gone on for a decade (2000 to 2010) and these continued under Horizon Online (introduced in 2010). Under “Impact”, some of the bullet points incorporate a summary of these issues.
“• The branch has appeared to have balanced, whereas in fact they could have a loss or a gain.
Our accounting systems [ie Horizon or the Post Office’s] will be out of sync with what is recorded at the branch
If widely known could cause a loss of confident in the Horizon System by branches • Potential impact upon ongoing legal cases where branches are disputing the integrity of Horizon Data
It could provide branches ammunition to blame Horizon for future discrepancies.” (emphasis added)
The attendees at this meeting included at least one member of Post Office (rather than Fujitsu) personnel, Andrew Winn of POL Finance. There were obviously legal cases going on at the time, hence the reference in the underlined bullet point to “ongoing legal cases”. If these were criminal cases, the Post Office would be the prosecuting authority, with certain important duties. If these were civil cases, the Post Office would be a party with disclosure obligations. An affected branch would believe it had balanced its accounts correctly; it would not have done so. There is an evident concern amongst those at the meeting which is recorded in this document that this issue should not become “widely known” in order to avoid causing “a loss of confidence in the Horizon System”. Fujitsu do not seem to have been particularly prompt in either identifying the problem or reacting to it.
Of the three solutions considered, the one that was adopted was issuing a debt recovery/refund even though there would be no discrepancy at the branch. Given one of the Common Issues was the status of a Branch Trading Statement, which the Post Office argued should legally have the status of a settled account with the SPM as agent, I note the suggestion adopted with more than passing interest. The solution explains “even though there is no discrepancy at the branch”. This means the Branch Trading Statement would be correct, or in balance. The discrepancy would not be in the Branch Trading Statement, it would be in POLSAP or Credence. That this is correct is shown in an associated document from Mr Jenkins (trial bundle reference F/777/2) where he stated that “the data used for the BTS will also have a zero value for Discrepancies at the end of the period.” BTS means Branch Trading Statement. A similar entry is in the document at F/1000/1 which states at /2 “Note that if the bug was not present, then the Discrepancy would have been transferred to Local Suspense and that would have been cleared, so there are a number of things wrong with the BTS.” (emphasis added). In other words, the Post Office itself was not considering the BTS as having the status of a settled account, based on these entries.
Further, an associated document entitled “Receipts and Payments Mismatch” states:
“1. The purpose of this note is to document a request that we have had from Post Office in terms of presenting details of what happened as a result of a bug in HNG-X
in September 2010 which caused a Receipts and Payments mismatch and also resulted in Discrepancies being lost.
The background to this is the fact that there was a BBC documentary broadcast on Monday 7 th February 2011 reporting on postmasters being unhappy about being pursued for losses by Postmasters on Horizon.
It should be noted that the issues described here relates to HNG-X (Horizon Online) and that the implementation of the accounting mechanisms in the two systems is totally different (but they do produce the same reports and support the same business process).”
(emphasis added)
The two references – one to “ongoing legal cases”, the second to a BBC documentary – show that there was a distinct sensitivity within both the Post Office and Fujitsu about keeping this information to themselves in order to avoid a “loss of confidence” in Horizon and the integrity of its data. A less complimentary (though accurate) way of putting it would be to enable the Post Office to continue to assert the integrity of Horizon, and avoid publicly acknowledging the presence of a software bug. The solution adopted was the issuing of TCs to the relevant branches. There was no publicity given to the SPMs at the time about the presence of this software bug in Horizon. An entry at F/1000/3 states:
“It should be noted that as Discrepancies are normally Losses, then a Lost Discrepancy would normally work in the Branches favour and so there is no incentive for the branch to report the problem. Also if we do amend the data to re-introduce the Discrepancy, this will need to be carefully communicated to the Branches to avoid questions about the system integrity.”
(emphasis added)
Mr Godeseth was asked about the lack of communication of this.
“Q. Are you surprised that this had not been communicated so people would just be warned about it?
A. There was obviously a fear that subpostmasters may be looking to exploit this because it gave -- there was a fear that people could see this as a way of defrauding the Post Office.
Q. So concealing it from SPMs who were honest was justified because of the expectation of dishonesty of subpostmasters in the network, in a nutshell?
A. In my view, this was a decision made by Post Office on how to manage this particular bug. You could interpret it the way that you have put it. Q. Do you agree with the way that I have put it to you?
A. I think I'm agnostic. I can see the -- I can see a rationale for not broadcasting this, but equally, if the objective is to be totally open and honest and take the risk of causing more chaos in the network then yes, I would have to agree with that.”
Indeed, the approach of the Post Office to notification of the existence of bugs to SPMs is an illuminating one. The Post Office communicates with SPMs generally in a number of ways. Methods that have been referred to in both of the Common Issues trial and also the Horizon Issues trial include “Branch News”, a type of newspaper that was used, and also “Memo View”, a pop up message that appears on Horizon terminals. During the preparation of this judgment, in view of the fact that judgments are required to be factually accurate, because dates can sometimes become lost in the midst of huge trial bundles and very detailed submissions, and also because the simplest way to arrive at a date when something has been done is (sometimes) simply to ask the parties for a date or a reference, on 7 October 2019 the parties were jointly asked the following:
“For each of the Dalmellington and Callendar Square bugs, could the parties please indicate (by means of an agreed fact but only if that is possible, if not each party’s contended for answer) their answers to the following two questions:
The date when the existence of each of these was communicated by the Post Office to SPMs;
A trial bundle reference to a contemporaneous document, email or communication from the Post Office demonstrating question 1.”
The parties could not agree on a date when the existence of these two bugs – Callendar Square being an acknowledged bug in the Bug Table dealt with below (from the 2nd Joint Statement of the two experts) and Dalmellington, which is now accepted to be a bug – was communicated to SPMs. The claimants’ answer to the two questions was the same for each bug, namely that there was “no date in any contemporaneous document from the Post Office in the trial bundle showing that such communication in fact took place and identifying the actual content of any such communication.” The Post Office’s answer was as follows:
For Callendar Square: “It has never been Post Office’s case that, once it was discovered, the existence of the Callendar Square bug was communicated by Post Office to all SPMs or communicated to SPMs who had been affected. Post Office's case is that it was unnecessary and would have been inappropriate to do so.” A number of references and further explanation then followed, stating that when “an SPM was affected by the bug, this was detected and corrected in the ordinary course” and also that “In some cases, SPMs who were affected by the Callendar Square bug may have been made aware that there was a problem in Horizon which had caused an error in their accounts which required correction.”
For Dalmellington: “It has never been Post Office’s case that, once it was discovered, the existence of the Dalmellington bug was communicated by Post Office to all SPMs or communicated to the SPMs who had been affected. Post Office's position is that it was unnecessary and would have been inappropriate to do so.” The Post Office also stated that “when an SPM was affected by the bug this was detected and corrected in the ordinary course” and that the Dalmellington bug “was prevented, by effective counter-measures, from causing even a single impact on branch accounts”.
The Post Office also stated that there were real disadvantages to notifying SPMs of this. The Post Office relied upon passages in Dr Worden’s report that summarise reasons why users of an IT system should not be given information about parts of that system which they did not encounter in their daily work; that there was “no point in trying to educate all the users in details and terminology of the system which will never concern them”; that the best thing to do is try and fix a bug or defect, rather than “create some new error message to the users”; that “automated messages from the system are only of limited help to users”; and other points in similar vein. The Post Office also provided documentary references to instances where “an SPM might have been made aware”, although these documents were not deployed in the evidence.
I will reproduce two entries from one of these documents, a PEAK with reference PC0103864. This shows that on 3 June 2004 an SPM reported “that he had a problem
with some transfers yesterday, he was transferring stock and cash between the aa main stock unit and the bb shared stock unit and although only one transaction shows for the transfer out the transactions were transferred into the bb stock unit twice giving the pm a discrepancy.”
The discrepancy in the cash account was £22,290 in total (shown in the entry for 8 June 2004) although the reconciliation error reported by the Host was £44,580. This was finally resolved on 5 August 2004 with the issue of an error notice. There is nothing to suggest that the SPM was told that this had been caused by a software bug.
The Post Office’s approach to this, in my judgment, entirely misses the point. In my judgment, the above passages are simply extraordinary. Two of the documents above are dated 2010, some 9 years ago. The PEAK is from 2004, 15 years ago. Their contents support the claimants’ case on the Horizon Issues. Fujitsu knew, to take Callendar Square as an example, that this bug existed in Horizon. They knew that it had affected branch accounts. It was not, as the Post Office puts it “unnecessary and inappropriate” to notify SPMs of this. I have listed the points on this bug at [425] above omitted by Mr Godeseth from his written evidence. Those same points all lead to the same conclusion in my judgment, namely that the Post Office ought to have notified, at the very least, all those SPMs whose branch accounts had been impacted by this bug that this had occurred, and that it had occurred as a result of a software bug. The fact that the integrity of Horizon data was a live issue at this time should not have influenced the decision to notify SPMs of a software bug. Further, the Post Office’s explanation in its submissions that SPMs had their accounts “corrected in the ordinary course” is not a suitable phrase, unless by “ordinary course” one means keeping the cause or reason for the correction secret and therefore hidden from the other party in the accounting transaction, namely the SPM. Also, one is not educating users in the details and terminology of the system (as suggested by Dr Worden) if one informs them that there is a software bug in the system and its symptoms are as follows. This is relevant to Horizon Issue 2, namely whether the Horizon IT system itself alerted SPMs of such bugs, errors or defects as described in Issue 1 above and if so how, and also Horizon Issue 9, which concerns transaction data and reporting functions available to SPMs. It is also relevant to Dr Worden’s consideration of countermeasures, which he considers includes vigilance by users (which means SPMs).
Mr Godeseth gave express evidence in his witness statement that the Receipts and Payments mismatch bug occurred in September 2010. That date too was factually incorrect. The issue notes refer to Fujitsu knowing about it far earlier, and Mr Godeseth accepted he had seen this document before his cross-examination. That document even records Fujitsu being taken to task by the Post Office about how long it had taken to react, as in “We have asked Fujitsu why it has taken so long to react to and escalate an issue which began in May. They will provide feedback in due course.” Not just the impression, but the express text in Mr Godeseth’s witness statement, was to the effect that the bug was discovered in September 2010 and almost immediately dealt with. That was far from the case, and that written evidence was simply wrong.
Mr Godeseth gave evidence about another bug, called the Local Suspense Bug, also in Horizon Online. His witness statement quoted almost word for word from a contemporaneous document prepared by Mr Jenkins. Almost all his information on
this bug had come from Mr Jenkins. The bug caused entries from local suspense accounts in 2010 to be reproduced in Horizon in two successive years. What was supposed to be only temporary data was retained, not deleted as it ought to have been, and the system then used it again. It affected certain tables in the branch database and the archiving strategy of deleted Stock Units. This affected 15 branches. The effects of the bug were brought to the attention of the Post Office by two SPMs “with the largest discrepancies” and Fujitsu became aware of it in January 2013. However, the note by Mr Jenkins of 15 May 2013 summarising it states that the problem was known about far earlier than that:
“In April 2011 a problem was found with the archiving strategy related to Stock Units that have been deleted in a Branch. A consequence of this is that some changes were made to the archiving strategy on 3rd July 2011. An unintended consequence of this change was that any Branch that deleted a Stock Unit at the end of 2010 which had local suspense transaction in that Stock Unit before it was deleted were left in the table used for constructing the BTS. This meant that as Trading Periods cycle around each year, these BTS records became visible in 2011 when the same Trading Period was reached.
The effect of these old records was that after the BTS was produced an incorrect figure was generated for the Opening Balance of the Local Suspense Account for the following period. This amount corresponded to the value of the historical record. These orphaned records were created between 16th November 2010 and 9th December 2010.
When the next Trading Period was balanced, then this incorrect Opening Figure would result in the total value for Local Suspense being calculated incorrectly and the SPMR being asked to make good an incorrect amount. It is at this point that transactions would be generated into the audit trail reflecting the fact that the SPMR had cleared the Local Suspense account for an incorrect amount. The audit trail operated correctly in the sense that it accurately recorded the transaction on the system.
This problem was not reported to Fujitsu in 2011/12 and only affected a small number of Branches and only for a single Trading Period. However the two branches with the largest discrepancies did report the issue to Post Office Ltd who could see the impact of the problem in their back end system and wrote off the loss or gain for the branch but did not ask Fujitsu to investigate further.
At the same Trading Period in 2012/13, the problem re-occurred and this time one of the affect Branches reported the problem to Fujitsu on 25th February 2013 (PEAK 223870) resulting in a detailed analysis of this issue and finding the orphaned BTS records. The root cause was determined by 28th February 2013 and a preliminary report was sent to Post Office Ltd. A further update was sent on 14th March 2013 with a full analysis of the issue and all the affected branches.”
(emphasis added)
Another software bug which Mr Godeseth gave evidence about was called the Dalmellington Bug. This is also known as the Branch Outreach Issue or Bug, as it affected what are called Outreach branches, which are those that have a core branch
(such as a rural or small branch post office) and an outreach branch, such as a mobile post office van, or village hall somewhere else (where post office services would be provided, say, one day a week). This “outreach” element is used, particularly in very rural areas, to provide post office services to remote communities. A SPM needs to scan or record a pouch from one branch (the core) into the outreach branch (the van or village hall) to record that the contents of the pouch are not in the core branch, but are in the outreach branch instead. This is a branch to branch cash remittance, if the pouch contains cash. Mr Godeseth had no first hand knowledge of this bug. As the claimants put it in opening submissions, this is a useful way of examining certain features in Horizon as the same person, the SPM, controls “both ends” of the transactions.
The Dalmellington Bug related to Horizon Online and actually included two potentially separate issues, and the combination of these. One related to what is called Forced Log Out, when the Post Log On script was not correctly closed down and was left on the stack of incomplete processes. The other related to the way the Pouch Delivery script operated; because the Post Log On script had not correctly closed down meaning the stack was not empty, the Pouch Delivery script thought it had not finished and attempted to repeat the last part of the script. This had the effect of recording the remittance transactions and printing of receipts. This would lead to duplicate pouch IDs. Pouches have bar codes that are scanned and each is supposed to have a unique ID. The duplicate pouch ID would have a value attached to it – the sum in the original pouch. It would however be recorded twice as a result of this.
What happened at Dalmellington was the SPM scanned £8,000 from her core branch into her outreach branch. She was lucky in this sense, in my judgment, in that she “controlled” both ends of the transaction. The transfer replicated four times. She had Horizon receipts in her outreach branch of £32,000 (4 times £8,000) and therefore a discrepancy of £24,000, as she had only transferred £8,000 in cash out of her core branch into it. This was investigated by SSC and she was issued with a TC a few weeks after it had occurred. This was caused by a software bug, and the bug is now admitted by the Post Office, although it describes it as having “transient impact”. That simply means it was corrected.
When investigating this Fujitsu found 112 occurrences affecting 88 different branches in the previous 5 years. They were as follows, taken from an internal Fujitsu presentation dealing with this:
Feb 2010 to Jan 2011 65 incidents
6 incidents
9 incidents
7 incidents
9 incidents
16 incidents
The same document showed fixes applied in April 2010, January 2011 and January 2016. It also showed only one call to SSC at Fujitsu in 2015, and none for any of the years 2010 to 2014. The presentation also showed that some branches that were affected were so-called “mediation branches” – in other words the SPMs would have been in the Second Sight mediation scheme that was brought to an end – but that none of the dates when those branches were affected matched the “mediation dates”, which means the periods for which ARQ data was to be used in the mediations to analyse exactly what had happened at each branch. There is nothing in the Fujitsu document used by Mr Godeseth for his evidence, to suggest that this analysis led to each branch being told of these software bug occurrences outside the mediation date range, and the items were usually corrected by means of TCs.
Mr Godeseth said that “Unfortunately this particular error is not subject to receipts and payments problems because it could be -- it could be the user doing something twice, it's -- the bug had the effect of making it look as though a user was simply doing something multiple times.” (emphasis added) He said that “the issue was not raised with Fujitsu until 2013”, it was put to him it was 2015 (based on the slides) but regardless of which of these dates is right, the Fujitsu slides show only one call to Fujitsu in 2015, none for the other years, although the reference to fixes in 2010 and 2011 suggests it was reported to Fujitsu earlier. If those fixes were intended to correct the bug they cannot have worked. Fujitsu maintained that they investigated and found all occurrences, and the Post Office maintained that they had corrected any financial impact of this bug. Whether this is correct or not in all cases, the evidence demonstrated that the effects of this bug were experienced for the period from 2010 until 2016.
In his re-examination, some of which I have already referred to in the context of UNIX user and privileges above, Mr Godeseth also said that having seen the OCPs and OCRs in cross-examination, he accepted data had been deleted but did not consider it to be transaction data. Initially he described it as “operational data” but said that was too wide a term, and:
“There possibly ought to be a different definition for it because equally data in the database which is telling you which stock unit, you know, what -- which TP [ie trading period] you're in, whether a stock unit is in use, all of those are operational data which do not go down and impact the transaction data, so for me, the definition of transaction data -- I should be able to get from one opening -- one set of opening figures to the end opening figures by going back to the transaction data in its absolutely raw form.”
He said it was what one went through to bring different totals together “to get to your end position in a trading statement.” Regardless therefore of whether the data which he accepted was being deleted is, or is not, properly described as “transaction data”, it would feed into the end position in a branch trading statement. It would therefore, in my judgment, have an impact upon branch accounts, regardless of the term used to categorise it.
By the end of Mr Godeseth’s evidence, and by virtue of the way he answered questions, the reality of the development and operation of both Legacy Horizon and Horizon Online was far clearer than it had been before the Horizon Issues trial. I
found Mr Godeseth to be a truthful, credible and helpful witness in his oral evidence. His evidence was of considerable assistance; he did not always answer questions the first time he was asked, but the final answer would be reached. Given his role throughout the whole of the life both of Legacy Horizon and Horizon Online, I found his evidence in cross-examination to be of central relevance to the Horizon Issues. His evidence in cross-examination was also, in my judgment, broadly supportive of the claimants’ general case.
There are, however, two aspects of Mr Godeseth’s evidence – though not his performance as a witness in the witness box - that were highly unsatisfactory and cannot pass without comment. The first aspect relates to his written evidence. The totality of his evidence, after his cross-examination, bore very little semblance to the picture that had been portrayed in his written witness statements. On some extremely important factual matters, such as the dates when Fujitsu had become aware of a particular bug, or the spreadsheet exercise by Anne Chambers prepared in 2015 (and disclosed in 2019), his written evidence was simply directly wrong. On others, such as the headline points omitted on the Callendar Square Bug that have been set out in [425] above, very important central elements detrimental to Fujitsu were simply omitted.
Legacy Horizon had, as has been explained, started life originally as something rather different to what it became, having initially been intended as a tri-partite project involving payment of benefits. It did not unfold in this way and became rather different. Horizon Online also did not have a happy birth. The pilot for it had to be stopped, and Fujitsu put it on what was called “red alert”. Mr Godeseth described this as “very serious”. The biggest issue was with Oracle, which was what Mr Godeseth was working on and hence knew the most about, but he explained that there were other problems going on at the same time. Some of these problems were put to him – and it must be remembered that this was a pilot scheme, with some problems to be expected – and they included cash being short on one day by £1,000 because a transaction for £1,000 did not show up on the online report facility; cash withdrawals being authorised on screen yet the printed receipt being declined (the customer very honestly brought the cash back next day having noticed the receipt wording); a similar problem with a cash deposit; and remming in figures all being doubled up. These are all somewhat – and indeed markedly - similar to some of the problems alleged by the different claimants’ witnesses in this litigation. These all occurred during the pilot scheme.
Not one of these different problems was referred to in Mr Godeseth’s witness statements, of which there were three. Witness statements are supposed to be accurate, and in a case such as this one with such centrally important issues, accuracy is clearly important. Quoting only selectively from, or wholly ignoring, contemporaneous documents prepared by (say) Mr Jenkins, who was the extensive source of much of the evidence, is not only unhelpful, it presents an entirely misleading evidential picture. It is not necessary to consider further how many personnel at Fujitsu may have assisted Mr Godeseth in producing such documents. Their content was wholly misleading in their original written form. Fortunately the cross-examination of Mr Godeseth led to a far clearer picture in so far as his evidence is concerned.
The second unsatisfactory aspect which arose from his evidence is the approach of Fujitsu as demonstrated in various documents, including the PEAKs and KELs, but also in particular in the Receipts/Payments Mismatch issue notes. To see a concern expressed that if a software bug in Horizon were to become widely known about it might have a potential impact upon “ongoing legal cases” where the integrity of Horizon Data was a central issue, is a very concerning entry to read in a contemporaneous document. Whether these were legal cases concerning civil claims, or criminal cases, there are obligations upon parties in terms of disclosure. So far as criminal cases are concerned, these concern the liberty of the person, and disclosure duties are rightly high. I do not understand the motivation in keeping this type of matter, recorded in these documents, hidden from view; regardless of the motivation, doing so was wholly wrong. There can be no proper explanation for keeping the existence of a software bug in Horizon secret in these circumstances.
The degree to which either, or both of, Fujitsu and/or the Post Office, expressly or constructively, knew exactly what and when, is for future trials in this litigation, and I make no findings in that respect in this judgment. They are not necessary in order to resolve the Horizon Issues and I do not speculate.
In my judgment, however, there are sufficient entries in the contemporaneous documents to demonstrate not only that Fujitsu has been less than forthcoming in identifying the problems that have been experienced over the years, but rather the opposite. The majority of problems and defects which counsel put to Mr Godeseth, and which were effectively admitted by him, simply would not have seen the light of day without this group litigation.
I do not know if this is because of concerns about the future of Horizon going forwards; or for Fujitsu to protect its corporate reputation; or for some other reason. It is unnecessary to speculate, and I do not do so. An internal Post Office IT risk management document from 2017 stated that “the HNG-X platform is end of life and is running on unsupported Windows software”, that it needs replacing, and also that the "Branch counter technology is aged and unreliable, with frequent hardware failures, resulting in branch disruptions." Mr Godeseth agreed with all of this. The unsupported platform is Windows NT4. As Mr Godeseth put it, “any technologist would tell you that was too old”. Windows NT4 is an operating system that was released in 1996, and Microsoft ended mainstream support for it in 2002 and extended support in 2004. Horizon now runs, not on NT4, but on Windows 10 and it is called HNG-A rather than HNG-X. This change to Windows 10 occurred in February 2017 and the roll out was administered by Computacentre, with Fujitsu providing the software. The Horizon system of 2019 is therefore a very different system than both Legacy Horizon and the earlier version of Horizon Online, HNG-X.
The original Horizon Online system was introduced in 2010, and was even then substantially based on the existing system, Legacy Horizon, itself by then at least 10 years old. That had been introduced in 2000, and therefore it should not be controversial to describe Horizon Online in its HNG-X form as old technology. There have of course been upgrades and other improvements since 2010, and other software additions and add-ons, but Horizon will by necessity require replacement at some point. I detected a degree of sensitivity in the litigation generally about any plans for replacing Horizon Online, as though the Post Office feared that any hint that it might
be replaced would be seized upon by the claimants (and/or assumed by the court) as accepting that the criticisms of Horizon Online by the claimants would be corroborated by any Post Office plans for its replacement. Such fears, if there are any, are groundless, at least from the court’s point of view. It is inevitable that technology systems, particularly complex systems from as long ago (in technology terms) as 2010 will need replacing at some point. Technology moves at an increasingly rapid pace. There is reference in some of the documents in the trial bundle, but not in the evidence, to something called “Thin Client”, about which Dr Worden said he had no knowledge. He thought it was a cloud based system. It is unnecessary to speculate about this either. It is sufficient to state that, in my judgment, any plans there may be for replacement of Horizon Online in the ordinary course of the Post Office’s wishes to improve its business are not something that I need to consider in order to resolve the Horizon Issues.
The degree to which Mr Godeseth’s evidence affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Very slightly before the end of Mr Godeseth’s cross-examination – about 30 minutes or so – the Post Office sought to bring the Horizon Issues trial to an end by issuing their recusal application seeking to remove me as the Managing Judge, and to have the trial re-started at some indeterminate point in the future before another judge. There was therefore a substantial interruption in the trial, and a period of 22 days before the Horizon Issues trial resumed and the next factual witness for the Post Office, Mr Parker, was called.
Mr Parker
Mr Stephen Parker is also a Fujitsu employee, and is now the Head of Post Office Application Support. He is therefore a very senior person. He first started work on what was then called the Royal Mail Group Account in 1997, which was before the introduction of Horizon. He has continued to provide support to the Post Office Account in the various roles he has occupied at Fujitsu throughout the whole of Horizon’s life, by which he meant both Legacy Horizon and Horizon Online.
He was originally a support consultant within the SSC, and stated that he was effectively the deputy manager of the SSC, providing 3rd line support. He was lead designer and part of development team for the internal website providing the support knowledge database (which is also called KEL), technical documentation management and operational change control. He assisted the SSC manager in the provision of the support service and operational management. He became the Manager of the SSC in March 2010. The SSC later expanded and provided support services to other Fujitsu customers, but the Post Office Horizon system is still the largest. He has between 25 and 40 staff in the SSC whom he manages. He said that he was deputy manager of the SSC when Mr Roll worked there, although he did not have that title formally, and that people in the SSC would know this because he would stand in for the manager in his absence, and make decisions on approving actions for him.
He had provided three different witness statements for the Horizon Issues trial, and his 2nd and 3rd statements were further to, and ultimately corrected, his earlier evidence, which arose in respect of what Mr Roll said had been done whilst he had been at the SSC, including injecting transactions during the time of Legacy Horizon. These were done using Riposte, the language then in use. During the Horizon Issues trial and just before the issuing of the recusal application, the Post Office’s solicitors wrote a letter to the claimants which stated the following. This letter was dated 20 March 2019, and was sent the night before Mr Parker was originally due to be crossexamined, his evidence then becoming delayed due to the recusal application:
“We understand from Fujitsu that the SSC has been carrying out further work to identify any PEAKs that show transactions being injected at the counter in Legacy Horizon in addition to those referred to in paragraphs 29 and 30 of Mr Parker's second statement. On Monday [ie the 6th day of the Horizon Issues trial] we learned that an SSC technician has:-
searched for all KELs that mentioned "RiposteMessageFile", "Ripostelmport" and "RiposteMessage';
collated the responsive KEL references (AChambers2226M, CObeng1029162824, DSeddon822M, MYoung5043M, RColeman1250R, acha2340K, ballantj498J, dsed344J, AHolmes3343J, DSeddon1753N, GMaxwell46141, PCarroll12541, RKing5135L, 81111, pcar847S, wbra716s);
re-searched the Peak system for any PEAKs which contained those KEL references; and
identified the following PEAKs: PC0105560, PC0106885, PC0063599, PC0063871, PC0065796, PC0066061 and PC0083998 (which cross refers to PC0076029).
We understand that the PEAKs referred to above relate to either:- • correcting configuration data after a PinPad change; or
reference data.
We also learned that during this exercise the SSC technician has also identified three examples of the marooned transaction scenario described in paragraph 38.2 of Mr Parker's second statement: PC0068495, PC0099141 and PC0079196.”
The search terms identified in the letter were different to the ones that Mr Parker had used in his 2nd witness statement. These were "RiposteMessageFile", "RiposteMessage", "LPO Delete", "Marooned" and "RiposteObject put". The reason this is important is the command “RiposteImport”, used by the SSC as a search term during the week commencing 18 March 2019, but not used for the purposes of Mr Parker’s 2nd witness statement in January 2019, is actually one of the commands that was in fact used to inject transactions. In other words, the exercise carried out for the purposes of the witness statement was wholly deficient, in that it did not search for one of the specific terms used to inject transactions.
When asked to explain this in his cross-examination, and the corrections that he made, Mr Parker stated the following:
“A I notified the legal team that Mr Simpkins had done some more work and as a result of that we changed -- we -- this letter was actually generated.
Q And you didn't think it was important to correct paragraph 29 of your second witness statement accurately to reflect the situation as it would have been on the day you were due to give evidence, or indeed today?
A That wasn't a choice I made personally. I was advised that we generated this letter.”
(emphasis added)
He did not say who made the choice to which he referred, namely not to correct his witness statement.
It certainly seems to me highly curious – at best - that the exercise performed for Mr Parker’s 2nd statement, by personnel at SSC who are expert in such things, should have omitted using the search term “RiposteImport”. Given the purpose of paragraph 29 and 30 of Mr Parker’s 2nd witness statement was to identify how many times data had been in fact injected, a search exercise that did not use “RiposteImport” was never going to give the full picture. That initial search exercise was plainly inadequate. It is also rather curious that, having done that, the exercise that was described in the letter of 20 March 2019 from the Post Office’s solicitors that correctly used “RiposteImport” as a search term should have been done so very closely before Mr Parker was originally due to be called as a witness, namely 21 March 2019. In re-examination, Mr Parker said that “the thought occurred to him that he could add some other search terms into the work that he had done previously” and “he came to me with the data”. Why this thought should have suddenly occurred to the person it occurred to, who was not a witness, and at the time that it did, could not be pursued. Mr Parker denied that he was trying to conceal anything, and said he was just trying “to get the right data for the court which can be difficult sometimes when you are going back fifteen years.” This explanation overlooks that the decision not to use “RiposteImport” was one taken for his witness statements. His first witness statement was dated 16 November 2018; his second was dated 29 January 2019, and his third was 28 February 2019. None of these are “going back fifteen years”.
Mr Parker had also given evidence in his 1st witness statement in respect of Mr Roll’s written evidence that Fujitsu could inject and affect branch transaction data without the knowledge of the SPM. He had originally said the following:
"As I explain below, those suggestions are incorrect and Mr Roll's account of Fujitsu's actions and powers is inaccurate and misleading".
He explained this paragraph as follows in his cross-examination.
“A. In that paragraph I am trying to make the point that the suggestion that we frequently changed branch transaction data without informing the branches that such actions were being actually taken is not correct. "Frequently" is a subjective term but I would not have described the rate at which we were changing branch transaction data as "frequently".”
One has only to read the original paragraphs as drafted to see that Mr Parker was not originally taking issue with frequency, or how often such actions were taken; he was stating in express terms that this could not be done by Fujitsu at all. For someone who has effectively spent the best part of 20 years in SSC, rising to a very senior post, such evidence must have been – and I find that it was - quite obviously incorrect. His explanation of this was as follows:
“Q Now, given the change of your evidence in your later statements about the ability remotely to access and inject transaction data, can you explain to the court what you were saying you believed about that sentence? What should the court now read for those words in the light of the three statements that it has before it?
A If I take the last sentence in isolation, which is what I think you are asking me to do, then I don't understand how I apply it, because I am -- I have been simply trying to say there that the frequency was not high, and that we would always involve the
Subpostmaster wherever possible if that sort of action was actually being taken. That's what I'm trying to say by that.”
This was a very poor attempt by Mr Parker at explaining away quite directly incorrect factual evidence in a witness statement on a very important area in the litigation, namely Fujitsu’s ability to inject transactions remotely. Mr Parker’s only recourse was to claim he had not been saying what he plainly had been saying in his 1st witness statement – that Fujitsu did not have the power to do it, and that Mr Roll’s evidence to the contrary was inaccurate and misleading. The power to do something cannot be equated with frequency.
Mr Parker also accepted that at the time he signed his 1st witness statement he had known that Fujitsu had the power to insert transactions into individual branch counters by using the correspondence server, a process Mr Roll described as “to piggy back through the gateway”. It was put to him that he had been giving his evidence initially from a position of not being very well informed about what could or could not be done. He denied that; he said “that would be wrong” and that in general terms he was confident of the information he gave. The obvious conclusion that I draw from this is that Mr Parker chose specifically to give the impression in his 1st witness statement that Fujitsu did not have the power (the word Mr Parker expressly chose) to inject transactions into the counter at branches, even though he knew that it did. This paints him in a very poor light as a credible witness. There is also no adequate explanation for why Mr Parker’s colleague should have decided, in the middle of the trial, to use the “RiposteImport” search term; equally, there is no adequate explanation for why this search term was not used for the exercise for Mr Parker’s witness statement. Injection of messages into the counter has been an extremely central issue in the long running dispute between the Post Office and its SPMs for a long time, and Mr Parker’s reliance in his re-examination upon the passage of time and the difficulty of “getting the right data” for the court is, in my judgment, a poor excuse. In reality, he had no credible explanation.
A series of propositions was put to him about the degree of expertise expected of all the members of SSC 3rd line support. He did not agree with all of the points put, and in particular did not agree that all those in the 3rd line support role would be expected to have detailed knowledge of the system, based on both documentation and the inspection of source code; or that they would be trained in the coding languages used within the application. He was, however, left rather exposed in terms of the accuracy of this evidence when it was shown that the points that he had been addressing in this part of the cross-examination – some of which he would not accept - were taken from a Fujitsu internal document, entitled “End to End Application Support Strategy” marked Fujitsu Restricted and Commercial in Confidence which he had himself drafted. Indeed, it is page 1 of that document that specifically identifies Mr Parker himself as the author of the document, and it was approved by the Head of Application Services at Fujitsu, an obviously very senior role. This refusal by Mr Parker to accept his own previously drafted points in what I consider to be an important contemporaneous document also paints him in a very poor light as a credible witness.
He accepted that Mr Roll was conscientious, skilled and experienced, and that he had given Mr Roll a reference when he left Fujitsu.
He, and others not identified in his statement (although he said who they were when he was asked, Mr John Simpkins and Mr Mark Wright), had prepared a spreadsheet that had been put to Mr Roll showing the total number of calls to SSC between 2001 and 2004. Mr Simpkins, for what it is worth, appears to have been the person behind the exercise referred to in the letter at [466] above, so he was obviously involved both in preparing evidence and doing searches. Mr Parker agreed that SSC during Mr Roll’s time needed to address coding issues, even if they did not necessarily have an estate wide impact, and he also agreed that the list of PEAKs that had been identified as involving Mr Roll was only those where he had been the person who put the final response on the PEAK, rather than where he had worked on the PEAK at all.
He also accepted that in categorising “software errors” he had excluded from the count those that were categorised as Avoidance Action, Administrative Response and Solicited Known Error.
He was taken to a PEAK in 1999 where an initial balance was multiplied twice “due to a known software error” in the amount of a discrepancy of over one million pounds - £1,082,544,32 to be exact. The PEAK ran to 14 pages. 10 people worked on it including those in Development as well as the SSC. He accepted that this would have involved people looking into code. It was, however, categorised as “Administrative Response”. That PEAK alone demonstrates to my satisfaction the lack of inaccuracy in the Fujitsu closure codes, but there were many others. Other PEAKs to which he was taken showed reconciliation errors, and a note by the developer identifying that what were “up to 10 unnecessary reconciliation errors each week” and that the PEAK was a regression of another PEAK; the investigation into it led to the conclusion that a subsequent software release had not caught a fix. It stated "Risks (of releasing and of not releasing proposed fix): Without this fix, there will be possibilities of system errors at counter and while doing reversal transaction". This software fix was “released to live”, which means the fix was actually written and introduced into the system. Mr Parker had not included this in his software category either, because the final person working on it had categorised this as Administrative Response.
In another PEAK, it was recorded that “the system is still playing up in that the screen is hanging in the middle of transactions -- PM did transaction ... but left office for 1 hour -- when he came back the monitor had 141 first-class stamps on screen totalling £38.07". Mr Parker accepted that this was not how the system was supposed to work, but the same PEAK also recorded the SPM becoming upset because the SSC were advising it was user error. The SPM wished to make a complaint about the person at SSC and the PEAK records “we both feel that this PM is complaining unjustly”.
The same PEAK recorded that ROMEC engineers had been to the site, and had actually seen the problem themselves, which were phantom transactions. It recorded
“it’s not just the PM’s word now”. Indeed, this PEAK is particularly concerning, in terms of the accuracy of PEAK closures by Fujitsu. It records in one of the Fujitsu entries the following:
“I now have pressing evidence to suggest that unwanted peripheral input is occurring, the likely source being the screen. This has been seen at Old Isleworth ...When the PM has been asked to leave the screen on overnight I have observed system activity corresponding to screen presses happening with no corresponding evidence of either routine system activity or human interference. The way forward now is to correlate this with Microtouch complied monitoring software and to this ends Wendy is arranging for installation of the kit on Friday..."
However, notwithstanding this, the subsequent entries include “Phantom transactions have not been proven in circumstances which preclude user error" and “In all cases where these have occurred a user error related cause can be attributed to the phenomenon?”.
The use of the phrase “in all cases” in respect of attributing user error wholly ignores that the ROMEC engineer is said to have seen the problem themselves, and it is not simply the word of the SPM in this instance. It ignores the activity observed overnight by the Fujitsu employee who made the entry in the PEAK, when the branch was not even open. Also, there is no reason, given the lengthy entries and studying the text throughout the PEAK in detail, why user error should be reached as a conclusion at all. This appears to be been used simply as an excuse to avoid an adverse conclusion to Fujitsu.
Notwithstanding the detailed entries in the PEAK itself that in my judgment plainly point to the contrary, the PEAK was still categorised as No Fault in Product.
In another, dealing with phantom transactions, one branch was described as “a problem office” that was accordingly being “monitored”. The PEAK recorded that “there have been incidents which showed a possible correlation between system activity and phantom transactions. These pointed to a touch screen problem and as a result the screen was replaced with a resistive model. As this produced no measurable improvement it has to be assumed that the problems were user related". This was also closed as No Fault in Product.
A KEL on phantom transactions stated "There have been several calls over the last few months where Postmasters have reported phantom sales. Items appear by themselves for which the PM has not pressed an Icon. These may be individual items or several of the same item. Sometimes when no one has been near the screen items may appear". These are similar problems being reported by a number of different SPMs. That alone should have suggested, not as a matter of computer expert evidence but simply basic common sense, that something was amiss. If a number of different and unconnected people all report the same symptom, then that could well suggest a common problem to them all. However, the KEL, which ran from 2000 to 2004, suggested that the cable between the screen and the base unit was the root cause. The dates on this KEL show that, even on the face of the document, the problem was manifested for some years.
Another PEAK that showed a problem described as “non-zero trading position on office rollover” was put to Mr Parker. This had been given a priority C status by Fujitsu, a status he accepted meant it was not taken account in the Service Level Agreement with the Post Office in liquidated damages thresholds (a type of financial penalty), which was a feature of those software PEAKs given higher priority status A and B. Status C also was non-critical. The description of the problem was described as “PM states that he has rolled over but the system is telling him that he hasn't -- PM states that he is in balance period 7 and he states he is getting the message 'wrong trading period MSG 31318 office balancing error'" and "Non-zero trading position ... on rollover of branch by user WMC002 to trading period 8".
Mr Parker described himself as “hesitant” to describe this as a payments mismatch issue, and said it was an “office balancing error”, even though later in the PEAK senior personnel at SSC recorded “NBSC have ruled out user error” and also that "The problem occurred on 15/09/10 when stock unit 02 rolled over. This was originally reported, as per KEL, BALLANTJL759Q, in call PC0204537 ... but for some reason the call was closed without being investigated. There is a known problem with the use of the Cancel button during the stock unit rollover. This is fully described in KEL WRIGHTM33145J".
(emphasis added)
He said he had not recognised the KEL number as being the payments mismatch
KEL. He accepted that in another entry in the PEAK the matter was stated to be “Severity: Critical” – which meant it could not be status C, defined as non-critical – and also included the entry “the branch accounts will need to be corrected.” This PEAK was however, on 1 November 2010, closed as category “Avoidance Action Supplied”. Mr Parker’s spreadsheet did not include “Avoidance Action Supplied” as constituting a software issue. He claimed that he had used “the definition my experience dictated rather than that in the document” and that “I forgot that that was exactly what I had done in that circumstance when I prepared that a few months ago.” Finally, after the question was put a third time, he accepted that it was a deliberate decision on his part to depart from the definition in Fujitsu’s own documentation.
Another PEAK also identified as category C, non-critical, related to pouches being remmed in twice, the duplicate pouch being £25,000 in value. The PEAK stated “this might be a software issue” but also “PM is unsure if this is user error if it is a software fault. NBSC have not confirmed if it was user error or if they can reverse it.” One entry stated "POL have been informed of the error. Hopefully they'll issue a TC to correct loss at the branch. The underlying problem caused by using previous button during or just after scanning pouch barcodes, is still under investigation". This was closed as a Solicited Known Error.
One PEAK which I found most illuminating, in terms of Mr Parker’s response to it when he was asked questions, is PEAK PC0229446. This is dated 17 November 2013, hence is Horizon Online, not in the period of the pilot project and sufficiently after the introduction of it, in 2010, for it not to be put down as the sort of issue that might occur in very early days of a system whilst minor issues were ironed out. The problem
was described in the PEAK as “PM doing cash declarations every now and again has major loss.” Entries in the PEAK included "PM has had cash declaration problem throughout the year and is losing a lot every now and again"; "He phoned up helpline told him can't [have] declared properly. He states that he loses £2,000 then jumps suddenly to £4,000, one point they lost £8,000 and is always losing money. PM stated that he has three post offices, only happens on this site"; and then "Done a declaration this morning and had a £6,000 also. It shows no error message when doing it. No report prints out only print-out of cash declarations". This passage of his crossexamination followed this:
“Q. Pausing there, if the PM is correctly reporting that, then that would be very serious for the Postmistress or Postmaster, wouldn't it? A. If it is being correctly reported, yes.
Q And it would not be the system working as it should.
A If we attribute it as a system fault, yes.”
This again was categorised by Fujitsu as Category C, non-critical. In a sense, those two questions and answers sum up a large part of the case. The reason that this plainly, in my judgment, does demonstrate a system fault is Fujitsu’s own PEAK recorded:
“Voiced NBSC quoted ref H18174172 to see what checks they have done themselves before transferring call to Horizon.
They stated they had trainers come into the office and ruled out user error.”
And
“NBSC states the user error checks were carried out by Auditors at the site and not over the phone.” (emphasis added)
This matter was assigned to Anne Chambers, whose name appears very often in many of the documents. The final part of her conclusion was (having said that “there were no known issues that would result in the variance being incorrect”):
“I can't tell why the declared cash doesn't match the expected cash figure, the branch need to make sure that what they have recorded on the system is correct, and investigate the anomalies.”
(emphasis added)
This is simply passing the buck back to a SPM in a branch. How an SPM could “investigate the anomalies” given they appeared to be generated by the Horizon system is not explained, nor could it sensibly be.
She also closed the response with the category “Final – no fault in product” and closed the call by defect cause “General – User”. I consider this PEAK to be an ideal illustration of what I consider to be the most extraordinary situation at SSC, and one which on the face of it is difficult to explain, given the function of SSC was to investigate faults:
Fujitsu routinely assigned non-critical Category C to matters that were really very important in their own right in any event, but of extreme importance to SPMs whose branch accounts were being directly affected. Mr Parker accepted that only Category A or B attracted financial penalties. It is not possible, on the evidence before the court, to conclude one way or the other whether it was this that affected the categorisation adopted by Fujitsu personnel.
Fujitsu would ignore information directly from the Post Office itself that demonstrated that a SPM was not at fault. ROMEC engineers observing specific matters occurring, or in this case the Post Office’s own auditors ruling out user error, were simply ignored. This effectively amounted to the Fujitsu SSC personnel positively over-ruling the Post Office’s own non-SPM personnel, who were giving them essential information about failures in the Horizon system.
Fujitsu would also close PEAKs in a variety of ways that entirely mischaracterised the issue. This particular one was “no fault in product” and defect cause “general – user”, in other words, it was caused by the SPM. This had been specifically ruled out by the Post Office’s own auditors who had visited the branch.
On the PEAKs that were used in the Horizon Issues trial, Fujitsu would routinely assign lower categories of importance to reported matters that were directly impacting SPMs’ branch accounts. They would also routinely assign user error to reported matters, not because they had uncovered user error, but because they could not explain what had occurred. It seems to have been used as a default setting. Given my findings in the Technical Appendix, this approach by Fujitsu simply cannot be justified in technical terms.
Fujitsu do not, on the face of these documents shown to Mr Parker, appear to me to have properly and fully investigated these myriad problems, nor did Fujitsu categorise such incidents correctly. They also seem to have moved away, in their investigations, from concluding that there were any issues with the software wherever it was possible for them to do so, regardless of evidence to the contrary, an approach that has been carried into the Fujitsu evidence for the Horizon Issues trial. In re-examination Mr Parker was asked about the way he had done his exercise, and also the point was put to him that the PEAKs to which he was taken were outwith the period when Mr Roll was employed at SSC. The implication of this was that Fujitsu may, or would, have been more accurate in the period in question rather than in later periods. Mr Parker said he could not possibly read all the PEAKs, that he considered his data to be accurate, and although “you may find a few PEAKs where the response code does not tally with the document” given the number of PEAKs involved “human beings will make those errors”.
I reject that explanation, such as it is. I do not consider that Mr Parker was interested in accuracy in any of his evidential exercises, and I do not consider that he was objective in the way he presented his evidence, although he sought to give the impression that he was. I consider that Mr Parker, and the team who assisted him, sought to portray the Horizon system – Legacy Horizon and Horizon Online – in a light as favourable as possible to Fujitsu, regardless of its own internal evidence to the contrary, and regardless of the facts. Mr Parker’s spreadsheet plainly did not calculate all the categories that were used by Fujitsu for problems with the system, and sought
to give the impression that Mr Roll had been involved in far fewer incidents of this type than were the case.
My conclusion about Mr Parker’s lack of accuracy, is supported by the changing story of Fujitsu about remote injection of messages; the need for correcting supplementary statements, including from Mr Parker himself; his admission that he knew Fujitsu had the power to do this, even though his 1st statement expressly said otherwise; the attempt to shift the blame for this on to Mr Roll; the assertions in his subsequent statement that Mr Roll had been unclear; and also by his attempt to portray his original evidence as dealing with frequency, not power or ability. My conclusions are also supported by his “deliberate decision” to depart from Fujitsu’s definition in its own internal document, which I have explained at [488] above. I also consider that the fact that someone such as Mr Parker had been on the Post Office Account for such a very long time, in his case throughout the whole life of Horizon, means he was hardly best placed to be objective in this type of evidence. But even making allowance for the natural reaction of an employee to wish to protect his employer’s interests, which many people may have sub-consciously, I find that Mr Parker’s evidence to the court was inaccurate to a significant degree.
In my judgment, the exercise done by Mr Parker to demonstrate the number of PEAKs that Mr Roll worked on involving software, included in the spreadsheet of categorisation he and others prepared, is of no evidential value whatsoever. It relies upon the categorisation of the PEAKs performed by the Fujitsu personnel themselves, which in the vast majority of documents used in the trial were simply wrong and/or misleading; and it also used a different definition to the one included in Fujitsu’s own documents such as the PEAKs above. He accepted that he had made a deliberate decision to use a definition other than the one in the Fujitsu documents, which he justified by saying was based on his experience. That deliberate decision alone is remarkable. I find that he was, by means of the exercise he was involved in, trying to downplay the extent of Mr Roll’s involvement in software issues, and also trying to downplay the extent of software problems that were experienced on Legacy Horizon during Mr Roll’s employment at SSC.
Although Mr Parker agreed, as it was put to him more than once, that accuracy is important, I do not consider his evidence in his witness statements to have been remotely accurate, even though he stoutly maintained that it was. He continued to maintain this in his re-examination, even though by then he can have been in no doubt that he had departed from Fujitsu’s own definition – which he had said he had forgot about. I found him a very unsatisfactory witness, who presented in his witness statements a misleading and one-sided sanitised version of actual problems and events that Fujitsu had experienced throughout his time at SSC. Although he provided tables with one of his witness statements that provided a detailed commentary on a large number of PEAKs and KELs, that commentary appearing in a column headed “Fujistu’s Comments” with two columns, “Response to Mr Coyne” and “Financial Impact on branch accounts” – that latter almost always “no impact”. The entries in those columns are almost entirely self-serving by Fujitsu and I find them to be of no evidential value. The text of the actual PEAKs and KELs themselves are far more useful, obviously entered at a time when the Fujitsu/Post Office position in the litigation was not at the forefront of the different authors’ minds, and these are what for the most part the experts have considered. Mr Parker’s table is of no assistance.
The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Membury
Mr William Membury is another Fujitsu employee, namely a Fujitsu Central Quality Partner and he is specifically focused on the Post Office account. He was taken ill during the early part of the Horizon Issues trial, and when (after dismissal of the recusal application) the trial resumed with outstanding factual evidence on 11 April 2019, he was not well enough to be called in person. His witness statement was dealt with as hearsay evidence and given the circumstances the claimants agreed to extend time for the relevant notice to be given under CPR Part 33.2.
Mr Membury has worked for Fujitsu since 1998 and became the Quality Risk and Compliance Manager for the Post Office account in 2011. He has overseen multiple audits of the Horizon system. Given the timing of this, these would have all been on Horizon Online. In 2014 he became a Central Quality Partner specifically focusing on the Post Office as well as the Payment Card Industry (PCI) Security Standards Subject Matter Expert (SME) for Fujitsu UK and Ireland. In 2015 he became the PCI SME for the Europe, Middle East, India and Africa regions. He returned to the Fujitsu Post Office Account as Head of Quality and Compliance in October 2018.
He described his statement as an overview of the auditing and development regimes in place in relation to Horizon to the extent that those points were not covered by the disclosed technical documents. The statement was said to provide a high-level overview of the audits to which Horizon had been subject since 2000; Fujitsu’s audit methodology and his role in the audits; and to describe the development of changes to Horizon and how that tied in with or into the audit process.
Mr Membury’s statement was very brief; it ran to only 5 pages of text. It also omitted some highly material matters. For example, he stated that the Post Office and Fujitsu agreed in 2010 that Horizon Online would be audited against the International Standard on Assurance Engagements (ISAE) No. 3402 assurance standard (Horizon ISAE 3402 Audit), entitled "Assurance Reports on Controls at a Service
Organization", which was issued in December 2009. This was developed to provide an international assurance standard for allowing public accountants to issue a report for use by user organisations and their auditors (user auditors) on the controls at a service organisation that are likely to impact or be a part of the user organisation's system of internal control over financial reporting. It provides assurance that the solutions in place to manage financial transactions are appropriate.
However, he completely omitted any reference to the Ernst & Young Management Letter for the year ending 27 March 2011 which has been referred to in [264], [393] and [524], which included concerns about privileges and other critical matters in relation to IT. Indeed, he made only two references to Ernst & Young. One was to state that:
“Ernst & Young have carried out the Horizon ISAE 3402 Audit since the 2012/2013 financial year (preparations for the audit began with Post Office in 2011)”
The second was a reference to how Ernst & Young performed the ISAE 3402 audit for the period 1 April 2014 to 31 December 2014. His failure to mention the 2011 Management Letter is, in my judgment, a serious omission.
He could not be asked any questions about his evidence; although that was not his fault, this goes to the weight which I attach to his evidence in any event.
I consider that Mr Membury’s evidence is of limited, if any, assistance in resolving the Horizon Issues. It does however continue the very one-sided picture presented by all the Fujitsu witness statements, which was to omit any reference to important contemporaneous documents that criticise or demonstrate any deficiencies with Horizon.
The absence of Mr Gareth Jenkins
It is entirely a decision of the parties which witnesses they choose to call in any proceedings in respect of any evidence. The position of one person, however, who did not appear in the Horizon Issues trial, must be considered in more detail than would be usual, as the claimants make considerable complaint about this. The person in question is Gareth Jenkins, a senior Fujitsu employee who, although he retired recently, was obviously widely available to the Post Office and the source of a great amount of information both to the Post Office’s witnesses of fact, and also to Dr Worden (although he was not separately identified in Dr Worden’s “sources of information” paragraphs in his 1st Report). The fact that he provided information to Dr Worden emerged during the latter’s cross-examination. Mr Jenkins had previously given expert evidence for the Post Office in some of the criminal prosecutions of SPMs, in particular that of Ms Misra, to whom I have referred above, who was convicted of criminal offences in Guildford Crown Court in 2010.
When the Post Office served its evidence of fact, there was no witness statement from Mr Jenkins, although many of their witnesses relied upon him as their source of information, he was referred to very often, and he obviously knew a great deal about Horizon. The extent and way in which Mr Jenkins had been closely involved was explained by Mr Godeseth in his cross-examination. Mr Godeseth had, in respect of the receipts and payments mismatch, originally stated in paragraph 42 of his 2nd witness statement that 60 branches were affected. He had corrected this to 62, a factual correction that was specifically made by him. The following passage of evidence is relevant to Mr Jenkins’ involvement in this.
“A. No, no, I didn't do a calculation to come up with the 60. I was quoting from other people.
Mr Justice Fraser: Someone just gave you the 60, did they?
A. I thought I was quoting from other people. I -- Gareth even said to me that in my statement I had said "approximately 60", so I was not -- clearly I didn't because the statement here doesn't contain that word. I had rather hoped it had when this was first brought to my attention, but no, I certainly did not do any specific calculation to come up with the 60 that I put into my original statement.
Mr Green: Did Gareth explain the change to you from 60 to 62? A. No.
Q. But how did he come to tell you what was in your original statement? What was that conversation?
A. I picked up the number -- to be -- my objective in this was to explain to the court the symptoms of the bug and how -- the technical aspect of it. I did not pay particular attention to getting the detail on how many branches were affected, correct. Q. Okay. So you have spoken to Gareth since your statement?
A. I don't think I have spoken to him about this in particular. I was -- as I say, when Gareth had said I had originally said "approximately 60" I was thinking that was quite neat, but that's not the case.
Q. Well, you said "Gareth even said to me that in my statement I had said 'approximately 60'", so he must have said that to you after your statement had been filed?
A. It was a comment in a document that we were exchanging.
Q. But you hadn't spoken to him about remote access since your first statement?
A. No.
Q. Why have you stayed off that topic with him?
A. Oh, sorry, this was just a comment. We have been exchanging documents, we have been commenting on documents, so it was not a particular conversation. It is merely a case of Gareth had commented on this when it was pushed back to us that I had originally said 60 and actually the answer was different.”
There is nothing wrong with Mr Godeseth correcting the number of affected branches from 60 to 62. There is also nothing wrong with him reproducing a number given to him by someone else, as long as it was properly identified that the information came from someone else, who ought to be identified (which it wasn’t in his statement, but was, in at least outline terms, in his cross-examination). The reason for reproducing these passages is simply to identify the extent to which Mr Jenkins was so closely involved in the litigation, a point relied upon by the claimants. There were a great many references throughout Mr Godeseth’s evidence, written and oral, about information he had obtained from Mr Jenkins.
When the Post Office served their evidence of fact, the claimants had asked the Post Office why there was no statement from Jenkins, whether Mr Jenkins was available to give evidence, and also whether he was involved as one of a team of what the claimants referred to as the “shadow experts”. This description was challenged by the Post Office, and the question of shadow experts is addressed further at [556] below. No explanation was given for Mr Jenkins’ absence in response to these requests, or in evidence in the trial, although it was confirmed that Mr Jenkins was not one of the team of so-called “shadow experts”.
There the matter might have rested. However, in the Post Office’s written closing submissions, an explanation of sorts was for the first time provided. This was in the context of two matters: firstly, by way of explanation of Mr Godeseth’s evidence, and potentially to downplay its impact; secondly, in relation to the claimants’ complaints about the second hand nature of some of the Post Office’s factual evidence because in large part this had emanated from Mr Jenkins. This explanation by the Post Office included the following passages in its written submissions:
“144. [The claimants] understandably complain that Mr Jenkins and the other source of Mr Godeseth’s information could have given some of this evidence first hand. However:
Taking into account that Mr McLachlan’s evidence specifically addressed things said or done by Mr Jenkins in relation to the Misra trial, Post Office was concerned that the Horizon Issues trial could become an investigation of his role in this and other criminal cases.
Moreover, Post Office was conscious that if it only adduced first hand evidence in the trial, it would end up having to call more witnesses than could be accommodated within the trial timetable.
Furthermore, so far as Post Office was aware, the relevant parts of Godeseth 2 were most unlikely to be controversial. For example, the Misra trial was a matter of public record, the four bugs were covered by contemporaneous documentation and Post Office had no reason to doubt Fujitsu’s account of the documents it held.”
In a footnote to paragraph 144.2 of the closing submissions, the Post Office added “…..As noted above, had its witnesses only given first hand evidence, Post Office estimates that some 34 additional witnesses would have been required.”
The following are relevant in my judgment:
Of primary importance is the principle that it is for each party to decide whom to call as a witness, and what evidence they may seek to obtain from any particular witness.
Mr Jenkins is an important and central person so far as the operation, efficacy and robustness of Horizon is concerned, and also in respect of the number of incidents over the years that have led to PEAKs, KELs, problems and fixes. That is reinforced by the Post Office’s own closing submissions that stated, at paragraph 138, that there were two possible candidates for the witness giving the overview of Horizon, namely Mr Godeseth and Mr Jenkins. It also stated that taking into account Mr Jenkins’ involvement in cases before the CCRC (including the Misra case) the decision was taken to use Mr Godeseth. For that decision to have been taken, it is implicit that Mr Godeseth must know enough about the system to give the evidence that he did.
The point in the footnote about a potential further 34 witnesses is not relevant to whether Mr Jenkins was, or was not, called by the Post Office. The Post Office did not at any stage apply to the court and explain that further time was required for the length of the Horizon Issues trial for the specific reason that relevant evidence of fact from the Post Office would require a particular minimum number of witnesses. However, given the primacy of point (1) above, it was entirely for the Post Office to decide whether to call Mr Jenkins or not, how many witnesses to call, whom those witnesses were to be and what the content of their evidence should include.
The Post Office chose to proffer a reason for Mr Jenkins’ absence in closing submissions. They were not obliged to explain. However, the reason in paragraph 144.1 quoted above is not a valid reason for his absence. The claimants would have been entitled, in cross-examination, to put to Mr Jenkins any previous inconsistent statements he had made on the same subject, but obviously only if there were any. These could, potentially, have included previous statements he may have made in earlier proceedings, but in order to be allowed to do that, such statements would have had to be inconsistent with his evidence in the Horizon Issues trial. Putting a previously inconsistent statement on a particular fact is permitted as part of crossexamination if a witness’ evidence on that fact is in issue. However, the Horizon Issues trial would not have become “an investigation of his role in this and other criminal cases” – the Horizon trial was about the Horizon Issues. Also, this type of cross-examination would only have arisen if any statement(s) in the Horizon Issues trial was or were not consistent with those he had made in previous cases in which he had been involved.
Regardless of the validity of the explanation given by the Post Office, Mr Jenkins was not called as a witness, and it merits repetition that it is entirely a matter for the Post Office which witnesses it called, and which it did not. No speculation is permitted as to what evidence he might have given in the Horizon Issues trial. It ought to be recorded however that no limit on the number of witnesses was imposed by the court on any party, and the Post Office could (had it wished) have called both Mr Godeseth and Mr Jenkins. Nobody forced the Post Office to choose one or the other.
There is established authority that in certain circumstances the court can draw adverse inferences from the absence of a witness. The claimants do not invite me to draw any adverse inferences from the absence of Mr Jenkins, and I do not do so.
The fact that the Post Office chose to advance certain evidence in the Horizon trial, that emanated from Mr Jenkins, by means of another witness saying “Mr Jenkins told me that…..” means that the claimants were deprived of the opportunity properly to test that evidence by asking Mr Jenkins about it. It also emerged in cross-examination, although not in his written report, that Dr Worden has also obtained certain information directly from Mr Jenkins. By dealing with the material in this way, and having deprived the claimants of the opportunity to cross-examine Mr Jenkins, the weight to be given to evidence emanating from Mr Jenkins is less than it would be otherwise.
There are two consequences of the Post Office choosing to adduce such evidence in this way. The first is that Mr Jenkins did not have to undergo any cross-examination on such evidence; the second is that such evidence would not be given the same weight by the court as though Mr Jenkins had given it himself. The Post Office would have known this in advance, as this is wholly conventional. How these different factors were weighed by the Post Office and its legal advisers in deciding whether or not to call him, is solely a matter for the Post Office. The end result is that Mr Jenkins did not appear and was not cross-examined.
However, I reject the suggestion that Mr Godeseth was insufficiently knowledgeable of the matters upon which he was cross-examined for his evidence to be of significant evidential weight. The Post Office’s Closing Submissions sought to dilute the effect of Mr Godeseth’s cross-examination by, at the same time, implicitly suggesting that
Mr Jenkins (or someone else, but not Mr Godeseth) would have been a far better
person to have answered those questions; yet explaining why Mr Jenkins had not been called. These submissions, though understandable from the Post Office’s point of view in forensic terms only, have little force. A clear choice was made by the Post Office not to call Mr Jenkins as a witness; that was their right. A clear choice was also made to call Mr Godeseth. The Post Office was are entitled to call such evidence as deemed necessary by both the Post Office and its advisers. They cannot, however, have matters both ways, and try to downplay Mr Godeseth’s evidence because he was called to deal with certain matters, and Mr Jenkins was not. The Post Office has to abide by the consequences of their choice of witnesses, in terms of the evidence now before the court to resolve the Horizon Issues trial. I have already explained my view of Mr Godeseth’s evidence in Part E above, and it need not be repeated.
Inaccurate Statements by the Post Office
Litigation in this jurisdiction is adversarial. What that means is each side advances their own case, and challenges or rebuts the case of the other side, and the court assesses the evidence before it adduced by the parties, makes necessary findings on the facts and applies the law. Parties in civil litigation will usually advance their own evidence of fact, and although mechanisms are available for compelling other witnesses to attend and answer questions, there can be some pitfalls in doing so and issuing a witness summons (what used to be called a subpoena) is a relatively rare step. The court can only therefore resolve the case on the evidence before it, although it can draw inferences, that is to say common sense conclusions, on the evidence it does have.
The Post Office has no obligation to assist the claimants in advancing their case against it. The Post Office has, however, maintained publicly that it was seeking to be “transparent” about Horizon, and prior to the litigation it made certain public statements in relation to the increasing disquiet on the part both of SPMs, and others who became involved either on their behalf (such as some Members of Parliament) or in an investigative way (such as the BBC Panorama programme and other journalists). These statements by the Post Office routinely and strongly insisted that there was nothing in the criticisms being levelled at the accuracy of Horizon, and that losses that were shown in SPMs’ branch accounts were caused either by carelessness or dishonesty on the part of the different SPMs who experienced what they considered to be unexplained discrepancies and losses. In 2015 a Parliamentary Select Committee held hearings into the mediation scheme that had been set up to attempt to address the claims by SPMs in respect of Horizon. I have not considered any evidence submitted to the Select Committee, or any details of the mediation scheme (which was being conducted by Second Sight) and refer to them here simply as a matter of chronological record.
The claimants rely upon a public statement released by the Post Office in 2015 after the BCC Panorama programme. Part of the statement was headed “The Horizon System” and stated:
“The Horizon Computer System
Horizon is robust and effective in dealing with the six million transactions put through the system every day by our postmasters and employees at 11,500 Post Office branches. It is independently audited and meets or exceeds industry accreditations. There have been 500,000 users of the system since it was introduced.
Nevertheless, rigorous re-investigations were undertaken into claims made by 136 mainly former postmasters that the system caused losses in their branches.
There is overwhelming evidence that the losses complained of were caused by user actions, including in some cases deliberate dishonest conduct. The investigations have not identified any transaction caused by a technical fault in Horizon which resulted in a postmaster wrongly being held responsible for a loss of money.
There is also no evidence of transactions recorded by branches being altered through ‘remote access’ to the system. Transactions as they are recorded by branches cannot be edited and the Panorama programme did not show anything that contradicts this.”
(emphasis added)
Another document from 2015 upon which the claimants rely is an internal email chain, which originated from Paula Vennells, then the Chief Executive of the Post Office, on 30 January 2015. This was prior to her appearance before the House of Commons Select Committee in February 2015. She posed the following question in an email sent internally to Mark Davies and Lesley Sewell, both of the Post Office:
“Dear both, your help please in answers and in phrasing those answers, in prep for the SC:
1) "is it possible to access the system remotely? We are told it is."
What is the true answer? I hope it is that we know this is not possible and that we are able to explain why that is. I need to say no it is not possible and that we are sure of this because of xxx and that we know this because we have had the system assured.”
(emphasis added)
Ms Vennells obviously needed to know whether the answer matched her understanding, which was – as she put it, both “I hope” and “I need” -- that it was not possible to access the system remotely. This query was passed on through various people, including at one stage from James Davidson who has both a Fujitsu and Post Office email address, who answered to Mark Underwood:
“As discussed, can you hook up with Kevin to review what answers have already been provided to second sight as this should form the Post Office response.”
The answer was provided by Mark Underwood on 30 January 2015 in an email which is part of the same email chain or string:
“Can Post Office or Fujitsu edit transaction data without the knowledge of a Subpostmaster?”
Post Office confirms that neither it nor Fujitsu can edit transaction data without the knowledge of a Subpostmaster.
There is no functionality in Horizon for either a branch, Post Office or Fujitsu to edit, manipulate or remove a transaction once it has been recorded in a branch's accounts.
The following safeguards are in place to prevent such occurrences:”
(various matters are then listed in the remainder of the email)
(bold present in original)
This then was subject, in the email chain, to a degree of refinement. Kevin Lenihan forwarded the email onwards to Mark Underwood and others, and stated:
“Mark / Mel,
James has had a look at your answer to Q1. And thinks there’s too much detail for Paula – this was written for a different type of audience. He has captured the same points but in a more appropriate format :- He states:-
Having looked again at the request from Paula, it appears that the fundamentals around this question (remote access) are not understood. I suggest that Paula is briefed along the lines of the following.
No transaction data is held locally in any branch. Transactions are completed and stored in a central database and copies of all data is sent to a secure audit database.
Sub-postmasters directly manage user access and password setting locally so system access (to create transactions) are limited to approved local personnel only who are responsible for setting their own passwords. Users are only created following an approval process which requires authorisation by the sub-postmaster. All subsequent transactions are recorded against the id used to log on to the system.
Once a transaction has been completed, there is no functionality (by design) for transactions to be edited or amended. Each transaction is given a unique number and ‘wrapped’ in a digital encryption seal to protect its integrity. All transactions are then posted to a secure and segregated audit server.
On approval, there is the functionality to add additional transactions which will be visible and have a unique identifier in the audit trail. This is extremely rare and only been used once since go live of the system in 2010 (March 2010)
Support staff have the ability to review event logs and monitor, in real time, the availability of the system infrastructure as part of standard service management processes.
Overall system access is tightly controlled via industry standard ‘role based access’ protocols and assured independently in annual audits for ISO 27001, Ernst and Young for IAS 3402 and as part of PCI audits.”
(emphasis added)
I do not see how the statement that “I do not know how the fundamentals around this question (remote access) are not understood” can sensibly be made. Nor do I know what the expression “the fundamentals around this question” in fact means, in plain English. The question from Ms Vennells was very straightforward. It was as follows, using her words, and separating out each clause of the enquiry:
“Is it possible to access the system remotely? We are told it is.”
“What is the true answer?”
Both of these are really very simple questions. Question 1. means can it be done, because Ms Vennells is being told it can be done. Question 2, “what is the true answer?” means she is seeking the true answer to question 1. This is not complicated, either in linguistic, computing or even business terms, nor is it difficult to understand. She then expressed her aspirations in terms of what the true answer might be. “I hope it is that we know this is not possible and that we are able to explain why that is. I need to say no it is not possible.” (emphasis added)
This trial has shown that the true answer to the enquiry she made in early 2015 was “yes, it is possible.” It has taken some years, and many tens of millions of pounds in costs, to reach that answer. Kevin Lenihan, who wrote that he did not understand “the fundamentals around this question (remote access)”, seems to me to have avoided providing a simple answer to a simple question. It is not necessary to consider why that was, whether from a lack of understanding on his part, or otherwise.
I am unaware of whether there is any other final answer to Ms Vennells internally to her very straightforward question, or of what she in fact told the Select Committee. None of the people through whom this email chain passed were called as witnesses in the Horizon Issues trial. However, the point in bold in the email quoted at [522] above is now known, as a result of this litigation, simply to be incorrect. Such editing can take place without the knowledge of the SPM. All and any previous statements by the Post Office stating that this cannot be done are simply factually wrong. The more detailed points at (1) to (6) of the email quoted at [523] provide only selected information and, in my judgment, are entirely off the point. They have the effect of obscuring what ought to be a simple answer, with a level of detail which makes the actual answer highly unclear. The answer at (1) applies only to Horizon Online. For Legacy Horizon, transaction data was held locally in the first instance, on what was called the counter. This was explained very clearly by the Post Office in its opening, including in its oral opening submissions. The first sentence of (3) in the list in the email is not correct. The statement in (4) – which may refer to the Transaction Correction tool – is correct (or at least, not incorrect) but only so far as it goes, and in my judgment crucially omits the ability to edit transactions without visibility to the SPM. It also fails to consider the existence of APPSUP permissions at all, which every member of SSC had. The experts are agreed that the APPSUP permission was very powerful and enabled an SSC employee to do pretty much whatever they wanted, to paraphrase the Post Office’s own expert. It can also be seen, particularly by the evidence that emerged in the cross-examination of Mr Godeseth, that in a great many cases the PEAKs show that user error was attributed to problems that were not user error at all, and that others within the Post Office (for example its own auditors and ROMEC engineers) had either witnessed for themselves and/or satisfied themselves that user error was not the cause of specific issues, but Fujitsu decided they were user error in any event.
The extent of permissions enjoyed by SSC was the subject of some express concern by Ernst and Young (the Post Office’s own auditors) in 2011, and although the position may have been regularised by 2015, certainly prior to 2011 (which covers the
Legacy Horizon years) the situation was very different. The email also states “he has captured the same points” yet the most important point to which an answer is given, or certainly the answer given in bold in the answer at [522] above, is not corrected, even though it is plainly wrong. The overall tenor of what Ms Vennells received, if it reflected the full content of this email chain, would have been that the Post Office and Fujitsu could not edit transaction data without the knowledge of the SPM. The evidence in this trial is directly to the contrary, and Fujitsu could do so. It is important to bear in mind the distinction between the way Legacy Horizon, and Horizon Online, work. Referring to keeping data on the “counters” is a Legacy Horizon expression. This refers to the way the data was held at the branch; these were discs (including mirror discs) at the branch which contained the data. The discs were contained in terminals on the counter. In Horizon Online, the data was held at a central database or server (the phrase used by the Post Office in its oral opening).
This tenor of these public statements made by the Post Office was maintained into, and as part of, the group litigation. For example, certain statements were made in the Generic Defence in these proceedings. A party’s Defence is the formal pleading document which sets out a party’s legal position in the litigation. Since the reforms in civil litigation adopted after what are called the Woolf Reforms, which led to the introduction of the Civil Procedure Rules in 1998 which govern all civil litigation in the jurisdiction, such pleadings must be accompanied by a Statement of Truth. These are, for larger companies, usually signed by directors or very senior personnel, which occurred in this case. That statement attests to the truth of the matters stated in the pleading.
This issue was dealt with in the pleadings in the following way. By this point the Post Office had accepted that there was some limited ability to do that which it had previously stated was not possible. In the Generic Statement of Case, the claimants set out the following part of their case:
“25. Further, the Defendant was, by itself and/or via its agent Fujitsu, able to alter branch transaction data directly and carry out changes to Horizon and/or transaction data which could affect branch accounts.
26. However, the Defendant has made public statements in the following terms:
"Horizon does not have functionality that allows Post Office or Fujitsu to edit or delete the transactions as recorded by branches" (Defendant's published reply to Second Sight's Briefing Report Part Two, concerning a review of the Horizon system); and
"Transactions as they are recorded by branches cannot be edited" (Defendant's published reply to the BBC Panorama documentary in relation to Horizon).
27. These statements were untrue, as the Defendant now admits.”
These paragraphs were pleaded to by the Post Office in its Generic Defence in the following terms:
“57. Paragraph 25 appears to be concerned with the editing or deletion of transaction data input by or on behalf of a Subpostmaster without his or her consent. Accordingly, Post Office assumes that it is not concerned with transactions such as Transaction Corrections which are sent to branches but must be accepted by or on behalf of the Subpostmaster before forming part of his or her branch account. As to the circumstances in which such transaction data can be edited or deleted without the consent of the Subpostmaster:
Neither Post Office nor Fujitsu has the ability to log on remotely to a Horizon terminal in a branch so as to conduct transactions.
A Post Office employee with "global user" authorisation can, when physically present at a branch, use a terminal within the branch to add a transaction into the branch's accounts. The purpose of "Global User" authorization is to allow access to the systems for during training and/ or audits. Any transactions effected by a Global User are recorded against a Global User ID and are readily identifiable as such.
Fujitsu (and not Post Office) has the ability to inject transactions into branch accounts (since the introduction of Horizon Online in 2010, transactions of this sort have been called "Balancing Transactions"). These transactions do not involve any removal or amendment of the transactions entered at the branch. Their intended purpose is to allow Fujitsu to correct errors or bugs in Horizon by cancelling the effect of an error or bug on a branch's data. They may be conducted only by a small number of specialists at Fujitsu and only in accordance with specific authorisation requirements. They are rarely used. To the best of Post Office's information and belief, only one Balancing Transaction has ever been made so as to affect a branch's transaction data, and this was not in a branch operated by a Claimant. A Balancing Transaction is readily identifiable as such.
There are a small number of Fujitsu specialists who have certain privileged user access rights which they could in theory use to amend or delete the transaction data for a branch. The intended purpose of privileged user rights is system support, not the alteration of branch transaction data. To have abused those rights so as to alter branch transaction data and conceal that this has happened would be an extraordinarily difficult thing to do, involving complex steps (including the writing of sophisticated computer programmes and circumvention of sophisticated control measures) which would require months of planning and an exceptional level of technical expertise. Post Office has never consented to the use of privileged user rights to alter branch data and, to the best of its information and belief, these rights have never been used for this purpose.
Post Office cannot conceive of a reason why any Fujitsu personnel would have sought to add, inject, amend or delete any transactions in any branch accounts so as to create a false shortfall. It would for all practical purposes be impossible for any of them to generate significant shortfalls without detection and, even if they were able to do so, they would be unable to take the benefit of such shortfalls for themselves.” (emphasis added)
This stance was maintained by the Post Office in the evidence served on its behalf for the Horizon Issues trial, until service of Mr Roll’s 2nd witness statement. To be fair to the Post Office, its origin was the witness statements served by Fujitsu employees, rather than Post Office employees. The position within the Fujitsu witness evidence,
prior to its correction by the later statements from Mr Parker, was that what Mr Roll said was possible on Legacy Horizon, and what he had himself done, was simply not possible. Indeed, Dr Worden considered it sufficiently clear that as an IT expert he felt able confidently to assert in his 1st Expert Report that he, Dr Worden, had “established” that Mr Roll’s evidence of fact in this respect was wrong. After service of Mr Roll’s 2nd witness statement, Fujitsu finally came clean and confirmed (via Mr Parker) that what Mr Roll said was correct. Data could be altered by Fujitsu on Horizon as if at the branch; under Legacy Horizon, transactions could be inserted at the counter in the way Mr Roll described. This could be done without the SPM knowing about this. Mr Godeseth also confirmed that it would appear as though the SPM themselves had performed the transaction. This is directly contrary to what the Post Office had been saying publicly for many years.
Therefore some of the earlier public statements made by the Post Office, and the important one contained in the Post Office’s pleaded Defence to which I have referred above, were factually untrue in at least one highly important respect. This concerns the ability of Fujitsu to insert transactions into a branch’s accounts remotely, without the SPM being aware of this, and without the transaction being identifiable in the transaction data as having been inserted remotely in this way. The term “remotely” means from a location elsewhere than within the branch itself.
Although the phrase “remote access” was used during the Horizon Issues trial, the parties did not always use the expression in the same way as one another. The Post Office’s definition was explained in oral opening in the following way:
“When I talk about remote access I'm talking about action taken remotely to either inject new transactions or to edit existing transactions or to delete existing transactions in a way that could change the accounting position of the relevant branch.”
For the purposes of this judgment, I too use the term “remote access” in this way. The true picture is now that, following the Horizon Issues trial, there is evidence of transactions recorded by branches being altered through remote access by Fujitsu; and transactions as they are recorded by branches being edited; that this can be done without a SPM’s knowledge or permission; and without being identifiable as having been done remotely. Dr Worden was asked about the 2015 Post Office statement in response to Panorama, which he said he had not read, and he explained that it was the audit records in the audit store that could not be edited. He was not prepared to say the 2015 statement was factually wrong. The BRDB records could be edited, and he also referred to the joint statement in which the two experts had agreed that they could not say that anything could not be done. He was very hesitant, if not reluctant, to state that the 2015 statement was wrong.
In my judgment the 2015 Post Office statement plainly is wrong. I find that the 2015 statement was not true. Again, to be fair to the Post Office, this was accepted in the litigation and at the hearing before Senior Master Fontaine. The quotation from the transcript of that hearing when this was addressed by the Post Office in oral submissions is as follows:
“Master, first of all, could I just deal with the remote access point? The letter to which my learned friend took you was, as you might expect, written by people who thought it was correct. The Horizon system is a very complicated system. It involves lots of departments in ... both in Fujitsu and in the Post Office. And the people who are responsible for the correspondence didn't know that, in fact, there were these two other routes. Very few people at Post Office knew that there were these two other routes. They were ... they were routes that are under ... essentially under the control of Fujitsu who's the expert independent contractor that is involved in the operation of the system. And it is a matter of enormous regret that the people who wrote that correspondence and made those submissions weren't aware of that but, you know, we are where we are; the point is that, the point having been discovered, the Post Office wasted no time in ... in bringing the truth ... the accurate ... and accurate set of facts to the knowledge of the claimants.”
(emphasis added)
The claimants rely upon the way this sequence unfolded, and the admission by the Post Office that its previously public statements were untrue, as making it particularly important that the true and accurate position was then provided by the Post Office. Indeed, the submissions I have quoted at [535] above make it clear that the court and the claimants were being told that the Post Office had wasted no time in bringing the truth, “the accurate set of facts”, to the knowledge of the claimants. As the analysis of the Defence and the evolution of the Fujitsu evidence of fact on remote access for the Horizon Issues trial demonstrates, the accurate set of facts did not emerge at that time either. The truth, the accurate set of facts, has only emerged as a result of the final supplementary statements for the Horizon Issues trial that were served, and only finally (so far as Fujitsu is concerned) after their witnesses had initially stated clearly in their witness statements that Mr Roll was wrong. Eventually these witnesses accepted that he was right. The supplementary witness statements that accepted this were dated 29 January 2019. Prior to that date, the opposite position had been maintained by the Post Office.
The Post Office in its opening submissions stated the following:
“Taking remote access as an example, the need for remote intervention affecting branch accounts will obviously be rare. On any view, the occasions on which privileged users at Fujitsu have exercised their ability to remotely inject, edit or delete branch transactions or accounting entries will represent a tiny percentage of the relevant transactions/accounting entries. And the occasions on which they have done so negligently or dishonestly will, in turn, represent a very small percentage of those occasions. So, compared with the volume of business recorded in branch accounts, the number of cases in which false data will have been remotely introduced will be extremely small (multiplying a small chance by a small chance). This is a “second order effect” (a small proportion of a small proportion) which is, by definition, extremely unlikely to have any significant impact on the robustness of Horizon.” (emphasis added)
That submission misses the point, in my judgment. It elides two matters, namely whether something is technically possible, and the number of times that it has in fact been done. The former, whether it was possible, had been expressly denied, and that denial is now shown to be wrong. The latter, the number of times it was done and
with what effect, is a different matter. There is also very little, if any, evidence that is relied upon by the Post Office to justify the assertion that it was a “tiny percentage” of times. That is a subjective assertion of very limited weight.
I consider the significance of the previously factually untrue statements to be considerable. The statement was made publicly by the Post Office, turned out not to be factually correct, and the Post Office gave an explanation and said the full set of facts was now available. The situation was pleaded to by the Post Office in its Generic Defence, with a statement of truth. That too turned out not to be correct, and the true position has only emerged in the Horizon Issues stage of the litigation as a result of the evidence of Mr Roll, which I have dealt with above. It was only following his written evidence that Mr Parker, and Mr Godeseth – both senior Fujitsu employees – prepared their supplementary witness statements correcting their first statements. These first statements, as I have explained above, were simply untrue in that important respect. These witnesses had previously stated that this was not possible. Mr Parker said Fujitsu did not have the power to do this.
I find as a fact that Fujitsu do have the ability to insert transactions into branch accounts on a remote basis (in other words, remote access as the Post Office defined it in opening submissions existed) and this could be done without these being visible to the SPM in that branch in question, either at the time or subsequently. It also follows that this could be done without an SPM in this position having knowledge of this, and therefore without consenting to it. Someone who does not know something is happening cannot give permission for it to happen. Mr Godeseth also gave important evidence about how such transactions would appear. It is not possible to say, in blanket terms, that permission was not sought and/or given in some cases. Indeed, Dr Worden accepted that the APPSUP permissions meant that – as he put it – Fujitsu personnel could do “pretty much anything”. On Day 18 he said in cross-examination the following:
“Q. Dr Worden, you knew that a central issue, not only a central issue legally but a very high-profile issue in the case, was the extent to which Post Office had remote access to the counters, didn't you? You knew that?
A. Yes, and what I'm talking about, what I was talking about was the extent to which this could happen without the knowledge of the subpostmaster.
Q. And that's the –
A. And we agreed in the joint statement that more or less Fujitsu or Post Office could do anything.”
He was asked about this answer two days later.
“Q. It is Day 18, page 67. I think it is lines 5 and 6 actually. It says: "Answer: And we agreed in the joint statement that more or less Fujitsu or Post Office could do anything."
A. Yes. We can see precisely what we agreed in a minute, obviously.
Q. In relation to the –
A. We agreed the experts couldn't demonstrate that they couldn't do everything. I mean, that's sloppy wording by me there. I think the joint expert statement says it better.”
The exact wording in the joint statement in respect of repairing transactions is as follows at 10.2 of the 4th Joint Statement:
“Certain facilities and procedures used by Fujitsu to repair the more common issues which arose in Horizon were standardised, and evidence of them persists. However, to repair less common issues which arose from time to time, standard tools and procedures might not have been sufficient, and evidence might not persist of what was done at the time. Even when evidence does persist, it may be extremely difficult for the experts to interpret it today, because of the scale and complexity of Horizon.
Therefore, it is usually difficult for the experts to make categorical negative statements of the form: ‘X or Y never happened’.”
This litigation is between the claimants and the Post Office. Fujitsu is not a party. Although one of the contracts between the Post Office and Fujitsu was uploaded to the electronic trial bundle on the penultimate day of evidence (by the Post Office’s solicitors) the litigation is not concerned with the details of that contractual relationship, other than (tangentially only) the cost to the Post Office of making ARQ requests, as the ARQ data is maintained by Fujitsu and charges are raised to the Post Office in this respect. However, Fujitsu’s involvement does cast something of a shadow. That contract has been addressed in terms of whether certain documents were, or were not, in the control of the Post Office. I have already dealt with the issue of PEAKs and KELs, and how the Post Office maintained initially it was not obliged to disclose these. Fujitsu personnel who were not witnesses were involved in providing information to the Post Office’s expert, Dr Worden. It was the Fujitsu witnesses who originally, and in my judgment unequivocally and directly, flatly contradicted what Mr Roll said could be done on the subject of remote access. I have also identified that in my judgment, the majority of the Fujitsu witnesses were more interested in following some sort of Fujitsu party line than they were in answering questions in cross-examination wholly frankly, although I exempt Mr Godeseth from that criticism. From what was said by the Post Office’s leading counsel to Master Fontaine at the very first case management hearing, those within the Post Office had relied upon Fujitsu in obtaining relevant information for its earlier public statements, and it plainly relied upon Fujitsu in terms of evidence of fact in the Horizon Issues trial, because it called a number of Fujitsu witnesses. I am unaware of the degree to which the Post Office has depended upon Fujitsu in the background to the Horizon Issues trial, and there is no reason why I should be, and I make it clear that I do not speculate on this. This information must have originated from Fujitsu at some point, but the Post Office must bear some responsibility for such incorrect statements having been made before, both publicly and in its pleadings.
Nor could it be said, in the Post Office’s and/or Fujitsu’s defence, that remote access was a side issue. It has been a very important central element of the whole dispute between SPMs and the Post Office over the Horizon system for many years. Mr Green for the claimants adopted a non-computing analogy, namely that of “a back door”. The origin of this was a phrase of Mr Roll, who referred to the back door to the system. The branch accounts can be likened to a locked room, to which the SPM has the key. This development in the case is comparable, after many years of the Post Office and Fujitsu maintaining that only the SPM has the key, that only the SPM could perform anything within the locked room, and if the Post Office and/or Fujitsu
needed access to the room the SPM would know because they would have to borrow the key, to now being an admission by the Post Office (and Fujitsu) (and a finding of fact by the court) that there is a back door which they could, and did, use without the SPM even knowing such a door existed. This analogy is not entirely apt, because software and remote access to edit transactions is not completely the same as physical access through a door, but it is an attempt to capture the general idea in lay terms.
The truth concerning remote access has now emerged in 2019, in group litigation that started in 2017. I find it notable that the truth did not emerge in the first Fujitsu witness statements that were originally served for the Horizon Issues trial. Such statements stand as the evidence in chief of witnesses of fact. They are supposed to be accurate. Minor corrections are not unusual and indeed are almost expected, as a trial approaches, as witnesses either research further or remember (when preparing for trial) other minor details. This topic, however, did not undergo that type of correction, and is a subject far more central and important than that. The truth only finally emerged in later statements, which were required to correct what I find were directly inaccurate statements in the first witness statements of Mr Godeseth and Mr Parker. There has been no adequate explanation for the contents of those first statements, which not only omitted this important fact, but contained evidence directly to the contrary. Those first witness statements were misleading. The statement in the Defence was misleading too. It ought also to be noted that the truth did not emerge internally within the Post Office in the email answers provided to internal inquiries in 2015 by senior Post Office personnel, such as the Chief Executive, who posed the specific question in preparation for providing evidence to a Select Committee and asked: “What is the true answer?”
She also said in the same email “I hope it is that we know this is not possible and that we are able to explain why that is”. The true answer is that, contrary to her aspiration, it was possible.
She also stated “I need to say no it is not possible and that we are sure of this because of xxx and that we know this because we have had the system assured.” The true answer to that was also “yes, it is possible”.
It is also difficult to believe that the signatory of the Statement of Truth in the Generic Defence would have signed that statement if she did not believe the contents of that pleading to be true. There is another relevant feature in this case, which is that the Generic Defence also includes a counterclaim seeking damages against the SPM claimants, including damages for fraud. Fraud is the most serious allegation that can be brought in civil litigation and there are special rules in relation to pleading it, which means that a pleading containing a fraud allegation should be subject to particular scrutiny before it is served.
It may therefore be that the Post Office itself fell into error as a result of information provided to it by Fujitsu on this important matter. It may be that some within the Post Office were themselves surprised by these revelations prior to, and during, the Horizon Issues trial. There is no need for me to speculate on this, and I do not do so. Certainly Mr Godeseth did not appear to have known about this for very long. Whatever the origin of this, and whether it came from Fujitsu, internally, being less than frank with the Post Office or not, the effect is that the Post Office has made specific and factually incorrect statements about what could be done with, or to, branch accounts in terms of remote access without the knowledge of the SPM. The evidence in this trial has made it clear that such remote access to branch accounts does exist; such remote access is possible by employees within Fujitsu; it does exist specifically by design; and it has been used in the past.
It follows that the previously stated public position of the Post Office to the contrary, in the statements to which I have referred above, is specifically wrong in fact.
Further, and as part of the Post Office serving supplementary witness statements from both Mr Godeseth and Mr Parker in respect of Mr Roll’s evidence, the Fujitsu witnesses adopted a position in the litigation, the rationale of which is somewhat difficult to understand. Responsibility for the incorrect and potentially misleading evidence of Mr Godeseth and Mr Parker (in their first statements) was effectively laid at the door of Mr Roll for giving what was said to be vague evidence, or evidence that, it was said, had not been fully understood by Fujitsu. Infallibility is a rare commodity, and everyone is capable of making mistakes. However, it is how one reacts to mistakes that is telling. In this instance, the initial reaction of the Fujitsu witnesses was to seek to shift the blame for their own misleading written evidence upon someone else. In this case, that “someone else” was their former Fujitsu colleague whose very evidence was responsible for exposing the full picture. This is not something that could have been arrived at lightly. To be fair to Mr de Garr Robinson, it was not a point he adopted on the Post Office’s behalf during the Horizon Issues trial. In my judgment, he was entirely sensible in not doing so. That it was done at all in the written Fujitsu statements speaks volumes, in my judgment, about Fujitsu’s reaction to being caught out.
Mr Parker’s supplementary statement blaming Mr Roll said the following:
“In paragraph 20 of Roll 2, Mr Roll describes a process by which transactions could be inserted via individual branch counters by using the correspondence server to piggy back through the gateway. He has not previously made this point clear. Now that he has, following a discussion with colleagues who performed such actions I can confirm that this was possible. I did not mention it in my first witness statement because, when faced with a less clear account in Mr Roll's first statement, my recollection was that if it was necessary for the SSC to inject a transaction data into a branch's accounts, it would have been injected into the correspondence server (injecting via the server was the default option which was followed in the vast majority of cases).”
The recusal application was issued the day after Mr Godeseth’s cross-examination had made it clear, not only that this remote access existed, but after he was taken in careful cross-examination through specific examples of Fujitsu personnel manipulating branch accounts, and leading to discrepancies in branch accounts. I am aware that criticism of the Post Office and Fujitsu in this respect may prove to be controversial, as earlier criticism of certain aspects of the Post Office’s case in Judgment (No.3) was not well received by it. However, if criticism is justified, I consider it would be detrimental to proper resolution of the group litigation if that criticism were to be withheld simply because it might lead to a further negative reaction by the Post Office. It is also an inherent part of the judicial function in any litigation to make findings, which may include criticisms where justified, that may be contrary to a litigant’s own view of the merits of their case. Some litigants are so convinced of the righteousness of their own position that they consistently refuse to accept any possible view of the litigation other than their own. Such a blinkered view is rarely helpful, and would be particularly unhelpful from a publicly owned institution.
I consider that criticism is justified of the Post Office’s incorrect previous statements – which included public and high profile statements such as the response to the BBC Panorama programme, whatever may have been said on the subject to the Select Committee, as well as in its own pleadings - about remote access without the knowledge of SPMs. The Post Office should have done its best to discern whether such remote access was possible when this subject first arose, and whether it had occurred, before these statements were made. Fujitsu should have been frank and unequivocal, internally, with the Post Office, so that there could be no possibility of incorrect statements on this important point being made publicly by the Post Office. The Fujitsu witnesses should not, in their first witness statements, have made the incorrect statements that they did. Had those initial statements been factually accurate, there would have been no need for the supplementary statements from them that eventually led to the true position being accepted by Fujitsu, and therefore by the Post Office. It is highly regrettable that such a situation as this should have developed. I am not making any findings on any future issues, whether of fact or law, yet to be tried.
There is however considerably more to resolving the Horizon Issues than simply making findings about remote access to branch accounts.
So-called “shadow experts” instructed by the Post Office
I return to the issue of so-called “shadow experts.” The parties in this case agreed that costs management would apply, and the Post Office in its costs budget for the Horizon Issues trial included, for the costs management hearing on 5 June 2018, an item for incurred expert costs in its costs budget in the sizeable amount of approximately £800,000. This was in addition to the amount incurred by that point in terms of the fees of Dr Worden. His costs at that stage were only £58,000. The sum of £800,000 broke down to about £300,000 paid to Fujitsu, and £500,000 paid to other experts, who were not being instructed to give expert evidence. Dr Worden was the expert who would be giving evidence to the court, which meant he would owe the relevant duties of independence to the court under the CPR. The description of these other experts was given in a skeleton argument for the Post Office in the following terms:
“The Defendant has spent around £500,000 on investigations by internally appointed experts for the purposes of determining its litigation strategy. The resulting material - which is privileged - has not been provided to the Defendant's expert for the purposes of this litigation.”
The claimants adopted the term “shadow experts” for these, a term with which the Post Office did not agree. At the cost management hearings, the claimants pointed out that by “internally instructed” this meant these experts were not instructed by the Post Office’s solicitors. It was also pointed out that there was no corresponding entry in the budget for any conference with any of their counsel, and this meant that this was “an entirely ring-fenced operation”. The Post Office did, however, reserve the right to recover their costs as costs in the litigation, and therefore this item was included in its costs budget. All of the material produced by these other experts was said to be privileged.
It is a highly unusual situation that entirely separate experts, instructed directly by a party, without the involvement either of that party’s solicitors or their counsel (all the more so when those experts are not even identified), are instructed on such a task, whatever that task might actually be. I recorded an adverse comment in the costs management order of 23 July 2018 stating that the Post Office’s incurred costs for experts were extraordinarily high, unreasonable and disproportionate; and that Fujitsu’s costs of assisting with the litigation, and the costs of these internally appointed experts did not, on the face of it, appear to be properly recoverable sums in the litigation. I should clarify that this adverse comment should not be taken as applying to Dr Worden’s costs that had been incurred at that stage, which were modest. It is not necessary to deal with this matter any further to resolve the Horizon Issues.
F. Documents and Available Information
There are certain categories or descriptions of classes of documents that have featured heavily in the evidence at the Horizon Issues trial. The path to disclosing them has not always been smooth. The majority, if not all, of the technical documents that relate to how Horizon was actually operating in fact in IT terms are in the possession of either the Post Office or (more usually) Fujitsu. The two most important categories, in my judgment, are Known Error Logs (also known as “KELs”) and PEAKs. The first of these records or logs known errors, which means errors with the Horizon system. The latter is a browser-based software incident and problem management system used by Fujitsu for the Post Office account, in other words for incidents and problems associated with Horizon that occur.
Disclosure had previously been a troublesome topic between these parties. I had in Judgment (No.3) on the Common Issues, been somewhat critical of certain aspects of the Post Office’s approach to the disclosure of some documents. These observations were not well received by the Post Office and some had been categorised, incorrectly, by Mr Parsons in his 14th witness statement supporting the recusal application as “critical invective”. Whether the contents of Judgment (No.3) encouraged the claimants to mount, or continue to advance, criticisms of the Post Office’s disclosure in the Horizon Issues trial (in the sense that they may have felt they had a “following wind”) is not clear. The Post Office, in the Horizon Issues trial (predominantly in solicitors’ correspondence, but also in some submissions), also criticised the claimants’ disclosure. For the sake of clarity, I wish to make it clear that the claimants had no such following wind.
The Horizon Issues are decided in this judgment based on all the evidence and the submissions. I deal with disclosure in the subsequent paragraphs of this judgment in order to assist and guide the parties concerning the future conduct of the group litigation. Some issues, such as whether certain Fujitsu documents were within the Post Office’s control, are likely to crop up again in respect of other types of documents in the future. The court’s interest in disclosure is to promote compliance
with the rules, and also to ensure cost effective progress of the litigation. The answers to the substantive Horizon Issues are no different because, for example, the experts were not given particular documents relating to the specific bugs in good time. For the most part, large numbers of KELs and PEAKs were provided to them, and the experts had an abundance of material. Some 5,000 KELs emerged from Fujitsu well after the trial ended, in October 2019. The experts did not consider these as they were not given them. Mr Parker had given evidence (at paragraph 61.9 of his 1st witness statement) that 1,491 KELs had been deleted at the time of writing that statement, but there was no reference to the existence of this far greater number that emerged after the trial was over.
The Post Office’s written closing submissions also included a separate section on disclosure as follows, which I will quote verbatim:
“I6. The Court’s interventions on disclosure
Post Office is concerned that the Court may have lost sight of the nature of the Orders made and of the approach which the Court ordered. It is striking that Cs have not themselves made any applications for specific disclosure nor have they advanced any complaint that particular disclosure orders have not been complied with.
The recent judgment of the Court of Appeal in Serafin v Malkiewicz & ors
[2019] EWCA Civ 852 @ para 118 is relevant:
“We are also highly troubled by the repeated demands and criticisms by the Judge regarding the Claimant’s disclosure, in circumstances where pre-trial disclosure had been completed by both sides at a time when both the Claimant and Defendants had been represented by solicitors and counsel, and no application for further disclosure had been made by the Defendants…” ”
These submissions identified what were said to have been “the Court’s interventions on disclosure” in the Horizon Issues trial. The authority identified by the Post Office, and stated to be relevant to that submission, concerned a defamation case in which the judge had demanded disclosure of a litigant in person, and made criticisms of him, both during the trial of that litigant’s libel claim, and in the judgment itself. In some instances the court in that case had, of its own volition, demanded production of certain documents from the litigant in person, and within a very short timescale. This had been part of findings made by the Court of Appeal in respect of procedural irregularity and the successful appeal against the judgment.
There were no “interventions on disclosure” by the court during the Horizon Issues trial, nor were there “demands and criticisms”. When asked during oral closing submissions about this passage in the Post Office’s written submissions citing Serafin, Mr de Garr Robinson did not demonstrate a great deal, if any, enthusiasm in advancing any claims of procedural irregularity having occurred during the Horizon Issues trial. He submitted that “this is not that case” – which does raise the question of why the passages from Serafin were included in the Post Office’s submissions in the first place.
There were, however the following matters that arose in this litigation during the Horizon Issues trial, which given paragraphs 1145 and 1146 of the Post Office’s closing submissions, should be recorded:
Redactions. In Judgment (No.3) I had identified the issue of unjustified redactions made by the Post Office of some contemporaneous documents that were used in the Common Issues trial. During the early part of the Horizon Issues trial some documents were referred to which again had redactions, which due to their nonredacted parts did not seem to me to be of a type that would necessarily attract privilege. In view of the previous history of this subject, I asked leading counsel for the Post Office to perform his own review of the redactions that had at that point been made of documents deployed in the Horizon Issues trial. This review resulted in some of the redactions in some of the documents being removed, and unredacted versions of those documents becoming available to the claimants, some of which were then used in cross-examination. Redactions of 2 documents were maintained on the grounds of legal professional privilege, which is entirely conventional. He also provided a helpful two page note explaining the approach that had been adopted. Asking counsel to do a review of this nature in these circumstances is entirely conventional. There was no short timescale imposed for this to be done. I asked for a summary of the position on 4 June 2019, which was after the interval which occurred as a result of the recusal application, and about 2 ½ months after I had asked him to perform the review. It was an outstanding matter, of a type sometimes referred to in a trial as “housekeeping”, which had not been entirely forgotten, but which had been overtaken in the latter weeks of March when the trial was temporarily halted.
A witness statement was ordered from the Post Office to explain the express, and factually incorrect, submissions made to the court by the Post Office about the Royal Mail’s refusal to produce, at the Post Office’s request, audit documents. The Royal Mail had, contrary to what the court had expressly been told by the Post Office, not even been asked by the Post Office for these. The Post Office’s leading counsel accepted that he had, entirely unwittingly, and on instruction, provided misleading information to the court and explained and apologised as soon as he discovered this. A witness statement from the Post Office, which was provided by Mr Parsons, was entirely justified in these circumstances.
A witness statement was also ordered from the Post Office in respect of the production of over 2,000 documents in disclosure by the Post Office in the interval in the trial period between 11 April 2019 (completion of the factual evidence), and the experts commencing their oral evidence on 4 June 2019. Given the timing and quantity of these, in the circumstances a witness statement (again, being provided by Mr Parsons) was justified. The first explanation given to the court, again on instruction, again proved to be wrong. These documents were originally said to have all been of some age (ten years or so) and discovered at Fujitsu on a “long-forgotten” server. One deployed the day after this explanation in cross-examination turned out to be dated 21 August 2018. A witness statement was again called for.
None of the above, in my judgment, fall into the category of the court losing sight of the nature of the orders that had been made concerning disclosure, or of demands or criticisms by the court, unjustified or otherwise. Further, the claimants made numerous complaints of the Post Office about disclosure. Indeed, the factually incorrect submissions made to the court about the audit documents and the Royal Mail came at the end of a court day when the claimants, by their leading counsel, expressly attempted to make a specific disclosure application. I was not prepared to hear that application at that point. Royal Mail was the correct respondent to such an application, not the Post Office. Royal Mail was not represented, nor even present in court, nor had any notice been given by the claimants to the Royal Mail of such an application. Each of these features is a far from promising start. I did however make directions for the making of a third party disclosure application against the Royal Mail, but this did not ultimately prove to be required. Once those at the Royal Mail were actually asked for these documents, the Royal Mail readily produced them. The claimants also made regular complaints about disclosure and would often point out that documents had been produced during the hearing. This is because documents were often produced during the hearing. This is something that had also occurred during the Common Issues trial.
The Post Office is, however, correct to point out that there have been no specific disclosure applications mounted by the claimants, with the exception of the one to which I refer in [566] above in respect of the Royal Mail. General complaints about disclosure only take a party so far, and that is not very far, if deficiencies are known about in advance of a trial, and no applications are made.
The parties had also voluntarily adopted Model C of the Disclosure Pilot, which I approved. This was done prior to the coming into force of the Pilot, the full title of which is Disclosure Pilot for the Business and Property Courts (“BPC”) which became mandatory in all BPC cases where the first CMC is held after 1 January 2019. This group litigation is not a BPC case and the first CMC was held before that date in any event. The Pilot is designed to run for a two year period. The Pilot is found at Practice Direction 51U, which is made under CPR Part 51.2. The features of the Pilot that are particularly relevant here are the Principles included in paragraph 2 of the Pilot. These state:
2. Principles, “document”, “adverse” and “known adverse documents”
2.7 Disclosure extends to “adverse” documents. A document is “adverse” if it or any information it contains contradicts or materially damages the disclosing party’s contention or version of events on an issue in dispute, or supports the contention or version of events of an opposing party on an issue in dispute.
2.8 “Known adverse documents” are documents (other than privileged documents) that a party is actually aware (without undertaking any further search for documents than it has already undertaken or caused to be undertaken) both
(a) are or were previously within its control and (b) are adverse.”
3.1 A person who knows that it is or may become a party to proceedings that have been commenced or who knows that it may become a party to proceedings that may be commenced is under the following duties (“the Disclosure Duties”) to the court—
to take reasonable steps to preserve documents in its control that may be relevant to any issue in the proceedings;
once proceedings have commenced against it or by it and in accordance with the provisions of this pilot scheme, to disclose, regardless of any order for disclosure made, known adverse documents, unless they are privileged;”
(emphasis added)
The fact that the parties chose Model C does not go to answer the complaints that the claimants have made about the Post Office’s disclosure. The Post Office had an obligation to disclose known adverse documents, as do the claimants as well. “Adverse” as defined plainly includes any documents that refer to bugs, errors or defects, or the operation of the Horizon system that led to potential impact upon branch accounts. This obligation is “regardless of any order for disclosure made”, in other words it exists whether any specific classes of documents are, or are not, already the subject of any order. The rational for this is obvious. Reliance upon the adoption of Model C is not therefore the answer to the claimants’ complaints in this respect. However, the fact that the claimants did not make applications for disclosure is relevant to the criticisms made at the Horizon Issues trial.
The Post Office has, essentially, submitted that the breadth of disclosure requests has been too wide. Particular criticism is that “Freeths’ [the claimants’ solicitors] did not control Mr Coyne’s requests”. I am not persuaded that it is the correct approach to matters of independent expert evidence to seek to “control” the documents that an independent expert wishes to see. The claimants did not issue any opposed applications for disclosure of documents which their expert wished to see, so a filter of some sort was applied in that sense to the documents Mr Coyne wished to see, and was not given.
Finally, disclosure is a particular feature of litigation in this jurisdiction. However, there was a clear perception in commercial litigation generally that the then-current disclosure regime had become inadequate and also that standard disclosure was often excessive in scale, cost and complexity. This is why the Disclosure Working Group was created in May 2016 at the initiative of Sir Terence Etherton, then the Chancellor of the High Court and now the Master of the Rolls. This led to the Disclosure Pilot.
It would be highly regrettable if disclosure in this case were to become yet another battleground between the parties. There are more than enough points in issue between the parties without adding endless disagreements about disclosure to that list. There are the following areas, however, of disclosure which should be recorded. In my judgment, the background to the evolution of the following categories does throw light on the parties’ different approaches to the litigation.
The Known Error Log or KELs
KELs are sometimes referred to in the singular – for example, in some places reference is made to the Known Error Log, as though it were a single Log. In practice, the Log as a whole includes multiple KELs, as they were referred to during the trial. If an incident or problem occurs, a PEAK is raised, and if the problem or incident refers to an existing issue which is known to have occurred before, reference will be made to that KEL entry, or more usually, it is said that the PEAK will refer to the KEL. Both
PEAKs and KELs have reference numbers; the former are usually all numbers after the prefix PC, whereas KELs have letters (taken from the identity of the employee who raised the KEL followed by four digits and a letter). Therefore, when a problem (which I shall refer to generally as an example as problem x) occurs for the first time, a KEL will be raised which will be given a reference. Subsequent PEAKs which identify the same problem x again, over time, afterwards will all refer to the same KEL. Sometimes a PEAK will be identified as including a problem which is similar to problems identified in more than one KEL, in which case both the KELs will be referred to within that PEAK. Any creation of a new KEL, or updating of an existing KEL, must be authorised by SSC before it can be seen by all users.
The experts agreed the following in the 2nd Joint Statement about both PEAKs and KELs: “KELs and PEAKs together form a useful source of information about bugs in Horizon but are a limited window on what happened. It is sometimes necessary to use evidence from both to try to understand, but even so they are not a comprehensive picture. It is to be expected that both KELs and PEAKs are incomplete in various respects.” It is also agreed that KELs will often give information about the impact of a bug or user error and they may also provide information about causes. There are other documents referred to as OCPs and OCRs. They are Operational Change Proposals and Operational Change Requests.
The subject matter of this litigation spans many years, starting from the first implementation of the Horizon system (what is now being called Legacy Horizon) in 2000. In the Letter of Claim from the claimants dated 28 April 2016, the Known Error Log was sought from the Post Office, that letter stating:
“We understand that Fujitsu maintained a 'Known Error Log' for Horizon and that such reports will have been provided to Post Office. Please see the list of the categories of documents relating to Fujitsu referred to below, that we request disclosure of.” Item 22 in the list of documents sought was “The 'known error log' kept by Fujitsu and provided to Post Office as referred to above, and all correspondence relating to the same.”
The answer in a letter from the Post Office’s solicitors against the specific item 22 was:
“In circumstances where you have not particularised any factual basis on which Horizon is defective, disclosure of these documents (if they exist) is not relevant, reasonable or proportionate.”
The suggestion in that letter that the Known Error Log was not relevant, is simply wrong, and in my judgment, entirely without any rational basis. The further suggestion, viewed with the hindsight now available, that the “known error log” may not exist, is disturbing. The claimants’ request used the precise title – “known error log” – and this clearly did exist. To suggest in an answer “if they exist” is somewhat misleading.
Item 23 in the same list of documents sought was “Internal memoranda from Fujitsu and POL referred to by Second Sight as identifying a `Horizon bug' within Horizon Online.”
The answer against that item was:
“We do not recognise the document to which you refer. Please provide further details.”
In my judgment, the documents sought in that entry must clearly include any PEAKs that identified a bug within Horizon Online. “Internal memoranda” is a plural reference, yet it was interpreted by the Post Office’s solicitors as though it were singular, and the request was for a single document, or a document with the title “internal memoranda”. This is, in my judgment, obstructive.
The claimants were not to be dissuaded, and sought the Known Error Log or KELs again. A reply from the Post Office’s solicitors on 13 October 2016 is relied upon by the claimants as showing that the Post Office was denying the relevance of the Known Error Log. This reply stated:
“The claims which you have particularised concern errors with the Core Audit Log. Following a review of the Known Error Log, Fujitsu have confirmed that there have been no logs in respect of Core Audit Log. The remainder of the Known Error Log does not relate to the claim which you have particularised and as such disclosure of this document is not relevant.”
(emphasis added)
Existence of the Known Error Log was at that stage accepted, but its relevance to the proceedings was now challenged. The Post Office’s solicitors stated that its contents “did not relate to the claim” and that “disclosure of this document is not relevant”. Disclosure of it was plainly resisted. The claimants did not therefore have it when the Generic Particulars of Claim was pleaded on 6 July 2017. In the Generic Defence, which is dated 18 July 2017, the Post Office changed its position, and now pleaded that the Known Error Log was not in its control. At paragraph 50(4) of the Generic Defence, the Post Office stated:
“It is admitted that Fujitsu maintain a "Known Error Log". This is not used by Post Office and nor is it in Post Office's control. To the best of Post Office's information and belief, the Known Error Log is a knowledge base document used by Fujitsu which explains how to deal with, or work around, minor issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide fixes have not been developed and implemented. It is not a record of software coding errors or bugs for which system-wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief, there is no issue in the Known Error Log that could affect the accuracy of a branch's accounts or the secure transmission and storage of transaction data.”
(emphasis added)
This was expressly challenged in paragraphs 3 and 4 of the Generic Reply.
The two main points to consider within that passage in the Defence are that the Post Office stated that it did not have control of the Known Error Log, and that entries within it related to minor/trivial matters and could not affect the accuracy of a branch’s accounts. This position was maintained, notwithstanding the terms of the
Post Office’s skeleton argument for the first case management hearing on 19 October 2017. That had stated:
“Cs’ response to the criticism of their case on Horizon is to argue that they do not have the access to information and documents that would allow them to plead any properly articulated case as to the “bugs” that they wish to blame for the shortfalls in their branches. Without prejudice to its position that the case is not properly pleadable, even at a generic level, Post Office is prepared to take steps to help Cs investigate Horizon, take an informed view as to whether they really wish to maintain their claims and, if so, decide how to proceed with them. Post Office wishes to put an end to the speculative attacks on Horizon and the related allegations as to its own conduct. It puts forward its proposals in paragraphs 2 to 8 of the Draft Order as a pragmatic compromise between the parties’ competing interests and concerns.” (italics present in original, emphasis by underlining added)
At the first case management hearing itself, upon specific enquiry by the court, this position was further maintained.
Disclosure of the Known Error Log was resisted by the Post Office on the grounds both of control, and content. So far as control is concerned, Mr Parsons had provided his 4th witness statement for that hearing, the first before me as the managing judge, which expressly stated at paragraph 35 “Despite Post Office explaining the irrelevance of the Known Error Log and that it was not within Post Office’s control…” and also at paragraph 37 “Due to the large amount of information involved and the fact that the Known Error Log is not in Post Office's control….”. This position was confirmed by the Post Office through its leading counsel in the following exchange with the court at that hearing, the transcript of that showing the following exchange.
“Mr Justice Fraser: Do you still maintain it is not in your control?
A: My Lord, yes. It’s a Fujitsu document.
Mr Justice Fraser: No. Just because it’s a Fujitsu document doesn’t mean it’s not in your control.
A: That’s right. At no point has Post Office ever suggested that it’s in its control so far as I’m aware in any of the letters that I – Mr Justice Fraser: Okay.
A: (Reading from pleading)
"To the best of Post Office's information and belief the known error log is a knowledge based document used by Fujitsu which explains how to deal with or workaround minor issues that can sometimes arise in Horizon for which often because of their triviality system-wide fixes have not been developed and implemented. It is not a record of software coding errors or bugs for which system-wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief there is no issue in the known error log that could affect the accuracy of branch accounts or the secure transmission and storage of transaction data."
This was the relevant passage in the Generic Defence that was read out and Post Office’s leading counsel then continued:
“It will be clear from what I've just read to your Lordship that Post Office thinks that this is a complete red herring.”
The position was then, again, expressly confirmed as follows.
“Mr Justice Fraser: I think it's going to be useful for me, certainly, for you to define all the different grounds upon which you currently resist disclosure of the known error log. Number (1) is you say it's not in your control.
A: Control.
Mr Justice Fraser: Number (2) you say the subject matter is not relevant. Is that right? A: My Lord, yes. An order that Post Office disclose documents relating to bugs and errors causing the branch account errors, if Post Office had control of the document and it dutifully complied with that order it would not be disclosing the known error log because, as far as it is aware, the known error log is not that kind of document.”
So far as the content of the Known Error Log is concerned, when asked by the court “does it contain any errors at all?” the following answer was given by the Post Office’s leading counsel:
“It contains things like there's a problem with printers. There's a printer. You have to kick it on the left-hand side to make the printer work. I mean there's a vast range of hardware problems of that sort and maybe some software problems …. but not the kind of bugs, errors and defects that the claimants are wishing to pursue in their particulars of claim so far as Post Office is aware.”
This exchange with the court then continued.
“Mr Justice Fraser: Well, that's the rider which slightly might concern---
A: Well, my Lord, unfortunately, that's all that Post Office can say because it's not Post Office's document. It's Fujitsu's document. Fujitsu are the experts.”
Given there was no witness statement from anyone at Fujitsu available on the subject at that hearing, that was as far as the matter could be taken on that occasion in terms of making any specific order relating to the Known Error Log. The Post Office had made clear that any order for disclosure that did not specify the Known Error Log, but was more general requiring disclosure of documents in relation to bugs and errors causing branch account errors, would not lead to disclosure of the Known Error Log in any event. A practical way forward, without requiring a fully argued separate application which would have included Fujitsu, was adopted whereby the IT experts were permitted to inspect the Known Error Log. The status of the Known Error Log as a disclosable document was resisted both on the grounds of control and relevance. It has to be acknowledged that this practical solution was one suggested by Mr de Garr Robinson.
It must therefore have come as a surprise to the Post Office, given the contents of its pleadings and the express submissions that it had made to the court, that both experts considered the Known Error Log to be highly relevant. This should also have led to the Post Office beginning to doubt what it was being told by Fujitsu, given the source of what the court was told about this was what Fujitsu had told the Post Office, as set out above. The explanation of what the Known Error Log was, what it contained, and its lack of relevance, was not remotely accurate.
Although a certain number of entries in the Known Error Log, which led to a significant number of different entries which are each called KELs being deployed in the trial and examined closely, were thereafter available, the issue of control of the Known Error Log and the Post Office’s earlier position resisting disclosure of this, did not go away. An entire appendix to the claimants’ opening submissions was devoted to criticism of the Post Office on disclosure, including its shifting position on the Known Error Log. In closing submissions, at the very end of the final day of the Horizon Issues trial, the claimants handed up a document entitled “Claimants’ Points in Reply”. Part F of that was a short document headed “KELs said not to be in the Post Office’s control” together with a variety of references. It ought also to be recorded here that some 5,000 KELs were later disclosed by the Post Office in October 2019, well after the trial ended, once the Post Office was told by Fujitsu that previous entries, which Fujitsu had previously told the Post Office were not retained, were in fact retained.
The subject of whether the Post Office had control of the Known Error Log therefore remained somewhat live. It simply was not, therefore, going to go away. The Post Office’s legal team had loaded version 12 of its contract with Fujitsu (“the Fujitsu contract”) onto the electronic bundle database on the last day of evidence, Day 20, for re-examination of its expert witness, Dr Worden. The contractual situation between the Post Office and Fujitsu was therefore before the court, which it had not been for the first case management hearing on 19 October 2017. It is also the case that “control” of documents (other than the KELs) held by Fujitsu could potentially crop up again in this litigation going forwards. It is cost effective and efficient to resolve that now, for the assistance of the parties.
I therefore gave both parties the opportunity to lodge further written post-hearing submissions restricted to whether the Known Error Log was in the Post Office’s control. It was a point relied upon by the claimants and it has to be resolved. There were two reasons for this. One is that in group litigation of this type, the parties’ approach to disclosure goes beyond a single document, or type of document. Guidance as to the court’s general approach will be of assistance in the litigation going forwards. There may well be other documents held by Fujitsu that need to be disclosed later, and the issue of control needs resolving. Secondly, the court is entitled to expect accurate evidence from parties on interlocutory matters, and accurate submissions.
The Fujitsu contract that was loaded onto the electronic bundle was a recent one. This document has a version history at the front which shows that it runs, with amendments therein identified, from 31 August 2006 to version 12 on 3 July 2017. Version 12 pre-dates the date of the Generic Defence, but in any event, clause 25 (which appears below at [602]) does not appear against any of the later version numbers in the version history of the Fujitsu contract. It is likely therefore that clause 25 in substantially the same (if not identical) form appeared in all the earlier versions of the Fujitsu contract prior to Version 12.
The claimants made the requested post-hearing written submissions based on the meaning of ‘control’, for the purposes of CPR Part 31.8, which is uncontroversial.
That rule provides as follows:
“(1) A party’s duty to disclose documents is limited to documents which are or have been in his control.
(2) For this purpose a party has or has had a document in his control if – (a) it is or was in his physical possession;
he has or has had a right to possession of it; or
he has or has had a right to inspect or take copies of it.”
Sub-paragraphs (a) to (c) are not exhaustive, but collectively those two subparagraphs mean, in this case concerning this document or documents, that if the Post Office has, or had, a right to possession of the Known Error Log, and/or a right to inspect or take or be provided with copies of it, it is within the Post Office’s control. Control will be established if a party has a contractual right to inspect or take copies of the document or documents in question. Given the terms of the Fujitsu contract, the claimants submitted that the Known Error Log is in the Post Office’s control and has been at all material times.
The Post Office’s written submissions on this were also received. These conceded, as it was put in paragraph 3, that the KEL was in its control. That paragraph stated at sub-paragraph (1):
“As at July 2017, Post Office understood the KEL to be a relatively unimportant internal working document produced and used by Fujitsu to assist in the performance of some services under the contract. On that basis, the KEL would not properly have been characterised as a “record…relating to the performance” of Fujitsu’s services under the Fujitsu Contract and would thus not have been within Post Office’s control. But, on the facts now known to Post Office, it would not contend the KEL to be outside its control.”
(emphasis added)
The title of the log – the Known Error Log, or known error log – includes rather obviously the two words “known” and “errors”. The Horizon Issues use the expression “bugs, errors and defects”. The presence of the word “error” in both might give an early indication of likely relevance of the log. Further, as shown in [603] below, in the Fujitsu contract the term records includes the phrase “full and accurate records” (emphasis added). Although records is used in the definition of Records – a point correctly identified by the Post Office, which it says is largely circular – it is clear that “full and accurate records” are included.
Clause 25 in version 12 of the Fujitsu contract states the following:
“25.8 In addition to its obligations under Clauses 25.2 and 25.3, Fujitsu Services shall provide the Court Case Support Services to Post Office in relation to prosecutions and other disputes (whether civil or criminal) with any third party including but not limited to any fraud, theft, breach of contract or impropriety (the “Court Case Support Services”). The Court Case Support Services shall include any matters whether they relate to Horizon, HNG-X or any other system provided by or on behalf of Fujitsu Services to Post Office, its agents or its subcontractors (including Post Office Service
Integrator and any Tower Contractor). Fujitsu Services shall provide the Court Case Support Services within the timeframes required by Post Office or the relevant court or other authority.
25.9 Without prejudice to Clause 25.3, the Court Case Support Services shall comprise:
the provision of copy reports;
the provision of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data) where such data is held by or in the control of Fujitsu Services;
the compilation of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data);
the interpretation of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data);
the provision of technical reports regarding technical aspects of any system (whether Horizon, HNG-X or otherwise);
live witness evidence at Court if any of the information provided (including without limitation that provided pursuant to Clauses 25.9.1 to 25.9.5) is challenged to the extent to which Fujitsu Services provided said information; and
the right of access to Records, including but not limited to information, reports and data, held by or in the control of Fujitsu Services, and the assistance of Fujitsu Services’ personnel with appropriate knowledge of the applicable Records (to the extent any such personnel remain employed or contracted to Fujitsu Services) for any independent experts and/or legal advisors instructed by Post Office and/or any other claimant(s) or defendant(s) and the Prosecution in any mediation, arbitration tribunal, court case or dispute in which Post Office is involved in relation to the Horizon and HNG-X or any other system provided by or on behalf of Fujitsu Services to Post Office.”
(emphasis added)
Records is defined in the Fujitsu contract in Schedule 1 “Interpretation”, in the following way:
“Records: means the full and accurate records relating to the performance of the Services.”
I consider that this term plainly includes the Known Error Log, as that relates to the performance of the Services, and in any event this contains data and hence also falls within clauses 25.9.2 and 3. I reject the lengthy analysis in the Post Office’s submissions on this point that the KEL is not properly characterised as being a record. It plainly is.
In my judgment the Post Office clearly has, and had, a contractual right to be provided with the Known Error Log by Fujitsu, given these are civil proceedings for (inter alia) damages sought by the claimants for breach of contract, and also fraud being alleged against the claimants by the Post Office by way of counterclaim. I do
not accept that the KEL is a type of document covered by the authority cited by the Post Office at paragraph 11 of its submissions on this point, namely working papers prepared by professionals for their own assistance in carrying out expert work for their clients; Hanley v JC & A Solicitors [2018] EWHC 2592 (QB) at [42]. That point is , in my judgment, plainly wrong. Given the terms of the contract between the Post Office and Fujitsu, it is not necessary to consider that point further.
I consider it verging on entirely unarguable, given the express terms of the Fujitsu contract which is now available to the court, that the Known Error Log was not in the control of the Post Office. Mr Parson’s witness statement had merely stated “it was not within Post Office’s control” and I simply cannot understand the basis for that statement, given the express terms of the contract that the Post Office had with Fujitsu. It plainly is in the Post Office’s control, given the terms I have reproduced above, and that point is now conceded.
The fact that the Post Office has submitted that in July 2017, on its understanding then, the KEL was not a “record” that “related to the performance” of the Fujitsu services demonstrates a worrying lack of knowledge on the part of the Post Office, about both Horizon, and Fujitsu’s record keeping. It also means, when this is put together with what the Post Office, by its leading counsel, submitted to the court at the 1st case management conference, that Fujitsu were extraordinarily inaccurate about the information it provided to the Post Office at that stage of these proceedings. As at 2017, the Horizon system (both Legacy Horizon and Horizon Online) had been in use for about 17 years. KEL appears in a table of Abbreviations/Definitions at F/87/3 in a document entitled “CS Support Services Operations Manual” that is dated 29 January 2001. That document is referred to in footnote 34 of Mr Coyne’s 1st report, and three passages within it state the following:
“4.5.1 Maintaining the Known Error Log on the SSC intranet site
The SSC generates and maintains a Known Error Log (KEL) system that uses searchable documents in HTML format. The mechanism for searching is a query entry in an intranet site. The KEL system is available to first, second, third and fourth line support units as well as SSC staff.
4.5.2 Transferring knowledge between support units
The SSC intranet site has KEL search facilities and other useful diagnostic data, documents and tools.
SSC and SMC staff raise KELs based on customer-observed symptoms. KELs are further maintained once the fault has been resolved.”
“4.7.1 Known Error Logs (KELs)
The intranet site holds known error details in Microsoft Word format, the contents of which may be searched for, in full text form. Documents are created to a defined template wherever possible. An application has been generated which limits the properties of the document to a subset of possible values, for clarity and ease of search. This application is made available to all support units.
The process for creating KEL entries outside of the SSC has not yet been formulated, but it is expected that no KEL will be allowed onto the system before it has been authorised by SSC staff.”
(bold present in original)
The Post Office plainly did not know this when making the submissions that were made in 2017. The Services Operation Manual had been in existence for 16 years by then, and this ignorance about the Known Error Log, which is clearly explained within that document, is most surprising.
Even without such an express obligation in the Fujitsu contract itself, I find it very difficult to believe that Fujitsu could mount any sort of coherent or legitimate objection to producing it at the request of the Post Office. The assertion that the Log was not in the control of the Post Office is, in my judgment, not a valid one.
Similarly unsustainable is the submission that was made to the court in 2017 that its contents were not relevant to the Horizon Issues, and that it did not contain references to bugs and errors having an impact on branch accounts. On that latter point, the Post Office’s submissions were based upon what the Post Office had been told by Fujitsu. Fujitsu were, again, being clearly inaccurate about what they had told the Post Office was actually contained in the Known Error Log.
Nor is the passage quoted above in the Generic Defence an accurate description of the Known Error Log. The Known Error Log does far more than “explain how to deal with, or work around, minor issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide fixes have not been developed and implemented”. The Known Error Log appears to be a comprehensive record of all the errors and defects of which Fujitsu have become aware over the life of Horizon. On the basis of the evidence that has been heard in the Horizon Issues trial, there are many issues in the Known Error Log that could affect the accuracy of a branch's accounts or the secure transmission and storage of transaction data, contrary to what is pleaded by the Post Office. Some KELs led to software fixes too. Indeed, Mr Coyne has used it in order to discover the existence of, and then investigate, the numerous different bugs from what was called “the Bug Table” in the 2nd Joint Statement, 11 of which Dr Worden agrees in his written evidence. Dr Worden himself has discovered some of these bugs, from the sample of the entries in the KELs that he had inspected. It is the KELs that are the origin of these expert conclusions, due to the text in the entries that they contain.
This discovery of these different bugs by both experts has directly flowed from the entries in the Known Error Log by Fujitsu, together with the associated PEAKs.
It is only fair to record that after contested disclosure had been considered at the first hearing before me on 19 October 2017 (dealt with in detail above), the Post Office’s leading counsel suggested the “pragmatic solution” that was giving Mr Coyne the opportunity to inspect it. It was because of this that the matter of control was not resolved at that point, as disclosure was permitted. Mr Coyne inspected the Known Error Log at Fujitsu’s premises. He found it highly relevant. The claimants received most of the KELs they had for the trial on 10 May 2018, with further KELs in respect of particular bugs disclosed on 16 November 2018, one month after the first expert
reports of Mr Coyne and Dr Worden had been served. Further KELs were provided in 2019, and a very substantial quantity of 5,000, well after the trial in October 2019. There has been no detailed explanation for that very late post-trial disclosure in any form of any witness statement from anyone at the Post Office or Fujitsu.
The KELs provided to the experts prior to their reports featured centrally not only in Mr Coyne’s two reports, but those of Dr Worden too. They were also the subject of a great deal of cross-examination by both sides. I consider that disclosure of the KELs to the experts – the pragmatic solution suggested by leading counsel for the Post Office - has been central in the discovery and investigation of the bugs, errors and defects that the experts agree were, or are, present in the Horizon system, and also of the bugs, errors and defects that are not agreed, but upon which both experts opine and in respect of which I make findings in the Technical Appendix. Dr Worden has also analysed them extensively, and devoted a separate appendix of his report to them. The best description of the KELs is taken from Dr Worden, who stated that they were “a rich source of evidence”. The notion that they were not relevant, or did not contain relevant material, is extremely difficult to fathom, and I do not understand why, or how, Fujitsu would or could sensibly choose to inform its own customer of many years, the Post Office, directly to the contrary. If Fujitsu’s description of the contents of the KELs had been taken at face value and not challenged, then the knowledge now available to the two experts and to the court about the extent of bugs, errors and defects would not have been available, with the obvious detrimental impact upon the fair resolution of the Horizon Issues.
The further developments, after the trial, that led to disclosure of another 5,000 KELs by the Post Office after the trial in October 2019 are dealt with at [627] below and following under the heading “Further post trial disclosure”.
PEAKs
Turning to PEAKs, upon inspecting the KELs at Fujitsu’s premises in 2018, Mr Coyne discovered the existence of a different category of documents called PEAKs, as these were referred to within different KELs. These documents were requested from the Post Office in July 2018. The claimants maintain that the Post Office were obstructive over this, but whether that is correct or not, over 218,000 different PEAKs were disclosed by the Post Office’s solicitors on 27 September 2018. This was two weeks before the date for exchange of experts’ reports. A further 3,866 were disclosed on 25 October 2018 – two weeks after the experts’ first reports. OCP documents (which are mentioned in PEAKs) were disclosed on 24 January 2019, and a number of PEAKs were identified by Mr Coyne in his supplementary report which he had identified but which had not been disclosed. These were provided (or extracts of them) on 18 February 2019, which is only three weeks before the Horizon Issues trial started.
A large quantity of further documentation was then disclosed by the Post Office, on dates that were not only well after the dates ordered for expert evidence to be served, but on dates that were even after the intended end date of the Horizon Issues trial itself. 2,590 OCRs were disclosed on 18 April 2019, and other PEAKs were disclosed on 31 May 2019, literally one working day before Mr Coyne was called to give evidence on 4 June 2019. This highly unsatisfactory situation led to my seeking an explanation. I consider an explanation was plainly required given the timescales. This is a large quantity of documents, dealing with central elements of the Horizon Issues, and the stage at which they had been provided was very late. PEAKs are important documents. I was told in submissions that Fujitsu had discovered an old database that had been copied more than ten years previously, and as soon as Fujitsu had told the Post Office this, matters had moved speedily and the Post Office had provided the contents of this old database to the claimants.
However, the day after this explanation was given, and during cross-examination, a PEAK dated 21 August 2018 was put to Dr Worden and the court was told this was one of the documents that had been disclosed on 31 May 2019. This plainly could not have come from “an old database” in excess of ten years of age which Fujitsu had only recently discovered. A further and more accurate explanation was plainly called for.
I had already ordered another witness statement to be provided by the Post Office’s solicitors, this one being the 18th witness statement of Mr Parsons, which was served on 19 June 2019. The contents of that witness statement were highly unsatisfactory. Firstly, it showed that the existence of the forgotten Fujitsu database came about because of questions raised by the Post Office’s solicitors “during preparation for the Horizon Issues trial at the beginning of April 2019”. The Horizon Issues trial in fact had an enforced interlude, from 21 March 2019 when the recusal application was issued, until that application was dismissed on 9 April 2019. In other words, those questions were asked very late in the trial process. Further, Mr Parsons’ witness statement stated, so far as the latter disclosure of PEAKs on 31 May 2019 is concerned, the following:
“20. In the course of investigations into the bugs referred to by Mr Coyne and/or Dr Worden, including in the Second Joint Statement, on 29 March 2019 Mr Lenton identified PCO273234 to Ms Simmonds as being a document relating to the Drop and Go Bug. For the next couple of months my firm continued to investigate these bugs and a number of further documents relating to these bugs came to light. PCO273234, along with 17 other documents (excluding redundant image files) that had come to my firm's attention when preparing for the Horizon Issues trial were disclosed in a single batch on 31 May 2019.”
(emphasis added)
Mr Lenton is a member of Fujitsu personnel. The 2nd Joint Statement was signed by the two experts on 25 February 2019. This passage means that one month after that document, Mr Lenton identified the specific document relating to the Drop and Go Bug. This is one week after Mr Godeseth had completed his evidence. I find the rather glossed-over reference in the witness statement to “continuing investigations” that went on “for the next couple of months” extraordinarily opaque, in the context of a trial that commenced on 11 March 2019, and a document identified on 29 March 2019. I also find the wholesale lack of any explanation of when the “further documents relating to these bugs came to light” – and indeed how – puzzling, again in a witness statement ordered specifically to explain what was already a highly unusual situation. Further, the PEAK relating to the Drop and Go bug was plainly identified by Mr Lenton, a Fujitsu employee, on 29 March 2019. It is in my judgment verging on the extraordinary, in the context of any trial that was actually underway, that it
took two months more for this PEAK to be disclosed to the claimants. This is particularly concerning in the context of this trial between these parties on these issues. It is also obvious from the witness statement that the PEAK in question dealing with the Drop and Go bug was not disclosed immediately, until “for the next couple of months my firm [ie the Post Office’s solicitors] continued to investigate these bugs.” There is no reason that investigations, whether ongoing or not, should delay disclosing a document of this nature to the other side, particularly in the middle of such a trial as this.
The conclusion to be drawn from this is that disclosure was given in a manner that could only have disrupted and delayed proper investigation of the issues contained in the documents. In this specific case, that includes the Drop and Go bug. That document should have been provided to the claimants very rapidly once it came to the attention of the Post Office’s solicitors, not kept “for the next couple of months”, as Mr Parsons puts it. In subsequent trials, if relevant documents of this nature are discovered by either side or their legal advisers, it is a matter of some importance that disclosure is given very rapidly. This is particularly important whilst the trial is underway. A failure to do so risks disruption, delay and increased costs.
The experts agreed the following about PEAKs and their content. “PEAKs record a timeline of activities to fix a bug or a problem. They sometimes contain information not found in KELs about specific impact on branches or root causes – what needs to be fixed. They are written, by people who know Horizon very well. They do not contain design detail for any change. They are generally about development activities and timeline rather than about potential impact. PEAKs typically stop when development has done its job, so they are not likely to contain information about follow-on activities, such as compensating branches for any losses.” It is also agreed, and indeed can be seen from the actual PEAKs themselves, that some of them record observations of financial impact. PEAKs do not however always use the symbol “£”, which was a search item used by Dr Worden to create a set of 259 KELs that contained that symbol. Even where they referred to a financial discrepancy, PEAKs sometimes just recorded the figure, or sometimes used “pds”, and did not use the symbol “£”.
Release Notes
A spreadsheet listing the Release Notes was produced which shows that there have been a great number of changes and updates both to Legacy Horizon and Horizon Online over the years. The total number of Release Notes is 19,842 at final version. Each expert provided an estimate of how many changes per week there have been to Horizon. Mr Coyne’s estimate is approximately 19 changes per week. Dr Worden’s estimate is 5 changes per working day. Given the number of working days per week, these figures are broadly the same. No details of the actual Release Notes were provided. The spreadsheet was not, in my judgment, complete, nor was it in useful usable form. No Release Notes detrimental to the Post Office’s case have been produced. The Post Office adopted an unhelpful approach to Release Notes, requiring the claimants to identify from the very difficult spreadsheet which Release Notes they would like to see, with an offer to provide those. On 14 February 2019 the claimants’ solicitors indicated which ones they did want to see. On 20 February 2019, the Post Office’s solicitors stated “we are currently taking instructions in relation to the requests”. Both the request from the claimants, and the response, are wholly unhelpful
and far too late. To be fair to the Post Office, a late request will undoubtedly lead to a late response. Further, the spreadsheet itself is almost wholly unusable.
I have commented upon the approach of the Post Office to disclosure in this litigation before, and not in favourable terms. At the first CMC in October 2017, I said in a short ruling on disclosure that the Post Office had been “obstructive” in its pre-action behaviour in this respect, and that its attitude to disclosure up to that point had been somewhat less than ideal. In Judgment (No.3) I made some further criticisms, which it is not necessary to repeat. It is regrettable that the Post Office’s approach to disclosure, admittedly this time in conjunction with Fujitsu, has continued in different respects to fall short of what is required. However, it does seem to be improving and it is to be hoped that such improvement will continue. Particularly in litigation such as this, with enormous distrust and suspicion on both sides, it is simply counterproductive to have that atmosphere added to by an unnecessarily combative stance on disclosure.
The claimants submitted in their opening that the Post Office’s approach to disclosure had the effect of impeding the claimants from obtaining a full view of the documents and the totality of the Horizon system. I agree with that submission. However, balanced against this, it does have to be remembered that the claimants did not issue any applications for disclosure, as I have already explained.
The Post Office’s case in this litigation will be subject to exactly the same degree of measured and independent assessment, and weighed in the same objective way, as the claimants’ case, and as it would be were these failures not present. However, there must be a change in approach to this group litigation. Disclosing documents late, and in the middle of trials, in such sizeable numbers, and failing (as clearly identified from the passage of Mr Parsons’ own witness statement referred to in [618] above) to act with expedition when clearly relevant documents are unearthed actually during a trial, has two distinct effects. It is disruptive to the proceedings, and it leads to an increase in the costs and/or causes delay. It is the antithesis of co-operation, which the Civil Procedure Rules expressly require. I consider that these passages within this judgment should provide sufficient encouragement to a more constructive approach to prompt disclosure of obviously relevant documents when (or if) they are discovered shortly before, or even during the course of, forthcoming trials.
A PEAK discovered on 29 March 2019 relating to the Drop and Go bug ought to have been disclosed to the claimants within a period of about 48 to 72 hours, not two months. With modern electronic communication, that is a readily achievable timescale.
Further post-trial disclosure
There was then yet further post-trial disclosure given by the Post Office during September 2019. However, the solicitors for the Post Office also notified the court in a letter of 3 October 2019 that further previous information given to the claimants and to the court in the Post Office’s e-disclosure questionnaire was, again, wrong. The letter to the court stated inter alia:
“We write further to the email sent by the Claimants' solicitors to the Managing
Judge's clerk on 27 September 2019 at 12:29, which brought to the Managing Judge's attention that 32 documents relating to the Horizon Issues Trial were disclosed by Post Office on 25 September 2019.
During the course of responding to a number of queries that the Claimants' solicitors had raised in relation to the disclosure of these documents (which can be located at [H/358] of the Horizon Issues Trial Bundle), it has come to Post Office's attention that a statement within its EDQ is incorrect and that there may be previous versions of the Known Error Log that have not been disclosed to the Claimants. Please find enclosed a letter which was sent to the Claimants' solicitors today explaining the background to these documents coming to light and the proposals which have been made in respect of providing any disclosure which may be sought by the Claimants.”
(emphasis added)
628. The actual statement that was wrong was further explained in the accompanying letter to the claimants, which stated:
“In respect of your query regarding the fact that intervening changes to KELs are not captured on the face of the KELs disclosed, we have made further enquires with Fujitsu to confirm our understanding that previous versions of the KELs are no longer held. We regret to say that these enquiries have revealed that our understanding was wrong.
As you will be aware, Post Office's EDQ stated that "[t]he KEL only contains the current database entries and is constantly updated and so the current version will not necessarily reflect the version that was in place at the relevant time. The previous entries /versions of the current entries are no longer available".
This statement was based on info provided by Fujitsu. In response to our recent enquiries, however, we were informed by Fujitsu on 30 September 2019 that this is incorrect and that previous versions of KELs are available.
This takes Post Office greatly by surprise. It relied on the information provided by Fujitsu at the time of filing its EDQ that such documents were not available. It is extremely sorry that this information was incorrect.
Regardless of whether these documents may or may not be "adverse", Post Office has taken immediate steps towards arranging for the previous versions of these KELs to be extracted and has instructed Fujitsu to begin producing a script to extract the documents into a readable HTML format. Unless you disagree, once this script has been produced we will ask Fujitsu to extract all of these versions.
When this has been done, again unless you disagree, we propose to take immediate steps to disclose previous versions of the KELs which were referred to by either Dr Worden or Mr Coyne in their expert reports or the joint statements or the bug table, or were for any other reason included in the trial bundle.
Disclosure of the previous versions of these KELs (where there are previous versions) will be given as a matter of urgency.
We invite you to tell us whether the Claimants also wish to be provided with disclosure of previous versions of the KELs which were not referred to by the experts or included in the trial bundle.
Given the seriousness of this matter, we propose to notify the Managing Judge of this development immediately and will send him a copy of this letter under cover of an explanatory email.”
(emphasis added)
This was correctly characterised by the Post Office’s solicitors as a serious matter. Further, when these KELs were produced and disclosed, it turned out that there were approximately 5,000 of them. This is an extraordinary number, and far more than were available to the experts for the trial.
There was no explanation of the way that what was obviously now incorrect information – that previous versions/entries of KELs were not retained by Fujitsu – had been communicated to the Post Office’s solicitors, or by whom, and there was no explanatory witness statement provided on a voluntary basis. I did not order one, nor did this disclosure interrupt preparation of the draft judgment, or lead to any application for further evidence or cross-examination.
I include it therefore for completeness. There are two points that arise from this.
It would have been a matter for argument, and an exercise in discretion, as to whether any further submissions or evidence (including cross-examination) would be permitted, had such an application been made. The Post Office were pragmatic and sensible, and essentially left this decision to the claimants. The claimants made no such application. However, had they done so, my preliminary view (without hearing any argument) is that I would not have permitted the Horizon Issues trial to be resumed and further evidence adduced. This is for three reasons. Firstly, there has to come a point when the parties are given a decision on the Horizon Issues. The trial had already been interrupted to a substantial degree by the recusal application. This trial was supposed to be completed in April 2019. Resuming it would have led to that date becoming sometime in early 2020. The over-riding objective requires consideration of other court users in other cases. Secondly, subsequent stages of the group litigation would, once again, become consequentially delayed. It is with a sense of wistfulness that I recall a statement at [16] in Judgment No.3 that this litigation would be speedily progressed. Thirdly, the parties have told me that they wish to mediate before Christmas 2019 and would like to know the outcome of the Horizon Issues trial to assist in that process. This would not be possible if the trial were resumed.
This extensive further disclosure does, however, rather reinforce the criticism which the claimants level at the Post Office and Fujitsu. The Post Office are the party in the litigation, and they are the party with the disclosure obligations. They obviously rely upon Fujitsu in this respect. Quite how Fujitsu, a reputable company with a worldwide reputation who have been involved in Horizon for nearly 20 years, could be so wrong on yet another important point going to Horizon is not explained, and will remain unclear. Given the statements made by Fujitsu, relied upon by leading counsel for the Post Office at an earlier hearing and repeated to the court, that the KELs did not contain any material relevant to bugs, errors and defects, a pattern has emerged on the part of Fujitsu. This is that its statements about KELs have been shown to be clearly wrong on more than one occasion, and in important respects.
Criticisms of the disclosure of the claimants
I turn therefore to the criticism that the Post Office has levelled at the claimants. It should be borne in mind that the bulk of the disclosure burden fell upon the Post Office, simply because of the nature of the Horizon Issues. However, that does not mean that the claimants were not themselves obliged to give relevant disclosure to the Post Office. This disparity is obvious in the number of documents disclosed by each party, which the Post Office in its Closing Submissions stated was some 517,965 documents by the Post Office, and only 1,525 by the claimants. Obviously that previous number does not include the 5,000 KELs disclosed by the Post Office in September and October 2019, after the submissions. The comparison in the figures demonstrates the different burden upon the claimants and the Post Office. It is also correct to identify that for many categories of very important documents, the Post Office is reliant upon Fujitsu. The claimants are not.
The Post Office submitted that despite it making appropriate and narrow requests for Model C disclosure of documents held by those claimants who were witnesses in the Horizon Issues Trial, the claimants have not engaged in discussions concerning the scope of the claimants’ disclosure nor provided disclosure of the documents sought by the Post Office.
The Post Office, in the context of its complaints about the claimants’ disclosure, submitted that “it was therefore not appropriate for claimants to serve claimantspecific evidence from current or former SPMs” because of the nature of the Horizon Issues and the terms of the Order that ordered witness statements, and that the Post Office sought to avoid making disclosure requests nearer to the trial. However, once the claimants did so, and served their nine witness statements in evidence in September 2018, the Post Office’s solicitors wrote to the claimants on 22 October 2018, 7 November 2018, 30 November 2018 and 20 December 2018 and did not receive any response. Even though the Common Issues trial was underway for part of this time, I do not see why such letters should have gone unanswered by the claimants’ solicitors. When the response did come, it was to the effect that there would be no further disclosure as this was not envisaged by existing orders.
The Post Office contrast this with the disclosure it was prepared to give in respect of specific witnesses, and also stated that “given that claimants had flouted that Order, Post Office sought to understand what searches (if any) had been conducted by claimants so that it could understand whether further disclosure would be required in light of claimants’ evidence. A further request for details by claimants was made by Post Office on 17 January 2019.”
I am invited to contrast this lack of response by the claimants, which it is said should be viewed in light of the approach adopted by the Post Office. One example of the further disclosure that Post Office had agreed to give which was outside that ordered by the Court was disclosure made in respect of the “operation of branches by Angela Burke, Akash Patny, Anup Patny, Jayesh Tank, Setpal Singh and Adrees Latif….”
The Post Office complains that the claimants “clearly intended disclosure to be a onesided exercise”, seeking disclosure of this nature but refusing to provide their own claimant-specific exercise. Mr Tank did search, in February 2019, within an archived Subpostmasters “Yahoo Groups” forum where he used to post (which means to make comments on the site) and after doing so, Mr Tank identified a post he made on 13 December 2011 relating to the £195.04 shortfall referred to by Ms Van Den Bogerd,
and also a post he made on 29 September 2014 relating to the £600 shortfall that he refers to in paragraphs 6-11 of his witness statement.
Mr Tank has also found a letter he received from Post Office’s Agents Accounting Team in Chesterfield dated 13 October 2014 which related to this shortfall (and which Mr Tank referred to in his forum post).
Mr Tank was cross-examined about this, and explained that at the time of his first witness statement he “hadn’t fully researched the whole background” and he did not realise at that time that the post he had made previously was as important as it had subsequently become, by which he meant in the trial. He also said he had only become aware of the £195 loss to which it related once he had read Mrs Van Den Bogerd’s witness statement. He also explained that he kept a box file and that was where he found the documents. Complaint is made of the fact that the documents that were disclosed should have been disclosed on 17 July 2018, the date for disclosure. I accept that, and even if one makes the assumption in the claimants’ favour that they did not know at that date that Mr Tank was to be a witness, certainly no later than the date of his first statement (which was 28 September 2018) disclosure ought to have been given. Mr Tank’s lack of understanding of relevance and disclosure should not have arisen, and the existence of his box file should have become known to the claimants’ solicitors far earlier than it was.
Further disclosure was also necessary in relation to Mr Latif. This came about because he served an amended witness statement on 1 March 2019, which came about following enquires with Mr Latif's former branch manager, who consulted the documents within the branch to confirm the date on which particular transaction corrections had been issued.
He explained this in his cross-examination:
" Q. First question about that: by amendment to your statement you now say that this was in around January 2018 rather than March 2018. A. Correct.
Q. What caused you to make that correction?
A. I had a look at the -- we hold the records for the information in the office, so I had my assistants look at the records, transaction logs and that's when I confirmed that it was January rather than March. (Inaudible) was logs in March as well. We made calls effectively every month to Horizon help desk concerning this issue.
Q. When do you say you asked your assistants to check about the date?
A. Yes.
Q. Sorry, when do you say that happened?
A. It was after I made the initial statement, I was checking.
Q. Roughly when, Mr Latif?
A. It would have been a few weeks ago, sir.
Q. So is it right that you didn't check those records from the branch before making your witness statement?
A. No, I thought I was correct but I double checked and made sure that actually in fact they were correct, those (inaudible), so I was right but initial incident happened in January when TA (inaudible) transaction acknowledgement, the TC, the corrections, they came in March.
Q. Okay. So you say, do you, that the TCs in relation to these transaction acknowledgements came in March 2018? A. Yes.
Q. And you say that's something you have checked from your records?
A. Yes."
The Post Office made requests for disclosure during the trial, on 15 March 2019 (which was the end of the first hearing week) requesting disclosure of various documents relating to Mr Latif and Mr Tank and their answers in cross-examination. This also included requests of CCTV recordings which Mr Tank said he had looked at. These requests were refused on 27 March 2019, and the claimants stated that this refusal was on the basis that they were either outside their control, no longer existed or were not relevant to the claimants’ evidence and were not relied upon at trial.
Documents that were held in Mr Latif's former branch were plainly outside of his control, even though his former branch manager had accessed them previously. However, that does not mean that the former branch manager could not at least have been asked for them. Neither the claimants did that, nor indeed did the Post Office (so far as I can tell). I consider the claimants’ attitude in this respect unhelpful, although it should be noted that for at least the documents in respect of Mr Latif, the Post Office could have taken its own steps in this regard.
There are some unsatisfactory aspects to the Post Office’s submissions about the CCTV footage. Mr Latif clearly said in his first witness statement at paragraph 7 “I also checked the CCTV to make sure that I had performed the transaction correctly.” Secondly, the Post Office’s closing submissions refer to requests for “the CCTV recordings which were consulted by Mr Tank for the purposes of drafting his witness statement”. He was asked about the CCTV during his evidence and did not say that he had consulted it for the drafting of his witness statement, he said that he had checked it at the time. Had the Post Office wished, it could have asked for that footage shortly after receiving that statement. It is also the case that the event in respect of which he referred to the CCTV occurred in July 2015. It may be that the CCTV images do not still survive, so long after the event in question, but this point was not pursued in cross-examination.
There is, however, one major difference between the claimants and the Post Office in this respect. The Post Office were entitled to, and did, explore with specific witnesses certain issues in respect of disclosure by asking the relevant person (in the above example, Mr Latif) questions about it. The claimants did not have this opportunity for the following reason. The majority of complaints made about the Post Office’s disclosure arise as a result of Fujitsu. Save for rare circumstances, such as Mr Parson’s specific witness statement about the Royal Mail issue, all that the claimants and the court are usually told is “Fujitsu say X” or “Fujitsu told the Post Office” such and such. Specific individuals are not identified, and they are certainly not called as witnesses to be asked questions about their approach, or (for example in respect of KELs) how they could have been so factually wrong in their explanation of the KELs’ contents. I have formed a dim view of the accuracy of information of this nature coming from Fujitsu, regardless of the person in that organisation responsible for communicating it.
Notwithstanding this, I wish to make one matter completely clear. If a witness refers to a particular document in their oral evidence, and the other party in this litigation wishes to see a copy of that document, then (absent issues of privilege) that ought to be disclosed, even more so if a specific request is made. This is by far the most sensible and proportionate way forward. It is in no party’s interests to engage in squabbling about relevance of a specific document in those circumstances. It costs far more to argue back and forth in solicitors’ letters than it does simply to give disclosure. In any event, the relevance of the document is made clear by the fact that it has been referred to orally in evidence.
It is also correct to say that notwithstanding the different burden upon the different sides in terms of disclosure in the Horizon Issues trial, which has undoubtedly been far heavier for the Post Office, this burden will not always be borne in these proportions. The Post Office also has the benefit of a far higher number of legal advisers. As at the date of this draft judgment, simply in terms of the total number of leading counsel engaged in the group litigation, the Post Office outnumbers the claimants by 4:1. In terms of the number of solicitors’ firms, the Post Office currently outnumbers the claimants by 2:1; although only one firm was acting for the Post Office on the Horizon Issues, and the second firm acted for the Post Office on its unsuccessful attempt to appeal the Common Issues.
Summary on disclosure
The Post Office is, in my judgment, somewhat at risk of downplaying certain unsatisfactory aspects of its own disclosure that have emerged during the Horizon Issues trial. For example, the incident concerning the incorrect statements made to the court about the Royal Mail refusing disclosure without an order of the court was described in the Post Office’s closing submissions as a “mix up” which was “regrettable but should not be the subject of criticism of Post Office.” I do not consider telling the court something in express terms which is factually wrong can properly or correctly be identified as merely a “mix up”. Two points that were expressly explained by its leading counsel were simply not factually accurate. The Royal Mail was not refusing disclosure, and it did not require an order of the court before it would provide certain documents. It had not even been asked for them.
Similarly, in its closing submissions the Post Office referred to the interlocutory discussions about the Known Error Log as a “debate about relevance”. That again downplays what had happened, including the positive submissions made, that the Log was specifically not relevant and even if an order were made in respect of documents showing bugs, errors and defects, the Log would not be disclosed (even if it were in the Post Office’s control, which it was stoutly maintained it was not). The submissions made by the Post Office on the adequacy of its own disclosure have been rather undermined, since the trial ended, by the production (emanating from Fujitsu, once again) of over 5,000 KELs, something rightly described by the Post Office’s solicitors as “a serious matter”.
The claimants have not been entirely co-operative on their own disclosure, however. I consider that the criticisms of the claimants for serving what was described as “claimant specific” evidence are unjustified. At least one of the experts, Mr Coyne, adopted a specific approach and investigated specific issues or occurrences, and there is no reason that both of the experts could not have done so. Dr Worden did not, but
that does not mean that the claimants were wrong to do so. Some of the criticisms made by the Post Office regarding the claimants are unjustified, therefore, but not all of them are. Although I do harbour the suspicion that the Post Office may have decided that, on disclosure at least, attack may be the best form of defence, some of the criticisms made in the Post Office’s closing submissions are justified. For example, for the claimants’ solicitors simply to state that “at present, it is not the Claimants’ intention to make another round of disclosure in relation to the Horizon Issues trial” as was done in July 2018, is not co-operative. Further, the reason that documents which Mr Tank had in his box, that were plainly relevant, were not disclosed was because Mr Tank had not been asked for them. Had he been asked – which is something that the claimants’ solicitors should have done, once it was known he would be a witness – these could and would have been disclosed. They certainly should have been.
The following points are of general application to both parties in this litigation. At some point a party who seeks to complain to the court about disclosure ought either to shelve its general disquiet on this subject, or make an application.
It is important to identify, so that the parties in this group litigation are aware, the court’s approach to disclosure. This is given for the purposes of guidance, and costeffective and efficient progress of the group litigation going forwards.
Care must be taken when explanations or submissions are made to the court. If further time is needed in order for accuracy to be achieved, that time should be sought. Those giving instructions to counsel must be astute to ensure that they are giving correct information.
All the substantive issues will be resolved on the evidence, the applicable law, and taking account of submissions. A party does not obtain a preliminary advantage or “head start” on the substantive issues because of any criticisms about disclosure.
Criticisms about the other party’s disclosure should be proportionate. Depending upon the circumstances and the nature of the document, the fact that disclosure is given late might be relevant, but simply because continuing disclosure is given by one party does not mean that there is anything sinister lurking in the background.
The court can always be asked to resolve disputes about documents. Mr Coyne sought a large number of different types of documents, but no applications were made by the claimants even though these were not provided. Criticisms of the Post Office’s disclosure by the claimants has to be seen in the light that the claimants did not consider it sufficiently important at the time to make any applications to the court. This is not to encourage applications. Sensible and well-advised parties who cooperate will usually deal with almost all disclosure issues consensually.
The parties are expected to comply with their disclosure obligations. If a document comes to light that should have been disclosed but has not been, it should be provided promptly. If this is discovered actually during a trial, it must be disclosed very rapidly indeed. As happened with the document referred to at [618] above, a document concerning the Drop and Go bug was discovered actually during the Horizon Issues trial, then (as Mr Parsons put it in his explanatory witness statement) “for the next couple of months” kept by the Post Office’s solicitors while “research” was done. It was then disclosed just before resumption of the trial “in a batch”. This is not working to the time scale either required, or expected, by the court, and it is not the correct way to deal with disclosure during a trial. This is a laissez-faire approach to disclosure of an important document during the trial itself, and the document should have been disclosed very promptly.
Both parties will be held to the same standards. The Post Office has, so far, and given the issues that have been tried, borne the bulk of the disclosure burden. The claimants are expected to observe their disclosure obligations in the same way, and will be expected to do so in the future when further issues and the claimants’ different cases are tried in 2020.
Witness statements will be ordered from a party when the court requires an explanation. Nothing else should be read into the ordering of a witness statement, other than it means precisely that -- an explanation is required.
It is obvious that the Post Office has had to rely upon Fujitsu to a large degree. However, given it was Fujitsu who told the Post Office what the Known Error Log contained – see further [586] above – Fujitsu has, so far, shown itself not to be entirely reliable in this respect. Fujitsu are also responsible for the Post Office making a directly incorrect important statement in its EDQ about retention of KELs, which led to the disclosure of about 5,000 of these some months after the trial closed.
Adoption of the Model C procedure does not mean that parties can keep adverse documents to themselves and not disclose them until a specific request is made. Such adverse documents should be disclosed promptly.
Finally, disclosure is very expensive. The court will be astute to guard against it becoming either satellite litigation or a weapon in the interlocutory arsenal.
I have exhorted the parties to co-operation many times before in this group litigation. I will continue to do so. Eventually, it is to be hoped that they will emerge into the broad sunlit uplands of the over-riding objective on disclosure matters.
G. The Experts’ Agreements
There were four experts’ agreements in this case. Experts’ agreements are very important. Permission to rely upon IT expert evidence was granted in the First CMC Order dated 25 October 2017. The two experts met on a number of occasions and agreed certain points.
As will be seen from Part H of this judgment, the experts approached their respective exercises in very different ways. However, they did reach some areas of agreement that were very helpful. Their joint statements or experts’ agreements did however also set out a considerable number of areas of disagreement. As the trial approached, I requested through the parties that the joint statements of the experts focus more on areas of actual agreement, rather than recitations of position and disagreements.
The 1st Joint Statement was dated 4 September 2018 and set out the following, inter alia, at the very beginning of the statement:
“Each expert's approach to writing his report, and to this joint memorandum which foreshadows their reports, could broadly be one of three possible approaches:
To focus mainly on negative points found in the disclosed documents about where Horizon may have fallen short.
To focus mainly on those aspects of Horizon which were intended to achieve robustness and reliability, and the evidence implying that they succeeded.
To provide the court with a clear foundation for understanding the design and operation of Horizon; then, building on that foundation, to provide a balanced assessment of the ways in which Horizon succeeded, whilst addressing any disclosed issues where Horizon may have fallen short.”
This was the first area of disagreement. Mr Coyne stated in the section immediately following, entitled Areas of Disagreement:
“In my opinion the technical issues can be answered without reference to the Claimants high level allegations document. The issues are about how Horizon and its interactive components operated and the processes employed by Post Office and Fujitsu in supporting these systems and the data within.
Whilst my report will take a balanced approach, it is the case that many of the issues require a deep focus on the occurrences of bugs, errors and defects as well as the potential for modification of transactional data. Whilst context will be provided as to how Horizon should work in typical circumstances, it is the non-typical operation where focus will be placed.”
This approach was not agreed by Dr Worden. He stated:
“I intend to take the balanced approach (c).
To provide a joint statement at this stage which will be useful to the court, under each Horizon issue the experts need first to agree what is worth agreeing having regard to the case which is being alleged. For instance, it would not help the court for the experts to list a number of detailed points as agreed, if one or the other expert thinks that those points are irrelevant or unimportant to the case alleged; or if collectively the points imply a view which, in the opinion of either expert, is not a balanced view. Therefore, the areas of agreement may at this stage be limited and more high level.
My approach will also focus on the Horizon system. It will consider Horizon within its wider business context, but I do not believe that the Horizon issues extend to offering any opinion on the wider business practices of Post Office.”
I consider that the balanced approach, which both experts stated they would be adopting, was the correct one. However, I do not find the following statement by Dr Worden in this 1st Joint Statement as helpful:
“under each Horizon issue the experts need first to agree what is worth agreeing having regard to the case which is being alleged. For instance, it would not help the court for the experts to list a number of detailed points as agreed, if one or the other expert thinks that those points are irrelevant or unimportant to the case alleged.”
It is a matter for the court to consider how helpful a particular point of detail, agreed by the experts, is or is not to the issues to be decided. Obviously the parties can make submissions about that, but it is not for an expert to decide that a point of detail, upon which he may be agreed with his opposite number, should however not be agreed because that point will not be helpful to the court; or that it should not be included in a joint statement. It is also not for experts to direct the court in any particular direction. The Horizon Issues were carefully drafted by the parties, approved by the court, and specifically included in an Order. They were the issues that the experts were to agree, and detailed points of an IT nature that arose within those Horizon Issues should, where possible, have been agreed.
Because Mr Coyne and Dr Worden approached their tasks so differently, the former inspecting a number of PEAKs and KELs about 10 times greater in number than the latter, this meant that a great amount of very detailed matters shown within those documents were not agreed by Dr Worden. He approached his expert exercise from what he described as “a top down” approach, a methodology that found its most stark manifestation in his statistical analysis of likelihood in his Section 8.
I will include within this judgment, not the full text of all four of the different Joint Statements, but some lengthy passages. This is to identify the large degree of agreement between the experts. I will however omit (with some isolated exceptions) the passages where the experts set out their separate, not-agreed, views.
The 1 st Joint Statement
I also quote at this stage from the 1st Joint Statement, signed on 4 September 2018, and the points of agreement and disagreement in respect of Horizon Issue 1. This is because in my judgment it is an important point of agreement, in respect of an important Horizon Issue, but also it is useful to set out the approach within the Joint Statements.
“2. Issues and Position
2.1 Issue 1
Issue 1 – To what extent was it possible or likely for bugs, errors or defects of the nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to 56 of the Generic Defence to have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions as alleged at §24.1 GPOC?
Areas of Agreement:
Evidence exists that bugs/errors/defects have caused actual discrepancies or shortfalls relating to Subpostmasters’ branch accounts/transactions.
Each time any IT system (including Horizon) is changed there is the potential to introduce new bugs/errors/defects.
Once bugs/errors/defects are discovered, they take time to resolve and therefore systems such as Horizon often continue to operate with bugs/errors/defects with or without workarounds in place.
Theoretically, bugs/errors/defects that existed within Horizon have the potential to cause apparent or alleged discrepancies or shortfalls in relating to Subpostmasters’ branch accounts/transactions.
In the event that any specific bug/error/defect had such an effect, the experts have differing views as to the ‘extent’ of the impact that such bugs/errors/defects may have had on branch accounts.”
(Emphasis added)
The statement continues:
“Areas of Disagreement:
Jason Coyne: As identified by select Known Error Logs (KELs), many thousands of bugs/errors and defects occurred and many of these known errors had impacts that recurred in different circumstances, therefore the reliability of Horizon to accurately process and record transactions is questionable.
Such facts undermine the reliability of Horizon to accurately process and record transactions.
Examples:
the Calendar Square/Falkirk bug – which led to Horizon failing to recognise transfers between different stock units, thereby affecting Branch accounts;
the Payments Mismatch defect – which affected at least 62 branches and was related to the process of moving discrepancies into the local suspense account, thereby affecting Branch accounts. This defect was not capable of being identified by Subpostmasters, who would have believed from Horizon that their account was balanced when it was not; and
the Suspense Account bug – which erroneously replicated suspense account items for 14 branches from 2010 in the same monthly trading periods in 2011 and 2012, thereby affecting Branch accounts.
In my opinion there is possibility of bugs/errors/defects existing in Horizon that have not yet been discovered. Further, most bugs/errors/defects that exist in Horizon have existed for extended periods of time before they were discovered.
I understand that Horizon had at least 19,842 ‘releases’ of new software, each one of these could have introduced new bugs/errors/defects.
Dr Worden:
My current preliminary view is that undetected bugs/errors/defects in Horizon were very unlikely to be the cause of permanent shortfalls or discrepancies in a branch's accounts.
It is necessary to define measures for the 'extent' of issue 1. At least two measures are possible: the number of such bugs and errors, and their net expected financial impact on claimants' branch accounts, compared to the total shortfall experienced by all claimants, which is of the order of £18 million. I intend to assess both measures. The latter measure may be more useful, because if any set of bugs has expected financial impact much less than £18M, it can do little to account for the claimants' shortfalls, either collectively or individually.
I propose to assess these measures from a number of different sources of information:
(a) the defects of Horizon cited by the claimants in para 1.3 of the outline; (b) the KELs identified by the claimants in para 1.4 of the outline; (c) other KELs; (d) data on their shortfalls provided by the claimants in their schedules of information; and (e) data from other sources (if available).
I am still scoping the data needed for (e). If it is available, I will provide it to Mr Coyne as soon as I have it.
My preliminary analyses of the data on (a) - (d) imply that:
For the majority of KELs, (which record known bugs), it can be shown, from an understanding of the robustness measures built into Horizon or otherwise, that they would have no permanent impact on branch accounts.
The net financial impact on claimants' branch accounts of any other bugs, including undiscovered bugs, is very small compared to the claimants' total shortfalls. I am not yet able to quantify this, but I intend to do so.”
In my judgment, this presages Dr Worden’s statistical analysis of likelihood, which I deal with as part of Section I. The following text used by Dr Worden should be considered:
“….if any set of bugs has expected financial impact much less than £18M, it can do little to account for the claimants' shortfalls, either collectively or individually” The phrase “much less than £18M” could, potentially, direct expert consideration of Horizon Issue 1 away from what it is in fact asking, towards a different issue entirely. The experts were not tasked only with investigating bugs, errors and defects that had a proven financial impact of (say) £18M, £10M, or any financial figure at all, and certainly not one in the millions of pounds. This was for the following reasons. It was never part of the Post Office’s position at the case management stage when the Horizon Issues were discussed and considered, that there should be any sort of threshold for financial impact, above which the experts should investigate, and below which they should not. The wording of Horizon Issue 1 has already been addressed above, and this uses the terms “possible or likely”. This notion of imposing a financial threshold must have come from Dr Worden.
Nor is the litigation intended to be a wide-ranging enquiry into all of the Post Office’s accounts with all of its SPMs over the 15 or 16 year period in question. The Post Office’s position is that Horizon is sufficiently robust such that it cannot be the cause of the different shortfalls complained of by the hundreds of different claimants. The counter-position to that by the claimants is not that particular discrete sets of bugs had, individually, an impact of millions of pounds upon their branch accounts. If that were the claimants’ case, then Dr Worden’s approach as set out in this joint statement might be understandable. Rather, the claimants’ case is that there are a large number of different bugs, errors and defects that each affected each of their branch accounts over time, with the end result being that each of them was (in accounting terms) identified as owing the Post Office sums in the tens of thousands, or thousands of pounds (or less) with the consequences which are well known. Some discrepancies were only a few pounds, but are said to have occurred many times. Some were said to be tens of thousands of pounds cumulatively. Some occurred many times; others less often, but with greater individual effect. One bug – of which more later – is said to “double up”, which would mean an exponential effect as a loss doubles, then quadruples and so on. If one imposes, at the beginning of the expert investigation process, a filter or threshold that looks for a “set of bugs” causing losses of “millions of pounds” that will have the effect of restricting the field of investigation. There is no reason for such a restriction, in my judgment.
On Horizon Issue 2, the experts were agreed, so far as that issue was concerned, that the extent to which any IT system can automatically alert its users to bugs within the system itself is necessarily limited, and while Horizon has automated checks which would detect certain bugs, there are types of bugs which would not be detected by such checks. The experts were also agreed, in respect of Horizon Issue 6, that whilst Horizon contains measures and controls for detecting system integrity concerns, the automatic mechanisms have failed in the past. The experts did not agree as to the ‘extent’ of prevention, detection, identification, reporting or risk reduction that the automatic and manual control measures delivered; however, the agreement that automatic mechanisms have failed in the past, which was contained in the 1st Joint Statement, is in my judgment important. It means that the Post Office cannot rely upon the presence of automatic mechanisms or processes as being an answer to the consequences of failures in system integrity.
It was also agreed by the experts that the causes of some types of apparent or alleged discrepancies and shortfalls may be identified from reports or transaction data available to the SPMs, but that other causes of apparent or alleged discrepancies and shortfalls may be more difficult or impossible to identify from reports or transaction data available to the SPMs in question, because of their limited knowledge of the complex back-end systems. Identification required the cooperation of PO staff and the SPMs. Again, I consider this expert agreement to be important. This was, essentially, the essence of the complaints raised by Mr Bates himself in the very earliest days of Legacy Horizon, namely that there was or were insufficient information or reports available to him to enable him to identify the causes of apparent or alleged discrepancies. The experts are agreed that, for some causes of apparent or alleged discrepancies this is indeed the case, due to the complexity of the back-end systems (back-end means not at the branch) and the limited knowledge a SPM would have.
The experts agreed that Horizon had evolved, and its robustness may have varied throughout its lifetime. They also agreed that the level of robustness may have increased or decreased as the system was changed. The existence of branch shortfalls was agreed, but they were not agreed at the stage of the first joint statement that this indicated any lack of robustness. They were also agreed, as at the date of their 3rd Joint Statement, that Horizon as it is in 2019 is relatively robust. However, given the span of the years under consideration is 2000 to 2010 for Legacy Horizon, and 2010 to date for Horizon Online, and given the agreement to which I have just referred that robustness may have increased or decreased, this only takes one so far.
On Horizon Issue 4, the experts agreed that there are a number of actual reported errors in data recorded within Horizon arising from (a) data entry, (b) transfer or (c) processing of data. Therefore, the potential exists. The experts do not agree as to “its “extent”.” Dr Worden saw this issue as a subset of Horizon Issue 3. He felt that because of the measures “built into Horizon”, errors of data entry, transfer and processing were very unlikely to affect branch accounts. The experts agreed on Horizon Issue 6 that the automatic measures and controls for detecting system integrity within Horizon had failed in the past; they could not agree as to the extent
that the automatic and manual control measures delivered. Dr Worden also made the point that bugs, when discovered, could not be fixed instantly and what he called “trade offs” were required when dealing with practical release management.
So far as remote access and Horizon Issue 7 was concerned, at the time of the 1st Joint Statement Dr Worden was unclear about this and the technical document provided to him by Mr Coyne had not been received in sufficiently timely manner such that he could consider it. By the time the Horizon Issues trial had started, the Fujitsu witnesses such as Mr Parker had corrected their earlier incorrect witness statements and accepted that this could be done. However, those statements were not served until November 2018 so would not have been in existence as at the time of the 1st Joint Statement.
On Horizon Issue 9, the experts agreed that the causes of some types of apparent or alleged discrepancies and shortfalls may be identified from reports or transaction data available to SPMs. Other causes of apparent or alleged discrepancies and shortfalls may be more difficult or impossible to identify from reports or transaction data available to SPMs, because of their limited knowledge of the complex back-end systems. Identification required the cooperation of PO staff and SPMs.
On Horizon Issue 10, it was agreed that by the very nature of rolling out fixes in any IT system, including those implemented by Fujitsu, these had the potential to affect transaction data or data in branch accounts. This could be seen as stating the obvious, but at least the experts agreed it. The experts were also agreed that the use of tools and facilities to do this should be auditable, however the maintenance of logs recording this would be dependent upon retention periods and the size of the logs in question. Again, this could be seen as stating the obvious. The experts were not agreed on Issue 12, how many times the ability to do this was used, if at all. Nor was there any agreement on Issue 13, the extent to which such facilities had the potential to affect the reliability of branch accounts; Issue 14, a detailed issue with 7 sub-issues concerning functionality; or Issue 15, which concerned Transaction Corrections or TCs.
The experts agreed, so far as the existence and use of permission controls is concerned, which is the essence of Horizon Issue 11, that the usage of tools and platforms which existed specifically for the purpose of accessing and modifying transaction data, should be auditable. It was agreed however that the maintenance of logs would be dependent upon retention periods and size. Such tools which were identified initially in the 1st Joint Statement under Horizon Issue 10 areas of disagreement by Mr Coyne, were at that stage split into the following:
Global Branches which would enable the input of transactions within Horizon as though it had come from an actual Branch;
the Branch Transaction Correctional tool;
the Transaction Information Processing repair tool.
The best description of “Global Branch” or “Global User” is contained in a Post Office document headed “Global User Process Management” which said that it allowed people such as auditors “to be able to go out to any branch and use the same log on to access any Horizon kit in Branch without having to request a single use password at each branch. The individual requiring a Global Use account has to be approved by a Manager showing on the Global User requester and approver list”. However, the document also identified that PWC had discovered that “the current Global User list inherited from Fujitsu contained employees who had left the Business. This raised concerns over the security of access to Horizon. Although this is a low risk in terms of Horizon access (the individual would have to gain access into a Branch first) it is still a risk and needed to be mitigated.” There were 32 such former employees. It was in respect of the permission control procedures concerning these tools that the documented authorisation process was identified, using documents called the “Master Service Change” or MSC documents and “Operational Control Procedures” or OCPs. These documents were requested by Mr Coyne but no actual disclosure application was issued. The document to which I have referred explaining Global User was in the trial bundle.
The 2 nd Joint Statement
The 2nd Joint Statement is dated 25 February 2019 and identified and collated the different bugs, most of which were given actual names, which had either been identified and agreed, or which Mr Coyne maintained were bugs. This statement was in landscape form and lengthy, and included what became to be referred to as “the bug table”. I adopt the terms used by the experts for the descriptions of the different bugs, or alleged bugs. The number of different bugs that were agreed by Dr Worden, by the time the trial ended, was 11. The description of this in the joint statement itself was
“The structure of the document captures i) A table of bugs/errors/defects containing evidence of financial impact upon branch accounts that both experts agree (or indicate if they do not), ii) all expert agreements grouped by Horizon issue, and iii) additional comments and observations input by the respective expert.”
Prior to the commencement of this litigation, the Post Office had acknowledged the existence of only two bugs in Horizon. Mr Coyne’s evidence was that there were 29 relevant to Horizon Issues 1 and 4, and he clarified in his cross-examination that the number that he considered had lasting impact upon branch accounts was 21. I deal with that cross-examination at [794] to [798] below.
The 2nd Joint Statement also said in its Introduction that:
“Because of time pressures and the complexity of the issues, we have not been able to address all the Horizon issues in this joint statement. We will issue a subsequent joint statement addressing those issues we have not yet addressed. For those issues we have addressed in this statement (issues 1, 2, 9, 14 and 15), the layout and references are not as polished as we would have wished.”
It is extremely useful for the court to have Joint Statements from experts, and I am grateful to the experts for producing these statements, in particular what was called “the bug table” in the 2nd Joint Statement, in what is a very complex case, and when they were under considerable time pressure. The bug table was very widely used in both cross-examination by the parties and indeed in their submissions.
The so-called “Bug Table”
The bug table, and my findings upon the bugs within it, are in my judgment central to the Horizon Issues. Evidence concerning bugs in the Bug Table formed a large part of the expert evidence that was tested in cross-examination. The different bugs, or alleged bugs, identified in the Bug Table were as follows. I have added to the following the summary of the position of the Post Office in Appendix 2 of its Closing Submissions, which in some cases accepted there was such a bug, and qualified that acceptance in most instances by references to what was said to be the impact of the bug. I deal with detailed findings on those bugs that are in issue in the Technical Appendix to this judgment. For the detailed submissions by the Post Office for each bug, the paragraph numbering in Appendix 2 started again at 1 for each bug. This did not cause particular difficulty, and because the Post Office explained that Appendix 2 had been put together by different parts having been drafted by different counsel/solicitor teams, this is understandable. I refer to it here, because otherwise a reader of the passages below might become confused as to why there seem to be so many paragraphs all (for example) numbered 2 in Appendix 2. The following summary of the Bug Table is intended to be a useful summary for this judgment, and is not restricted solely to the entries within the 2nd Joint Statement.
Receipts and Payments Mis-match bug.
This was agreed in the bug table as an “acknowledged bug” which had an impact on branch accounts. Dr Worden added in his comments that “Therefore, the extent of this bug is well established, in the GJ analysis.” GJ is Gareth Jenkins. The identified year of its effect was 2010. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. In the summary in Appendix 2, the Post Office stated that “In the event, however, this bug resulted in transient impact only.”
Callendar Square/Falkirk bug.
This too was agreed in the bug table as an “acknowledged bug” which had an impact on branch accounts. Its identified years of effect were 2000 – 2006. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had “transient impact”. Dr Worden added different comments, including that “the bug arose from a fault in the underlying Riposte software, so it is not surprising that it took Fujitsu some time to understand it, or that they had to rely on the suppliers to fix it. It does not show poor system design or support by Fujitsu”. That latter sentence is a little surprising, and seemingly very defensive on Fujitsu’s behalf. This is because the Horizon Issues are all about the operability and functionality of Horizon, not who is to blame for the presence of any bugs, errors or defects. Riposte software was part of Legacy Horizon. The fact that it is a product designed and supplied by Escher, and not Fujitsu, is not relevant to the dispute between the Post Office and the claimants. In any event, and as accepted and recited in paragraph 2 of the Summary of Appendix 2 dealing with this bug, “Mr Coyne asserts that Bug 2: Callendar Square is a bug with lasting financial impact and in JS2, Dr Worden appears to agree that there is strong evidence of this….”
Suspense Account bug.
This was agreed in the bug table as an “acknowledged bug” and its identified years are 2010 to 2013. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. Dr Worden’s comments do not include that it had an agreed impact upon branch accounts. This is because of a concept he introduced (expanded in his reports) of “transient effect”. What this meant was that if there was an impact on branch accounts by something within Horizon, but that was then corrected by a TC, he concluded it had a “transient effect” upon branch accounts and dealt with it differently. I address this concept below. Other comments of his in relation to this acknowledged bug are:
“It was a transient effect arising not from a fault in the software, but from a change in database archiving policy in 2010. The delay in correcting it arose from a failure of communication between PO and Fujitsu. Because the bug would only manifest itself annually for any affected branch, the effects of this delay were not widespread.
Peak PC0223870 shows that Fujitsu were able to identify the branches affected, even when Subpostmasters did not report it. There is evidence that the branches were compensated, as I would expect from the normal error correction processes.”
Dalmellington bug/Branch Outreach Issue.
This was not acknowledged as a bug in the bug table, but is effectively accepted by the Post Office as a bug. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had “transient impact”. Indeed, for some time during the trial the Post Office would refer to this as “an issue”. I find that it is a bug – that much does not seem to be controversial, as paragraph 2 of the detailed part of Appendix 2 relating to this bug states the following:
“This is a bug which Mr Coyne states has lasting impact on branch accounts. Post
Office submits that there was no lasting impact on branch accounts.”
The years of its effect were 2010 to 2015. It is named in respect of the branch where it was “discovered” in 2015, although the investigation undertaken by Fujitsu in that year identified previous occurrences. There were over 110 occurrences of it over this 5 year period (the actual number is probably 116, but at least 114) with 88 branches affected. Some were impacted multiple times. It is what is called a “cash remming error” and only happened in certain circumstances. It was explained further by Mr Godeseth in his evidence at [445] onwards above. It would, in my judgment, have an impact upon branch accounts – it is the extent of that impact that is in issue, whether it was lasting or transient. Dr Worden relied upon what he described as “a well-tested process of reconciliation and TCs to detect and correct errors in cash remming (used 20,000 times per year)” and said it was “straightforward for Horizon to detect any discrepancy between a “rem out” and the corresponding “rem in” (a mismatch arising either from a miscount, or a multiple count of a pouch) and then a TC can be issued.” He also added that “this process catches and corrects remming errors, whatever their cause - including if they arise from, or are provoked by, software faults.” He therefore implicitly accepted that the Dalmellington bug was a software fault, although he did not say so in terms. I find that it was plainly was. He was also reliant upon the process of TCs to correct it. Further, the bug was there for 5 years and was not discovered, although its effects doubtless were. I find that this was a software bug that impacted upon branch accounts and I will come to the nature of its effect in my findings on the bugs that are not agreed.
Remming In bug.
This was not acknowledged as a bug in the bug table. However, Dr Worden accepted it was a bug or defect in his cross examination and it is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is again said to have had transient impact. The years it was said to have been present in terms of effect were “March – August 2010 and recorded as fixed approx. 2011”. Mr Coyne said there were 14 branches affected. Dr Worden’s entry in the Joint Statement relied upon TCs, and stated “As for the Dalmellington bug, above – PO had a robust process for detecting and correcting remming errors, whatever their origin. So, there were no lasting effects on branch accounts.” In paragraph 2 of the detailed part of Appendix 2 relating to this bug, the Post Office submitted that “any discrepancy would be transient as instances of this bug are caught by automatic reporting.”
(emphasis added)
The answer to this is therefore dependent upon my answer to the point concerning lasting and transient impact.
Remming Out bug.
This was split into two in the bug table, 6(i) which was identified in KEL acha508S and 6(ii) which was identified in KEL GMaxwell3853P. The former was identified in February/April 2007, recorded as fixed approx. 2007, with Mr Coyne identified 57 branches affected; and the latter was in May 2005 with one branch being affected. They were both remming out issues, hence the grouping by the experts in the bug table. It was not acknowledged as a bug in the bug table. However, it is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Again, for both of these iterations, 6(i) and 6(ii), Dr Worden used the same wording as he had in the entry for bug 5, the Remming In bug, namely “As for the Dalmellington bug, above – PO had a robust process for detecting and correcting remming errors, whatever their origin. So, there were no lasting effects on branch accounts” (emphasis added). In paragraph 2 of the detailed part of Appendix 2 dealing with this bug, the Post Office stated that this “comprises two separate issues, only one of which was a bug. Any discrepancy caused by either issue would be transient as instances of both issues were caught by automatic reporting.” It was also said that Mr Coyne had conflated two unrelated issues under the heading “Remming Out”. Given this was an entry in the 2nd Joint Statement agreed by the experts, it is a little unfair to state that Mr Coyne had done this, as the entry was obviously agreed by Dr Worden. However, it was split into 6(i) and 6(ii) and I will consider each separately.
Local Suspense Account issue, not the same as 3. Suspense Account bug. This was reported in 2010 and recorded as fixed in September 2010. It was not acknowledged in the bug table, but is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, although it is one of those bugs said to have had transient impact. Mr Parker had identified 33 branches affected. Mr Coyne recorded what he said were four associated KELs, namely acha5259Q (for which there were 6 PEAKs)(this KEL is mistakenly recorded twice in column 3, with acha5838T only mentioned in the text in that column); cardc2043L (10 PEAKs); PorterS199P (3 PEAKs); and acha5838T (which states there are “two different but similar problems” and appears in the text, but not in the list of KELs at the end of the text by Mr Coyne). Dr Worden’s comments were that
“The KEL acha5259Q implies that PO and Fujitsu were able to identify all occurrences of the problem, without being notified by any Subpostmaster. I would therefore expect them to have corrected any impact on branch accounts as part of normal error correction processes.
I would not expect evidence of all corrections to accounts to have survived to the present day. PEAKs and KELs are not used to record corrections of financial impact.”
He also relied upon a statement by Mr Parker that there was ‘Temporary financial impact which would have been cancelled out in the following period by a corresponding discrepancy’.
This introduces another concept, similar if not identical to Dr Worden’s “transient impact”, and that is “temporary financial impact”. Dr Worden is correct that PEAKs and KELs do not record corrections of financial impact; they do however record financial impact, because when an SPM reaches SSC (which raises the PEAK) they often start by recording “SPM says he/she has a problem in that……” and financial impact is often then recorded. The Post Office in paragraph 2 of the detailed part of Appendix 2 dealing with this bug states that “any discrepancy would not be lasting”. This does therefore accept, albeit implicitly, that it would have an impact upon branch accounts. It is also submitted that it was a “teething problem” from the early days of the Horizon Online pilot scheme.
8. Recovery Issues.
The Post Office does not accept these are bugs at all. These are not agreed as bugs by the experts, and there are four different types included in the table, with years of effect from 2010 to 2018. Mr Coyne’s entry stated that “The text within the PEAKs and KELs suggests that in each case a branch account discrepancy would be evident and would require correction by the Post Office.” Dr Worden stated that:
“The KELs and PEAKs cited by Mr Coyne are not indicative of errors in Horizon. They provide guidance on how to correct discrepancies caused by human errors or other errors in transaction recovery ('recoverable transactions')
Because there were many such errors, there were many calls to the help desk and many PEAKs and KELs. Normally, correction of errors involved back office reconciliation and issuing TCs. This was accurate and effective; I have derived an upper limit of £2 per branch per month on the mean impact of erroneous TCs.
One important KEL acha959T was guidance to the back office MSU, not for Subpostmasters”
There was a minor typographic error in the submissions in paragraph 4 of the detailed part of Appendix 2 which referred to bug 9. However, the four different types of recovery issues are addressed in the subsequent paragraphs and the conclusion paragraphs deal with bug 8. This was later corrected in a sheet of corrections, which clarified that paragraph 4 of the detailed part of Appendix 2 should have stated "Mr Coyne states that Bug 8: Recovery Issues is a bug with lasting financial impact. Post Office submits that it is not a bug at all". The relevant findings on these issues are dealt with in the Technical Appendix.
Reversals.
This was not acknowledged as a bug in the bug table but is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Indeed, Dr Worden’s entry in the 2nd Joint Statement in relation to this stated “Transaction reversals are a complex area which, like recoverable transactions, are less familiar to Subpostmasters and are more prone to human error. They lead to many calls to the help line and to many KELs and PEAKs - not necessarily related to any fault in Horizon.” This can be seen as yet another example of attributing fault to the SPMs where possible, but given it is now accepted as a bug it is not necessary to consider that any further. Mr Coyne’s entry in the 2nd Joint Statement stated “In April 2003 due to a failure in regression testing, Horizon version S30 was released by Fujitsu and this introduced a bug where the value of transactions reversed by Subpostmasters was shown twice in the amount of the reversal in branch accounts.”
Data Tree Build Failure discrepancies.
This is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. The PEAK dealing with this reads “Data trees have been failing to build fully, and the system has not been detecting this.” Data trees were part of Legacy Horizon, and is used to build a summary (or picture, the word used by the Post Office in its Appendix 2) of the accounts. The building of data trees is a software function, and given “the system” is Horizon and is supposed to detect failures of this nature, it is difficult to see how it could ever have been in issue that this was a bug. Mr Coyne’s entry in the bug table recites that “Dugannon branch suffered a £43,000 discrepancy but the cause was not immediately known. £52,814.29 at the Yate Sodbury Branch. £9,368.40 at the Appleby Westmoreland branch.” These are sizeable sums and arose in the branch accounts. That this is a bug was de facto accepted by Dr Worden in the bug table due to the text of his entry which states:
“There was a bug which has potential impact on branch accounts, early in the lifetime of Horizon. Soon after it arose, the error was trapped and detected by DEP and was then soon fixed.
The fault was easily noticeable at branches before the error trapping which was soon introduced and would be even more noticeable after that. Only three branches appear to have been affected, as described by Mr Coyne.
Because it was so noticeable at the branch, and the Peak is concerned with a software error rather than any other cause, I would expect any discrepancies in branch accounts to have been corrected.”
(emphasis added)
This is therefore undoubtedly a bug, but findings in respect of this therefore depend upon the nature of its impact on branch accounts, which are made in the Technical Appendix.
Girobank discrepancies.
Mr Coyne considered this was a bug. Dr Worden did not and stated that “the first fault concerns reports. A fault in a report is not a discrepancy in branch accounts, and only causes one if it causes a person to make a mistake.” This is now essentially accepted as a bug by the Post Office, but it is submitted it had no branch impact. It is included in paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact.” The detailed part of Appendix 2 dealing with this submits that there is no evidence of any financial impact upon branch accounts, let alone a lasting impact. This was in the early days of Legacy Horizon. There are said to be six distinct issues arising under this heading, and further detail on this is included within the Technical Appendix. I do not know if what are called “giros” are still in use in 2019.
Counter-replacement issues.
There were two KELs associated with this dealt within the bug table. It is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Mr Coyne considered that when replacing a counter within a branch, the process could result in “the total loss of a transaction”. Dr Worden stated that the cause, recorded in the first KEL (which was created in December 2000 and last updated in July 2007) was that Riposte was coming online from the Recovery mode too early, and causing messages to be overwritten. The nature of the correction was stated as being “to find the overwritten transactions for reconciliation we need to look at the Ripostemirror messagestore' followed by detailed instructions”. Riposte, as has been explained, was part of Legacy Horizon and was a product provided by another company, Escher, but it is plainly part of Horizon. Dr Worden also stated that “the incident arose from a hardware replacement (probably from a hardware fault) not from a fault in Horizon. It is a different kind of recovery issue.” In my judgment the hardware is part of Horizon. Findings in respect of this are made in the Technical Appendix.
Withdrawn stock discrepancies.
The Post Office does not accept these as a bug. Mr Coyne maintains that it is, and in the bug table he extracted part of a PEAK that stated these “Can cause confusion and unexpected (though hopefully temporary) discrepancies at branches by allowing them to declare stock which has already been withdrawn.” Dr Worden stated that “some impact on branch accounts cannot be ruled out, although it is small”. The Post Office detailed submissions in Appendix 2 of its Closing Submissions concentrated on the fact that withdrawn stock – which of course is part of the way that the Post Office manages its business, adding and removing types of products from time to time – this removal of products is done by means of an update to reference data, “and not a change to the core code in the system.” This again is concentrating on the code, rather than the way the system operates. Findings on this are made in the Technical Appendix.
Bureau discrepancies.
These relate to foreign exchange, hence the name. It must be differentiated from bug 23 (also foreign currency, but entitled Bureau de Change). This one arose in 2017. Mr Coyne considered it to be a bug, and Dr Worden effectively agreed in that his entry stated “This appears to be a system error with impact on branch accounts. Although it is possible that a subsequent discrepancy between branch accounting and POLSAP would reveal the problem, leading to a correction (e.g. see Peak PC0265443, and Mr Coyne's para 3.146), I cannot be certain of this.” It is now accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact
(although they were resolved)”. The detailed part of Appendix 2 states that Mr Coyne has drawn together two distinct issues. Paragraph 4 of the detailed part also states the following:
“Bug 14: Bureau Discrepancies is a bug with the potential for lasting financial impact. There are two distinct issues which fall under this heading. With regards to the first issue the branch was made good and a fix was implemented. The second issue was not a bug in Horizon nor an issue which could have impacted branch accounts; it created what was essentially a cash flow problem for the branch.”
Findings on this are made in the Technical Appendix, but so far as the first issue, given a software fix was required, it is undoubtedly a bug, and it is correct to record (as the Post Office do) that there was the potential for lasting financial impact, as without that, there would be no need for the “branch to be made good” because that means the Post Office corrected the branch account discrepancy caused by the bug.
Phantom Transactions.
There are three different issues grouped under this heading. The Post Office does not accept these as a bug. Mr Coyne in his cross-examination stated that this was referable to Horizon Issue 4. He also accepted that some of the issues were not bugs in Horizon. Dr Worden relied upon the fact that what he called “the master PEAK” was closed as “no fault in product”; however, the evidence of fact on this is dealt with above at [209] to [213] when the PEAK was put to Mrs Van Den Bogerd. Dr Worden stated that “There is no evidence for bugs in Horizon with impact on branch accounts.” That entry was obviously before the cross-examination, which in my judgment provided greater factual information. I do not consider reliance can be placed upon the Fujitsu conclusion in the PEAK. Findings on this are dealt with in the Technical Appendix.
Reconciliation issues.
The Post Office does not accept these as a bug. There are a number of different issues grouped together in this heading. The issue was that the SPM was shown a discrepancy on his or her screen. Mr Coyne accepted that the discrepancy would not be shown in the branch accounts; Dr Worden stated that “as it concerns an issue in reporting, the software fault (which was fixed after 5 months) had no direct impact on branch accounts. The only effect of an error in this report would be to mislead or confuse the Subpostmaster - probably leading him to check his figures more carefully and costing him some time.” That accepts a software fault; the fault was in fact a miscounting of the number of files by the system. However, there are six different issues grouped under this heading and the process of reconciliation, which is effectively the comparison by Horizon of two different sets of data, is part of its design function. Mr Coyne in his cross-examination also stated that this was referable to Horizon Issue 4, rather than Horizon Issue 1. The recovery messages were held in the branch messagestore. Findings on this are dealt with in the Technical Appendix.
Branch Customer discrepancies.
This was an entry in respect of Horizon Issue 4 and recorded as such in the bug table. The Post Office does not accept these as a bug. Mr Coyne accepted in his crossexamination that the entry in the PEAK did not suggest any impact upon branch accounts. The entry in the bug table originated in a single PEAK, although there were two PEAKs that relate to the same issue, which was a discrepancy between the financial records held by the Post Office (from Horizon) and a bank’s records, after a counter crashed whilst a transaction was being processed. Findings on this are dealt with in the Technical Appendix.
Concurrent logins.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. In paragraph 2 of the detailed part of Appendix 2 dealing with this bug, the following is stated by the Post Office: “Post Office accepts that Bug 18: Concurrent Logins had a potentially lasting financial impact. There is no evidence of any discrepancy in the PEAKs referred to by the experts.” The problem was in the early days of Legacy Horizon when it was possible for users to log in to two terminals at once. Dr Worden in the bug table made the following useful summary statement:
“discrepancies could occur - manifesting themselves as a receipts/payments mismatch. This had the potential to affect branch accounts. The mismatch would bring it to the attention of the Subpostmaster, who would require it to be investigated, except possibly in the case of small mismatches, which he might pass off as an error in the branch (e.g. of counting stock).
“….Fujitsu believed it was a problem with the underlying Riposte software, and passed it to Escher. In September 2000, the problem was 'Now formally fixed in Build 223 update 19 which was released overnight.' However, the new release from Escher did not, as it was expected to, fix the problem. Escher denied that is was a bug in Riposte, but Fujitsu believed in July 2001 that 'This is clearly a bug in the Supplier's code'.”
Dr Worden considered that “these faults had to the potential to produce discrepancies in branch accounts, of small amounts, for a short period of time”. How the bug is therefore characterised depends upon my findings in relation to lasting and transient impact, which is dealt with below. There is no doubt, in my judgment, that this was a bug, and Fujitsu clearly recognised and recorded that in 2001. The fact that the bug was in the code supplied by Escher, as part of Riposte, and not that of Fujitsu is wholly irrelevant for the purposes of the Horizon Issues.
Post & Go/TA discrepancies in POLSAP.
It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. The entry relates to Horizon Issue 4 rather than 1. “Post & Go” are self-service terminals, that are no longer available in branch Post Offices, and are only available in Crown and WH Smith main branches. They are basically selfservice kiosks, designed to avoid queuing, and a customer can weigh a letter or parcel, the terminal will print the relevant stamps/labels, and the item is then posted. The Post Office submitted that this issue was irrelevant to the Horizon Issues trial as ”this issue does not relate to branches that are the focus of this trial”. Findings on this are dealt with in the Technical Appendix.
Recovery Failures.
The entry in the bug table identifies this as relating to Horizon Issue 4. Post Office does not accept these as a bug and states that there is no evidence of a bug in Horizon. There are three different PEAKs. Mr Coyne accepted in his cross-examination that this (or these) should be removed from the table that he used as the originator for his list of bugs (which was Mr Coyne’s Table 1). Dr Worden stated in the bug table that “there was some implication of hardware faults, with a replacement of a base unit, but the PEAK has no evidence of software faults in Horizon.” In one of the PEAKs, he concluded that there was no evidence of any fault in Horizon. Findings on this are dealt with in the Technical Appendix.
Transaction Correction Issues.
This is de facto accepted as a bug by the Post Office, but it is submitted it had no branch impact. Mr Coyne’s entry was that “Transaction Correction bugs/errors and defects do not cause discrepancies with branch accounts but” and he then listed various consequences. In his report he referred to “technical flaws”. Dr Worden, in respect of some PEAKs, accepted they were a bug, such as PEAK PC0129587 where he stated “In my opinion this bug would result in an inconvenience to the Subpostmaster (inability to rollover to the next TP) but would not result in inaccurate processing of any TC, or any impact on branch accounts”. The bug is included in the list of paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact.”
These are submitted by the Post Office as there being “no evidence of any financial impact upon branch accounts”. Findings on this are dealt with in the Technical Appendix.
Bugs/errors/defects introduced by previously applied PEAK fixes.
This is de facto accepted as a bug by the Post Office, but it is submitted it had no branch impact. It is included in paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact”. It is also submitted by the Post Office that “a number of PEAKs arose in testing. The PEAKs that arose in the live environment do not indicate evidence of branch impact.” It is listed in the bug table as concerning Horizon Issue 4. Mr Coyne stated in the Joint Statement that “Branch accounts would be affected by this bug which would cause a discrepancy when handling cheques where the value of the cheque would be doubled” although he accepted that the SPM in question was processing a cheque in a different manner to that recommended. Dr Worden used the term “a fault” in his entries, maintained that there was no impact on branch accounts if the SPM followed correct procedures, and also in respect of another branch relied upon the fact that the effect was an error of 2p only. Findings on this are dealt with in the Technical Appendix.
Bureau de change.
These arose in 2005, 2006 and 2010, and should be differentiated from bug 14, which arose in 2017, was during Horizon Online and is referred to as Bureau Discrepancies. The Post Office does not accept this as a bug. The Post Office also states that Mr Coyne has identified three issues with the same heading, and submits that they all relate to user error. Dr Worden’s entry in the bug table in relation to one of them states “Analysing the second KEL (2010) I noted: ‘Impact small until bug fixed - rounding errors 10-5 in exchange rates.’ The Post Office submitting that this is not a bug is a somewhat bold submission, given the KEL to which Dr Worden refers expressly states that the problem is a different exchange rate appearing on the HNG-X rateboard and the Horizon rateboard. It states “Problem – there is a bug in the code” (emphasis added). Findings on this are dealt with in the Technical Appendix.
Wrong branch customer change displayed.
Mr Coyne concluded this was a bug and part of his entry in the 2nd Joint Statement was “the KEL explains that “the cash amount entered is multiplied by the Qty and hence the new stack total is wrong”, this Horizon bug was due to incorrect reference data and led to an incorrect amount of change being displayed on the branch screen leading to the operator to provide the branch customer with the wrong amount of money thereby leaving a discrepancy in Branch Accounts. It is possible that the amount of change shown on screen is more than the actual money tendered by the customer.” Dr Worden’s entry was “When analysing this KEL I noted ‘Sounds like a genuine problem which may have led to giving the customer the wrong amount - i.e. not recoverable.’ It is now accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. It is also said that it is a reference data bug, and that therefore once discovered could be quickly fixed by changing the relevant reference data. Findings on this are dealt with in the Technical Appendix.
Lyca top up.
It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. It is also a reference data bug, and paragraph 2 of the detailed part of the submissions on this in Appendix 2 states “Lyca Top Up is a bug with the potential for lasting financial impact. This is also [a] reference data bug. As set out above, the experts have agreed that that while reference data bugs may be a significant proportion of the bugs with financial impact, once discovered, they could be quickly fixed (by a change to the reference data) once the bug is correctly identified.” It is also submitted that it was identified through Fujitsu’s automatic reporting. Findings on this are dealt with in the Technical Appendix.
TPSC250 Report.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. The origin of the experts discovering this bug is a KEL raised by Anne Chambers in February 2005, and last updated in April 2008. Findings on this are dealt with in the Technical Appendix. I do however find it notable that although Dr Worden says the amounts involved are what he calls “small”, he accepts it is as a “back end reporting problem” – which is nothing to do with the SPM, by definition – and also the KEL states that “the accounting tree has not handled this properly when calculating the daily recon figures and it has resulted in a mismatch…” The mismatch in that case was 66p. The accounting tree is part of Horizon. The KEL also appears to be incomplete because only the first page is present. The section that follows the heading “Evidence” is entirely missing. There is also nothing in terms of the text available that demonstrates what occurred in 2008, even though it is plain for the page that is available that the KEL was updated then. The Post Office submits that there are five separate issues drawn together within this bug. Findings on this are dealt with in the Technical Appendix.
TPS.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. One of the Fujitsu documents referenced in the 2nd Joint Statement states that the Transaction Repair Tool or TRT is being used “to repair 1 harvester exceptions for” a particular branch and “There is no correction to be performed and hence no call for confirmtiprepair - this is just an oddity performed by that very flaky mails code.” (emphasis added) There were 40 associated PEAKs, and Mr Coyne observed that both the credit and debit sides of a transaction were doubled, so the net impact of the bug was zero, although he drew attention to the entry in two PEAKs that suggested that SSC requested confirmation of any gain or loss at the counter. Dr Worden believed it was a back-end reporting problem, although the chances of impact upon branch accounts were small. Findings on this are dealt with in the Technical Appendix.
Drop and Go.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. This occurred in July 2017 and related to a duplicate “Drop and Go” transaction for £100. This was performed twice; the branch was debited with £200, but the customer credited with only £100 (which had been the amount “topped up” by the customer). Mr Coyne considered it a bug with impact upon branch accounts; Dr Worden stated in his entry “My analysis of this KEL was ‘Possible financial impact. Seems very visible on the counter. Script = reference data - therefore fixed easily’. Paragraph 2 of the detailed part of Appendix 2 on this bug states “Peak PC0260269 relates to an issue involving a Drop and Go transaction (a £100 mobile phone top up) that timed out on Horizon” (emphasis added). This does not appear to be correct, and I do not believe that the transaction relates to a mobile phone. However, I deal with this further in the findings in the Technical Appendix.
Network Banking Bug.
This related to a KEL which was raised in 2004 and updated in 2005, with 12 associated PEAKs in the range 2004 to 2010. The Post Office does not accept this as a bug. Mr Coyne stated that “Horizon appears to mis-handle communications, leading to errors within network banking and in turn causing the potential for branch account discrepancies.” One of these was pension transactions being declined yet the customer’s bank account being debited; when the customer complained, the SPM refunded the £50 in question. Dr Worden considered it was “mainly about a communication problem from BT, outside Horizon” but also wanted to investigate further due to an entry that referred to a “CNIM own goal”. The Post Office maintained there were two separate issues and the £50 was paid out due to “user error”. Findings on this are dealt with in the Technical Appendix.
The names given to the different bugs above came, sometimes, from the names applied to the bugs by the Fujitsu personnel who knew about these bugs and worked upon them over the years. These names would sometimes appear in the PEAKs themselves that identified the bugs or which represented the views reached by different personnel within Fujitsu who investigated the background, sometimes identifying that previous incidents had similar, or sometimes even identical, characteristics to earlier occurrences of the bug, and who worked to provide fixes or solutions to some of them, or came to conclusions that there was nothing wrong with Horizon and attributed blame, fault or responsibility to the SPM. Some of the names are self-explanatory. Some, such as Callendar Square or Dalmellington, are the names of places where that bug was first identified by Fujitsu.
Certain terms were used, dependent upon whether a particular bug was agreed as being a bug (such as the Callendar Square bug), or something else, such as “issues” or “discrepancies.” However, as at the end of the trial, it can be seen from the above entries that the experts’ agreements, the clarification provided by each during crossexamination, and the other developments in the trial such as (one assumes) some reflection by the Post Office on the actual text of PEAKs and KELs, lead to the following situation. Of the 29 different entries in the bug table, eight were alleged by the Post Office not to be bugs at all. They are those numbered 8, 13, 15, 16, 17, 20, 23 and 29. This is set out in paragraph 4 on the first page of Appendix 2 to the Post Office’s Closing Submissions. This careful terminology therefore became a little clearer. The remaining number can properly therefore be described as bugs, even on the Post Office’s own case, although their effect was challenged.
This means that the remainder of what is, in my judgment, a lengthy list, were accepted to be bugs. In my judgment, that number of accepted bugs, even on the Post Office’s own case about their effect, and the non-acceptance of the eight others, is a sizeable number. It is approximately a ten-fold increase in the number admitted by the Post Office prior to the Horizon Issues. It is correct that the Post Office did not accept lasting impact on a number – a point I deal with below under “Lasting and transient impact” – and maintained that only 9 “had the potential to cause lasting impact, but were resolved by the Post Office and Fujitsu” but, in my judgment, that is rather to miss the point. The wording of Horizon Issue 1, agreed by the parties, and (one assumes) a major issue that both parties wished to have resolved in order sensibly to progress this group litigation (and omitting the references to the pleadings) is:
“To what extent was it possible or likely for bugs, errors or defects of the nature alleged… to have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions.”
The use of the words “possible” and “potential” in this issue makes it clear, in my judgment, that the number of “bugs, errors or defects” that fall to be considered is, even on the Post Office’s own submissions, 21 different bugs. I find that to be a significant total.
Part only of Appendix 2 to the Post Office’s Closing Submissions dealing with bug 16, Reconciliation Issues, was, by administrative error, not submitted by the date for closing submissions, or until some time after oral submissions were delivered. The Post Office’s solicitors realised this in September 2019. There were three missing pages. I am satisfied that these were drafted prior to the date for service, and I allowed these to be submitted late, together with a clarification document, and short responsive written submissions by the claimants (as the claimants had been deprived of the opportunity to address these three missing pages orally in July). I consider that I am permitted to do this under the court’s general case management powers, and that to do this was to adopt a reasonable and proportionate course. The Post Office’s Closing Submissions was a very substantial document, and it was in my judgment sensible to admit the 3 pages only that had been omitted from one of the appendices.
Other elements of the 2nd Joint Statement were also agreed. These included that Horizon had produced over 3 million sets of monthly branch accounts (a point relevant to Dr Worden’s Section 8 statistical exercise), and also that the experts had different views on “branch impact”. The relevant passages in the Joint Statement were that:
“Mr Coyne refers to any discrepancy that caused a loss (or gain) within branch accounts that needed corrective action as an “impact to branch accounts”. Dr Worden only considers an effect or impact on branch accounts where a discrepancy loss (or gain) was not rectified by a correction such as a Transaction Correction.”
This in particular demonstrates an important difference in the approach of the experts. Given, as I have explained above, that the Horizon Issues do not refer to Transaction Corrections, which even the Post Office’s own witnesses explained had not been addressed because the Post Office’s solicitors said these were outside the scope of the Horizon Issues trial, I consider Mr Coyne’s approach to “impact on branch accounts” to be the correct one. Dr Worden’s approach minimises or avoids impact on branch accounts that have been caused by bugs, errors or defects, even where those that are (or have been) present in the Horizon system have in fact affected the branch accounts of SPMs. It is a feature that I take into account when I consider which of the two experts’ approaches and evidence I prefer. I also explain this further in “Lasting and Transient Impact” below.
The 3rd Joint Statement continued the process of setting out the experts’ agreed position by working through the remaining Horizon Issues. This statement dealt with Horizon Issues 3 to 8 and was dated 1 March 2019. The experts agreed the following points in respect of each. The numbering is mine. One entry below relates to one passage of Dr Worden’s separate and not agreed view under Horizon Issue 3. It was not the only such entry – in different places the experts would record their own separate views.
The 3 rd Joint Statement
Horizon Issue 3
“1. Irrespective of how you define the detail of robustness, in line with most other large-scale computer systems, Horizon's robustness has generally improved. From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events. This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.
Computer systems are considered more robust if access to the back-end databases is restricted tightly.
It was possible for some of the Horizon support staff working at Fujitsu to modify the Horizon back-end branch database. In 2012, Post Office's auditors observed that there were inappropriate system privileges assigned to the APPSUP role (which allowed amendments to the BRDB).
Post Office does not consult the full audit data (unfiltered ARQ Data) before deciding how to handle discrepancies and issuing Transaction Corrections.
PEAKs show that some defects have lain undetected in Horizon for extended periods without being diagnosed and fixed.
During the life of Horizon there have been 19,842 changes made to it via the Fujitsu/Post Office release mechanism.
It is common modern IT development practice to make frequent incremental builds and releases of software.
Specific release note detail has not been provided. Of the 19,842 changes, we would expect that many were minor changes. It is likely that others contained changes to improve the system or to fix bugs and defects.
The effectiveness of various countermeasures changed throughout the life of Horizon.
Countermeasures are basic elements of practical IT system design
Countermeasures work by limiting the impact of Horizon bugs/error and defects on branch accounts. Countermeasures do not always eliminate the effects of adverse events (they are not perfect) but they are often effective in the area where they are deployed; that is why they have become basic elements of practical IT systems design.
It is difficult to measure the extent of the robustness of Horizon, apart from how it might limit the extent of impact on branch accounts, as in Issue 1.
There are indications that in its first year of operation, and in the first year after the introduction of Horizon Online, the system suffered from more problems than in other years. One might expect a higher level of problems in these early periods. The extent to which these problems were serious, or evaded countermeasures, or caused discrepancies in branch accounts, is not agreed.”
Dr Worden stated the following, which was not agreed, following the entry immediately above. “The main way in which the experts have assessed the extent of robustness of Horizon is to ask to what extent failures of its robustness impacted branch accounts. I have addressed this in Horizon Issue 1. In answering this question, fluctuations over the years are of less importance than the sum over all years, if the sum over all years is small. I have found the sum of impacts on branch accounts over all years to be very small”. (emphasis added). This approach is considered further by me when I address Dr Worden’s Section 8 statistical analysis.
The agreed passages continued. “The users of any IT system play a role in assuring its robustness. The designers of a system should not make unrealistic assumptions about the users of the system. Unrealistic assumptions would lead to inappropriate design, making the system less usable.
As Horizon has changed throughout its lifetime, the existence and effectiveness of any countermeasures has too. To have considered the time dependence of all robustness countermeasures over 20 years, would have made the expert reports impossibly lengthy. There was not the time to do so.
Many software bugs can have the same effects as a user error (as illustrated, for instance, by the Dalmellington bug, which produced a remming error)”.
(emphasis added)
Horizon Issue 4
“1. Bugs, errors and defects identified in relation to Horizon Issue 1 are often relevant to Issue 4 in that they are ultimately errors arising from the processing of data in Horizon.
There is evidence within the PEAKs and KELs of bugs/errors/defects within Horizon arising from parts (a), (b) and (c) of this issue that occurred without causing financial discrepancies as well as some that occurred causing financial discrepancies.
Reference data is critical to the operation of Horizon and errors in reference data have led to discrepancies in branch accounts.
(emphasis added)
Of the bugs which in the experts' opinion had the potential to produce discrepancies in branch accounts there may be some involvement of reference data in Bureau Discrepancies, Bureau de Change, Wrong branch customer change displayed, Lyca top-up, and Drop and Go. (Rows 14, 23, 24, 25, and 28 of the bugs table in the second expert joint statement). It is notable that these bugs all concerned specific products (arising from the reference data defining those products).
So, while reference data bugs may be a significant proportion of the bugs with financial impact, once discovered, they could be quickly fixed (by a change to the reference data) once the bug is correctly identified.
Reconciliation between transactions recorded on Horizon and transactions recorded by Post Office's clients is largely automated. Detected discrepancies were subject to manual corrective fixes and/or the issue of Transaction Corrections/Error Notices to the Subpostmasters.
The adequacy of Post Office back office processes to prevent discrepancies in branch accounts can be measured by the quality of the TC process. This quality includes:
The processes of consideration of available data
The level of errors observed in the process
The level of complaints or disputes raised following a TC
The level of upheld complaints following a TC
The level of financial impact of erroneous TCs
Errors in third-party data have led to discrepancies in branch accounts, through erroneous TCs being issued on Subpostmasters.
PO does not control the level of errors made by its third-party client organisations (which may lead to errors in TCs), or the delays in their processes (which may lead to delays in TCs).
(emphasis added)
PO can and should ensure, by careful investigation of disputed TCs, that only a small proportion of errors by PO clients lead to losses for Subpostmasters, provided that the Subpostmasters are in good control of their branches and have the required information available.”
Horizon Issue 6
“1. It is agreed that there are many measures and controls within Horizon that existed to prevent, detect, identify report or reduce the risk of varying errors.”
Horizon Issue 7
“1. Fujitsu could access all transaction data recorded by Horizon.