Public Law Project v The Information Commissioner

This decision relates to an appeal brought under section 57 of the Freedom of Information Act 2000 (“the FOIA”). The appeal is against the decision of the Information Commissioner (“the Commissioner”) contained in a Decision Notice (“DN”) dated 26 July 2022 (reference IC-130877- C0S1), which is a matter of public record.

Full details of the background to this appeal, the complainant’s request for information and the Commissioner’s decision are set out in the DN. The appeal concerns a request to the public authority in question, the Home Office (“HO”) for information relating to the criteria used to assess which marriage referrals should be investigated. The HO provided a copy of two annexes, withholding some content under sections 31(1)(a) (prejudice to prevention or detection of crime) and 40(2) (third party personal information) of the FOIA. The Commissioner’s decision was that the HO was entitled to rely on sections 31(1)(a) and 40(2) of FOIA to withhold the requested information.

The Commissioner maintains the position set out in his DN; namely that the HO was entitled to rely on sections 31(1)(a) and 40(2) of the FOIA to withhold the requested information from the complainant. The Appellant now appeals against the DN. The Commissioner opposes the appeal and invites the Tribunal to uphold the DN.

On 27 November 2020, the complainant wrote to the HO and requested information in the following terms:

“Please provide the following information:

(1)

Does the MRAU still use a triage model or similar system to decide which marriage referrals should be investigated as potential shams?

(2)

Does the model use nationality as a factor in assessing marriage referrals? If so, please provide a copy of the relevant Ministerial authorisation for the purposes of the Equality Act.

(3)

Please provide copies of any equality impact assessments or data protection impact assessments completed in relation to the model.

(4)

Please provide copies of any internal policies, guidance or standard operating procedures which deal with the process of handling marriage referrals and the use of the model.

In the event that you determine some of the information I have requested to be exempt from disclosure, please redact exempt information with black boxes, instead of snipping or excerpting, and please state which category of exemption you believe applies to the information.

If it is not possible to provide the information requested due to the information exceeding the cost of compliance limits identified in s.12 FOIA, please provide advice and assistance as to how I can refine my request, as required under s.16 FOIA.”

On 17 December 2020, the HO responded. It advised that the Marriage Referral Assessment Unit (MRAU) does use a triage model, and that the HO does have an Equality Impact Assessment (EIA) for the marriage assessment process. They disclosed two annexes with relation to the request and citied section 31(1)(a) and 40(3)(a) of the FOIA for some redactions to the requested information.

On 14 January 2021, the complainant requested an internal review. Following this, after several chasers by the complainant, and holding responses, the HO wrote to the complainant on 22 June 2021. It upheld its position in relation to section 31(1(a) and clarified its reliance on section 40(2) of the FOIA regarding personal information contained within the annexes, the HO also advised that they did not hold any further information in relation to the complainants third point within their internal review request, with relation to the full analysis of the triage model on different nationalities.

The complainant contacted the Commissioner on 22 September 2021, to complain about the way their request for information had been handled as follows:

“The Home Office has refused to disclose the criteria used by its sham marriages triage model. The internal review concluded that publication of the criteria would prejudice their ability to detect and deter sham marriages and would not be in the public interest. I disagree with the refusal to disclose the criteria. I think that public law standards require transparency about how the system works and refusal to disclose the criteria is not justifiable on public interest grounds.

The internal review states that the Home Office does not hold any further information about the impacts of the triage model on different nationalities. However, in light of the EIA, I believe that the Home Office does in fact hold such information. Even if the review referred to in the EIA was conducted by a third party, I think the Home Office likely holds information about it.”

The complainant added:

“I consider that the Home Office must disclose the criteria used by the triage model and could explain why they say they do not hold information about the further review of the nationalities involved. In particular, they could explain who conducted the review and who – if not the Home Office.”

As part of his investigation, the Commissioner viewed the withheld information in this case which consists of the two annexes which were redacted before disclosure to the complainant.

As part of his considerations, the Commissioner searched online for details of the named parties within the documentation to ascertain whether they were sufficiently “high ranking” to fall within the scope of the complainant’s request. He found that none of the redacted parties’ details were in the public domain. The Commissioner therefore considers that it would be unlawful to disclose these details.

The complainant provided some detailed arguments which included the criteria used by the triage model, information in the MRAU Guidance, and the failure of the HO to disclose its full analysis of the impact of the triage model on different nationalities, and generally, they said that it is not credible to claim that the disclosure of the criteria used by the triage model would prejudice the prevention or detection of sham marriages. The HO has disclosed some of the criteria in the remainder of the EIA. Given that the HO could disclose these criteria without prejudicing the prevention or detection of crime, the Appellant argues, it would not seem apparent why further disclosure would create a risk of harm.

The complainant agreed with the redaction of HO official names, however, they questioned the flowchart redactions and gave examples of position titles and descriptions of actions to be done as being unlikely to be personal data as it could not be used to identify a natural person, and regarding the EIA, they said that the HO EIA indicated that there were further and more detailed analysis of the equality impacts of the model “review of the nationalities involved has been conducted.”

The Commissioner put some of the concerns directly to the HO, specifically where they said they were evidenced by information in the annexes disclosed and asked for its views. Where relevant, they were included in the decision-making

The matters that must be considered, to establish whether section 31(1) is engaged have been frequently rehearsed in the case-law and are summarised thus:

In relation to point (ii), the nature of the prejudice, it is necessary to demonstrate a causal link between the disclosure and the harm claimed.

The disclosure of information under the FOIA may have a negative effect and, in particular on the voluntary supply of information to a public authority in the future.

As to the chance of prejudice, point (iii), it is not necessary to show that the prejudice would be significant (although the extent of the prejudice is relevant to the public interest balance). However, disclosing the information must have; “a very significant and weighty chance” of causing prejudice that is; “real, actual and of substance” – see: Department for Work and Pensions v Information Commissioner and FZ [2014] UKUT 0334 (AAC) and R (Lord) v Secretary of State for the Home Department [2003] EWHC 2073 (Admin).

In this context, the term “would prejudice” means that it must be more probable than not that the prejudice would occur. “Would be likely to prejudice” is a lower test – here, the chance of prejudice must be more than a hypothetical possibility: there must be a; “real and significant risk” of prejudice (see: Hogan and Oxford City Council v Information Commissioner EA/2005/0026).

The exemption in section 31(1) is a qualified exemption. If a public authority finds that the conditions for applying the exemption are satisfied, it must go on to consider whether, in all the circumstances of the case, the public interest in maintaining the exemption outweighs the public interest in disclosing the information (section 2 FOIA).

In relation to prejudice-based exemptions, such as the exemption under section 31 FOIA, there is always an inherent public interest in maintaining the exemption. This should be considered when the public interest test is applied.

The material time to consider the public interest balance is the time of the response to the request – see: Montague v Information Commissioner & Department of International Trade [2022] UKUT 104 (AAC) (‘Montague’).

Section 1 FOIA

When determining whether, or not information is held the Commissioner and Tribunal should apply the normal civil standard of proof i.e., on the balance of probabilities. The Tribunal has repeatedly confirmed that this is the relevant test; see, for example, Linda Bromley v the Information Commissioner and the Environment Agency (EA/2006/0072) at [13] Malcolm v Information Commissioner EA/2008/0072, at [24]; Dudley v Information Commissioner EA/2008/008, at [31], and Councillor Jeremy Clyne v IC and London Borough of Lambeth EA/2011/0190.

The Commissioner investigated the matter and held that the HO appropriately relied upon section 31(1) FOIA to the withheld information on the grounds that: -

“i)

The Commissioner is satisfied that the harms which the Home Office claims would be likely to occur if the withheld information was disclosed relates to the applicable interests within section 31, namely the prevention or detection of crime and the operation of immigration controls.

ii)

Prejudice to the Home Office’s law enforcement functions in sections 31(1)(a) and (e) would be likely to occur if the withheld information were to be disclosed and such prejudice is real, actual and of substance and there is a causal link between disclosure of the withheld information and the prejudice against which the exemptions are designed to protect.

iii)

The public interest in maintaining the exemptions under sections 31(1)(a) and (e) outweighs the public interest in disclosing the withheld information.”

The Commissioner further went on to conclude that the personal data held within the withheld information would be exempt under section 40(2) FOIA.

The Appellant’s Grounds of Appeal inferred that: Ground 1. The Commissioner erred in concluding that disclosure of the withheld information would be likely to prejudice either the prevention or detection of crime or the operation of immigration controls such that the exemptions under sections 31(1)(a) and (e) are not engaged –: Ground 2. The Commissioner erred in concluding that the public interest in maintaining the exemptions under section 31(1)(a) and (e) outweighed the public interest in disclosure – Ground 3. The Commissioner erred in failing to reach a decision as to whether the Home Office held further information about the equality impacts of the triage model on different nationalities.

The Commissioner maintained her position as outlined in the DN and resisted the appeal. The Commissioner set out additional observations in respect of the Appellant’s Grounds of Appeal.

The Commissioner remained satisfied, that disclosure of the withheld information to the public under the FOIA would assist sham marriage organisers or organised crime gangs in their attempts to facilitate sham marriages and / or entrance to the country illegally and that therefore disclosure would be likely to prejudice the prevention or detection of crime and the operation of immigration controls. The Commissioner maintained that this is the case with the criteria alone, without the weights given to the criteria. Regarding the disclosure of some of the criteria, having made further enquiries with the HO, the Commissioner understands that two of the criteria listed on page 6 (and referred to on page 7) of the EIA, namely, “number of shared travel events” and “age difference between the couple” can now be disclosed. An amended redacted version of the EIA will be provided to the Appellant in the hearing bundle.

However, the fact that two of the criteria has been disclosed does not mean that disclosure of the remaining criteria withheld would not be likely to prejudice the detection of crime and the operation of immigration controls for the reasons set out in the DN. The Commissioner maintained that disclosure of the remaining criteria would be likely to so prejudice and that the exemptions under sections 31(1)(a) and (e) are engaged with respect to the criteria withheld.

The Commissioner argued, there is a very strong public interest in protecting the ability of public authorities to enforce the law and in protecting society from the impact of crime. Further, when considering the public interest in preventing crime, it is important to take account of all the consequences that can be ‘anticipated as realistic possibilities’ – see London Borough of Camden v Information Commissioner & Voyias [2021] UKUT 190 (AAC) [11]. Whilst the HO did express the view to the Commissioner, that disclosing the withheld information may indirectly place the public at risk, the Commissioner accepted that it may not have been correct to go as far as to suggest that disclosure would result in any material risk to human life and therefore accepts that it would not be a factor to weigh into the balance. The Commissioner nonetheless remained of the view that the balance lays in favour of withholding the information withheld.

In addition, the Commissioner submitted that disclosure of the withheld information would not provide the public with any further information on issues such as the special significance of transparency and accountability in the context of algorithmic decision making, nor new technologies. The Commissioner contended that the specific public interest, referred to by the Appellant has been met by the HO’s disclosure to date. The HO has previously confirmed to the Appellant that the HO has disclosed its full analysis of the impact of the triage model on different nationalities. Disclosure of the information withheld from the EIA will not add anything further to the information already disclosed and will not assist the public further in determining whether the model is discriminatory.

In relation to the equality impacts of the triage model on different nationalities, the HO advised the Commissioner that, as part of the production of the EIA, a review of the nationalities involved in the marriage process was carried out and that this review was then copied into the EIA. The HO confirmed that no further review exists. The Commissioner stated that he had no reason to doubt the word of the HO regarding the above and as such is satisfied that, on the balance of probabilities, no such further information is held. The Commissioner would invite the Tribunal to issue a substituted decision notice to reflect this. The Commissioner averred that should the Tribunal require further information from the HO concerning this ground, it would be appropriate either for the Tribunal to join the HO as a party to the appeal or alternatively exercise its case management power to require the HO to provide further information or documentation to the Tribunal.

The Appellant maintained its appeal against the Commissioner’s conclusion that s. 31(1)(a) and/or (e) are engaged. The Appellant argued that there is no publicly available information that supports the conclusion that the undisclosed factors would permit individuals successfully to ‘game the system’, or to do so in a way that could not be mitigated or guarded against.

The Appellant accepted that, if and to the extent that, the Tribunal finds that the exemption(s) were engaged, that would have relevance to the public interest assessment. The Appellant did not agree that that interest is necessarily “very strong”: the strength of the interest will depend upon the basis upon which, and extent to which, engagement is found. The contention that there is no further transparency/accountability interest in disclosure, given the information already disclosed, is wrong.

The Appellant contended that transparency and accountability are not secured, in respect of an algorithmic system, by knowing simply that a system exists and is operating in a particular field. That is not the sense in which the Justice and Home Affairs Committee used the words; “what technology is currently being used”. Transparency and accountability are secured only by knowing what that technology is doing: i.e., how it makes its decisions. The need for transparency has been repeatedly emphasised in relation to algorithmic systems. It is notable, in this regard, that the HO has stated that the purpose of the criteria is to enable the HO to; “justify why specific features are used and significantly reduce the opportunity for explicit discrimination that a rules-based approach can introduce.” It is entirely unclear how the HO proposes to; ‘justify why specific features are used’ while refusing to disclose those features.

The Appellant stated for the same reasons, further disclosure would assist in understanding whether the model is discriminatory. Further, that the model was introduced to reduce ‘the opportunity for explicit discrimination’ in the application of the HO’s previous rules-based approach. It is important to consider whether the model has/has not achieved that objective; or has achieved the objective only by substituting concealed or implicit discrimination for explicit discrimination. That is particularly so given that the EIA indicates that the model was developed (i) in order to reduce the risk of discrimination, but (ii) was produced and trained by ‘leveraging’ “historic outcomes and associated data on sham marriages”, which will ex-hypothesi have been reached under the old system, with its acknowledged opportunity for explicit discrimination. It is argued that none of this can properly be assessed absent disclosure of the criteria.

The Appellant noted also that the HO has previously accepted that the balance of public interests tended in favour of disclosure of characteristics; “to understand the justification behind any indirect discrimination linked to a protected characteristic.” The Appellant agreed: but it followed that the same balance of public interest requires disclosure of the model factors. According to the Appellant, it is clear from the EIA that the model affects certain nationalities disproportionately, in that a higher proportion of individuals with that nationality fail triage, relative to the proportion of such nationals who are assessed. The Appellant averred that on its face, and contrary to what is set out in the EIA, that is indirect discrimination. Again, it is argued that it cannot be properly investigated absent disclosure of the factors.

The Appellant submitted on the balance of probabilities; more detailed analysis does exist. The Appellant stated that it is implausible that no preparatory or other work was carried out in relation to the material set out in the EIA. The graph referred to has presumably been derived from an analysis via Microsoft Excel or a similar programme, containing the base figures and the relevant assessment and calculations. Accordingly, the Appellant submitted that such information is likely to be held, and that disclosure of that information should be ordered.

Reuben Binns provided written a witness evidence to the Tribunal. Mr Binns is an Associate Professor of Human-Centred Computing at the Department of Computer Science, University of Oxford. Mr Binns stated that he has no formal relationship with the Appellant but has spoken at some of its conferences.

Mr Binns addressed, in his evidence: the importance of transparency, the risks of discrimination in the operation of algorithms, and specific possible risks that arise in relation to the MRAU Model.

In relation to transparency Mr Binns stated as follows:

“Transparency is important for public trust, particularly in the context of public decision- making, and especially concerning the use of algorithms, models, or ‘artificial intelligence’. Transparency is important for public trust because without it, processes are opaque, impersonal, and it cannot be ascertained whether they involve various sources of unfairness, injustice, and error. Transparency is achieved by disclosing aspects of the operation of the relevant decision-making system, so that its operation can be publicly scrutinised and assessed by those subject to it, but also by interested observers, civil society bodies, academics, and suchlike. Transparency can involve disclosure of many aspects of an automated system, including who built it, its purpose, results of testing, and more, but it always includes the sources of data used to develop a system, and relatedly, the features it uses to make decisions during deployment. Relevant academic and professional guidance stresses the importance of disclosure of these features: so, for example, algorithms used in the public sector must disclose their sources and decision-making process so ‘accountable, transparent and public'2 debate can occur. Ensuring that an algorithmic tool is ‘understandable to the community should mean that the resulting debate can be better informed’. According to guidance produced by the Alan Turing Institute and the ICO, public transparency about algorithms may ‘help the public to have a meaningful involvement in the ongoing conversation about the deployment of AI and its associated risks and benefits.’

In the context of the MRAU Model and marriages that are selected for various levels of investigation to determine whether they are shams, it is important that the public is able to have meaningful involvement in the debate about the risks and benefits of any model, algorithm or AI system deployed for such purposes. The MRAU triage model, like any model used to make or support high stakes decisions, has both risks and benefits. But it is not possible for affected individuals, or concerned citizens, to engage meaningfully in debate about those risks and benefits if they are unaware of even the features used to make such decisions.”

Concerning the risks of discrimination in the operation of algorithms, Mr Binns detailed:

“Algorithms are decision-making procedures, typically automatically executed by computers, that take an input and create an output; for instance, a credit risk algorithm might be given someone’s salary, borrowing history, and bank account information as inputs, and produces a risk score as an output.

When they are used to make decisions which affect particular individuals, algorithms can both directly and indirectly discriminate.5 Speaking broadly, they can directly discriminate when one or more of the inputs to the decision are protected characteristics: the person in question will be treated differently from other people, because of that protected characteristic. They indirectly discriminate when, even if the input involves no direct use of protected characteristics, the algorithm nevertheless has a worse effect on some people with a protected characteristic than others. Since many salient features about a person are to some extent statistically correlated with protected characteristics, some degree of indirect discrimination is frequently an issue when algorithms are based on statistical models.

The primary way to detect direct discrimination in an algorithm is to examine its inputs to see if any of them are a protected characteristic, or a proxy for such a characteristic. This analysis therefore only requires a list of the inputs to decisions, not any detail on the weights of particular inputs (assuming that if an input is used at all, it must have at least some effect on the output, however small). It may not always be obvious that an input is a proxy for a protected characteristic, but this is possible; for instance, an input might be inextricably linked to a protected characteristic, such as where attendance at a gender-segregated school is Inextricably linked to gender.

There are several preliminary kinds of analysis that may indicate prima facie evidence of indirect discrimination in an algorithm. One involves testing how it performs on representative samples of people with different protected characteristics to see if it has worse effects on some protected characteristics than others. This analysis requires access to the outputs of the algorithm, disaggregated by the characteristics of interest, e.g. to test whether an algorithm is indirectly discriminating by gender, one needs to see the distribution of outputs disaggregated by gender.

Another approach is to look at the inputs to the decision, alongside auxiliary data, to assess whether there are correlations between the input features and a protected characteristic. For instance, it might be that certain occupations are correlated with gender; in which case, knowing that an individual’s occupation is an input to the decision, combined with auxiliary data about the gender distribution within occupations, may provide clarity as to whether the use of that input may be causing indirect discrimination by gender.

Where prima facie indirect discrimination exists, I understand that (as a matter of law) it has to be justified. Any case for and against justification would also typically involve studying the input features that (individually or in combination) may be correlated with a protected characteristic to ascertain, among other things, whether there might be alternative input features which are so related to protected characteristics and would therefore avoid the discriminatory effect, and whether such alternative input features would be proportionate.

Prima facie evidence of indirect discrimination can be found by looking at the distribution of outcomes of a model disaggregated by protected characteristics, but knowledge of the input features is also relevant to assessing why the model may be indirectly discriminating, and also whether that indirect discrimination is justified. It should be noted that while these approaches require access to inputs and outputs of the model, the ‘inner workings’, such as the exact weights of particular inputs, the type of model being used, or its internal logic, do not necessarily need to be revealed.”

Turning to discrimination in the MIRAU Model, Mr Binns outlined that:

“I have considered below some specific possible risks of discrimination that arise in relation to the MRAU Model. The EIA states "[t]his automated triage process, developed in accordance with Analytical Quality Assessment (AQA) best practice to leverage historic outcomes and associated data on sham marriages in order to identify referrals of couples where there is an indication of potential sham activity, whilst still reducing the risk of discrimination".

The Home Office has stated an objective of the model is to; ‘reduce the opportunity for explicit discrimination associated with its previous rules-based approach’;

“However, just like human and rules-based approaches, algorithms trained on statistical data are equally capable of discrimination (both indirectly and in some cases, directly also6). This is in large part due to the fact that statistical models accurately reproduce human biases in the data that they are trained on. In the context of the MRAU Model, biases could enter at various stages.

First, there may be bias in the decisions to refer cases by registrars. While referral is mandatory for any couple who are not exempt, registrars also use their discretion to refer couples in cases where they suspect a sham marriage. If registrars exercising this discretion were to have implicit or explicit biases, then the selection of cases on which the triage model is trained will be biased. This may result in disproportionately more or less data on particular populations, such that the resulting triage model performs better or worse on those populations.

Second, there may be bias in the observations made by the registrars. The Home Office has disclosed that these observations are used as one of the inputs to the MRAU Model. For instance, it may be that registrars are unconsciously biased in favour of or against couples who interact in certain culturally, religiously, age-related, or nationality-specific ways. This would mean that this input to the model would encode the biases of the registrars, making marriages in those populations appear more likely to be a sham than they really are.

Third, there may be bias in the historical investigations made by Home Office officials, specifically the Immigration Intelligence staff who undertake investigations of marriages referred to them. If there were systematic biases in these historical investigations, and these have been used as training data for the triage model, then the model would inherit those same biases. For instance, suppose that Home Office Immigration Intelligence staff had an unconscious bias against partners from specific nationalities; or believed that couples with a 10-year age gap are much more likely to be engaging in a sham marriage than couples with a 5-year age gap. It may be that those specific nationalities are no more likely than others to engage in sham marriages, or that in reality couples with a 10-year age gap are no more likely to be engaging in a sham marriage than couples with a 5-year age gap. However, such unfounded biases and beliefs in the minds of Home Office staff might lead those staff to be more rigorous in their investigations of such couples. This would result in couples from those nationalities or with those age gaps being determined shams at a higher rate than the actual ground truth rate for those groups. The historical data would appear to confirm those biases, purportedly showing that couples with a 10-year age gap are more likely to be shams, even though this would not reflect the true base rates. These inaccurate disparities in the training data would be accurately reproduced in the model trained on them, therefore reproducing the underlying biases.

Fourth, even if there are no unconscious biases at any point in the training data creation, or during the recording of new cases, either from registrars or from Home Office staff, there could still be bias in the model. Models are trained to maximise performance across all classes. But it may be that the statistical relationships that best predict sham marriages differ for statistical minority populations compared to the majority. Perhaps a large age difference is associated with sham marriages for the majority, but actually predicts genuine marriages for specific minority populations for whom age differences are the norm. In such cases, the most accurate model would one which is systematically less accurate on the minority population. This problem might be solved by explicitly factoring in nationality and building a different model which was more accurate on the minority population, but the Home Office has claimed in the EIA that nationality is not used as an input.

Finally, it may be that input features that are genuinely statistically related to sham marriages, are also (for possibly unrelated reasons), statistically related to nationality, age, or other protected characteristics. In such cases, even without using those features explicitly, a model trained on such data would be more likely to classify couples with those characteristics as ‘red’. Even if there were equal proportions of data on all nationalities, sham marriages may be simply harder to predict accurately for some nationalities, because there is more variation in practices and marriage customs.

While it is not possible to say from the material provided thus far which of these biases are present and to what degree, the ‘Triage Fail / Country’ figure on page 10 of the EIA does demonstrate that some nationalities are more likely to fail triage than others. This could be due to a combination of the biases presented above, and / or due to differences in the base rates of sham marriages involving those nationalities. As the EIA itself states, ‘some nationalities have a greater chance of failing the triage process than others‘. The EIA states that this ‘is not considered, in itself, to be indirect discrimination but a record of those nationalities involved’. But an algorithm which disadvantages people with a protected characteristic when compared to people without it is indirectly discriminating unless it can be objectively justified. It may be that the use of such a model could still be objectively justified and therefore argued to be not indirectly discriminating; but to my knowledge no such objective justification has been offered by the Home Office.”

Prof. Tomlinson provided written witness evidence to the Tribunal. Prof. Tomlinson is a Professor of Public Law at the University of York. Further, he was previously employed by the Appellant as a research director but remains involved with the Appellant as an Associate Research Fellow.

Prof. Tomlinson provided an array of information such as his background, information regarding the Appellant, background to the appeal, the impact on individuals, the engagement of the exemption, public interest balance and information not searched for.

Prof. Tomlinson provided the following information in relation to the impact on individuals and indirect discrimination:

“Based in part on its general research experience and expertise relating to algorithmic decision- making, PLP is concerned that the triage model algorithm may have wider effects, specifically apparently discriminatory effects, as certain nationalities appear to ‘fail triage’ at a high rate. The redacted EIA states that no protected characteristic is used as one of the criteria, and nor is nationality. However, it also includes another (incompletely labelled) graph, showing that couples in which one person is of Bulgarian, Greek, Romanian, and Albanian nationality are flagged for investigation at a rate of between 20% and 25%. These nationalities are most likely to fail triage, on the basis that, according to the EIA, "[t]he combination of criteria that work together to produce an outcome are more likely to be present in notifications to marry from couples that have one of these nationalities present".

The rate at which these Bulgarian, Greek, Romanian, and Albanian nationalities are flagged for investigation is higher than the rate for any other nationality. By contrast, around 10% of couples involving someone of Indian nationality and around 15% of couples involving someone of Pakistani nationality fail triage. No other nationalities are labelled on the graph. According to the graph in the EIA, nationals from Pakistan, India, Albania and Romania are involved in large numbers of marriages (over approximately 1,250) triaged by the tool. In comparison, nationals from Greece and Bulgaria are involved in less than 500 of the marriages triaged by the tool, yet Bulgarian, Greek, Romanian, and Albanian nationals fail triage (and are flagged for investigation) at a rate of between 20% and 25%. Whilst nationals from Romania, Albania, Pakistan and India are triaged in similar numbers by the tool, couples involving Romanian and Albanian nationals fail triage (and are flagged for investigation) at a significantly higher percentage (over 20%) than couples involving Pakistani or Indian nationals (10-15%).

PLP has taken steps to seek to assess the relative impact of the triage model against available nationality data. This data does not directly relate to the issues raised by the triage model, since it is not concerned with marriages between relevant nationalities. However, it is the best evidence that PLP has been able to obtain from public sources. I have reviewed Office for National Statistics (ONS) published data on the "Population of the UK by country of birth and nationality: July 2020 to June 2021" (the ONS data). The data sources the ONS uses show international migration through two main indicators: nationality and country of birth. I have therefore looked at the tables within the ONS data relating to nationality and sex and country of birth.

If the MRAU were being applied consistently to all nationalities, we might expect to see each nationality failing triage at a rate proportionate to the number of non-British migrants. It is possible that inconsistencies may be attributable to an indirectly discriminatory effect of the algorithm. Without disclosure of the requested information, it is not possible to assess this with certainty, but I would suggest that there is at least a prima facie indication of indirect discrimination here.

The information disclosed to date by the Home Office is extremely limited and it is thus impossible to use to draw firm conclusions. However, the limited evidence available suggests that certain nationalities are affected disproportionately. These concerning trends can only be understood if the Home Office provides more detailed information about the MRAU model in line with PLP's FOI request.

As explained above, the EIA graph at suggests that Bulgarian, Greek, Romanian, and Albanian nationals are impacted at higher rates by the triage tool compared to other nationalities, and subsequently by Home Office investigations into the legitimacy of their relationship to prove, or disprove the indication that they might be planning to enter into a sham marriage. The Home Office accepts that it is in the public interest "to understand the justification behind any indirect discrimination linked to a protected characteristic". Given the model impacts particular nationalities disproportionately, this seems to equate to indirect discrimination.

This position seems to be supported by the data analysis above which suggest these nationalities also seem to be being picked out when they form relatively small proportions of migration into the UK by nationality.

Disclosure of the criteria is necessary, and of compelling public interest, in order to assess whether the model is discriminatory. The above issues point to the public interest being strongly in favour of disclosure of the information/ criteria sought; this is explored further below.”

Prof. Tomlinson stated in relation to the engagement of the exemption that:

“For the reasons set out in PLP’s Grounds, PLP maintains its appeal against the Commissioner's conclusions that s. 31(1)(a) (the prevention or detection of crime) and/or (e) (the operation of the immigration controls) FOIA are engaged. I focus here on the proposition that disclosure of the remaining five risk factors would permit individuals successfully to ‘game the system’, i.e. to alter their behaviour so as to seek to present a lawful marriage, or to do so in a way that could not be mitigated or guarded against.

From an academic perspective, research has highlighted that some factors are not capable of being gamed, for example, when the features of the algorithm are fixed and unalterable. A common example is race. Other examples include sex, disability, physical appearance, or nationality. Commentators note that criteria not based on user behaviour offer ‘no mechanism for gaming from individuals who have no direct control over those attributes.’

Furthermore, even if some factors are in principle capable of being ‘gamed,’ it is often the case that there is a low risk of this occurring in practice. For instance, it might require extensive efforts or coordination, or mean that claims by individuals are made without supporting evidence which negatively impacts the sustainability of their claims. Cofone and Strandburg put this point as follows: ‘because effective gaming requires fairly extensive information about the features employed and how they are combined by the algorithm, it will often be possible for decision makers to disclose significant information about what features are used and how they are combined without creating a significant gaming threat.’ It is worth noting in this respect that the three criteria that were unredacted in the EIA (i.e., (a) registrar observation of unusual couple behaviour; (b) number of shared travel events; and (iii) age between the couples) are not all fixed and unalterable and therefore are in theory capable of being gamed – but they were disclosed.

Overall, it is of significant public importance that concerns about ‘gaming’ are not used and do not come to be used as a blanket excuse not to disclose the workings of an automated system. I believe that what is required is a close analysis of the level of risk associated with disclosing a particular part of an automated system. This assessment should not turn on the broad nature of the task the system is undertaking or the general context in which it operates but on a detailed appraisal of whether any particular criteria are capable of being gamed and, in cases where they are so capable, a realistic assessment of the actual likelihood of gaming occurring. It is not clear from the current information we have whether this was the approach taken when considering the application of the exemption in this particular case. PLP's expertise would enable proper and objective analysis and consideration to be undertaken.”

Turning to the Public Interest Balance, Prof. Tomlinson contended:

“The opacity of automated systems in the immigration system general, and specifically matters concerning the MRAU triage model, is of profound societal concern. It is usually impossible for individuals who are subject to an automated system to understand the reasons for the decision made about them and, therefore, to submit these systems to appropriate scrutiny, including in relation to whether they comply with legal requirements imposed on public decision-making. This is a problem not only for individuals subject to such decisions but also for the established wider public interest in the maintenance of the rule of law generally. These concerns about opacity apply directly, and in particular, to the automated system used to trigger investigations into sham marriages. Without access to information about how an algorithm operates, an individual is unable to seek an appropriate remedy when things go wrong, including where they may have been the victim of indirect discrimination. As the Justice and Home Affairs Committee has noted, “a lack of transparency risks undermining trust in the justice system, and the rule of law”

PLP also has broader concerns about transparency and algorithmic decision making. More generally, the Home Office is increasingly using a range of automated systems to implement immigration policy, and various Home Office policy documents state that the use of digital technologies will be a growing trend in the coming years. Automated immigration systems are the subject of a growing body of research literature (which I cover in detail in my book, Experiments in Automated Immigration Systems (Bristol University Press, 2022)). By way of a recent and prominent example, the EU Settlement Scheme was underpinned by multiple layers of automation. The operation of this system, and how it fits into the wider development of Home Office’s use of such technologies, is set out at length in my article in the Oxford Journal of Legal Studies (‘Justice in Automated Administration’ (2020) 40(4) Oxford Journal of Legal Studies 708). It is also worth noting that the Central Digital and Data Office (CDDO) has developed an Algorithmic Transparency Standard7 to "help public sector organisations provide clear information about the algorithmic tools they use, and why they’re using them"; these are clearly concerns about these issues.

In respect of the risks of “gaming,” the Home Office has a potential legitimate interest in ensuring their systems cannot be abused or manipulated. However, it is vital that any assessment of the risk of gaming is informed by an analysis of the risk of gaming in the context of the particular system and that reliance on any such legitimate interest is justified. Little is known about the operation of this automated system, but the analysis that can be undertaken suggests that the actual risk of gaming is minimal, for the reasons I set out above. It appears blanket reasoning is being provided and what is required is a much closer analysis of the risks. It is, in my view, highly improbable that the risks of gaming, on a proper analysis, justify the extensive lack of transparency that is currently being maintained.

The balance of interest is in favour of disclosure. Disclosure of the withheld information would aid public understanding in an issue of profound significance, not just for those affected or potentially affected by the triage model algorithm, but also for general understanding of the operation of and effects of that and similar algorithmic decision-making processes.”

Lastly, with reference to information not searched for, Professor Tomlinson evidenced as follows:

“Lastly, I briefly refer to the third part of the appeal, namely PLP’s appeal against the Information Commissioner’s apparent conclusion that there is no further information held by the Home Office concerning the impacts of the triage model on different nationalities. This was not contained in the Decision Notice but is said to be based on a statement from the Home Office in correspondence of 19 July 2022 that “the DSA conducted a review of the nationalities involved in the marriage process. This review has been copied into the EIA. No further review exists.”

For obvious reasons, the question of what information is or is not held by the Home Office is not a matter within my personal knowledge, and I do not therefore address it in detail. However, I note that it seems very unlikely that (as the Home Office asserts) the information in the EIA was produced without any preparatory or other work being performed. In particular, it seems very likely that the Home Office holds the data underlying the graph in the EIA, and (very probably) a spreadsheet containing analysis on that data. It also seems likely that the Home Office may hold other preliminary workings on its conclusions, and potentially analysis of the indirect discrimination issue. The Home Office appears to have understood the question to be primarily about whether a “further review” has been conducted, but that is not what the internal review request says.”

The Appellant made the following submissions to encapsulate the argument replied upon by the Appellant in their skeleton argument by which the Tribunal must determine. The Appellant dealt with three issues it believed was at the forefront of the appeal.

The Appellant accepts that, if and to the extent that the Tribunal finds that the exemption(s) were engaged, that would have relevance to the public interest assessment. The Appellant does not agree that that interest is necessarily “very strong”: the strength of the interest will depend upon the basis upon which, and extent to which, engagement is found, arguing - :

“The contention that there is no further transparency/accountability interest in disclosure, given the information already disclosed, is wrong. Transparency and accountability are not secured, in respect of an algorithmic system, by knowing simply that a system exists and is operating in a particular field. Transparency and accountability are secured only by knowing what that technology is doing: i.e., how it makes its decisions. The need for transparency has been repeatedly emphasised in relation to algorithmic systems. It is notable, in this regard, that the Home Office has stated that the purpose of the criteria is to enable the Home Office to “justify why specific features are used and significantly reduce the opportunity for explicit discrimination that a rules-based approach can introduce”. It is entirely unclear how the Home Office proposes to ‘justify why specific features are used’ while refusing to disclose those features.

For the same reasons, further disclosure would assist in understanding whether the model is discriminatory. It is notable, as set out above, that the model was introduced to reduce ‘the opportunity for explicit discrimination’ in the application of the Home Office’s previous rules-based approach. It is important to consider whether the model has achieved that objective; has not achieved that objective; or has achieved the objective only by substituting concealed or implicit discrimination for explicit discrimination.

The Tribunal acknowledged the witness statements of Joseph Tomlinson and Ruben Binns as read and understood. We are extremely grateful for the care and industry applied by these witnesses to assist the Tribunal and properly presented in the Appellants cause.

The Tribunal also wish to express our gratitude to the meticulous and helpful submissions made by Counsel on behalf of the Appellants.