Written Ministerial Statement

The content of this written ministerial statement is as received at the time from the Minister. It has not been subject to the official reporting (Hansard) process.

Department of Education - Alleged Security Breaches in relation to Post-primary School Entrance Test Papers of November 2009: Report of the Department of Education’s Investigation

Download this statement as a PDF (1.36 mb)

Mr O’Dowd (The Minister of Education): On 29 May, 2011, a primary school principal and Chair of the Catholic Principals Association (CPA), reported to the Department of Education (DE) an alleged security breach involving an entrance test paper that was sat by P7 children in November 2009. The CPA also issued a public statement on the matter on the same day. This is attached at Annex A. The test paper in question was used by grammar schools within the Post-primary Transfer Consortium (PPTC) for the purposes of deciding Year 8 admissions in Transfer 2010.

On June 7, 2011, I stated to the Assembly that I would keep it informed of DE’s investigations into this matter. These investigations have reached a stage where this statement can now be provided.

The Chair of the CPA informed the DE that on or around the 23rd or 24th of May, 2011 a primary school principal had been approached by a parent whose child had been a P7 pupil in the 2009/10 school year. The parent passed to the primary school principal a set of test papers purporting to be the entrance test papers used by grammar schools within the PPTC and sat by some P7 children in November 2009. These papers were passed by the primary school principal to the CPA Chair who reported the issue to DE and publicly on May 29, 2011. The CPA’s Press Release on the subject stated:

“The allegation is that this information and the test papers were provided by a member of staff at one of the Catholic Grammar Schools administering the test”.

The CPA sent these papers to GL Assessment (GLA) - the company that produced the PPTC entrance test that was used for Transfer 2010: the CPA sent to GLA a copy of a Maths paper on 31 May, 2011 and a copy of the English paper on 1 June 2011. On June 8, 2011, the CPA sent copies of both papers to DE.

The papers are those that were sat by some P7 children in November 2009 for the purposes of grammar school admission. This has been confirmed by the GLA that produced these papers as the entrance tests that PPTC grammar schools used for admissions purposes in that year. Approximately half of grammar schools are within the PPTC. This consortium contracted with the GLA for their provision of an entrance test – to be sat by P7 children in November 2009, for the purposes of enabling schools within the consortium to make admissions decisions using the results of the test.

Three separate security concerns were raised and/or reported by the CPA as a result of their possession of these papers. These were expressed in the Press Release at Annex A:

• a breach in the pre-test security arrangements of the November 2009 test: The CPA Press Release of May 29, 2011 stated that the primary school principal who had received these November 2009 test papers from a parent on or around the 23rd or 24th of May, 2011, had also been informed by this parent that “they had access to the content of the 2009 GL Assessment Test Papers prior to their child sitting the tests”. In this respect the CPA was alleging a very serious breach in the pre-test security arrangements of this entrance test.
• alleged breaches in secure test development (i.e. the alleged repetition of questions) across 2009 and 2010. Having received the 2009 Test paper, the CPA Chair showed it to a small group of children who had sat the November 2010 test paper. He was concerned to find that they considered some of the 2009 test items to be the same as those they had faced a year later. The CPA Press Release stated: “some of the questions encountered by the 2009 pupils may actually have reappeared in the 2010 process”.
• the evident breach in the post-test security of the November 2009 test – potentially affecting the fairness of the November 2010 & 2011 test. The arrangements governing the PPTC test are such that not only should no-one see or possess a test paper before it is sat, but also no-one but the GLA should possess a test paper afterwards. The GLA contractually required any school that used the November 2009 test to return all copies of it to GLA once the test was finished, “to be counted back in”. Once it was clear, therefore, that the English and Mathematics test papers which found their way into the possession of the CPA in late May 2011 were the PPTC test papers of November 2009, it was clear that a breach in this post-test security had occurred. Indeed, the CPA Press Release stated that part of the allegation that it was reporting was that “an unknown number of parents had this paper available to them to assist their children in preparation for the 2010 test”. The CPA was concerned as to the potential for this particular breach to have presented “an unknown number of participants with an unfair advantage in their preparation for the tests”.

DE’s investigation has firstly been into these three different, but related, alleged breaches. The investigation has also sought assurances from the relevant bodies about future security. The investigation has been conducted by DE officials with assistance from the PPTC, CPA, GLA and the independent Education and Training Inspectorate. I have sought and received regular updates during the course of the investigation. In relation to these three different kinds of breaches, I can now report that the investigation has found as follows.

A breach in the pre-test security arrangements of the November 2009 test: It has proved difficult for the investigation to establish any information beyond the basic allegation in this area. The CPA has confirmed to DE that the primary school that reported this allegation did not receive or pass on any documentary information relating to, or supporting, the claim that a parent possessed the November 2009 test papers prior to their child sitting the November 2009 test. It was also confirmed to DE that both the primary school principal who reported the breach to the CPA, and the parent who reported the breach to that principal, wish to remain anonymous.

The PPTC and GL Group have also sought to obtain further information here. They have twice written to the CPA Chair seeking the information necessary to take an investigation into this specific allegation further – on 22 June, 2011 and again on 31 August, 2011. The information they sought included details about the Catholic Grammar School at which, it is alleged, the breach occurred. The Chairperson confirmed to them on September 6, 2011 that he could not provide this information and that he was required to maintain the confidence with which information had been provided to him.

As things stand then, the position on this allegation is that it remains unsupported by any evidence. My department will gladly take from any source any information that may enable further investigation on this point – but until that point must consider that further investigation is not possible.

Alleged breaches in secure test development (i.e. the alleged repetition of questions) across 2009 and 2010. The facts here are that:

• the CPA arranged for a small number of children who sat the PPTC Entrance Test of November 2010 to be shown, in late-May 2011, the PPTC Entrance Test papers of November, 2009. According to the recollections of these children in this context, the CPA considered that “some of the questions encountered by the 2009 pupils may actually have reappeared in the 2010 process”.
• both the 2009 and 2010 PPTC test papers contained, at the outset, a distinct set of questions called “Familiarisation Questions”. To quote the GLA: “The 2009 and the 2010 Entrance Tests both began with two pages of familiarisation (‘Practice’) test items. In the English paper, this consisted of a total of 10 questions, namely: three comprehension questions relating to a short passage of text; four spelling questions; and three punctuation questions. The Maths paper contained a total of nine practice questions in the familiarisation section, covering a range of question types featured in the examination paper. We can confirm that the same familiarisation questions were used for both the English and Maths papers in the 2009 and the 2010 sittings” (emphasis added).
• the PPTC have confirmed the nature and role of “familiarisation questions: “These questions, as the term ‘familiarisation’ suggests, are solely for the purpose of familiarising candidates with the style and layout of the questions on the paper and with the correct way to enter their answers on the multiple-choice answer sheet. The content of the questions is immaterial and the repeated use of the same questions for this function is irrelevant to any score obtained.
• both the PPTC and GLA have confirmed to my department that, aside from the repeated use of these familiarisation items which do not form part of the assessment-proper, no questions found in the 2009 test paper were repeated in the 2010 test paper – nor could this kind of repetition occur.

The 2009 and 2010 test papers both “began” with identical “familiarisation” sections. That may explain the CPA’s allegation. Beyond that, however, and in terms of the questions that comprised the actual tests, there was no repetition of questions.

The evident breach in the post-test security of the November 2009 test – potentially affecting the fairness of the November 2010 & 2011 test: Both the PPTC and GLA consider the evident breach in post-test security to be a serious breach in the contractual arrangements. All schools within the PPTC have contracted with the GLA for the provision of an entrance test and in so doing have signed a “Declaration of Use” document that commits them to the GLA’s requirements in terms of security before, during and after the sitting of tests. All such schools are also provided with operational instructions on the “Security of GLA Entrance Assessment Question Papers and Associated Materials”. All concerned consider that the evident breach in post-test security occurred within one of the PPTC schools – and indeed those are the specific terms of the allegation. The GLA is clear that there was no breach in its own secure handling of test papers and that the breach occurred within a school setting. That has been accepted by the PPTC. It appears, therefore, that a school is responsible for a security breach that is a contravention of that school’s contractual agreement with the GLA.

I consider that a matter for the PPTC and the GLA. They have concluded that whilst the CPA maintains the confidentiality of the sources who reported the security breach to them and at the request of those sources, it is not possible to identify the original source of the breach and how exactly it occurred.

My concern and responsibility here, and that of my department, is the functionality of the post-primary admissions process and the obvious need for this process to reach robust and fair admissions decisions. In that context, the investigation has considered the question of whether the post-test security breach in relation to the November 2009 test papers may have affected the admissions processes of 2010 and 2011 in line with the concerns expressed by the CPA. Did this breach, which may have allowed a past-paper to be available to children sitting the 2010 test or preparing for the 2011 test, present “an unknown number of participants with an unfair advantage in their preparation for the tests?”

The Chair of the PPTC wrote to DE on this issue on July 4, 2011. His position was that the availability of the November 2009 test prior to the 2010 and 2011 test could not particularly aid preparation. The PPTC’s argument was that, despite the PPTC’s efforts to have a secure and confidential test based on the delivery of the Key Stage 2 curriculum and not generating a preparation industry, “practice papers” for their annual assessments are annually produced (for sale) by various companies. They argued that these companies draw on the general availability of alternative GLA assessment products comparable to the PPTC test. They further argued that in a context where such specimen papers were widely available and used, some children’s alleged access to the November 2009 test as a past-paper is unlikely to have been significant in terms of admissions decisions.

My officials, with my agreement, sought the independent and expert advice of the Education and Training Inspectorate (ETI) on this position of the PPTC. In the light of the ETI advice, DE considered that, whilst it would be impossible to prove categorically that children who had access to the 2009 materials did not have an advantage, in all probability they did not, given the plethora of test preparation materials freely available and schools’ experiences of putting children through such tests for many years.

It is also relevant to mention here that the GLA has informed my department that, at the request of the PPTC, they produce a ‘Sample English Familiarisation Booklet’ and a ‘Sample Mathematics Familiarisation Booklet’. The GLA described the purpose of these booklets as allowing “children and parents to see examples of the 11+ test questions prior to the examination and gain an understanding of how the questions would be laid out in the actual examination”. The PPTC has informed my department that “every parent who received the PPTC pre-registration pack, whether or not they registered their child for the 2009 and the 2010 Entrance Assessments, would have been sent the sample familiarisation booklets”. The circulation of these booklets adds to the fact that any November 2009 test paper illicitly in circulation before tests in subsequent years would have been available in a context where much practice material was also available. In line with the view of the ETI, therefore, my department’s investigation has concluded that children alleged to have accessed the 2009 assessment materials will not have had any greater advantage conferred on them by that experience. Furthermore, the questions used in the main assessments were different from one year to the next.

Future security: In line with its ultimate responsibility for a functional post-primary admissions process, my department has sought and received assurances from the GLA and PPTC as to how they will in future seek to strengthen their security arrangements to avoid a future breach. A letter from the Managing Director of the GLA received by my office on 10 October, 2011, provides the following:

Both GL Assessment and the PPTC take the security of the transfer tests extremely seriously. As such, we have undertaken a number of communications with the schools participating in the transfer tests administered by GL Assessment and the PPTC in November 2011. These include:

The reissuing of the “Declaration of Use for the GL Assessment Special Access Test Series with the Post Primary Transfer Consortium (PPTC)” form, which stipulates the requirements of the test administration process and confirms the school’s acceptance of these requirements. All headteachers have now signed and returned these forms to the PPTC. GL Assessment has also received copies of these signed forms.

The PPTC held a meeting with the headteachers of all of its schools on 9 September 2011 in relation to this year’s transfer tests. Security arrangements were discussed at length. Three head teachers were not able to attend but the PPTC have subsequently made contact with each of them individually to reiterate the importance of security and the security measures that schools are required to enforce.

The PPTC is also updating the Administration Instructions for the transfer tests which it sends to schools in advance of the Entrance Assessments and these will include further instructions about security…

GL Assessment never releases any past papers and we never allow schools to use papers for their own purposes or reproduce our questions. All schools involved are fully aware of the strict procedures they are contractually obliged to follow.

The process for distributing, storing and implementing the tests includes a number of precautionary measures:

All schools involved in administering the Entrance Assessments are required to sign the Declaration of Use form. All schools administering the 2009 and 2010 transfer tests signed this document and returned it to the PPTC.

The Declaration of Use form requires the Principal to report any missing materials to the Chair of the PPTC, who will then report this to GL Assessment. Neither the PPTC nor GL Assessment received any such notification during either the 2009 or the 2010 Entrance Assessment process.

All GL Assessment transfer test materials are printed at registered printers, specialising in the security of assessments and the protection of confidentiality. Prior to delivery, the materials are stored in our own secure facility and once cleared for release, they are delivered via secure couriers in tamperproof packaging.

When the test items are delivered to the schools, they can only be signed for by the named contact, which is usually the Principal or a member of the Admissions team. The materials are then taken to secure storage within the school.

On receipt of the test materials after testing has taken place, GL Assessment checks and counts in the materials against those delivered to ensure every one has been returned. Once the marking has been completed, all test papers are securely pulped.

We always endeavour to do everything in our power to ensure the security of our tests and we will continue to do so. However, as is always the case, the ultimate responsibility for security has to lie with the schools who administer the assessments.

CONCLUSION

There is no suggestion of any weakness in how the GLA have maintained their security in the past, nor in how they will seek to maintain this in the future. There is also a general belief amongst those principally involved that the breach in post-test security which did occur in relation to the November 2009 test occurred within one of the member schools of the PPTC. That is a specific aspect of the allegation.

The issue that remains a concern is that, as the GLA put it, “the ultimate responsibility for security has to lie with the schools who administer the assessments”. This was the case in the period when the breach occurred and it will remain to be the case going forward. It appears that someone within a school setting took a copy of the November 2009 test paper at some point before 23rd May, 2011 – in direct contravention of the security arrangements with which the PPTC, under contract to the GLA, seek to govern their administration of their entrance tests. Whether the measures that the PPTC have taken with their member schools to avoid any kind of recurrence of this will be effective remains to be seen. There is no evidence that a serious breach has occurred, but the potential for one within a school setting has been demonstrated.

Under Article 30 of the Education Order, 2006, my department has issued guidance to all post-primary schools on their admissions processes – and by law all of these schools are required to “have regard” to this guidance. For Transfer 2010, DE issued its Transfer 2010 Guidance. DE then updated this document to apply to all subsequent years and published this as “Guidance to Primary School Principals, Post-primary Schools’ Board of Governors and Principals, and Education and Library Boards on the Process of Transfer from Primary to Post-primary School from September 2010”. It is available on the DE website at: http://www.deni.gov.uk/post_primary_transfer_policy_from_september_2010_-_pdf_822kb.pdf. This document recommends that all schools do not use academic admissions criteria on educational grounds. For schools that do not follow this recommendation and that choose to continue academic selection by administering independent entrance tests, this document makes them aware in detail of the risks for which they assume responsibility. These include the potential for a breach in test security. I re-emphasise the importance of the relevant schools understanding this information.

I consider, however, that this episode stresses the inherent difficulties involved in making sure that an entrance test will always be completely secure. An independent entrance test is a high stakes test reliant on a security regime only as strong as its weakest link among all those involved within a large number of schools.

As my department’s transfer guidance reflects, the law, as it stands, may not prohibit independent entrance tests. It does, however, require schools to use admissions criteria that can robustly select for admission the number of children that a school may admit. Schools must use functional admissions criteria and my department will always act in line with this. If schools are determined not to follow DE Guidance and wish to remain able to apply academic admissions criteria on the basis of independent entrance tests, they must ensure their testing arrangements are secure and have contingency arrangements against any breaches that occur.

ANNEX A: 29 MAY, 2011 PRESS RELEASE FROM CATHOLIC PRINCIPALS ASSOCIATION

The Catholic Principals Association has in the past few days been made aware of an allegation regarding the operation of the Post Primary Consortium Transfer Test process in 2009 & 2010 which if substantiated is a cause of great concern.

A member of our Association has reported that they have been approached by a parent who has stated that they had access to the content of the 2009 GL Assessment Test Papers prior to their child sitting the tests and that subsequently an unknown number of parents had this paper available to them to assist their children in preparation for the 2010 test.

The allegation is that this information and the test papers were provided by a member of staff at one of the Catholic Grammar Schools administering the test.

Fintan Murphy, Chair of the CPA stated,

“This allegation, if substantiated, is extremely serious as it undermines the integrity of the whole process of using unregulated tests. If true, it provided an unknown number of participants with an unfair advantage in their preparation for the tests and would therefore call into question the accuracy and integrity of the results and grades of all of the pupils who have sat the tests. Many principals have, since these tests began, had expressed grave concerns about the integrity and security of this process and the suggestion that the tests may have been leaked does not come as a great surprise to me personally. ”

GL Assessment who provide the tests, have always been very clear that participants in these tests should never have sight of any of the tests prior to their participation and have consistently refused to provide past papers to those schools or individuals who sought them as this could potentially impact on the standardisation process of the results. GL Assessment have made clear that they do not issue ‘past papers’ and have stated that tests are provided on a confidential basis to the Post Primary Transfer Consortium. Our understanding of the process is that GL Assessment create the tests each year by extracting questions from a database of questions. It is therefore possible that questions from one test could reappear in future tests.

As supporting evidence of these allegations the CPA has been provided with copies of the papers which are reported to be the official 2009 test papers.

Fintan Murphy CPA Chair stated, “Our initial attempt to investigate the accuracy of this claim involved showing them to participants in the 2009 tests who recall significant elements of the tests as having been those sat in 2009; though this is dependent on the recollection of young people who sat the test 18 months ago.”

“To seek further confirmation of the accuracy of this information we will be passing the tests to GL Assessment and to the Post Primary Transfer Consortium to confirm if these are the official tests as used in 2009. We will also be passing the information to the Department of Education to inform them of our concerns.”

“Initial investigations by CPA also lead us to concerns that not only were the 2009 tests available to a number of participants but we believe that some of the questions encountered by the 2009 pupils may actually have reappeared in the 2010 process. Again this is based on the recollection of a participant in the 2010 process. If this proves to be accurate, it further undermines the process and would clearly have provided a group of pupils with a clear advantage.”

On Saturday, a significant number of pupils will have received letters confirming their rejection by grammar schools across N. Ireland largely based on their performance in the unregulated tests. Undoubtedly this will have caused much upset and distress for those 11 year old children and their families. If it is confirmed that some of children sitting these tests did so with an advantage over others; gained through access to official test papers then the whole process of the use of the tests will be called into question; not just this year but for future years. We would therefore again call on the Catholic grammar schools to end their participation in these tests and to embrace a non selective system of education.

CPA believes that this allegation, if proven, will lead to a significant number of appeals this year and potentially to judicial reviews of the use of the tests by schools this year in their selection of pupils.

CPA calls for an immediate response from both GL Assessment and the Post Primary Transfer Consortium into these allegations. In particular we would wish for;

1. A confirmation or denial that the papers which have been provided to the CPA are copies of the official papers used in 2009.
2. An explanation as to the source of these papers and how they could have entered the public domain if they are the official papers.
3. A confirmation or denial that questions included on these papers appeared in the 2010 process.
4. Clarity around the security measures which have been in place regarding the test papers in the past 2 years.
5. Confirmation that a full investigation will be undertaken into these allegations in the coming weeks.

Fintan Murphy Chair, Catholic Principals Association