Reading Comprehension Assessment Tools: Why Two Scores Disagree
You have a student whose reading comprehension assessment tools disagree with each other. The school psychologist's battery put comprehension at the low end of average. The measure you gave in session put it two standard deviations lower. Nobody made an error.
This is the one construct on your assessment shelf where the format of the task is not a delivery detail. It is the construct. A cloze item, an open-ended question about a story the student read aloud, and a multiple-choice item attached to a passage the student can still see are three different cognitive tasks wearing the same label.
The field has known this for a long time, and the number attached to it is worth carrying around.
The finding that should change how you read a comprehension score
In 2014, Janice Keenan and Chelsea Meenan published Test Differences in Diagnosing Reading Comprehension Deficits in the Journal of Learning Disabilities.
They gave 995 children four comprehension measures — the Gray Oral Reading Test, the Qualitative Reading Inventory, the Woodcock-Johnson Passage Comprehension test and the Peabody Individual Achievement Test — and asked a simple question. Of the children each test placed in the lowest ten percent, how many did the other tests also place there?
The average overlap between tests in diagnosing comprehension difficulties was only 43 percent.
That is the authors' own figure, reported in the abstract of record. Fewer than half. And the paper reports the same instability at the other end: the tests disagreed about who the top performers were just as much as they disagreed about who was struggling.
One more detail from that study matters more than it first appears. Agreement was better for younger children and worse for older ones. In the early grades, a comprehension deficit is usually a decoding deficit wearing a comprehension label, so every test picks it up. By the time decoding is fluent, the tests are free to measure whatever else they measure — and they diverge.
That is the practical shape of the problem. The students on your caseload where a comprehension score matters most are the ones where comprehension scores agree least.
The earlier work this sits on
Keenan and Meenan were not first. The 2014 paper extends Reading Comprehension Tests Vary in the Skills They Assess: Differential Dependence on Decoding and Oral Comprehension, by Janice Keenan, Rebecca Betjemann and Richard Olson, published in Scientific Studies of Reading in 2008 — the citation of record is held on the University of Colorado's research profile for that work.
Before either, Kate Nation and Margaret Snowling asked the same question in the British Journal of Educational Psychology in 1997, in Assessing Reading Difficulties: The Validity and Utility of Current Measures of Reading Skill. Their study of 184 children aged seven to ten found something precise and still useful:
While sentence completion was well accounted for by individual differences in word recognition, text comprehension was more heavily dependent on listening comprehension.
Read that twice. A sentence-completion comprehension task behaved like a decoding measure. A passage-comprehension task behaved like a language measure. Both were sold as comprehension.
Nothing in the intervening years has overturned this, and no publisher disputes it. The formats are different on purpose. What the field has never produced is a convention telling you which one a given referral question calls for.
And the finding is not confined to American instruments. Danielle Colenbrander, Lyndsey Nickels and Saskia Kohnen published Similar but Different: Differences in Comprehension Diagnosis on the Neale Analysis of Reading Ability and the York Assessment of Reading for Comprehension in the Journal of Research in Reading in 2017, holding the record in the same place with two widely used instruments outside the United States, as catalogued by Australian Catholic University's research bank.
Three research groups, three decades, two continents, and the same result. This is about as settled as anything in reading assessment gets.
Four formats, and what each one is actually asking
Before the instruments, the formats. Almost every comprehension score you will ever read comes from one of four tasks.
Cloze
A word is removed from a sentence or short passage and the student supplies it. The task rewards syntactic and semantic fit at a local level. It is quick, it scales, and it is heavily dependent on word-level skill — which is the Nation and Snowling finding in one line.
Silent reading, then questions with the passage gone
The student reads, the passage is withdrawn, and the questions are asked. This is as much a memory task as a comprehension task, and the research literature says so directly. Hua and Keenan made this explicit in The Role of Text Memory in Inferencing and in Comprehension Deficits, in Scientific Studies of Reading in 2014:
When text memory was perfect, group differences disappeared, indicating that poor comprehenders can generate inferences as well as controls, if they have the relevant information in memory.
Oral reading, then open-ended questions
The student reads aloud, which contaminates nothing but does mean the comprehension score sits downstream of a fluency performance. Open-ended questions are scored by the examiner, which introduces judgement the other formats do not have.
Multiple choice with the passage available
The student can look back. This is the format most like actual academic work and the one most vulnerable to a student who is skilled at locating text without understanding it.
None of these is wrong. They are four measurements of four overlapping things. A student can be genuinely strong at one and genuinely weak at another, and both scores can be accurate.
What none of these tests measures
There is a second problem underneath the first, and it is older than any instrument on this page.
In 2002 the RAND Reading Study Group published Reading for Understanding: Toward an R&D Program in Reading Comprehension, written by Catherine Snow. Among its research priorities was the development of
an assessment system for reading comprehension that includes the design of valid and reliable measures of self-regulated, strategic reading that are sensitive to instructional interventions.
That was a statement about what did not yet exist. Three years later, David Francis, Jack Fletcher, Hugh Catts and J. Bruce Tomblin took up the point directly in Dimensions Affecting the Assessment of Reading Comprehension, a chapter in Children's Reading Comprehension and Assessment, published by Routledge in 2005 and catalogued in the University of Iowa research repository.
Their observation is the one to carry into a report. The RAND group had described three categories of comprehension outcome: knowledge, application of content to the world, and engagement with the ideas and style of a text. Francis and colleagues noted that contemporary assessments concentrate on content knowledge alone, that approaches addressing integration and evaluation of information remain uncommon, and that assessments of application or engagement are, in their word, rarer still.
So the instruments do not merely disagree with each other. They agree in measuring the narrowest of the three things the construct is generally taken to include. A comprehension standard score is a measure of knowledge extraction from a short passage under timed or untimed conditions. It is not a measure of whether a student can use what they read, and no publisher on this page claims that it is.
What the publishers themselves say
This is the part worth keeping. Every claim below is what the issuing publisher publishes about its own instrument, with the page linked. Where a publisher does not say something, this article does not say it either.
Woodcock-Johnson V: one battery, three tests, three different tasks
The clearest illustration of the whole problem is inside a single battery. Riverside Insights' Woodcock-Johnson V ships three separate tests whose names all point at comprehension, and they are not interchangeable.
Passage Comprehension and Paragraph Reading Comprehension both appear in the WJ V Achievement battery, with Paragraph Reading Comprehension marked as new, according to Riverside's own WJ V achievement and CHC mapping document. Both are coded to the same two broad abilities, Grw and Gc.
Riverside describes Paragraph Reading Comprehension in its own test spotlight as a task in which students read short stories silently and then answer comprehension questions read aloud by the examiner, with adaptive administration for individuals from 6 to 80 years and older.
Then there is Story Comprehension, described in its own spotlight as a test in which the examinee listens to short stories from an audio recording and answers questions read aloud by the examiner. It is coded to listening comprehension and oral language, not reading.
So one publisher, in one current battery, sells you a cloze-lineage reading test, a silent-reading-plus-oral-questions test and a listening test, and the word comprehension is in all three names. The WJ V is normed on data collected from 2022 to 2024, per Riverside's WJ V brochure.
GORT-5: oral reading, open-ended questions, and an explicit design claim
PRO-ED's Gray Oral Reading Tests, Fifth Edition, Normative Update yields five scores — Rate, Accuracy, Fluency, Comprehension and an Oral Reading Index — across ages 6-0 to 23-11, in two equivalent forms. Sixteen passages, each followed by five comprehension questions.
The design claim is the interesting part, and PAR states it plainly on its own GORT-5 listing:
The open-ended format ensures that the items are passage-dependent.
Passage-dependent means a student cannot answer from general knowledge alone. That is a real design virtue and it is not the same virtue as construct breadth. A comprehension score built from five open-ended questions per passage, scored by you, is a different animal from a thirty-item multiple-choice subtest.
TORC-4: five subtests, only one of which looks like comprehension
PRO-ED's Test of Reading Comprehension, Fourth Edition is a 2009 instrument for ages 7-0 to 17-11 that assesses silent reading comprehension through five subtests:
Relational Vocabulary — matching word sets on semantic connections
Sentence Completion — selecting word pairs to complete sentences
Paragraph Construction — arranging unordered sentences into a coherent paragraph
Text Comprehension — multiple-choice questions about short passages
Contextual Fluency — identifying word boundaries in dense text, in three minutes
Four of those five are not passage comprehension by any ordinary reading of the phrase. One is a timed word-boundary task. If a report hands you a TORC-4 composite and you read it as "comprehension" in the same sense as a GORT-5 comprehension score, you have misread it — and the publisher's own subtest list is what tells you so.
WIAT-4 and KTEA-3: the two batteries you will meet most often
Pearson's Wechsler Individual Achievement Test, Fourth Edition was published in 2020, covers ages 4:0 to 50:11, and carries a Reading Comprehension subtest inside its core reading composite.
Pearson's Kaufman Test of Educational Achievement, Third Edition was published in 2014, covers ages 4:0 to 25:11 across grades pre-K through 12, and lists Reading Comprehension among its nineteen subtests.
Neither storefront page publishes a task description for its comprehension subtest. Format, item type, whether the passage stays in front of the student, timing — none of it is on the public product page for either instrument. That information lives in the examiner's manual, which means the practitioner reading a report cannot check it from the open web. It is worth naming that plainly rather than filling the gap with an assumption.
GMRT: the group test in the school file
Riverside's Gates-MacGinitie Reading Tests run from pre-reading through an adult reading level. Riverside's GMRT brochure says the tests measure each stage along what it calls the comprehension continuum, from listening skills to mature reading comprehension, and that the mature comprehension passages are taken from previously published books and periodicals in a variety of writing styles.
This is the score most likely to be sitting in a school file with no protocol attached, and it comes from a group-administered instrument. A group-administered score and an individually administered score are not the same evidence, whatever the two standard scores look like side by side.
QRI-7: the one that is honest about not being norm-referenced
Pearson's Qualitative Reading Inventory, Seventh Edition, by Lauren Leslie and JoAnne Schudt Caldwell, assesses comprehension through retellings, explicit and implicit questions, and look-backs — the student may return to the passage.
It is an informal inventory, not a norm-referenced test, and nothing about it pretends otherwise. That makes it the most useful instrument on this list for instructional planning and the least useful for anything that needs a standard score. Being clear about which of those two jobs you are doing is most of the skill.
The storefront question, and a finding that did not repeat
Regular readers of this site will know the pattern: the same test is often sold at different qualification levels by different storefronts. Our post on achievement test qualification levels found the TOWL-4 listed at one level by two publishers and a different level by a third, and the math assessment tools post found two more instances.
This time the finding did not repeat, and that is worth reporting too. The GORT-5 is listed at Test Level B by PRO-ED and at Qualification Level B by PAR. They agree. The TORC-4 is Test Level B at PRO-ED. The WIAT-4 and the KTEA-3 are both Qualification Level B at Pearson.
It is worth knowing what the letter actually buys. PRO-ED's qualifications policy defines Level A as products for which "there are no special qualifications to purchase," and Level B as available to holders of a master's degree in psychology, education, occupational therapy, social work, counseling or a related field with formal training in the ethical administration, scoring and interpretation of clinical assessments, among several other routes.
Pearson's qualifications policy uses the same three letters and draws Level B in almost the same words, reserving Level C for instruments that, in Pearson's phrasing, "require a high level of expertise in test interpretation."
Two publishers, two policies, one vocabulary, and no industry body behind either of them. That is the finding our qualification levels post sets out at length, and the comprehension shelf is a well-behaved corner of it rather than an exception to it.
So on the comprehension shelf specifically, the letters line up. A practitioner who can purchase one of these can generally purchase the others — which removes the purchasing question and leaves only the harder one, which is what the score means once you have it.
What this does to a re-test
Here is where the 43 percent figure stops being an academic curiosity and starts costing somebody money.
A student is assessed in the spring with one battery. You pick them up in the autumn and, for perfectly good reasons — you own it, you are trained on it, it has two forms — you re-test with a different instrument. The comprehension score moves.
You now cannot tell whether the student changed or the instrument did. The Keenan and Meenan result says that for a student at the extremes, a change of instrument alone is enough to move them in or out of the lowest decile roughly half the time.
This is the same hazard the progress monitoring tools post describes for benchmarks and percentiles across monitoring systems, and the reasoning is identical: comparability lives inside an instrument, not across instruments. If the point of the re-test is to show change, use the same instrument and its alternate form. If you must switch, say in the report that you switched and that the two scores are not directly comparable.
That sentence costs you nothing and protects the family from a conclusion the data will not carry.
Administering is not interpreting
The Association of Educational Therapists' fact sheet describes the educational therapist as administering formal and informal educational assessments for which they are both qualified and trained, and interpreting assessment results. The same fact sheet's DO NOT list says the educational therapist does not:
administer cognitive, intelligence, or psychological tests (unless otherwise qualified to do so)
The parenthetical is part of the sentence and belongs in any quotation of it. AET's Code of Ethics is the binding document, and section II.I.D directs the educational therapist toward related-field professionals for interpretation — the boundary our post on reading a psychoeducational report works through in detail.
For comprehension specifically the line lands in a useful place. Describing what a given comprehension score is made of is reporting on a published test design. Concluding from a comprehension score that a student has a specific learning disability is not, and the answer to "can I identify a disorder from this?" is a flat no — the same answer the executive function assessment tools post gives for its shelf.
What the evidence base does and does not say about the teaching
It would be easy to slide from "the tests disagree" to "nobody knows how to teach comprehension." The published record does not support that, and the distinction matters.
The Institute of Education Sciences practice guide Improving Reading Comprehension in Kindergarten Through 3rd Grade, released in September 2010, carries five recommendations with explicit evidence levels: teaching comprehension strategies is rated strong; teaching text structure and establishing a motivating context are rated moderate; guided discussion of text meaning and strategic text selection are rated minimal. The full guide carries the evidence appendix behind each rating.
A second guide covers the older students where the test-agreement problem is worst. Providing Reading Interventions for Students in Grades 4-9, released in March 2022, rates three of its four recommendations as strong evidence: building decoding skills for multisyllabic words, purposeful fluency-building activities, and a set of comprehension-building practices.
Those practices cover vocabulary and conceptual knowledge, question-and-answer work about text, a procedure for identifying main ideas in sections of text, and comprehension monitoring. Practice with challenging text is rated moderate. The overview document is the short version.
Those ratings are about instruction, not measurement, and one does not transfer to the other. A strong evidence rating for strategy instruction tells you nothing about which comprehension test should be used to detect its effect — which is, if anything, the sharper unanswered question. Our post on whether educational therapy is evidence-based works through what these clearinghouse ratings do and do not license.
The National Center on Intensive Intervention's Academic Screening Tools Chart is the other place practitioners look, and its own framing is worth quoting: the Technical Review Committee rates each submitted tool against established criteria but did not compare it to other tools on the chart. A tool absent from the chart was not judged and found wanting. It was not submitted.
A sequence, not a decision
This article will not tell you which comprehension test to give a described student. That is a clinical decision and it depends on facts about the student that no article has. What it can give you is the order of the questions.
What is the referral question? Eligibility, instructional planning and progress documentation call for different instruments, and only the first of the three needs a standard score at all.
What already exists in the file? If a recent comprehension score exists, the value of a second one from a different format is mostly in the contrast, not the number.
Is decoding fluent? If it is not, a comprehension score is substantially a decoding score and the Keenan and Meenan agreement pattern says every instrument will tell you roughly the same thing.
Which format matches the question? A student who comprehends well with look-backs and poorly without has told you something specific — but only if you chose the formats deliberately.
Are you qualified to purchase and administer it? See the qualification levels post for the letters and what they actually govern.
Will you re-test? If so, choose an instrument with alternate forms now rather than switching later.
Where the published record runs out
Being plain about the gaps is more useful than papering over them.
Neither Pearson product page publishes a task description for the WIAT-4 or KTEA-3 Reading Comprehension subtests. Format, item count and whether the passage remains available are in the manuals only. Do not assume; open the manual.
PRO-ED and PAR both describe the GORT-5 comprehension items as open-ended and passage-dependent, but neither storefront page states whether the story remains in front of the student while the questions are asked. That is a manual question.
The Keenan and Meenan study used earlier editions — the GORT-3, the QRI-3, the Woodcock-Johnson Passage Comprehension test and the PIAT. Its finding is about test formats, which have not changed, but nobody has published the same comparison on the current editions.
Riverside's public materials do not say how Passage Comprehension and Paragraph Reading Comprehension differ in task design, only that both exist and both are coded to Grw and Gc.
The GMRT normative update year is not stated in the brochure reviewed for this article.
The assessment system RAND called for in 2002 has not obviously arrived. No instrument reviewed here is presented by its publisher as a measure of self-regulated strategic reading that is sensitive to instructional intervention, which is what that report asked the field to build.
If any of those matters to a decision you are making this week, the manual or the publisher is the place to settle it, not this page.
Frequently asked questions
Why do two reading comprehension tests give different scores for the same student?
Because they are different tasks. Keenan and Meenan's 2014 study of 995 children found the average overlap between comprehension tests in identifying the lowest-performing ten percent was 43 percent, and the disagreement was larger for older students whose decoding was already fluent.
Which reading comprehension assessment tool is the most accurate?
The question does not have an answer as posed. Accuracy is relative to a construct, and the instruments define comprehension differently on purpose. Choose on the basis of the referral question and the format, and say in the report which format you used.
Can an educational therapist administer a reading comprehension test?
AET's fact sheet describes educational therapists as administering formal and informal educational assessments for which they are both qualified and trained, and its DO NOT list excludes cognitive, intelligence and psychological tests unless otherwise qualified. Purchase eligibility is a separate question, set by each publisher's qualification level.
Is a cloze test a real comprehension measure?
It is a real measure of something, and the research says that something leans heavily on word-level skill. Nation and Snowling found sentence-completion performance was well accounted for by word recognition, while text comprehension depended more on listening comprehension.
Should I re-test comprehension with a different instrument?
Not if the purpose is to show change. Comparability lives inside an instrument. Use the same test and its alternate form where one exists, and if you must switch, state in the report that the scores are not directly comparable.
Does the WJ V have more than one reading comprehension test?
Yes. Riverside's published materials list both Passage Comprehension and a new Paragraph Reading Comprehension in the WJ V Achievement battery, plus a separate Story Comprehension test coded to listening comprehension rather than reading.
Where to go next
If you work across reading, writing and mathematics, the companion pieces on written expression assessment tools and math assessment tools apply the same publisher-first method to their own shelves, and the post on grade equivalent scores covers the score type most likely to be misread in a comprehension report.
On the intervention side, reading intervention programs compared and what "Orton-Gillingham based" actually means cover what happens after the assessment. For the wider scope questions, the practitioner scope map and the role comparison pillar are the places to start. If you assess remotely, what publishers permit for remote administration is the companion to this one.
We publish practitioner-facing explainers like this one, sourced to the issuing bodies and the publishers themselves. Join the educator list and we will send the next one to you.
Comments