top of page

NCII Tools Chart Ratings: What a Full Bubble Requires

Writer: Andrea Chernin
Andrea Chernin
Nov 9
19 min read

Updated: 22 hours ago

You have been on the NCII tools charts. Most practitioners have. You scanned the rows for full circles, noticed that one tool had more of them than another, and treated that as the answer to a purchasing question.


That reading is wrong, and not for the reason you would expect. A full bubble is not a grade. It is a pass mark against a threshold that changes from column to column, and in two of the columns you care about most, almost nothing on the chart has ever earned one.


This is the fourth time this site has found the same shape in a different place. The letter on a test publisher's catalogue page is a purchasing rule rather than a scale. A continuing education hour is not a portable currency. Here, a full bubble is not one standard.

The short version


  • The four marks are convincing evidence (full bubble), partially convincing evidence (half bubble), unconvincing evidence (empty bubble) and data unavailable (a dash). Those are NCII's own words.

  • The numeric bar behind a full bubble is different in every column. It ranges from a lower-bound coefficient of 0.80 down to 0.40, depending on which standard is being rated.

  • Every bar is set on the lower bound of a confidence interval, not on the coefficient itself.

  • On the academic progress monitoring chart, the column for decision rules about changing instruction carries a dash on 203 of 216 rated rows and not one full or half bubble anywhere.

  • The ratings describe data that developers chose to submit, in response to an annual call.

  • A dash is not a failing grade. It means nobody sent the evidence.

What the four marks actually are


NCII states the vocabulary plainly in its own Tool Chart Users Guide, dated November 2019.


For each of the standards, the TRC reviews data submitted by developers of the programs and tools and provides a rating of convincing evidence (full bubble), partially convincing evidence (half bubble), unconvincing evidence (empty bubble), or data unavailable (dashed line).

Two things in that sentence are worth slowing down for. The subject of the review is data submitted by developers. And the rating is of evidence, not of the tool.


The same guide adds a sentence that sits in a sidebar and is easy to skip:


The TRCs rate submitted tools against technical standards but do not compare them to other tools on the same chart.

So the charts are not league tables. Two tools with identical bubbles were never compared with each other, and nothing on the chart was built to tell you which of them is better.

A full bubble is a different number in every column


This is the part nobody writes for practitioners. Click a column heading on either academic chart and NCII shows you the rating rubric it applied. The rubrics are specific, they are public, and they do not agree with one another about what a full bubble costs.



  • Classification Accuracy. A full bubble needs three design questions answered yes and "the lower bound of the confidence interval around the Area Under the Curve (AUC) estimate ≥ 0.80 and Sensitivity ≥ 0.80 and Specificity ≥ 0.80."

  • Reliability. A full bubble needs a model-based approach, or two other appropriate types of reliability from two representative samples, and for each type "the lower bound of the confidence interval around the median estimate met or exceeded 0.70."

  • Validity. A full bubble needs at least two justified validity analyses from a representative sample and a lower bound that "met or exceeded 0.60," or a value "within an acceptable range given the expected relationship with the criterion measure(s)."



  • Reliability of the performance level score. Lower bound at or above 0.70.

  • Validity of the performance level score. Lower bound at or above 0.60.

  • Sensitivity to student learning: reliability of slope. Lower bound at or above 0.50.

  • Sensitivity to student learning: validity of slope. Lower bound at or above 0.40.


Read that ladder again. The identical full circle means 0.80 in one place, 0.70 in another, and 0.40 in another. A tool with full bubbles across a row has cleared four different bars, and the chart renders all four with the same picture.


Four columns, four thresholds, one symbol. The symbol is the thing practitioners read and the threshold is the thing that decides it.

The threshold sits on the lower bound, not on the coefficient


Every numeric criterion above is applied to the lower bound of a confidence interval around a median coefficient. That is a deliberately conservative choice, and it is worth understanding rather than skipping.


A reported reliability of 0.74 does not earn a full bubble if the interval around it reaches down to 0.68. The chart asks what the worst plausible value is, and then compares that to the bar.


This site has made that argument before, from the other direction. The post on standard error of measurement has a section on exactly this asymmetry: the clearinghouse reads the lower bound and the practitioner reads the point estimate. What is new here is that the bound is not compared to one number. It is compared to a different number in each column.

The two columns that matter most are nearly empty


Every rated cell on these charts is in the page, so the distribution can be counted rather than guessed. We counted it on 9 November 2026, directly from the charts as published.


The academic progress monitoring chart carries 19 distinct tools across 216 rated rows. Column by column:


  • Reliability of performance level: 143 convincing, 22 partially, 51 unconvincing.

  • Validity of performance level: 109 convincing, 50 partially, 57 unconvincing.

  • Reliability of slope: 101 convincing, 11 partially, 45 unconvincing, 59 dashes.

  • Validity of slope: 8 convincing, 2 partially, 43 unconvincing, 163 dashes.

  • Alternate forms: 97 convincing, 44 unconvincing, 75 dashes.

  • Decision rules for setting and revising goals: 0 convincing, 22 partially, 13 unconvincing, 181 dashes.

  • Decision rules for changing instruction: 0 convincing, 0 partially, 13 unconvincing, 203 dashes.


Nothing on that chart has ever earned a full bubble for its decision rules. Not one row, on either decision-rules column. And the column that asks whether the growth estimate the tool produces is actually valid carries convincing evidence on 8 of 216 rows.


The performance-level columns, by contrast, are mostly full. So the chart is at its strongest exactly where a progress monitoring tool is least distinctive, and at its emptiest where the whole purpose of the instrument lives.


You do not buy a progress monitoring tool for its performance-level score. You buy it for the slope and the rule. Those are the two columns the chart cannot fill in.

What a dash means, and what it does not


NCII defines the dash as "data were not provided." That is a statement about a submission, not about an instrument.


A tool with a dash in the slope-validity column may have excellent slope validity that nobody studied, or studied and did not submit. A tool with an empty bubble in the same column submitted evidence that did not clear 0.40. Those are different situations and the chart distinguishes them.


The practical consequence runs the other way from intuition. An empty bubble tells you more than a dash, because an empty bubble means somebody looked.


NCII's own advice for a dash is in the users' guide, under the heading "Ask for More Information," and it is blunter than most federal prose:


Similarly, if a program or tool that you currently use or are interested in learning about is not on the chart, contact the developer of that program.

The screening chart is mostly full bubbles, and the one that splits is the one that counts


The academic screening chart carries 45 distinct tools across 459 rated rows, and it looks very different from the progress monitoring chart.


  • Reliability: 387 convincing of 459 rows.

  • Validity: 347 convincing.

  • Classification accuracy, fall: 194 convincing against 197 partially convincing, 36 unconvincing and 32 dashes.

  • Classification accuracy, winter: 191 convincing, 179 partially.

  • Classification accuracy, spring: 206 convincing, 163 partially.


Reliability and validity are close to a formality on this chart. Classification accuracy is the column that genuinely divides the field, and it is the one that decides whether a cut score flags the right students.


There is a structural reason it looks that way. NCII rates classification accuracy "separately for each criterion measure and time of year," for up to two criterion measures and three time points, and states that data beyond that "may be reported, but will not be rated." So the column is three columns, and a tool can be convincing in spring and partially convincing in autumn.


What counts as risk in that rating


The rubric's second design question asks whether "risk [was] adequately defined within an RTI approach to screening (i.e., 10th - 20th percentile)." That is the definition of the thing being predicted. It is not the definition you may be using, and it is not the definition the intervention chart uses, which we come to below.

"National" means 150 students from three census divisions


The screening chart carries a Sample Representativeness column with seven text labels rather than bubbles. The definition of the top label is a single footnote:


A national sample consists of at least 150 students across at least three of nine geographical divisions defined by U.S. Census Bureau.

Those are the divisions published by the Census Bureau. One hundred and fifty students across three of nine divisions is a defensible minimum for a technical review. It is a long way from what most readers hear in the word "national."


Across the 459 rated rows: 95 are National with Cross-Validation, 91 National without, 18 Regional with, 179 Regional without, 10 Local with, 61 Local without, and 5 report No Evidence. So the most common sample label on the academic screening chart is regional, uncross-validated.


NCII's own one-page brief on sample representativeness is worth reading beside that column, because it describes a second route to representativeness that the labels do not expose: a group of samples counts if the analyses across them produce approximately the same results.


The companion briefs on classification accuracy and validity are the other two worth having open. The first explains the rating with an airport security analogy and states the bar plainly: the charts "rate a screening tool highest when it has both a sensitivity rate and a specificity rate of 80% or higher."


Both briefs close by pointing readers to the National Center on Improving Literacy for the screening process itself, which is a useful admission in its own right. The chart rates the instrument. It does not describe the process the instrument sits inside.

Some bubbles rest on evidence NCII no longer accepts


This is the finding that changes how you read an old row, and it is published in a footnote on NCII's one-page brief on reliability:


Test-retest reliability may be shown for some tools because NCII accepted it in the past. However, our current standards no longer consider it sufficient evidence of reliability.

So the charts are not a single snapshot taken under one rulebook. A full bubble awarded some years ago may rest on a form of evidence the current standards would not count, and the chart does not stamp each cell with the standard it was judged against.


The forms of reliability NCII now assesses are alternate form, internal consistency, inter-rater, and model-based indices of item quality. If you are looking at a reliability bubble and the underlying data are test-retest coefficients, you are looking at a rating from an earlier era of the chart.

The legend's quietest marker, and where it never appears


Both academic charts carry a legend entry for a superscript d, meaning disaggregated data available. It is the only thing on the chart that speaks to whether a rating holds for a subgroup.


Counted on 9 November 2026, the marker appears 71 times across the 459 screening rows, on 11 of the 45 tools — and only in the Reliability and Validity columns. It appears in none of the three classification accuracy columns, on any tool, in any season.


On the progress monitoring chart it appears 41 times across 216 rows, on 4 of the 19 tools, in the reliability, validity and both slope columns.


Put plainly: on the chart built to help you pick a screener, there is no row anywhere telling you that the cut score performs comparably for a subgroup.


The separate Bias Analysis column is a different thing again — it reads Provided or Not Provided, and on the screening chart it reads Provided on 348 of 459 rows. That records that an analysis exists. It does not show you what the analysis found.


The users' guide anticipates exactly this gap and tells the reader what to do about it:


For example, what if a program that you are interested in does not have disaggregated data available for a subgroup that is important to you? Ask the vendor!

That is good advice. It is also the clearinghouse telling you that the most equity-relevant question on the chart is one the chart does not answer.

Who decides what gets rated


NCII's review process page sets out the mechanics, and three details matter for how you read a row.


  • Reviews are conducted by Technical Review Committees of content and methodological experts, who also develop the standards and rubrics they then apply.

  • "Calls for submissions occur on an annual basis," and vendors "are invited to respond." Submission is voluntary and periodic.

  • "A strict conflict of interest protocol is followed in instances of a TRC member submitting their own tool or intervention for review." So committee members may submit their own instruments, under a declared protocol.


None of that is a criticism. It is a public, documented process, and NCII says what it is. But it does mean the chart is a map of who applied, not a map of what exists — which is the single most useful thing to know before treating an absence as a verdict.

The intervention chart rates studies, and it is five years old


The academic intervention tools chart works on a different unit from the other two, and says so in its own description:


The chart includes information and ratings on the quality of research studies about different interventions, not ratings on the interventions themselves. The ratings are about study quality only.

This site's post on reading program evidence ratings has a section on that distinction and on how it differs from the What Works Clearinghouse's unit of analysis, and we are not going to restate it here. What belongs in a post about the ratings themselves is the arithmetic.


The chart carries 64 distinct programs across 119 rated studies. Column by column:


  • Study design: 55 convincing, 43 partially, 21 unconvincing.

  • Participants: 84 convincing, 35 unconvincing — and this column has no half bubble at all. It is pass or fail.

  • Fidelity of implementation: 48 convincing, 26 partially, 45 unconvincing.

  • Targeted measures: 82 convincing, 7 partially, 15 unconvincing, 15 dashes.

  • Broader measures: 74 convincing, 2 partially, 3 unconvincing, 40 dashes.


Nearly as many studies failed the fidelity rating as cleared it. And the Participants rubric sets the risk bar for the whole chart at "below 30th percentile on local or national norm; or sample mean below 25th percentile," which is a different and looser definition of risk than the 10th to 20th percentile the screening rubric uses.


Three more numbers from the same chart. Disaggregated effect-size data: 95 of the 119 studies report None, 13 report a percentile-based subsample, 6 a demographic one and 5 both. Review elsewhere: 44 studies have been reviewed by both the What Works Clearinghouse and Evidence for ESSA, 38 by Evidence for ESSA only, 5 by the WWC only, and 32 by neither.


And the final column counts other research "potentially eligible for NCII review" that has not been reviewed — 50 studies list zero, and the remaining 69 list between one and nine.


The chart's own front matter records that it was last updated September 2021. The screening chart says August 2026 and the progress monitoring chart November 2024. Three charts, three vintages, one house style.


The effect size column is not a rating


Worth saying plainly, because the column sits beside the bubbles and is read as though it were one. NCII computes Hedges g corrected for small-sample bias across every study so the figures are comparable, and prefers adjusted posttest means.


It also notes that by What Works Clearinghouse guidance "an effect size of .25 or greater is considered to be 'substantively important.'" An effect size is a result. The bubbles beside it are judgements about the study that produced it.


The borrowing runs deeper than the formula. NCII's design rubric states that it "follows guidance from the What Works Clearinghouse (WWC) in determining attrition bias" and points at pages 13 and 14 of the WWC Procedures and Standards Handbook. So one federal standard is nested inside another, and the handbook version is named on the chart.

The chart is not a shortlist, and NCII says so twice


Each academic chart's own description carries this, verbatim and in full:


The presence of a particular tool on the chart does not constitute endorsement and should not be viewed as a recommendation. All tools that meet the criteria for review are posted on the chart, regardless of results. The chart represents all tools that were reviewed, not those that were "approved."

A correction to this site's own record. In August 2026 we noted internally that we could not locate that second and third sentence as a literal string on any NCII page, and we stopped quoting them. They are there.


They sit in the "About this chart" description on all three academic charts, confirmed on 9 November 2026. We were looking in the wrong document, and we are stating that here rather than quietly starting to quote it again.

What is out of scope here


This post is about what NCII's ratings mean. Several adjacent questions are not ours to answer and are named rather than answered:


  • Whether any tool on these charts is appropriate for a particular student. That is a professional judgement, and the chart is explicit that it is not a recommendation.

  • Whether a screening result establishes a disability, an eligibility category or an entitlement to services. Those are matters of statute, regulation and district policy. Nothing here is legal advice. Take eligibility questions to your state education agency and, for anything turning on entitlement, to an attorney licensed where you practise.

  • Whether the tools you can license as an independent practitioner are the same tools a district can license. Most of these systems are sold to schools. Our post on progress monitoring tools compared covers who the access terms actually name.

  • What any individual coefficient means for one student's score. That is the standard error question, and it has its own post.

Can a full bubble tell you a tool will work for your caseload?


No.


A full bubble says that evidence submitted by a developer cleared a threshold set for that column, on samples that may be regional and uncross-validated, in a review that did not compare the tool to anything else on the chart. It is a floor on technical adequacy. It is not a prediction about a student, and the chart says in its own words that it is not a recommendation.

A sequence, not a decision


If you use these charts the way a technical reviewer would, the order runs roughly like this.


  • Decide which chart answers your question. Screening identifies risk, progress monitoring quantifies growth, and the intervention chart rates studies about programs. They are not interchangeable.

  • Read the column heading's rubric before reading the column. The threshold is in there, and it is not the threshold next door.

  • Separate the dashes from the empty bubbles. One is a missing submission and the other is a tested shortfall.

  • Check the sample label and the grade band on the specific row you care about, not on the tool as a whole.

  • Look for the superscript marker on the rating you are relying on, and expect not to find it.

  • Then click the rating. NCII publishes the submitted data behind each cell, and the users' guide recommends exactly this when two tools share a rating.

  • Write down what the chart did not tell you, and ask the developer for it in writing.

When several organisations share a vocabulary


Nearly every post on this site has run into the same problem from a different direction, and the pattern is now the point rather than the coincidence.



One consequence of how the charts are built is worth following up. Every reading construct reviewed on the screening chart is a rate measure, so the untimed instruments practitioners use to plan instruction appear nowhere on it. NCII keeps them on a separate, unrated Example Diagnostic Tools list instead, which is the subject of the comparison of phonics screeners and phonics inventories.

What the profession that credentials you says about any of this


Nothing, and that absence is now predictable enough to be worth stating as a pattern.


The AET Code of Ethics, Revision #9 of September 2025, runs to 2,978 words. Counted as whole words on 9 November 2026, it contains "chart" zero times, "rating" zero, "clearinghouse" zero, "tool" and "tools" zero, "vendor" zero, "developer" zero, and the bare word "evidence" exactly once — in a clause about representing your own qualifications.


"Reliability," "validity" and "screening" each appear once. The AET Fact Sheet, at 761 words of readable page text, carries "chart," "rating," "clearinghouse," "evidence," "research" and "screening" at zero each.


What the Code does say is the thing that actually binds you, at Section Two, I.B: select and use appropriate assessment instruments, "recognizing their limitations with respect to reliability, validity, and bias."


That clause is a duty to read the rubric, not a duty to prefer the tool with the most circles. It is also, read against this chart, a duty to notice that the bias column records whether an analysis happened and not what it showed.


This is the ninth consecutive post on this site to find the Code silent on its own subject. The conclusion has stopped being surprising: AET regulates conduct, not method.

Where the published record runs out


Several things a practitioner would want here are not published anywhere we could find, and the honest move is to say which.


  • Why the thresholds differ. NCII publishes 0.80, 0.70, 0.60, 0.50 and 0.40 for different standards and does not publish the reasoning behind each figure. A descending ladder from classification accuracy to slope validity is defensible on measurement grounds, but the justification is not on the charts.

  • Which standard each existing cell was judged against. The reliability brief states that test-retest is no longer sufficient. Nothing on a given row says whether that row predates the change.

  • What the bias analyses found. The column reports only that one was provided.

  • Why the three charts carry three different update dates, and what happens to a 2021 rating under a 2026 rubric.

  • Anything about the behaviour charts. NCII publishes six charts and this post counted three. The behaviour screening chart and its two behaviour siblings were not examined, and nothing here should be read as describing them.

  • Whether a rating transfers to a different name for the same instrument. The progress monitoring chart's own row label reads "Acadience Reading K-6 (aka DIBELS Next)", which is the name history this site covered in its post on the four systems; the publisher's own account of that split is the place to start, and the chart does not tell you which edition a rating was earned on.

  • The source universe here is narrow and deliberately so. This post rests on NCII's own charts, rubrics, briefs and users' guide, plus two AET documents and two external definitions. There is no independent methodological literature on these rubrics that we could locate, and padding the citations with secondary commentary would have made the post look better sourced without making it more true.

Common questions


Does a full bubble mean the same thing in every column?


No. It means the submitted evidence cleared that column's threshold. The thresholds published on the two academic charts run from a lower bound of 0.80 for classification accuracy down to 0.40 for validity of slope.


Is a dash the same as an empty bubble?


No. A dash means "data were not provided." An empty bubble means evidence was submitted and did not meet the full or half bubble criteria. An empty bubble carries more information than a dash, because it means somebody measured.


Why do so few tools have rated decision rules?


The rubric asks for a basis that is "strongly evidence-based," analysed from measurement collected at least weekly, from a sample of students in need of intensive intervention. On the academic progress monitoring chart, no tool has a full or half bubble for decision rules about changing instruction, and 203 of 216 rated rows carry a dash in that column.


Does being on the chart mean NCII recommends the tool?


No, and NCII says so in the chart's own description: the presence of a tool "does not constitute endorsement and should not be viewed as a recommendation," and the chart "represents all tools that were reviewed, not those that were 'approved.'"


If a tool I use is not on the chart, is that a bad sign?


Not by itself. Submission is voluntary and happens through an annual call. The users' guide tells readers to contact the developer and ask them to consider submitting. An absence is evidence about who applied.


What does the superscript d mean?


Disaggregated data are available for that rating. On the academic screening chart it appears on 11 of 45 tools and only in the reliability and validity columns, never in classification accuracy.


Can I use these ratings to tell a family a tool is reliable for their child?


Be careful with the word. A reliability bubble describes a coefficient estimated on a sample, often a regional one, with a lower bound at or above a threshold. It supports a statement about the instrument's technical adequacy in that sample. It does not support a claim about the precision of one student's score, which is a separate question with its own answer.


Where do I find the actual numbers behind a rating?


Click the rating. NCII publishes the submitted data behind each cell, and recommends doing exactly that when two tools share a rating and you need to tell them apart.

Related reading


If this post was useful, the closest neighbours on this site are the post on standard error of measurement, which covers the lower-bound asymmetry in depth; the post on reading program evidence ratings, which compares five rating systems; and the post on progress monitoring tools compared, which covers what the four systems are and who may license them.


For the wider frame, NCII's own pages on the data-based individualization process and the Taxonomy of Intervention Intensity are free and better starting points than any summary of them. The taxonomy's underlying paper has an ERIC record if you want the citation rather than the handout.


And if the phrase "ESSA tier" turns up beside a bubble in a vendor's marketing, the tiers are defined by IES and have nothing to do with NCII's rubric. Two systems, two units, one piece of sales copy.

If this is the work you do


Illuminate keeps a list of educational therapists, learning specialists and executive function coaches who want the practitioner-facing material rather than the parent-facing version, and who get referrals through us when a family in their area needs the kind of work they do.


Recent Posts

See All
Standard Error of Measurement: What That Band Means

Every score report prints a confidence interval and almost no practitioner acts on it. What the standard error of measurement is, why the band's width is a software setting rather than a property of t

 
 
 

Comments


bottom of page