Equating Oral Reading Fluency Scores from Reading Passages

item response theory continuous response model equating R 2020

A non-peer reviewed opinion about how one can equate oral reading fluency scores from two reading passages with different difficulty levels using Samejima’s Continuous Response Model.

Cengiz Zopluoglu (University of Miami)
05-05-2020

Suppose you fit Samejima’s Continous Response Model (CRM) to several reading passages (or MAZE forms) and estimated item parameters for all passages/forms in a way that they are all on the same scale. There are a couple of ways you can do this (concurrent calibration, chain equating, or more traditional methods, as explained in this paper by Shojima). You can read this post to learn more about this model, how to estimate its parameters, and what these parameters mean.

So, let’s assume that the parameters for all reading passages are on the same scale, and you have 20 of these reading passages (Form 1, Form 2, …, Form 20).

\(a\) \(b\) \(\alpha\)
Form 1 1.246 -1.068 0.965
Form 2 1.469 -0.218 0.990
Form 3 3.594 -1.026 0.937
Form 4 1.708 -0.729 1.108
Form 5 1.759 -0.625 1.060
Form 6 3.887 -1.687 0.944
Form 7 2.076 0.838 0.980
Form 8 0.876 0.153 0.977
Form 9 1.169 -1.138 1.039
Form 10 1.319 1.254 0.996
Form 11 3.041 0.426 1.013
Form 12 1.974 -0.295 0.999
Form 13 2.015 0.895 0.998
Form 14 1.743 0.878 1.068
Form 15 1.249 0.822 0.989
Form 16 4.029 0.689 1.076
Form 17 2.115 0.554 0.923
Form 18 0.617 -0.062 1.029
Form 19 2.341 -0.306 1.006
Form 20 1.302 -0.380 1.011

Let’s say that Form 4 is administered in the Fall semester, and Form 11 semester is administered in the Spring Semester. The scores from these two reading passages may not be directly comparable because they may differ in their difficulty (\(b\) parameter), and also in other characteristics (\(a\) parameter and \(\alpha\)). The difference in the \(b\) parameter is probably most critical here. It wouldn’t make much sense to compare the ORF scores from Fall and Spring without any equating between the two forms, and practitioners would probably appreciate an easy to understand score conversion table between two reading passages. I don’t know much about CBM-Reading literature. I am aware that there are already methods used and published to create such a table using traditional equating methods (e.g., Albano and Rodriguez, 2012). There may also be other works published. Here, I would like to accomplish the same thing using an IRT approach when Samejima’s CRM is used to scale many reading passages.

Form 4 and Form 11 has the following item parameters:

\[(a_4,b_4,\alpha_4)=(1.708,-0.729,1.108)\] \[(a_{11},b_{11},\alpha_{11})=(3.041,0.426,1.013)\]

Also, let’s assume that the maximum possible ORF score is 250 for both forms.

We need to remember a few formulas for CRM. In this model, the probability of an individual with a latent parameter \(\theta_i\) obtaining a score of \(x\) or higher on an item with three parameters (\(a\), \(b\), and \(\alpha\)) is specified as,

\[\begin{equation} P_{ij}(X\geq x|\theta_{i},a_j,b_j,\alpha_j)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{v}{e^{-\frac{t^2}{2}}dt} \tag{1} \end{equation}\]

\[\begin{equation} v=a_j\times (\theta_i-b_j-\frac{1}{\alpha_j}ln\frac{x_{ij}}{k_j-x_{ij}}) \tag{2} \end{equation}\]

where \(k_j\) is the maximum possible score for the item.

\(x_{ij}^*\) is a nonlinear transformation of the observed continous score,

\[\begin{equation} x_{ij}^* = ln(\frac{x_{ij}}{k_j-x_{ij}}), \tag{3} \end{equation}\]

and its conditional PDF is a normal distribution with a mean of \(\alpha_j(\theta_i - b_j)\) and standard deviation of \(\alpha_j/a_j\).

\[\begin{equation} x^* \sim N(\alpha_j(\theta_i - b_j),\alpha_j/a_j). \tag{4} \end{equation}\]

Let’s also remember that the maximum likelihood estimate of \(\theta\) in CRM has a closed formula and can easily be computed based on the item parameters.

\[\begin{equation} \hat\theta_i = \frac{\sum_{j=1}^{n}a_j^2(b_j+\frac{x_{ij}^*}{\alpha_j})}{\sum_{j=1}^{n}a_j^2} \tag{5} \end{equation}\]

Now, given that what we know, let’s do a quick exercise. Let’s say a student got a score of 120 from Form 4. What would be the expected score if we administered Form 11 to the same student?

Step 1: Transform the observed score from Form 4 (Eq. 3).

\[x_{ij}^* = ln(\frac{120}{250-120})=-0.080\]

Step 2: Given the item parameters for Form 4 and the transformed score, obtain an estimate of latent ability, \(\hat\theta_i\) (Eg. 5). I know it will sound absurd for those who are familiar with other IRT models to estimate latent trait using a single item. We can do it in CRM. Note that when there is only one item, the ML estimate simplifies. So, if you write Eq.5 for a single item, the estimate is equal to \[\hat\theta_i = b_j + \frac{x_{ij}^*}{\alpha_j}.\]

Alternatively, you can think about it as the following. For a given \(x_{ij}^*\), we know its expected value is \(\alpha_j(\theta_i - b_j)\). So, we can get the same equation if we solve the following equation for \(\theta\),

\[x_{ij}^* = \alpha_j(\theta_i - b_j).\] Either way, we can get the estimate of \(\theta\) based on the Form 4 score.

\[\hat\theta_i = -0.729 + \frac{-0.08}{1.108}= -0.801 \]

Step 3: Given the theta estimate in Step 2 and item parameters for Form 11, compute the expected transformed score (mean of the conditional distribution) in Form 11.

\[x_{ij}^* = \alpha_j(\theta_i - b_j) = 1.013*(-0.801-0.426) = -1.243\]

Step 4: Transform the expected score in Form 11 back to its original scale.

\[ x_{ij} = \frac{e^{x_{ij}^*}*k_j}{e^{x_{ij}^*}+1}\]

\[ x_{ij} = \frac{e^{-1.243}*250}{e^{-1.243}+1}=55.98\]

This indicates that a student who received a score of 120 from Form 4 is expected to receive a score of 55.98 (~ 56) from Form 11. This is not surprising as Form 11 is more difficult than Form 4.

Below is an R code to produce some sort of conversion table for all possible scores from 1 to 249. You can change the item parameters and maximum score to get the conversion table for any hypothetical two forms.

# Generate CRM item parameters

set.seed(123)

a     <- rlnorm(20,.5,.5)
b     <- rnorm(20,0,1)
alpha <- rnorm(20,1,.05)

ipar <- cbind(a,b,alpha)

colnames(ipar) = NULL
rownames(ipar) <- paste0("Form ",1:20)

#################################

# two forms to be equated

  form1 = 4
  form2 = 11

ipar1 <- ipar[form1,]
ipar2 <- ipar[form2,]
max.score = 250

raw <- as.data.frame(as.matrix(1:249))
raw$Form2 = NA
raw$Theta = NA
colnames(raw)[1] <- "Form1"

for(i in 1:nrow(raw)){
  z1 = log(raw[i,1]/(max.score-raw[i,1]))
  theta = (z1/ipar1[3])+ipar1[2]
  z2 = ipar2[3]*(theta-ipar2[2])
  raw[i,2]=(exp(z2)*max.score)/(exp(z2)+1)
  raw[i,3]= theta
}

round(raw,2)
    Form1  Form2 Theta
1       1   0.50 -5.71
2       2   0.95 -5.08
3       3   1.37 -4.71
4       4   1.79 -4.44
5       5   2.20 -4.24
6       6   2.60 -4.07
7       7   3.00 -3.93
8       8   3.40 -3.80
9       9   3.79 -3.69
10     10   4.18 -3.60
11     11   4.57 -3.51
12     12   4.96 -3.42
13     13   5.35 -3.35
14     14   5.74 -3.28
15     15   6.13 -3.21
16     16   6.52 -3.15
17     17   6.90 -3.09
18     18   7.29 -3.04
19     19   7.68 -2.98
20     20   8.06 -2.93
21     21   8.45 -2.88
22     22   8.84 -2.84
23     23   9.23 -2.79
24     24   9.62 -2.75
25     25  10.01 -2.71
26     26  10.40 -2.67
27     27  10.79 -2.63
28     28  11.18 -2.60
29     29  11.57 -2.56
30     30  11.97 -2.53
31     31  12.36 -2.49
32     32  12.76 -2.46
33     33  13.15 -2.43
34     34  13.55 -2.40
35     35  13.95 -2.37
36     36  14.35 -2.34
37     37  14.75 -2.31
38     38  15.16 -2.28
39     39  15.56 -2.25
....

We can also plot the expected score for both forms as a function of theta.

Citation

For attribution, please cite this work as

Zopluoglu (2020, May 5). Cengiz Zopluoglu: Equating Oral Reading Fluency Scores from Reading Passages. Retrieved from https://github.com/czopluoglu/website/tree/master/docs/posts/crm-equating/

BibTeX citation

@misc{zopluoglu2020equating,
  author = {Zopluoglu, Cengiz},
  title = {Cengiz Zopluoglu: Equating Oral Reading Fluency Scores from Reading Passages},
  url = {https://github.com/czopluoglu/website/tree/master/docs/posts/crm-equating/},
  year = {2020}
}