Suppose you fit Samejima’s Continous Response Model (CRM) to several reading passages (or MAZE forms) and estimated item parameters for all passages/forms in a way that they are all on the same scale. There are a couple of ways you can do this (concurrent calibration, chain equating, or more traditional methods, as explained in this paper by Shojima). You can read this post to learn more about this model, how to estimate its parameters, and what these parameters mean.
So, let’s assume that the parameters for all reading passages are on the same scale, and you have 20 of these reading passages (Form 1, Form 2, …, Form 20).
| \(a\) | \(b\) | \(\alpha\) | |
|---|---|---|---|
| Form 1 | 1.246 | -1.068 | 0.965 |
| Form 2 | 1.469 | -0.218 | 0.990 |
| Form 3 | 3.594 | -1.026 | 0.937 |
| Form 4 | 1.708 | -0.729 | 1.108 |
| Form 5 | 1.759 | -0.625 | 1.060 |
| Form 6 | 3.887 | -1.687 | 0.944 |
| Form 7 | 2.076 | 0.838 | 0.980 |
| Form 8 | 0.876 | 0.153 | 0.977 |
| Form 9 | 1.169 | -1.138 | 1.039 |
| Form 10 | 1.319 | 1.254 | 0.996 |
| Form 11 | 3.041 | 0.426 | 1.013 |
| Form 12 | 1.974 | -0.295 | 0.999 |
| Form 13 | 2.015 | 0.895 | 0.998 |
| Form 14 | 1.743 | 0.878 | 1.068 |
| Form 15 | 1.249 | 0.822 | 0.989 |
| Form 16 | 4.029 | 0.689 | 1.076 |
| Form 17 | 2.115 | 0.554 | 0.923 |
| Form 18 | 0.617 | -0.062 | 1.029 |
| Form 19 | 2.341 | -0.306 | 1.006 |
| Form 20 | 1.302 | -0.380 | 1.011 |
Let’s say that Form 4 is administered in the Fall semester, and Form 11 semester is administered in the Spring Semester. The scores from these two reading passages may not be directly comparable because they may differ in their difficulty (\(b\) parameter), and also in other characteristics (\(a\) parameter and \(\alpha\)). The difference in the \(b\) parameter is probably most critical here. It wouldn’t make much sense to compare the ORF scores from Fall and Spring without any equating between the two forms, and practitioners would probably appreciate an easy to understand score conversion table between two reading passages. I don’t know much about CBM-Reading literature. I am aware that there are already methods used and published to create such a table using traditional equating methods (e.g., Albano and Rodriguez, 2012). There may also be other works published. Here, I would like to accomplish the same thing using an IRT approach when Samejima’s CRM is used to scale many reading passages.
Form 4 and Form 11 has the following item parameters:
\[(a_4,b_4,\alpha_4)=(1.708,-0.729,1.108)\] \[(a_{11},b_{11},\alpha_{11})=(3.041,0.426,1.013)\]
Also, let’s assume that the maximum possible ORF score is 250 for both forms.
We need to remember a few formulas for CRM. In this model, the probability of an individual with a latent parameter \(\theta_i\) obtaining a score of \(x\) or higher on an item with three parameters (\(a\), \(b\), and \(\alpha\)) is specified as,
\[\begin{equation} P_{ij}(X\geq x|\theta_{i},a_j,b_j,\alpha_j)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{v}{e^{-\frac{t^2}{2}}dt} \tag{1} \end{equation}\]
\[\begin{equation} v=a_j\times (\theta_i-b_j-\frac{1}{\alpha_j}ln\frac{x_{ij}}{k_j-x_{ij}}) \tag{2} \end{equation}\]
where \(k_j\) is the maximum possible score for the item.
\(x_{ij}^*\) is a nonlinear transformation of the observed continous score,
\[\begin{equation} x_{ij}^* = ln(\frac{x_{ij}}{k_j-x_{ij}}), \tag{3} \end{equation}\]
and its conditional PDF is a normal distribution with a mean of \(\alpha_j(\theta_i - b_j)\) and standard deviation of \(\alpha_j/a_j\).
\[\begin{equation} x^* \sim N(\alpha_j(\theta_i - b_j),\alpha_j/a_j). \tag{4} \end{equation}\]
Let’s also remember that the maximum likelihood estimate of \(\theta\) in CRM has a closed formula and can easily be computed based on the item parameters.
\[\begin{equation} \hat\theta_i = \frac{\sum_{j=1}^{n}a_j^2(b_j+\frac{x_{ij}^*}{\alpha_j})}{\sum_{j=1}^{n}a_j^2} \tag{5} \end{equation}\]
Now, given that what we know, let’s do a quick exercise. Let’s say a student got a score of 120 from Form 4. What would be the expected score if we administered Form 11 to the same student?
Step 1: Transform the observed score from Form 4 (Eq. 3).
\[x_{ij}^* = ln(\frac{120}{250-120})=-0.080\]
Step 2: Given the item parameters for Form 4 and the transformed score, obtain an estimate of latent ability, \(\hat\theta_i\) (Eg. 5). I know it will sound absurd for those who are familiar with other IRT models to estimate latent trait using a single item. We can do it in CRM. Note that when there is only one item, the ML estimate simplifies. So, if you write Eq.5 for a single item, the estimate is equal to \[\hat\theta_i = b_j + \frac{x_{ij}^*}{\alpha_j}.\]
Alternatively, you can think about it as the following. For a given \(x_{ij}^*\), we know its expected value is \(\alpha_j(\theta_i - b_j)\). So, we can get the same equation if we solve the following equation for \(\theta\),
\[x_{ij}^* = \alpha_j(\theta_i - b_j).\] Either way, we can get the estimate of \(\theta\) based on the Form 4 score.
\[\hat\theta_i = -0.729 + \frac{-0.08}{1.108}= -0.801 \]
Step 3: Given the theta estimate in Step 2 and item parameters for Form 11, compute the expected transformed score (mean of the conditional distribution) in Form 11.
\[x_{ij}^* = \alpha_j(\theta_i - b_j) = 1.013*(-0.801-0.426) = -1.243\]
Step 4: Transform the expected score in Form 11 back to its original scale.
\[ x_{ij} = \frac{e^{x_{ij}^*}*k_j}{e^{x_{ij}^*}+1}\]
\[ x_{ij} = \frac{e^{-1.243}*250}{e^{-1.243}+1}=55.98\]
This indicates that a student who received a score of 120 from Form 4 is expected to receive a score of 55.98 (~ 56) from Form 11. This is not surprising as Form 11 is more difficult than Form 4.
Below is an R code to produce some sort of conversion table for all possible scores from 1 to 249. You can change the item parameters and maximum score to get the conversion table for any hypothetical two forms.
# Generate CRM item parameters
set.seed(123)
a <- rlnorm(20,.5,.5)
b <- rnorm(20,0,1)
alpha <- rnorm(20,1,.05)
ipar <- cbind(a,b,alpha)
colnames(ipar) = NULL
rownames(ipar) <- paste0("Form ",1:20)
#################################
# two forms to be equated
form1 = 4
form2 = 11
ipar1 <- ipar[form1,]
ipar2 <- ipar[form2,]
max.score = 250
raw <- as.data.frame(as.matrix(1:249))
raw$Form2 = NA
raw$Theta = NA
colnames(raw)[1] <- "Form1"
for(i in 1:nrow(raw)){
z1 = log(raw[i,1]/(max.score-raw[i,1]))
theta = (z1/ipar1[3])+ipar1[2]
z2 = ipar2[3]*(theta-ipar2[2])
raw[i,2]=(exp(z2)*max.score)/(exp(z2)+1)
raw[i,3]= theta
}
round(raw,2)
Form1 Form2 Theta
1 1 0.50 -5.71
2 2 0.95 -5.08
3 3 1.37 -4.71
4 4 1.79 -4.44
5 5 2.20 -4.24
6 6 2.60 -4.07
7 7 3.00 -3.93
8 8 3.40 -3.80
9 9 3.79 -3.69
10 10 4.18 -3.60
11 11 4.57 -3.51
12 12 4.96 -3.42
13 13 5.35 -3.35
14 14 5.74 -3.28
15 15 6.13 -3.21
16 16 6.52 -3.15
17 17 6.90 -3.09
18 18 7.29 -3.04
19 19 7.68 -2.98
20 20 8.06 -2.93
21 21 8.45 -2.88
22 22 8.84 -2.84
23 23 9.23 -2.79
24 24 9.62 -2.75
25 25 10.01 -2.71
26 26 10.40 -2.67
27 27 10.79 -2.63
28 28 11.18 -2.60
29 29 11.57 -2.56
30 30 11.97 -2.53
31 31 12.36 -2.49
32 32 12.76 -2.46
33 33 13.15 -2.43
34 34 13.55 -2.40
35 35 13.95 -2.37
36 36 14.35 -2.34
37 37 14.75 -2.31
38 38 15.16 -2.28
39 39 15.56 -2.25
....We can also plot the expected score for both forms as a function of theta.
