Methods For Improving Test Efficiency And Accuracy In A Computer Adaptive Test (cat)

Abstract

A method for test item selection is provided that includes a computer implemented test battery having at least two or more sections with a plurality of test items. An ability estimate is calculated from an earlier section(s) of the at least two or more sections and an initial item and subsequent items for a subsequent section are selected from the plurality of test items based upon the ability estimate(s) from the earlier section(s). Use of a more informative initial ability estimate in the item selection process can improve interim ability estimation accuracy and item selection efficiency while keeping item exposure and item usage rates at acceptable levels.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority under 35 U.S.C. .sctn.119 to provisional application Ser. No. 61/912,774, filed Dec. 6, 2013, which is hereby incorporated in its entirety.

BACKGROUND

[0002] I. Field of the Disclosure

[0003] The present disclosure relates to computer adaptive testing. More specifically, but not exclusively, the present disclosure relate to methods for improving ability estimation accuracy and item selection efficiency in a Computer Adaptive Test (CAT).

[0004] II. Description of the Prior Art

[0005] In educational assessments, a test battery is usually composed of several related sections based on content categories. The inter-relationships among different sections can be used to improve the test efficiency in a Computer Adaptive Test (CAT) scenario.

[0006] Therefore, a primary object, feature, or advantage of the present disclosure uses an examinee's ability estimate from an earlier section to inform the selection of the initial item and the subsequent items in a later section.

[0007] To date, a number of studies have been conducted on selecting the initial item based on an examinee's scores from earlier related tests using a variety of methods. However, none of the methods used in these studies have provided satisfactory results. Furthermore, using prior ability estimates to drive the starting point for a subsequent test section has not been widely used and practiced.

[0008] Therefore, it is another object, feature, or advantage of the present disclosure to provide a method for using the ability estimates from a previous test or test section(s) as prior information to inform the selection of the initial item of a subsequent test or section of a test.

[0009] Another object, feature, or advantage of the present disclosure is to improve methods for item selection and ability estimation in a CAT for accurately measuring an examinee's ability.

[0010] A still further object, feature, or advantage of the present disclosure is to provide methods for initial item selection based on ability estimates from a previous section.

[0011] One or more of these and/or other objects, features or advantages of the present disclosure will become apparent from the specification and claims that follow.

SUMMARY

[0012] The present disclosure improves test efficiency and accuracy in a Computer Adaptive Test (CAT).

[0013] One exemplary method is for test item selection. One example of such includes a computer and a computer implemented test battery having a plurality of test sections. Each test section has a set of test items selected from a plurality of test items. An item selection process is also provided. At least one section from the plurality of test sections is administered to an examinee using the computer. The computer is configured to receive the examinee's responses to the set of test items in the one section. The item selection process for at least one subsequent test section is informed based on scores from the one test section or previous test sections.

[0014] According to another aspect, a method for test item selection is provided. A computer and computer implemented test battery are included along with at least two or more test sections having a plurality of test items. One test section of the at least two or more test sections is administered to an examinee using the computer. The computer is adapted to receive the examinee's responses to a set of the plurality of test items for the one test section. An initial ability estimate for the examinee's responses to the set of the plurality of test items in the one test section is calculated. One or more test items from the plurality of test items are selected to include in a subsequent test section to the one test section of the at least two or more test sections based upon the initial ability estimate from at least the one previous test section.

[0015] According to still another aspect, a method for test item selection is provided. The item selection method includes a computer and a computer implemented test battery having at least two or more test sections with a plurality of test items. One section of the at least two or more test sections are administered to an examinee using the computer. The computer receives the examinee's responses to a set of the plurality of test items for the one test section. An initial ability estimate is calculated for the examinee's responses to the set of the plurality of test items in the one test section. The plurality of test items are minimized to a subset of test items based upon the initial ability estimate from at least the one previous test section. A next test item is selected from the subset of test items for the subsequent test section.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Illustrated embodiments of the present disclosure are described in detail below with reference to the attached drawing figures, which are incorporated by reference herein, and where:

[0017] FIG. 1 is a flowchart of a process for item selection in computer adaptive testing in accordance with an illustrative embodiment; and

[0018] FIG. 2 is a block diagram providing an overview of an item selection process for computer adaptive testing in accordance with an illustrative embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] The present disclosure provides for various computer adaptive testing methods. One exemplary method includes a method for item selection and ability estimation in a computer adaptive test. The accuracy and efficiency of a computer adaptive test is improved by making the test more succinct and/or accurate, and thus more effective. What results is a testing platform using a computer adaptive test that can estimate an examinee's ability using the examinee's response information relating to a specific set of operational test items.

I. Item Selection in a Computer Adaptive Test (CAT)

[0020] In educational assessments, a test battery is usually composed of several related sections based on content categories. The inter-relationships among different sections can be used to improve the test efficiency in a computer adaptive testing (CAT) scenario. For example, an examinee's ability estimate from an earlier section can inform the selection of an initial item and the subsequent items in a later section. The methods used for item selection and ability estimation in CAT are essential for accurately measuring an examinee's ability.

[0021] a. Illustrative Embodiments for Item Selection in a CAT for a Test Battery

[0022] FIG. 1 provides an illustrative flowchart diagram for an exemplary process for item selection in CAT for a given test battery. Aspects of the disclosure contemplate interfacing with, displaying and receiving examinee input via a computer adaptive testing platform. A workstation, such as the one illustrated in FIG. 2, may be used to administer a computer implemented test battery. The workstation may be part of a computer network in which any embodiment of the present disclosure may be implemented. As shown, the workstation or computer network (not shown) includes, for example, a server, workstation, scanner, a printer, a datastore, and other linked networks. The computer networks may be configured to provide a communication path for each device of a computer network to communicate with other devices. Additionally, the computer network may be the internet, a public switchable telephone network, a local area network, private wide area network, wireless network and any of the like. In various embodiments, an automated computer adaptive testing module, operated, at least in part, by an operating protocol, may be executed on a server and/or workstation, similar to the workstation shown in FIG. 2. For example, in one embodiment or aspect of the application, the server may be configured to execute a computer adaptive testing module using an operating protocol individually or co-operatively with one or more workstations. The scanner may be configured to scan textual content and output the content in a computer readable format. Additionally, the printer may be configured to output the content to a print media, such as paper. Furthermore, data associated with any one of the computer adaptive testing modules, test script administration process or interim ability estimation process may be stored on the datastore. The datastore may additionally be configured to receive and/or forward some or all of the stored data. Moreover, in yet another aspect, some or all of the computer network or workstation may be subsumed within a single device. Although a workstation is depicted in FIG. 2, it is understood that the disclosure is not limited to operation within a computer workstation, but rather, the disclosure may be practiced in or on any suitable computer network or electronic device. Accordingly, the workstation depicted in FIG. 2 is for illustrative purposes only and thus is not meant to limit the present disclosure in any respect.

[0023] Using, for example, a test script administration process or application, a section or item selector is used to select a first test section from a test battery. The selected test section may be displayed using one of the aforementioned interface pieces or other like electronic device, whereby for example a plurality of test items from a first selected section are displayed. Through a workstation, computer network, or other like electronic device, examinees' input, answers or responses are received in response to the plurality of test items from a first selected section of a test battery being presented using a test scripted administration process or application. Operably configured on a workstation, such as the one illustrated in FIG. 2, a computer network or other like electronic device is an interim ability estimator using one or more estimation methods for calculating an examinee's ability estimate for the first selected section of test items for a test battery. In the following description, several estimation methods are discussed, which the interim ability estimator application or module may use to calculate an examinee's ability estimate for a first selected section of a plurality of test items in a test battery. One or more operating protocols on the workstation shown in FIG. 2, or a computer network, or another like electronic device may be used to inform selection of an initial test item for next test section in the test battery based upon an examinee's ability estimate from the first or previous selected section.

[0024] One comparative method selects a first item of a later section randomly from a pool of items most informative near the initial fixed theta value and then uses one or more estimation methods to estimate an examinee's interim ability. For example, the maximum likelihood (ML) estimator can be used to estimate an examinee's interim ability upon responding to each item, which was then used for selecting next items in subsequent sections of the test battery. Other exemplary methods herein use a Bayesian approach. The ability estimates from the first section may be used as prior information to select the first item of a subsequent or second section in a test battery. The expected a posteriori (EAP) estimation method can also be used at every step to obtain an examinee's interim ability which may then be used for selecting subsequent items in subsequent sections of a test battery.

[0025] b. Illustrative Application(s)

[0026] In practice, the existing CAT programs involving multiple test sections usually use the same initial ability for all examinees without considering the interrelatedness between the test sections. This practice is limited because it involves arbitrary choices of ability values and items for each examinee. If the initial ability estimate is away from the true ability of the examinee, the CAT procedure will have an inaccurate start. For example, starting the test with a mid-level difficulty item for a high- or low-proficiency examinee would take longer to arrive at an accurate estimate of his/her ability. This, to a large degree, affects the efficiency of item selection. Moreover, initialization at the same ability estimate for all examinees leads to first items in the test that are always chosen from the same subset in the pool. Hence, these items are overexposed (Van der Linden & Pashley, 2010).

[0027] In the present disclosure, using an initial ability estimate predicted by each examinee's performance on earlier related test section(s) provides an individualized initialization for adaptive tests. That is, different examinees will start at different initial ability estimates. Further, using an initial theta predicted by each examinee's performance on earlier section(s) provides an individualized prior that is located near each examinee's true ability. The use of a predicted initial theta and an individualized prior that is continuously improved during the test using additional information obtained from the individual examinee will improve item selection and speed up convergence of the ability estimates. Finally, the present invention helps improve item exposure. The empirical initialization of the test entails leads to a variable entry point to the pool, and hence offers a more even exposure of its items (Van der Linden & Pashley, 2010). The item selection and ability estimation procedure proposed in the present invention can be integrated with the current algorithm for delivering the CAT tests.

[0028] II. Other Embodiments and Variations

[0029] The present disclosure is not to be limited to the particular embodiments described herein. In particular, the present disclosure contemplates numerous variations in the type of ways in which embodiments of the disclosure may be applied to computer adaptive testing. The foregoing description has been presented for purposes of illustration and description. It is not intended to be an exhaustive list or limit any of the disclosure to the precise forms disclosed. It is contemplated that other alternatives or exemplary aspects that are considered are included in the disclosure. The description is merely examples of embodiments, processes or methods of the invention. For example, the methods used to select an initial item in a subsequent section based upon ability estimates from a previous section are not limited to those disclosed herein. It is understood that any other modifications, substitutions, and/or additions may be made, which are within the intended spirit and scope of the disclosure. For the foregoing, it can be seen that the disclosure accomplishes at least all of the intended objectives.

Claims

1. A method of test item selection comprising; providing a computer and a computer implemented test battery comprising: a plurality of test sections in the computer implemented test battery, each test section having a set of test items selected from a plurality of test items; an item selection process; administering one test section from the plurality of test sections to an examinee using the computer, wherein the computer receives the examinee's responses to the set of test items in the one test section; and informing the item selection process for at least one subsequent test section based on scores from the one test section or previous test sections.

2. The method of claim 1 further comprising: calculating an initial ability estimate from the examinee's responses to the set of test items in the one test section.

3. The method of claim 2 further comprising: basing the item selection process for the at least one subsequent test section on the initial ability estimate.

4. The method of claim 1 wherein the computer implemented test battery comprises a computer adaptive test battery further comprising related tests or test sections.

5. The method of claim 1 further comprising: matching item difficulty in the at least one subsequent section with an estimated examinee ability from the one test section or previous test sections.

6. The method of claim 1 further comprising: improving interim ability estimation accuracy and item selection efficiency by informing the item selection process with ability estimates from the one test section or previous test sections.

7. A method of test item selection comprising; providing a computer and a computer implemented test battery comprising at least two or more test sections having a plurality of test items; administering one section of the at least two or more test sections to an examinee using the computer, wherein the computer receives the examinee's responses to a set of the plurality of test items for the one section; calculating an initial ability estimate for the examinee's responses to the set of the plurality of test items in the one test section; and selecting one or more test items from the plurality of test items to include in a subsequent test section to the one test section of the at least two or more test sections based upon the initial ability estimate from at least the one previous test section.

8. The method of claim 7 further comprising: providing an item selection process for selecting the one or more test items based upon the initial ability estimate from at least the one previous section.

9. The method of claim 7 wherein the computer implemented test battery comprises a computer adaptive test.

10. The method of claim 8 further comprising: factoring into the item selection process interrelatedness between the at least the one test section and the subsequent test section.

11. The method of claim 7 further comprising: minimizing overexposure and over usage of selected test items from the plurality of test items by basing item selection in the subsequent test section on the ability estimate from at least the one previous section.

12. The method of claim 7 further comprising: minimizing the plurality of test items to a subset of test items based upon the initial ability estimate from at least the one previous section.

13. The method of claim 8 further comprising: improving interim ability estimation accuracy and item selection efficiency by informing the item selection process with ability estimates from the one test section or previous test sections.

14. A method of test item selection comprising; providing a computer and a computer implemented test battery comprising at least two or more test sections having a plurality of test items; administering one section of the at least two or more test sections to an examinee using the computer, wherein the computer receives the examinee's responses to a set of the plurality of test items for the one test section; calculating an initial ability estimate from the examinee's responses to the set of the plurality of test items in the one test section; minimizing the plurality of test items to a subset of test items based upon the initial ability estimate from at least the one previous test section; and selecting a next test item from the subset of test items for the subsequent test section.

15. The method of claim 14 further comprising: adapting the next test item selection to scores from at least the one previous section.

16. The method of claim 14 further comprising: calculating an initial theta for the subsequent test section based on an examinee's ability from at least the one previous test section, wherein the at least one previous test section is related to the subsequent test section.

17. The method of claim 14 further comprising: selecting the next test item from the subset of test items to include in the subsequent test section to the one test section of the at least two or more test sections based upon ability estimates from at least the one previous test section.

18. The method of claim 14 further comprising: factoring interrelatedness between the at least two or more test sections into the next test item selection.

19. The method of claim 14 further comprising: minimizing overexposure and over usage of selected test items from the plurality of test items by basing next test item selection in the subsequent test section on an interim ability estimate arrived at from the initial ability estimate.

20. The method of claim 7 further comprising: improving interim ability estimation accuracy and item selection efficiency by informing the next test item selection with an interim ability estimates resulting from the initial ability estimates informed by ability estimates obtained from one test section or previous test sections.