U.S. patent application number 12/222385 was filed with the patent office on 2009-06-25 for integrated assessment system for standards-based assessments.
Invention is credited to Brian Benson, John Robert Bergan, Luke Ferris, Michael Rattee.
Application Number | 20090162827 12/222385 |
Document ID | / |
Family ID | 40789087 |
Filed Date | 2009-06-25 |
United States Patent
Application |
20090162827 |
Kind Code |
A1 |
Benson; Brian ; et
al. |
June 25, 2009 |
Integrated assessment system for standards-based assessments
Abstract
A computer integrated assessment system for standards-based
assessments wherein the assessments conform to recorded standards,
the system includes algorithms for directing the generation of a
plan of a series of customized assessments aligned to respective
selected standards of the recorded standards wherein each of the
customized assessments are set for different times within a period
of time; and algorithms for enabling a user to generate a test in
one of the customized assessments wherein one of the selected
standards of the one customized assessment is displayed to the user
during generation of questions for the test. During generation of
tests, repeating of questions or questions on selected subject
matter can be prevented. Individuals or groups can be selected to
the customized assessments. Accuracy in grading of answer sheets by
scanning is improved by an algorithm determining the lightest and
darkest answer mark to determine the intended answer. Additionally
student handheld devices can be used to answer questions in test
with automatic grading or scoring.
Inventors: |
Benson; Brian; (Tucson,
AZ) ; Bergan; John Robert; (Tucson, AZ) ;
Ferris; Luke; (Tucson, AZ) ; Rattee; Michael;
(Tucson, AZ) |
Correspondence
Address: |
DONALD W. MARKS
600 SOUTH 29TH ROAD
ARLINGTON
VA
22202
US
|
Family ID: |
40789087 |
Appl. No.: |
12/222385 |
Filed: |
August 7, 2008 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
60963675 |
Aug 7, 2007 |
|
|
|
60963676 |
Aug 7, 2007 |
|
|
|
Current U.S.
Class: |
434/350 ;
434/354 |
Current CPC
Class: |
G09B 7/02 20130101; G06F
16/285 20190101 |
Class at
Publication: |
434/350 ;
434/354 |
International
Class: |
G09B 7/00 20060101
G09B007/00 |
Claims
1. A computer integrated assessment system for standards-based
assessments wherein the assessments conform to recorded standards,
the system comprising: means for directing the generation of a plan
of a series of customized assessments aligned to respective
selected standards of the recorded standards wherein each of the
customized assessments are set for different times within a period
of time; and means for enabling a user to generate a test in one of
the customized assessments wherein one of the selected standards of
the one customized assessment is displayed to the user during
generation of questions for the test.
2. A computer integrated assessment system as defined in claim 1
further comprising: means for preventing repeating a question in a
subsequent assessments.
3. A computer integrated assessment system as defined in claim 1
further comprising: means for allowing review of tests for each
customized assessment by respective selected participants.
4. A computer integrated assessment system as defined in claim 1
further including the ability to scan test answer sheets wherein
the lightest and darkest answer marks are determined and used to
determine the marked answer for question.
5. A computer integrated assessment system as defined in claim 1
further including means to administer tests using student handheld
input devices.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is related to U.S. Pat. Nos. 6,322,366,
6,468,085 and 7,065,516 and U.S. patent application Ser. No.
11/009,708, all of which patents and application are expressly
incorporated in their entirety herein. This application claims
benefit under 35 U.S.C. .sctn. 119(e)(1) of U.S. Provisional Patent
Applications 60/963,675 and 60/963,676 which are expressly
incorporated in their entirety herein. Additionally the disclosure
in U.S. Published Patent Application 2003/00044762 is expressly
incorporated in its entirety herein.
BACKGROUND OF THE INVENTION
[0002] A major challenge facing educational programs in the
21.sup.st century is to promote learning aimed at the achievement
of valued goals or standards. In an effort to assist educators to
meet this challenge, an educational management system has been
designed to assist programs to promote goal-directed
standards-based learning for example as described in U.S. Pat. No.
6,322,366 and No. 6,468,085. The present patent application details
additional innovations that enhance the usefulness of the system
for learners involved in a variety of standards-based educational
programs. A particularly important group of such learners is
comprised of elementary and secondary school students receiving
instruction aimed at the achievement of federal, state, and local
standards
[0003] In one educational management system, instruction to promote
goal-directed learning is informed by assessment information
indicating the capabilities that a learner has acquired and those
that the learner will be ready to learn in the future. Item
Response Theory (IRT) is used to estimate the probability that a
learner will be ready to acquire capabilities reflecting goals that
have not yet been mastered. There is room for innovations that
enhance the construction of assessments and the use of assessment
information to inform goal-directed standards-based learning.
BRIEF SUMMARY OF THE INVENTION
[0004] In the educational management system, instruction to promote
goal-directed learning is informed by assessment information
indicating the capabilities that a learner has acquired and those
that the learner will be ready to learn in the future. Item
Response Theory (IRT) is used to estimate the probability that a
learner will be ready to acquire capabilities reflecting goals that
have not yet been mastered. There are included innovations that
enhance the construction of assessments and the use of assessment
information to inform goal-directed standards-based learning.
[0005] The present invention can be summarized in an integrated
assessment system for standards-based education. The system
including: (a) Assessment planning within a Benchmark Planner,
allowing for a series of customized assessments aligned to
standards and delivered on a schedule determined by the user. (b)
Automatic benchmark test generation wherein these tests are: (i)
Part of an overall benchmark plan that may cover up to an entire
academic year. (ii) Able to restrict specific items from being
re-tested if included in an earlier benchmark test during that
planning period. (iii) Alignment of test items in accordance with
the specifications used in the selected benchmark plan. (iv)
Capable of optimizing overall printed test length by associating a
flexible number of items with a block of text to reduce overall
test length and minimizing white space. (c) Benchmark test review
wherein the review process contains: (i) Agency review of tests
constructed using the benchmark planner. (ii) Multi-phase review,
allowing distinct groups to participate in the review process.
(iii) Ability to align the items with the associated standard
within the review process. (iv) Use of distinct phases of review,
e.g. Not Reviewed, Accept, and Replace. (d) Manual construction of
tests using Test Builder, including: (i) The ability to hand-enter
test questions with text, equations and images, import printable
tests, or search a pre-populated test item bank. (ii) Ability to
create a standards-based assessment by aligning items with a
standard, or to circumvent standard alignment. (iii) Multi-phase
construction, e.g. Construction phase, tryout phase, item review,
and publication phase. (e) Offline test administration including:
(i) The ability to scan answer sheets using optical scanning
technology using proprietary answer sheets, or to print and
subsequently scan answer sheets on plain paper that have been
filled out by the student. (ii) A scanner controlled by a
client-side computer connected to the Internet. (iii) Software
installed on the client computer, used to connect to the student
assessment database to upload assessment item responses. (iv)
Bar-coded answer sheets linking the answer sheet to a specific test
ID. (v) An algorithm scoring lightest and darkest mark on an answer
sheet, to enhance accuracy in determining what constitutes a marked
response. (vi) The ability for the client to automatically submit
images of scanned answer sheets in the event of a processing
problem. (vii) Ability to use both offline and online
administration within a single assessment. (f) Online test
administration, including: (i) Test entry through an online student
center using dual password identification at the student login and
test login levels, or through classroom administration using
handheld input devices. (ii) Real-time progress updates for the
student during test administration. (iii) Ability to administer
tests with students using either individual computers or hand-held
response pads, obviating the need to have a computer for each
student in the testing environment. Test administration using
response pads further including: (1) Software installed on a single
client computer, connected to a response pad receiver. (2) A
central display device, either an instructor's computer or
whiteboard, to display test questions, or handheld devices capable
of displaying test questions for the student. (3) Handheld response
pads using wireless technology to transfer student responses to the
client-side application managing the student response pads. (4)
Automated transfer of data from the client-side application
managing the student response pads to the assessment database. (iv)
Test questions and answers transferred to the client computer after
each response is saved, reducing the risk of data loss in the event
of hardware failure. (g) Test monitoring capability, with the
ability for teachers to view individual student item responses, and
notation of correct/incorrect responses by question. (g)
Combinatorial assessments, where standards mastery may be
determined from diverse data sources such as class assignments and
online/offline assessments. (h) Combinatorial assessments make it
possible to use item parameters along with a continuous ability
score to compute the probability that a student will achieve any of
a series of goals in a scale comprised of a set of goals.
[0006] The invention also may be summarized in an integrative data
analysis system to promote standards mastery. The system includes:
(a) Innovations making it possible to: (i) Combine multiple tests
into a single assessment, (ii) Combine parts of tests to make a new
test, and (iii) combine information from tests, class assignments,
and other data sources into one scale. (b) The ability to estimate
the test score needed to achieve standards mastery and to identify
objectives to be mastered to achieve the required test score. (i)
Required learning objective estimates are linked to the Benchmark
Planner. (c) A Risk Assessment initiative to determine whether or
not standards have been achieved. The initiative includes: (i)
Predictive abilities to determine which students are on course to
meet standards and which students are at risk for not meeting
standards. (ii) A model basing estimates of the risk that students
will not meet standards on data gathered during a previous year.
(iii) Assessment of the validity of estimates based on computation
of new estimates when new ata have been collected on all measures
involved in the assessment. (d) Use of multiple tests to determine
standards mastery. Tests used to determine mastery may include
multiple test types (e.g. benchmark, formative, and/or State
test).
BRIEF DESCRIPTION OF THE DRAWINGS
[0007] FIGS. 1, 4, 7, 9, 11, 13, 15, 17, 20, 23, 28, 29, 31, 33,
35, 37, 39, 41, 43, 45, 47, 49, 56, 58, 60, 63, 65 and 69 are step
diagrams setting forth the operation and construction of the
integrated assessment system of the present invention.
[0008] FIGS. 2, 3, 5, 6, 8, 10, 12, 14, 16, 18, 19, 21, 22, 24, 25,
26, 27, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 51, 52, 53, 54,
55, 59, 61, 62, 64, 66, 67, 69 and 70 are illustrations of computer
screens of the integrated assessment system of the present
invention setting forth various steps the above step diagrams.
DETAILED DESCRIPTION OF THE INVENTION
[0009] Standards-based education requires that instruction be
targeted toward the achievement of shared goals articulated in
standards. For example, currently educational agencies across the
nation are targeting instruction toward the achievement of shared
goals reflected in state and local standards. The effective pursuit
of shared goals in a given educational agency (e.g., school
district or charter school) calls for a coordinated agency-wide
effort designed to insure consistency and continuity in instruction
and assessment related to instruction. The desired consistency and
continuity are typically reflected in the agency's curriculum and
assessment plans, which spell out the goals of instruction and the
sequence in which those goals will be pursued and assessed.
[0010] An integrated assessment system is developed to support the
construction, scheduling, and administration of customized
assessments based on input from large numbers of teachers and
administrators who are working together toward the achievement of
shared goals. The system supports agency-wide assessment planning
covering multiple assessments during the school year. It integrates
planning with test construction, agency-wide assessment review, and
agency-wide and classroom based test scheduling. The system
supports agency-wide online and/or offline test administration and
automated and/or manual scoring.
Assessment Planning with Benchmark Planner
[0011] Two kinds of assessment planning occur in standards-based
education. The most familiar involves planning to meet the needs of
individual students in the classroom setting. For example, a
teacher may construct a brief classroom quiz to measure specific
capabilities currently targeted for instruction. Performance on the
quiz may be used as a basis for recommending specific instructional
activities designed to promote mastery of the targeted skills.
[0012] A second form of planning is long-range planning involving
multiple assessments designed to assess student performance related
to curriculum plans coordinating instruction across multiple grades
for an entire school year. Technology supporting this type of
planning has been lacking. An innovative tool called Assessment or
Benchmark Planner has been developed to support long-range planning
involving multiple assessments aligned to standards. The Assessment
or Benchmark Planner makes it possible to plan a series of
customized assessments aligned to standards and to schedule the
delivery of those assessments at successive times across the school
year. The planning process begins with the selection of the year
during which the plan will be in effect, and the subject and
associated standards to which the assessment items called for in
the plan will be aligned. The selected standards are often state
standards. However, agencies have the option of entering their own
local standards using Scale Builder technology. A Plan Transfer
feature allows the user to transfer a plan from a previous year to
the current year. A Copy feature allows the user to make a copy of
a plan. The Transfer and Copy features provide planning continuity
across time and across related planning initiatives.
[0013] Many people typically have a stake in the development of
Benchmark assessments. The Benchmark Planner allows the agency to
initiate the process of giving people a voice in the development
process by specifying individuals who will have the responsibility
of reviewing draft assessments developed in accordance with the
plan. This is accomplished using the Set Test Reviewers feature.
The completed plan can also be printed and disseminated to
interested parties.
[0014] For each benchmark test, the user has the option of
specifying the number of items to be used to assess each standard
(benchmark) to be measured on the test. For example, the user may
indicate that four items should be selected to measure each
standard. A projected delivery date for the assessment is also
automatically recorded for the assessment. The standards
potentially available for assessment are displayed in a series of
check boxes. The user checks the standards to be assessed on each
benchmark. In addition, the user may overwrite the number of items
allotted to each standard. For example, suppose that the user has
indicated that there should be four items for each standard
assessed on the test. The user may overwrite the global
specification of four items. For instance, the user might choose to
include three items to measure achievement related to a particular
standard.
[0015] When the benchmark plan for one or more assessments is
complete, the user may indicate that the plan is complete and save
the plan. The completion of all plans at a given grade level
provides a multi-assessment benchmark plan covering an entire
school year for that grade level. The completion of plans at
multiple grade levels provides a global plan for assessments to be
conducted at all of the selected grade levels.
Test Construction Using Generate Benchmark Tests
[0016] When a benchmark plan for a given test has been completed, a
benchmark test may be automatically generated using the Generate
Benchmark Tests feature. This feature is unique in important ways.
It treats each benchmark test as part of an overall benchmark plan
that generally covers an entire school year. This treatment makes
it possible to impose useful restrictions on the entire series of
tests associated with the plan. If an item has appeared on a
previous benchmark assessment, it can be restricted from appearing
on subsequent tests. Likewise, if a previous test includes one or
more questions referring to a particular text, the text and any
questions referring to it can be excluded from subsequent benchmark
assessments. The restrictions help to insure that student
performance reflects student skill and not merely responses to
specific test items.
[0017] The process of generating a test is initiated by selecting
the benchmark plan to be used in guiding the construction of the
assessment. Next the user selects the item banks to be called upon
to generate the test. The third step is to indicate a library which
will be used to store and retrieve the test and to specify the
subject and grade level for the test. Subject and grade-level
information is used to automatically generate a title for the test.
The final step is to press the Generate Test button.
[0018] When the Generate Test button is pressed, the system selects
items aligned with standards in accordance with the specifications
in the selected benchmark plan. It keeps items linked to a common
text or image together, and it implements an algorithm to order
items in ways that save paper required to print test booklets in
the event that offline test administration is elected. If items
needed to meet some of the requirements of the benchmark plan are
unavailable, the system prints a report identifying the number of
required items and indicating the standard(s) to which they are to
be assigned. These reports can play an important role in guiding
item development activities.
Benchmark Test Review
[0019] After a series of benchmark tests have been generated in the
system, the agency for which the tests were constructed has the
option of engaging in test review. Test review serves a special
function in standards-based education because benchmark tests are
designed to assess many students receiving instruction through the
efforts of many educators. All of these educators have a stake in
the assessment process. In order to meet the assessment needs of
teachers and administrators responsible for student instruction, it
is important for the views of all stakeholders to be represented in
shaping benchmark assessments.
[0020] A two-phase test review process makes it possible for
stakeholders to review assessments and achieve needed modifications
prior to test publication and administration. The first phase is
called initial review. Any number of reviewers can participate in
the initial review process. For example, a school district might
designate all fifth-grade teachers as initial reviewers of a
fifth-grade benchmark math test. By contrast, they might form a
review committee to serve as initial reviewers. The second phase is
a final review. There is only one final reviewer. The final
reviewer has access to all of the initial reviews. It is the final
reviewer's responsibility to produce a review that guides
construction of the final version of the benchmark test.
[0021] Both the initial and final reviews are carried out using the
Test Review feature of the system. A review is initiated by
selecting the benchmark assessment to be reviewed. The reviewer has
the option of displaying certain categories of items. For example,
initially a reviewer might wish to see all items. At a later time,
a reviewer interested in checking his or her judgments might elect
to see only those items that had been previously reviewed.
[0022] Each item to be reviewed is displayed along with the
standard to which it is aligned. The review status of the item is
displayed following the item. There are three status categories,
which may be selected by the reviewer: Not Reviewed, Accept, and
Replace. Not Reviewed is initially selected for all items that have
not yet been reviewed. A comment box is provided below the three
status categories.
[0023] The reviewer has the option of saving a review at any time.
The reviewer also may delete the review. Finally, the reviewer may
indicate that the review is complete. When a final reviewer
indicates that a review is complete, a message is sent to an
assessment staff. After receiving the message, an assessment staff
member goes over the final review with the final reviewer. This
part of the review process is included to facilitate the
construction of reliable and valid benchmark assessments.
[0024] The assessment staff member activates a replace button,
which appears next to each item for which the review status is
Replace. When the button is pressed, the appropriate item bank is
searched and possible replacement items are displayed. An item may
then be selected from the list of replacement items and inserted
into the test. When replacements are completed, the test is ready
for publication.
Test Construction Using Test Builder
[0025] Assessment options included in the system include the
capability to construct assessments using a feature called Test
Builder. Test Builder is used mainly at the classroom level to
enable teachers to construct class quizzes and formative
assessments used to guide instruction related to objectives that
are immediate targets for instruction. Test construction is
initiated in Test Builder by entering the title of the test and
selecting a test library in which to store the test for later
retrieval. The next step is to enter general test-taking
instructions. There are three basic options for entering items into
the test. One option is to import a printable version of the entire
test from a word-processing file. This option provides the
capability to print test booklets and automatically score scanned
answer sheets for assessments developed outside the system and
administered offline. The second option is to construct items using
text editing, equation editing, and image importation features
provided in Test Builder. This option allows users to construct
their own items, and to edit, or delete items from the test. The
third option is to search recorded item banks. A variety of search
options are available. Users may search for items aligned to a
particular objective. They may conduct a key word search of
objectives, and they may search for groups of items all linked to
the same text or image. Search features also include the capability
to automatically generate all or part of a test by designating the
objectives to be included in the test and the number of items to be
included for each of the selected objectives. Items selected from
the banks may be copied. The copies may be edited to customize the
items to user needs.
[0026] In keeping with the standards-based approach to assessment,
each item included in an assessment constructed using Test Builder
is aligned to a standard. As indicated earlier, all items in The
banks are aligned to standards. Thus, items selected from The banks
for inclusion on a test constructed using Test Builder are aligned
to standards. When a new item is constructed in Test Builder, the
user is requested to select the standard to which the item is
aligned. Although the system is specifically designed to support
standards-based assessment, there are cases in which users may not
wish to align items to standards. The system allows for the
creation of tests that are not aligned to standards. This is
accomplished by creating a dummy objective for each item on the
test.
[0027] When a new test is developed, the test is automatically
assigned a status labeled Construction Phase. When construction is
completed, the user may make a series of changes in the status.
Following construction, the user may change the test status to
Tryout Phase. During the Tryout Phase the test may be scheduled for
tryout administration, but scores will not be saved. The user also
has the option to change test status to Item Review Phase. In the
Item Review Phase the test may be subjected to test review in the
manner described for benchmark tests. Finally, the user may change
the status to Publication Phase. When test status is changed to
Publication Phase, the test can be scheduled and examinee responses
will be saved. After a test is published, it cannot be changed or
deleted. This rule assures the ability to trace examinee responses
back to the test that was actually taken. Although a published test
cannot be changed, it can be copied, and the copy can be
edited.
Scheduling Tests
[0028] After a test has been published, it can be scheduled for
online and/or offline administration. Scheduling options are
included to accommodate benchmark and classroom formative
assessments. Since benchmark tests are typically administered
agency wide, an approach that makes it possible to schedule tests
for large numbers of schools and classes can save those involved in
the scheduling process large amounts of time. A Bulk Scheduler
feature that makes it possible to schedule agency-wide assessments
quickly and easily. Bulk Scheduler allows the user to schedule a
test for all schools or selected schools in an agency. The user has
the additional option of scheduling the test for all classes in the
set of selected schools or selected classes in those schools. After
schools and classes have been chosen, the user specifies the dates
within which the test is scheduled for administration. The user may
specify a user name and password for students who will be taking
the test online. In addition, the user may specify a date for
posting assessment results to appropriate audiences (e.g., students
and parents).
[0029] Although benchmark tests are typically scheduled for large
groups of students, other types of assessments such as class
quizzes are appropriately scheduled at the class level. Test
scheduling at the class level is accomplished using the Class
Calendar feature. This feature allows the user to schedule the
dates for a class test, a user name and password for the test, and
the dates when scores will be posted. As soon as scheduling
information has been entered, it appears along with other events on
the teacher's Class Calendar.
Offline Test Administration with Scanline
[0030] Assessment innovations incorporate a feature called Scanline
that includes the ability to scan answer sheets in order to support
offline test administration. The ability to scan answer sheets has
been well established for many years. Scanline supports established
optical scanning technology using proprietary answer sheets.
Scanline also supports more recently developed technology making it
possible to print and subsequently scan plain paper answer
sheets.
[0031] The operation of Scanline requires a scanner controlled by a
client-side computer connected to the Internet. Scanline software
is downloaded from the Internet to the client machine. The software
makes it possible to scan proprietary answer sheets and send
information regarding examinee responses over the Internet to a
server, which automatically scores the responses. In the case of
plain paper scanning, the software may send information regarding
student responses. However, scanned images may also be sent to a
server.
[0032] Scanline includes a number of innovations that increase the
ease of use of plain-paper scanning technology and that enhance the
ability to detect and correct scanning errors using the plain paper
approach. Scanline technology identifies form types and form
characteristics automatically. This feature enhances ease of use
because it supports dynamic form printing and scanning. When a user
prints an answer sheet for a selected test, the answer sheet
includes a barcode containing the test-administration ID. When the
sheet is subsequently scanned, the barcode is read on the client
machine. Web services are then called that indicate to the client
the number of items on the test and the number of alternatives
associated with each item. For example, the barcode might indicate
that the test contained 35 items and that items 1 through 12 were
true-false items and that the remaining items were 4-alternative
multiple choice items. This information would be used to control
the information processed by the scanner. Dynamic scanning
minimizes the time required to process scanning information. For
example, if there are only 35 items on a test, the scanner would
process only 35 items even though the form type might be capable of
including many more items than 35.
[0033] One of the special problems associated with plain-paper
scanning involves the determination of what constitutes a marked
response. Plain paper scanning necessarily involves printing answer
sheets on multiple printers. The output of printers may vary
substantially along the light-to-dark dimension. This fact creates
circumstances in which an unmarked alternative printed on one
printer may be much darker or lighter than the same alternative
printed on another printer. One approach for determining whether or
not an alternative has been marked is to set darkness threshold
expressed in pixels. If the alternative exceeds the threshold, it
is classified as marked. Printer variation can make that approach
unreliable. In the case in which printer output is dark, unmarked
alternatives may exceed the threshold and be incorrectly classified
as being marked.
[0034] To address the classification problem, an algorithm uses the
lightest mark for an alternative as an anchor against which to
judge other alternatives. The current implementation of the
algorithm recognizes three categories along the light-dark
dimension: Light, regular, and dark. When the classification for
the lightest bubble has been established, a multiplier is applied
to establish the threshold for determining the percentage of pixels
required in the annotated space to classify the bubble as marked. A
different multiplier is used for each of the three categories in
order to insure adequate classification accuracy. The thresholds
for the three categories and the multipliers are determined by
empirical tests.
[0035] When a user experiences a problem during plain-paper
scanning, it is helpful for the user to be able to easily and
clearly communicate the problem to technical support personnel. An
innovation design assists users to address scanning problems.
Scanline stores images of scanned answer sheets on the client
machine. When a user encounters a problem, the user opens up a
scanning history feature. This feature shows each scanned image and
the status of the image. For example, if there is a scanning
problem, the history indicates that an error has occurred and
specifies the nature of the error. The user then has the option of
making needed adjustments and rescanning the sheet or submitting
the image a server. For example, if the sheet was initially scanned
upside down, the user may choose to rescan it. On the other hand,
if the nature of the problem is not readily apparent the user may
send the image to a technical support specialist for further
review. The capability to pinpoint and view problem images of
previously scanned sheets and to send them over the Internet with
one click increases scanning accuracy and greatly simplifies the
task of identifying scanning errors.
Online Assessment
[0036] Assessment options include online assessment as well as
offline assessment. Moreover, both options are available within a
single assessment. Online assessment is carried out in a virtual
student center. A dual password approach provides two levels of
security for online assessment. Each student is assigned a username
and password that allows entry into the student center. A second
username and password enables the student to enter the testing
environment for the particular assessment that the student is
scheduled to take. When the student logs into the test, his or her
name appears at the top of the test. This helps proctors to insure
that the students scheduled to take the test are actually the
individuals who do take the test.
[0037] At the beginning of each online assessment, the student is
provided with instructions explaining how to navigate through the
assessment, how to indicate her/his response, and how to review
questions. The online testing feature affords flexible navigation,
which enables the student to go to any item at any time. Contextual
materials such as narratives or charts appear above the question to
which they are attached. If the amount of such material is
extensive, it appears in a window that permits the student to use a
scroll bar to view all of the contextual content. Students may
respond to items in a variety of ways depending on the type of
item. For example, to respond to items in a multiple-choice format,
the student points and clicks on the "radio button" next to her/his
response and then clicks Save My Answer. For these and all other
questions, the system will automatically take the student to the
next question once the save button is clicked.
[0038] As the student proceeds through the test, the numbers for
questions already answered are in gray and those not yet answered
are in blue. A test completion status bar indicates the proportion
of items that have been completed, and a test summary screen lists
the questions that have been answered and those that have not been
answered. If the student has inadvertently omitted one or more
items, the summary can alert the student to the omissions.
Online Assessment with Mercury
[0039] Assessment innovations include a feature called Mercury that
provides the ability to adminster assessments using hand-held
response pad systems. These systems allow students to enter
responses to assessment questions on hand-held units. The units
then wirelessly transmit the student's responses to a receiver
where they are read and recorded by Mercury, on the instructor's
computer in a classroom or computer lab environment. Mercury
includes features that increase the ease and efficiency with which
devices of this type may be used for administration of assessments.
These innovations also help to ensure that data is accurately
collected, particularly in the event of technical problems such as
hardware failure.
[0040] Mercury includes an application that is installed on the
computer located in the classroom being used to administer the
assessments. The application has the ability to communicate
directly and automatically with the Galileo database over the
Internet via web services located on servers. Because this
communication is seamless and automatic it eliminates several of
the steps that would otherwise be required for the user. The
advantages of this approach are described in detail in the
following discussion of the steps require for this type of
assessment administration.
[0041] In order to administer an assessment, the first step that a
teacher must take is to start the Mercury application. Once the
program is started, it uses web services to communicate
automatically with the Galileo database. The data that is passed to
Mercury includes information such as the available students and the
scheduled tests. The need for the teacher to take any extra steps
to manually download information in order to start administration
of a test is eliminated. For example, the teacher is not required
to log into the Galileo servers and download the available tests
before they can be selected for administration. The list of tests
is automatically available.
[0042] The next step in administration is for the teacher to
distribute the hand held units to the students. Students then start
working on the assessment using the units to enter their answers to
the questions. Their responses are received by the Mercury program
and saved to the Galileo database via web services. This process is
entirely automatic without the teacher being required to take any
action. There is no need for manual uploading of student responses
at the end of test administration. This increases the ease of use
for the teacher because there are fewer tasks for them to perform.
The accuracy of the resulting data will also be increased because
there is no need for the teacher to remember and successfully
perform the necessary steps to complete a manual data upload of
student responses. Because the responses are being recorded
continually, there is also greater protection in the event of
hardware failure. For example, should the hard drive on the
computer running the Mercury program fail in the middle of test
administration, the data loss would be quite limited. Any student
responses that had been entered prior to the time of the failure
will have already been recorded on the Galileo servers.
Test Monitoring
[0043] With both methods of online test administration, test
monitoring is possible through the use of Galileo and Mercury.
While a group of students is taking a test, teachers may log into
the Galileo test administration screen or use links from within the
Mercury application to view a monitoring screen showing all
students currently taking the test, which questions have been
answered, and whether each question was answered correctly or
incorrectly.
Integrative Data Analysis System to Promote Standards Mastery
[0044] Standards-based assessment initiatives generally include
assessment information gathered by multiple agencies. For example,
under the No Child Left Behind Act, statewide assessments of
standards mastery are required each year at specified grade levels.
These tests are often accompanied by local standards-based
assessment initiatives such as benchmark assessment programs
implemented by local school districts. Although both types of
assessment are typically aimed at measuring the mastery of state
standards, the data from these assessments are generally not linked
in ways that provide flexible data combinations to support accurate
mastery classification and provide information that can be used to
promote standards mastery. An integrative data analysis system
links assessment data from local educational agencies to data from
super ordinate agencies such as state departments of education in
ways that promote accurate mastery classification and the
achievement of shared goals such as those reflected in state
standards.
[0045] Two kinds of data play a key role in standards-based
initiatives: continuous data and categorical data. It is virtually
universal practice in standards-based initiatives to provide test
scores for an assessment on a continuous distribution. Standard
Item Response Theory (IRT) techniques (e.g. Thissen & Wainer,
2001) is used to score tests such as benchmark assessments. In
standards-based initiatives, it is also customary to segment the
score continuum into categories to determine mastery of standards
(e.g. Cizek, 2001). For example, a score continuum might be
segmented into categories such as exceeds the standard, meets the
standard, approaches the standard, and falls far below the
standard. Ability scores such as those yielded using IRT lend
themselves to statistical techniques appropriate for use with
continuous data. By contrast, mastery classifications call for
statistical procedures appropriate for use with categorical data.
Integrative data analysis system accommodates scores of both
types.
Combinatorial Assessments
[0046] Innovative technology involving a continuous score
distribution is related to technology described in U.S. Pat. No.
6,322,366 B1 (Nov. 27, 2001) and U.S. Pat. No. 6,468,085 B1 (Oct.
22, 2002) in which a continuous ability score is used along with
item parameters to compute the probability that a student will
achieve any of a series of goals in a scale comprised of a set of
goals. This technology makes it possible to determine standards
mastery from diverse data sources. For example, mastery of one
standard in a scale might determined by grades on a sample of work
such as a class assignment. Another might come from online
assessment.
[0047] The present application introduces technology innovations
based on existing ATI patents. These innovations make it possible
to combine multiple tests into a single assessment, to combine
parts of tests to make a new test, and to combine information from
tests, class assignments, and other data sources into one scale.
These combinatorial assessments can be utilized along with other
assessment information to guide instruction toward the mastery of
standards.
[0048] ATI's combinatorial innovations have a number of practical
benefits. Combining data from different sources into a single
assessment can be expected to increase the reliability of the
assessment because test reliability is a direct function of test
length (e.g. Nunnally & Bernstein, 1994). The data sources
being combined are those directly linked to instruction. For
example, class assignments and class quizzes are routine features
of instruction. When data gleaned from these instructional
mainstays is combined in ways that yield psychometrically sound
assessments, it is possible to assess the relationship between
those assessments and other high-stakes assessments such as
statewide tests used to determine student mastery of standards. The
relationship between combinatorial assessments and high-stakes
assessments provides a measure of the extent to which performance
measured as part of instruction is assessing the same thing as
high-stakes measures of student performance used to evaluate
schools and students.
[0049] In some cases it is useful to combine parts of benchmark
tests with short quizzes. For example, a benchmark test may provide
information suggesting an intervention targeting certain
capabilities. After the intervention, a short formative assessment
may be given to determine whether or not the targeted capabilities
have been acquired. It may be useful to create and score a new
combinatorial test substituting scores on items from the quiz for
corresponding scores on items from the benchmark. The new test
might then be used to revise estimates of the risk that students
may have of not meeting standards as measured on a statewide
test.
[0050] Ability scores computed from combinatorial assessments can
play an important role in guiding instruction. Using the continuous
IRT ability score and item parameters to estimate the probability
that a student will be able to perform tasks reflecting the mastery
of particular standards. This information can be used to determine
what capabilities to target for instruction.
[0051] Many variations, modifications and changes may be made in
the above described example without departing from the scope and
spirit of the invention.
* * * * *