U.S. patent application number 11/231975 was filed with the patent office on 2006-01-19 for system and method for processing test reports.
This patent application is currently assigned to Grow.Net, Inc.. Invention is credited to David Coleman, Steve Delvecchio, Ben Fishman, Patrick Haugh, Mark Malaspina, Kito Mann, David Waxman, Jason Zimba.
Application Number | 20060014129 11/231975 |
Document ID | / |
Family ID | 35599861 |
Filed Date | 2006-01-19 |
United States Patent
Application |
20060014129 |
Kind Code |
A1 |
Coleman; David ; et
al. |
January 19, 2006 |
System and method for processing test reports
Abstract
A system and method enable processing and displaying of test
results in accordance with information and specifications provided
by the client requiring such services. Skills examined by a test
are identified, and skill-by-skill analysis of test results is
provided. Performance within constituent skill categories are
compared to applicable standards, thereby providing
criterion-referenced conclusions about a student's performance
within each skill category tested. Evaluation and recommendation
statements are generated for students or groups of students based
on their test performance and procedures for automatically
generating such statements are provided. Test results are also used
to identify skills in which individual students and groups have the
greatest potential for growth. Test reports are produced in print
and electronically using the same electronic document structures
and data source files to ensure consistency between the two display
mechanisms. The analyses and reports generated provide
instructional information tailored to the student's or group's
needs, as identified by analysis of test performance. Teachers and
administrators can track the progress of students and groups
through useful, accurate, and easily accessible test reports.
Inventors: |
Coleman; David; (New York,
NY) ; Delvecchio; Steve; (Seattle, WA) ;
Fishman; Ben; (New York, NY) ; Haugh; Patrick;
(New York, NY) ; Malaspina; Mark; (Brooklyn,
NY) ; Mann; Kito; (New York, NY) ; Waxman;
David; (New York, NY) ; Zimba; Jason;
(Brooklyn, NY) |
Correspondence
Address: |
ROTHWELL, FIGG, ERNST & MANBECK, P.C.
1425 K STREET, N.W.
SUITE 800
WASHINGTON
DC
20005
US
|
Assignee: |
Grow.Net, Inc.
New York
NY
|
Family ID: |
35599861 |
Appl. No.: |
11/231975 |
Filed: |
September 22, 2005 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10072820 |
Feb 8, 2002 |
|
|
|
11231975 |
Sep 22, 2005 |
|
|
|
60267840 |
Feb 9, 2001 |
|
|
|
Current U.S.
Class: |
434/322 |
Current CPC
Class: |
G09B 7/02 20130101 |
Class at
Publication: |
434/322 |
International
Class: |
G09B 3/00 20060101
G09B003/00; G09B 7/00 20060101 G09B007/00 |
Claims
1. A method of providing test performance results for an individual
test-taker comprising: obtaining the individual test-taker's
overall score achieved on test questions within a subject area that
was tested on the test; providing a statement relating to the
test-taker's overall score achieved on the test questions within
the subject area, the statement being selected from a group of
predefined statements, each statement of the group of predefined
statements being reflective of a different level of performance
with respect to the test questions within the subject area; listing
two or more skills within the subject area; for each of the listed
skills, providing a score achieved by the test-taker on those of
the test questions within the subject area deemed to test each of
the listed skills; for each of the listed skills, providing a
comparative score representative of the score achieved by
individual test-takers within a comparative population of
test-takers on those of the test questions deemed to test each of
the listed skills; and providing a statement for each of the listed
skills relating to the score achieved by the test-taker on those of
the test questions deemed to test each of the listed skills, the
statement being selected from the group of pre-defined statements
based on a comparison between (1) the score achieved on the skill
by the individual test-taker and (2) the score achieved on the
skill by the individual test-takers within the comparative
population of test-takers.
2. The method of claim 1, wherein the comparative population of
test-takers consists of a population selected from the group of
populations comprising: (a) all test-takers who took the particular
test for which the performance results are being provided; and (b)
all test-takers who took the particular test for which the
performance results are being provided and whose overall score
achieved on questions within the subject area corresponds to a
specific one of the predefined statements of the group of
predefined statements.
3. The report of claim 1 wherein the score reported for each skill
comprises: (a) a percentage of correct responses on those of the
test questions within the subject area deemed to test the skill; or
(b) a number of correct responses on those of the test questions
within the subject area deemed to test the skill.
4. The report according to claim 1, wherein the comparative score
comprises a percentage of correct responses by the test-takers
within the comparative population on those of the test questions
within the subject area deemed to test the skill for which the
score is reported; or a number of correct responses by the
test-takers within the comparative population on those of the test
questions within the subject area deemed to test the skill for
which the score is reported.
5. The report according to claim 4, wherein the comparative score
comprises the average percentage or number of correct responses of
all test-takers within the comparative population or a range of the
percentage or number of correct responses of all test-takers within
the comparative population.
6. A method for generating statements for individual test-takers of
a group of test-takers with respect to separate skills within a
subject area assessed on a test, the method comprising: (a)
obtaining overall test scores achieved on test questions within the
subject area as a whole for all test-takers of the group of
test-takers; (b) obtaining a set of separate skills within the
subject area; (c) for each of the separate skills, obtaining the
scores achieved on those of the test questions deemed to test the
skill for all test-takers of the group of test-takers; (d)
obtaining a score-to-statement mapping between (i) possible overall
scores achievable by individual test-takers on the test questions
within the subject area, and (ii) a set of statements used to
describe the overall performance of individual test-takers with
respect to the subject area, such that each statement of the set of
statements correspond to an associated range of possible scores
achievable by individual test-takers on the test questions within
the subject area; and (e) selecting from the information obtained
in steps (a) through (d) one of the statements to describe the
performance of each individual test-taker on each skill assessed on
the test.
7. A method of providing test performance results for an individual
test-taker comprising: obtaining the individual test-taker's
overall score achieved on test questions within a subject area that
was tested on the test; providing a statement relating to the
test-taker's overall score achieved on the test questions within
the subject area, the statement being selected from a group of
predefined statements, each statement of the group of predefined
statements being reflective of a different level of performance
with respect to the test questions within the subject area; listing
two or more skills within the subject area; for each of the listed
skills, providing a score achieved by the test-taker on those of
the test questions within the subject area deemed to test each of
the listed skills; and providing a statement for each of the listed
skills relating to the score achieved by the test-taker on those of
the test questions deemed to test each of the listed skills, the
statement being selected from the group of pre-defined statements
relating to the test-taker's overall score achieved on the test
questions within the subject area.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. application Ser.
No. 10/072,820, filed Feb. 8, 2002, which claims the benefit of
U.S. Provisional Application No. 60/267,840, filed Feb. 9, 2001,
which is hereby incorporated by reference.
COPYRIGHT AND TRADEMARK NOTICE
[0002] A portion of the disclosure of this patent document contains
material that is subject to copyright and trademark protection. The
copyright owner has no objection to the facsimile reproduction by
anyone of the patent document or the patent disclosure, as it
appears in the Patent and Trademark Office patent files or records,
but otherwise reserves all copyright and trademark rights
whatsoever.
FIELD OF THE INVENTION
[0003] This invention relates to systems and methods for analyzing
and interpreting test results and for displaying test results
together with diagnostic information and instructional
materials.
BACKGROUND OF THE INVENTION
[0004] A standardized test consists of questions or tasks that are
given to students for completion under similar testing conditions,
thus enabling comparisons among the students' performance. The term
"standardized test" is used here expansively to denote assessments
of various sorts.
[0005] Standardized tests are employed in a wide variety of ways in
our society. For example, standardized test results play an
important role in certain employers' decisions concerning hiring
and promotion decisions, certain government agencies'
determinations of whether to license professionals, and certain
educational institutions' admissions decisions.
[0006] In addition, standardized tests are increasingly used within
K-12 education as a means of assessing students' progress in
various disciplines such as math, reading, social studies, and
language arts. At least 48 states now assess students' reading and
mathematics skills at the elementary, middle, and high school
levels. Increasingly, the results of end-of-year tests are seen as
an important way to measure educational progress at the state and
local level, and the consequences of these tests are growing; for
example, in some places, students may be held back from advancing
to the next grade based on their standardized test results.
[0007] A particular standardized test (a "Test") is designed to
measure the performance of a test-taker in a given field or domain
(a "Subject"). Subjects could include an academic discipline (e.g.,
college mathematics); (b) a professional field (e.g., tax
accounting); or (c) a practical endeavor (e.g., driving).
Individual test-takers are known as "Students", and aggregations of
Students are known as "Groups". Groups can exist at different
levels of hierarchy, such as the class or school level, or can be
based on non-hierarchical relationships such as shared ethnicity,
etc.
[0008] A Test is designed to measure Students' abilities to carry
out certain tasks in that Subject and/or Students' knowledge about
that Subject ("Skills", sometimes known in the literature as
"Attributes" or "Rules"). A Test is designed for test-takers within
a given ability range or at a certain point within a course of
study, meaning that the Test has a certain "Level" associated with
it.
[0009] All of the information that can be said about a Student or
Group, based on its performance on a Test, is known as the "Test
Results". Test Results is a broad concept, which can encompass both
numerical and evaluation statements, either about Students' overall
performance or performance in a specific Skill. Some examples of
Test Results are as follows: a list of Student's total scores on a
monthly diagnostic test; a chart tracking the average performance
of girls in a school from one year to the next on the yearly math
exam; or statements of "Needs Improvement", "Good Work", or "Review
Vectors" associated with Students' performance in the various
Skills assessed on a science exam.
[0010] Users are the individuals that use the Test Results for a
given purpose. Some typical Users include Educators, Parents, and
Students. For example, Educators. (as defined below) may use Test
results to guide instruction of Students, or to evaluate the
overall progress of a class or school. Likewise, Parents may use
Test Results by following up with their child's teacher to make
sure the child receives additional instruction in a Skill.
[0011] Here, Educator is an extremely general term and can refer to
any individual associated with the training or instructing of
Students. For example, in the K-12 context, an Educator can include
without limitation teachers, tutors, reading specialists,
remediation specialists, or administrators of various kinds (such
as a school principal, superintendent, or state education
official). In other contexts, an Educator could be a job trainer,
flight trainer, or professor, for example.
[0012] An organization that wants to process and display the
results of a Test to a certain group of Users is known as the
"Client". For example, a Client may be a local school district that
wishes to process and display the students' results on a statewide
standardized exam. In other cases, the Client could be the test
publishing organization, which wants a way to effectively process
and display the Test Results. Here, more generally, the term Client
is used to refer to the organization or organizations that may
provide inputs into the system, such as lists of Students and lists
of Students' responses to tasks on the Test.
[0013] It is important to note that standardized tests have
different reference methods. For example, some standardized tests
are "Norm-referenced", meaning that an individual student's
performance is implicitly compared against the performance of other
students. Other standardized tests are "Criterion-referenced",
meaning that students' performance is implicitly compared against
performance standards in that Subject as established by pedagogical
experts in that Subject. Criterion-referencing is common in a
number of contexts, including licensure and certification exams
within the professions, K-12 accountability measures, college
entrance exams, and elsewhere. According to Gandal in The State of
States: A Progress Report (1999), increasing numbers of states are
moving away from norm-referenced tests that compare students to
national averages and toward criterion-referenced exams that
measure students' ability to master standards-based material.
[0014] Standardized tests may be given only at the end of a course
of study (such as at the end of a grade in school), or they may be
given at various times throughout the year to measure students'
progress. Their purposes may be evaluative or diagnostic or some
combination of both.
[0015] Also, standardized tests may be structured in various ways.
For example, a standardized test consists of one or more questions
(known here as "Items"). Items may be of one or more types; for
example, two common types of Items are multiple choice Items (which
require a student to choose the best response among various
possible answers) and constructed response Items (which require a
student to compose the student's own answer). Other types of Items
could include tasks of other natures, in other forms, delivered by
other media.
[0016] Finally, standardized tests may be administered and scored
in diverse ways. For example, they can be administered in various
media, such as in paper and through a networked computer. Scoring
of Items can be performed manually, electronically, or in some
combination thereof. Items are scored with respect to a "Scoring
Guide" for that Test, which may include an answer key for scoring
multiple choice questions and/or rubric guides for scoring essays
and other types of open-ended questions.
[0017] Innovations in standardized testing continue to reshape the
field, particularly within the field of psychometrics, the science
of interpreting test results by means of statistical and cognitive
models. Now some types of standardized tests involve a testing
process in which different Students are given different Items and
complex scoring methodologies are used to generate aggregate (and
in some cases, Skill-specific) Scores that are comparable across
Students.
[0018] Regardless of their subject, level, reference method,
design, structure, administration, or scoring method, all
standardized tests share a common feature: Test Results are
received by individuals who want to utilize the results in certain
ways. In some cases, the Users are only interested in aggregate
information that specifies how well students performed overall on a
test. For example, a school administrator may want to review the
school's mean student performance on a test in a given subject from
year to year, as one method of evaluating the school's progress in
that subject over time.
[0019] Often, however, Users want significant diagnostic
information that goes beyond students' overall test performance.
For example, an Educator may want to know how well a given student
performed on a particular Skill examined on a given Test. The
Educator also may be interested in how well certain groups of
students performed on particular Skills. Furthermore, the Educator
may want to understand what instructional strategy is most
appropriate for individuals and groups, based on results from that
test. Other recipients of Test Results (such as Students or
Parents) often desire similar information.
[0020] In order for Test Results to be useful to Users, Test Result
information should be processed and displayed in a manner that
permits Users to understand the results, navigate between different
displays of the results, and take action based on the results.
Current methods of processing and displaying Test Result
information have various flaws: For example, the Skill categories
in which the results are displayed are not useful, and the methods
used to generate Skill-Item associations are crude. Likewise, the
conclusions that are reached about individual Students and Groups,
based on the Results, are often difficult for Users to understand
and are based on sub optimal methods of generating statistical
conclusions. Finally, the display of the Test Results itself leaves
much to be desired, as current methods (such as U.S. Pat. Nos.
5,934,909 and 6,270,351) fail to enable Users to see Test Results
and related instructional materials in a way that facilitates
action.
[0021] These problems with the existing methods of processing and
displaying Test Results are endemic across all forms of
standardized testing, including such diverse fields as corporate
training and higher education. Indeed, in the K-12 context, various
experts have sharply critiqued current systems and methods for
processing and displaying Test Result information. For example, the
National Educational Goals Panel (1998) has concluded that printed
reports given to Parents about the Test Results in K-12 "are not
very clear". As a result, the nonprofit organization Public Agenda
has concluded in its Reality Check (1998) that Parents "appear to
lack a solid grasp of their schools' academic goals," as well as
the "information essential to properly evaluate how well their
children and schools are doing."
[0022] Similarly, even though many Educators in grades K-12 are
told to use data to inform their instructional practice, they are
not positioned to do so because current systems and methods do not
render the information meaningful or comprehensible. As researchers
at the UCLA Center for Research on Evaluation, Standards, and
Student Testing have concluded, "The practice of applying
large-scale data to classroom practice is virtually nonexistent"
(2001).
[0023] The system and method described in this invention address
deficiencies in the current methods for the processing and
displaying of Test Results, with application to all forms of
standardized testing.
OBJECTS AND SUMMARY OF THE INVENTION
[0024] The objects of the invention are several, primarily related
to the importance of ensuring that Test Results are processed and
displayed in such a way that Users can respond practically and
effectively to the demonstrated educational needs of Students and
Groups: [0025] 1. One object of the invention is to enable the
processing and displaying of Test Results based on information and
specifications provided by a Client requiring such services, and
performed according to strict routines and operating processes. The
information and specifications to be provided by the Client
includes the Test itself, performance information such as the
Student Scores on the Test Items, the identity of the Students and
Groups who took the Test, specifications approved by the Client
with respect to the displays to be shown Users, and educational
content related to the Subject of the Test. [0026] 2. Another
object is to produce Skill-by-Skill analyses of Test Results in a
wide array of circumstances, for example even when the Client is
itself unable to identify which particular Skills are actually
being examined by a particular Test. [0027] 3. Another object is to
ensure that the Skill categories deployed in performing the test
analysis are both pedagogically useful and statistically sound.
Reporting Test Results in Skill categories that are overbroad, too
specific, inconsistent with the educational goals and practices of
Users, or statistically ungrounded can render the results useless
as a basis for instructional action by Users. Accordingly, a
reliable process is needed that will establish useful,
statistically sound Skill categories. [0028] 4. Another object is
to introduce enhanced instructional insight into certain
psychometric techniques for establishing correspondences between
Items and Skills. These techniques take as their input a "coding
matrix" that is typically binary in form; the results of the
techniques could be improved if the coding matrix reflected more
nuanced pedagogical insight about the extent to which an Item
assesses a particular Skill. [0029] 5. Another object is to
generate evaluation statements and recommendation statements about
Students based on their performance. Users of Test Results do not
simply want to see numerical Scores for each Student. Instead, they
want meaningful verbal statements about how Students performed,
both on the Test as a whole and in particular Skills, as well as
concrete recommendations about steps for improvement. Therefore it
is useful to establish practical procedures for assigning such
statements based on Student Skill performance. [0030] 6. Another
object is to produce Criterion-referenced conclusions about
Students' performance on individual Skills. Many Tests are
Criterion-referenced as a whole, meaning that Students' overall
performance on the Test is compared against performance standards
in that Subject that have been established by pedagogical experts.
Users rely on these standards to know where their Students stand in
mastery of a Subject. An object of this invention, therefore, is an
efficient method for leveraging existing Criterion-referencing of
Students' overall scores in order to generate approximate
Criterion-referenced statements about Students' performance on the
Skills assessed by the Test. [0031] 7. Another object is to use
Test Results to identify the Skills in which Students and Groups
have the greatest potential for growth. Users often want to
understand how Students and Groups can most efficiently achieve
mastery of various Skills. A method for identifying Skills where
rapid progress might be expected, especially a method that is
understandable to Users and computationally efficient, would be
valuable to Users. [0032] 8. Another object is to generate
diagnostic statements, including Skill-specific recommendations,
for Groups of Students. Users will find Group-level recommendations
particularly useful to the extent that the recommendations are
aligned with the recommendations given for individual Students and
take into account the effects of different Group-level
instructional actions on individual Students. [0033] 9. Another
object is to generate meaningful statements summarizing the
performance of Groups. It is important that these statements resist
regression towards the mean, as Students are conceptually
aggregated into larger and larger Groups such as schools, school
districts, and states. [0034] 10. Another object is to establish a
system and method for obtaining a list of desired statistical
analyses from an external source (here called the "Client");
performing calculations according to these specifications; and
extracting the resulting information into electronic document
structures that can be used to facilitate print and electronic
displays of data. These statistical specifications could range
widely and could include, for example, performance data that is to
be calculated across time or by racial group. [0035] 11. Another
object is to produce reports about Test Results and instructional
information both in print and electronically, using the same
electronic document structures and data source to ensure
consistency between the two displays, and to encourage Users to see
electronic displays through the Internet by means of access
information included in the printed documents. [0036] 12. Another
object is to provide Users with instructional materials that are
organized in the same Skills assessed on the Test; that offer Users
instructional responses in each Skill that are appropriate for
Students with varying levels of mastery; and that are organized
within each Skill in the same categories, or mastery levels, that
are used to describe Students' performance in the Skills tested.
This is highly effective in encouraging Users to act on the Test
Results to address their Students' particular needs. [0037] 13.
Another object is to enable Users to navigate directly among key
electronic displays of Test Result information and instructional
materials. It is critical, for example, to enable Users to go
directly from seeing an overall Skill-by-Skill analysis of a
Student or Group to additional information on that Skill, such as
the breakdown of the Group into different mastery levels within
that Skill alone. [0038] 14. Another object is to permit Users,
such as teachers, to track Students' progress in Skills over time
using diagnostic measures that are simple and efficient to
implement. Test Results, though one useful measure of Student
performance, should be supplemented by ongoing assessments in the
classroom in order to be trustworthy. In order for this
supplementation to be effective, the ongoing assessments should be
practical to access, distribute, grade, and record. They also
should be calibrated to generate evaluation statements that are
tightly aligned with the evaluation statements given to Students
and Groups based on the Test Results themselves.
[0039] The system of the invention involves several Operators that
operate and coordinate multiple components and subcomponents, as
detailed in later sections. Physically, the system may be
constructed from a network of interoperating servers and desktop
computers, which form a Local Area Network (LAN), connected to the
Internet through a switch and secure firewall.
[0040] The method of the invention consists of several steps:
[0041] First, the system receives information about the Test
itself, performance information such as the Student Scores on the
Test Items, the identity of the Students and Groups who took the
Test, specifications approved by the Client with respect to the
displays to be shown Users, and educational content related to the
Subject of the Test.
[0042] Second, the system executes analysis procedures to identify
the Skills examined on the Test and determine Scores, i.e.
numerical measures of Student and Group performance by Skill
(assuming these are not originally provided to the system as an
input). In doing so, the system implements routines to ensure that
the Skill categories themselves are pedagogically useful and
statistically sound.
[0043] Third, based on these numerical scores and on Client
specifications about the information to be displayed, the system
generates and stores various quantitative and qualitative
indicators and recommendation statements by Student and Group, both
in the aggregate and by Skill.
[0044] Fourth, the system organizes educational content based on
how the Test scores themselves are reported, including by breaking
down the content into the same Skills that are assessed on the
Test, and by breaking down the content within a Skill into the same
categories, or mastery levels, used in the evaluation statements
for Students and Groups.
[0045] Fifth, in accordance with the specifications of the Client,
the system generates displays of the Test Results and instructional
information, and distributes these related displays to Users in
print and via the Internet. The print reports may include access
information so the Users can access their accounts online.
[0046] Sixth, the website operated by the system enables Users to
obtain instructional information for each Skill at different
mastery levels and provides direct navigational access between
critical views of different Test Results. In one embodiment, the
system additionally provides Educators with diagnostic assessment
tools by Skill for use in the classroom. These assessments are
scored by hand using the same evaluation statements as employed in
assessing Students' and Groups' Test performance, and therefore
they enable Educators to track Student performance by Skill in an
ongoing way.
BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The subject matter regarded as the invention is particularly
pointed out and distinctly claimed in the concluding portion of the
specification. The invention, however, both as to organization and
method of operation, together with features, objects, and
advantages thereof may best be understood by reference to the
following detailed description when read with the accompanying
drawings in which:
[0048] FIG. 1 is a block diagram illustrating the system.
[0049] FIG. 2 is a block diagram illustrating the technical
components of the system.
[0050] FIG. 3 is a flowchart illustrating the operation process of
the entire system.
[0051] FIG. 4 is a block diagram illustrating the operation
components of the Data Intake Module component of the system.
[0052] FIG. 5 is a block diagram illustrating the Data Storage
Module component of the system.
[0053] FIG. 6 is a flowchart illustrating the operation process of
the Analysis Module component of the system, according to one
embodiment of the system.
[0054] FIG. 6a is a block diagram illustrating the Item-Skill
Analyzer component of the system.
[0055] FIG. 7 is a flowchart illustrating the operation process of
the Skill Set Generator component of the system.
[0056] FIG. 8 is a diagram illustrating Skill Organizations,
described in the Skill Set Generator component of the system.
[0057] FIG. 9 is a flowchart illustrating the operation process of
the Skill Item-Table Generator component of the system.
[0058] FIG. 10 is a flowchart illustrating the operation process of
the Item-Skill Analyzer component of the system.
[0059] FIG. 10a is a flowchart illustrating the operation process
of the Student Skill Score Calculator component of the system.
[0060] FIG. 11 is a flowchart illustrating the operation process of
the Student Skill Performance Evaluator component of the
system.
[0061] FIG. 12 is a flowchart illustrating the Criterion Mapping
execution, described in the Student Skill Performance Evaluator
component of the system.
[0062] FIG. 12a is a flowchart illustrating the Room to Grow.TM.
prioritization, described in the Student Skill Performance
Evaluator component of the system.
[0063] FIG. 13 is a flowchart illustrating the operation process of
the Aggregation Analyzer component of the system.
[0064] FIG. 14 is a flowchart illustrating the operation process of
the Utility Aggregation algorithm, described in the Aggregation
Analyzer component of the system.
[0065] FIG. 14a is a diagram illustrating the different class
breakdowns possible within the First Scenario, described in the
Aggregation Analyzer component of the system.
[0066] FIG. 14b is a diagram illustrating the different class
breakdowns possible within the Second Scenario, described in the
Aggregation Analyzer component of the system.
[0067] FIG. 14c is a diagram illustrating the different class
breakdowns possible within the Third Scenario, described in the
Aggregation Analyzer component of the system.
[0068] FIG. 14d is a flowchart illustrating the operation process
of the Statistics Summarizer component of the system.
[0069] FIG. 15 is a flowchart illustrating the operation process of
the Content Manager component of the system.
[0070] FIG. 16 is a flowchart illustrating the operation process of
the Display Module component of the system.
[0071] FIG. 17 is a diagram of a Data Display Requirement and a
diagram of a corresponding Electronic Document Structure, described
in the Display Module component of the system.
[0072] FIG. 18 is a diagram of a portion of an Instructional
Response Report, described in the Display Module component of the
system.
[0073] FIG. 19 is a diagram of a sample printed report, described
in the Display Module component of the system.
[0074] FIG. 19a is a diagram of a complex data display in print,
described in the Display Module component of the system.
[0075] FIG. 20 is a diagram of a Results Report and a diagram of a
corresponding Electronic Display, described in the Display Module
component of the system.
[0076] FIG. 21 is a flowchart of the operation process of the user
authentication process, described in the Display Module component
of the system.
[0077] FIG. 22 is a flowchart illustrating the operation process of
a Display Website, described in the Display Module component of the
system.
[0078] FIG. 23 is a diagram illustrating a "Welcome" display of a
Display Website, described in the Display Module component of the
system.
[0079] FIG. 24 is a diagram illustrating a "Main Menu" display of a
Display Website, described in the Display Module component of the
system.
[0080] FIG. 25 is a diagram illustrating a "Students' Overall
Performance" display of a Display Website, described in the Display
Module component of the system.
[0081] FIG. 26 is a diagram illustrating a "Skill Profile of a
Student" display of a Display Website, described in the Display
Module component of the system.
[0082] FIG. 27 is a diagram illustrating a "Skill Profile of a
Group" display of a Display Website, described in the Display
Module component of the system.
[0083] FIG. 28 is a diagram illustrating a "Performance in a Skill
across Students" display of a Display Website, described in the
Display Module component of the system.
[0084] FIG. 29 is a diagram illustrating a "Listing of
Instructional Tools" display of a Display Website, described in the
Display Module component of the system.
[0085] FIG. 30 is a diagram illustrating an "Instructional Tools"
display of a Display Website, described in the Display Module
component of the system.
[0086] FIG. 31 is a diagram illustrating potential hyperlinks in a
Display Website, described in the Display Module component of the
system.
[0087] FIG. 32 is a diagram illustrating a potential electronic
display of "Performance in a Skill across Groups", described in the
Display Module component of the system.
[0088] FIG. 33 is a flowchart illustrating the operation process of
a Display Website with an Ongoing Assessment Component, described
in the Display Module component of the system.
[0089] FIG. 34 is a diagram illustrating a "Welcome" Display for a
Display Website with an Ongoing Assessment Component, described in
the Display Module component of the system.
[0090] FIG. 35 is a diagram illustrating an "Introduction to
Ongoing Assessments" Display for a Display Website with an Ongoing
Assessment Component, described in the Display Module component of
the system.
[0091] FIG. 36 is a diagram illustrating a "Performance in a Skill
across Students" Display for a Display Website with an Ongoing
Assessment Component, described in the Display Module component of
the system.
[0092] FIG. 37 is a diagram illustrating an "Update Student Skill
Assignments" Display for a Display Website with an Ongoing
Assessment Component, described in the Display Module component of
the system.
[0093] FIG. 38 is a diagram illustrating an "Ongoing Assessment"
Display for a Display Website with an Ongoing Assessment Component,
described in the Display Module component of the system.
[0094] FIG. 39 is a diagram illustrating a "Focus on a Student"
Display for a Display Website with an Ongoing Assessment Component,
described in the Display Module component of the system.
FULL DESCRIPTION OF THE DRAWINGS
1. System Overview and Requirements
[0095] FIG. 1 illustrates the four components of the system, each
of which is more fully described in subsequent sections. These
components are the Intake Module 102, the Data Storage Module 112,
the Analysis Module 120, and the Display Module 142.
[0096] The system is activated by a System Operator, which may be
instantiated as a human or a computer running a script. The System
Operator carries out the following steps, as shown in FIG. 3.
[0097] In step 302, the System Operator notifies the Intake
Operator to activate the Intake Module 102, following the flowchart
in FIG. 4. The Intake Module has two functions: First, the Intake
Module populates the Information Repository 114 with inputs from
the Client concerning the Test, the Students, the Users, as well as
specifications about how the Client wants information to be
displayed to Users. Second, the Intake Module populates the Content
Repository 116 with instructional information from the Client
related to the Subject of the Test.
[0098] In step 304, the System Operator signals the Analysis
Operator to activate the Analysis. Module 120, following the
flowchart in FIG. 6. Analogous to the Intake Module 102, the
Analysis Module 120 has two functions: First, the Analysis Module
120 uses the information stored in the Information Repository 114
to generate numerical information, qualitative statements, and
recommendations based on the performance of Students and Groups on
the Test, and the Analysis Module 120 writes that information to
the Information Repository 114 upon completion. Second, the
Analysis Module 120 organizes the Content in the Content Repository
116 so that instructional information can subsequently be presented
to Users in a manner that is tightly integrated with information
about the Test Results themselves.
[0099] In step 306, the System Operator signals the Display
Operator to activate the Display Module 142, following the
flowchart in FIG. 16. The Report Generator 144 extracts information
from the Information Repository 114 and Content Repository 116
respectively, and assembles that information into electronic
reports, called Results Reports and Instructional Information
Reports respectively. The Print Displayer 150 and Electronic
Displayer 152 then utilize these reports to create displays viewed
by Users.
[0100] FIG. 2 illustrates potential technical components of such a
system, which are the following:
[0101] The system may be constructed from a network of
interoperating servers and desktop computers, which form a Local
Area Network (LAN). This LAN can be built using Fast Ethernet
technology over twisted pair cables (such as 100BaseT). Each of the
servers and computers may be equipped with a compatible network
adapter, and connected to a central Network Switch 204 via
appropriate network cables. To segment the network into logical
components for security, this Network Switch 204 may ensure Layer 2
through Layer 4 access control over traffic between the computers
and servers, using a switch such as the Cisco.RTM. Catalyst.RTM.
2900 series. The lines between machines in FIG. 2 illustrate the
permitted access routes.
[0102] There may be an uplink connection between the Firewall 202
and the Network Switch 204. The Firewall 202 may be an
enterprise-class firewall appliance similar to as the Cisco.RTM.
Secure PIX.RTM. 500 series. The purpose of this Firewall 202 is to
restrict traffic between the LAN and the Internet on every
direction and every network port unless specified otherwise in the
following paragraphs.
[0103] The Data Intake Computer 206 facilitates the transfer of
data from external test data sources into the LAN. In one
embodiment, this Data Intake Computer 206 should have access to
enough storage space either internally or through Flat File Storage
208 space to permanently store every data file transferred into the
LAN. In a typical configuration, this Data Intake Computer 206 can
be an Intel.RTM. Pentium.RTM. III-class machine with at least 20
gigabytes of storage space running the Microsoft.RTM. Windows.RTM.
2000 operating system. The Flat File Storage 208 may be a similar
machine but with additional space on its hard drive. The Firewall
202 may be configured to allow Internet communications to and from
the Data Intake Computer 206.
[0104] An operator may run programs familiar to those skilled in
the art on the Data Intake Computer 206 to parse the data file and
load it into one or Data Servers 214. The Data Servers 214 may run
relational database management systems (RDBMS) that allow for
efficient querying and storage of tabular data. These Data Servers
214 may run Oracle8i.TM. Standard Edition on top of a UNIX
operating system (OS) such as Sun.TM. Solaris.TM. 7OS. These Data
Servers 214 may be powerful enough to allow for concurrent,
multi-user access (both read and write) to the data that is stored
within them. Typically, these Data Servers 214 can be
enterprise-class servers such as the Sun.TM. Enterprise.TM. 420R
with one or more 450 MHz CPUs and 100 or more gigabytes internal
disk storage. Additional external disk storage can be added as
total amount of data on the machine grows by attaching external
disk arrays such as Sun.TM. StorEdge.TM. A1000 disk arrays to the
data servers via a fast network connection or SCSI-style
connectors. To ensure security of the data stored within these Data
Servers 214, detailed access control may be implemented using
mechanisms provided by the DBMS, according to methods familiar to
those skilled in the art.
[0105] For documents that are delivered by mail or by hand, a
secure storage area can be used to store sensitive physical
documents such as test booklets. Security policies of permitting
access to electronic media may serve as a reference when granting
access to this Physical Document Storage 212.
[0106] Several computers may access the Data Servers 214 to perform
a variety of functions. Both the Pedagogical Analysis Computer 216
and the Psychometric Analysis Computer 218 may be installed with
client software that is compatible to the DBMS used on the data
server. The operators of the two aforementioned computers are
likely to be authorized users as defined by the access control
profile stored in the DBMS. They then may have the privileges to
extract data from the DBMS over the LAN onto their local computers
to perform further data processing. These two computers may be
powerful desktop machines that can perform calculation of large
data sets efficiently. Intel.RTM. Pentium.RTM. III class machines
with more than 512 MB of RAM would be suitable.
[0107] The Psychometric Computer 218 may be configured to run
standard statistical packages such as such as SAS.RTM., SPLUS.RTM.,
Mathematica.RTM., BILOG.RTM., etc. Using those statistical packages
as a software platform, additional interactive programming and
iterative refinement of the psychometric programming code can be
carried out. Throughout the development of the code, the artifacts
may be stored in a Code Library 222, which could be a workgroup
server-class machine running the Microsoft.RTM. Windows.RTM. 2000
operating system that houses a configuration management system
similar to StarBase's StarTeam.TM. system. Once source code is
finalized, the code can be transferred to the Application Server
220, which may be an enterprise-class server running the UNIX
operating system that allows the same code to run faster on the
data set. An example of an appropriate Application Server would be
a Sun.TM. Enterprise.TM. 420R with one or more 450 MHz CPUs and 100
or more gigabytes of internal disk storage. To ensure
interoperability of code between the Psychometric Analysis Computer
and the Application Server, both machines may run analysis software
from the same statistics software vendor.
[0108] Similar to the Data Intake Computer 206, a Content
Management Computer 224 may be configured to serve as storage for
content transfer from outside the LAN. This Content Management
Computer 224 may run the Microsoft.RTM. Windows.RTM. 2000 operating
system and contain copies of a collection of content editing
software for various types of text, images, or other content
resources. Content that is imported from external source may be
stored in their native format in a Content Feed Storage 228 device
connected to the LAN. This allows selective incorporation of
external content into the Content Server 234.
[0109] The Content Server 234 may run multimedia content repository
software that allows multiple types of content resources to be
stored on a single system. Software packages, such as Oracle.RTM.
Internet File System (iFS), are candidates for such a Content
Repository 230. An enterprise-class server running the UNIX
operating system, similar to the one used for the Data Servers 114,
may be used to power the Content Repository 230.
[0110] Content that is generated by internal content authors may be
created and edited using one or more Content Authoring Computers
228 running similar software and hardware as the Content Management
Computer 224. These machines may have user-friendly interfaces that
authors can use to add and modify content that is stored in the
content repository. Software packages such as Arbortext.TM.
Epic.TM. Editor may be suitable as the front-end for these
content-editing tasks.
[0111] For extracting and merging useful data and content, and
generating results reports, a Results Reports Publishing Computer
236 may be configured to access the Data Servers 214 using a
database access protocol such as Java Database Connectivity (JDBC).
The Result Reports Publishing Computer 236 may be a powerful
personal computer running the Microsoft.RTM. Windows.RTM. 2000
system with one or more Intel.RTM. Pentium.RTM. III processors. The
output of the publishing computer is a self-contained report
encoded in a markup language such as XML. These structured
documents may then be stored in a Report Server 240 for further
processing. Similarly, content from the Content Repository 230 may
be selected and extracted by an Instructional Response Report
Publishing Computer 238 and placed inside the report repository in
a markup language such as XML or HTML. The hardware configuration
of the two report-generating machines may be similar: Intel.RTM.
Pentium.RTM. III-class personal computers with one or more CPUs
running the Microsoft.RTM. Windows.RTM. 2000 system.
[0112] The Report Server 240 may be an enterprise-class machine
similar to those used for the Data Server 214 and Content Server
234. For the storage of the markup documents, for example, in the
XML or HTML format, a well-defined directory structure using the
native file system of the UNIX operating system may be sufficient.
The Report Repository 238 may include more than one Report Server
240 if needed for scalability.
[0113] To ensure the quality of the reports in the Report
Repository 238, a Quality Assurance Computer 242 may contain a
custom program that is configured to extract all or a sample of
reports and run programs to ensure the consistency and completeness
of both types of reports. The Quality Assurance Computer 242 may be
an Intel.RTM. Pentium.RTM. III-class personal computer with one or
more CPUs running the Microsoft.RTM. Windows.RTM. 2000 system. Any
errors discovered during this quality assurance process may prompt
rework in all or some of the previous steps.
[0114] It may be necessary to convert the markup documents stored
in the Report Servers into formats that a Physical Imaging Device
246 can interpret and print. A separate Print Converter Computer
244 can be used for this purpose. A typical output language for the
print converter may be the Adobe.RTM. PostScript.RTM. format. A
possible configuration for this Print Converter Computer 244 is a
workstation-class machine with one 450 MHz processor running a UNIX
operating system such as Sun.TM. Solaris.TM. operating system. It
is also possible to use a data merge print language that generates
the PostScript format. This Print Converter Computer 244 may also
be responsible for scheduling and spooling print data to one or
more Physical Imaging Device 246. Possible imaging devices may be
high-speed laser printer such as the Xerox.RTM. DocuTech.RTM.
series.
[0115] To convert and serve the files in a human-friendly HTML
format to authorized users via the World Wide Web, a Web
Application Server 246 may be placed between the Internet and the
Report Server 240. The Firewall 202 may need to be configured to
allow Internet access to the Web Application Servers 246 over the
Hypertext Transfer Protocol (HTTP). One possible configuration of
the Web Application Servers 246 would be a workstation-class UNIX
server similar to the one used as the Print Converter Computer 244.
One or more Web Application Servers 246 may be installed as a
cluster for increased performance and fail-over in case of hardware
or software failure. Report-specific code may add visual
embellishments and hypertext navigation structure using a web
template language such as JavaServer Pages.TM. (JSP). A
JSP-compatible engine such as Apache Tomcat may be used on the Web
Application Server 248.
[0116] Also, the Web Application Server 248 may manage the User
authentication process, ensuring that only authorized Users have
access to the various electronic displays. The Web Application
Server 248 may accomplish this by making queries to the Data
Servers 214 using a protocol such as Lightweight Directory Access
Protocol (LDAP) or Structured Query Language (SQL) to retrieve data
stored about a User who is attempting to log on. The retrieved
information may consist of a unique Log In ID and password for a
User, as well as the User Electronic Display Rules that define what
a particular User is able to view.
[0117] Finally, authorized users on the Internet may use a personal
computer that can run web browser software such as Microsoft.RTM.
Internet Explorer.RTM. 6 or Netscape.TM. Navigator.TM. 6 to access
the reports served up by the Web Application Servers 246.
2. Intake Module
[0118] The Intake Module 102 consists of two parallel processes by
which a range of information is received from the Client,
assembled, and stored. This information relates to such matters as
the Test itself, the identity of the Students and Groups who took
the Test, and the specifications approved by the Client with
respect to the displays to be shown Users. The information also
includes educational content sent by the Client to be shown to
Users in conjunction with the Test Results.
[0119] FIG. 4 illustrates the operation process of the Intake
Module. The System Operator activates the Intake Module 102 by
notifying the Intake Operator (which could be person or a computer
implementing a script), which notifies in parallel the Data
Receiver Operator and the Content Receiver Operator, so that the
Data Receiver 104 and the Content Receiver 108 are activated in
steps 402 and 406 respectively.
[0120] 2.A. Data Receiver
[0121] Once the Data Receiver Operator activates the Data Receiver
104 in step 402, the Data Receiver receives information from the
Client about a wide range of matters related to the Test and stores
that information temporarily, allowing the Data Assembler 106 to
assemble the data in step 404.
[0122] FIG. 5 illustrates the information that is delivered by the
Client to the Data Receiver 104 and eventually stored by the Data
Assembler in step 404:
1. Test Specification Data
[0123] For each Test that is to be analyzed by the Analysis Module
120 and reported upon by the Display Module 142, the Client may
provide "Test Specification Information" 508. Depending on the
Client's preferences, needs, or abilities, the type of Test
Specification Information 508 provided by the Client may vary:
[0124] The Client may provide the actual form of the Test, or a
facsimile thereof. The Client may choose to provide the Scoring
Guide for the Test, but if not provided by the Client, a Scoring
Guide can be developed by experts skilled in the tested
Subject.
[0125] In addition to the actual form of the Test, the Client may
choose to provide information about the cognitive demands made by
the Test. This information is known as a "Skill Set" and is defined
in more detail in the description of the Analysis Module 120. If
the Client does not provide the Skill Set, the Skill Set will be
generated by the Item-Skill Analyzer 130.
[0126] Finally, in addition to the actual form of the Test and the
Skill Set, the Client may choose to provide complete information
showing the degree of relevance of each Skill in the Skill Set to
each Item on the Test. This information is known as a "Coding
Matrix" and is defined in more detail in the description of the
Analysis Module 120. If the Client provides the Skill Set but not
the Coding Matrix, then the Coding Matrix will be generated by the
Analysis Module 120.
2. Goal Specification Information
[0127] The Client may provide "Goal Specification Information" 510,
namely (1) a list and description of the educational goals in the
subject examined by the Test Items, appropriate to the Level of the
Test, and (2) a list and description of the criteria to be used for
determining Students' mastery of such goals.
3. Student Identifying Information
[0128] The Client may provide "Student Identifying Information" 512
such as Student names, identification numbers, or other personal
information.
4. Student Test Performance Information
[0129] The Client may provide "Student Test Performance
Information" 514 that enables determination (according to the
methods described in the Analysis Module 120) of each Student's
performance with respect to the Skills examined on the Test.
Depending on the Client's preferences, needs, or abilities, this
particular information could be provided in one of three ways: as a
listing of each Student's response to each Item on the Test; as a
listing of each Student's Score for each Item on the Test; or as a
listing of each Student's Score for each Skill examined on the
Test.
[0130] An Item Response is a particular Student's response to
particular Test item. An Item Response may be obtained in the form
of a pencil mark on a bubble sheet; an alphanumeric character in an
electronic file; a physical document bearing the Student's writing,
artwork, computations, or other responses; as a digitized (scanned)
electronic version of such a document; as analog or digital
recordings; or in any other form.
[0131] In addition to this information, the Client also may choose
to provide Student Overall Test Performance Information. This
information is numerical or qualitative measures of Students' level
of accomplishment on the Test as a whole, examples of which are as
follows: [0132] total number of Items answered correctly; [0133]
percent of Items answered correctly; [0134] total number of points
earned, using an internal scoring metric; [0135] percent of points
earned; [0136] rank measures (such as percentiles or quartiles)
based on the above numbers; [0137] scale scores computed from
individual Item performance by means of psychometric models
familiar to those skilled in the art of measuring student test
performance (scale scores being overall numerical measures that
permit comparison of test results from one administration of the
exam to another); and [0138] qualitative evaluations of Students'
performance (such as "Meeting the Standards", "Far Below
Standards", etc.) based on the above numbers. 5. Student
Performance History
[0139] To permit the generation of displays showing trends in
Student accomplishment, Student educational needs, the
effectiveness of educational programs, or similar metrics, the
Client may provide "Student Performance History Information". This
consists of information concerning the performance of Students on
prior Tests, including either Overall Performance information or
Skill-Specific Performance information or both.
6. Student Demographic Information
[0140] To permit the formation of Groups along dimensions of
interest to the Client, the Client may provide Student Demographic
Information 518 about each Student. Such information may include
information such as the Students' age, race, ethnicity, gender,
and/or socioeconomic status.
7. User Information
[0141] In order to enable Users to view the displays produced by
the Display Module 142, the Client may provide "User Information"
519 that uniquely identifies Users. For example, the Client could
provide names, identification codes, titles, and contact
information of the Users.
8. Group Identifying Information
[0142] To permit the formation of Groups along dimensions of School
organization, the Client may provide "Group Identifying
Information" 520. This may include listings of any or all of
classrooms, schools, districts, ad hoc tutoring groups, training
centers, etc.
9. Associative Information
[0143] The Client also may provide information that associates the
various actors and entities that are referred to during the
processing and displaying of Test results. "Associative
Information" 522 includes the correspondence between specific
Students and their respective Groups, hierarchical associations
among Groups and institutions (i.e., which classes are within which
school), and personnel associated with a specific Group for a given
Subject (i.e., which teacher teaches math to a certain class).
[0144] Associative Information may be gathered from many sources,
including without limitation class rosters, school enrollment
lists, human resource files, payroll lists, district personnel
charts, and student course schedules.
10. Display Requirements
[0145] For a given Test, the Display Requirements consist of
specifications of the displays that may be made available to Users.
As will be discussed in more detail with respect to the Display
Module 140, Users may receive displays of data or displays of
instructional material or, most commonly, displays of both.
[0146] As implemented in this system and method, the Client may
approve the particular type of displays it wants the Users to
receive. There are five aspects to this decision: [0147] (1) What
kinds of Test Result information to display in print and
electronically? [0148] (2) What kinds of instructional information
to display in print and electronically? [0149] (3) In what format
to display Test Result and instructional information in print?
[0150] (4) In what format to display Test Result and instructional
information electronically? [0151] (5) Which Users should see which
kinds of information? (Display Requirements typically will vary
from User to User. For example, customarily the Client may want
Users who are teachers to see displays of individual information
about many Students, but the Client may want Users who are Students
to see only information about themselves.)
[0152] According to the terminology used here, the five types of
Display Requirements 524 are as follows: [0153] (1) Data
Requirements for Results Reports [0154] (2) Content Requirements
for Instructional Response Reports [0155] (3) Print Presentation
Requirements [0156] (4) Electronic Presentation Requirements [0157]
(5) User Display Rules Data Requirements for Results Reports
[0158] Data Requirements for Results Reports are text files that
specify what data is needed in order to generate each display that
concerns Student Test Results. For example, it may be desirable to
display all Students' national percentile rank in a given Subject
and Level of a Test. In that case, advantageous data would be the
Students' names, the Subject and Level of the Test, and each
Student's national percentile rank in that Test, in addition to the
explanatory information needed concerning the Student's teacher,
class, and school.
[0159] In another example, the Client may want to display how a
specific Group of Students (as defined by gender, race, or
socio-economic status) performed in a specific Skill. In that case,
it would be advantageous to obtain numerical and qualitative
information about each Student's performance in each Skill, as well
as information that will enable the appropriate Group of Students
to be identified.
[0160] Finally, the Data Requirements for Results Reports may also
include specifications of the particular recommendation statements
that should be made about Students or Groups. These recommendation
statements, of course, can vary by the type of User being
addressed: The types of recommendation statements given to
Educators, for example, may differ from the recommendation
statements given to Students. The Data Requirements for Results
Reports also may explain how the Evaluation Statements should be
presented to Users, and how those statements relate to evaluation
statements that may already be in use by the Users. This
relationship serves as an input to the statistical analysis
performed in the Student Skill Performance Evaluator 134, as shown
in the discussion of "Coarse Graining" therein.
Content Requirements for Instructional Response Reports
[0161] The Content Requirements for Instructional Response Reports
are text files that specify the type of the instructional
information that is to be presented along with the Test Results.
These requirements will vary by User. More details about this
instructional information is included below in the description of
the Content Receiver 108 and the Content Assembler 110.
[0162] For example, the Display Requirements identify whether the
instructional information to be displayed may consist of some
various combinations of the following elements: [0163] Overview of
the Subject at a particular Level. [0164] Definition of a
particular Skill within that Subject, at a particular Level [0165]
Analysis of how the Skill aligns with educational goals, such as
the relevant state and local standards [0166] Common difficulties
that Students face with respect to the Skill [0167] Listing of
related Skills [0168] Instructional strategies that Educators might
use to help Students improve in the Skill, including strategies
organized by difficulty level [0169] Activities that Educators may
use to help Students improve in the Skill, including activities
organized by difficulty level [0170] Activities or games that
Students and/or Parents can use to build mastery of a Skill [0171]
Sample problems in the Skill that Educators may provide to
Students, including problems organized by difficulty level and
problems organized or by subskill within the Skill
[0172] As suggested in this last item, the Content Requirements
specify which types of content need to be organized by difficulty
level. The Client may decide that sample problems should be
organized by difficulty level so that they can be distributed to
Students appropriately according to their needs. On the other hand,
the Client may decide that instructional overviews in a Skill
should not be specific to a difficulty level, so that teachers can
use that information in instructing their class as whole.
[0173] It is important to understand that different Users may
desire different types of instructional information: For example,
teachers may benefit from lesson plans; an administrator may
benefit from materials useful for training teachers to better
understand Skills; Parents may benefit from materials that
recommend books appropriate to a Student's interest and level of
performance; and Students may benefit from games that build mastery
of Skills.
[0174] Also, the Content Requirements approved by the Client may
vary by Skill: For example, instructional material that is used to
teach Skills in math is typically structured quite differently than
the instructional material used to teach Skills in social
studies.
[0175] Finally, it is important to note that a Client may approve
multiple Content Requirements even for a given Skill, in order to
allow the Print Displayer 150 and the Electronic Displayer 152 to
show Students different views depending on Students' needs. For
example, the Content Requirements for one type of Users (i.e.,
Students) with respect to a given Test could specify two
information structures: one for sample problems and for fun
activities. In such a scenario, the Content Requirements also will
include decision-rule information about which Students should see
which type of information.
Print Presentation Requirements
[0176] Print Presentation Requirements are electronic text files
and prototypes that together specify the printed document or
documents that a particular type of User will receive. For example,
the Print Presentation Requirements may specify that a Student will
receive a 2-page document, one side of which is a form letter to
the Student's parent from the principal and the other side of which
is a "Skill Profile Table", where the precise data involved in that
table is specified in the Data Requirements.
[0177] The Print Presentation Requirements include all necessary
specifications to transform the raw elements of the Test Results
and instructional information into printed documents for the Users.
These specifications, for example, will include the colors and
styling to be used on the documents, the page length, and the
location of the various displays of Test Result and instructional
information.
Electronic Presentation Requirements
[0178] Analogous to the Electronic Presentation Requirements, the
Electronic Presentation Requirements are electronic text files and
prototypes that specify all of the types of electronic displays
that a particular type of User will see, and under what
circumstances the User should see those displays. In the case of a
website that a User will access, the Electronic Presentation
Requirements may include all the necessary elements (such as color,
navigation elements, etc.) that would allow one skilled in the art
of web design to create a website that shows all of the information
in the manner specified.
User Display Rules
[0179] The final aspect of the Display Requirements that may be
approved by the Client is the "User Display Rules", which are
electronic text files containing descriptions of which types of
information is to be seen by which Users. There are User Print
Display Rules and User Electronic Display Rules, each for the
respective display medium.
[0180] User Display Rules simply define what information particular
types of Users are permitted to see: For example, the User Print
Display Rules could specify that a local job training coordinator
should see a printed document that describe the performance of all
job training classes at a job training institute, while a regional
job training coordinator should see a printed document that
describes the performance of a all job training classes in that
region.
[0181] In the school context, User Display Rules may specify the
year of the Test Results to be shown to a teacher. For example, in
typical cases, 3rd grade Students typically take exams in the late
spring before school ends. A 4th grade teacher in the fall should
see Students' results based on the new class organization for that
school year.
[0182] Also, User Display Rules can specify more detailed criteria.
In particular, User Electronic Display Rules may specify a high
degree of customization of displays on a website, given the dynamic
nature of that medium: For example, one User Electronic Display
Rule could be that if the User is a Parent of a Student who
performed at or below a certain overall score on a Test, the User
should see an electronic display that warns them that their
Students may be facing significant academic challenges. Another
User Electronic Display Rule could be that a User who is a tutor or
teacher should see an electronic display of a complete packet of
Instructional Response Materials that would be made available with
respect to each Student whose Test Results are displayed on the
website.
[0183] Physically, step 402 may be accomplished by means of the
Data Intake Operator, who could be an individual or a computer
operating according to a script. With respect to the receipt of
electronic information, the Data Intake Operator may operate the
Data Intake Computer 206, which is connected to the Internet 202
via a firewall 204, and can obtain information from the Client in
methods that are familiar to those skilled in the art, such as
e-mail of tables, File Transfer Protocol, or other means.
[0184] After the Data Intake Computer 206 receives the electronic
information, it may store this information internally or in the
attached Flat File Storage mechanism 208, which may store
information using a file system, a relational database, or
both.
[0185] With respect to physical information, such as actual Test
forms and Scoring Guides, the Data Intake Operator may this
information by familiar methods, such as regular mail or personal
delivery. The Data Intake Operator simply holds this information
for use by the Data Assembler 106.
[0186] When step 402 is complete, the Data Intake Operator notifies
the Intake Operator.
2.B. Data Assembler
[0187] Once the Intake Operator is notified that the Data Receiver
has completed processing, the Intake Operator notifies the Data
Assembler Operator. The Data Assembler 106 takes the information
that had been stored temporarily by the Data Receiver 104 and,
using methods known to those skilled in the art, assembles and
stores the data in the Information Repository 108 so that it can be
retrieved efficiently by the various components of the Analysis
Module 120 that rely upon that information.
[0188] In most cases, the data received by the Data Receiver 104
will be in electronic form, such as Student data files that include
Student names and Test Result information. With respect to this
information, the Data Assembler 106 uses methods familiar to those
skilled in the art to clean the information; remove records that
are incomplete, duplicate, or inconsistent; and store the
information, either in data tables that reside in a computerized
database or in electronic data files that reside in a computer
directory structures.
[0189] The Data Assembler 106 also stores system-specific
identification numbers for potential Users of the system, including
randomized passwords that can be utilized by Users to access
electronic displays produced by the Electronic Display 152. This
information is known as "User Access Information" and is discussed
more fully in the discussion of the operation of the Electronic
Display 152 below.
[0190] In other cases, the data received by the Data Receiver 104
will be in paper form, such as actual physical specimens of the
form of the Test. In that case, the Data Assembler 106 may simply
store the physical documents within the Information Repository 114,
in a way that facilitates secure storage and efficient retrieval of
these documents later on.
[0191] Physically, step 404 may be accomplished by means of the
Data Intake Operator as follows. With respect to the electronic
information, the Data Intake Operator operates the Data Intake
Computer 206 and employs data cleansing and extraction using
commercially available products such as Ascential Software.TM.
DataStage XE and Informatica PowerCenter.RTM. to read information
from the Flat File Storage mechanism 208; remove incomplete,
duplicate, or inconsistent information; and store it in organized
files within the Data Server 214, which is part of the Information
Repository 210.
[0192] The Data Intake Operator can generate the User Access
Information by means of software products offered commercially or
as shareware, which produce random passwords according to protocols
tested for security.
[0193] The Data Server 214 typically runs on a UNIX operating
system or other commonly used system. The Data Server 214 typically
may be built using a SQL-based database management system (DBMS)
such as Oracle or other type of DBMS.
[0194] With respect to physical documents that may be received by
the Data Receiver 402, the Data Assembler Operator organizes and
stores such documents in the Physical Document Storage 212 for the
purpose of enabling efficient retrieval in the future.
[0195] When step 404 is complete, the Data Assembler Operator
notifies the Intake Operator.
[0196] 2.C. Content Receiver
[0197] As data is received by the Data Receiver 104 and assembled
by the Data Assembler 106 for storage in the Information Repository
114, a concurrent process occurs whereby educational content is
received and assembled by the Content Receiver 106 and the Content
Assembler 108 respectively.
[0198] At the same time that the Intake Operator notifies the Data
Receiver Operator to activate the Data Receiver 104 in step 402,
the Intake Operator also notifies the Content Receiver Operator to
activate the Content Receiver 108 in step 406.
[0199] The Content Receiver 108 receives Content Information from
the Client applicable to the Subject and Level of the Test and
stores this information temporarily for use in Step 408 by the
Content Assembler 110.
[0200] Content Information is educational information that can help
guide instructional responses to Student Test Results, and may be
customized according to the User. For example, Content Information
for Educators may consist of lesson plans and sample problems that
they can use with their Students to address their Students' needs.
Similarly, Content Information for Students may consist of
activities, reading passages, and self-quizzes that Students may
use to improve their understanding of a Subject. FIG. 5 shows that
among the items held in the Content Repository 504 are Subject
overviews 566, Skills definitions 568, classroom activities for
teaching Skills 570, lesson plans for teaching Skills 572, and
sample problems in Skills 574. This is not an exhaustive list.
[0201] Physically, step 406 may be accomplished by means of the
Data Intake Operator, who could be an individual or a computer
operating according to a script, as follows. The Data Intake
Operator operates the Content Management Computer 224, which may be
connected to the Internet 202 via a firewall 204, and which may
obtain information from the Client in methods that are familiar to
those skilled in the art (such as through FTP, e-mail, and other
means). After the Content Management Computer 224 receives the
Content Information, it may store this information in the attached
Content Feed Storage mechanism 226. When step 406 is complete, the
System Operator is notified and activates the Content Assembler 106
in step 408.
[0202] In certain cases, the Content Information received by the
Data Receiver 104 may be in paper form, such as actual physical
specimens of lesson plans. In that case, the Data Receiver 104 may
convert the paper images into electronic text, using methods
familiar to those skilled in the art such as advanced scanning
techniques, or, if such techniques are unavailable, using human
typists and a word processing program. This electronic text is then
stored in the attached Content Feed Storage mechanism 226.
[0203] When step 406 is complete, the Content Receiver Operator
notifies the Intake Operator. 2.D. Content Assembler
[0204] Once the Intake Operator is notified that the Content
Receiver has completed processing, the Intake Operator notifies the
Content Assembler Operator. The Content Assembler Operator
activates the Content Assembler 110 in step 408. The Content
Assembler 110 takes the information that had been stored
temporarily by the Content Receiver 406 and, using methods known to
those skilled in the art, assembles and stores the data in the
Content Repository 116 so that the information can be retrieved
efficiently by the Content Manager 140 during operation of the
Analysis Module 120.
[0205] The object of the Content Assembler 110 in step 408 is to
organize all of the Content Information received by the Content
Receiver 406, and store it electronically in the Content Repository
116. The information held in the Content Repository 116 may be
organized by Skill and Level within a given Subject, either
according to the judgment of the Intake Operator or by means of
electronic tagging of the material by the Client prior to
delivery.
[0206] Physically, step 408 may be accomplished by means of the
Content Assembler Operator in the following way. The Content
Assembler Operator operates the Content Management Computer 224
and, employing common data extraction methods and sorting methods
as well as organizational judgment, assembles the information from
the Content Feed Storage mechanism 226 and store it in files on the
Core Content Server 232 within the Content Repository 230.
[0207] In order to facilitate the work of the Content Manager 140
of the Analysis Module 120, these materials may be organized in
different files to the extent possible by Subject, Level, and
Skill. However, the Content Manager 140 will organize this material
more systematically on the basis of the Skill definitions arrived
at during the processing of the Analysis Module 120.
[0208] When the processing of the Content Assembler 110 is
completed, the Content Assembler Operator notifies the Intake
Operator.
[0209] At the point when both the Data Assembler Operator and the
Content Assembler have notified the Intake Operator that their
respective processes are completed, the Intake Operator notifies
the System Operator that the processing of the Intake Module 102 is
completed. At this point, the System Operator notifies the Analysis
Operator to activate the Analysis Module in step 304.
3. Data Storage Module
[0210] The Data Storage Module 112 is an integral module in this
system, because it facilitates the information transfer among
different system components. The precise steps involved are
detailed in the flowcharts of the various components. It is useful,
nonetheless, to provide an overview of the three components of the
Data Storage Module 112: the Information Repository 114, the
Content Repository 116, and the Report Repository 118.
3.A. Information Repository.
[0211] The first component of the Data Storage Module 112 is the
Information Repository 114. As described in step 404, the Data
Assembler 404 outputs information to the Information Repository
114.
[0212] Also, as will be described more fully below, various
components of the Analysis Module 120 read information from, and
write information to, the Information Repository 114. In this way,
the Analysis Module 120 can store intermediate and final
calculations that express Test Results for Students and Groups in
ways that are meaningful for the Users.
[0213] Finally, as part of the functioning of the Display Module
142, the Results Report Generator 146 accesses information from the
Information Repository 114 in order to assemble Instructional
Response Reports, which are then stored in the Report Repository
118.
[0214] FIG. 5 shows that the Information Repository holds many
different data types:
[0215] The following is data that may be stored in the Information
Repository 502 if the Client does not provide full information
about which Skills are examined on a given Test: Item Scores for
each Student 526, Skill Sets 428, Skill Organizations 530,
Skill-Item Tables 532, Items Assessing a Skill 534, Skill-Specific
Item Scores for each Student 536, Item Score Arrays for each
Student 538, Fall-off Ratios 540, Raw Coding Matrices 542, Refined
Coding Matrices 544, Quality Rating Information for Refined Coding
Matrices 546, Student Skill Scores 548, and Obtained Skill Scores
for each Skill 550. In other cases, the Client will provide some
but not all of the above information.
[0216] In addition, the following information is calculated and
stored in the Information Repository 502 by the Analysis Module
120: Student Evaluation Statements by Skill 552; Group Skill Scores
554; Group Evaluation Statements 556; Pedagogical Constraints 558;
and Summary Statistics 562.
[0217] 3.B. Content Repository
[0218] The second component of the Data Storage Module 112 is the
Content Repository 116. Initially, the Content Repository 116 holds
instructional materials developed prior to activation of the system
and received via the Content Receiver 108 in step 406. As described
more fully below, the Content Manager 140 then organizes and
customizes the materials in the Content Repository 116.
[0219] FIG. 5 shows some of the types of instructional materials
that may be held in the Content Repository 504: Subject overviews
566, Skills definitions, 568, classroom activities for teaching
Skills 570, lesson plans for teaching Skills 572, and sample
problems in Skills 574.
[0220] As part of the functioning of the Display Module 142, the
Instructional Response Report Generator 148 accesses information
from the Content Repository 116 in order to assemble Instructional
Response. Reports, which are then output to the Report Repository
118. This is not an exhaustive list.
[0221] 3.C. Report Repository
[0222] The third component of the Data Storage Module 112 is the
Report Repository 118. FIG. 5 illustrates the key components held
in the Report Repository 506: namely the Results Reports 576,
Instructional Response Reports 578, Print Templates 580, Electronic
Templates 582, Electronic Document Structures for Results Reports
584, and Electronic Document Structures for Instructional Response
Reports 586. These components are each discussed more fully below
in the description of the Display Module 142. As will be discussed,
the Print Displayer 150 and Electronic Displayer 152 access these
reports and templates in order to produce their respective displays
for Users.
4. Analysis Module
[0223] FIG. 1 is a block diagram showing the relation of the
Analysis Module 120 to the rest of the System.
[0224] FIG. 6 is a flowchart showing the functioning of the
Analysis Module 120
Overview of the Analysis Module
[0225] A great deal of analysis must be done in order to turn raw
assessment data into useful information for Educators, Parents, and
Students. Much of this work is accomplished by the Analysis Module
120.
[0226] As described below, the Analysis Module 120 performs a
substantial number of calculations, the results of which are stored
in the Information Repository 114. The precise type of data
outputted by the Analysis Module 120 depends on the extent to which
data is not otherwise inputted into the System by the Client.
[0227] FIG. 5 represents a block diagram of the Data Storage
Module, including the Information Repository 502 and its
contents.
[0228] If the Client provides no information with respect to the
Skills that are examined on the Test, then the Analysis Module 120
must calculate the following information and output it to the
Information Repository 114: Item Scores for each Student 526, Skill
Sets 428, Skill Organizations 530, Skill-Item Tables 532, Items
Assessing a Skill 534, Skill-Specific Item Scores for each Student
536, Item Score Arrays for each Student 538, Fall-off Ratios 540,
Raw Coding Matrices 542, Refined Coding Matrices 544, Quality
Rating Information for Refined Coding Matrices 546, Student Skill
Scores 548, and Obtained Skill Scores for each Skill 550. In other
cases, the Client will provide some but not all of the above
information.
[0229] In addition, the Analysis Module 120 typically calculates
the following information and outputs it to the Information
Repository 114: Student Evaluation Statements by Skill 552; Group
Skill Scores 554; Group Evaluation Statements 556; Pedagogical
Constraints 558; F Scores, P Scores, and A Scores by Skill 560; and
Summary Statistics 562.
[0230] The Analysis Module 120 accomplishes several discrete tasks.
First, in those cases where the Client does not provide detailed
information about the particular Skills examined on the Test, the
Analysis Module 120 analyzes the Test to generate such a list of
the Skills examined as well as a detailed chart concerning the
degree to which each Item on the Test assesses each Skill. This
task is accomplished by an algorithmic process explained below and
is based on analysis of the Test itself, as well as Student
response data.
[0231] Second, the Analysis Module 120 calculates various
quantitative and qualitative indicators of performance and
recommendations for individual Students and Groups within the
Skills that have been tested. After determining these indicators,
the Analysis Module 120 stores these indicators in the Information
Repository 114 so they can be displayed to Users via the Display
Module 142.
[0232] Finally, by activating in step 632 the Content Manager 140,
the Analysis Module 120 generates appropriate instructional
material, organized by skill and difficulty, so that this material
can be presented to Educators, Parents, and Students in an
integrated manner with the Test Result information. This
instructional material will enable them to act effectively in
response to the Test Result information.
[0233] As detailed in the description of its components and
subcomponents, the Analysis Module 120 identifies and circumvents
many of the deficiencies in the existing methods of processing and
displaying Test Results. Later in this document is a detailed
description of the system and method of each component of the
Analysis Module 120. By way of introduction, however, what follows
is an outline of the steps followed by the Analysis Module 120, as
well as key features of this system and method:
1. Responding Flexibly to Different Levels of Information from the
Client, in Order to Identify How STUDENTS Performed on Particular
Skills
[0234] The core of the Analysis Module 120 is a determination of
how Students have performed on particular Skills examined on the
Test. Importantly, the Client may provide different types of
information about Students' Skill performance. This system and
method are designed to respond appropriately depending on the level
of information provided, with the Analysis Module 120 filling in
gaps where required.
[0235] The first step of the Analysis Module 120 is simply to
determine in step 602 whether the Items of the Test are scored
already. If not, the Item Scorer 122 is activated in step 604 to
produce Student Scores in all of the Items of the Test.
[0236] The next steps of the Analysis Module 120 respond to much
more complex scenarios regarding the extent of information provided
by the Client. At one extreme, the Client may not provide (and may
not even possess) any information on which Skills are being Tested.
If in step 606 that is the case, then the Analysis Module 120
activates in step 622 the Item-Skill Analyzer 130, which (as
described below) generates, as its final output, a list of the
Skills examined on the Test (also known as the "Skill Set") and a
numerical matrix that describes the degree to which each Item
assesses each Skill (also known as the "Refined Coding
Matrix").
[0237] At the other extreme, the Client in step 608 may have
provided all necessary information: namely, the individual
Students' Scores on each particular Skill assessed on the Test. In
that case, the Analysis Module 120 activates the Student Skill
Performance Evaluator 134 directly in step 626, omitting all the
various prior steps.
[0238] Between the two extremes are several different potential
scenarios, all of which are addressed explicitly by the Analysis
Module 120:
[0239] In some cases, as governed by steps 606, 608, and 610, the
Client may provide only the Skill Set and Refined Coding Matrix. In
that case, the Analysis Module 120 activates step 624 the Student
Skill Score Calculator 132, which performs the necessary
calculations to obtain each Students' Score in each Skill.
[0240] In some cases, as governed by steps 606, 608, 610, and 612
the Client may provide only the Skill Set and a Raw Coding Matrix.
In that case, the Analysis Module 120 activates in step 620 the
Coding Matrix Refiner 128, which takes as its input the Raw Coding
Matrix and generates the Refined Coding Matrix (to be used by the
Student Skill Score Calculator 132).
[0241] In some cases, as governed by steps 606, 608, 610, 612, and
614, the Client may provide only the Skill Set and a Skill-Item
Table. In that case, the Analysis Module 120 activates in step 618
the Raw Coding Matrix Generator 126, which takes these as its
inputs and generates the Raw Coding Matrix (to be used by the
Coding Matrix Refiner 128).
[0242] Finally, in some cases, as governed by steps 606, 608, 610,
612, and 614, the Client may provide only the Skill Set. In that
case, the Analysis Module 120 activates in step 616 the Skill-Item
Table Generator 124, which takes the Skill Set as its input and
generates the Skill-Item Table (to be used by the Raw Coding Matrix
Generator 126).
[0243] As shown above, these components enable the system to
respond flexibly and comprehensively, regardless of the extent of
the Skill information that is provided by the Client. Each of these
various components will be detailed more precisely below.
2. Generating Sound Skill Categories
[0244] One important feature of the Item-Skill Analyzer 130 is that
it produces a list of the Skills examined on the Test. In
particular, the system and method employed in the Item-Skill
Analyzer 130 result in a list of Skills that are pedagogically and
statistically sound categories in which to report Test Results.
[0245] As background, it should be noted that when Test Results are
displayed to Users, the results for any Student or Group typically
are shown in two ways: aggregated and Skill-specific. For example,
a Student's aggregate results might be reported as the Student's
percent correct on the Test as a whole, or as a percentile that
measures the Student's performance on the Test as a whole against
the performance of other Students. A Student's Skill-specific
results, by contrast, measure only the Student's performance on
those Items on the Test that measure a specific Skill.
[0246] Existing Skill categories used in Test reports are not
useful for the intended recipients of the results. In many cases,
teachers typically do not understand these Skill categories, or
find them overbroad. For example, the organization that reports the
exam results for the New York state testing program uses the math
reporting category "Modeling/Multiple Representation", a term that
typical teachers do not understand. Similarly, the organization
that reports the exam results for the California state testing
program uses the reading reporting category "Functional Reading", a
term that typical teachers find overbroad.
[0247] Reporting in Skill categories that are not useful may occur
because the organization that reports the exam results decides, for
simplicity, to base the reporting categories on educational
language that already exists, such as broad language that the state
has adopted about its educational goals or standards.
Unfortunately, using reporting categories based on such language is
problematic because the categories are then too broad and
non-descriptive, and may or may not faithfully represent the
content of the Test.
[0248] On the other extreme, certain organizations report exam
results using Skill categories that are very narrowly defined
around the precise types of questions asked on the exam. Because
these categories are built around capturing the specific items on a
particular Test itself, these categories are not useful for
teachers or others interested in using the Test Results to guide
instruction more broadly. Indeed, such narrow categories may even
be instructionally misleading, to the extent they do not help
instructors understand the range of Skills that their students may
be expected to master on future Tests.
[0249] Another problem arises when the organization that reports
the results uses a purely statistical method, such as a factor
analysis, to determine which Items to group together for reporting
purposes. Because this approach does not involve systematically the
insight of people with instructional expertise, it often leads to
reporting categories that are not clearly defined or understood by
the recipients of the Test Results.
[0250] By contrast, the Item-Skill Analyzer 130 constitutes a
reliable process for developing appropriate Skill categories for a
given Test. As will be discussed in more detail below, the
Item-Skill Analyzer 130 maximizes the pedagogical insight of the
reporting categories, subject to statistical constraints, in order
to create Skill reporting categories that are both educationally
useful and psychometrically sound.
3. Generating Non-Binary Raw Coding Matrices Based on Instructional
Insight
[0251] Another important aspect of the Analysis Module 120 is that
the Raw Coding Matrix Generator 126 produces Raw Coding Matrices
that are particularly useful because they contain instructionally
nuanced information about the extent to which each Item assesses
various Skills.
[0252] Several well-known approaches in the current art take, as
their inputs, pedagogical estimates of the degree to which each
Skill is examined on each Item (which estimates are here called the
Raw Coding Matrix), in order to output what is known as a Refined
Coding Matrix, which is the best estimation of the same, taking
into account statistical features of Students' responses to the
Test Items.
[0253] However, the pedagogical estimates used as inputs by these
approaches are typically simple `yes/no` judgments about whether a
Skill is or is not assessed on an Item. In other words, the Raw
Coding Matrix typically is in binary form, with each cell
containing 1 or 0, corresponding to a simple `yes/no` decision
about whether or not each Skill is assessed by each Item.
[0254] These approaches would be substantially enhanced by a
reliable process for creating a Raw Coding Matrix that contains
nuanced, non-binary pedagogical estimates about the extent to which
each Skill is assessed by each Item.
[0255] To address this issue, the Analysis Module 120 activates the
Skill-Item Table Generator 124, which instantiates a robust process
of deliberation whereby those with instructional experience
generate a non-binary, instructionally accurate coding matrix
representing qualitatively the degree of relevance of each Skill to
each Item, rather than simply a binary, `yes/no` determination of
relevance. FIG. 9 is flowchart showing the functioning of the
Skill-Item Table Generator 124 in one embodiment.
4. Calculating Student Skill Scores
[0256] After the Item-Skill Analyzer 130 is completed (in the event
that the Client did not provide the Skill Set) or after the Coding
Matrix Refiner 128 is completed (in the event that the Client did
provide the Skill Set but did not provide Student Scores in each
Skill), then the Analysis Module 120 activates in step 624 the
Student Skill Score Calculator 132.
[0257] The Student Skill Score Calculator 132 embodies a system and
method for transforming a Refined Coding Matrix and information
about Students' Scores on all Test Items into a table that reflects
Students' Scores in all Skills. This is accomplished by employing
any of several methods from among the body of psychometric methods
familiar to those skilled in the art.
5. Generating Evaluation Statements (Including Formative
Statements) for Students
[0258] It is important to realize that Users of Test Results do not
simply want to see numerical Scores that describe Students'
performance on the Test. Instead, they want verbal statements
(known here as "Evaluation Statements") about how Students and
Groups performed, both on the Test as a whole and in particular
Skills. In particular, Users of Test Results may want to know about
recommended instructional strategies for addressing weaknesses and
for helping Students and Groups to advance still further in areas
of their strength (here known as "Formative Statements").
[0259] Accordingly, after the Student Skill Score Calculator 132 is
completed, the Analysis Module 120 activates the Student Skill
Performance Evaluator 134. The Student Skill Performance Evaluator
134 assigns Evaluation Statements (including Formative Statements)
to individual Students based on their performance on the Test. The
system and method of the Student Skill Performance Evaluator 134
constitute a repeatable process for assigning Evaluation Statements
to Students based on Students' performance in particular
Skills.
[0260] For example, in one embodiment, the Student Skill
Performance Evaluator 134 assigns each Student one of three
absolute, Formative Statements for each Skill, such as the
following: "Likely to Need Help with Fundamentals", "Likely to Need
Additional Instruction and Practice", and "Likely to Be Ready for
Advanced Work". These Formative Statements (i.e., statements that
recommend one course of action over another) might be applied to
specific Students and Skills, resulting (for example) in
suggestions that one Student work on the fundamentals of
subtraction, while another Student proceed to advanced work in that
subject. These Formative Statements can be very useful for Users,
such as Educators who can take such recommended courses of actions
in the classroom.
[0261] It may also be the case that Client may provide additional
requirements for the types of Evaluation Statements that it wishes
displayed. These requirements, stored in the Information Repository
114 by the Data Assembler 106, can be applied by the Student Skill
Performance Evaluator 134 as well. For example, the Client might
wish that when a Student receives his or her Test Results, the
Student will see that one of a particular group of two Evaluation
Statements (for instance, "Try Harder" and "Good Work") that
corresponds best to the Student's overall performance and to the
Student's particular performance in individual Skills.
6. Leveraging Overall Criterion-Referencing to Enhance
Diagnosis
[0262] One embodiment of the Student Skill Performance Evaluator
134 described below is a system and method for obtaining
Criterion-referenced conclusions about Students' performance in
particular Skills.
[0263] As stated in the Introduction, many Tests are
Criterion-referenced, meaning that Students' performance is
implicitly compared against performance standards in that Subject
that have been established by pedagogical experts. Students who
take a Criterion-referenced Test are typically assigned an
Evaluation Statement based on their overall Test Results, which
statement reflects experts' interpretation of those particular
results. For example, Students who perform poorly on a particular
Criterion-referenced 4th grade math test may be deemed as "Not
Meeting Standards" in 4th grade math, while those who perform well
may be deemed as "Exceeding Standards."
[0264] However, Criterion-referenced tests typically are not used
to generate criterion-based evaluations about a Student's
performance in a given Skill. For example, a Student who takes a
4th grade math test typically is not evaluated against particular
criteria in the various Skills tested. This occurs because
pedagogical experts usually do not establish criteria with respect
to Students' performance on particular Skills that are examined on
the Test, but rather only with respect to Students' overall scores
on the Test.
[0265] Indeed, it would be extremely time-consuming for pedagogical
experts to perform Criterion-referencing in each Skill on a Test,
because that would require the experts to assign evaluative
judgments to the range of actual Student scores attained within
each Skill. Therefore, in the K-12 environment,
Criterion-referencing is usually used only to measure student
overall performance on high-stakes, end-of-year tests. This
situation is unfortunate, because it would be useful for Users to
know a Student's performance on a Skill with respect to
expert-established criteria of mastery.
[0266] Therefore, the Student Skill Performance Evaluator 134, in
one embodiment, instantiates an algorithm to produce
Criterion-referenced information about each Student's performance
in each Skill. Namely, the Student Skill Performance Evaluator 134
leverages (1) the existing Criterion-referencing of Students'
overall scores, and (2) empirical data about the relationship
between Students' overall scores and their scores within a Skill,
in order to produce an approximate Criterion-referencing for
performance within each of the Skills assessed by the Test.
7. Prioritizing Skills Within a Diagnostic Report Based on
Potential for Growth
[0267] Users of Test Results often want to understand how Students
(and Groups) can most efficiently improve their mastery of Skills.
Unfortunately, there does not exist in the prior art a method of
prioritizing Skills within a diagnostic report so that the Users'
attention is drawn first to those Skills in which a Student (or
Group) performed relatively poorly while most Students who took the
Test performed relatively well. Such Skills often present unique
opportunities for rapid progress, and therefore this information
may be helpful for Users, such as for an Educator who is deciding
which Skill to focus on with a Student.
[0268] In one embodiment of the Student Skill Performance Evaluator
134 described below, that component instantiates an algorithm that
prioritizes the Skills for a Student based on the Student's (or
Group's) results on a Test, using potential for growth as a
consideration.
[0269] That application of the Student Skill Performance Evaluator
134 (described more fully below) is advantageous for two reasons:
First, the algorithm employed is relatively straightforward, so
that Users may easily understand the method and feel confident
about its application. Second, the algorithm is not computationally
intensive, so that it can be implemented relatively quickly over
large numbers of Tests, each taken by large number of Students. The
application of this algorithm does not scale drastically as the
number of Students, Items, or Skills increases.
8. Generating Formative Statements for Groups
[0270] After the Student Skill Performance Evaluator 134 has
completed its processing, the Analysis Module 120 activates the
Aggregation Analyzer 136 in step 628. Among other forms of
aggregation, the system and method embodied in the Aggregation
Analyzer 136 can obtain appropriate recommendations, or "Formative
Statements" for Groups of Students.
[0271] As might be expected, Educators would like to know how to
approach instructing their class based on its performance on a
Test. However, there does not exist in the art an adequate method
of aggregating individual Students' Formative Statements over a
Group in order to generate a single Formative Statement for the
Group.
[0272] An obvious approach to aggregating Formative Statements
might proceed as follows: (1) calculate a single average Score for
the Group; (2) imagine that this score represents the score of a
single Student; and then (3) assign the Group the same Formative
Statement that a Student would receive, had he or she received the
Group average Score.
[0273] However, an approach based on averaging will not be
appropriate for determining the best way to address the needs of a
Group collectively. This is because averaging does not take into
account the varying degree to which the different actions
recommended by different Formative Statements will affect different
individual Students.
[0274] The Aggregation Analyzer 136 in one embodiment described
below offers a superior approach. This embodiment of the
Aggregation Analyzer 136 instantiates an algorithm that aggregates
individual Formative Statements that have been offered for
Students, in order to generate a single Formative Statement for the
Group as a whole. In this embodiment, the particular algorithm
functions by selecting the Formative Statement for the Group that
corresponds to the instructional action that maximizes the
educational benefit accruing to all of the Group members.
9. Maintaining Diversity in Aggregation Across Larger and Larger
Groups
[0275] In some applications, the goal of aggregating the Evaluation
Statements of a Group is not to recommend a single action to be
pursued with the Group as a whole, but, rather, simply to give a
sense of which Evaluation Statement best typifies the Group. Here
again, though, averaging is impractical. For example, if the
Evaluation Statements range from "Poor" to "Intermediate" to
"Good", and if the tested population contains Students with
well-distributed variations in their Scores, then in practice, as
Students are aggregated into larger and larger Groups, those
Groups' Evaluation Statements tend, under an averaging mechanism,
to converge rapidly toward "Good". Thus, averaging gives rise to a
regression to the mean in which all Groups of large enough size
(more than 10 Students or so) receive very similar information,
even though their data may be meaningfully different.
[0276] In one embodiment described below, the Aggregation Analyzer
136 instantiates an algorithm that measures the central tendency of
individual Students' Evaluation Statements in a manner that resists
such regression toward the mean.
10. Generating Numerical Data About the Performance of Students and
Groups
[0277] Users of Test Results obviously want to see a wide of range
of data that reflects the performance of Students on the Test.
Therefore, after the Aggregation Analyzer 136 is completed, the
Analysis Module 120 activates the Statistics Summarizer 138. Thus,
the Statistics Summarizer 138 takes as its input the specific types
of displays sought by the Client and produces the appropriate
tables of data for individual Students and Groups.
[0278] For example, based on the types of displays sought by the
Client, the Statistics Summarizer 138 might calculate trends in
Students' overall percentage correct across past administrations of
similar Tests. Similarly, the Statistics Summarizer 138 might
calculate the percentage of students of each race (or by other
category such as gender, socioeconomic class, etc.) who have been
deemed to meet that state's educational standards in the Subject of
the Test.
11. Instructional Material That Truly Assist Users
[0279] The Content Manager 140 ensures that instructional materials
are organized in the same Skills assessed on the Test; offer Users
instructional responses in each Skill that are appropriate for
Students with varying levels of mastery; and are organized within
each Skill in the same categories, or mastery levels, that are used
to describe Students' performance in the Skills tested. This is
highly effective in encouraging Users to act on the Test Results to
address their Students' particular needs.
Preliminary Definitions and Notation
[0280] The Analysis Module 120 may process many Tests for any given
Client as part of any single Reporting cycle. For purposes of
illustration, however, the description of the Analysis Module 120
often focuses on a single Test for the sake of clarity.
[0281] The mathematical definitions of many terms are included in
the Technical Glossary. As a preliminary matter, however, it is
important to note the following terms: The number of Items on this
Test is denoted n. A fixed but arbitrary ordering is assigned to
these n Items; the i.sup.th Item of the Test may then be denoted
.quadrature..sub.i. The full set of Items on the Test is denoted
T={.quadrature..sub.1, .quadrature..sub.2, . . . ,
.quadrature..sub.n}. The symbol T denotes the Test itself. As a
matter of convention, the index i will always range over the values
1, 2, . . . , n. Thus, any quantity with a subscript i refers to a
particular Item on Test T. The number of Students taking the Test
is denoted p. As a matter of convention, the index j will always
range over the values 1, 2, . . . , p. Thus, any quantity with a
subscript j refers to a particular Student taking Test T.
[0282] Certain typographical conventions are also employed here:
Italic lower-case letters (as x) represent numbers. Boldface
lower-case letters (as v) represent vectors of numbers, or vectors
of other entities (such as Skills, Evaluation Statements, etc.).
Boldface upper-case letters (as V) represent a set of vectors
defined in some particular way.
[0283] Depending on the nature of the displays that are produced
(different displays carry different statistical demands), the
number of Students taking the Test should be relatively large. For
most displays, p.gtoreq.1000 or so is required.
Functioning and Physical Instantiation of the Analysis Module
[0284] The Analysis Module 120 may consist physically of (a) an
Analysis Module Operator (human or computer) that follows the
flowchart FIG. 6 to operate the Analysis Module 120 when notified
to do so in step 304 by the System Operator; (b) one or more
Psychometric Analysis Computers 218 with network access to one or
more application servers 220 running commercial analysis software
(such as SASS, SPLUS.RTM., Mathematica.RTM., MULTILOG, etc.), which
computers operate various components of the Analysis Module 120
and/or execute algorithms implemented by these components; and (c)
a Code Library 222, which is simply an electronic file storage
system (such as a hard drive) for storing custom-written code as
necessary for implementing different components of the Analysis
Module 120.
[0285] Each component of the Analysis Module 120 is operated by an
Operator as well (human or, in many embodiments, computer). When
the Analysis Operator notifies the Operator of a particular
component within the Analysis Module 120, that component is
activated, performs the processing required, and then notifies its
Operator that it has finished. The Operator transmits this
information to the Analysis Module Operator, which requires this
flow-control information to implement the flowchart FIG. 6.
[0286] Each component of the Analysis Module 120 draws inputs from
the Information Repository 114 and/or Content Repository 116;
applies computer algorithms and/or prompts human expertise to
generate new information from the inputs; writes the output
information to the Information Repository 114 in electronic form;
and notifies its Operator that processing is complete.
[0287] More specifically, using the terminology employed here, the
Analysis Module 120 uses the information held in the Information
Repository 112 to generate: [0288] a Skill Set; [0289] Scores and
Evaluation Statements for each Student for each Skill in the Skill
Set; [0290] Scores and Evaluation Statements for each Group for
each Skill in the Skill Set; and [0291] Statistical Summary
Information for Students.
[0292] These products, described more fully below, are written to
the Information Repository 114 so they can be assembled into
Reports by the Display Module 142.
[0293] In addition, the Analysis Module 120 uses the information
held in the Content Repository 116 to generate content suitable for
Educators, Parents, and Students that is broken down by Skill and
mastery level, corresponding to the Skills and mastery levels used
in reporting the Test Results themselves. This content is written
to the Content Repository 118 so it can be assembled by the Display
Module 142.
[0294] The following individual components can all be activated
during the operation of the Analysis Module 120: [0295] Item Scorer
122 [0296] Skill-Item Table Generator 124 [0297] Raw Coding Matrix
Generator 126 [0298] Coding Matrix Refiner 128 128 [0299]
Item-Skill Analyzer 130 [0300] Student Skill Score Calculator 132
[0301] Student Skill Performance Evaluator 134 [0302] Aggregation
Analyzer 136 [0303] Content Manager 140
[0304] The above components function as follows.
4.A. Item Scorer
[0305] As indicated in step 602, the Item Scorer 122 is activated
in step 604 in the event that the Client does not provide each
Student's numerical Score for each Item. In that case, the Item
Scorer 122 uses the Scoring Guide for the Test to assign each
Student an Item Score for each Item on the Test.
[0306] For each Item, the Item Score for that Item is a number that
represents a given Student's level of accomplishment on a
particular Test item. The Item Score Range for a Test Item is a
number that represents the range of Item Scores that are possible
in that Item. For a multiple-choice Item, the Item Score for the
Item is typically 0 or 1. For constructed-response Items (such as
essay questions, for example), the Item Score typically ranges from
0 to some integer greater than 1. Larger Item Scores represent
greater levels of accomplishment on the Item in question.
[0307] The Item Scorer 122 assigns each Student a list of Item
Scores, one Item Score for each Item on the Test. This list of Item
Scores is called the Student's Item Score Vector. The Item Scorer
122 records these Item Score Vectors in the Information Repository
114, where they can be used by the other components of the Analysis
Module 120 later in the analysis process.
[0308] The system used to generate Item Scores will depend on the
nature of the Test Items and of the Scoring Guide.
[0309] For example, if all of the Test Items are in multiple choice
format, then the Item Scorer 122 can be instantiated in the obvious
way as a computer program running on the Psychometric Analysis
Computer 218 (which has network access) that (1) reads Student Item
Responses from the Information Repository 114 in electronic form,
as lists of alphanumeric characters (each Student's list containing
one character representing that Student's response to each Item on
the Test); (2) reads the Scoring Guide from the Information
Repository 114 in electronic form as a list of correct response
characters, one correct response character for each Item on the
Test; (3) assigns the Item Score 0 or 1 to each Student for each
Item, according to whether the Student's Item Response for that
Item matches the corresponding response character in the Scoring
Guide; and (4) writes these Item Scores to the Information
Repository 114 in electronic form, where they constitute the stored
Item Score Vectors for the Students.
[0310] For Test Items in constructed-response format, the Item
Scorer 122 can be instantiated as a rubric-scoring routine, such as
is commonly employed in the art, in which humans with skill in
evaluating Student work in the Subject of the Test are prompted to
input scores for each Item for each Student, by following a Scoring
Guide. (The scorers may evaluate Students' actual Responses, or
facsimiles or reproductions thereof.) Such routines typically
involve, in addition to human scorers, human arbitrators who are
prompted to assign final scores whenever human scorers disagree
markedly as to the Score merited by a particular Response.
[0311] In the constructed-response case, the Item Scorer 122
generates Item Scores for each Student in such a way, and then
writes them to the Information Repository 114 in electronic form,
where they constitute the stored Item Score Vectors for the
Students.
[0312] Once the Item Score Vectors for the Students taking the Test
have been written to the Information Repository 114, they can serve
as inputs to the other components of the Analysis Module 120.
[0313] 4.B. Skill-Item Table Generator
[0314] The Skill-Item Table Generator 124 described here is
valuable because in many cases, the Client will provide Students'
scores on Test Items, in addition to a list of the Skills that are
assessed on the Test, but will not provide any information
concerning how those Skills are tested by particular Items. In that
case, i.e., when Decisions 606, 608, 610, 612, and 614 are all
negative, it will be necessary to activate the Skill-Item Table
Generator 124.
[0315] FIG. 9 is a flowchart showing the functioning of the
Skill-Item Table Generator 124 in one embodiment.
[0316] The Skill-Item Table Generator 124 may consist physically of
(a) an Operator (human or computer) that follows the flowchart in
FIG. 9 to operate the Skill-Item Table Generator 124 when notified
to do so in step 616 by the Analysis Operator or in steps 1004,
1010, or 1016 by the Item-Skill Analyzer Operator; and (b) the
Psychometric Analysis Computer 218 (or a human moderator) executing
a programmed routine whose algorithm reproduces that represented
within FIG. 9 and described below. The embodiment below describes a
computer-driven version.
[0317] The programmed routine of the Skill-Item Table Generator 124
prompts a group of human judges, called Pedagogical Operators, to
assess the relevance of a given set of Skills S.sub.T to a given
set of Test Items T. The result of this process is a Skill-Item
Table I(S.sub.T, T) that reflects the detailed judgment of the
Pedagogical Operators as to which Skills are required for the
successful completion of which Items, and in what qualitative
degree. The number of Pedagogical Operators is flexible.
[0318] An important feature of the Skill-Item Table Generator 124
is that it produces nuanced information about the extent to which
each Skill is assessed by each Item. Rather than simply a `yes/no`
decision about whether a Skill is assessed by an Item, the
Skill-Item Table Generator 124 produces qualitative judgments about
the extent to which each Item assesses each Skill.
[0319] Given a Skill Set, which is a list of the Skills required
for a given Test, a Skill-Item Table is a table in which (a)
successive rows of the table correspond to successive Skills in the
Skill Set; (b) successive columns in the table correspond to
decreasing degrees of relevance that a Skill may have for
successfully responding to a Test Item, as judged by the
Pedagogical Operators; (c) cells of the table contain various Item
designations, reflecting the degree to which it is judged that the
designated Item assesses each Skill; and (d) no Item appears more
than once in any given row.
[0320] For a Test T with five Items and Skill Set S.sub.T with four
Skills, a Skill-Item Table might look like the following:
TABLE-US-00001 Items PRIMARILY Items SECONDARILY Assessing this
Skill Assessing this Skill Skill 1 .quadrature..sub.1,
.quadrature..sub.3 .quadrature..sub.4 Skill 2 .quadrature..sub.4,
.quadrature..sub.5 Skill 3 .quadrature..sub.1 Skill 4
.quadrature..sub.3, .quadrature..sub.4, .quadrature..sub.5
.quadrature..sub.2
[0321] According to this Skill-Item Table, Item .quadrature..sub.4
primarily assesses Skills 2 and 4, but only secondarily assesses
Skill 1. Depending on the protocols for the algorithm, the
Skill-Item Table may also contain columns for tertiary relevance,
etc.; or, at the other extreme, it may only contain a single
column. It is not required that every Item appear somewhere in the
Skill-Item Table. This is to allow for the possibility that certain
Items on the Test may be deemed unsuitable for use in providing
Skills diagnosis.
[0322] In general form, the Skill-Item Table Generator 124
functions in the following way (with various embodiments noted): In
step 902, the computer displays to the Pedagogical Operators or
provides to them in printed form a Skill Set, the Test Items, the
Scoring Guide, a written outline of the algorithm to be followed,
and printed guidelines that facilitate careful judging of
Skill-Item relevance. The computer also offers the Pedagogical
Operators access to the actual responses of the Students that took
the test, should they wish to review them during judging.
[0323] In step 903, the computer enters a loop that is indexed by
the control index b; the initial value of the control index is
b=0.
[0324] In step 904, the Pedagogical Operators construct a "Content
Map" C.sub.b in the following way. [0325] 1. The computer cycles
through the Items on the Test, prompting the Pedagogical Operators
to code them in turn as follows. [0326] 1.a. The Pedagogical
Operators are prompted to focus on the first Item in the Test.
[0327] 1.b The Pedagogical Operators are prompted to reach
consensus as to which Skills are assessed by that Item in any
degree whatsoever.
[0328] One Embodiment of 1(b) is as Follows: [0329] 1.b.i The
Pedagogical Operators are prompted to focus on the first Skill in
the Skill Set. [0330] 1.b.ii The Pedagogical Operators are prompted
to vote "yea" or "nay" on the issue of whether or not the chosen
Item assesses the Skill in any degree.
[0331] Votes are tallied by inputting them on a computer keyboard.
[0332] 1.b.iii The computer tallies the votes. If a majority of the
Pedagogical Operators has voted "yea," then the chosen Skill is
entered next to the chosen Item in the Content Map C.sub.0 and
displayed as shown in 1(c) below. [0333] 1.b.iv The computer
repeats 1.b(i)-1.b(iii) for all Skills in the Skill Set. [0334]
1.b.v At the completion of 1.b(iv), it is possible that no Skill
will have been determined to assess the chosen Item. That is, it is
possible that each Skill in the Skill Set will have failed to earn
a majority of "yea" votes for the chosen Item. In this event, the
computer enters those Skills receiving the maximal number of "yea"
votes into the Content Map C.sub.b alongside the chosen Item. If no
Skill receives any "yea" votes, then the computer leaves the row of
the Content Map C.sub.b corresponding to the chosen Item blank.
[0335] 1.c. 1(a) and 1(b) are repeated for all Items on the Test.
As the routine progresses through the Items, the group's
conclusions for each Item are entered into a Content Map, C.sub.b,
which is a table with the following structure. This Content Map, as
well as all Content Maps and Skill-Item Tables constructed during
the functioning of the Skill-Item Table Generator, are visually
displayed to the Pedagogical Operators throughout the process, for
example in separate windows on the computer screen. TABLE-US-00002
C.sub.b Item Skills Assessed by the Item .quadrature..sub.1
.quadrature..sub.2 .quadrature..sub.3 .differential.
[0336] Upon completion of 1(c), step 904 has been completed. [0337]
Also, as noted in step 906, as the routine progresses through the
Items, each Operator is prompted to keep records of any Items he or
she may wish to revisit in step 2 below.
[0338] The Pedagogical Operators may likely want to revisit the
analysis of the Items, as their understanding of the relationship
between the Items and the Skills may have evolved during the
process of carrying out steps 904 and 906. Thus, in steps 908 and
909, the Pedagogical Operators are given the opportunity to
construct a new Content Map, entitled C.sub.b+1, as follows: [0339]
2.a. The group is prompted to focus on the first Item in the Test.
[0340] 2.b. The group is prompted to reach consensus as to whether
the list of Skills assessed by the Item, as represented in Content
Map C.sub.b, should be revised.
[0341] One Embodiment of 2(b) is as Follows: [0342] 2.b.i. The
Pedagogical Operators vote "aye" or "nay" on the issue of whether
the list of Skills Assessing the Item, as represented in Content
Map C.sub.b, should be revised. Votes are tallied by inputting them
on a computer keyboard. [0343] 2.b.ii. The computer tallies the
votes. If "aye" votes form the majority, then the computer reports
that the consensus is that the list of Skills must be revised.
[0344] 2.b.iii. If one or more "aye" votes were cast, but did not
form a majority, then each of those who voted "aye" is prompted to
explain his or her reasoning. Then the vote is repeated as in
2.b(i) and 2.b.(ii). If in this second vote "ayes" form the
majority, then the computer reports that the consensus is that the
list of Skills must be revised. If "aye" votes do not form the
majority, then the computer reports that the consensus is that the
list of Skills must not be revised. [0345] 2.c. If the consensus is
that the list of Skills should be revised, then the experts reach a
new consensus as to which Skills are assessed by the Item in any
degree whatsoever, as in 1(b) above. [0346] 2.d. The computer
repeats 2(a)-2(c) for all Items on the Test. As the group
progresses through the Items, the computer enters the group's
conclusions for each Item into a new Content Map C.sub.b+1. Also,
as noted in step 909, as the routine progresses through the Items,
each Operator is prompted to keep records of any Items he or she
may wish to revisit in further iterations.
[0347] Upon completion of 2(d), step 908 has been completed.
[0348] The computer next evaluates Decision 912, by comparing the
(b+1).sup.st Content Map, C.sub.b+1, to the b.sup.th Content Map,
C.sub.b.
[0349] If the two content maps are identical, then in step 916 the
computer designates C.sub.b as the Content Map, C.
[0350] If the two content maps are not identical, then the computer
evaluates Decision 912, by comparing the value of b to the value of
a `kickout` number, N.sub.max, which is hard-coded into the routine
in advance.
[0351] If b<N.sub.max, then in step 910 the computer increments
the value of the loop control index b by unity and repeats steps
904, 906, 908, 909.
[0352] If b>N.sub.max, then in step 918 the computer prompts the
group to select by consensus the best Content Map C from among all
of the Content Maps C.sub.0, . . . , C.sub.b. This allows earlier
Content Maps to re-enter the discussion in case the group feels it
has gone astray in its revisions.
[0353] One Embodiment of Step 918 is as Follows: [0354] 3.a. Each
Content Map is subjected to a vote in turn. The Pedagogical
Operators are prompted to vote "aye" or "nay" on the question
whether the given Content Map is acceptable. The votes are input
via keyboard, and the computer tallies the votes. [0355] In the
logically possible (but extremely unlikely) event that no Content
Map recieves any "aye" votes, the Pedagogical Operators are
prompted to vote once again, with the constraint imposed that each
Pedagogical Operator must vote for at least one Content Map. [0356]
3.b. From among the Content Maps receiving the maximal number of
"aye" votes, the computer selects one at random as the consensus
choice C.
[0357] In step 920, the computer `transposes` the Content Map C in
the obvious way to produce a Skill-Item Table I.sub.binary with one
column, as illustrated here: the Content Map C TABLE-US-00003 C
Item Skills Assessed by the Item .quadrature..sub.1 S.sub.1,
S.sub.3 .quadrature..sub.2 S.sub.4 .quadrature..sub.3 S.sub.1,
S.sub.4 .quadrature..sub.4 S.sub.1, S.sub.2, S.sub.4
.quadrature..sub.5 S.sub.2, S.sub.4
[0358] becomes `transposed` as: TABLE-US-00004 I.sub.binary Skill
Items Assessing this Skill S.sub.1 .quadrature..sub.1,
.quadrature..sub.3, .quadrature..sub.4 S.sub.2 .quadrature..sub.4,
.quadrature..sub.5 S.sub.3 .quadrature..sub.1 S.sub.4
.quadrature..sub.2, .quadrature..sub.3, .quadrature..sub.4,
.quadrature..sub.5
[0359] In step 922, the Pedagogical operators are prompted to vote
on the question whether they wish to refine the Skill-Item Table
I.sub.binary by sorting the Items assessing each Skill into Items
primarily assessing that Skill and Items secondarily assessing that
Skill. The votes are input via keyboard and tallied by the
computer, which then displays the majority decision (breaking ties
at random or by a hard-coded prejudice to one side or another).
[0360] In Decision 924, the computer chooses a course of action
depending on the outcome of the vote. If the Pedagogical Operators
have voted not to refine the Skill-Item Table I.sub.binary, then in
step 926 the computer writes I.sub.binary to the Information
Repository 114, bringing the functioning of the Skill-Item Table
Generator 124 to completion.
[0361] If the Pedagogical Operators have voted to refine the
Skill-Item Table I.sub.binary, then in step 927 the computer enters
a refinement loop indexed by the control index b, which is
initially set to b=0.
[0362] In step 928, the Pedagogical Operators construct a
Primary/Secondary Content Map C.sub.b in the following way. [0363]
4. The computer prompts the Pedagogical Operators to re-examine the
Content Map C, cycling through the Items in turn. [0364] 4.a. The
group is prompted to focus on Item .quadrature..sub.1. [0365] 4.b
Among the Skills in the Content Map C appearing next to Item
.quadrature..sub.1, the group is prompted to reach consensus as to
which Skills are Primarily Assessed by Item .quadrature..sub.1, and
which Skills are Secondarily Assessed by Item
.quadrature..sub.1.
[0366] The group may construe the terms `Primarily` and
`Secondarily` as they wish, or they may allow themselves to be
constrained by guidelines and examples such as those specified
below, which clarify the Pedagogical Operators' role and
standardize their judgments, thereby contributing to an efficient
coding process: [0367] To decide which of two Skills, S.sub.1 and
S.sub.2, is primarily assessed by an Item, and which is secondarily
assessed by it, suppose that a Student has answered the Item
incorrectly, and ask the following question: [0368] If a Student of
this Subject, being tested at this Level, answers this question
incorrectly, then, in your judgment, is the reason much more likely
to be a lack of competence in S.sub.1 or a lack of competence in
S.sub.2? [0369] If the reason is much more likely to be a lack of
competence in S.sub.1, then S.sub.1 is primarily assessed by the
Item and S.sub.2 is secondarily assessed by the Item. [0370] If the
reason is much more likely to be a lack of competence in S.sub.2,
then S.sub.2 is primarily assessed by the Item and S.sub.1 is
secondarily assessed by the Item. [0371] If neither reason is much
more likely than the other, then the Skills are either (1) both
primarily assessed by the Item, or, particularly when other Skills
are assessed by the Item, (2) both secondarily assessed by the
Item.
[0372] Example: TABLE-US-00005 Skills Secondarily Skills Primarily
Assessed Assessed Item by the Item by the Item John has 9 apples.
Mary has 6 Addition - Addition - more apples than John. Bill
Problem Solving Computation has 17 times as many apples
Multiplication - as Mary. How many apples Problem Solving does Bill
have? Multiplication - Computation John has 3 apples. Mary has 9
Addition - Multiplication - times as many apples as John. Problem
Solving Computation Bill has 998 more apples than Multiplication -
Mary. How many apples does Problem Solving Bill have? Addition -
Computation
[0373] One Embodiment of 4(b) is as Follows: [0374] 4.b.i. The
group is prompted to focus on the first Skill appearing next to
Item .quadrature..sub.1 in the Content Map C. [0375] 4.b.ii. The
group is prompted to vote "Primary" or "Secondary" on the issue of
whether Item .quadrature..sub.1 primarily assesses that Skill or
secondarily assesses that Skill. The votes are input via keyboard,
and the computer tallies the votes.
[0376] 4.b.iii. In accordance with the outcome of the vote (ties
being ruled "Primary"), the computer places the chosen Skill in the
appropriate column in a new Content Map C.sub.b like the one shown
below. (The majority vote was "Primary" in this example.)
TABLE-US-00006 C'.sub.0 (shown in progress) Skills Primarily Skills
Secondarily Item Assessed by this Item Assessed by this Item
.quadrature..sub.1 S.sub.1 .quadrature..sub.2 .quadrature..sub.3
.quadrature..sub.4
[0377] 4.b.iv. The computer repeats 4.b(i)-4.b(iii) for all Skills
appearing next to Item .quadrature..sub.1 in the Content Map C.
[0378] 4.c. The computer repeats 4(a) and 4(b) for all of the Items
in T. As the group progresses through the Items, the computer
enters the group's conclusions for each Item into the Content Map
C.sub.b as in the following example: TABLE-US-00007 C'.sub.b Skills
Primarily Skills Secondarily Item Assessed by this Item Assessed by
this Item .quadrature..sub.1 S.sub.1, S.sub.3 .quadrature..sub.2
S.sub.4 .quadrature..sub.3 S.sub.1, S.sub.4 .quadrature..sub.4
S.sub.2, S.sub.4 S.sub.1 .quadrature..sub.5 S.sub.2, S.sub.4
[0379] Upon completion of 4(c), step 928 has been completed. [0380]
Also, as noted in step 930, as the routine progresses through the
Items, each Operator is prompted to keep records of any Items he or
she may wish to revisit in further iterations.
[0381] The Pedagogical Operators may likely want to revisit the
analysis of the Items, as the group's own understanding of the
terms `Primary` and `Secondary` may have evolved during the process
of carrying out steps 928 and 930. Thus, in steps 932 and 933, the
Pedagogical Operators are given the opportunity to construct a new
Primary/Secondary Content Map, C'.sub.b+1, by cycling through the
Items in turn as follows: [0382] 5.a. The group is prompted to
focus on Item .quadrature..sub.1. [0383] 5.b. The group is prompted
to reach consensus as to whether the sorting of Skills Assessed by
the Item, as represented in Content Map C'.sub.b, should be
revised.
[0384] One Embodiment of 5(b) is as Follows: [0385] 5.b.i. The
group is prompted to vote "aye" or "nay" on the issue of whether
the sorting of Skills should be revised. The votes are input via
keyboard and tallied by the computer. [0386] 5.b.ii. If "aye" votes
form the majority, then the Pedagogical Operators are informed that
the consensus is that the sorting of Skills must be revised. [0387]
5.b.iii. If one or more "aye" votes were cast, but did not form a
majority, then each of those who voted "aye" is prompted to explain
his or her reasoning. Then the vote is repeated. If in this second
vote "ayes" form the majority, then the Pedagogical Operators are
informed that the consensus is that the sorting of Skills must be
revised. If "aye" votes do not form the majority, then the
Pedagogical Operators are informed that the consensus is that the
sorting of Skills must not be revised. [0388] 5.c. If the consensus
is that the sorting should be revised, then the experts are
prompted to reach a new consensus as to which Skills are Primarily
Assessed by the Item and which Skills are Secondarily Assessed by
the Item, as in 4(b) above. [0389] 5.d. The computer repeats
5(a)-5(c) for all Items on the Test. As the routine progresses
through the Items, the computer enters the group's conclusions for
each Item into a new Content Map C'.sub.b+1. Also, as noted in step
933, as the routine progresses through the Items, each Operator is
prompted to keep records of any Items he or she may wish to revisit
in further iterations.
[0390] Upon completion of 5(d), step 932 has been completed.
[0391] The computer next evaluates Decision 936, by comparing the
(b+1).sup.st Primary/Secondary Content Map, C'.sub.b+1, to the
b.sup.th Primary/Secondary Content Map, C'.sub.b.
[0392] If the two content maps are identical, then in step 940 the
computer designates C'.sub.b as the Content Map, C'.
[0393] If the two content maps are not identical, then the computer
evaluates Decision 938, by comparing the value of b to the value of
a `kickout` number, M.sub.max, which is hard-coded into the routine
in advance.
[0394] If b<M.sub.max, then in step 934 the computer increments
the value of the loop control index b by unity and repeats steps
928, 930, 932, 933.
[0395] If b>M.sub.max, then in step 942 the computer prompts the
group to select by consensus the best Content Map C' from among all
of the Content Maps C'.sub.0, . . . , C'.sub.b, in the same manner
as in step 918.
[0396] Next, in step 944 the computer `transposes` the Content Map
C' in the obvious way to form the Skill-Item Table I(S.sub.T,
T).
[0397] The Pedagogical Operators are then prompted in step 945 to
identify the Fall-off Ratio(s) that numerically express the
differences between the qualitative assignments `Primary` and
`Secondary`.
[0398] For example, take the case of a Skill-Item Table that has
columns for "Skills Primarily Assessed by this Item" and "Skills
Secondarily Assessed by this Item". If the Operators express the
Fall-Off Ratio of Primarily Assessed Skills to Secondarily Assessed
Skills as 10:1, then this means that the Operators regard the
so-called Secondary Skills as highly secondary. If the Fall-Off
Ratio were 2:1, however, then this means that the Pedagogical
Operators regard Secondary Skills as relatively important.
[0399] Thus, for example, the computer can prompt each Pedagogical
Operator with a menu of choices, such as the following:
TABLE-US-00008 Reference value In Your Judgment . . . of Fall-Off
Ratio . . . Are Secondary Skills much less relevant 8:1 to the Item
than the Primary Skills? . . . Are Secondary Skills almost as
relevant 1.5:1 to the Item as the Primary Skills are?
[0400] The Pedagogical Operators can input the value of the
Fall-Off Ratio that they feel best captures their judgment, based
on the reference values shown in the table. The computer records
the values .quadrature..sub.t of these input Ratios, and then
calculates the consensus Fall-Off Ratio .quadrature. as the
geometric mean of the input values:
.quadrature.=(.quadrature..sub.1.ident..quadrature..sub.N).sup.1/-
N, where N is the number of Pedagogical Operators. (When the
numbers being chosen are ratios, the geometric mean is a more
sensible measure of central tendency than the arithmetic mean.)
[0401] When the Skill-Item Table includes columns for Tertiary
relevance, etc., then there will be more than one Fall-Off Ratio,
so as to permit comparison between Primary/Secondary Skills,
Secondary/Tertiary Skills, etc.: .quadrature..sub.PS,
.quadrature..sub.ST, . . .
[0402] In step 946, the computer writes the Skill-Item Table
I(S.sub.T, T) and the Fall-Off Ratio(s) to the Information
Repository 114, and then finally in step 948 the computer notifies
the Operator of the Skill-Item Table Generator 124 that processing
is complete.
[0403] For simplicity, the flowchart FIG. 9 does not represent the
logically possible (but extremely unlikely) event in which every
Item fails to have any Skills listed next to it in the initial
Content Map C.sub.0. If this happens, the Client is notified, and a
new Skill Set must be provided. (This circumstance cannot arise
when the Skill-Item Table Generator 124 is being activated as a
subprocess of the Item-Skill Analyzer 130.)
[0404] 4.C. Raw Coding Matrix Generator
[0405] The next component to be described is the Raw Coding Matrix
Generator 126. The Raw Coding Matrix Generator 126 uses the
qualitative correspondences between Skills and Items as represented
in the Skill-Item Table, as well as the Fall-Off Ratio(s) that
encapsulate the strength of the pedagogical distinction(s) between
Primary/Secondary, Secondary/Tertiary, etc., to produce a numerical
matrix that expresses the correspondence between each Item and each
Skill.
[0406] The Raw Coding Matrix Generator 126 may consist physically
of (a) an Operator (human or computer) that activates the Raw
Coding Matrix Generator 126 when notified to do so in step 618 by
the Analysis Operator or in steps 1006, 1012, or 1018 by the
Item-Skill Analyzer Operator; and (b) a computer (or a human
moderator) connected to the network executing a programmed routine
that retrieves a Skill-Item Table from the Information Repository
114, converts it to a numerical Raw Coding Matrix, and then writes
the Raw Coding Matrix to the Information Repository 114. The
embodiment below describes one algorithm for accomplishing this,
assuming for definiteness that the programmed routine is being
carried out by a computer.
[0407] Once the Raw Coding Matrix Generator 126 is activated, the
computer retrieves a Skill-Item Table from the Information
Repository 114. The computer first converts the Skill-Item Table
into a larger table, with one row for each Skill and one column for
each Item. Such a conversion functions in the natural way as
follows: TABLE-US-00009 Items PRIMARILY Items SECONDARILY Assessing
this Skill Assessing this Skill Skill 1 .quadrature..sub.1,
.quadrature..sub.3 .quadrature..sub.4 Skill 2 .quadrature..sub.4,
.quadrature..sub.5 Skill 3 .quadrature..sub.1 Skill 4
.quadrature..sub.3, .quadrature..sub.4, .quadrature..sub.5
.quadrature..sub.2
[0408] B. TABLE-US-00010 Item 1 Item 2 Item 3 Item 4 Item 5 Skill 1
Primarily Not assessed Primarily Secondarily Not assessed assessed
assessed assessed Skill 2 Not Not assessed Not Primarily Primarily
assessed assessed assessed assessed Skill 3 Primarily Not assessed
Not Not assessed Not assessed assessed assessed Skill 4 Not
Secondarily Primarily Primarily Primarily assessed assessed
assessed assessed assessed
[0409] Then, the computer transforms this larger table into a
numerical matrix, with the same number of rows and the same number
of columns as the table, and with a numerical value between 0 and 1
associated with each cell. This value represents numerically the
degree to which a given Item assesses a given Skill, and reflects
prior pedagogical judgments made about the correspondence between
each Item and each Skill, as reflected in their expressed Fall-Off
Ratio(s).
[0410] Mathematically, the computer effects this transformation in
such a way that (a) the sum of the numbers across all the Skills
for a particular Item must equal 1; and (b) the previously
determined Fall-Off Ratio(s) govern the comparative numerical
measures of Skill relevance within a given Item.
[0411] For example, take the above table in the case where the
Fall-Off Ratio stipulates a primary to secondary ratio of 2:1. In
that case, the Raw Coding Matrix would be as shown below. For
example, in the case of Item 4, where Skill 1 is secondarily
assessed and Skills 2 and 4 are primarily assessed, the numerical
assignments of Item 4 to Skills 1, 2, and 4 respectively would be
0.2, 0.4, and 0.4: TABLE-US-00011 Item 1 Item 2 Item 3 Item 4 Item
5 Skill 1 0.5 0 0.5 0.2 0 Skill 2 0 0 0 0.4 0.5 Skill 3 0.5 0 0 0 0
Skill 4 0 1 0.5 0.4 0.5
[0412] (In this embodiment, in the event that an Item has only a
single Secondary coding, that Item-Skill cell is assigned the
number 1. In another embodiment, one could regard an Item with only
a "Secondary" coding as reflecting the Operators' judgment there is
a Skill required for the Item not represented in the Skill Set, in
which case the Item-Skill cell would be assigned 0.33--reflecting
the fact that a non-assessed Skill was primarily assessed).
[0413] This Raw Coding Matrix is sometimes called a Q-matrix,
except that a Q-matrix traditionally contains as matrix elements
only 0's and 1's.
[0414] Once the computer has transformed the Skill-Item Map into a
numerical Raw Coding Matrix by means of a transformation such as
the above, it writes the Raw Coding Matrix to the Information
Repository 114 and notifies the Operator that processing is
complete.
[0415] 4.D. Coding Matrix Refiner
[0416] The Raw Coding Matrix generated by the Raw Coding Matrix
Generator 126 represents a set of careful judgments about which
Skills in the Skill Set are important for successfully answering
each Item on the Test--and, at least qualitatively, in what degree
these Skills are required. These judgments are highly valuable in
generating meaningful diagnostic information based on the results
of a Test.
[0417] However, the question of which Items on a Test "belong
together" is not entirely a question of expert opinion. If judgment
suggests that all of the Items in a particular cluster assess the
same Skill, then it should be the case, for example, that Students
who get one of those Items wrong tend to get many of them wrong.
This is a hypothesis that can be tested empirically by examining
the Student response data. If the correlations between the Items in
the cluster is not high enough, then the original judgment may need
to be modified.
[0418] Those skilled in the art of psychometrics recognize complex
methods for using correlations found in the Student response data
(such as the relationship between a Student's answers on different
sets of Items) to refine the entries in a Raw Coding Matrix in this
way. The Coding Matrix Refiner 128 serves to implement such a
routine, and the matrix resulting from this implementation is
called here the Refined Coding Matrix. Thus, in short, the Coding
Matrix Refiner 128 transforms the Raw Coding Matrix into a Refined
Coding Matrix.
[0419] The Coding Matrix Refiner 128 may consist physically of (a)
an Operator (human in most embodiments) that activates the Coding
Matrix Refiner 128 when notified to do so in step 620 by the
Analysis Operator or in step 1008, 1014, or 1020 by the Item-Skill
Analyzer Operator; and (b) a computer (or a human moderator)
connected to the network executing a programmed routine that
retrieves a Raw Coding Matrix from the Information Repository 114,
retrieves Student response data from the Information Repository 114
as appropriate to the psychometric routine being employed, uses the
Student response data to refine the Raw Coding Matrix, and then
writes the resulting Refined Coding Matrix to the Information
Repository 114, along with Quality Rating Information as described
in more detail below. The embodiment below assumes for definiteness
that this programmed routine is being carried out by a
computer.
[0420] One simple method the computer might implement to transform
a Raw Coding Matrix into a Refined Coding Matrix would be to
perform commonly employed factor-analytic calculations that
identify spurious Skill assignments for particular Items. More
complex methods for refining the Raw Coding Matrix include
approaches based on multidimensional Item Response Theory and
cognitive diagnosis routines such as Tatsuoka's Rule Space
methodology (Tatsuoka, K. K., Architecture of knowledge structures
and cognitive diagnosis, P. Nichols, S. Chipman & R. Brennan,
Eds., Cognitively Diagnostic Assessment. Hillsdale, N.J.: Lawrence
Erlbaum Associates, 1995) (incorporated herein by reference) and
DiBello and Stout's Unified Cognitive/Psychometric Diagnostic
Assessment model (DiBello, L., Stout, W., and Roussos, L. Unified
Cognitive/Psychometric Diagnostic Assessment likelihood-Based
Blassification Techniques. In P. Nichols, S. Chipman, and R.
Brennan, Eds., Cognitively Diagnostic Assessment. Hillsdale, N.J.:
Lawrence Erlbaum Associates, 1995) (incorporated herein by
reference).
[0421] All of these methods typically produce one form of quality
rating or another that reflects whether and to what extent the
connections posited by a Coding Matrix are reflected in the
patterns of Student response data.
[0422] To take a `toy model` example of the foregoing, suppose that
a Test has two Skills, Addition and Subtraction, and three Items.
The Raw Coding Matrix for this Test has been given as
TABLE-US-00012 Item 1 Item 2 Item 3 Addition 1 0.5 0 Subtraction 0
0.5 1
[0423] Thus, the best pedagogical judgment has determined that Item
2 assesses Addition and Subtraction equally.
[0424] Of course, whether or not Item 2 really assesses Addition
and Subtraction equally is a question that admits of empirical
analysis; that is, the Students' responses to the Test Items can be
analyzed with this question in mind. For example, if the
correlation r.sub.21 that holds between a Student's score on Item 2
and the Student's score on Item 1--a pure Addition Item--is higher
than the correlation r.sub.23 that holds between Student's Score on
Item 2 and the Student's Score on Item 3--a pure Subtraction
Item--then this offers statistical evidence that Item 2 actually
assesses ability in Addition more directly than it assesses ability
in Subtraction. Thus, a simple refinement of the Raw Coding Matrix
above could be given by TABLE-US-00013 Item 1 Item 2 Item 3
Addition 1 0.5((1 + r.sub.21)/(1 + r.sub.21/2 + r.sub.23/2)) 0
Subtraction 0 0.5((1 + r.sub.23)/(1 + r.sub.21/2 + r.sub.23/2))
1
provided, at least, that the highly exceptional case
r.sub.21=r.sub.23=-1 does not obtain. (For example, the condition
r.sub.21=-1 can only obtain when every Student who answered Item 1
correctly also answered Item 2 incorrectly, and conversely; this
occurrence would never arise in practice.)
[0425] For example, if r.sub.21 is as high as 0.7, but r.sub.23 is
only 0.2, then the Raw Coding Matrix TABLE-US-00014 Item 1 Item 2
Item 3 Addition 1 0.5 0 Subtraction 0 0.5 1
[0426] would become the Refined Coding Matrix TABLE-US-00015 Item 1
Item 2 Item 3 Addition 1 0.586 0 Subtraction 0 0.414 1
[0427] Since the correlation r.sub.21 is greater than the
correlation r.sub.23, the refinement has increased the weight of
Item 2 in Addition, at the expense of the weight of Item 2 in
Subtraction.
[0428] Correlation coefficients can also serve as a simple example
of Quality Rating Information for a Refined Coding Matrix. For
example, the matrix elements Q.sub.ki from the Refined Coding
Matrix (where k=1 for Addition, k=2 for Subtraction, i=1 for Item
1, i=2 for Item 2, and i=3 for Item 3) can be compared with
correlation coefficients to form a quality rating index such as
q=9-(1/2(1+r.sub.12)-Q.sub.11Q.sub.12).sup.2-(1/2(1+r.sub.23)-Q.sub.12Q.s-
ub.13).sup.2-(1/2(1+r.sub.13)-Q.sub.11Q.sub.13).sup.2-
(1/2(1+r.sub.12)-Q.sub.21Q.sub.22).sup.2-(1/2(1+r.sub.23)-Q.sub.22Q.sub.2-
3).sup.2-(1/2(1+r13)-Q.sub.21Q.sub.23).sup.2.
[0429] Taking the first term in parentheses as an example, one sees
that if Item 1 correlates poorly with Item 2 (so that
1/2(1+r.sub.12) is small), and yet Item 1 is grouped with Item 2 in
the same Skill (so that Q.sub.11Q.sub.12 is large), then the
squared term in parentheses will have to be large, because it is
the difference of a small number and a large number. Wlien the
squared term is large, it will detract from the quality q in virtue
of the minus sign. Thus, one sees that when the Refined Coding
Matrix "disagrees with the data," its quality rating will be
relatively low. As an illustration of this, the quality rating of
the Refined Coding Matrix (with the weights adjusted) is higher
than the quality rating of the Raw Coding Matrix (with 0.5/0.5
entries for Item 2) in the above example:
q.sub.refined-q.sub.raw=0.028>0.
[0430] This shows that, at least according to this simple model,
the quality of the Coding Matrix has gone up as a result of
Refinement.
[0431] In a conceptually similar manner, more sophisticated and
theoretically motivated analyses yield Quality Rating Information
that allows those skilled in the art to evaluate the quality of a
Refined Coding Matrix from the statistical point of view.
[0432] However, whatever method is used, it is important to note
that the method will rely on accurate instructional judgment to
provide it with a Raw Coding Matrix as an input. Typically, the Raw
Coding Matrices employed reflect crude `yes/no` judgments as to
Skill-Item relevance; this compromises the effectiveness of the
routines in generating instructionally valid Skill-Item
assignments. By contrast, the Raw Coding Matrices employed in this
invention are more instructionally accurate because they reflect
nuanced, non-binary judgments about Skill-Item relevance. As a
result, any routine implemented in the Coding Matrix Refiner 128
becomes a more powerful source of diagnostic information.
[0433] 4.E. Item-Skill Analyzer
[0434] FIG. 6ais a block diagram of the Item-Skill Analyzer 130
showing its subcomponents: the Skill Set Generator 650, the Coding
Matrix Inspector 652, and the Skill Set/Coding Matrix Selector 654.
FIG. 10 is a flowchart showing the functioning of the Item-Skill
Analyzer 130.
[0435] Existing methods typically calculate Test Results in
reporting categories that are not useful for the intended
recipients of the results. In many cases, teachers typically do not
understand these reporting categories, or find them overbroad. For
example, the organization that reports the exam results for the New
York state testing program uses the math reporting category
"Modeling/Multiple Representation", a term which typical teachers
do not understand. Similarly, the organization that reports the
exam results for the California state testing program uses the
reading reporting category "Functional Reading", a term which
typical teachers find overbroad.
[0436] This may occur because the organization that reports the
exam results decides, for simplicity, to base the reporting
categories on educational language that already exists, such as
broad language that the state has adopted about its educational
goals or standards. Unfortunately, using reporting categories based
on such language is problematic because the categories are then too
broad and non-descriptive.
[0437] On the other extreme, certain test preparation organization
report exam results using categories that are very narrowly defined
around the precise types of questions asked on the exam. Because
these categories are built around capturing the specific items on
the test itself, these categories are not useful for teachers or
others interested in using the Test Results to guide instruction.
Indeed, such narrow categories may even be instructionally
misleading, to the extent they suggest do not help instructors
understand the range of skills that their students may be expected
to master.
[0438] Another problem arises when the organization that reports
the exam results uses a purely statistical method, such as a factor
analysis, to determine which Items on the Test to collect together
for reporting purposes. Because this approach does not
systematically involve the insight of people with instructional
expertise, it often leads to reporting categories that are not
clearly defined or understood by the recipients of the Test
Results.
[0439] The Item-Skill Analyzer 130 instantiates a repeatable
process by which individuals can develop reporting categories for a
given test according to a shared protocol. One embodiment of this
process would maximize the pedagogical insight of the reporting
categories, subject to statistical constraints, and therefore would
be both educationally useful and psychometrically sound.
[0440] This is especially useful when Clients (a) regard themselves
as working towards an explicit set of educational goals or
standards defined by them, but (b) purchase their Tests from
external vendors. Under these circumstances, the connection between
the Skills required for success on the Test and the educational
goals embraced by the Client may not be explicit. Indeed, the
Client may never even have examined this connection in detail.
[0441] The Item-Skill Analyzer 130 makes it possible to establish
such a connection in an efficient, reliable, and robust manner.
Given a particular Test, and given the response data from Students
taking the Test, the Item-Skill Analyzer 130 generates (a) a Skill
Set S.sub.T, (b) an "Organization," .quadrature..sub.T, which is a
grouping of the Skills in the Skill Set S.sub.T into pedagogically
related families of Skills (Organizations are defined in detail
below); and (c) a Refined Coding Matrix Q(S.sub.T, T). These items
are written to the Information Repository 114 for use in the
remainder of the System's operations.
[0442] The Item-Skill Analyzer 130 may consist physically of (a) an
Operator (human or computer) that activates the Item-Skill Analyzer
130 when notified to do so by the Analysis Operator; and (b) a
computer (or a human moderator) executing a programmed routine that
follows the sequence of steps in FIG. 10. The embodiment below
assumes for definiteness that this programmed routine is being
carried out by a computer.
[0443] The process embodied in the Item-Skill Analyzer 130 is as
follows: First, the computer activates the Skill-Set Generator 650
in step 1002. As part of its functioning (described below), the
Skill-Set Generator 650 outputs a list of Proposed Skill Sets
S.sub.T.sup.1, . . . , S.sub.T.sup.N and their associated
Organizations .quadrature..sub..quadrature..sup.1, . . . ,
.quadrature..sub.T.sup.N to the Information Repository 114.
[0444] In step 1004, the computer activates the Skill-Item Table
Generator 124 with the first Proposed Skill Set S.sub.T.sup.1 as
the input. As part of its functioning as described above, the
Skill-Item Table Generator 124 outputs to the Information
Repository 114 a Skill-Item Table I(S.sub.T.sup.1, T) corresponding
to S.sub.T.sup.1.
[0445] With reference to step 904 of the Skill-Item Table Generator
124 flowchart, when the computer activates the Skill-Item Table
Generator 124 in step 1004, it enforces the rule that, with
reference to part 1.b(ii) of that process, the Proposer of that
Proposed Skill Set is required to vote "aye" for an Item whenever
the Item is a Justifying Item for him or her in the given Skill.
(These terms are defined below.)
[0446] In step 1006, the computer activates the Raw Coding Matrix
Generator 126 with the first Skill-Item Table I(S.sub.T.sup.1, T)
as the input. As part of its functioning as described above, the
Raw Coding Matrix Generator 126 outputs to the Information
Repository 114 a Raw Coding Matrix Q.sup.0(S.sub.T.sup.1, T)
corresponding to I(S.sub.T.sup.1, T).
[0447] In step 1008, the computer activates the Coding Matrix
Refiner 128 with the first Raw Coding Matrix Q.sup.0(S.sub.T.sup.1,
T) as the input. As part of its functioning as described above, the
Coding Matrix Refiner 128 outputs a Refined Coding Matrix
Q(S.sub.T.sup.1, T) and its associated Quality Rating Information
q.sub.1 to the Information Repository 114. (This quality
information may consist of more than a single number, and may also
include qualitative statements useful to those skilled in the art
of psychometrics for evaluating Coding Matrices.)
[0448] In steps 1010, 1012, and 1014, the computer carries out for
the second Proposed Skill Set S.sub.T.sup.2 a process strictly
analogous to the steps 1004, 1006, and 1008 that were carried out
for the first Proposed Skill Set S1. At the termination of step
1014, therefore, the Information Repository 114 will contain a
Skill-Item Table I(S.sub.T.sup.2, T), its associated Organization
.quadrature..sub.T.sup.2, a Raw Coding Matrix
Q.sup.0(S.sub.T.sup.2, T), a Refined Coding Matrix Q(S.sub.T.sup.2,
T), and Quality Rating Information q.sub.2 associated with
Q(S.sub.T.sup.2, T).
[0449] Strictly analogous steps are carried out for successive
Proposed Skill Sets in the list S.sub.T.sup.1, . . . , S
.sub.T.sup.N, until, in steps 1016, 1018, and 1020, the final
Proposed Skill Set S.sub.T.sup.N is processed. The Information
Repository 114 now contains, for each Proposed Skill Set, an
Organization that groups the Skills in the Skill Set into
pedagogically related families; a Skill-Item Table; a Raw Coding
Matrix; a Refined Coding Matrix; and Quality Rating Information
associated with the Refined Coding Matrix.
[0450] At the completion of step 1020, the computer activates in
step 1022 the Coding Matrix Inspector 652. As part of its
functioning (described below), the Coding Matrix Inspector 652
outputs to the Information Repository 114 a sub-list of the
original list of Proposed Skill Sets (containing M Skill Sets,
where M.ltoreq.N).
[0451] In step 1024 the computer activates the Skill Set/Coding
Matrix Selector 654. As part of its functioning (described below),
the Skill Set/Coding Matrix Selector 654 writes to the Information
Repository 114 a single Skill Set S.sub.T, its associated
Organization .quadrature..sub.T, and its associated Refined Coding
Matrix Q(S.sub.T, T).
Subcomponents Activated by the Item-Skill Analyzer
[0452] In broad outline, the Item-Skill Analyzer 130 is a system
for prompting two groups of human judges (called Pedagogical
Operators and Psychometric Operators, respectively) to interact
with one another to examine a set of Test Items and a set of
Student response data for those Test Items. The two groups are
prompted, in an efficient and repeatable way, to use the skills of
their respective arts to generate, from those Items and those data,
a Skill Set S.sub.T, an Organization .quadrature..sub.T, and a
Refined Coding Matrix Q(S.sub.T, T) that together express the
structure of the Test in a pedagogically and statistically sound
manner.
[0453] The dynamics of the Item-Skill Analyzer 130 and its
subcomponents are structured in recognition of the fact that
Pedagogical Operators and Psychometric Operators tend to exhibit
opposing tendencies. Pedagogical Operators seek to define Skill
Sets aligned with their pedagogical outlook, potentially at the
expense of statistical reliability. Psychometric Operators, on the
other hand, seek to define Skill Sets that lead to results that are
statistically reliable (as understood by people trained in the
art), potentially at the expense of making diagnostic
recommendations that are understandable for Educators, Parents, or
Students.
[0454] When one of these groups works in isolation, or when its
interaction with the other group is insufficiently structured, the
resulting Skill Set has the weaknesses characteristic of the group
that produced it. By contrast, the flow of the Item-Skill Analyzer
130 achieves an optimization between the two groups' tendencies
that produces results that are simultaneously useful and
reliable.
[0455] The dynamics of the Item-Skill Analyzer 130 and its
subcomponents, one embodiment of which is described in detail
below, have the following fundamental characteristics.
[0456] First, the algorithm is iterative in nature, consisting of a
series of proposals and counterproposals between the two groups.
The basic dynamics of these proposals and counterproposals must be
determined in advance. Schematically, if one group is Group A and
the other group is Group B, then the process may be chosen to take
any one of the following forms: AB, BA, ABA, BAB, etc.
[0457] The Group acting first in the process proposes a number of
potential Skill Sets, and in each successive phase of the process,
the number of potential Skill Sets is diminished, until only a
single Skill Set survives.
[0458] Subject to the chosen structure of proposals and
counterproposals, the process by which each Group generates or
selects Skill Sets and Coding Matrices may be decided by the Group
itself or may be governed by pre-existing decision-making criteria,
including (1) required approval by a fixed percentage of a Group
with respect to Proposed Skill Sets and Coding Matrices; (2)
culling from a summation of individual or subgroup determinations
of Proposed Skill Sets' and Skill-Item Matrices' fitness according
to a pre-determined rubric (these determinations could be in the
form of quality ratings, rank orderings, gradated determinations of
fitness, etc.).
[0459] Now that the operation of the Item-Skill Analyzer 130 as a
whole has been described, it is useful to describe one embodiment
of its first subcomponent, the Skill Set Generator 650.
[0460] 4.E. 1. Skill Set Generator 650
[0461] The Skill Set Generator 650 may consist physically of (a) an
Operator (human or computer) that activates the Skill Set Generator
650 when notified to do so by the Item-Skill Analyzer Operator; and
(b) a computer (or a human moderator) connected to the network
executing a programmed routine that follows the sequence of steps
in FIG. 7. The embodiment below assumes for definiteness that this
programmed routine is being carried out by a computer.
[0462] The programmed routine of the Skill Set Generator 650
prompts each member of a group of human judges, called Pedagogical
Operators, to generate lists of Skills, each of which, in that
person's judgment, captures the array of cognitive, artistic, or
other demands made on Students by a given set of Test Items T. The
number of Pedagogical Operators is flexible.
[0463] Importantly, the Skill Set Generator 650 ensures that every
list of Skills proposed by a Pedagogical Operator arises from a
comprehensive and, in the Pedagogical Operator's judgment, coherent
breakdown of the Subject of the Test (called an Organization).
Deriving the proposed lists of Skills from an Organization ensures
that, even though the Test itself may only assess a limited portion
of the Subject being assessed, nevertheless each proposed list of
Skills will present a coherent picture of the Subject, with no gaps
or overlaps. This feature of the Skill Set Generator 650 is highly
valuable to Users, who seek to use the Skill Set in a Display as a
way of thinking about the Subject they are trying to teach or
learn.
[0464] FIG. 7 is a flowchart showing the functioning of the Skill
Set Generator 650.
[0465] To understand the operation of the Skill Set Generator 650,
one can consider the following brief example of a Test, which is a
set of six constructed-response Items that might be administered to
third-grade mathematics students. [0466] 1. What is 13+7? [0467] 2.
What is 7.times.9? [0468] 3. Is 16.times.19879 odd or even? [0469]
4. What is the name of the following shape? .DELTA. [0470] 5. What
fraction is represented by this diagram? ##STR1## [0471] 6. Which
decimal is larger? 0.6 or 0.06
[0472] In step 702, the first Pedagogical Operator is prompted to
review the Test.
[0473] In step 704, the first Pedagogical Operator is prompted to
generate a list of the discrete Skills from the Subject that are
required in order to answer all Items on the Test correctly. These
lists will be called "Item Skills Lists."
[0474] For the example Test above, the Item Skills Lists of the
first Pedagogical Operator might be as follows: TABLE-US-00016 Item
Skills Lists for the first Pedagogical Operator Item 1 Item 2 Item
3 Item 4 Item 5 Item 6 Skills Required Basic Basic Basic Naming
Recognizing Ordering Operations Operations Operations Shapes
Fractions Decimals on Whole on Whole on Whole Numbers Numbers
Numbers Properties of Whole Numbers
[0475] In step 706, the first Pedagogical Operator is prompted to
list all of the Skills that appear in at least one Item Skills
List. This will be called the "Comprehensive Skills List".
TABLE-US-00017 Comprehensive Skills List for one Pedagogical Expert
Basic Operations on Whole Numbers Properties of Whole Numbers
Naming Shapes Recognizing Fractions Ordering Decimals
[0476] In step 708, the first Pedagogical Operator is prompted to
generate a tree that arranges the Skills in the Comprehensive
Skills List into families based on pedagogical relatedness.
[0477] To do this, the first Pedagogical Operator is prompted to
judge which Skills from the Comprehensive Skills List are most
closely related to one another, and to group these Skills together.
These groups are called Skill Families, and the Pedagogical
Operator is prompted to give them names.
[0478] For example, the first Pedagogical Operator may judge that a
Skill called "adding fractions with like denominators" and a Skill
called "adding fractions with unlike denominators" both belong to a
Skill Family called "adding fractions."
[0479] The first Pedagogical Operator is next prompted to judge
which Skill Families are most closely related to one another, and
to group these into Skill Families of Second Order, and to give
these names.
[0480] For example, the first Pedagogical Operator may judge that a
Skill Family called "adding fractions" and a Skill Family called
"multiplying fractions" both belong to a Skill Family of Second
Order called "basic operations with fractions."
[0481] Next, the first Pedagogical Operator is prompted to judge
which Skill Families of Second Order are most closely related to
one another, and to group these into Skill Families of Third Order,
and to give these names.
[0482] This process continues until at some order the first
Pedagogical Operator has obtained a single over-arching Skill
Family. (This single Skill Family may be considered the Operator's
view as to the Level and Subject of the Test itself.) This is
called the "Organization" for the first Pedagogical Operator. An
example of an Organization 802 is shown in FIG. 8.
[0483] In step 710, the first Pedagogical Operator is prompted to
"Regularize" his/her Organization. Regularizing an Organization
means collapsing all single-track paths in the corresponding tree
as far as possible from the bottom up, combining Skill Family names
in the process. More precisely, regularization is an iterative
process that proceeds as follows: If any leaf in the tree is an
"only child," then the name of the leaf is appended to the name of
the parent, and the leaf is deleted. This process is repeated on
the resulting tree, and iterated until the tree remains fixed. A
regularized tree, known as the "Regularized Operation" is either a
single node, or else a tree in which each leaf has a parent with
degree greater than one. An example of a Regularized Organization
804 is shown in FIG. 8.
[0484] Note that Regularizing an Organization is a process that can
be easily performed by a computer; hence, in some embodiments, the
task of Regularization can be lifted from the Pedagogical Operator.
This is especially worthwhile when the number of nodes in the
Organization is large.
[0485] Next, the first Pedagogical Operator is prompted to use
his/her Organization to generate a series of Proposed Skill Sets. A
Proposed Skill Set is a list of Skills and/or Skill Families
(henceforth called, simply, Skills) with the property that a direct
path from the root of the tree to any leaf in the tree encounters
exactly one Skill or Skill Family in the list. The Regularized
Organization in FIG. 8 generates four Proposed Skill Sets:
TABLE-US-00018 Proposed Proposed Proposed Proposed Skill Set
S.sub.T.sup.1 Skill Set S.sub.T.sup.2 Skill Set S.sub.T.sup.3 Skill
Set S.sub.T.sup.4 Basic Operations Basic Operations Whole Whole
Numbers on Whole on Whole Numbers Numbers Numbers Whole Number
Whole Number Geometry: Geometry: Properties Properties Naming
Shapes Naming Shapes Geometry: Geometry: Fractions: Fractions and
Naming Shapes Naming Shapes Recognizing Decimals Fractions
Fractions: Fractions and Decimals: Recognizing Decimals Ordering
Fractions Decimals Decimals: Ordering Decimals
[0486] Note that each Skill in a Proposed Skill Set corresponds to
a node in the Organization that generated it. This node has a set
of ultimate descendants within the Organization, which are Skills
that form leaves in the Organization. These Skills must earlier
have appeared as entries in the Item Skills List of some Item (or
Items). Thus, for any Skill in any Proposed Skill Set generated by
an Organization, there is a nonempty set of Items on the Test whose
Skill requirements, in that Pedagogical Operator's opinion, would
not properly be represented in the Proposed Skill Set if that Skill
were deleted from the Skill Set. These will be called the
Justifying Items for this Skill for that Pedagogical Operator.
[0487] Note that the process of enumerating all of the Proposed
Skill Sets that can be generated by a given Organization is a
process that can be easily performed by a computer; hence, in some
embodiments, the task of generating Proposed Skill Sets can be
lifted from the Pedagogical Operator. This is especially worthwhile
when the number of nodes in the Organization is large.
[0488] In steps 714, 716, 718, 720, 722, and 724, the precisely
analogous process is carried out for the second Pedagogical
Operator as was carried out in steps 702, 704, 706, 708, 710, and
712 for the first Pedagogical Operator.
[0489] This process continues until, in steps 726, 728, 730, 732,
734, and 736, the precisely analogous process is carried out for
the last Pedagogical Operator as was carried out in steps 702, 704,
706, 708, 710, and 712 for the first Pedagogical Operator.
[0490] The Pedagogical Operators may be prompted simultaneously (in
parallel), provided each Pedagogical Operator has access to his/her
own terminal. (In the case in which a human moderator executes the
programmed routine, using paper questionnaires, etc., to prompt the
Pedagogical Operators, parallel processing is trivial to
implement.)
[0491] In step 738, the computer collects together all of the
distinct Proposed Skill Sets that are generated by the
Organizations of all the Pedagogical Operators.
[0492] In step 740, the computer prompts each Pedagogical Operator
to rank the assembled Proposed Skill Sets. A Proposed Skill Set
earns a high rank if, in the Pedagogical Operator's opinion, it
offers (1) a usefully precise set of diagnostic categories for
Teachers working in the Subject and Level of the Test; and (2) a
breakdown of the Subject and Level consonant with the instructional
approaches of most Teachers in the audience of the Report.
[0493] In step 742, the computer averages the rankings of the
Pedagogical Operators to produce a "Consensus Ranked List" of
Proposed Skill Sets. For the sake of economy, the analysis can be
continued with only a certain number or proportion of Proposed
Skill Sets, chosen from the top of the Consensus Ranked List. This
number or proportion can be hard-coded into the programmed routine.
The resulting collection of Proposed Skill Sets is called the
Reduced Consensus List of Skill Sets.
[0494] In step 746, the computer writes each Skill Set STb in the
Reduced Consensus List S.sub.T.sup.1, . . . , S.sub.T.sup.N to the
Information Repository 114, along with the Organization
.quadrature..sub.T.sup.b that generated it. (Note that the
.quadrature..sub.T.sup.b will not in general be distinct, although
this fact is not significant for the functioning of the Item-Skill
Analyzer 130.)
[0495] Finally, in step 748, the computer notifies the Operator of
the Skill Set Generator 650 that processing is complete.
[0496] 4.E.2. Coding Matrix Inspector 652
[0497] After the Skill Set Generator 650 has completed its
processing, the Information Repository 114 contains a Reduced
Consensus List of Skill Sets S.sub.T.sup.1, . . . , S.sub.T.sup.N;
an Organization .quadrature..sub.T.sup.b for each Skill Set
S.sub.T.sup.b in the Reduced Consensus List that groups the Skills
in the Skill Set S.sub.T.sup.b into pedagogically related families;
a Skill-Item Table I(S.sub.T.sup.b, T) for each Skill Set
S.sub.T.sup.b in the Reduced Consensus List; a Raw Coding Matrix
Q.sup.0(S.sub.T.sup.b, T) for each Skill Set S.sub.T.sup.b in the
Reduced Consensus List; a Refined Coding Matrix Q(S.sub.T.sup.b, T)
for each Skill Set S.sub.T.sup.b in the Reduced Consensus List; and
Quality Rating Information q.sub.b associated with the Refined
Coding Matrix Q(S.sub.T.sup.b, T).
[0498] The Coding Matrix Inspector 652 may consist physically of
(a) an Operator (human or computer) that activates the Coding
Matrix Inspector 652 when notified to do so by the Item-Skill
Analyzer Operator; and (b) a computer (or a human moderator)
executing a programmed routine that follows the sequence of steps
outlined below. The embodiment below assumes for definiteness that
this programmed routine is being carried out by a computer.
[0499] The programmed routine of the Coding Matrix Inspector 652
prompts each member of a group of human judges, called Psychometric
Operators, to shorten the Reduced Consensus List of Skill Sets
S.sub.T.sup.1, . . . , S.sub.T.sup.N based on the Quality Rating
Information of the associated Refined Coding Matrices. The number
of Pedagogical Operators is flexible.
[0500] The programmed routine operates according to the following
algorithm:
[0501] First, the computer reads from the Information Repository
114 the list of Refined Coding Matrices Q(S.sub.T.sup.1, T), . . .
, Q(S.sub.T.sup.N, T). Then, each Psychometric Operator is prompted
to view each Refined Coding Matrix Q(S.sub.T.sup.b, T) and
familiarize himself/herself with its Quality Rating Information
q.sub.b. The Psychometric Operators are prompted to record notes on
each Refined Coding Matrix for use later in the routine.
[0502] The Psychometric Operators can signal the completion of
their review of a given Refined Coding Matrix by touching a key on
the computer keyboard or entering a computer command.
[0503] After each Psychometric Operator has been prompted to review
each Refined Coding Matrix, the computer prompts the Psychometric
Operators to discuss each Refined Coding Matrix in turn. The
Psychometric Operators signal the computer when the discussion of a
given Refined Coding Matrix has completed.
[0504] Following this discussion, the computer prompts each
Psychometric Expert to rate the suitability of each Refined Coding
Matrix in binary fashion (suitable/unsuitable). The ratings are
inputted via the computer keyboard, and the computer tallies the
votes.
[0505] If a Refined Coding Matrix is judged unsuitable by a certain
percentage of the Psychometric Operators (which percentage can be
hard-coded into the programmed routine), then the computer
designates the Refined Coding Matrix as unsuitable.
[0506] If all of the Skill-Item Matrices are deemed unsuitable by
the group, then the Psychometric Experts are each prompted to cast
a vote for the single Refined Coding Matrix they would reinstate.
After the votes are input into the computer and the computer counts
them, a Refined Coding Matrix is chosen randomly from among those
receiving the maximal number of votes, and this Refined Coding
Matrix is reinstated as suitable.
[0507] The computer writes to the Information Repository 114 the
suitable Refined Coding Matrices Q(S.sub.T.sup.b, T), numbering
M.gtoreq.1 in all, along with their corresponding Skill Sets
S.sub.T.sup.b, Organizations .quadrature..sub.T.sup.b, and
Skill-Item Tables I(S.sub.T.sup.b, T). The remaining list of Skill
Sets S.sub.T.sup.1, . . . , S.sub.T.sup.M is now called the Culled
List of Skill Sets.
[0508] Finally, the computer notifies the Operator of the Coding
Matrix Inspector 652 that processing is complete.
[0509] 4.E.3. Skill Set/Coding Matrix Selector 654
[0510] After the Coding Matrix Inspector 652 has completed its
processing, the Information Repository 114 contains a Culled List
of Skill Sets S.sub.T.sup.1, . . . , S.sub.T.sup.M; an Organization
.quadrature..sub.T.sup.b for each Skill Set S.sub.T.sup.b in the
Culled List, which groups the Skills in Skill Set S.sub.T.sup.b
into pedagogically related families; a Skill-Item Table
I(S.sub.T.sup.b, T) for each Skill Set S.sub.T.sup.b in the Culled
List; a Raw Coding Matrix Q.sup.0(S.sub.T.sup.b, for each Skill Set
S.sub.T.sup.b in the Culled List; a Refined Coding Matrix
Q(S.sub.T.sup.b, T) for each Skill Set S.sub.T.sup.b in the Culled
List; and Quality Rating Information q.sub.b associated with each
Refined Coding Matrix Q(S.sub.T.sup.b, T).
[0511] The Skill Set/Coding Matrix Selector 654 may consist
physically of (a) an Operator (human or computer) that activates
the Skill Set/Coding Matrix Selector 654 when notified to do so by
the Item-Skill Analyzer Operator; and (b) a computer (or a human
moderator) executing a programmed routine that follows the sequence
of steps outlined below. The embodiment below assumes for
definiteness that this programmed routine is being carried out by a
computer.
[0512] The programmed routine of the Skill Set/Coding Matrix
Selector 654 prompts a group of human judges, called Pedagogical
Operators, to select a single Skill Set S.sub.T from among the
Skill Sets in the Culled List of Skill Sets S.sub.T.sup.1, . . . ,
S.sub.T.sup.M based on pedagogical considerations. The number of
Pedagogical Operators is flexible.
[0513] The programmed routine operates according to the following
algorithm:
[0514] First, the computer reads from the Information Repository
114 the Culled List of Skill Sets S.sub.T.sup.1, . . . ,
S.sub.T.sup.M, along with the Organizations
.quadrature..sub.T.sup.b and Refined Coding Matrices
Q(S.sub.T.sup.b, T) corresponding to each Skill Set S.sub.T.sup.b
in the Culled List. Each collection [0515] (S.sub.T.sup.b,
.quadrature..sub.T.sup.b, Q(S.sub.T.sup.b, T)) that is read by the
computer from the Information Repository 114 is called a
"Proposal."
[0516] Next, the computer provides each Pedagogical Operator with
access to the Test Items for his/her reference.
[0517] Then, each Pedagogical Operator is prompted to view each
Proposal and consider it from a pedagogical point of view, using
the same criteria applied during step 740 of the functioning of the
Skill Set Generator 650, together with two additional criteria:
[0518] Pedagogical Operators may also take into account the
coherence, completeness, scope, and sensibility of the Organization
.quadrature..sub.T.sup.b in a Proposal, as it serves to communicate
the structure and organization of the Subject to Users. [0519]
Pedagogical Operators may also take into account the extent to
which the Skill-Item connections represented in the Refined Coding
Matrix reflect their own assessment of the relevance of each Skill
to each Item.
[0520] The Pedagogical Operators can signal the completion of their
review of a given Proposal by touching a key on the computer
keyboard or entering a computer command.
[0521] After each Pedagogical Operator has been prompted to review
each Proposal, the computer prompts the Pedagogical Operators to
discuss each Proposal in turn as a group. The Pedagogical Operators
signal the computer when the discussion of a given Proposal has
completed.
[0522] Following this discussion, the computer prompts each
Pedagogical Operator to rank the Proposals. The ranking of each
Pedagogical Operator is inputted via the computer keyboard.
[0523] After the rankings are all input into the computer and
aggregated by the computer using a point system (e.g., 0 points for
the top ranking, -1 points for the bottom ranking, etc.), a
Proposal (S.sub.T, .quadrature..sub.T, Q(S.sub.T, T)) is chosen
randomly from among those receiving the maximal number of
points.
[0524] The computer then writes to the Information Repository 114
the final Skill Set S.sub.T, Organization .quadrature..sub.T, and
Refined Coding Matrix Matrices Q(S.sub.T, T).
[0525] Finally, the computer notifies the. Operator of the Skill
Set/Coding Matrix Selector 654 that processing is complete.
[0526] 4.F. Student Skill Score Calculator
[0527] The purpose of the Student Skill Score Calculator 132 is to
represent each student's performance in each Skill by a single
number. This number is called the Student's Skill Score for the
Skill in question.
[0528] It is important to understand that the Skill Score may have
many different interpretations. First, the Skill Score may
represent a normative level of performance; for example, the
percent of those Items requiring the Skill that were answered
correctly by the Student. Second, the Skill Score may represent an
estimate of the Student's ability in the Skill, as commonly
discussed in Multidimensional Item Response Theory. Third, the
Skill Score may represent an estimate of the Student's probability
of mastery of the Skill, as commonly discussed in the theory of
cognitive diagnosis. Fourth, the Skill Score may represent the
Student's percentile rank, based on normative criteria such as
these. Finally, the Skill Score need not be interpreted as
normative data at all, instead representing simply a label that
uniquely identifies the Student's set of responses to the Items
requiring the Skill in question. Different Displays in a given
Report may show different varieties of Skill Score, and any given
Display may show several different varieties of Skill Score
simultaneously.
[0529] The Student Skill Score Calculator 132 may consist
physically of (a) a human Operator who activates the Student Skill
Score Calculator 132 according to the flowchart in FIG. 10awhen
notified in step 624 to do so by the Analysis Module Operator; and
(b) a computer connected to the network that executes appropriate
code as described in more detail below.
Processing Performed by the Computer Code
[0530] The Student Skill Score Calculator 132 takes from the
Information Repository 114 the following inputs: [0531] Data
Requirements for Results Reports; [0532] the Refined Coding Matrix;
and [0533] each Student's Item Score Vector.
[0534] The Student Skill Score Calculator 132 writes to the
Information Repository 114 the following outputs: [0535] Each
Student's Skill Score Vector (or Vectors, in the case of multiple
types of Skill Scores as suggested above).
[0536] The transformation of these inputs into these outputs is
accomplished by computer code. According to the technical
terminology specified in the Glossary, the computer code executed
during the operation of the Student Skill Score Calculator 132
effectively applies one or more "Skill Score Mappings" to each
Student's "Item Score Array", in order to create one or more "Skill
Score Vectors" for each Student. This statement can be made clearer
by describing several explicit embodiments of Skill Score
Mappings.
[0537] As specified in the Glossary, an Item Score Array is a
two-dimensional array of numbers containing a Student's Item
Scores, sorted by Skill. Each row of the array corresponds to a
different Skill. The numbers in a given row are the Student's Item
Scores for the Items requiring that Skill, where, in the context of
the Student Skill Score Calculator 132, a given Item i is said to
require a given Skill k when its column i in the Refined Coding
Matrix has a non-zero entry in row k (i.e. when the matrix element
Q.sub.ki.noteq.0).
[0538] A simple example of an Item Score Array can be given as
follows.
Item Score Array Example
[0539] Suppose the Skill Set is given by [0540] S.sub.T={"Skill 1",
"Skill 2", "Skill 3", "Skill 4"},
[0541] and the Refined Coding Matrix is gven by TABLE-US-00019 Item
1 Item 2 Item 3 Item 4 Item 5 Skill 1 0.5 0 0.788 0.2 0 Skill 2 0 0
0 0.4 0.5 Skill 3 0.5 0 0 0 0 Skill 4 0 1 0.212 0.4 0.5
[0542] Then, for example, the Items requiring Skill 1 are Item 1,
Item 3, and Item 4.
[0543] Suppose next that a hypothetical Student has incorrectly
answered Items 1 and 5, but has correctly answered Items 2, 3, and
4. So the Student's Item Score Vector is given by [0544]
y=(0,1,1,1,0).
[0545] By sorting these Scores in rows corresponding to Skills, the
Student's two-dimensional Item Score Array is found as .quadrature.
= ( ( 0 , 1 , 1 ) ( 1 , 0 ) ( 0 ) ( 1 , 1 , 1 , 0 ) ) .
##EQU1##
[0546] For example, the first row of the array contains the Item
Scores for just those Items requiring Skill 1 (Items 1, 3, and 4).
The Student's Score of 0 for Item 1 appears as the first number in
the first row of the array; the Student's Score of 1 for Item 3
appears as the second number in the first row of the array; and the
Student's Score of 1 for Item 4 appears as the third number in the
first row of the array. The other rows are completed similarly.
[0547] In this example, there is only one Item requiring Skill 3,
namely Item 1. So the third row of the Item Score Array contains
only one number, namely, the Item Score 0 that was received by the
Student on Item 1.
Explicit Embodiments of Skill Score Mappings
[0548] A Skill Score Mapping is any mathematical function that
transforms a Student's Item Score Array .quadrature. into a Skill
Score Vector, denoted s=(s.sub.1, . . . , s.sub.m). Here, s.sub.1
denotes the Student's Skill Score for Skill 1, and so on. Some
embodiments of Skill Score Mappings are as follows.
[0549] 1. A given Display may include information how many Items
requiring each Skill were answered correctly by the Student. This
list of numbers is an example of a Skill Score Vector for the
Student. To calculate this Skill Score Vector, the computer code
would simply add the entries in each row of the Student's Item
Score array: .quadrature. = ( ( 0 , 1 , 1 ) ( 1 , 0 ) ( 0 ) ( 1 , 1
, 1 , 0 ) ) .times. .times. transforms .times. .times. to .times. :
.times. .times. ( 2 1 0 3 ) = s raw .times. .times. scores
##EQU2##
[0550] This transformation is a Skill Score Mapping.
[0551] 2. Another Display may include information on the Student's
probability of mastery of each Skill. This would call for a more
complex Skill Score Mapping, .quadrature. = ( ( 0 , 1 , 1 ) ( 1 , 0
) ( 0 ) ( 1 , 1 , 1 , 0 ) ) .times. .times. transforms .times.
.times. to .times. : .times. .times. ( .quadrature. 1 .quadrature.
2 .quadrature. 3 .quadrature. 4 ) = s probabilities . .times. of
.times. .times. mastery ##EQU3##
[0552] Here the probabilities of mastery .quadrature..sub.k emerge
from the Student's Item Scores in a much more complicated way than
simple addition, through the implementation of a cognitive
diagnosis routine.
[0553] 3. Generalized Percent Correct.
[0554] In one embodiment of the Student Skill Score Calculator, a
Skill Score Mapping is implemented which generates Generalized
Percent Correct values in each Skill. Here the term "Generalized
Percent Correct" reflects the fact that Refined Coding Matrices may
split Items among different Skills. Therefore, a Student's
performance on a given Skill must take into account not only which
of the Items requiring that Skill the Student answered correctly,
but also the varying extent to which those Items require the
Skill.
[0555] The following constitutes a method for generating Students'
Generalized Percent Correct for a given Skill. First, given the
Refined Coding Matrix for the Test, the Total Weight for each Skill
is calculated and stored in the Information Repository 114. The
Total Weight in a Skill is found by multiplying [the n.sup.th value
in the row of the Refined Coding Matrix corresponding to the Skill
in question] by [the Item Weight of the n.sup.th Item], and adding
these products across the row. In other words, the Total Weight in
a Skill can be thought of as the maximum Score possible in a Skill,
taking into account the various maximum Scores of every Item and
the varying degree to which any Item assesses any Skill.
[0556] The Student's point total in the Skill is then found by
multiplying [the n.sup.th value in the row of the Refined Coding
Matrix corresponding to the Skill in question] by [the Student's
Item Score for the n.sup.th Item], and adding these products across
the row Dividing this point total by the Total Weight in a Skill
yields the Student's Generalized Percent Correct in the Skill. For
example, in the special case in which all Items are multiple-choice
Items with possible Item Scores of 0 or 1 (so that the Item Weight
of every Item is 1), and in which no single Item assesses more than
one Skill, the Generalized Percent Correct is equal to [the number
of Items requiring the Skill that were answered correctly by the
Student] divided by [the total number of Items requiring the
Skill].
[0557] 4. Labeling Item Score Arrays.
[0558] As a final example of a Skill Score Mapping, a Display may
involve specifying, for a given Skill, precisely which Items among
the Items requiring the Skill were answered correctly by the
Student. For this to be possible, the Student Skill Score
Calculator 132 must write the Student's Skill Score Vector to the
Information Repository 114 in the form of a vector of "labels" that
allow the Student's Item Score Array .quadrature. to be
reconstructed. This can be accomplished by means of the following
Skill Score Mapping, which regards each row in the Item Score Array
as an integer in base-2 representation, and simply expands the
integer in base-10: .quadrature. = ( ( 0 , 1 , 1 ) ( 1 , 0 ) ( 0 )
( 1 , 1 , 1 , 0 ) ) -> .times. ( 0 2 0 + 1 2 1 + 1 2 2 0 2 0 + 0
2 1 0 2 0 1 2 0 + 1 2 1 + 1 2 2 + 0 2 3 ) = ( 3 1 0 7 ) = s labels
##EQU4## (Note that the number of Items requiring each Skill is
necessary in order to recover the original Item Score Array from
the vector of labels s=(3,1,0,7); this information exists in the
Information Repository 114 within the Refined Coding Matrix.)
[0559] In the embodiments 1, 3, and 4 above, it is easy to see that
the Skill Score Mapping treats each Skill independently when it
calculates the Skill Scores; that is, the mapping proceeds row by
row, and there is no interaction between one Skill and another
Skill in calculating the Skill Scores. Such a Skill Score Mapping
is called a Factorizable Skill Score Mapping. Skill Scores leading
to probabilities of mastery may or may not arise from Factorizable
Skill Score mappings.
[0560] Thus, Non-Factorizable Skill Score Mappings form another
embodiment of the Student Skill Score Calculator 132.
Non-Factorizable Skill Score Mappings do not treat a Student's
performance on each Skill as independent from the Student's
performance on other Skills. As an example of a Non-Factorizable
Skill Score Mapping, those skilled in the art can estimate the
probability that a Student has mastered a particular Skill taking
into account the Student's performance on a particular Test; for
example, by using Cognitive Diagnosis theories such as Tatsuoka's
Rule Space Methodology or dibello and Stout's Unified Model, both
cited above.
[0561] A given Cognitive Diagnosis routine may or may not require
(or permit) the introduction of explicit curricular intelligence
into the calculation of this estimated probability of mastery.
However, one might easily imagine that, among those skilled in the
pedagogy of the Subject of the Test, it is almost universally
agreed that a Student can never master Skill k.sub.1 without first
mastering Skill k.sub.2. This knowledge makes it possible (and
necessary) to introduce an `override` into the estimate of the
probability of mastery for Skill k.sub.1. For example, if a Student
has very low mastery of skill k.sub.2, then it is extremely
unlikely that the Student can have mastered skill k.sub.1. This
curricular `override` produces an explicit dependence of one Skill
Score on another.
[0562] Skill Score Mappings are implemented by the Student Skill
Score Calculator 132 as follows.
[0563] FIG. 10ais a flowchart showing the functioning of the
Student Skill Score Calculator 132.
[0564] In step 1050, the Operator of the Student Skill Score
Calculator 132 retrieves the Data Requirements for Results Reports
from the Information Repository 114.
[0565] In step 1052, the Operator reviews the Data Requirements for
Results Reports to determine which Skill Score Mappings must be
implemented (probabilities of mastery, percent correct values,
etc.) in order to meet the requirements.
[0566] In step 1054, the Operator searches the Code Library to
determine whether code exists in the Code Library to implement all
of the required Skill Score Mappings. In Decision 1056, if all of
the necessary code does not exist in the Code Library, the Operator
uses the skills of his/her art in Step 1058 to write the necessary
code and add it to the Code Library. Upon searching the Code
Library again in Step 1054, he/she is guaranteed to find all of the
necessary code in Decision 1056.
[0567] Having found the necessary code in Decision 1056, the
Operator loads the code from the Code Library into the Psychometric
Analysis Computer in step 1060 and executes it in step 1062. This
sets the computer in motion.
[0568] In step 1064, the computer retrieves from the Information
Repository 114 the Refined Coding Matrix, as well as each Student's
Item Score Array.
[0569] In step 1066, the computer then effects a transformation of
each Student's Item Score Array into that Student's Skill Score
Vector(s), according to the particular Skill Score Mapping(s) being
implemented.
[0570] In step 1068, the computer writes the Student Skill Score(s)
to the Information Repository 114, and then, in step 1070, notifies
the Operator that the processing of the Student Skill Score
Calculator 132 is complete.
[0571] 4.F. Student Skill Performance Evaluator
[0572] The purpose of the Student Skill Performance Evaluator 134
is to evaluate each Student's performance in each Skill in a verbal
(non-numerical) fashion. It is important to realize that Users of
Test Results do not simply want to see numerical Scores that
describe Students' performance on the Test. Instead, they want
verbal statements (known here as "Evaluation Statements") about how
Students and Groups performed, both on the Test as a whole and in
particular Skills. In particular, Users of Test Results may want to
know about recommended instructional strategies for addressing
weaknesses and for helping Students and Groups to advance still
further in areas of their strength. Evaluation Statements that
recommend actions of one kind or another are here known as
"Formative Statements."
[0573] Accordingly, the Student Skill Performance Evaluator 134
assigns Evaluation Statements (including Formative Statements) to
individual Students based on their performance on the Test. The
system and method of the Student Skill Performance Evaluator 134
constitute a repeatable process for assigning Evaluation Statements
to Students based on Students' performance in particular
Skills.
[0574] For example, in one embodiment, the Student Skill
Performance Evaluator 134 assigns each Student one of three
absolute, Formative Statements for each Skill, such as the
following: "Likely to Need Help with Fundamentals", "Likely to Need
Additional Instruction and Practice", and "Likely to Be Ready for
Advanced Work". These Formative Statements (i.e., statements that
recommend one course of action over another) might be applied to
specific Students and Skills, resulting (for example) in
suggestions that one Student work on the fundamentals of
subtraction, while another Student proceed to advanced work in that
subject. These Formative Statements can be very useful for Users,
such as Educators who can take such recommended courses of actions
in the classroom.
[0575] It may also be the case that Client may provide additional
requirements for the types of Evaluation Statements that it wishes
displayed. These requirements, stored in the Information Repository
114 by the Data Assembler 106, can be applied by the Student Skill
Performance Evaluator 134 as well. For example, the Client might
wish that when a Student receives his or her Test Results, the
Student will see that particular one of a group of two given
Evaluation Statements (for instance, "Try Harder" and "Good Work")
that corresponds best to the Student's overall performance and to
the Student's particular performance in individual Skills.
[0576] The Student Skill Performance Evaluator 134 thus draws its
Evaluation Statements from a certain range. Formally, a "Range of
Evaluation Statements" is here defined to be a set of jointly
exhaustive, mutually exclusive Evaluation Statements. To say that a
Range of Evaluation Statements is jointly exhaustive means that any
Obtained Skill Score can be comprehended under the scope of at
least one of the Evaluation Statements in the Range. To say that a
Range of Evaluation Statements is mutually exclusive means that
there is no Obtained Skill Score that can be comprehended under the
scope of more than one Evaluation Statement in the Range.
[0577] Some examples of Ranges of Evaluation Statements could
include: [0578] R={`Rapid Growth Potential`, [0579] `Low Growth
Potential`} (growth-based Evaluation Statements are discussed in an
embodiment below); [0580] R={`Likely to Need Help With
Fundamentals`, [0581] `Likely to Require Additional Instruction and
Practice`, [0582] `Likely to Be Ready for Advanced Work`}; (these
particular Evaluation Statements can be assigned to Students based
on a "Coarse-Grained Criterion Mapping," as described in an
embodiment below, or can be assigned by other means, including
comparisons of relative performance of Students); and [0583]
R={`Top Quartile`, [0584] `Second Quartile`, [0585] `Third
Quartile`, [0586] `Bottom Quartile`} (which are necessarily
assigned on the basis of relative comparisons).
[0587] Several Ranges of Evaluation Statements may be involved in
the various Displays of a particular Report. For example, the
Displays in a given Report may involve the following four Ranges of
Evaluation Statements: [0588] R.sub.1={`Likely to Need Help With
Fundamentals`, [0589] `Likely to Require Additional Instruction and
Practice`, [0590] `Likely to Be Ready for Advanced Work`}, [0591]
R.sub.2={`Room to Grow`, [0592] `Where You Shine`, [0593]
`Neither`}, [0594] R.sub.3={`Recommended Starting Point`, [0595]
`Not a Recommended Starting Point`}, [0596] R.sub.4={`Low
Probability of Mastery`, [0597] `High Probability of Mastery`,
[0598] `Impossible to Estimate Probability of Mastery`}.
[0599] Not every Evaluation Statement in each Range need be overtly
displayed in an actual display shown to Users. For example, in the
case of R.sub.3 above, the display would likely highlight the
single Skill that maps to the Evaluation Statement `Recommended
Starting Point`--the other Skills' status as `Not a Recommended
Starting Point` being reasonably indicated merely by omission or
implication.
[0600] Using the Evaluation Statements in a given Range, the
Student Skill Performance Evaluator 134 "evaluates" numerical
scores and generates "conclusions" from them. Informally, it is as
if the Student Skill Performance Evaluator 134 accomplishes the
task of saying, "Given all these Skill Scores, what would one best
conclude?" Obviously, the mathematical details of the code
operating within the Student Skill Performance Evaluator 134 must
depend intimately on the semantic universe represented by the Range
of Evaluation Statements. The Client constrains this process
through the Data Reporting Requirements, as described in more
detail below.
[0601] When Evaluation Statements are displayed by the Display
Module 142, they allow Users to view and organize information not
only according to numerical measures such as percent correct or
scale score, but also according to qualitative criteria, which
criteria may include summative evaluations ("Mastery", "Area of
Difficulty", etc.) as well as formative statements, which are
recommendations for action of one kind or another ("Ready for
Advanced Work", etc.).
Physical Instantiation
[0602] The Student Skill Performance Evaluator 134 may consist
physically of (a) a human Operator who activates the Student Skill
Performance Evaluator 134 according to the flowchart in FIG. 11
when notified in step 624 to do so by the Analysis Module Operator;
and (b) a computer connected to the network that executes
appropriate code as described in more detail below.
Processing Performed by the Computer Code
[0603] The Student Skill Performance Evaluator 134 takes from the
Information Repository 114 the following inputs: [0604] Data
Requirements for Results Reports; [0605] each Student's Skill Score
Vector(s); [0606] other Student Performance Information such as may
be required to generate Evaluation Statements in a particular
embodiment, such as those described in detail below.
[0607] The Student Skill Score Calculator 132 writes to the
Information Repository 114 the following outputs: [0608] Each
Student's Vector(s) of Evaluation Statements (or Vectors, in the
case of multiple correspondences between Skill Scores and Ranges of
Evaluation Statements as suggested above).
[0609] The transformation of these inputs into these outputs is
accomplished by computer code. Using the technical terms defined in
the Glossary, the computer code executed during the operation of
the Student Skill Performance Evaluator 134 effectively applies one
or more "Score-Statement Mappings" to one or more of each Student's
Skill Score Vectors, in order to produce one or more "Vectors of
Evaluation Statements" for the Student.
[0610] As a very brief example of an explicit embodiment of such a
Score-Statement Mapping, consider a mapping that transforms any
Skill Score above 90 into the Evaluation Statement "Good Job!";
transforms any Skill Score between 80 and 90 into the Evaluation
Statement "Practice Makes Perfect!"; and transforms any Skill Score
of 80 or below into the Evaluation Statement "Keep Trying!" This
transformation rule is a Score-Statement Mapping.
[0611] More sophisticated embodiments of Score-Statement Mappings
will be described below. However, since these explicit embodiments
have lengthy descriptions, it is helpful to describe first the
overall functioning of the Student Skill Performance Evaluator 134
itself, which is the component of the Analysis Module 120 that
implements Score-Statement Mappings.
[0612] Score-Statement Mappings are implemented by the Student
Skill Performance Evaluator 134 as follows.
[0613] FIG. 11 is a flowchart showing the functioning of the
Student Skill Performance Evaluator 134.
[0614] In step 1102, the Student Skill Performance Evaluator
Operator retrieves from the Information Repository 114 the Data
Requirements for Reporting. These stipulate the Ranges of
Evaluation Statements to be used in the Displays, and they also
stipulate constraints on the mapping of Skills Scores to these
Evaluation Statements, as happens for example in Coarse Graining,
discussed below.
[0615] By reviewing the Data Requirements for Reporting, the
Operator determines in step 1104 which Score-Statement Mapping(s)
must be implemented in order to provide data for the Displays.
[0616] In step 1106 the Operator searches the Code Library for the
code necessary to implement these mappings via a computer
program.
[0617] If in Decision 1108 the necessary code does not exist, then
the Operator uses the skills of his/her art to write the necessary
code, adding it to the Code Library in step 1110. When the Operator
next searches the Code Library in step 1106, the necessary code
will be found.
[0618] In Decision 1108 the Operator then finds the necessary code
in the Code Libarary, and in step 1112 loads it into the computer,
executing it in step 1114.
[0619] The computer in step 1116 retrieves Student Skill Scores, as
well as other Student Performance Information as required by the
algorithm(s) being implemented; see below for embodiments. In step
1118 the computer carries out the algorithm(s) to assign each
Student one or several Vectors of Evaluation Statements, as for
example in the embodiments below. Then in step 1119 the Students'
Vectors of Evaluation Statements are written to the Information
Repository 114, and the computer in step 1120 notifies the Operator
that its processing is complete.
Explicit Embodiments of Score-Statement Mappings
[0620] Intuitively, a Score-Statement Mapping converts the
numerical data in a Skill Score Vector into the Skill-by-Skill list
of verbal statements or recommendations in a Vector of Evaluation
Statements. As defined in the Glossary, a Skill Score Mapping
.quadrature..sub.R is any mathematical function that transforms a
Student's Skill Score Vector s=(s.sub.1, . . . , s.sub.m) into a
Vector of Evaluation Statements e=(e.sub.1, . . . , e.sub.m). Here,
s.sub.1 denotes the Student's Skill Score for Skill 1 and e.sub.1
denotes the Student's Evaluation Statement for Skill 1, drawn from
the Range of Evaluation Statements R. Usually the Range of
Evaluation Statements under discussion is understood, so
.quadrature..sub.R is simply written .quadrature..
[0621] It is also useful to consider the nature of "Factorizable"
Score-Statement Mappings, .quadrature.=(.quadrature..sub.1, . . . ,
.quadrature..sub.m). As defined in the Glossary, the finite set
consisting of all distinct Skill Scores in a particular Skill k
that have been obtained by at least one Student is denoted O.sub.k
and called the set of "Obtained Skill Scores" for Skill k. Thus,
given any Student's Skill Score Vector s, its k.sup.th component
s.sub.k must be drawn from the set O.sub.k. The set of all Skill
Score Vectors that may be assembled by drawing k.sup.th components
from O.sub.k is therefore called the "Logical Space of Skill Score
Vectors," defined in the Glossary and denoted U. With these terms
fixed, a Score-Statement Mapping .quadrature.: S.fwdarw.E(R) is
said to be Factorizable when it "treats Skills independently,"
i.e., when there exist m Skill-specific mappings [0622]
.quadrature..sub.k:O.sub.k.fwdarw.R such that the action
.quadrature.(s)=e decomposes as [0623] .quadrature.((s.sub.1,
s.sub.2, . . . , s.sub.m))=(.quadrature..sub.1(s.sub.1),
.quadrature..sub.2(s.sub.2), . . . , .quadrature..sub.m(s.sub.m))
for every s.di-elect cons.U. Intuitively, a Factorizable
Score-Statement Mapping is one that considers each Skill
independently; whereas a non-Factorizable Score-Statement Mapping
is one that "takes the entire Skill Score Vector into account."
[0624] When the mappings .quadrature..sub.k exist, it will be said
that ".quadrature. Factorizes." An economical notation for a
Factorizable Score-Statement Mapping is
.quadrature.=(.quadrature..sub.1, . . . , .quadrature..sub.m)
[0625] Some embodiments of Skill Score Mappings are as follows.
Embodiments: Automated Skill-Specific Criterion Referencing
[0626] Many Tests are Criterion-referenced, meaning that Students'
performance is implicitly compared against performance standards in
that Subject as established by pedagogical experts in that Subject.
Students who take a Criterion-referenced Test are typically
assigned an Evaluation Statement based on their Test Results, which
statement reflects experts' interpretation of those particular
results. For example, Students who perform poorly on a particular
Criterion-referenced 4th grade math test may be deemed as "Not
Meeting Standards", while those who perform well may be deemed as
"Exceeding Standards." These standards are typically absolute
standards, although they could also be based on a relative
comparison to a standard group, e.g. "All Students nationwide
taking this Test."
[0627] However, Criterion-referenced tests typically are not
designed to generate criterion-based evaluations about a Student's
performance in a given Skill. For example, a Student who takes a
4th grade math test generally is not evaluated against particular
criteria in the various Skills tested. This occurs because test
developers usually do not ask pedagogical experts to establish
criteria with respect to particular Student response vectors on the
Items related to a given Skill. Instead, such experts usually
establish criteria only with respect to Students' overall scores on
the Test. (For example, in the K-12 environment,
criterion-referencing is usually used only to measure student
overall performance on high-stakes, end-of-year tests.)
[0628] This situation is unfortunate for several reasons. First of
all, Users often want to understand a Student's performance on a
Skill with respect to the same criteria of mastery that is applied
to Students' overall scores on the Test. More fundamentally,
although a given Student may perform poorly overall on a Test, that
Student may still have high ability in one or more Skills. One of
the most important goals of diagnostic Test reporting is to uncover
these hidden strengths. As long as Criterion Referencing only
applies to the Student's overall Score, this is impossible.
[0629] It is conceivable, of course, that pedagogical experts could
perform Skill-specific Criterion-referencing in each Skill on a
Test. However, this process is extremely time-consuming and is
rarely carried out.
[0630] What is needed, therefore, is an efficient computer
algorithm that leverages (1) the existing criterion-referencing of
Students' overall scores, and (2) empirical data about the
relationship between Students' overall scores and their scores
within a Skill, in order to produce an approximate
criterion-referencing for performance within each of the Skills
assessed by the Test.
[0631] One helpful way to approach this problem is through the
notion of probability. Probabilities are a useful way to deal with
situations in which we have limited information. In the present
situation, we are trying to determine, roughly speaking, whether or
not a Student has high ability in a given Skill. We don't have much
information to work with; just the Student's Score in the Skill,
which often arises from the Student's responses to just a few Test
Items. In a situation of uncertainty like this, it is helpful to
use probability to make a good decision. What is the probability
that the Student has high ability in the Skill? Can we use what
knowledge we have about the Student's Score in order to estimate
this probability and make the most accurate determination possible
under the circumstances?
[0632] A particular algorithm based on this approach is defined in
this embodiment. The algorithm consists of an explicit
Score-Statement Mapping (Factorizable in this embodiment), in which
the Evaluation Statements that evaluate Students' performance by
Skill inherit their meaning from the expert Criterion-referencing
that had been used to evaluate students' overall performance. This
example of Automated Skill-Specific Criterion Referencing is called
the Criterion Mapping.
Construction of the Criterion Mapping .quadrature.*
[0633] The following description specifies precisely how to
construct mathematically the sought-after Criterion Mapping
.quadrature.*. To be useful in large-scale applications, this
specification should be implemented on a computer by one skilled in
the art of programming mathematical algorithms.
[0634] It is a feature of this Embodiment that the Criterion
Mapping .quadrature.* is Factorizable. One may therefore select a
fixed but arbitrary Skill--denoted Skill K--and construct
explicitly a mapping .quadrature.*.sub.K: O.sub.K.fwdarw.R. The
Criterion Mapping will then be completely specified in terms of its
Skill-by-Skill components as .quadrature.*=(.quadrature.*.sub.1, .
. . , .quadrature.*.sub.m).
[0635] For the sake of brevity, .quadrature.*.sub.K is referred to
as a Criterion Mapping, although, strictly speaking, it is a
component of the Criterion Mapping .quadrature.*.
[0636] FIG. 12 is a flowchart illustrating the major steps in one
possible Criterion Mapping routine.
[0637] In step 1202, the Student Skill Performance Evaluator
Operator loads code for implementing the Criterion Mapping from the
Code Library, and in step 1204 he/she executes this code. The
computer then performs the following steps.
[0638] In step 1205, the computer reads from the Information
Repository 114 the list of Absolute Levels of Overall Performance;
these are part of the Data Requirements for Results Reports. The
meaning of these Absolute Levels of Overall Performance is as
follows.
[0639] As discussed above, the Test under consideration in this
embodiment has been Criterion-referenced. That is to say, certain
ranges of overall scores have been defined by human experts, which
ranges are regarded by persons under the umbrella of the testing
system as indicating ascending absolute levels of overall
performance. As an example, such a Criterion-referencing can be
compactly expressed as follows: TABLE-US-00020 Scale Score Absolute
Level Range of Overall Performance 350-500 Far Below Standards
501-600 Below Standards 601-720 Meeting Standards 721-800 Far Above
Standards
[0640] As per this table, each Student has had assigned to him or
her an Absolute Level of Overall Performance, based on his or her
overall score. In this example, there are four Absolute Levels of
Overall Performance, .quadrature..sub.1, . . . ,
.quadrature..sub.4: [0641] .quadrature..sub.1 Far Below Standards
[0642] .quadrature..sub.2 Below Standards [0643] .quadrature..sub.3
Meeting Standards [0644] .quadrature..sub.4 Far Above
Standards.
[0645] In general, there will be M Absolute Levels of Overall
Performance, .quadrature..sub.1, . . . , .quadrature..sup.M, which
can be taken without loss of generality to be arranged in ascending
order of performance. For brevity, a Student with Absolute Level of
Overall Performance .quadrature..sub.a is referred to as a "Level-a
Student."
[0646] The Range R for the sought-after Criterion Map
.quadrature..sub.K*:O.sub.k.fwdarw.R is that same collection of
verbal statements provided by the criterion referencing of the Test
as a whole. Abbreviating these verbal statements as
.quadrature..sub.1, .quadrature..sub.2, . . . , .quadrature..sub.M,
the Range R for the Criterion Map .quadrature..sub.K* is thus
R={.quadrature..sub.1, .quadrature..sub.2, . . . ,
.quadrature..sub.M}. The absolute levels of performance
.quadrature..sub.1, .quadrature..sub.2, . . . , .quadrature..sub.M
in R are called Skill-Specific Absolute Levels of Performance.
[0647] In sum, the Criterion Map .quadrature..sub.K* ascribes an
absolute level of performance to a Student's numerical performance
in a Skill, extending, by statistical inference, the same
vocabulary that expresses absolute levels of performance for the
Test as a whole: namely, the criterion-referenced Absolute Levels
of Overall Performance, .quadrature..sub.1, .quadrature..sub.2, . .
. , .quadrature..sub.M.
[0648] In step 1206, the computer reads from the Information
Repository 114 each Student's Skill Score s.sub.k in each Skill k,
as well as each Student's Absolute Level of Overall
Performance.
[0649] In step 1208 the computer determines from the data read in
step 1206 those Skill Scores o.sub.k.sup.b in each Skill k that
were obtained by Students taking the Test. In this way, the
computer constructs the sets of Obtained Skill Scores
O.sub.k={o.sub.k.sup.1, . . . , o.sub.k.sup.r(k)}. Here, r(k) is
the number of distinct Skill Scores obtained by Students in Skill
k.
[0650] As an example that will be considered for the sake of
concreteness from time to time throughout the embodiment, consider
a hypothetical multiple-choice test in elementary-school
mathematics, with seven Skills in the Skill Set S.sub.T={S.sub.1, .
. . , S.sub.7}. Suppose that five of the Items on this Test Assess
Skill S.sub.7, which happens to be the Skill of "Addition."
Suppose, further, that none of these five Items are `split`
(meaning that none of the five Items Assess any other Skill besides
Addition.) Then, if the Skill Scores in question represent
Generalized Percent Correct values, O.sub.7 will be given by
O.sub.7={0.00, 0.20, 0.40, 0.60, 0.80, 1.00},
[0651] provided (as is usually the case for a typical Test and a
for large enough group of Students) that there is at least one
Student among the Test-takers who got one out of the five Addition
Items correct, at least one Student among the Test-takers who got
two out of the five Addition Items correct, and so on.
[0652] Returning to the computer's execution, in step 1210 the
computer enters a loop controlled by the index K, initializing the
value of K at 1. The index K is a Skill index, ranging from 1 to m.
The computer has thus begun looping through the Skills in
order.
[0653] In step 1212 the computer enters a loop controlled by the
index b, initializing the value of b at 1. The index b is an
obtained-score index, ranging from 1 to r(K). The computer has thus
begun looping through the Obtained Skill Scores O.sub.K,
o.sub.K.sup.b, in Skill K.
[0654] At this point, the algorithm can bring the methods of
probability to bear. Specifically, we would like next to answer the
following question: What is the probability that a given Student
has a certain level of ability in a given Skill?
[0655] Of course, all we know about the Student is his or her Skill
Score in the Skill. Thus, the algorithm will focus on the question
in this form: What are the chances that a given Student has a
certain level of ability in a given Skill, given that he or she
attained a certain Score in the Skill? Very high Scores will cause
us to raise our estimate of the chances that the Student has high
ability in the Skill. Very low Scores will cause us to lower our
estimate of the chances that the Student has high ability in the
Skill. The algorithm provides a rational method for making this
intuition precise.
[0656] In step 1214 the computer estimates a certain probability
distribution, denoted .quadrature..sub.K.sup.b. This calculation
proceeds as follows: [0657] 1. For each value of a=1, . . . , M,
count the number p.sub.a of Level-a Students. [0658] 2. For each
value of a=1, . . . , M and each value of b=1, . . . , r(k), count
the number g.sub.Kab of Level-a Students with Skill Score
s.sub.K=o.sub.K.sup.b. [0659] 3. For each value of a=1, . . . , M
and each value of b=1, . . . , r(k), calculate the fraction
f.sub.Kab=g.sub.Kab/p.sub.a [0660] of Level-a Students with Skill
Score s.sub.K=o.sub.K.sup.b. [0661] 4. For each value of b=1, . . .
, r(k), use the M fractions f.sub.K1b, f.sub.K2b, . . . , f.sub.KMb
to estimate the probability distribution
.quadrature..sub.K.sup.b={.quadrature..sub.K(1|b),
.quadrature..sub.K(2|b), . . . , .quadrature..sub.K(M|b)} [0662]
over the space of Absolute Levels of Overall Performance
{.quadrature..sub.1, . . . , .quadrature..sub.M}.
[0663] Here, .quadrature..sub.K(a|b) represents the probability
that the Absolute Level of Overall Performance .quadrature..sub.a,
among all the .quadrature..sub.M, best describes a Student's true
level of mastery of Skill K, given that the Student has earned a
Skill Score o.sub.K.sup.b in the Skill. As discussed above, the
Student's Score in the Skill has a strong effect on our assessment
of the probability that the Student has this or that level of
ability. This is why the probability .quadrature..sub.K(a|b)
depends on the index b: it is this index that specifies the Student
Skill Score, o.sub.K.sup.b. Of course, the probability
.quadrature..sub.K(a|b) also depends on the index a: it is this
index that specifies the particular level of mastery
.quadrature..sub.a we are considering attributing to the
Student.
[0664] Returning to the hypothetical five-Item Test to illustrate
.quadrature..sub.K(a|b) by example, suppose that the overall
criterion referencing for this Test recognizes four Absolute Levels
of Overall Performance: [0665] .quadrature..sub.1 Far Below
Standards [0666] .quadrature..sub.2 Below Standards [0667]
.quadrature..sub.3 Meeting Standards [0668] .quadrature..sub.4 Far
Above Standards.
[0669] In this case, recalling that there six Obtained Skill Scores
in the Skill S.sub.7="Addition," namely the Generalized Percent
Correct Values O.sub.7={0.00, 0.20, 0.40, 0.60, 0.80, 1.00}, the
quantity .quadrature..sub.7(1|6) represents the probability that
the statement "Far Below Standards" best describes a Student's
absolute level of performance in Addition, given that the Student
has earned a Skill Score of 1.00 in Addition. (This probability is
reasonably expected to be low.)
[0670] The discrete probability distribution
.quadrature..sub.K.sup.b={.quadrature..sub.K(1|b),
.quadrature..sub.K(2|b), . . . , .quadrature..sub.K(M|b)} can be
estimated from the M fractions f.sub.K1b, f.sub.K2b, . . .
,f.sub.KMb in many different ways, according as the Operator
applies the skills of his art in writing the code that effects the
Criterion Mapping. However, in any event, the transformation used
should have the following properties: [0671] i
.quadrature..sub.K(a|b) is an increasing function of f.sub.Kab; and
[0672] ii. .quadrature..sub.K(a|b) is a decreasing function of
f.sub.Kxb for x.noteq.a.
[0673] The reason for constraint (i) can be better understood by
returning to the hypothetical example. All other things being
equal, if 75% of those Students who are "Below Standards" overall
earned a Skill Score of 0.40 in Addition, then one is more
convinced that earning a score of 0.40 in Addition is a performance
deserving of the label "Below Standards" than if it were the case
that only 20% of those Students who are "Below Standards" earned a
Skill Score in Addition of 0.40.
[0674] On the other hand, even if 75% of those Students who are
"Below Standards" overall did earn a Skill Score of 0.40 in
Addition, one can also say that if 95% of those Students who are
"Far Below Standards" earned a Skill Score of 0.40 in Addition,
then one is more convinced that earning a score of 0.40 in Addition
is a performance deserving of the label "Far Below Standards"--and
hence less convinced that earning a score of 0.40 in Addition is a
performance deserving of the label "Below Standards"--than if it
were the case that only 60% of those Students who are "Far Below
Standards" earned a Skill Score in Addition of 0.40. This is the
motivation for constraint (ii).
[0675] If desired, one can adopt a very simple rule for generating
the probabilities .quadrature..sub.K from the fractions f:
.quadrature..sub.K.sup.0(a|b)=f.sub.Kab/(f.sub.K1b+f.sub.K2b+.varies.+f.s-
ub.KMb), where the superscript 0 indicates that this is an
estimate. It is easy to see that this rule satisfies constraints
(i) and (ii). (To economize on notation, the superscript 0 will
henceforth be suppressed.)
[0676] Once the probability distribution .quadrature..sub.K.sup.b
has been estimated, the computer in step 1216 increments the loop
index b by unity and checks in Decision 1218 whether the new value
of b is greater than r(K), which would signal the completion of the
Obtained Skill Score loop.
[0677] If the new value of b is less than or equal to r(K), then
the computer returns to step 1214 to begin the probability
distribution .quadrature..sub.K.sup.b estimation cycle once
again.
[0678] Once the computer has cycled through all the possible values
of b, the computer will have determined, for each Skill Score, the
probability that a Student with that Skill Score has an ability in
that Skill best described by a certain Level. Thus, if the Student
has a Skill Score in Addition of 0.20, we may assess the chances as
follows: TABLE-US-00021 Addition Score: "Far Below "Below "Meeting
"Far Above 0.20 Standards" Standards" Standards" Standards" Chances
that the 70% 20% 8% 2% Student's level of mastery in Addition are
best described by the statement . . .
[0679] In this example, if the Student earns a Score of only 0.20,
then the chances are low (2%) that the Student's level of mastery
in Addition is best described as "Far Above Standards."
[0680] But the situation may look very different for a Student who
earns a score of 0.80 in Addition: TABLE-US-00022 Addition Score:
"Far Below "Below "Meeting "Far Above 0.80 Standards" Standards"
Standards" Standards" Chances that the 10% 25% 25% 40% Student's
level of mastery in Addition are best described by the statement .
. .
[0681] Suppose, then, that in this example we are faced with a
Student who has earned an Addition Score of 0.80. What
Skill-Specific Absolute Level of Performance should we assign to
this Student?
[0682] The table above shows that the chances are greatest that the
Student's level of mastery is best described by the statement "Far
Above Standards."
[0683] Thus, once the relevant probability distributions have been
calculated, one option for assigning the Skill-Specific Absolute
Levels of Performance is to (a) focus on a particular Student; (b)
look at the Student's Score in the Skill in question; (c) look at
the probabilities for the different Level options, given the
Student's Score; and (d) choose the Level with the highest
probability, i.e. the greatest likelihood.
[0684] This embodiment of the Criterion Mapping is a useful
option.
[0685] Another option is also useful, particularly when the Student
Score data in question are extremely "noisy," as may happen for any
number of reasons: few Students taking the Test, very few Items
requiring a given Skill, a great deal of "Item splitting," and so
on. In the case of noisy data, it can be very hard to obtain good
estimates of the above probabilities. And when the accuracy of the
probability estimates is compromised, it may not be appropriate to
base Students' Level assignments on a simple examination of these
probabilities.
[0686] Even in such a situation, however, the algorithm can
proceed, as described in the remaining steps laid out in FIG. 12.
In this embodiment, one first narrows the field of possibilities by
imposing reasonableness constraints, and only then does one return
to the use of probability to make the best possible choice.
[0687] If the new value of b is greater than r(K), then in step
1220 the computer constructs the Reduced Search Space, .sub.K. This
is accomplished as follows. [0688] 1. To begin with, the Universal
Search Space, .sub.K, consists of all possible mappings
.quadrature..sub.K from the set O.sub.k={o.sub.K.sup.1, . . . ,
o.sub.K.sup.r(K)} to the set R={.quadrature..sub.1,
.quadrature..sub.2, . . . , .quadrature..sub.M}. The mappings
.quadrature..sub.K in .sub.K are called "Candidate Mappings." The
Criterion Mapping .quadrature..sub.K* is necessarily contained in
the Universal Search Space .sub.K. [0689] 2. The Criterion Mapping
.quadrature..sub.K* will be selected by defining one or several
objective functions that assign "ratings" to Candidate Mappings.
The Criterion Mapping will be selected by examining these ratings
for a wide array of Candidate Mappings. [0690] 3. Note, however,
that the number of mappings in .sub.K is given by M.sup.r(K), which
is on the order of log.sub.10M.sup.r(K).about.5-12 in practice.
Values on the larger end of this range make exhaustive searching
impracticable in most applications. Where possible, exhaustive
searching of .sub.K can of course be employed, in the same manner
used below to search the Reduced Search Space .sub.K. Where this is
not possible, those skilled in the art of combinatorial
optimization can apply many techniques in order to find efficiently
a Candidate Mapping .quadrature..sub.K.sup.+.di-elect cons..sub.K
with a high rating. The Candidate Mapping .quadrature..sub.K.sup.+
can then be designated as the sought-after Criterion Mapping
.quadrature..sub.K*. Alternatively, as mentioned above, the
Universal Search Space can be reduced, as described next. [0691] 4.
The following reasonableness constraints for Candidate Mappings
drastically reduce the region of Universal Search Space .sub.K that
must be examined. [0692] i. If o.sub.K.sup.x>o.sub.K.sup.y, then
.quadrature..sub.K(o.sub.K.sup.x) represents a higher level of
performance than .quadrature..sub.K(o.sub.K.sup.y). [0693] That is,
if o.sub.K.sup.x>o.sub.K.sup.y, then we must have
.quadrature..sub.K(o.sub.K.sup.x)=.quadrature..sub.r and
.quadrature..sub.K(o.sub.K.sup.y)=.quadrature..sub.s with r>s.
In other words, .quadrature..sub.K must be order-preserving. [0694]
ii. .quadrature..sub.K(min(O.sub.K))=.quadrature..sub.1 and
.quadrature..sub.K(max(O.sub.K))=.quadrature..sub.M. [0695] That
is, .quadrature..sub.K maps endpoints to endpoints.
[0696] It may be advantageous that a Criterion Mapping should obey
these constraints. After all, condition (i) simply says that higher
Skill Scores should never lead to lower estimates of ability. And
condition (ii) is almost tautological: the first condition says
that when a Student's Skill is Score is the lowest Score possible,
then the best estimate of the Student's ability in the Skill, based
only on that Score, is given by the lowest ability rating
available. And the second condition says that when a Student's
Skill is Score is the highest Score possible, then the best
estimate of the Student's ability in the Skill, based only on that
Score, is given by the highest ability rating available.
[0697] This is not to say that the constraints are without effect.
Apart from reducing the size of the Search Space, these constraints
will also sometimes result in a Student receiving a Level
assignment that is not the most probable assignment. Thus, as
discussed above, imposing these constraints is the most appropriate
option when the probability estimates themselves are highly
uncertain.
[0698] On the other hand, the probabilities obviously cannot be
ignored. Having opted first of all to require that the Level
assignments satisfy the conditions (i) and (ii), the remaining
steps of the algorithm fold the probabilities back into the
calculation, so that the final Level assignments can be chosen in a
way that minimizes the expected number of inaccurate assignments
made during the process. [0699] 5. The Reduced Search Space
.sub.K.OR right..sub.K consists of all mappings .quadrature..sub.K
satisfying the constraints above. The number of Candidate Mappings
.quadrature..sub.K in .sub.K is (r(K)+M-3)!/((M-1)!(r(K)-2)!)
[0700] a number typically on the order of log.sub.10(|
.sub.K|).about.2 or 3 in applications, which means that global
optimization by the method of exhaustive searching is practical.
Here the exclamation point represents the factorial function,
N!.ident.N(N-1)(N-2).varies.(3)(2)(1). [0701] 6. One skilled in the
art of programming can cause the computer to construct the space
.sub.K by simply generating a list of all many-to-one
correspondences between O.sub.K and R satisfying constraints (i)
and (ii).
[0702] In step 1222, the computer culls the list of Candidate
Mappings in .sub.K by means of a series of one or more Objective
Functions. In the present embodiment this is accomplished as
follows. [0703] 1. Note that arbitrary mapping .quadrature..sub.K
chosen at random from .sub.K will ascribe Skill-Specific Absolute
Levels of Performance to the Skill Scores o.sub.K.sup.b.di-elect
cons.O.sub.K without any regard for the actual probability
.quadrature..sub.K(a|b) that a Student who scored
s.sub.K=o.sub.K.sup.b in Skill k actually has level of mastery in
the Skill that is best described, among all the Evaluation
Statements in R, by .quadrature..sub.K(o.sub.K.sup.b). In short,
most Candidate Mappings .quadrature..sub.K will commit many errors.
[0704] 2. Thus, given a Candidate Mapping .quadrature..sub.K, an
"Error of Type (a, x)" is an erroneous assignment committed by the
mapping in which a Student, whose level of mastery in the Skill is
actually best described by the Evaluation Statement
.quadrature..sub.a, is nevertheless assigned the Evaluation
Statement .quadrature..sub.x.noteq.a by the Candidate Mapping
.quadrature..sub.K based on his or her Skill Score. [0705] 3. Note
that any given Candidate Mapping .quadrature..sub.K will commit a
certain number of Errors of Type (a, x). The expected number of
such errors is denoted E(.quadrature..sub.K; a, x), and can be
calculated by the computer based on the probability distributions
.quadrature..sub.K.sup.1, . . . , .quadrature..sub.K.sup.r(K) as
follows: [0706] 3.A. First calculate the total number of Students
p.sub.K.sup.b obtaining Skill Score o.sub.K.sup.b in Skill K. (Note
that p.sub.K.sup.b=g.sub.Kab+.varies.+g.sub.KMb.) [0707] 3.B. Then,
in terms of quantities already defined, E(.quadrature..sub.K; a, x)
is given by E(.quadrature..sub.K; a,
x)=.SIGMA.[.delta.(.quadrature..sub.K(o.sub.K.sup.b),
.quadrature..sub.x)p.sub.K.sup.b.quadrature..sub.K(a|b):b=1, . . .
, r(K)]. [0708] Here, .delta. is the discrete delta function, i.e.
.delta.(p, q).ident.1 if p=q, and .delta.(p, q).ident.0 if
p.noteq.q. [0709] 4. To construct the Objective Function .phi., the
Operator must have specified the kinds of errors he/she would most
like to avoid. For instance, in the hypothetical example [0710]
.quadrature..sub.1 Far Below Standards [0711] .quadrature..sub.2
Below Standards [0712] .quadrature..sub.3 Meeting Standards [0713]
.quadrature..sub.4 Far Above Standards the intended meaning
attached to the Absolute Levels of Overall Performance is
bifurcated, so that, broadly speaking, Levels 1 and 2 represent
failure, whereas Levels 3 and 4 represent success. In such a case,
one might wish, in assigning the Skill-Specific Absolute Levels of
Performance, to minimize the number of errors in which a Student
who should have been assigned either .quadrature..sub.1 or
.quadrature..sub.2 is actually assigned either .quadrature..sub.3
or .quadrature..sub.4; or conversely. One might call errors of
these kinds Crossing Errors, because in these cases, a Student's
erroneous Skill-Specific Absolute Level of Performance causes him
or her to cross the boundary between passing and failing, or
conversely. Preferences such as this can also be part of the Data
Requirements for Results Reports as well.
[0714] Alternatively, the Operator might choose (or the Data
Requirements for Results Reports might stipulate) to minimize the
total number of erroneous assignments without regard for the
detailed type; or the number of erroneous assignments of the type
(a, x) where x differs from a by two or three, etc. All of these
choices can be implemented in the series of Objective Functions as
described below. [0715] 5. Define an objective function .phi.:
.sub.K.fwdarw. to be -1 times the expected number of errors, of the
kind specified in (4) as being most important, that are committed
by a Candidate Mapping .quadrature..sub.K in .sub.K. Thus, for
example, if in (4) Crossing Errors are identified as most
important, one would put
.phi.=-(E(1,3)+E(1,4)+E(2,3)+E(2,4)+E(3,1)+E(3,2)+E(4,1)+E(4,2)).
[0716] Or, as another example, if in (4) the total number of
errors, without prejudice as to the Type, are identified as most
important, then one would put
.phi.=-(E(1,2)+E(1,3)+E(1,4)+E(2,1)+E(2,3)+E(2,4)+E(3,1)+E(3,2)+E(3,4)+E(-
4,1)+E(4,2)+E(4,3)).
[0717] In this embodiment, the Operator may also hard-code into the
computer program a tunable parameter .quadrature..sub.1 defined
below that allows the global optimization to relaxed, if desired,
so that additional criteria (different kinds of errors, or
considerations in addition to error) can play a role. [0718] 6. The
computer calculates .phi.(.quadrature.(.quadrature..sub.K) for each
.quadrature..sub.K.di-elect cons. .sub.K. [0719] 7. The computer
discards all .quadrature..sub.K.di-elect cons. .sub.K except those
that are within .quadrature..sub.1 standard deviations of
.phi..sub.max.ident.max
{(.phi.(.quadrature..sub.K):.quadrature..sub.K.di-elect cons.
.sub.K}.
[0720] Here, .quadrature..sub.1>0 is a pre-specified parameter;
some experimentation on the part of the Operator who writes the
code for the Criterion Mapping is helpful in determining values of
.quadrature..sub.1 that yield .quadrature..sub.K with acceptable
error rates while presenting following steps of the construction
with a fair diversity of options. (Suitable values of
log.sub.10.quadrature..sub.1 usually fall in the range , although
one can of course set .quadrature..sub.1=0 to allow the objective
.phi. to completely determine the solution. One skilled in the art
can also implement a subroutine to calculate a value of
.quadrature..sub.1 that satisfies various statistical criteria.)
[0721] 8. The set of .quadrature..sub.K remaining after (7) is the
Second Reduced Search Space .sub.K.sup.2. If desired, a secondary
objective function .phi..sub.2: .sub.K.sup.2.fwdarw., based on
criteria other than the expected number of errors, can now be
defined. A beneficial such function is the entropy,
.phi..sub.2=-.SIGMA.[z.sub.aln(z.sub.a):a=1, . . . ,M] [0722] where
z.sub.a.ident.|.quadrature..sub.K.sup.-1(.quadrature..sub.a)|/p
[0723] is the percentage of Students Tested who are assigned
.quadrature..sub.a by the Candidate Mapping .quadrature..sub.K.
[0724] The purpose of using the entropy .phi..sub.2 as a secondary
objective function is to introduce a bias in favor of Candidate
Mappings that `spread out` the Students to a greater extent. As a
hypothetical example, suppose that two Candidate Mappings
.quadrature..sub.K' and .quadrature..sub.K'' in .sub.K.sup.2 have
very similar error rates, and are given as in the following table:
TABLE-US-00023 Obtained Skill-Specific Skill-Specific Skill
Absolute Level Absolute Level Number of Score of Performance of
Performance Students 0.sub.K.sup.b
.quadrature..sub.K'(0.sub.K.sup.b)
.quadrature..sub.K''(0.sub.K.sup.b) p.sub.K.sup.b 0.00
.quadrature..sub.1 .quadrature..sub.1 1000 0.20 .quadrature..sub.1
.quadrature..sub.2 2000 0.40 .quadrature..sub.2 .quadrature..sub.2
5000 0.60 .quadrature..sub.3 .quadrature..sub.3 10000 0.80
.quadrature..sub.4 .quadrature..sub.3 5000 1.00 .quadrature..sub.4
.quadrature..sub.4 200
[0725] Then the profile of the Student population within Skill K is
as follows: TABLE-US-00024 .quadrature..sub.K' .quadrature..sub.K'
Number of Students 3000 1000 assigned .quadrature..sub.1 Number of
Students 5000 7000 assigned .quadrature..sub.2 Number of Students
10000 15000 assigned .quadrature..sub.3 Number of Students 5200 200
assigned .quadrature..sub.4 .phi..sub.2(.quadrature..sub.K')
.apprxeq. 1.29 .phi..sub.2(.quadrature..sub.K'') .apprxeq. 0.82
[0726] In this situation .quadrature..sub.K' may be chosen over
.quadrature..sub.K'', because without compromising accuracy, it
offers Users a richer diagnosis (i.e., there is less `clumping` of
Students in the middle Levels .quadrature..sub.2 and
.quadrature..sub.3). This is reflected in the entropy values,
.phi..sub.2(.quadrature..sub.K')>.phi..sub.2(.quadrature..sub.K'').
[0727] 9. The computer repeats (6)-(8), this time with the Second
Reduced Search Space .sub.K.sup.2 and the second objective function
.phi..sub.2. [0728] The process may be continued as many times as
desired, with successive objective functions .phi..sub.3,
.phi..sub.4, . . . and successively Reduced Search Spaces
.sub.K.sup.3, .sub.K.sup.4, . . . , until either (i) only one
candidate mapping is left; or (ii) all of the objective functions
have been applied.
[0729] Once all the objective functions have been applied, the
computer checks in Decision 1224 whether there is exactly one
Candidate Mapping remaining, in which case this is the sought-after
Criterion Mapping, or whether there are many Candidate Mappings
remaining. If there are many Candidate Mappings remaining after the
objective functions have been applied, then these Candidate
Mappings are all equally suitable by definition, so in step 1226
the computer selects one of them at random.
[0730] In step 1226 the computer reads from the Information
Repository 114 a "Coarse Graining." This Coarse Graining is part of
the Data Requirements for Results Reports, and specifies (a) the
number and wording of the Skill-Specific Evaluation Statements that
will ultimately be shown in Displays, which may differ from the
number M and the wording of the Skill-Specific Absolute Levels of
Performance .quadrature..sub.1, . . . , .quadrature..sub.M; and (b)
the rule for translating the Skill-Specific Absolute Levels of
Performance .quadrature..sub.1, . . . , .quadrature.M generated in
the previous steps of the algorithm.
[0731] The translation rule allows the computer to `bin` the
Students into a smaller number of levels than the
.quadrature..sub.1, . . . , .quadrature..sub.M levels resulting
from the previous steps of the algorithm. This is often
advantageous, especially when M is greater than three, because a
smaller number of final Evaluation Statements allows for both
improved accuracy (because the diagnostic ranges are wider) and
also improved usability (because it is pointless to offer
practitioners more options for action than they can realistically
resolve in their practice).
[0732] The computer reads the protocol for Coarse Graining from the
Data Requirements for Results Reports in the form of an
order-preserving mapping D: R.fwdarw.R' onto a Range of Evaluation
Statements R' with strictly fewer Evaluation Statements than R
(i.e., |R'|<|R|). For example, if there are more than three
Absolute Levels of Overall Performance, such as, TABLE-US-00025
.quadrature..sub.1 Far Below Standards .quadrature..sub.2 Below
Standards .quadrature..sub.3 Meeting Standards .quadrature..sub.4
Far Above Standards,
then the Criterion Mapping found in the previous steps of the
algorithm, [0733] .quadrature..sub.K*:O.sub.k.fwdarw.{`Far Below
Standards`, . . . , `Far Above Standards`},
[0734] can be Coarse-Grained with a Coarse Graining D given by
TABLE-US-00026 D(`Far Below Standards`) .quadrature. `Likely to
Need Help with Fundamentals` D(`Below Standards`) .quadrature.
`Likely to Require Additional Instruction and Practice` D(`Meeting
Standards`) .quadrature. `Likely to Require Additional Instruction
and Practice` D(`Far Above Standards`) .quadrature. `Likely to Be
Ready for Advanced Work`.
[0735] Now the Coarse-Grained Criterion Mapping
D.quadrature..sub.K* assigns each Student one of three
Skill-Specific Absolute Levels of Performance in Skill K: [0736]
.quadrature..sub.K*:O.sub.k.fwdarw.{`Likely to Need Help with
Fundamentals`, [0737] `Likely to Require Additional Instruction and
Practice`, [0738] `Likely to Be Ready for Advanced Work`}.
[0739] Using this rule, the computer in step 1230 calculates an
Evaluation Statement D(.quadrature..sub.K*(s.sub.k)) for each
Student in Skill K, based on his or her score s.sub.K in that
Skill.
[0740] In step 1232 the loop control index K is incremented by
unity, and in Decision 1234 the computer checks to see whether the
new value of K is greater than the overall number of Skills, m. If
not, then the computer returns to step 1212 to resume the algorithm
with another Skill.
[0741] If in Decision 1234 the value of K is greater than m, then
the computer in step 1236 writes all Students' Evaluation
Statements in all Skills to the Information Repository 114 and
notifies the Operator in step 1238 that processing is complete.
Some Important Features of .quadrature.* in this Embodiment
[0742] 1. A given Student's Skill-Specific Absolute Levels of
Performance, as generated by the Criterion Mapping .quadrature.*,
derive their meaning via statistical inference from the same
Absolute Levels of Overall Performance that are assigned to the
body of Students as a whole. Thus, the Criterion Mapping
.quadrature.* allows Users to understand a Student's performance on
a Skill with respect to the same criteria of mastery that are
applied to Students' overall scores on the Test.
[0743] 2. Evaluating .quadrature.* for a given student is an
efficient computational process. Even when the Criterion Mapping
.quadrature.* is constructed by means of a computer program written
in a relatively slow, high-level programming language, such as
Mathematica; and even when this code is executed on a modest
computer (IBM Thinkpad 600X, a laptop with a 500 MHz Pentium III
processor); and even when the data set to be analyzed is quite
large (p.about.100,000 Students, m.about.15 Skills, and
r(k).about.10 Obtained Skill Scores, on average, across Skills);
the construction of the map still takes only a matter of minutes.
Furthermore, the mapping .quadrature..sub.K itself need only be
constructed once, at the outset of processing-and, once
constructed, it can be cast as a simple lookup table with .about.15
entries, as in the examples .quadrature..sub.K' and
.quadrature..sub.K'' above. Assigning Evaluation Statements to the
Students via the Criterion Mapping then merely amounts to doing a
lookup for each Student; hence, the processing time per student is
negligible.
[0744] 3. The combination of the primary objective function .phi.
defined in terms of error rates, and the secondary objective
function .phi..sub.2 defined in terms of entropy, allows the
Criterion Mapping to provide diagnostic richness without
appreciably affecting accuracy.
[0745] 4. Coarse Graining allows the Criterion Mapping to be at
once more accurate and more usable.
Embodiment: Prioritizing Skills based on Growth Opportunities.
[0746] Users of Test Results often want to understand how a Student
(or Group) can improve its mastery of Skills. Unfortunately, there
does not exist in the prior art a method of prioritizing Skills
within a diagnostic report so that the Users' attention is drawn
first to those Skills in which rapid growth may be possible. The
present invention remedies this problem.
[0747] On the other hand, recommending that Users focus on one
Skill over another is a sensitive undertaking, because Users must
be confident that they are not being asked to pass over important
Skills without good cause. Users may be much more receptive to
Skill prioritization, and may therefore be more likely to act on
the recommendations in the Test Report, when the method used to
prioritize the Skills is relatively straightforward, so that Users
may easily understand the method and feel confident about its
application. The routine specified below allows this requirement to
be satisfied.
[0748] Of course, as a practical matter, it is also important that
any such prioritization method be not computationally intensive, so
that it can be implemented relatively quickly over large numbers of
Tests, each taken by large number of Students. For example, the
time for implementing such a method should not scale exponentially
with the number of Items on the Test. The routine specified below
allows this requirement to be satisfied also.
[0749] The routine specified below is one method for specifying
priorities for individual Students based on opportunities for
growth, called the "Room to Grow.TM. Method." The analogous process
for identifying priorities for Groups based on the Room to Grow.TM.
Method is described during the discussion of the Aggregation
Analyzer 136.
Construction of the Room to Grow.TM. Mapping
.quadrature..sub.RTG
[0750] The Room to Grow.TM. Mapping .quadrature..sub.RTG is a
understandable way to identify effectively those Skills in which a
particular Student performed relatively poorly, but in which most
Students who took the Test performed relatively well. Such
"surprising" Skills often present unique opportunities for rapid
progress, and therefore this information may be helpful for Users,
such as for an Educator who is deciding which Skill to focus on
with a Student.
[0751] The description below will specify one way in which to
construct (mathematically) the Room to Grow.TM. Mapping
.quadrature..sub.RTG. To be useful in large-scale applications,
this specification could be implemented on a computer by one
skilled in the art of programming mathematical algorithms.
[0752] The Room to Grow.TM. Mapping .quadrature..sub.RTG as
constructed here will be a Score-Statement Mapping, that is, it
will convert vectors of Skill Score values into vectors of
Evaluation Statements.
[0753] The Skill Scores may represent Generalized Percent Correct
values, estimated probabilities of mastery, etc. Meanwhile, the
Evaluation Statements in question are [0754] R={`Room to Grow.TM.
Skill`, `Not a Room to Grow.TM. Skill`} or statements with
essentially equivalent meaning, which have been read by the Student
Skill Performance Evaluator 134 in step 1102. When Results are
Displayed, those Skills that are deemed "Room to Grow.TM. Skills"
are highlighted, e.g. with graphical elements or by collecting them
together in a separate list.
[0755] FIG. 12ais a flowchart illustrating the major steps in one
possible Room to Grow.TM. prioritization routine.
[0756] In step 1250, the Student Skill Performance Evaluator
Operator checks the Information Repository 114 to determine whether
or not it contains the aggregate Skill Scores for Groups required
to run the particular Room to Grow.TM. prioritization being
implemented. If in Decision 1252 aggregate Skill Scores for Groups
do not exist in the Information Repository 114, then in step 1254
the Student Skill Performance Evaluator Operator activates the
Aggregation Analyzer 136, either by serving as its Operator or by
notifying its Operator. In steps 1302, 1304, 1306, 1308, 1310,
1312, 1314, 1315, 1316, 1317, and 1318, the Aggregation Analyzer
136 generates the required Group aggregate Skill Scores, writes
them to the Information Repository 114, and notifies the Operator
that processing is complete. In step 1258, the Student Skill
Performance Evaluator Operator then proceeds to load the Room to
Grow.TM. code from the Code Library (which, if necessary, had been
written by the Student Skill Performance Evaluator 134 in step
1110).
[0757] In step 1260 the Student Skill Performance Evaluator
Operator executes the code. The computer then performs the
following routine.
[0758] In step 1264 the computer enters a loop controlled by the
index j, initializing the value of j at 1. The index b is a Student
index, ranging from 1 to the number of Students p. The computer has
thus begun looping through the Students.
[0759] In step 1266 the computer calculates the current Student's
"Room to Grow.TM. points" in each Skill, using a "Room to Grow.TM.
function." This function estimates the growth potential for a given
Skill. The Room to Grow.TM. function can be constructed in numerous
ways, one of which is decided upon by the Operator in step 1104.
[0760] A. In the first family of methods, the Room to Grow.TM.
function estimates growth potential by taking into account (1) how
well the Student scored in the Skill; and (2) how well a comparison
group scored in the Skill. Intuitively speaking, Skills in which
the Student scored poorly, but in which the comparison group scored
well, represent "gaps" that the Student can be expected to address
easily. [0761] To be precise, the Room to Grow.TM. function
.rho.(x,y) in this case is a function that takes two real
arguments, both in the continuous range [0,1]. The first argument
is interpreted as the Student's Skill Score, normalized if
necessary so that it ranges from 0 to 1. The second argument is
interpreted as a `benchmark` Skill Score in the Skill, also
normalized to the range [0,1]. Examples of possible benchmark Skill
Score values might include the average Skill Score in this Skill of
all Students taking the Test, or the average Skill Score in this
Skill of all Students taking the Test who earned overall Scores in
a certain range, etc. Benchmark Skill Scores are calculated in step
1254 during the functioning of the Aggregation Analyzer 136. [0762]
Under this framework, the Room to Grow.TM. function can be chosen
heuristically so long as it satisfies the following properties:
[0763] i. .rho.(x,y)>0 for all x and y in [0,1]. [0764] ii.
.rho.(1,y)=0 for all y.di-elect cons.[0,1] [0765] iii. .rho.(x,0)=0
for all x.di-elect cons.[0,1] [0766] iv. For all y.di-elect
cons.[0,1], x'>x''.rho.(x',y)<.rho.(x'',y) [0767] v. For all
x .di-elect cons.[0,1], y'>y''.rho.(x,y')>.rho.(x,y'').
[0768] For example, a satisfactory functional form is
.rho.(x,y)=y(1-x). If this form is chosen, then the Room to
Grow.TM. function .quadrature. measures the growth potential in a
Skill, for a Student with Skill Score s.sub.k in the Skill, as
.quadrature.(s.sub.k, b.sub.k)=b.sub.k(1-s.sub.k).
[0769] Here, b.sub.k represents the benchmark Percent Correct in
Skill k. This has an understandable motivation as follows: [0770]
Insofar as the term s.sub.k is the level of performance of the
Student in Skill k, and insofar as the maximum possible level of
performance in the Skill k is 1, the term 1-s.sub.k represents a
"resource" that is still available to the Student provided his or
her ability in the skill increases. [0771] Insofar as the term
b.sub.k is a fraction between 0 and 1 that represents a realistic
level of performance for some standard comparison group, the term
b.sub.k measures the "efficiency" with which the group can be
expected to "convert" the resource. [0772] The product of
"resource" times "efficiency" gives the "yield"--in this case, the
improvement in the level of performance in the given Skill. [0773]
B. In the second family of methods, the Room to Grow.TM. function
estimates growth potential by taking into account not only (1) how
well the Student scored in the Skill; and (2) how well a comparison
group scored in the Skill; but also (3) how strongly emphasized the
Skill was on the Test. Intuitively speaking, progress will be more
rapid with the Student focuses on Skills that carry a great deal of
weight on the Test. [0774] To be precise, the Room to Grow.TM.
function .rho.(x,y,z) in this case is a function that takes three
real arguments. The first two arguments are interpreted as in (a).
The third argument is interpreted as the Total Weight in a Skill,
w.sub.k. Total Weights of Skills are calculated by the Student
Skill Score Calculator 132 in step 1066, during the calculation of
Generalized Percent Correct results. [0775] Under this framework,
the Room to Grow.TM. function can be chosen heuristically so long
as it satisfies the following properties: [0776] i.
.rho.(x,y,z)>0 for all x and y in [0,1] and all z>0. [0777]
ii. .rho.(1,y,z)=0 for all y.di-elect cons.[0,1] and all z>0.
[0778] iii. .rho.(x,0,z)=0 for all x.di-elect cons.[0,1] and all
z>0. [0779] iv. .rho.(x,y,z).fwdarw.0 as z.fwdarw.0, for all x
and y in [0,1] [0780] v. For all y.di-elect cons.[0,1] and all
z>0, x'>x''.rho.(x,y,z)<.rho.(x'',y,z) [0781] vi. For all
x.di-elect cons.[0,1] and all z>0,
y'>y''.rho.(x,y'',z)>.rho.(x,y'',z) [0782] vii. For all x and
y in [0,1], z'>z''.rho.(x,y,z')>.rho.(x,y,z'') [0783] This
method is advisable only when the distribution of maximum point
totals among the Skills (a) reflects explicit educational choices
on the part of the Client; and (b) is approximately stable from one
administration of the test to another. [0784] C. In the last family
of methods, which may improve accuracy, but at the expense of
understandability, the Room to Grow.TM. function estimates growth
potential by positing a probabilistic model of teaching and
learning, and using the model to predict where the greatest growth
can occur. [0785] To be precise, in this approach one adopts:
[0786] i. A multicomponent-ability Item Response Theory model, and
[0787] ii. A probabilistic model of instruction and ability, taking
place within the multicomponent ability model, that allows one to
assess the expected magnitude of group improvement in the given
Skill, following additional instructional focus. This expected
magnitude serves then as the Room to Grow.TM. function .rho..
[0788] The multicomponent-ability Item Response Theory model in (i)
is a framework within which one asserts that each Student in the
testing population has an independent ability in each Skill
Category. The higher the Student's ability in a given Skill, the
better able the Student is to answer Items in that Skill correctly.
The probabilistic model in (ii) amounts to a series of probability
distributions, one for each Skill. The k.sup.th probability
distribution P.sub.k(s.sub.2|s.sub.1) gives the probability density
that a Student who achieved a Skill Score s.sub.1 in Skill k will,
after additional instruction in Skill k, achieve the Skill Score
s.sub.2 in Skill k. [0789] It is to be expected that in any simple
and reasonable model of instruction and learning, the expected
improvement for any given Student upon additional instruction,
<s.sub.2-s.sub.1>=.intg.s.sub.2P.sub.k(s.sub.2|s.sub.1)ds.sub.2-s.s-
ub.1, [0790] will be positive. This expected increase in Skill
Score is then the Room to Grow.TM. function:
.rho..sub.k=.intg.s.sub.2P.sub.k(s.sub.2|s.sub.1)ds.sub.2-s.sub.1.
[0791] As a very simple model of P.sub.k(s.sub.2|s.sub.1) that
serves to illustrate the ideas, it can be supposed that given a
particular Student, it is known only that (a) the Student's score
will increase upon additional instruction; and (b) all other
Students will perform the same. Then P.sub.k(s.sub.2|s.sub.1) might
be taken proportional to .theta.(s)n.sub.k(s), where
.theta.(s).ident.0 for s.ltoreq.s.sub.1 and .theta.(s).ident.1 for
s>s.sub.1 [0792] and n.sub.k(S) is "the curve" in Skill k, that
is, n.sub.I(s) is the probability density for Students to perform
at each Skill Score for the Skill in question. (A model this crude
will in general have unsatisfactory properties, but it is described
here for illustrative purposes.)
[0793] In step 1268, the computer selects those Skills which, for
Student j, have the highest number of Room to Grow.TM. points. The
selection can be governed by any number of rules. For example, one
may wish to ensure a uniform User experience by always choosing the
three Skills (for example) with the highest Room to Grow.TM. point
totals, breaking ties, for example, in favor of Skills with higher
benchmark Skill Score values.
[0794] One may also wish to detect the "strongest signal" in the
Room to Grow.TM. point total information. Thus, the computer may
(a) calculate the standard deviation .quadrature. of the m Room to
Grow.TM. point values; (b) eliminate those Skills lying more than
0.2.quadrature. (for example) from the maximum; and (c) of these,
choose up to three (for example).
[0795] In step 1270, the computer assigns an appropriate Evaluation
Statement (such as `Room to Grow.TM. Skill`) to the Student for
those Skills selected in step 1268, and assigns an appropriate
Evaluation Statement (such as `Not a Room to Grow.TM. Skill`) to
the Student for those Skills not selected in step 1268.
[0796] In step 1272, the computer increments the value of the
Student index j by unity, and checks in Decision 1274 whether j is
greater than the overall number of Students p. If not, then the
computer returns to step 1266 to process the next Student.
[0797] If in Decision 1274 the Student index j exceeds the total
number of Students p, then in step 1276 the computer writes all
Students' Evaluation Statements in all Skills to the Information
Repository 114 and in step 1278 notifies the Student Skill
Performance Evaluator Operator that processing is complete.
[0798] 4.G. Aggregation Analyzer
[0799] Aggregation simply means combining information about each
Student in a Group in order to produce information about the Group
as a whole. Aggregation is extremely important in providing
Educators with an overall picture of their Students, since it is
often impractical for Students to receive one-on-one attention.
[0800] Some forms of aggregation are extremely simple, such as
averaging together Student scores to produce a score for the Group.
Other forms of aggregation are much more complex, such as
determining a Group's overall Evaluation Statement on the basis of
the Evaluation Statements and/or numerical Scores of the Group
Members. The different aggregations required for different Displays
are specified in the Data Requirements for Results Reports.
[0801] Broadly speaking, there are four kinds of aggregation that
the Aggregation Analyzer 136 can implement in different embodiments
to provide data for different Displays:
[0802] Score-to-Score Aggregation is any process whereby individual
Students' Skill Scores are used to determine a Skill Score for the
Group. For example, a simple averaging of each Student's
Generalized Percent Correct to obtain an Average Percent Correct
for the Group constitutes Score-to-Score Aggregation.
[0803] Statement-to-Statement Aggregation is any process whereby
individual Students' Evaluation Statements are used to determine an
overall Evaluation Statement for the Group. For example,
determining a Group's "Room to Grow" Skills, based purely on which
Skills are "Room to Grow" Skills for the individual Students in the
Group, constitutes Statement-to-Statement Aggregation.
[0804] Score-to-Statement Aggregation is any process whereby
individual Students' Skill Scores are used to determine an overall
Evaluation Statement for the Group. For example, determining a
Group's "Room to Grow" Skills, based purely on the numerical Scores
of Students in the Group, constitutes Score-to-Statement
Aggregation.
[0805] Full-Information Statement Aggregation is any process
whereby individual Students' Skill Scores and individual Students'
Evaluation Statements are used to determine an overall Evaluation
Statement for the Group. For example, determining a Group's "Room
to Grow" Skills, based on both the numerical Scores of Students in
the Group as well as a knowledge of which Skills are "Room to Grow"
Skills for the individual Students in the Group, constitutes
Score-to-Statement Aggregation.
[0806] The Displays in a given Report produced under this system
may include instances of all these forms of aggregation.
Physical Instantiation
[0807] The Aggregation Analyzer 136 may consist physically of (a) a
human Operator who activates the Aggregation Analyzer 136 according
to the flowchart in FIG. 13 when notified to do so in step 628 by
the Analysis Module Operator or in step 1254 by the Student Skill
Performance Evaluator Operator; and (b) a computer connected to the
network that executes appropriate code as described in more detail
below.
Processing Performed by the Computer Code
[0808] The Aggregation Analyzer 136 takes from the Information
Repository 114 the following inputs: [0809] Data Requirements for
Results Reports; [0810] Skill Scores for all members of all Groups
to be aggregated over; and [0811] Evaluation Statements for all
members of all Groups to be aggregated over.
[0812] The Aggregation Analyzer 136 writes to the Information
Repository 114 the following outputs: [0813] Each Student's Vector
of Evaluation Statements (or Vectors, in the case of multiple types
of aggregation as suggested above).
[0814] The transformation of these inputs into these outputs is
accomplished by computer code, the processing ranging in complexity
from simple averaging to more complex forms of aggregation as in
the embodiments below.
[0815] FIG. 13 is a flowchart showing the process of operation of
the Aggregation Analyzer 136.
[0816] In step 1302, the Aggregation Analyzer Operator retrieves
the Data Requirements for Results Reports from the Information
Repository 114. Based on the Data Requirements, the Operator
determines in step 1304 which form(s) of aggregation must be
implemented in order to create the Displays in the Report. For
example, it may happen that three different kinds of aggregation
are required to produce all of the required Displays: [0817] 1. For
each Group, and for each Skill, the Generalized Percent Correct
values of each Group member in that Skill must be averaged in order
to produce the Group's Average Percent Correct. [0818] 2. For each
Group, the Group's Average Percent Correct values for each Skill
must be used to determine the Group's "Room to Grow" Skills. [0819]
3. For each Group, and for each Skill, the Evaluation Statements of
each Group member in that Skill must be combined with the
Generalized Percent Correct values of each Group member in that
Skill, to produce the Group's overall Evaluation Statement.
[0820] Each of these aggregations is to be accomplished by running
a computer program that (a) reads data from the Information
Repository 114; (b) carries out the particular form of aggregation
required (for example, one of those described in the embodiments
below); and (c) writes the results to the Information Repository
114.
[0821] Having determined in step 1304 the forms of aggregation
required, the Aggregation Analyzer Operator consults the Code
Library in step 1306, to see whether the Code Library contains code
for implementing the required form(s) of aggregation.
[0822] Code may exist in the Code Library for performing some of
the desired aggregations, but not for performing others. If in
Decision 1308 all of the necessary code does not exist (as
naturally arises, for example, in the case of Displays that have
not been provided previously to any Client), the Operator in step
1310 uses the skills of his/her art to write the necessary code,
depositing it in the Code Library so it will be available for use
in reporting to future Clients. (In this way, the Analysis Code
Library becomes richer over time.)
[0823] The necessary code now exists in the Code Library, so when
in Decision 1308 this code is found to exist, the Operator in step
1312 loads it into the computer, executing it in step 1314.
[0824] In step 1315 the computer accesses the Information
Repository 114 and, for all Groups to be analyzed, retrieves the
Group members' Scores and Evaluation Statements in all Skills. In
step 1316 computer then calculates aggregate data for the Group as
instructed in the computer code (for example by following an
algorithm such as one of those specified in the embodiments below).
Having done so, the computer in step 1317 writes the Group
aggregate data to the Information Repository 114 and in step 1318
notifies the Operator that processing is complete.
Embodiment: Aggregation of Formative Evaluation Statements
[0825] Despite the benefits of differentiated instruction (a term
that refers to separating Students into work groups based on their
levels of mastery, and pursuing different strategies with each work
group), teachers are often faced with the need to adopt a single
strategy for an entire class. For example, this happens during
curriculum planning at the start of an academic term or at the
start of a content-specific unit. It also happens during direct
instruction, when the teacher explains concepts to the class as a
whole.
[0826] Many methods exist in the prior art for determining
diagnostic information about individual Students based on their
Test performance. In some instances, this diagnostic information
may include "Formative Statements" about each student--statements,
that is, which recommend a particular course of action that should
likely be taken by the Student's Educator(s) and/or Parent(s), or
even by the Student himself or herself.
[0827] For example, Formative Statements about Student Test Results
might include suggestions that one Student work on the fundamentals
of subtraction, while another Student proceed to advanced work in
that subject. These Formative Statements can be very useful for
Users, such as Educators can take such recommended courses of
actions during direct instruction.
[0828] But Educators often have to address Groups of Students
collectively. Therefore, Educators can benefit from generalizing
the Formative Statements they receive for each Student to obtain
and `overall picture` of the Group as a whole. Unfortunately,
however, this need has not been addressed in the prior art, with
the result that Educators have not until now been able to obtain
formative evaluation statements for Groups of Students. The present
invention remedies this problem.
[0829] An obvious approach to aggregating Formative Statements
might proceed as follows: (1) calculate a single average score for
the Group; (2) imagine that this score represents the score of a
single Student; and then (3) assign the Group the same Formative
Statement that a Student would receive, had he or she received the
Group average Score.
[0830] However, an approach based on averaging will not be
appropriate for determining the best way to address the needs of a
Group collectively. This is because averaging does not take into
account the varying degree to which the different actions
recommended by different formative statements will affect different
individual Students.
[0831] The Aggregation Analyzer 136 in various embodiments offer a
superior approach. Once the Student Skill Performance Evaluator 134
is completed, this embodiment of the Aggregation Analyzer 136
instantiates an algorithm that aggregates individual Formative
Statements that have been offered for Students, in order to
generate a single formative statement for the Group as a whole. In
this embodiment, the particular algorithm functions by selecting
the Formative Statement for the Group that corresponds to the
instructional action that maximizes the educational benefit
accruing to all of the Group members.
Specific Embodiment: Utility Aggregation of Formative
Statements
[0832] In the example of Utility Aggregation described here, the
Range of Formative Evaluation Statements being aggregated consists
of three Formative Statements, denoted R={f, p, a}. These take
these statements to be arranged in order, that is, beginning with
an action more appropriate to a Student with low mastery (f),
progressing to an action more appropriate to a Student with an
intermediate level of mastery (p), and ending with an action more
appropriate to a Student with high mastery (a).
[0833] For example, we might have: [0834] f=`Work on Fundamentals`;
[0835] p=`[Provide] Additional Instruction and Practice`; and
[0836] a=`[Pursue] Advanced Work`.
[0837] Or, for example, if Users are using the information in
Reports to inform their curriculum planning decisions at the start
of an academic term, we might have: [0838] f="Set modest goals
during curriculum planning"; [0839] p="Set intermediate goals
during curriculum planning"; and [0840] a="Set high goals during
curriculum planning".
[0841] Utility Aggregation works on a Skill-by-Skill basis. In its
simplest form, it is a Statement-to-Statement Aggregation process.
That is, given: [0842] a Group of Students; [0843] each Student's
assigned formative statement in the Skill (f, p, or a); and [0844]
pedagogical constraint parameters as discussed below; the Utility
Aggregation process returns a single formative statement (f, p, or
a) appropriate for the Group as a whole. Background: Utility Models
and Pedagogical Constraints
[0845] As specified more precisely below, Utility Aggregation is
based on a 3-by-3 payoff matrix, Z.sup.0, of the form Z ~ = ( Z ff
Z fp Z fa Z pf Z pp Z pa Z af Z ap Z aa ) . ##EQU5##
[0846] In a typical matrix element of Z.sup.0, of the form {tilde
over (Z)}.sub.xy, the first subscript index refers to an action
taken by the Group, and the second index refers to an action
appropriate to the individual. Here, looking at the first row of
{tilde over (Z)}, the number Z.sub.ff represents the educational
benefit that accrues to an individual Student in a Group, when the
Student merits the recommendation f, and when the Student's
Instructor chooses to follow the recommendation f with the Group as
a whole. The number Z.sub.fp represents the (presumably smaller)
educational benefit that accrues to an individual Student in a
Group, when the Student merits the recommendation f, and when the
Student's Instructor chooses to follow the recommendation p with
the Group as a whole. And finally, the number Z.sub.fa represents
the (presumably still smaller) educational benefit that accrues to
an individual Student in a Group, when the Student merits the
recommendation f, and when the Student's Instructor chooses to
follow the recommendation a with the Group as a whole. The elements
in the p and a rows have similar interpretations.
[0847] As stated above, in the matrix element {tilde over
(Z)}.sub.xy, the first index refers to action taken by the Group,
and the second index refers to the action appropriate to the
individual. To make more intuitive the notation that follows, we
will begin capitalizing the first index, to emphasize that this
component refers to a Group. Thus, the matrix elements of {tilde
over (Z)} will appear as Z.sub.Ff, etc. However, bear in mind that
`F` and `f` both refer to the same Evaluation Statement, namely
f.
[0848] It is necessary that these matrix elements obey the
following reasonableness constraints: [0849]
Z.sub.ff.quadrature.Z.sub.fp.quadrature.Z.sub.fa [0850]
Z.sub.pp.quadrature.Z.sub.pf [0851] Z.sub.pp.quadrature.Z.sub.pa
[0852] Z.sub.aa.quadrature.Z.sub.ap.quadrature.Z.sub.af [0853]
Z.sub.ff.quadrature.Z.sub.pf.quadrature.Z.sub.af [0854]
Z.sub.pp.quadrature.Z.sub.fp [0855] Z.sub.pp.quadrature.Z.sub.ap
[0856] Z.sub.aa.quadrature.Z.sub.pa>Z.sub.fa.
[0857] Essentially, these constraints require that the matrix
elements become smaller as one moves away from the diagonals in any
direction (horizontally or vertically). The first set of
inequalities mean that any given Student receives maximal benefit
when the Group adopts the approach that is most appropriate to him
or her, and receives less and less benefit, the more disparate the
Group approach becomes from his or her preferred approach. The
second set of inequalities mean that the Educational institution as
a whole receives maximal benefit (in terms of meeting its goal of
reaching every Student) when it provides the approach that matches
the Student's needs, and receives less and less benefit, the more
disparate the approach it provides is from the Student's needs.
[0858] Decision-making models based on a payoff matrix are commonly
used in economic theory and in the study of ethics. However, this
invention represents the first time they have been applied to the
problem of making sound recommendations to instructors.
[0859] Decision-making models based on a payoff matrix can be
difficult to implement in practice, because it is often difficult
to choose values for the matrix elements. The Aggregation Analyzer
136 solves this problem by prompting one or more persons to use
pedagogical expertise in setting constraints on the possible
elements of the payoff matrix. (This prompting need be carried out
at most once per Client.)
[0860] For example, suppose that a person with pedagogical
expertise, when faced with a class of at least one-third of the
students meriting the f recommendation and the remainder meriting
the p recommendation, would always adopt the f approach with the
class. If the decision-making rule imposed by the payoff matrix is
to respect this mandate, then the matrix elements themselves will
have to satisfy a mathematical constraint arising from the special
role of the number 1/3 in the mandate. Such a constraint is called
a Pedagogical Constraint.
[0861] Given a set of Pedagogical Constraints, the aggregation
routine below calculates a payoff matrix Z.sup.0 that satisfies
these constraints.
[0862] This payoff matrix, by itself, leads to a simple and
reasonable aggregation rule, based only on the diversity of
Students' recommended actions (f, p, or a) within the Group. (This
is a Statement-to-Statement Aggregation process.) This rule can be
implemented whenever simplicity is a goal of the aggregation
process (for example, when the methods used in the Analysis Module
120 will be explained to Users).
Specific Utility Aggregation Algorithm
[0863] FIG. 14 is a flowchart showing the sequence of steps that
can be carried out in order to implement Utility Aggregation.
[0864] In step 1402, the Operator of the Aggregation Analyzer 136
accesses the Information Repository 114 to determine whether or not
it contains Pedagogical Constraints for the current Client.
[0865] If in Decision 1404 Pedagogical Constraints do not exist,
then the Operator must convene a group of one or more human
Decision Makers in step 1406 and initiate a sub-process consisting
of steps 1408, 1410, 1412, 1414, 1416, 1418, 1420, 1422, 1424,
1426, 1428, and 1430. In this sub-process, prompts are given to the
Decision Maker(s) in order to elicit their decisions; the
Pedagogical Constraints corresponding to these decisions are
calculated; and the values of these Pedagogical Constraints are
then written to the Information Repository 114. This sub-process
can be executed by a human moderator or a computer; for
definiteness, this embodiment is written as though the steps are
performed by a computer.
[0866] Note that steps need be carried out only once per Client;
thus, in the typical case, the Operator will proceed from Decision
1404 directly to step 1432. However, in those cases in which steps
1406, 1408, 1410, 1412, 1414, 1416, 1418, 1420, 1422, 1424, 1426,
1428, and 1430 must be carried out, they are as follows.
[0867] In step 1408 the computer enters a loop controlled by the
index s and initiates the value of s to unity. The index s is a
"Scenario" index, and the loop will cycle through three different
Scenarios. Each Scenario involves a hypothetical class of 25
Students who, it is imagined, are about to receive direct
instruction in a Skill.
[0868] In the first Scenario (s=1), the class has no Students who
merit the highest recommendation a. FIG. 14ais a drawing that
illustrates figuratively the 26 different possible Class Breakdowns
consistent with this Scenario: [0869] all 25 Students in the f
column; [0870] 24 Students in the f column and one Student in the p
column; [0871] 23 Students in the f column and two Students in the
p column; and so on, ending with [0872] all 25 Students in the p
column.
[0873] In the second Scenario (s=3), the class has no Students who
merit the lowest recommendation f. FIG. 14bis a drawing that
illustrates figuratively the 26 different possible Class Breakdowns
consistent with this Scenario: [0874] all 25 Students in the p
column; [0875] 24 Students in the p column and one Student in the a
column; [0876] 23 Students in the p column and two Students in the
p column; and so on, ending with [0877] all 25 Students in the a
column.
[0878] In the third Scenario (s=3), the class has no Students who
merit the intermediate recommendation p. FIG. 14cis a drawing that
illustrates figuratively the 26 different possible Class Breakdowns
consistent with this Scenario: [0879] all 25 Students in the a
column; [0880] 24 Students in the a column and one Student in the f
column; [0881] 23 Students in the a column and two Students in the
f column; and so on, ending with [0882] all 25 Students in the f
column.
[0883] For the sake of clarity, steps 1410, 1412, 1414, 1416, 1418,
1420, 1422, 1424, 1428, and 1430 will be described for each value
(1, 2, or 3) of the Scenario index s.
[0884] Scenario 1
[0885] In step 1410 (with s=1), Scenario 1 is described to the
Decision Makers. At this point the computer enters a loop in step
1412 controlled by the loop index t and initializes the value of t
to unity. The index t is a Class Breakdown index; as described
below, in cycling through the loop the computer will present the
Decision Makers with a succession of the 26 different Class
Breakdowns B.sub.1, . . . , B.sub.26 belonging to the Scenario 1
(which Class Breakdowns are shown in FIG. 14a).
[0886] In step 1414 the Decision Makers are shown Class Breakdown
B.sub.t, and the computer pauses while the Decision Makers discuss
which approach they would take with the class as a whole (the
possible choices being stipulated as f or p in Scenario 1), given
the distribution of Student needs represented by that Breakdown.
The Decision Makers can signal the completion of the discussion by
striking a key on the computer keyboard.
[0887] After the Decision Makers have finished discussing Class
Breakdown B.sub.t, the computer in step 1416 prompts each Decision
Maker to vote for the approach (f or p) he or she would take with
the class as a whole. Given a Breakdown B.sub.t, each
Decision-Maker votes "f" or "p". The votes are cast by typing them
on the computer keyboard.
[0888] In step 1418 the computer tallies the votes, storing the
percentage of the Decision Makers voting "f" (this quantity is
denoted F.sub.t) as well as the percentage of the Decision Makers
voting "p" (this quantity is denoted P.sub.t). Note that
P.sub.t=1-F.sub.t.
[0889] In step 1420 the computer increments the value of the Class
Breakdown index t by unity, and in Decision 1422 checks whether the
value of t exceeds 26, the number of Class Breakdowns. If in
Decision 1422 the value of t is less than or equal to 26, the
computer repeats steps 1414, 1416, 1418, and 1420 for the Class
Breakdown B.sub.t corresponding to the incremented value of t.
[0890] After the Decision Makers have voted on all 26 Class
Breakdowns, the computer in step 1420 will increment the value of t
to 27, and the Decision in 1422 will cause the computer to proceed
to step 1424, the calculation of a Pedagogical Constraint based on
the Decision Makers' votes for the Class Breakdowns in Scenario 1.
This Pedagogical Constraint is denoted a and calculated as
a.ident.t.sub.0/25 where t.sub.0 is the smallest value of t in the
interval 2, . . . , 25 for which the "Dissent Quantity"
(F.sub.t-1+F.sub.t+F.sub.t+1)(P.sub.-1+P.sub.t+P.sub.t+1) takes on
its maximal value. This condition identifies the point in the
voting at which the group of Decision Makers reached their maximal
level of dissent (or, in the case of one Decision Maker, the Class
Breakdown B.sub.t0 for which the Decision Maker first changed his
or her initial vote).
[0891] As an example of such a calculation, suppose that there are
five Decision Makers, and that the voting for the Class Breakdowns
in Scenario 1 plays out in the following typical fashion:
TABLE-US-00027 Dissent Class "f" "p" Quantity Value Break- "f" "p"
Pct. Pct. (F.sub.t-1 + F.sub.1 + F.sub.t+1) .times. of t down votes
votes F.sub.1 P.sub.1 (P.sub.t-1 + P.sub.1 + P.sub.t+1) 1 B.sub.1 7
0 1.00 0.00 2 B.sub.2 7 0 1.00 0.00 0.00 3 B.sub.3 7 0 1.00 0.00
0.00 4 B.sub.4 7 0 1.00 0.00 0.00 5 B.sub.5 7 0 1.00 0.00 0.00 6
B.sub.6 7 0 1.00 0.00 0.00 7 B.sub.7 7 0 1.00 0.00 0.00 8 B.sub.8 7
0 1.00 0.00 0.400 9 B.sub.9 6 1 0.86 0.14 0.762 10 B.sub.10 6 1
0.86 0.14 1.385 11 B.sub.11 5 2 0.71 0.29 1.385 12 B.sub.12 6 1
0.86 0.14 1.642 13 B.sub.13 5 2 0.71 0.29 1.840 14 B.sub.14 4 3
0.57 0.43 2.010 15 B.sub.15 5 2 0.71 0.29 2.128 16 B.sub.16 4 3
0.57 0.43 2.245 17 B.sub.17 2 5 0.29 0.71 2.128 18 B.sub.18 2 5
0.29 0.71 1.853 19 B.sub.19 2 5 0.29 0.71 1.642 20 B.sub.20 1 6
0.14 0.86 1.105 21 B.sub.21 0 7 0.00 1.00 0.400 22 B.sub.22 0 7
0.00 1.00 0.00 23 B.sub.23 0 7 0.00 1.00 0.00 24 B.sub.24 0 7 0.00
1.00 0.00 25 B.sub.25 0 7 0.00 1.00 0.00 26 B.sub.26 0 7 0.00
0.00
[0892] The value of t.sub.0 in this example is t.sub.0=16, because
the Dissent Quantity first takes on its maximal value (2.245)
there; and it is easily seen from the voting pattern that this is
where the dissent of the group is most pronounced. In step 1424,
this example would therefore lead to the Pedagogical Constraint
a=16/25=0.64.
[0893] In step 1426 the computer writes to the Information
Repository 114 the value of the Pedagogical Constraint calculated
in step 1424 and increments the Scenario index by unity.
[0894] Scenario 2
[0895] In step 1410 (with s=2), Scenario 2 is described to the
Decision Makers. At this point the computer enters a loop in step
1412 controlled by the loop index t and initializes the value of t
to unity. The index t is a Class Breakdown index; as described
below, in cycling through the loop the computer will present the
Decision Makers with a succession of the 26 different Class
Breakdowns B.sub.1, . . . , B.sub.26 belonging to the Scenario 2
(which Class Breakdowns are shown in FIG. 14b).
[0896] In step 1414 the Decision Makers are shown Class Breakdown
B.sub.t, and the computer pauses while the Decision Makers discuss
which approach they would take with the class as a whole (the
possible choices being stipulated as p or a in Scenario 2), given
the distribution of Student needs represented by that Breakdown.
The Decision Makers can signal the completion of the discussion by
striking a key on the computer keyboard.
[0897] After the Decision Makers have finished discussing Class
Breakdown B.sub.t, the computer in step 1416 prompts each Decision
Maker to vote for the approach (p or a) he or she would take with
the class as a whole. Given a Breakdown B.sub.t, each
Decision-Maker votes "p" or "a". The votes are cast by typing them
on the computer keyboard.
[0898] In step 1418 the computer tallies the votes, storing the
percentage of the Decision Makers voting "p" (this quantity is
denoted P.sub.t) as well as the percentage of the Decision Makers
voting "a" (this quantity is denoted A.sub.t). Note that
A.sub.t=1-F.sub.t.
[0899] In step 1420 the computer increments the value of the Class
Breakdown index t by unity, and in Decision 1422 checks whether the
value of t exceeds 26, the number of Class Breakdowns. If in
Decision 1422 the value of t is less than or equal to 26, the
computer repeats steps 1414, 1416, 1418, and 1420 for the Class
Breakdown B, corresponding to the incremented value of t.
[0900] After the Decision Makers have voted on all 26 Class
Breakdowns, the computer in step 1420 will increment the value of t
to 27, and the Decision in 1422 will cause the computer to proceed
to step 1424, the calculation of a Pedagogical Constraint based on
the Decision Makers' votes for the Class Breakdowns in Scenario 2.
This Pedagogical Constraint is denoted c and calculated as
c.ident.(1-t.sub.0)/25 where t.sub.0 is the smallest value of t in
the interval 2, . . . , 25 for which the "Dissent Quantity"
(P.sub.t-1+P.sub.t+P.sub.t+1)(A.sub.t-1+A.sub.t+A.sub.t+1) takes on
its maximal value. This condition again identifies the point in the
voting at which the group of Decision Makers reached their maximal
level of dissent (or, in the case of one Decision Maker, the Class
Breakdown B.sub.t0 for which the Decision Maker first changed his
or her initial vote).
[0901] In step 1426 the computer writes to the Information
Repository 114 the value of the Pedagogical Constraint calculated
in step 1424 and increments the Scenario index by unity.
[0902] Scenario 3
[0903] In step 1410 (with s=3), Scenario 3 is described to the
Decision Makers. At this point the computer enters a loop in step
1412 controlled by the loop index t and initializes the value of t
to unity. The index t is a Class Breakdown index; as described
below, in cycling through the loop the computer will present the
Decision Makers with a succession of the 26 different Class
Breakdowns B.sub.1, . . . , B.sub.26 belonging to the Scenario 3
(which Class Breakdowns are shown in FIG. 14c).
[0904] In step 1414 the Decision Makers are shown Class Breakdown
B.sub.t, and the computer pauses while the Decision Makers discuss
which approach they would take with the class as a whole (the
possible choices being stipulated as f, p, or a in Scenario 3),
given the distribution of Student needs represented by that
Breakdown. The Decision Makers can signal the completion of the
discussion by striking a key on the computer keyboard.
[0905] After the Decision Makers have finished discussing Class
Breakdown B.sub.t, the computer in step 1416 prompts each Decision
Maker to vote for the approach (f, p, or a) he or she would take
with the class as a whole. Given a Breakdown B.sub.t, each
Decision-Maker votes "f", "p" or "a". In Scenario 3, the voting is
constrained by the following two rules: (1) each Decision-Maker
must vote "f" for B.sub.1, and (2) each Decision-Maker must vote
"a" for B.sub.26. The votes are cast by typing them on the computer
keyboard.
[0906] In step 1418 the computer tallies the votes.
[0907] In step 1420 the computer increments the value of the Class
Breakdown index t by unity, and in Decision 1422 checks whether the
value of t exceeds 26, the number of Class Breakdowns. If in
Decision 1422 the value of t is less than or equal to 26, the
computer repeats steps 1414, 1416, 1418, and 1420 for the Class
Breakdown B.sub.t corresponding to the incremented value of t.
[0908] After the Decision Makers have voted on all 26 Class
Breakdowns, the computer in step 1420 will increment the value of t
to 27, and the Decision in 1422 will cause the computer to proceed
to step 1424, the calculation of two Pedagogical Constraints based
on the Decision Makers' votes for the Class Breakdowns in Scenario
3. These Pedagogical Constraints are denoted b and d, and are
calculated as follows: [0909] 1. The computer determines the
following two numbers for each Decision Maker n: t.sub.1n, the
largest value oft for which the Decision-Maker voted "f" on the
corresponding Breakdown B.sub.t, and t.sub.2n, the smallest value
of t for which the Decision-Maker voted "f" on the corresponding
breakdown. [0910] 2. The computer calculates the average values of
t.sub.1n and t.sub.2n over the group (averaging over n), calling
the results t.sub.1 and t.sub.2, respectively. [0911] 3. If
t.sub.1.gtoreq.t.sub.2, then the computer calculates the numbers b
and d via the formulas b=max(a, 1-((t.sub.1+t.sub.2)/2-1)/25)
d=min(c, 1-((t1+t.sub.2)/2-1)/25); [0912] whereas if
t.sub.1<t.sub.2, then the computer calculates the numbers b and
d via the formulas b=max{a, 1-(t.sub.1-1)/25} d=min{c,
1-(t.sub.2-1)/25}. [0913] Examination of the voting process will
show that the min and max expressions here, which ensure (i) that
the value of b must be at least as large as the value of a, and
(ii) that the value of d must be no larger than the value of c, are
necessary to ensure that the voting respects the following two
reasonableness conditions: [0914] i. "p" students should not be
more influential in driving the class towards more advanced
teaching than "a" students are (this requires a.ltoreq.b); and
[0915] ii. "p" students should not be more influential in driving
the class towards less advanced teaching than "f" students are
(this requires c.ltoreq.d).
[0916] In step 1426 the computer writes to the Information
Repository 114 the values of the Pedagogical Constraints calculated
in step 1424 and increments the Scenario index by unity.
[0917] In Decision 1428 the computer now determines that the
Scenario index is greater than 3, and proceeds to step 1432 where
it checks the Information Repository 114 for the presence of the
payoff matrix Z.sub.0. If in Decision 1434 the computer does not
find the payoff matrix, it loads the Pedagogical Constraints a, b,
C, and d from the Information Repository 114 in step 1436, and in
step 1438 calculates the payoff matrix Z.sub.0by means of the
following formula, Z 0 = ( ~ - 1 ~ .times. 1 - ( b / ( 1 - b ) )
.times. ( ( 1 - a ) / a ) - ( 1 - a ) / a ~ ~ .times. 1 - ( 1 - a )
/ a - ( 1 - d ) / d - ( 1 - c ) / c ~ ) ##EQU6## writing Z.sup.0 to
the Information Repository 114 in step 1440.
[0918] This brings the computer to step 1442, which, in a typical
implementation of the Aggregation Analyzer 136, would actually be
reached after a short series of steps and decisions as follows:
1402, 1404, 1432, 1434, 1442. In step 1442 the computer reads the
payoff matrix Z.sup.0 from the Information Repository 114.
[0919] In step 1444 the computer enters a loop controlled by the
index k, initializing the value of k at 1. The index k is a Skill
index, ranging from 1 to m. The computer has thus begun looping
through the Skills in order.
[0920] In step 1446 the computer loads Student Evaluation
Statements in Skill k for all Students in the first Group to be
analyzed.
[0921] Using the Student Evaluation Statements in Skill k, the
computer calculates in step 1448 the Group's "Population Vector"
v=(v.sub.f, v.sub.p, v.sub.a). Here, the number v.sub.f is between
0 and 1, and represents the percentage of Students in the Group who
have Evaluation Statement f in Skill k. Likewise, the number
v.sub.p is between 0 and 1, and represents the percentage of
Students in the Group who have Evaluation Statement p in Skill k;
and the number v.sub.a is between 0 and 1, and represents the
percentage of Students in the Group who have Evaluation Statement a
in Skill k. (The components of the Population Vector v satisfy the
relationship v.sub.f+v.sub.p+v.sub.a.quadrature.1.)
[0922] In step 1450, the computer calculates the per-person utility
or benefit associated with the three Group-level actions f, p, or
a, by multiplying the population vector v by the payoff matrix
Z.sup.0 to yield the Utility Vector u.ident.Z.sup.0v: u = ( Z ff Z
fp Z fa Z pf Z pp Z pa Z af Z ap Z aa ) .times. ( v f v p v a ) = (
Z ff .times. v f + Z fp .times. v p + Z fa .times. v a Z pf .times.
v f + Z pp .times. v p + Z pa .times. v a Z af .times. v f + Z ap
.times. v p + Z aa .times. v a ) .ident. ( u F u P u A ) ##EQU7##
where the nine matrix elements Z.sub.ff, Z.sub.fp, . . . ,
Z.sub.ap, Z.sub.aa are given in terms of the Pedagogical
Constraints a, b, c, and d as above.
[0923] In the Utility Vector u=(u.sub.F, u.sub.p, u.sub.A), the
first number UF represents the per-person utility for the Group
that is associated with the Group-level action f; the second number
up represents the per-person utility for the Group that is
associated with the Group-level action p; and the third number
u.sub.A represents the per-person utility for the Group that is
associated with the Group-level action a.
[0924] In step 1452 the computer determines the maximal utility
component u.sub.F, u.sub.p, or u.sub.A. If there is a single
largest component, then the computer selects that component as the
maximal utility component. In the rare event of ties, the computer
must follow a tie-breaking rule such as the following: [0925] i. If
u has two maximal components, then: [0926] When they are u.sub.F
and u.sub.P, the maximal utility component is u.sub.P. [0927] When
they are u.sub.F and u.sub.A, the maximal utility component is
u.sub.A. [0928] When they are u.sub.P and u.sub.A, the maximal
utility component is u.sub.A. [0929] ii. If all three components of
u are equal, then the maximal utility component is u.sub.P.
[0930] In step 1454 the computer selects in the obvious way the
Group Evaluation Statement corresponding to the maximal utility
component determined in step 1452, selecting f when the maximal
utility component is u.sub.F, selecting p when the maximal utility
component is u.sub.P, and selecting a when the maximal utility
component is u.sub.A.
[0931] In step 1456 the computer writes the Evaluation Statement
for the Group to the Information Repository 114.
[0932] In step 1458 the computer increments the value of the Skill
index k by unity and in Decision 1460 checks whether the value k is
greater than the number of Skills m. If k is not greater than m,
then the computer returns to step 1448 to begin the process of
assigning the Group and Evaluation Statement in the next Skill
(steps 1448, 1450, 1452, 1454, 1456, and 1458).
[0933] When the computer finds in Decision 1460 that the value of k
is greater than the number of Skills m, then the computer proceeds
in step 1462 to check the Information Repository 114 for the
presence of unprocessed Groups. If in Decision 1464 the computer
finds that there are additional Groups to be processed, then it
returns to the Skill loop in step 1444. When the computer finds in
Decision 1464 that there are no Groups remaining, it proceeds to
step 1466 and notifies the Operator that the aggregation process is
complete.
[0934] The Utility Aggregation process as described in the above
embodiment aggregates formative statements for any Group of
Students in such a way as to deliver maximal educational benefit to
the Students in the Group, as that notion is given meaning by
pedagogical expertise. Interestingly, whether or not the Decision
Makers implicitly or explicitly considered utilitarian principles
in making their judgments, there is nevertheless a linear utility
model that reproduces their insight for the Breakdowns considered
by them.
[0935] The process is efficient because the Pedagogical Constraints
can be calculated once for each Client (and need not be calculated
for each Test analyzed on behalf of that Client), and the same is
true for the payoff matrix Z.sup.0.
Embodiment: High-Dispersion Group Aggregation
[0936] The Utility Aggregation method described in the above
embodiment seeks to maximize the utility of a chosen action when a
uniform choice must be made for a Group. In some applications,
however, the goal of aggregating the Evaluation Statements of a
Group is not to recommend a single action to be pursued with the
Group as a whole, but, rather, simply to give a sense of which
Evaluation Statement best typifies the Group. Here again, though,
averaging is impractical.
[0937] In the second embodiment, the Aggregation Analyzer 136
instantiates an algorithm that measures the central tendency of
individual Students' ordered evaluation statements in a manner that
resists such regression toward the mean. A significant advantage of
the method is that is extremely simple to explain to Users: the
method simply selects that Evaluation Statement which appears most
commonly in the Group. The deceptive simplicity of this method
renders surprising its effectiveness at maintaining high-dispersion
recommendations. The effectiveness of this "Plurality Method"
arises from the following two circumstances: [0938] 1. The set of
all Population Vectors v=(v.sub.poor, v.sub.intermediate,
v.sub.good) satisfying v.sub.poor+v.sub.intermediate+v.sub.good=1
forms an equilateral triangle in the three-dimensional space of all
vectors (x, y, z); and [0939] 2. The Plurality Method divides this
set symmetrically into three regions A.sub.poor,
A.sub.intermediate, and A.sub.good, all having the same area:
[0940] A.sub.poor={(x, y, z): x>y and x>z} [0941]
A.sub.intermediate={(x,y, z): y>x and y>z} [0942]
A.sub.good={(x, y, z): z>x and z>y}.
[0943] Population Vectors drawn from the region A.sub.poor are
assigned the Group Evaluation Statement "Poor" by the Plurality
Method; Population Vectors drawn from the region A.sub.intermediate
are assigned the Group Evaluation Statement "Intermediate" by the
Plurality Method; and Population Vectors drawn from the region
A.sub.Good are assigned the Group Evaluation Statement "Good" by
the Plurality Method. With no one region having greater area than
any other, the assignments have "equal representation"--at least,
absent any knowledge of the actual distribution of Evaluation
Statements within typical Groups-and therefore enjoy high
dispersion.
Embodiment: Growth-Based Recommendations for Groups
[0944] Just as there does not exist in the prior art a method of
prioritizing Skills within a diagnostic Student report so that the
Users' attention is drawn first to those Skills in which rapid
growth may be possible, there also does not exist in the prior art
a method of making growth-based recommendations for Groups of
Students. In this embodiment, an easily understood method for
making such recommendations within the Aggregation Analyzer 136 is
described.
[0945] Given the Skill Score Vectors s=(s.sub.1.sup.j,
s.sub.2.sup.j, . . . , s.sub.m.sup.j) for each Student j in a
Group, the code executed within the Aggregation Analyzer 136 can
simply calculate the average Skill Score of the Group in each Skill
k: s.sub.k.sup.avg={tilde over
(g)}.sup.1(s.sub.k.sup.1+s.sub.k.sup.2+ . . . s.sub.k.sup.g) where
s.sub.k.sup.j is the Skill Score in Skill k for the j.sup.th
Student in the Group, and the Group contains g Students. The code
can then calculate the Group's Room to Grow.TM. points in each
Skill, using whichever Room to Grow.TM. function .quadrature. has
been implemented in the Student Skill Performance Evaluator (see
step 1266 as well as the description thereof given during the
description of the Room to Grow.TM. Mapping .quadrature..sub.RTG),
selecting the Group's Evaluation Statement in each Skill (`Room to
GroWTM Skill` or `Not a Room to Grow.TM. Skill`) based on Room to
Grow.TM. points, again by whichever method has been implemented in
the Student Skill Performance Evaluator (see steps 1268 and
1270).
[0946] 4.H. Statistics Summarizer
[0947] Users typically want to see a wide of range of data about
the performance of Students and Groups on Tests. Therefore, it is
important that a system and method for processing and displaying
test results perform all the necessary calculations to produce the
displays of Test Results approved by the Client.
[0948] The Client specifications, called "Data Requirements for
Results Reports", are stored in the Information Repository 114. The
Statistics Summarizer 138 uses these requirements to determine the
types of data calculations needed to be performed, performs these
calculations on data stored in the Information Repository 114, and
then stores the final data in the Information Repository, so that
the Report Generator 144 can retrieve the final data and generate
the appropriate types of displays.
[0949] The Statistics Summarizer 138 is activated in step 630 as
follows: After the Aggregation Analyzer 136 is completed, the
Operator of the Aggregation Analyzer 136 notifies the Operator of
the Analysis Module 120. The Operator of the Analysis Module 120
then notifies the Operator of the Statistics Summarizer 138 who
activates the Statistics Summarizer 138.
[0950] FIG. 14aillustrates the method of operation of the
Statistics Summarizer 138:
[0951] In step 1480, the Operator of the Statistics Summarizer 138
executes code, which retrieves from the Information Repository 114
the Data Requirements for Results Reports. The Data Requirements
for Result Reports are text files that specify what data is needed
in order to generate each display that concerns Student Test
Results. For example, a Client may want to display how a specific
Group of Students (as defined by gender, race, or socioeconomic
status) performed in a specific Skill. The required data here would
include numerical and qualitative information about each Student's
performance in each Skill, as well as information that will enable
the appropriate Group of Students to be identified.
[0952] Based on the information retrieved in step 1480, the
Operator in step 1482 determines which data calculations need to be
performed. For example, if the Data Requirements for Results
Reports include a requirement that a group's overall math
performance be shown as the percentage of girls who passed versus
the percentage of boys who passed, then the data calculations would
be simply to (a) divide [the number of girls who passed] into [the
total number of girls] to obtain a percentage, and (b) divide [the
number of boys who passed] into [the total number of boys].
[0953] In step 1484, the Operator searches the Code Library 222 for
code to implement the statistical calculations. In the above
example, the code sought would be quite simple: merely the ability
to divide one number into another and calculate a percentage.
[0954] Obviously, more complex calculations will be needed
depending on the Data Requirements for Results Reports. For
example, it might be necessary to calculate trends in Students'
overall percentage correct across past administrations of similar
Tests. Similarly, the Statistics Summarizer 138 might calculate
"disaggregated data" that breaks down by Group of Students, such as
the percentage of students of each race (or by other category such
as gender, socioeconomic class, etc.) who have been deemed to meet
that state's educational standards in the Subject of the Test.
Indeed, one skilled in the art will recognize that many possible
data calculations will involve much more sophisticated processes,
such as obtaining standard deviations, stanine scores, and linear
regressions, and therefore more sophisticated code.
[0955] In step 1486, the Operator determines whether all necessary
code exists. If not, then the Operator adds the necessary code to
the code library in step 1488. The Operator can do this by
programming code or by obtaining the code from elsewhere, including
by purchase.
[0956] In step 1490, having determined that the necessary code
exists, the Operator loads the code from the Code Library 222, and
in 1492 the Operator executes the code. When executed, the code
retrieves Test information from the Information Repository 114 and
performs the necessary calculations, according to the
specifications provided in the Data Requirements for Results
Reports. Finally, in step 1494, the code writes this information to
the Information Repository.
[0957] Physically, the Statistics Summarizer 138 may consist of:
(a) a human Operator who follows the flowchart FIG. 14ato operate
the Statistics Summarizer 138 when notified to do so by the
Analysis Operator; (b) one or more desktop computers 218 with
network access to the Data Servers 214 and to one or more
application servers 220 running commercial analysis software (such
as SAS.RTM., SPLUS.RTM., Mathematica.RTM., BILOG.RTM., etc.); and
(c) a Code Library 222, which is simply an electronic file storage
system (such as a hard drive) for storing custom-written code as
necessary for implementing different components of the Statistics
Summarizer.
[0958] 4.I. Content Manager
[0959] The Content Manager 140 ensures that the Content Repository
116 contains all of the content that will be required for creating
Instructional Response Reports in step 1622.
[0960] FIG. 15 is a flowchart illustrating the process of operation
of the Content Manager 140.
[0961] The Content Manager 140 may consists physically of (a) a
human Operator who performs the steps indicated in the flowchart
FIG. 15 when notified in step 632 to do so by the Analysis Module
Operator; (b) a Content Management Computer 224, with network
access to the Content Server 234, which is used by the Content
Manager Operator; (c) one or more Authors, who create new content
when notified in step 1508 to do so by the Content Manager
Operator; and (d) one or more Content Authoring Computers 228, with
network access to the Content Server 234, used by the
Author(s).
[0962] In step 1502 the Content Manager Operator reads from the
Information Repository 114 the Content Requirements for
Instructional Response. The Content Requirements for Instructional
Response will specify, among other things, whether the
Instructional Response should include content for each Skill in the
Skill Set and for each Skill Family in the Organization; whether
the Instructional Response should include content for Skills other
than those in the Skill Set; whether the Instructional Response
should include content for Skill Families or other larger groupings
of Skills in the Organization and/or outside of the Organization;
whether the Instructional response should include content other
than that associated with subject areas, such as content intended
to help Users integrate the Reports into their workflow, whether
that workflow consists of teaching, training teachers, supervising
teachers, tutoring Students, or Studying; and so on.
[0963] In the most common case, the Content Requirements for
Instructional Response will mandate, among other things, an
Instructional Response that includes content for each Skill in the
Skill Set and for at least some Skill Families in the Organization.
Thus in step 1503 the Content Manager Operator reads the Skill Set
and Organization from the Information Repository 114.
[0964] Using the Content Requirements for Instructional Response
and the Skill Set and Organization, the Content Manager Operator
creates in step 1504 a list of required "Content Categories," along
with specified "Elements" for each. These terms are explained
next.
[0965] A Content Category is a subject-matter heading for content
that will be delivered to Users in the form of Instructional
Responses. Thus, a partial list of Content Categories for
fourth-grade mathematics might look like the following: [0966] 1.
Addition and Subtraction [0967] 2. Addition and
Subtraction-Computation [0968] 3. Addition and Subtraction-Problem
Solving [0969] 4. Problem Solving in Mathematics [0970] 5. Holding
grade-wide conferences with Teachers [0971] 6. Doing math with your
child [0972] 7. Differentiated instruction
[0973] An Element is a potential component of an Instructional
Response associated with a given Content Category. Different
Content Categories naturally have different Elements that are
germane to them, as in this partial example (Elements are listed
here in no particular order): TABLE-US-00028 Content Category
Element Element Element Element Element . . . Addition and
Explanation Common Diagnostic Activity Activity for Subtraction of
category student Problem for Students difficulty Students Likely to
Likely to Need Help Be Ready With for Fundamentals Advanced Work
Addition and Explanation Lesson Plan Common Practice Diagnostic . .
. Subtraction- of category student problem problem Computation
difficulty Addition and Explanation Lesson Plan Common Practice
Diagnostic . . . Subtraction- of category student problem problem
Problem Solving difficulty Problem Solving in Explanation Lesson
plan Common Practice Diagnostic . . . Mathematics of category
student problem problem difficulty Holding grade- Motivation
Planning a Questions to . . . wide conferences for a gradewide ask
teachers with Teachers gradewide conference in a conference
gradewide conference Doing math with Motivation Practice Book list
. . . your child for doing problem math with your child
Differentiated Theory of Using Help Instruction differentiated
differentiated teachers in instruction instruction in your school
your use classroom differentiated instruction
[0974] The breakdown of instructional content into Content
Categories, with various Elements native to each, allows different
Users to receive arrays of content that are not only targeted to
their particular role within the educational system, but that also
are tightly tied to the Skills being tested, and that, in addition,
allow a given Content Category to be presented to Students at range
of different levels--in fact, at the same range of levels reported
to users in the Test Results as levels of mastery.
[0975] Some Elements may be understood within the Content Manager
140 as having sub-Elements as well; for example, a "Practice
Problem" Element may necessarily connote a collection of
sub-Elements consisting of a Problem Statement, one or more
Solution Methods, and one or more Answers.
[0976] With the list of required Content Categories and Elements as
a reference, the Content Manager Operator in step 1506 inventories
the Content Repository 116 to identify which required Elements of
which required Content Categories cannot be found there. Depending
on the physical instantiation of the Content Repository 116, this
inventory may be carried out in different ways, for example by
clicking through a directory structure or querying an XML
database.
[0977] Based on the results of this inventory, the Content Manager
Operator in step 1508 notifies one or more Authors of the Elements
that must be authored. In step 1510 the Author(s) use the skills of
their art to create these Elements and write them to the Content
Repository 116. Finally, in step 1512 the Author(s) notify the
Content Manager Operator that content creation is complete.
Glossary of Technical Terms
[0978] The definitions and notation presented in this Glossary are
intended to clarify and standardize the descriptions of the
Analysis Module's components.
>Item Score
[0979] For each Item, the Item Score is a number that represents a
given Student's level of accomplishment on a particular Test item.
For a multiple-choice Item, the Item Score is typically 0 or 1. For
constructed-response Items, the Item Score typically ranges from 0
to some integer greater than 1.
>Item Score Range, H(.quadrature..sub.i)
[0980] The Item Score Range H(.quadrature..sub.i) is a set of real
numbers giving the range of possible Item Scores for Item
.quadrature..sub.i, as stipulated by the Scoring Guide. For
example, we might have H(.quadrature..sub.1)={0,1} for a
multiple-choice item .quadrature..sub.1, and
H(.quadrature..sub.2)=[0,3] for a `three-point question`
.quadrature..sub.2.
>Weight of an Item
[0981] The top score of the Item Score Range is known as the weight
of Item .quadrature.i.di-elect cons.T, and is denoted W.sub.i>0.
The weight of each Item may be stipulated by the designer of the
Test, or by the agency giving the Test, or by another external
entity. Among common motivations for the weighting are (a)
correspondence to the perceived difficulty of the different Items,
or (b) correspondence to the expected time required for the
different Items. For example, the Test may have been created such
that each multiple choice Item has a weight 2 and each constructed
response (e.g., short answer or essay) Item has a weight of 5.
>Item Score Vector, y=(y.sub.1, . . . , y.sub.n)
[0982] An Item Score Vector is an ordered n-tuple of numbers
y=(y.sub.1, Y.sub.n), where the i.sup.th number y.sub.i represents
a Student's Item Score for the i.sup.th Item on the Test. Note that
y.sub.i.di-elect cons.H(.quadrature..sub.i) for each i, and that y
is therefore an element of the n-fold Cartesian product
H(.quadrature..sub.1).times.H(.quadrature..sub.2).times. . . .
.times.H(.quadrature..sub.n). >Logical Space of Item Score
Vectors, H
[0983] Any logically-possible Student Item Score Vector y must be
an element of the n-fold Cartesian product
H.ident.H(.quadrature..sub.1).times.H(.quadrature..sub.2).times. .
. . .times.H(.quadrature..sub.n).
[0984] The space H is therefore called the Logical Space of Item
Score Vectors.
>Actual Space of Item Score Vectors, Y
[0985] The Actual Space of Item Score Vectors Y.OR right.H is
defined as Y=.orgate.{y.sub.j}:j=1, . . . ,p where y.sub.j.di-elect
cons.H is the Item Score Vector of the j.sup.th Student taking the
Test. To put it another way, a vector y.di-elect cons.H is a member
of the Actual Space of Item Score Vectors Y iff there exists at
least one Student who obtained y as his or her Item Score Vector.
Not every Item Score Vector in H will be found among thep Item
Score Vectors of Students taking the Test. For example, in a very
difficult test, it may happen that no Student attains the Item
Score Vector y.sup.perfect=(y.sub.1.sup.max, . . .
,y.sub.n.sup.max) where y.sub.i.sup.max.ident.max(H.sub.i). For
purposes of algorithmic efficiency, it is often beneficial to
ignore many of the logically possible Item Score Vectors, and to
work with the relatively small set of Item Score Vectors actually
obtained by Students taking the Test. On the other hand, it is
usually easier to define algorithms and mappings in terms of the
larger space, restricting attention to the smaller space during
implementation. >Skill Set, S.sub.T
[0986] A Skill Set S.sub.T is a set of Skills, posited as being
necessary and sufficient for successfully responding to the Items
on Test T The number of Skills in a Skill Set S.sub.T will always
be denoted m. As a matter of convention, the index k will always
range over the values 1, 2, . . . , m. Thus, any quantity with a
subscript k refers to a particular Skill in the Skill Set
S.sub.T.
>Coding Matrix, Q(S.sub.T, T)
[0987] A Coding Matrix Q(S.sub.T, T) is an m-by-n matrix of
non-negative numbers Q.sub.ki in which rows correspond to Skills in
a Skill Set S.sub.T, and columns correspond to Items on the Test T.
Typically, the matrix element Q.sub.ki lies in the interval [0,1]
and is posited as representing numerically the degree to which the
i.sup.th Item in Test T assesses the k.sup.th Skill in S.sub.T. For
example, in the special case in which each Item assesses a single
Skill, the Coding Matrix will consist only of 0's and 1's, with a
single 1 appearing in each column. A Coding Matrix is sometimes
called a Q-matrix, except that a Q-matrix traditionally contains as
matrix elements only 0's and 1's. A Raw Coding Matrix is a Coding
Matrix that reflects purely pedagogical judgment. A Refined Coding
Matrix is a Coding Matrix that takes into account both pedagogical
judgment and statistical features of Students' responses to Items
on the Test.
>Skill-Specific Item Score
[0988] If the symbol .quadrature..sub.k.sup.i denotes the i.sup.th
Test Item requiring Skill k--where an Item i is said to "require" a
Skill k whenever the relevant matrix element Q.sub.ki of the
Refined Coding Matrix is nonzero--then a Student's Score on Test
Item .quadrature..sub.k.sup.i is correspondingly denoted
y.sub.k.sup.i and called a Skill-Specific Item Score. Note that
each y.sub.k.sup.i is one of the Item Scores y, in the Student's
Item Score Vector y.
>Skill-Specific Item Score Vector, y.sub.k=(y.sub.k.sup.1,
y.sub.k.sup.2, . . . , y.sub.k.sup.l(k))
[0989] If (k) denotes the number of Items on a Test requiring Skill
k, then the ordered (k)-tuple of Skill-Specific Item Scores
y.sub.k.sup.i for a fixed Skill k is called the Skill-Specific Item
Score Vector and denoted y.sub.k. Intuitively, the Skill-Specific
Item Score Vector y.sub.k is just that part of the Student's Item
Score Vector y that pertains to the particular Items requiring
Skill k.
>Logical Space of Skill-Specific Item Score Vectors, H.sub.k
[0990] Any logically-possible Skill-Specific Item Score Vector yk
must be an element of the (k)-fold Cartesian product
H.sub.k.ident.H(.quadrature..sub.k.sup.1).times.H(.quadrature..sub.k.sup.-
2).times. . . . .times.H(.quadrature..sub.k.sup.(k)).
[0991] The space H.sub.k is therefore called the Logical Space of
Skill-Specific Item Score Vectors. H.sub.k is the analogue of the
Logical Space of Item Score Vectors H, except this time for a
specific Skill S.sub.k.di-elect cons.S.sub.T.
>Actual Space of Skill-Specific Item Score Vectors, Y.sub.k
[0992] For a given Skill S.sub.k.di-elect cons.S.sub.T, the Actual
Space of Skill-Specific Item Score Vectors Y.sub.k.OR right.H.sub.k
is defined as Y.sub.k=.orgate.{y.sub.kj}:j=1, . . . ,p, where
y.sub.kj.di-elect cons.H.sub.k is the Skill-Specific Item Score
Vector for Skill S.sub.k of the j.sup.th Student taking the Test.
To put this another way, a vector y.sub.k.di-elect cons.H.sub.k is
a member of the Actual Space of Skill-Specific Item Score Vectors
Y.sub.k for Skill S.sub.k iff there exists at least one student who
obtained y.sub.k as his or her Skill-Specific Item Score Vector for
that Skill. Not every Skill-Specific Item Score Vector in H.sub.k
will be found among the p Skill-Specific Item Score Vectors of
Students taking the Test. For example, in a very difficult test, it
may happen that no Student attains the Skill-Specific Item Score
Vector y.sub.k.sup.perfect=(y.sub.k.sup.1.sub.max, . . . ,
y.sub.k.sup.(k).sub.max) where
y.sub.k.sup.i.sub.max.ident.max(H(.quadrature..sub.k.sup.i)).
>Item Score Array, .quadrature.
[0993] When all m of a Student's Skill-Specific Item Score Vectors
y.sub.k are collected together, the resulting collection of numbers
is called an Item Score Array, and denoted .quadrature. = ( y 1 y 2
.differential. y m ) . ##EQU8##
[0994] Item Score Arrays are two-dimensional arrays, but they are
not rectangular, because each Skill may have a different number (k)
of Items Assessing it.
>Logical Space of Item Score Arrays, H
[0995] Any logically-possible Item Score Array .quadrature. can be
regarded as an element of the m-fold Cartesian product
H.ident.H.sub.k.times.H.sub.k.times. . . . .times.H.sub.k.
[0996] The space H is therefore called the Logical Space of Item
Score Arrays.
>Actual Space of Item Skill Arrays, .quadrature.. The Actual
Space of Item Skill Arrays .quadrature..OR right.H is defined as
.quadrature.=.orgate.{.quadrature..sub.j}:j=1, . . . ,p where
.quadrature..sub.j.di-elect cons.H is the Item Score Array of the
j.sup.th Student taking the Test. To put it another way, an Item
Skill Array .quadrature..di-elect cons.H is a member of the Actual
Space of Item Skill Arrays .quadrature. iff there exists at least
one student who obtained .quadrature. as his or her Item Skill
Array. >Skill Score, s.sub.k
[0997] A Skill Score s.sub.k is any number reflecting a Student's
performance in a particular Skill S.sub.k.di-elect
cons.S.sub.T.
>Obtained Skill Score in a Skill
[0998] A number x is called an Obtained Skill Score in a Skill if
there is at least one Student who has obtained a Skill Score
s.sub.k=x for the Skill in question.
>Set of Obtained Skill Scores in a Skill, O.sub.k
[0999] For each Skill S.sub.k.di-elect cons.S.sub.T, there are
finitely many Obtained Skill Scores. We denote these numbers by
o.sub.k.sup.1, o.sub.k.sup.2, . . . , o.sub.k.sup.r(k). Here, r(k)
represents the number of Obtained Skill Scores in Skill k. The
superscript indices are understood to be assigned in such a way
that o.sub.k.sup.1.ltoreq.o.sub.k.sup.2.ltoreq. . . .
.ltoreq.o.sub.k.sup.r(k) for each k. The set
O.sub.k={o.sub.k.sup.1, o.sub.k.sup.2, . . . , o.sub.k.sup.r(k)} is
called the Set of Obtained Skill Scores in Skill k. >Skill Score
Vector, s
[1000] A Skill Score Vector s=(s.sub.1, . . . , s.sub.m) is an
ordered m-tuple of numbers, where the k.sup.thnumber represents the
Skill Score s.sub.k for the k.sup.th Skill in the Skill Set
S.sub.T. Note that each component s.sub.k of s is an element of the
relevant set of Obtained Skill Scores O.sub.k.
>Logical Space of Skill Score Vectors, U
[1001] Any logically-possible Skill Score Vector s must be an
element of the m-fold Cartesian product
U=O.sub.1.times.O.sub.2.times. . . . .times.O.sub.m.
[1002] The space U is therefore called the Logical Space of Skill
Score Vectors. U is the analogue of the Logical Space of Item Score
Vectors H, but this time the scores in question are Skill Scores,
not Item Scores.
>Actual Space of Skill Score Vectors, S
[1003] The Actual Space of Skill Score Vectors S.OR right.U is
defined as S=.orgate.{s.sub.j}:j=1, . . . , p,
[1004] where s.sub.j.di-elect cons.U is the Skill Score Vector of
the j.sup.th Student taking the Test. To put it another way, a
vector s.di-elect cons.U is a member of the Actual Space of Skill
Score Vectors S iff there exists at least one student who obtained
s as his or her Skill Score Vector for that Skill.
>Skill Score Mapping, .quadrature.
[1005] A Skill Score Mapping is any mapping .quadrature.:
.quadrature..fwdarw. .sub.m. Symbolically, the action of
.quadrature. is given by .quadrature.(.quadrature.)=s. In words, a
Skill Score Mapping converts an Item Score Array
.quadrature..di-elect cons..quadrature. into a Skill Score Vector
s.di-elect cons.S. Note that the Actual Space of Skill Score
Vectors S is the image of the Actual Space of Item Score Arrays
.quadrature. under the mapping .quadrature.:
S=.quadrature.(.quadrature.). >Factorizable Skill Score Mapping,
m ##EQU9##
[1006] A Skill Score Mapping .quadrature.: .quadrature..fwdarw.
.sub.m is said to be Factorizable if there exist mappings
.quadrature..sub.k: H.sub.k.fwdarw. so that .quadrature.((y.sub.1,
y.sub.2, . . . , y.sub.m))=(.quadrature..sub.1(y.sub.1),
.quadrature..sub.2(y.sub.2), . . . , .quadrature..sub.m(y.sub.m))
for each .quadrature.=(y.sub.1, y.sub.2, . . . , y.sub.m) in H.
When the action of .quadrature. does not decompose in this way, we
say that .quadrature. is non-Factorizable. A convenient notation
for a Factorizable Skill Score Mapping is m ~ ##EQU10## >Total
Weight in a Skill, w.sub.k
[1007] Given a Coding Matrix Q(S.sub.T, T) with matrix elements
Q.sub.ki, the Total Weight in Skill k is
w.sub.k=.SIGMA.[Q.sub.ki:i=1, . . . , n].
[1008] For example, in the special case in which all Items are
multiple-choice Items with Item Score Ranges {0,1}, and in which no
single Item assesses more than one Skill, the Total Weight in a
Skill w.sub.k is simply equal to the total number of Items
assessing Skill k.
>Generalized Percent Correct in a Skill, c.sub.k
[1009] Given a Coding Matrix Q(S.sub.T, T) with matrix elements
Q.sub.ki, along with a particular Student's Item Score Vector y,
that Student's Generalized Percent Correct in Skill k is
c.sub.k=w.sub.k.sup.-1.SIGMA.[Q.sub.kiy.sub.i:i=1, . . . , n].
[1010] For example, in the special case in which all Items are
multiple-choice Items with Item Score Ranges {0,1}, and in which no
single Item assesses more than one Skill, the Generalized Percent
Correct c.sub.k is equal to the number of Items Assessing Skill k
answered correctly by the Student, divided by the total number of
Items Assessing Skill k.
>Score-Statement Mapping, .quadrature.
[1011] A Skill Score Mapping .quadrature..sub.R is any mathematical
function : S.fwdarw.E(R) that transforms a Student's Skill Score
Vector s=(s.sub.1, . . . , s.sub.m) into a Vector of Evaluation
Statements e=(e.sub.1, . . . , e.sub.m). Here, s.sub.1 denotes the
Student's Skill Score for Skill 1 and e.sub.1 denotes the Student's
Evaluation Statement for Skill 1, drawn from the Range of
Evaluation Statements R. Usually the Range of Evaluation Statements
under discussion is understood, so .quadrature..sub.R is simply
written .quadrature..
>Factorizable Score-Statement Mappings,
.quadrature.=(.quadrature..sub.1, . . . , .quadrature..sub.m)
[1012] A Score-Statement Mapping : S.fwdarw.E(R) is said to be
Factorizable when it "treats Skills independently," i.e., when
there exist m Skill-specific mappings
.quadrature..sub.k:O.sub.k.fwdarw.R such that the action
.quadrature.(s)=e decomposes as .quadrature.((s.sub.1, s.sub.2, . .
. , s.sub.m))=(.quadrature..sub.1(s.sub.1),
.quadrature..sub.2(s.sub.2), . . . , .quadrature..sub.m(s.sub.m))
for every s.di-elect cons.U. Intuitively, a Factorizable
Score-Statement Mapping is one that considers each Skill
independently; whereas a Non-Factorizable Score-Statement Mapping
is one that "takes the entire Skill Score Vector into account."
When the mappings .quadrature..sub.k exist, it will be said that
".quadrature. Factorizes." An economical notation for a
Factorizable Score-Statement Mapping is
.quadrature.=(.quadrature..sub.1, . . . , .quadrature..sub.m). 5.
Display Module
[1013] The system and method of the Display Module 142 are designed
to address very specific problems in the current systems and
methods of displaying Student Test Results and associated
instructional information. Namely, existing display processes do
not present Users with the data and information that they need in
order to respond to Students' needs. Some of these problems are as
follows:
[1014] First, current display methods do not utilize simultaneous
print and Internet delivery of information. In particular, current
methods fail to invite Users online via printed reports that
include passwords for accessing an account on the Internet to see
additional information presented electronically.
[1015] Second, current display methods do not provide critical
instructional information to Users together with the Test Results.
Therefore, they fail to render the Test Results meaningful for
Users who seek to take action in response to students'
instructional needs as revealed by the Test.
[1016] Third, current display methods do not permit Users to
navigate between critical information easily. For example, in
current methods of electronic display, an Educator cannot click on
a Skill name to see her class's Student-by-Student breakdown in
that Skill.
[1017] Fourth, current display methods do not permit Users to track
Students' progress in Skills over time using diagnostic measures
that are simple and efficient to implement.
[1018] The Display Module 142 addresses each of these issues using
systems and methods not previously employed in the current art.
[1019] It is important to understand that the application of the
Display Module 142 need not be integrated with the application of
the Analysis Module: Although the Display Module 142 obviously can
display the Test Results produced by the Analysis Module 140, the
Display Module 142 alternatively could take as its input any type
of Test Results that are stored and organized within the
Information Repository 114.
[1020] For example, some methods of computerized testing (such as
methods known as "adaptive testing") assess different Students with
different Items, and complex scoring methodologies are used to
generate Scores that are comparable across Students. Provided that
Aggregate and Skill-Specific Scores are stored in the Information
Repository 114 together with the other requisite information such
as Student Identifying Information and Associative Information, the
Display Module 142 can display such Scores in the same way that it
displays the Scores generated from a Test in which all Students
take the same Items.
[1021] The Display Module 142 consists of three components: the
Report Generator 144, the Print Displayer 150, and the Electronic
Displayer 152. The system and method of each of these components is
described in detail below.
[1022] 5.A. Report Generator
[1023] After the Analysis Module 120 is completed in step 304, the
System Operator is notified and notifies the Display Operator to
activate the Display Module 142 in Step 306. The Display Operator
then notifies the Report Generator Operator, which initiates the
Report Generator process by activating the Results Report Generator
in step 1602, as shown in FIG. 16. The Display Operator may be an
individual or a computer operating according to a script. The
Report Generator Operator is typically one or more individuals.
[1024] The Report Generator 144 consists of two subcomponents: the
Results Report Generator 146 and the Instructional Response Report
Generator 148. It is not necessary for both of these report
generators to run, although it may be advantageous if Users see
Test Result information together with information about how Users
can respond instructionally to students' needs.
[1025] These subcomponents of the Report Generator 144 extract
information from the Information Repository 114 and Content
Repository 116 respectively, and they assemble that information
into electronic reports called respectively Results Reports and
Instructional Information Reports. (FIG. 5 illustrates the key
components held in the Report Repository 506: namely the Results
Reports 576, Instructional Response Reports 578, Print Templates
580, and Electronic Templates 582. These components are each
discussed more fully below.) The Print Displayer 150 and Electronic
Displayer 152 then utilize these reports to create displays viewed
by Users.
[1026] One important benefit of the method described here, of
course, is that a single electronic report can be used to generate
a particular display in print and a display on the Internet.
Because the electronic report functions as a single source for both
types of displays, this helps eliminate the possibility of data
being inconsistent between a print and electronic display of the
same report. In addition, this method takes advantage of
efficiencies in parallel processing, as the Results Reports and
Instructional Reports can be produced concurrently, rather than
sequentially.
[1027] Even more important, the system and method described here
enable flexible processing and display of Test Results and
instructional materials so that the displays can be generated in
multiple formats and media for various Users, according to the
demands of the Client.
[1028] 5.A. 1. Results Report Generator
[1029] The Results Report Generator 146 extracts information from
the Information Repository 114 in order to create electronic
reports that describe Student Test Results in a particular manner.
These so-called "Results Reports" can take various forms. For
example, a Result Report could be generated on each class in a
school, showing the national percentile ranking of each Student in
that class based on the Students' performance on a Test. There are
many other potential illustrations of data performance, however,
which may be accommodated in order to satisfy the demands of the
Users of Test Results.
[1030] The method employed by the Results Report Generator 146
comprises several steps, as detailed in flowchart form in FIG.
16:
[1031] After the Results Report Generator 146 is activated by the
Report Generator Operator in step 1602, the Results Report
Generator 146 retrieves the Data Requirements for Results Reports
from the Information Repository 114 in step 1604.
[1032] As explained in the description of the Intake Module 102,
the Data Requirements for Results Reports are electronic
specifications that detail the data needed to generate each display
concerning Student Test Results that the Client wants shown to
Users.
[1033] For example, a Client may wish for teachers to see their
students' national percentile rank in a given Subject and Level of
a Test. In that case, the required data would be the Student's
names, the Subject and Level of the Test, and each Student's
national percentile rank in that Test, in addition to the
associative information needed to link the particular Students with
a class, teacher, and school.
[1034] In a more involved case, the Client may want to display how
a specific Group of Students (as defined by gender, race, or
socio-economic status) performed in a specific Skill, plus
recommendation statements about how a teacher should proceed in
teaching each Student about that Skill. The required data here
would include numerical and qualitative information about each
Student's performance in each Skill, as well as information that
will enable the appropriate Group of Students to be identified.
[1035] For purposes of clarity, here is a list of the Data
Requirements of certain displays that may be offered to the Client
for inclusion in the Data Requirements for Results Reports.
[1036] Displays of "Overall Performance of Students" are visual
depictions of the following information fields at a minimum: [1037]
Student Identifying Information for each Student [1038] Student
Associative Information for each Student [1039] Student Overall
Performance Information for each Student
[1040] These displays also could depict other information fields,
such as: [1041] Student Demographic Information for each Student
[1042] Student Performance History for each Student
[1043] Displays of "Overall Performance of Groups" are visual
depictions of the following information fields at a minimum: [1044]
Group Identifying Information for each Group [1045] Group
Associative Information for each Group [1046] Group Overall
Performance Information for each Group
[1047] These displays also could depict other information fields,
such as: [1048] Group Demographic Information for each Group [1049]
Group Performance History for each Group
[1050] Displays of "Skill Profile of a Student" are visual
depictions of the following information fields at a minimum: [1051]
Student Identifying Information for each Student [1052] Student
Associative Information for each Student [1053] Scores by Skill for
each Student and/or Verbal Evaluation Statement by Skill for each
Student
[1054] These displays also could depict other fields of
information, such as: [1055] Student Demographic Information for
each Student [1056] Student Performance History for each Student
[1057] Student Overall Performance Information for each Student
[1058] Displays of "Skill Profile of a Group" are visual depictions
of the following information fields at a minimum: [1059] Group
Identifying Information for each Group [1060] Group Associative
Information for each Group [1061] Scores by Skill for each Group
and/or Verbal Evaluation Statement by Skill for each Group
[1062] These displays also could include other fields of
information, such as: [1063] Group Demographic Information for each
Group [1064] Group Performance History for each Student [1065]
Group Overall Performance Information for each Group
[1066] Displays of "Performance in a Skill across Students" are
visual depictions of the following information fields at a minimum:
[1067] Student Identifying Information for each Student [1068]
Associative Information for each Student [1069] Scores in one or
more Skills for each Student and/or Verbal Evaluation Statements in
one or more Skill for each Student
[1070] These displays also could depict other fields of
information, such as: [1071] Student Demographic Information for
each Student [1072] Student Performance History for each Student
[1073] Overall Performance Information for each Student
[1074] Displays of "Performance in a Skill across Groups" are
visual depictions of the following information fields at a minimum:
[1075] Group Identifying Information for each Group [1076]
Associative Information for each Group [1077] Scores in one or more
Skills for each Group and/or Verbal Evaluation Statements in one or
more Skill for each Group
[1078] These displays also could include other fields of
information, such as: [1079] Group Demographic Information for each
Group [1080] Group Performance History for each Group [1081]
Overall Performance Information for each Group
[1082] These displays are discussed at greater length in the
description of the Electronic Displayer 152 below.
[1083] The next step 1606 is for the Results Report Generator 146
to define an electronic document structure for Results Reports
based on the Data Requirements.
[1084] In order to understand this step more fully, consider the
case of a Client that wants the principal of a school to see many
tables of the same type, namely tables which show Performance in a
Skill across Students, for multiple classes and multiple Skills.
Those skilled in the art will recognize that the electronic
document structure to be defined in step 1606 advantageously
permits incorporation, at a minimum, of the following information:
[1085] Test Event Name [1086] Group Identifying Information for
that Group, including the teacher name and class identification
[1087] Skill Name [1088] Student names in that Group, associated
with the appropriate Verbal Evaluation Statement for each Student
in each Skill
[1089] FIG. 17 uses this example to show how the Report Generator
Operator creates an electronic document structure based on
particular Data Display Requirements. In that example, Data Display
Requirement 1736 specifies various fields, including teacher name
1702, school 1704, subject and level 1706, class 1708, date of test
1710, name of a skill examined 1712, and lists of students assigned
each of three Evaluation statements based on their performance in
that skill 1714, 1716,1718.
[1090] The Results Report Generator Operator, by means of the
judgment of one skilled in the art, transforms the Data Display
Requirement into an electronic document structure 1738. Many of the
elements of the electronic document structure are simply
transpositions from the Data Display Requirements: namely, elements
1720, 1722, 1724, 1726, 1728, and 1729. In the case of student
names, however, each of the requirements 1714, 1716, and 1718 is
transformed into a structural element to be filled by a string of
names 1730, 1732, and 1734.
[1091] (Discussed in more detail below, FIG. 20 extends the example
shown in FIG. 17, as the particular Data Display Requirement and
electronic document structure in FIG. 17 form the basis of the
Results Report and ultimately the Electronic Display in FIG.
20.)
[1092] The next step 1608 in the operation of the Results Report
Generator 146 is to store these electronic document structures in
the Report Repository 118. Next, in step 1610, the Results Report
Generator 146 extracts Test Result data from the Information
Repository 114 (such as the names of Students and their performance
on the Test as a whole and individual Skills) and loads that data
into electronic document structures to create "Results Reports."
These Results Reports will serve as the basis for the displays
generated by the Print Displayer 150 and the Electronic Displayer
152.
[1093] An optional step 1612 in the operation of the Results Report
Generator 146 is to store the Results Reports in the Report
Repository 118. This would enable the Results Reports to be
accessed at a later date by the Print Displayer 150 and Electronic
Displayer 152.
[1094] Once the process of the Results Report Generator 146
terminates, the Report Generator Operator notifies the Display
Operator.
[1095] Physically, the Results Report Generator 146 may function,
in one embodiment, as follows:
[1096] The Results Report Publishing Computer 236 first obtains the
Data Requirements for Results Reports from the Data Servers 214.
The Data Servers 214 runs on a UNIX operating system or other
commonly used system. The Data Servers 214 typically may be built
using a SQL-based database management system (DBMS) such as
Oracle.RTM. or other type of DBMS. The Results Report Publishing
Computer 236 uses database access technologies such as Java
Database Connectivity (JDBC) to obtain this information over the
internal network.
[1097] The Report Generator Operator then uses an editing tool
(such as an XML DTD editing tool) to create electronic document
structures that reflect the Data Requirements. XML, or eXtensible
Mark-up Language, is a widely known, flexible, and
platform-independent software language used to create electronic
structure documents. DTDs, or Document Type Definitions, are the
corresponding language that describes the explicit structure of
such documents.
[1098] The Results Report Publishing Computer 236 then stores these
electronic document structures as DTD files on the Report Server
240. The Report Server 240 operates on a UNIX operating system and
stores the DTD in its file system.
[1099] Finally, the Results Report Publishing Computer 236 creates
individual Results Reports based on a DTD. Java technology is used
to extract the Test Result information from the Data Servers 214,
and Java and XML technology are used to transform the data based on
the rules defined by the DTD and create Results Reports in the form
of XML documents.
[1100] The Results Report Publishing Computer 236 typically stores
these Results Reports in a hierarchical structure (for example
organized by teacher/low-level administrator, high-level
administrator) on the Report Server 240. These Results Reports
could be stored using a file system or a XML-based database
management product.
[1101] 5.A.2. Instructional Response Report Generator
[1102] The Instructional Response Report Generator 148 is analogous
to the Results Report Generator 146. Just as the Results Report
Generator 146 creates Results Reports that are used to display
information concerning the Test Results, so the Instructional
Response Generator 148 creates electronic Instructional Response
Reports that are then used to display suggested Instructional
Response Materials based on Student and Group performance.
[1103] This function responds to a critical problem in the existing
methods of displaying Test Results: Existing methods do not provide
Educators with relevant instructional information to address the
needs of their Students revealed by their Students' Test Results.
This is unfortunate, because the moment in which Test Results are
reported is an ideal time at which to communicate with Educators
about recommended instructional courses of action.
[1104] The deficiency in existing methods is evident. For example,
in U.S. Pat. No. 5,934,909, Ho et al. describe a method in which
the results of individually-tailored tests are used to create a
list of additional questions in areas of Students' weakness. This
information, however, does not provide Educators with guidance
about how to instruct their Students in these areas.
[1105] Likewise, in U.S. Pat. No. 6,270,351, Roper describes an
individual education program tracking system. This system involves
the generation of reports describing a Student's educational
priorities among various disciplines. However, these reports are
not based on an analysis of a student's standardized test scores,
and so the reports do not address Students' actual strengths and
weaknesses within a given subject. Furthermore, the reports do not
include any instructional recommendations that an Educator can use
in teaching the Students.
[1106] Furthermore, organizations that display basic Test Result
information for Students and Groups do not display Instructional
Response Material based upon Students' actual results. For example,
one existing organization offers a test reporting application on
CD-ROM, but this application does not provide instructional
recommendations in the various Skills that take into account the
different performance levels of Students in those Skills.
[1107] What is needed, therefore, is a system for (1) taking
instructional content concerning the Subject of the Test and the
particular Skills examined and (2) displaying instructional
information for Educators with regard to Students and Groups in a
manner that is integrated with the presentation of the Test Results
themselves.
[1108] The operation of the Instructional Response Report Generator
comprises several steps:
[1109] After activation in step 1614, the Instructional Response
Report Generator 148 retrieves the Content Requirements from the
Information Repository 114 in step 1616.
[1110] As described in the discussion of the Intake Module 102, the
Content Requirements are text files that specify the type of the
instructional information that is to be presented along with the
Test Results. These requirements may vary by User. For example, the
Content Requirements might specify that Users who are Educators
receive Instructional Response Materials consisting of lesson plans
organized by Skill, while Users who are Students receive
Instructional Response Materials consisting of sample problems
organized by Skill.
[1111] The next step 1618 is for the Instructional Response Report
Generator 148 to define an electronic document structure for
Instructional Response Reports based on the Content Requirements.
This process is analogous to Step 1606 in which the Results Report
Generator defines an electronic document structure for Results
Reports based on the Data Requirements.
[1112] For example, take the case of a Client who wants to provide
Users who are Students with sample Test problems that they can work
on. In that case, the Content Requirements approved by the Client
would state that the material for a Skill contains three sample
problems, with two-sentences of explanation about the correct
answer to the problem. In that case, the electronic document
structure is simply an electronic template that is structured to
receive three separate problems with two-sentence explanations.
[1113] The Content Requirements approved by the Client may
typically vary by Skill: For example, instructional material that
is used to teach Skills in math is typically structured quite
differently than the instructional material used to teach Skills in
social studies.
[1114] It is important to note that a Client may approve multiple
Content Requirements even for a given Skill, in order to allow the
Print Displayer 150 and the Electronic Displayer 152 to show
Students different views depending on the Client's determination of
students' needs. For example, the Content Requirements for one type
of User (i.e., Students) with respect to a given Test could specify
two information structures: one presenting the information in
English, and another presenting the information in Spanish. In such
a scenario, the Content Requirements also may include decision-rule
information about which Students may see which type of information.
In a similarly way, the Print Displayer 150 can query the database
and, based on any manner of information stored there, determine for
each Student whether to create a display of sample problems or fun
activities.
[1115] Next, in step 1620, the Instructional Response Report
Generator 148 stores the electronic document structures for the
Instructional Response Reports, as defined by the Content
Requirements, in the Report Repository 148.
[1116] Next, in step 1622, the Instructional Response Report
Generator 148 extracts instructional content (such as the sample
problems and explanations) from the Content Repository 116 and
inserts it into the electronic document structures to create
Instructional Response Reports.
[1117] An optional step 1624 in the operation of the Instructional
Response Report Generator 148 is to store the Instructional
Response Reports in the Report Repository 118. This would enable
the Instructional Response Reports to be accessed at a later date
by the Print Displayer 150 and Electronic Displayer 152.
[1118] Once the process of the Instructional Response Report
Generator 146 terminates, the Report Generator Operator notifies
the Display Operator.
[1119] Physically, the Instructional Response Reports Generator may
function, in one embodiment, as follows:
[1120] The Instructional Response Report Publishing Computer 246
first obtains the Content Requirements for the Instructional
Response Reports from the Data Servers 214. The Data Servers 214
runs on a UNIX operating system or other commonly used system. The
Data Servers 214 typically may be built using a SQL-based database
management system (DBMS) such as Oracle.RTM. or other type of DBMS.
The Instructional Response Report Publishing Computer 236 uses
database access technologies such as Java Database Connectivity
(JDBC) to obtain this information over the internal network.
[1121] The Operator then uses an editing tool, such as an XML DTD
editing tool, to create electronic document structures that reflect
the Content Requirements. XML, or eXtensible Mark-up Language, is a
widely known, flexible, and platform-independent software language
used to create electronic structure documents. DTDs, or Document
Type Definitions, are the corresponding language that describes the
explicit structure of such documents.
[1122] The Instructional Response Report Publishing Computer 236
then stores these electronic document structures as DTD files on
the Report Server 240. The Report Server 240 operates on a UNIX
operating system and stores the DTD in its file system.
[1123] Finally, the Instructional Response Report Publishing
Computer 236 creates Instructional Response Reports based on the
DTDs. Java technology is used to extract the instructional content
materials from the Content Server 214, and Java and XML technology
are used to transform the data based on the rules defined by the
DTD and create Instructional Response Reports in XML format.
[1124] FIG. 18 shows a portion of an actual Instructional Response
Report in XML format 1802. This portion of the report concerns
strategies for teaching multiplication and division problem-solving
Skills. The strategies are divided according to Student mastery
level: strategies for those Students needing help with
"Fundamentals" in the Skill 1804; strategies for those Students
requiring additional instruction and "Practice" in the Skill 1806;
and strategies for those Students ready for "Advanced" work in the
Skill 1808. (As presented here for illustration purposes, only the
"Practice"-level strategies can be seen in detail; the
"Fundamentals"- and "Advanced"-level strategies have not been
expanded for full view.)
[1125] The Instructional Response Report Publishing Computer 236
stores these Instructional Response Reports in a hierarchical
structure (for example organizing the reports in terms of User
type, from the Student to Parent to teacher to principal to
superintendent) on the Report Server 240. These are typically
stored using the file system, but also could be stored using a
XML-based database management product.
[1126] 5.B. Print Displayer
[1127] Once the processing of the Results Report Generator and the
Instructional Response Report Generator has terminated, the Report
Generator Operator notifies the Display Operator. As shown in FIG.
16, the Display Operator then simultaneously activates the Print
Displayer 150 in step 1626 and the Electronic Displayer 152 in step
1642. These components use the Results Reports to generate displays
for the Users.
[1128] In step 1626, the Print Displayer retrieves Print
Presentation Requirements from the Information Repository.
[1129] Print Presentation Requirements are electronic text files
and prototypes that together specify the printed document or
documents that a particular type of User may receive. For example,
the Print Presentation Requirements may specify that a Student may
receive a 2-page document, one side of which is a form letter to
the Student's parent from the principal and the other side of which
is a "Skill Profile Table", where the particular types of data in
that table are specified in the Data Requirements.
[1130] The Print Presentation Requirements include all
specifications necessary for transforming the raw elements of the
Test Results and Instructional Response Materials into printed
documents for the Users. These specifications, for example, include
the colors and styling to be used on the documents, the page
length, and the location of the various displays of Test Results
and Instructional Response Materials.
[1131] The range of possible formats is familiar to those skilled
in the art of data presentation, and includes such means as tables,
pie charts, bar charts, scatter plots, and other types of charts,
icons, or graphs. In presenting the information, the displays may
sort the information by an ordering rule as applied to one or more
of the displayed fields.
[1132] Based on these specific Print Presentation Requirements, the
Print Displayer 150 generates corresponding Print Templates in step
1630 and stores them in the Report Repository 118 in step 1632.
Print Templates are electronic documents written in a templating
language known to those skilled in the art; their generation can be
carried out by the Operator of the Print Displayer 150, or in some
cases by a computer that receives as its input specific, electronic
Print Presentation Requirements.
[1133] Next, in step 1634, the Print Displayer 150 extracts User
Print Display Rules from the Information Repository 114. In step
1636, the Print Displayer 150 extracts the Print Templates, Results
Reports, and Instructional Response Reports from the Report
Repository 118, and, using the User Print Display Rules, the Print
Displayer merges the templates with the reports to create an
electronic print stream.
[1134] This print stream is then sent to a printer for electronic
imaging in step 1638, and once that imaging is complete the printed
documents may be distributed to Users in step 1640.
[1135] FIG. 19 shows an example of a printed report 1902 produced
by this method. This particular printed report 1902 does not
include any Instructional Response Materials, only Test Results
combined with static explanatory text. The Test Results shown are
displays of Students' Overall Performance 1904, the Group's
Performance by Skill 1906, and Verbal Evaluation Statements
corresponding to the Group's Performance by Skill 1908. (The names
in this report 1902 are not the names of actual people.)
[1136] This example also shows how the printed documents produced
by the Print Displayer can provide the User with access
information, so that the User can log on to a computer network and
can see the displays produced by the Electronic Displayer. The
access information box 1910 shows the User the appropriate website
address for viewing the displays produced by the Electronic
Displayer, together with a personalized Login ID and Password to
enable the User to log onto the website.
[1137] In one embodiment, printed documents produced by the Print
Displayer (as well as the electronic displays produced by the
Electronic Displayer) show Skill families that are produced by the
Skill Set Generator 650 (a component of the Item-Skill Analyzer
130). In the example shown above, these Skill families 1912 are
grouped appropriately in the numerical chart of the Group's Score
in each Skill, as well as in an explanatory section relating each
Skill to local educational standards 1914. So for example the Skill
family of "Geometry and Measurement" is composed of the individual
Skills "Time and Units of Measure" and "Geometric Terms and
Shapes".
[1138] Those skilled in the art will recognize that displays of
Test Results in print (as well as in electronic form) may be quite
complex, and they will be able to implement these displays using
commercially available software, such as graphing software. The
complexity of the displays of Test Results may depend upon the
precise Data Requirements for Results Reports as well as the
precise Presentation Requirements.
[1139] FIG. 19ais an example of a particular print display. The
chart in 1952 is a print display that portrays the percentage of
Students in each racial/ethnic group who met grade 10 math
standards in each of four years. This chart reflects the Data
Requirements for Results Reports (which specify the required
percentages for each Group in each year), as well as the Print
Presentation Requirements (which specify the chart format and
layout.)
[1140] The chart in 1952 also illustrates that the Summary
Statistics that may be prepared by the Statistics Summarizer 138
often include both "disaggregated" data and "trend data".
Disaggregated data is data that reflects breakdowns of Student
results according to Groups, while trend data is data that is
tracked across time. One skilled in the art will recognize other
forms of Summary Statistics that might be demanded as well.
[1141] Once the processing of the Print Displayer 150 is
terminated, the Display Operator is notified.
[1142] Physically, the Print Displayer may function in one
embodiment as follows:
[1143] The Print Converter Computer 244 retrieves the Print
Presentation Requirements from the Information Repository 114 using
standard file transfer methods, such as opening a visual display
program (e.g., Microsoft Visio.RTM.) or a word processing document
(e.g., Microsoft Word). These requirements could consist of visual
prototypes and text descriptions of what the print documents may
look like for each type of User. Based on these Requirements, the
Print Display Operator uses a templating language familiar to those
skilled in the art such as XSL (eXtensible Style Language) or other
language to create Print Templates, which then are stored as files
on the Report Server.
[1144] Then Print Converter Computer 244 uses common database
methods and file-system operations to retrieve User Print Display
Rules from the Information Repository 1634, as well as Print
Templates, Results Reports, and Instructional Response Reports from
the Report Repository 118.
[1145] Then, based on the User Print Display Rules, the Print
Converter Computer 244 uses a standard XSL transformation engine,
which interprets the XSL and outputs the print stream consisting of
Test Results and Instructional Response Information. The print
stream can be a complete description of the documents to be
printed, such as an Adobe PDF.RTM. file could give. A typical
output language for the print converter may be the Adobe.RTM.
PostScript.RTM. format. It is also possible to use a data merge
print language that generates the PostScript format; IBM's Advanced
Function Presentation (AFP) and Xerox's Variable data Intelligent
PostScript Printware (VIPP) are two examples of such a language, in
which the database master file creates the rules for the printer to
interpret, and the data file contains all of the variable data for
a particular print job.
[1146] The printing itself could be performed using on a variety of
types of Physical Imaging Device 246. Depending on the Print
Presentation Requirements of the User, the most appropriate machine
or machines to be used may vary by such factors as the size of the
overall print job and the type of printing (e.g., color or
black-and-white) desired. Possible imaging devices are high-speed
laser printers such as the Xerox.RTM. DocuTech.RTM. series.
[1147] 5.C. Electronic Displayer
[1148] Current methods of test reporting involve the production of
print reports that contain certain information about Students' Test
Results. In some instances, companies that report on tests provide
teachers with basic information on CD-ROM. However, current methods
do not permit teachers, parents, and others to use the Internet to
view electronic displays of rich test result information (including
detailed Student and Group performance on specific Skills) and
information about suggested instructional responses for responding
to Student needs.
[1149] What is needed, therefore, is a production system for the
print and Internet display of information about Test Results and
suggested instructional responses. In order to eliminate the
potential for consistency errors across the two media, this
production system advantageously may extract data files from a
common data storage system. In addition, because the printed
documents that display information are often the single method
which a school system or test reporting company uses to communicate
directly with the Users, it may be advantageous that the printed
documents contain the password information that enable the Users to
access the website securely
[1150] As described here, the Electronic Displayer 152 generates
electronic displays that are incorporated into a website, enabling
Internet access in the manner described above. However, as also
discussed, the electronic displays that are produced may also be
transmitted to Users using other electronic means.
[1151] To reiterate the process flow, once the Results Report
Generator 146 and the Instructional Response Report Generator 148
have completed processing, the Display Operator is notified by the
Report Generator Operator and then activates the Print Displayer
150 in step 1626 and the Electronic Displayer 152 in step 1642.
[1152] In step 1642, the Electronic Displayer retrieves the
Electronic Presentation Requirements from the Information
Repository.
[1153] Electronic Presentation Requirements are electronic text
files and prototypes that together specify the types of electronic
displays that a particular type of User may receive. For example,
the Electronic Presentation Requirements may specify that an
Educator will see electronic displays as part of a website in which
specific views are shown, with particular navigation elements to
allow the Educator to go from one view to the next.
[1154] The Electronic Presentation Requirements include all
necessary specifications to present Test Results and Instructional
Response Materials by means of electronic displays for Users. These
specifications, for example, may include the colors and styling to
be used on the electronic displays, the types of data presentation
methods (such as tables, pie charts, bar charts, or scatter plots)
employed, and the display interface used to show instructional
materials.
[1155] Based on these Electronic Presentation Requirements, the
Electronic Displayer 152 generates the corresponding Electronic
Templates in step 1646 and stores them in the Report Repository 118
in step 1648. The generation of electronic templates can be carried
out by the Operator of the Electronic Displayer 150 using a
templating language familiar to those skilled in the art, or in
some cases by a computer that receives as its input specific
Electronic Presentation Requirements.
[1156] Next, in step 1650, the Electronic Displayer 152 extracts
User Electronic Display Rules from the Information Repository 114.
In step 1652, the Electronic Displayer 152 extracts the Electronic
Templates, Results Reports, and Instructional Response Reports from
the Report Repository 118, and, using the User Electronic Display
Rules, merges the templates with the reports to create electronic
displays.
[1157] A complete example elucidates this process of creating
electronic displays:
[1158] Assume that the Data Requirements for Results Reports
specify that one report to be shown to Users (in this case
Educators) is a display of "Performance in a Skill across Students"
that visually depicts the performance of every Student in several
4th grade classes with respect to the specific Skill of
subtraction, and that certain additional information (such as
teacher name) also may be provided to Users in these displays.
[1159] Further assume that the Student Skill Performance Evaluator
134 has stored the appropriate Evaluation assignments in the
Information Repository 114, so that each Student has a Verbal
Evaluation Statement in the Skill of subtraction based on the
Student's Score in that Skill. In this case, the potential Verbal
Evaluation Statements for that Skill are "Likely to Need Help with
Fundamentals", "Likely to Need Additional Instruction and
Practice", or "Likely to Need Advanced Work".
[1160] Further assume that the Electronic Presentation Requirements
specify that the format for this display is a single table with
three columns that correspond to, and are labeled by, each of the
three Verbal Evaluation Statements, and that each Student in the
class should be listed alphabetically within the column labeled by
the Verbal Evaluation Statement corresponding to that Student's
performance in that Skill. Finally assume that the Electronic
Presentation Requirements also specify the style and the
explanatory text to be included within that particular display.
[1161] In this example, the figures demonstrate the simple
progression from (1) a Data Requirement for Results Reports 1734,
to (2) the corresponding Electronic Document Structure 1736, to (3)
the corresponding Results Report 1802, and to (4) the corresponding
electronic display of that report 2004.
[1162] This example also illustrates two important issues
concerning display formats: First, effective displays need not
include numerical information, because simple verbal evaluation
statements often are compelling and descriptive, particularly for
the majority of Users who are unfamiliar with the interpretation of
numerical data. Second, a single display typically relies on
multiple field-sorting principles. In this example, the Verbal
Evaluation Statements are arranged from left to right in a specific
order, and the Students are ordered from top to bottom, in
ascending alphabetical order by last name.
[1163] Next, in step 1654, the Electronic Displayer shows
electronic displays to Users. This can be done by means of a
variety of methods. In one embodiment, the displays could be sent
to users via e-mail attachments, or they could be saved onto
CD-ROMs and distributed.
[1164] According to one embodiment, the electronic displays to be
shown to any particular User may be accessible as part of a single
application, allowing the User to navigate among different
views.
[1165] Potential methods for doing so would involve an application
on CD-ROM using standard CD-ROM authoring tools, or an application
that can run on an internal network. The method described here,
however, is to create a web-based application that enables Users to
access the application by logging onto the User Computer 248 from
anywhere with an Internet connection.
[1166] In one embodiment, the User may then enter appropriate
log-in information, such as a personalized Login ID and Password.
This involves an authentication process, as shown in FIG. 21. In
step 2102, the User makes the request to access a certain set of
electronic displays. This request may take the form, for example,
of an attempted log-in on a website. Next in step 2104, the
Electronic Displayer retrieves User permission information from the
Information Repository 114, and in step 2106 the Electronic
Displayer evaluates whether the User has permission to access this
particular set of electronic displays. If so, then the Electronic
Displayer enables the User to access the electronic displays 2108,
but if not, then the Electronic Displayer informs the User that the
User cannot access the electronic displays.
Embodiment
[1167] In one potential embodiment of the website, the web
application that may be approved by the Client would function as
follows:
[1168] FIG. 22 illustrates the flowchart of a website used to
present electronic displays to the Users. First, the User enters
the website via an authentication protocol in step 2202 (discussed
more fully in FIG. 21), and sees a Welcome Screen in step 2204. In
step 2206, the Welcome Screen prompts the User to select a
particular class (say a 4th grade class called class 401) and a
particular test event (say, 3rd grade math exam). An example of
this display 2302 is shown in FIG. 23
[1169] After making a selection, the User is shown the Main Menu
page for that class and test event in step 2208, from which point
the User in step 2210 can select one of three types of electronic
displays. (An example of this display 2402 is shown in FIG.
24.)
[1170] One display that can be accessed from the Main Menu page is
a list of Students showing Students' overall performance in step
2214. (An example of this display 2502 is shown in FIG. 25.)
[1171] From this display, the User can opt in step 2216 to see
another list of Students (for another class of the same
teacher-User, for instance), or can click on a hyperlinked Student
name to access a Skill Profile for that Student in step 2218. (An
example of this display is shown in FIG. 25. From this display, the
User can opt to see another list of Students by using the dropdown
menu 2504, or can click on a hyperlinked Student name such as shown
by 2506 to access a Skill Profile for that Student.)
[1172] From the view of a Skill Profile for a Student in Step 2218,
the User is prompted for next actions in step 2220. From these
prompts, the User can opt to see a Skill Profile for another
Student, or the User can click on a Skill name and be taken to a
view of Performance in that Skill across Students 2226. (An example
of this display 2602 is shown in FIG. 26. From this display, the
User can opt to see a Skill Profile for another Student by using
the dropdown menu 2604, or the User can click on a hyperlinked
Skill name such as shown by 2606 to view the Performance in that
Skill across Students.)
[1173] A second display that can be accessed from the Main Menu
page 2208 is a display of a Skill Profile of the Group as a whole
in step 2222. This display could contain the Skill-by-Skill scores
for the class as a whole, as well as recommendation statements by
Skill, for how the teacher should proceed instructionally with the
class.
[1174] From this view, the User is prompted for next actions in
step 2224. The User can opt to see the Skill Profile of another
Group, or the User can click on a Skill name and be taken to a view
of Performance in that Skill across Students 2226. (An example of
this display 2702 is shown in FIG. 27. From this display, the User
can opt to see a Skill Profile for another Group by using the
dropdown menu 2704, or the User can click on a hyperlinked Skill
name such as shown by 2706 to view the Performance in that Skill
across Students.)
[1175] The display of Performance in a Skill across Students in
step 2226 shows the User a breakdown of all the Students in that
Skill. From there, the User can choose in step 2228 to see the same
display with respect to another Skill in step 2226, to see the
Skill Profile 2218 for a Student listed in the breakdown, or to see
Instructional Response Materials in a Skill at a particular level
in step 2234. (An example of this display 2802, called here the
"Flexible Groupings" page, is shown in FIG. 28. From this display,
the User can opt to see the same display with respect to another
Skill by using the dropdown menu 2808, or the User can click on a
hyperlinked Student name 2810 to see the Skill Profile for that
Student, or the User can click on an explicit instruction to see
instructional materials in that Skill 2806.)
[1176] Finally, a third display that can be accessed from the Main
Menu 2208 is a listing of all Instructional Response Materials
available in a particular subject 2230. From there, the User can
choose 2232 to view Instructional Response Materials in a
particular Skill at a particular Level 2234. (An example of a
display listing the various Instructional Response Materials 2902
is shown in FIG. 29.)
[1177] The Instructional Response Materials in a particular Skill
at a particular Level 2234 can contain a variety of different kinds
of instructional information, organized in various ways as
described above in the Instructional Reports Generator. From the
Instructional Response Materials in a particular Skill 2234, the
User can choose 2236 to print those materials 2238, see materials
for another Skill 2234, or see the display of Performance in that
Skill across Students 2226. (An example of this display 3002 is
shown in FIG. 30.)
[1178] According to this embodiment, the Instructional Response
Materials in a particular Skill at a particular Level 2234 may
advantageously be organized within the same Evaluation categories
that are used to describe Student's mastery of a given Skill
elsewhere on the website, such as in the display of Performance in
a Skill across Students 2226. (In FIG. 30, the "Activities" portion
of the materials are divided 3002 according to the categories of
Fundamentals, Practice, Advanced shown in the column headings 2804
of FIG. 28.)
[1179] At all times 2206, 2210, 2216, 2224, 2232, 2220, 2228, and
2236 when prompted, the User can return to the Main Menu 2208, can
see help and information screens, can change user authentication
information, and can log out. These functions are typically
available on many commercially available websites.
[1180] The foregoing description of one embodiment of the system
illustrates several important features:
[1181] The first important feature of the website described here is
that Users can navigate directly between different displays of
Tests Results, and directly between displays of Test Results and
Instructional Response Materials. In FIG. 22, these direct
navigational paths were as follows: [1182] 1. From the display of
the List of Students with Overall Performance for each Student
2214, the User can choose a Student name in step 2216 and see the
display of the Skill Profile for that Student in step 2218. [1183]
2. From the display of the Performance in a Skill across Students,
the User can choose a Student name in step 2228 and see the display
of the Skill Profile for that Student in step 2218. [1184] 3. From
the display of the Skill Profile of the Group in step 2222 the User
can choose a Skill name and see the display of the Performance in
that Skill across Students in step 2226. [1185] 4. Similarly, from
the display of the Skill Profile of a Student in step 2224 the User
can choose a Skill name and see the display of the Performance in
that Skill across Students in step 2226. [1186] 5. The User can
navigate directly between the displays of the Performance in a
Skill across Students in step 2226 and the display of Instructional
Response Materials in that Skill in step 2234.
[1187] In one embodiment, the direct navigation occurs through the
links of "hyperlinked" text. For instance, when a User sees the
Skill Profile of a Group as shown in FIG. 27, 2702, the User can
click on a Skill name 2704 to be taken directly to a display of the
Skill Performance in that Skill across Students, as shown in FIG.
28, 2802. However, those skilled in the art will understand that
this type of navigation can also be operationalized by means other
than hyperlinked text.
[1188] The second important feature of the website is that when
Users are viewing any particular display, they can directly access
the same type of display loaded with different content. For
example, consider a User who teaches more than one class (or
teaches one class in more than one subject). If that User is
viewing the Skill Profile of a Group 2702 as shown in FIG. 27, then
the User can simply choose to see another the Skill Profile of
another Group by clicking on the dropdown menu 2706 and selecting a
class and subject. Similarly, when viewing the display of the
Instructional Response Materials for a Skill 3002 as shown in FIG.
30, the User may use the dropdown menu 3004 to access Instructional
Response Materials in other Skills.
[1189] Dropdown menus are one way to enable direct navigation to
the same type of display loaded with different content, and can
function with respect to both data and instructional displays.
However, those skilled in the art can also implement this feature
by other technical means.
[1190] These two features of this embodiment of the website are
shown in one particular way in the embodiment illustrated in FIG.
22. However, it may be noted that this approach has broader
application. As shown in FIG. 31, a website with several types of
displays can permit a wide variety of direct connections between
different displays. (FIG. 20 shows how a few potential hyperlinks
(2006, 2008, 2010) and dropdown menus (2012, 2014, 2016) can be
implemented on any single screen.) Regardless of the precise
navigational elements of any screen, the critical feature is that
the User not have to return to a central menu between viewing
different displays of Tests Results or Instructional Materials.
[1191] Importantly, these navigational tools may be used with
respect to the presentation of Group information as shown in FIG.
32, which is a diagram of a potential display of Performance in a
Skill across Groups 3202. This display enables a principal to see
how different classes in a given grade performed in a given skill
and contains several key navigational features: [1192] 1. The
column-headings of Evaluation Statements are hyperlinked 3204 to
Skill-Specific Instructional Response Materials at an
identically-named, corresponding difficulty level, so that the
principal could learn how the teachers in the school could best
approach the teaching of that Skill with Students of different
proficiency levels. [1193] 2. The class numbers are hyperlinked
3206 to the respective displays of the Skill Profiles for that
class. [1194] 3. A link at the bottom of the page 3208 takes the
principal to Skill-Specific Instructional Response Materials in
that Skill. [1195] 4. Dropdown menus enable the principal to see
the same display of "Performance in a Skill across Groups" for
additional Subjects and Levels 3210, Tests 3212, and Skills
3214.
[1196] A third important feature of this embodiment of the website
is that the Instructional Response Materials are integrated with
the Test Results themselves. As described above, Users directly
navigate between Skill names and Instructional Response Materials.
More important, the same terminology used to present Test Result
information by Skill corresponds to the categories used to present
instructional strategies by Skill.
[1197] Take, for example, the display of Performance in a Skill
across Students shown in FIG. 28, 2802. This display associates
three particular Evaluation Statements 2804 with each Student for
the Skill of "Multiplication/Division--Problem Solving". The three
Evaluation Statements utilized are "Likely to Need Help with
Fundamentals", "Likely to Need Additional Instruction and
Practice", and "Likely to Be Ready for Advanced Work".
[1198] If the User clicks on the hyperlink text for the "Teaching
Tools" 2806, the User then sees Instructional Response Materials
for the Skill of "Multiplication/Division--Problem Solving", shown
in FIG. 30, 3002. Here, the User can obtain instructional
information in that Skill.
[1199] Importantly, the suggested activities in that Skill are
organized 3004 according to difficulty level using precisely the
same Evaluation Statements that were used in the display of
Performance in a Skill across Students 2802: Help with
Fundamentals. This correspondence is important because it enables
the User (and Educator in this case) to provide Students with
different activities in a Skill, based on each Student's particular
needs.
[1200] This correspondence is achieved because the content of the
Instructional Response Materials are based on the "Content
Requirements for Instructional Response Reports", which are
approved by the Client as described in the Intake Module 102. These
Content Requirements may specify the circumstances in which
Instructional Response Materials should be disaggregated into
difficulty levels, rather than at a single difficulty level. For
example, referring to the XML document 1802 shown in FIG. 18 (which
was generated from the Content Requirements and served as the basis
for the Instructional Response Materials 3002 shown in FIG. 30), it
can be seen that the Content Requirements in that case specified
that the "activities" portion of the Instructional Response
Materials should be disaggregated into three difficulty levels
1804, 1806, 1808.
[1201] In many cases, Users when presented with instructional
material organized by difficulty level can more accurately respond
to Student needs. Of course, the particular displays shown here are
but one method of integrating Test Results with Instructional
Response Materials, and those skilled in the art doubtless could
make subtle modifications to the approach specified here.
[1202] For example, those skilled in the art can program the web
application to enable the User to navigate directly between a
display showing the Skill Profile for a Student 2218 and a display
showing an entire packet of Instructional Response Materials, such
that those materials are customized for that particular Student and
appropriate to the Student's mastery level in each particular
Skill. (According to the terminology used here, this functionality
would be appropriate in the event that the User Electronic Display
Rules, discussed above with respect to the Intake Module 102,
specify that a User see an electronic display of a complete packet
of Instructional Response Materials for each Student whose Test
Results are displayed on the website.) Other such minor changes in
navigation are possible as well.
Another Embodiment
[1203] In another potential embodiment, FIG. 33 illustrates a
website that advantageously provides Users with additional tools to
examine Students' progress in various Skills and to keep track of
students' mastery levels over time.
[1204] In many environments (both in K-12 education and other
educational settings), standardized tests are given to Students
only at periodic intervals, say once every year or once every few
months. In the meantime, Educators have no organized way in which
to track their Students' progress in particular Skills over time.
Although some companies offer online testing programs so that
Students can take online exams weekly or even daily, these options
are not practical for many Educators because educational
institutions (such as K-12 schools) rarely have sufficient
computing resources for the Students.
[1205] It would be possible for Educators to adapt materials (such
as commercially available workbooks or personally produced problem
sets) to generate assessments for use in tracking Student progress
in specific Skills. However, this is unrealistic for many
Educators, who typically have broad responsibilities and little
extra time to perform the work of adapting materials.
[1206] What is needed, therefore, is a system and method that
provide Educators with an efficient manner of using
non-computer-intensive means to track Students' progress in
different Skills, particularly a method that maintains a common
language between the standardized tests themselves and the ongoing
assessment process.
[1207] FIG. 33 illustrates the flowchart of a website that provides
Educators with this functionality. The steps 3332, 3334, 3336,
3338, 3340, 3342, 3344, and 3322 are the processes that have been
added incrementally to the flowchart illustrated in FIG. 22.
[1208] The User navigates through the website shown in FIG. 33 in
generally the same way as the User would navigate through the
website shown in FIG. 22. However, when the User is viewing the
Performance in a Skill across Students in step 3328, the User has
available an important additional option in selection 3330: The
User can view (step 3336) and then choose (3338) to print (step
3340) an Ongoing Assessment in that Skill. In step 3342, the User
administers and evaluates the Ongoing Assessment printed in
3340.
[1209] FIG. 36 is a diagram of a "Performance in a Skill across
Students" display for the Skill of Character, here known as a
"Flexible Groupings" view. This diagram shows the hyperlink 3604
entitled "Ongoing Assessments in this topic". When pressed, this
hyperlink enables the User to view the display 3336 of an Ongoing
Assessment in the Skill of Character.
[1210] FIG. 38 illustrates a sample portion of an Ongoing
Assessment in the Skill of Character 3802. (The Skill of analyzing
character being a typical reading comprehension skill examined in
grades K-12.) As shown, the Ongoing Assessment includes four
questions that can be copied and given to students 3804, as well as
a scoring guide 3806 that can be used by teachers to evaluate the
results.
[1211] An Ongoing Assessment is one of the Instructional Response
Materials generated by the Electronic Displayer 152. In order for
Ongoing Assessments to be produced, the Content Manager 140
advantageously may ensure that the Content Repository 116 includes
all of the appropriate components for Ongoing Assessments of each
Skill: namely, Items, scoring guides for Items, total score metric,
and teacher reference materials. The list of required components
advantageously may be set forth as part of the Client-approved
specifications about the instructional materials in the Intake
Module 102 stage, which are known as "Content Requirements for
Instructional Reports".
[1212] The Content Requirements for Instructional Response Reports
also advantageously may specify that the scoring guides produce
Evaluation Statements for Students are the same Evaluation
Statement categories used to describe Students' Test Results. This
last specification enables teachers to use Ongoing Assessment
results in conjunction with the Test Results themselves; when the
standardized test results and the Ongoing Assessment results are
both expressed in terms of the same Evaluation Statements, teachers
can easily track their Students' progress over time, even across
multiple standardized tests and Ongoing Assessments.
[1213] FIG. 36 and FIG. 38 illustrate this point. In FIG. 36, each
Student is categorized within one of three categories based on the
Student's performance on the standardized test (here known as the
ISAT 2002): These three categories are "Likely to Need Help with
Fundamentals", "Likely to Require Additional Instruction and
Practice", and "Likely to Be Ready for Advanced Work". These three
categories are also employed in the scoring of the Ongoing
Assessment 3802 in the following way: As shown in 3808, a teacher
who uses this Ongoing Assessment is told that score of 0-1 points
puts a Student in the "Fundamentals" column, 2-4 points puts a
Student in the "Practice" column, and 5-6 points puts a Student in
the "Advanced Work" column.
[1214] The User can then input the results of the Ongoing
Assessment via the input page 3702 shown in FIG. 37 into the three
categories of mastery described above 3704. As a result of this
input the Flexible Groupings screen 3328 may now display the
revised information about each Student's mastery level in the
Skill, including if desired each Student's performance history in
the Skill.
[1215] Users also can obtain Ongoing Assessments from the Main Menu
as follows: From viewing the Main Menu in step 3308, the User can
navigate in step 3310 to a listing of all the Ongoing Assessments
by Skill in a given Subject in step 3322. Examples of a Main Menu
screen and the Ongoing Assessments by Skill in a given Subject
screen are shown in FIG. 34 and FIG. 35, as elements 3402 and 3502
respectively. From this view, the User can select in step 3334 to
see an Ongoing Assessment in a particular Skill in step 3336. For
example, if the User clicks on the word "Character" as seen in
element 3504, then the User can, in step 3336, see an Ongoing
Assessment in the Skill of analyzing character.
[1216] An important feature of the Ongoing Assessment system
described in this embodiment is its ease of use. Students do not
have to gain access to a computer to take the assessments (much
less an Internet-connected computer). Teachers do not need
late-version browser software to update and track their students'
progress, thanks to the simplicity of the HTML form interface. And
finally, teachers do not have to devote substantial time to
administering or scoring the brief assessments, thanks to their
three-tiered scoring system.
[1217] Those skilled in the art will further recognize that the
embodiment described above can be supplemented with other types of
particular functionality, depending upon the requirements of the
Client:
[1218] In another embodiment, a Client might specify that Users be
able to see certain Student information in addition to Test Result
information, such as attendance data, course grades, homework
assignments, etc. It will be evident to those skilled in the art
that the core functionality of the website described above can
easily be complemented with additional electronic data displays
concerning these types of data, according to methods familiar to
those skilled in the art of web design. For instance, it would be
possible to show a Student's attendance and grade records on the
same view as the view of the Skill Profile for a Student in step
2218, or in a display that is directly hyperlinked to that
view.
[1219] Physically, the Electronic Displayer 152 may function, in
one embodiment, as follows:
[1220] The Web Application Server 248 retrieves the Web
Presentation Requirements from the Information Repository 114,
which includes visual prototypes and text description. Based on
these Requirements, the Electronic Display Operator operates the
Report Publishing Computer 236, and creates Electronic Templates
using languages familiar to those skilled in the art, such as Java
Server Pages and XSL. The Electronic Report Publishing Computer 236
then stores these Web Templates as files on the Report Server
240.
[1221] The Web Application Server 248 uses common database methods
and file-system operations to retrieve User Electronic Display
Rules from the Information Repository 1634, as well as Print
Templates, Results Reports, and Instructional Response Reports from
the Report Repository 118. The Web Application Server 248 merges
the Electronic Template with the reports, creating resulting
documents in Hypertext Markup Language (HTML) format that include
Test Results and Instructional Response Materials. These documents
in HTML form can be transmitted over the Internet for presentation
with commonly available web browsers.
[1222] The Web Application Servers 246 makes the reports on the
Report Server 240 available to authorized users via the World Wide
Web. To convert and serve the files in a human-friendly HTML
format, a Web Application Server 246 may be placed between the
Internet and the Report Server 240. The Firewall 202 may need to be
configured to allow Internet access to the Web Application Servers
246 over the Hypertext Transfer Protocol (HTTP). A possible
configuration of the Web Application Servers 246 may be a
workstation-class UNIX server similar to the one used as the Print
Converter Computer 244. One or more Web Application Servers 246 may
be installed as a cluster for increased performance and fail-over
in case of hardware or software failure. Report-specific code adds
visual embellishments and hypertext navigation structure using a
web template language such as JavaServer Pages.TM. (JSP). A
JSP-compatible engine such as Apache Tomcat may be used on the Web
Application Server 248.
[1223] Also, the Web Application Server 248 manages the User
authentication process, ensuring that only authorized Users have
access to the various electronic displays. The Web Application
Server 248 accomplishes this by making queries to the Data Servers
214 using a protocol such as Lightweight Directory Access Protocol
(LDAP) or Structured Query Language (SQL) to retrieve data stored
about a User who is attempting to log on. The retrieved information
consists of a unique Log In ID and password for a User, as well as
the User Electronic Display Rules that define what a particular
User is able to view.
[1224] Once the processing of the Electronic Displayer 152 is
terminated, the Display Operator is notified. Once the Print
Displayer 150 and the Electronic Displayer 152 have finished
processing, the System Operator is notified and terminates the
entire system.
CONCLUSION OF THE DESCRIPTION OF THE INVENTION
[1225] The foregoing is illustrative of the present invention and
is not to be construed as limiting thereof. Although a few
exemplary embodiments of this invention have been described, those
skilled in the art will readily appreciate that many modifications
are possible in the exemplary embodiments without materially
departing from the novel teachings and advantages of this
invention. Accordingly, all such modifications are intended to be
included within the scope of this invention as defined in the
claims. In the claims, means-plus-function clauses are intended to
cover the structures described herein as performing the recited
function and not only structural equivalents but also equivalent
structures. Therefore, it is to be understood that the foregoing is
illustrative of the present invention and is not to be construed as
limited to the specific embodiments disclosed, and that
modifications to the disclosed embodiments, as well as other
embodiments, are intended to be included within the scope of the
appended claims. The invention is defined by the following claims,
with equivalents of the claims to be included therein.
* * * * *