U.S. patent application number 10/832003 was filed with the patent office on 2004-11-11 for fusion classification for risk categorization in underwriting a financial risk instrument.
This patent application is currently assigned to General Electric Company. Invention is credited to Aggour, Kareem Sherif, Bonissone, Piero Patrone, Iyer, Naresh Sundaram, Messmer, Richard Paul, Subbu, Rajesh Venkat, Yan, Weizhong.
Application Number | 20040225587 10/832003 |
Document ID | / |
Family ID | 33309734 |
Filed Date | 2004-11-11 |
United States Patent
Application |
20040225587 |
Kind Code |
A1 |
Messmer, Richard Paul ; et
al. |
November 11, 2004 |
Fusion classification for risk categorization in underwriting a
financial risk instrument
Abstract
A system, process and computer program product for underwriting
a financial risk instrument application represented by at least one
risk attribute is provided. Decision engines examine the at least
one risk attribute associated with the financial risk instrument
application and assign the application to one of a predetermined
set of risk classes. A fusion engine compares the risk classes
assigned by each of the decision engines and fuses the assigned
risk classes into an aggregated result representative of the risk
of the financial risk instrument application. The fusion engine
includes a first multi-classifier fusion module that uses an
associative function to fuse the assigned risk classes into a first
aggregated result and a second multi-classifier fusion that uses a
non-associative function to fuse the assigned risk classes into a
second aggregated result. A comparison engine selects one of the
first aggregated result generated from the first multi-classifier
fusion module and the second aggregated result generated from the
second multi-classifier fusion module and compares it with a
production result generated from the production decision engine.
The comparison engine generates an underwriting decision for the
financial risk instrument application according to the
comparison.
Inventors: |
Messmer, Richard Paul;
(Rexford, NY) ; Bonissone, Piero Patrone;
(Schenectady, NY) ; Aggour, Kareem Sherif;
(Schenectady, NY) ; Subbu, Rajesh Venkat; (Clifton
Park, NY) ; Yan, Weizhong; (Clifton Park, NY)
; Iyer, Naresh Sundaram; (Clifton Park, NY) |
Correspondence
Address: |
General Electric Company
CRD Patent Docket Rm 4A59
Bldg. K-1
P.O. Box 8
Schenectady
NY
12301
US
|
Assignee: |
General Electric Company
|
Family ID: |
33309734 |
Appl. No.: |
10/832003 |
Filed: |
April 23, 2004 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10832003 |
Apr 23, 2004 |
|
|
|
10425721 |
Apr 30, 2003 |
|
|
|
Current U.S.
Class: |
705/35 |
Current CPC
Class: |
G06Q 40/00 20130101;
G06Q 40/08 20130101 |
Class at
Publication: |
705/035 |
International
Class: |
G06F 017/60 |
Claims
1. A system for underwriting a financial risk instrument
application represented by at least one risk attribute, comprising:
a plurality of decision engines that each examines the at least one
risk attribute associated with the financial risk instrument
application and assigns the application to one of a predetermined
set of risk classes; a fusion engine that compares the risk classes
assigned by each of the plurality of decision engines and fuses the
assigned risk classes into an aggregated result representative of
the risk of the financial risk instrument application, wherein the
fusion engine comprises a first multi-classifier fusion module that
uses an associative function to fuse the assigned risk classes into
a first aggregated result and a second multi-classifier fusion that
uses a non-associative function to fuse the assigned risk classes
into a second aggregated result; a production decision engine that
assigns the financial risk instrument application to one of a
predetermined set of risk classes according to the at least one
risk attribute associated with the application and generates a
production result representative of the risk of the application;
and a comparison engine that selects one of the first aggregated
result generated from the first multi-classifier fusion module and
the second aggregated result generated from the second
multi-classifier fusion module and compares with the production
result generated from the production decision engine, the
comparison engine generating an underwriting decision for the
financial risk instrument application according to the
comparison.
2. The system according to claim 1, further comprising a fusion
confidence estimation engine that estimates a degree of confidence
in the aggregated result of the fusion engine, wherein the fusion
confidence estimation engine estimates a degree of confidence in
the first aggregated result generated from the first
multi-classifier fusion module and a degree of confidence in the
second aggregated result generated from the second multi-classifier
fusion module.
3. The system according to claim 2, wherein the comparison engine
uses the estimated degree of confidences to select between the
first aggregated result generated from the first multi-classifier
fusion module and the second aggregated result generated from the
second multi-classifier fusion module.
4. The system according to claim 1, further comprising a production
confidence estimation engine that estimates a degree of confidence
in the production result generated from the production decision
engine.
5. The system according to claim 1, further comprising a comparison
confidence estimation engine that estimates a degree of confidence
in the underwriting decision made for the financial risk instrument
application by the comparison engine.
6. The system according to claim 1, wherein the associative
function used by the first multi-classifier fusion module comprises
triangular norm operators.
7. The system according to claim 1, wherein the non-associative
function used by the second multi-classifier fusion module
comprises convex combination and averaging operators.
8. A system for underwriting a financial risk instrument
application represented by at least one risk attribute, comprising:
a plurality of decision engines that each examines the at least one
risk attribute associated with the financial risk instrument
application and assigns a preference from a set of predetermined
risk classes for the application, wherein the preference of risk
classes provides a conviction in applicability of each risk class
assigned to the financial risk instrument application; a fusion
engine that compares the preferences of risk classes generated by
each of the plurality of decision engines and fuses the preferences
of risk classes into an aggregated result representative of the
risk of the financial risk instrument application, wherein the
fusion engine comprises a first multi-classifier fusion module that
uses an associative function to fuse the preferences of risk
classes into a first aggregated result and a second
multi-classifier fusion that uses a non-associative function to
fuse the preferences of risk classes into a second aggregated
result; a production decision engine that assigns the financial
risk instrument application to a preference from a set of
predetermined risk classes according to the at least one risk
attribute associated with the application and generates a
production result representative of the risk of the application;
and a comparison engine that selects one of the first aggregated
result generated from the first multi-classifier fusion module and
the second aggregated result generated from the second
multi-classifier fusion module and compares with the production
result generated from the production decision engine, the
comparison engine generating an underwriting decision for the
financial risk instrument application according to the
comparison.
9. The system according to claim 8, further comprising a fusion
confidence estimation engine that estimates a degree of confidence
in the aggregated result of the fusion engine, wherein the fusion
confidence estimation engine estimates a degree of confidence in
the first aggregated result generated from the first
multi-classifier fusion module and a degree of confidence in the
second aggregated result generated from the second multi-classifier
fusion module.
10. The system according to claim 9, wherein the comparison engine
uses the estimated degree of confidences to select between the
first aggregated result generated from the first multi-classifier
fusion module and the second aggregated result generated from the
second multi-classifier fusion module.
11. The system according to claim 8, further comprising a
production confidence estimation engine that estimates a degree of
confidence in the production result generated from the production
decision engine.
12. The system according to claim 8, further comprising a
comparison confidence estimation engine that estimates a degree of
confidence in the underwriting decision made for the financial risk
instrument application made by the comparison engine.
13. A computer-implemented process for underwriting a financial
risk instrument application represented by at least one risk
attribute, comprising: examining the at least one risk attribute
associated with the financial risk instrument application with a
plurality of decision engines; using each decision engine to assign
the application to one of a set of predetermined risk classes;
fusing the assigned risk classes into an aggregated result
representative of the risk of the financial risk instrument
application, wherein the fusing comprises: applying an associative
function to fuse the assigned risk classes into a first aggregated
result; and applying a non-associative function to fuse the
assigned risk classes into a second aggregated result; selecting
one of the first aggregated result and the second aggregated
result; comparing the selected aggregated result with a production
result generated from a production decision engine, and generating
an underwriting decision for the financial risk instrument
application according to the comparison of the selected aggregated
result with the production result.
14. The process according to claim 13, further comprising
estimating a degree of confidence in the first aggregated result
and in the second aggregated result.
15. The process according to claim 14, wherein the selecting
comprises comparing the estimated degree of confidences associated
with the first aggregated result and the second aggregated
result.
16. The process according to claim 13, further comprising
estimating a degree of confidence in the underwriting decision made
for the financial risk instrument application.
17. The process according to claim 13, wherein the associative
function comprises triangular norm operators.
18. The process according to claim 13, wherein the non-associative
function comprises convex combination and averaging operators.
19. A computer-implemented process for underwriting a financial
risk instrument application represented by at least one risk
attribute, comprising: examining the at least one risk attribute
associated with the financial risk instrument application with a
plurality of decision engines; using each decision engine to assign
a preference from a set of predetermined risk classes for the
application, wherein the preference of risk classes provides a
conviction in applicability of each risk class assigned to the
financial risk instrument application; fusing the preferences of
risk classes into an aggregated result representative of the risk
of the financial risk instrument application, wherein the fusing
comprises: applying an associative function to fuse the preferences
of risk classes into a first aggregated result; and applying a
non-associative function to fuse the preferences of risk classes
into a second aggregated result; selecting one of the first
aggregated result and the second aggregated result; comparing the
selected aggregated result with a production result generated from
a production decision engine, and generating an underwriting
decision for the financial risk instrument application according to
the comparison of the selected aggregated result with the
production result.
20. The process according to claim 19, further comprising
estimating a degree of confidence in the first aggregated result
and in the second aggregated result.
21. The process according to claim 20, wherein the selecting
comprises comparing the estimated degree of confidences associated
with the first aggregated result and the second aggregated
result.
22. The process according to claim 19, further comprising
estimating a degree of confidence in the underwriting decision made
for the financial risk instrument application.
23. A computer-readable medium storing computer instructions for
instructing a computer system to underwrite a financial risk
instrument application represented by at least one risk attribute,
the instructions comprising: examining the at least one risk
attribute associated with the financial risk instrument application
with a plurality of decision engines; using each decision engine to
assign the application to a one of a set of predetermined risk
classes; fusing the assigned risk classes into an aggregated result
representative of the risk of the financial risk instrument
application, wherein the fusing comprises: applying an associative
function to fuse the assigned risk classes into a first aggregated
result; and applying a non-associative function to fuse the
assigned risk classes into a second aggregated result; selecting
one of the first aggregated result and the second aggregated
result; comparing the selected aggregated result with a production
result generated from a production decision engine, and generating
an underwriting decision for the financial risk instrument
application according to the comparison of the selected aggregated
result with the production result.
24. The computer-readable medium according to claim 23 further
comprising instructions for estimating a degree of confidence in
the first aggregated result and in the second aggregated
result.
25. The computer-readable medium according to claim 24, wherein the
selecting comprises instructions for comparing the estimated degree
of confidences associated with the first aggregated result and the
second aggregated result.
26. The computer-readable medium according to claim 23, further
comprising instructions for estimating a degree of confidence in
the underwriting decision made for the financial risk instrument
application.
27. The computer-readable medium according to claim 23, wherein the
associative function comprises triangular norm operators.
28. The computer-readable medium according to claim 23, wherein the
non-associative function comprises convex combination and averaging
operators.
29. A computer-readable medium storing computer instructions for
instructing a computer system to underwrite a financial risk
instrument application represented by at least one risk attribute,
the instructions comprising: examining the at least one risk
attribute associated with the financial risk instrument application
with a plurality of decision engines; using each decision engine to
assign a preference from a set of predetermined risk classes for
the application, wherein the preference of risk classes provides a
conviction in applicability of each risk class assigned to the
financial risk instrument application; fusing the preferences of
risk classes into an aggregated result representative of the risk
of the financial risk instrument application, wherein the fusing
comprises: applying an associative function to fuse the preferences
of risk classes into a first aggregated result; and applying a
non-associative function to fuse the preferences of risk classes
into a second aggregated result; selecting one of the first
aggregated result and the second aggregated result; comparing the
selected aggregated result with a production result generated from
a production decision engine, and generating an underwriting
decision for the financial risk instrument application according to
the comparison of the selected aggregated result with the
production result.
30. The computer-readable medium according to claim 29, further
comprising instructions for estimating a degree of confidence in
the first aggregated result and in the second aggregated
result.
31. The computer-readable medium according to claim 30, wherein the
selecting comprises instructions for comparing the estimated degree
of confidences associated with the first aggregated result and the
second aggregated result.
32. The computer-readable medium according to claim 29, further
comprising instructions for estimating a degree of confidence in
the underwriting decision made for the financial risk instrument
application.
33. A system for underwriting a financial risk instrument
application represented by at least one risk attribute, comprising:
a plurality of decision engines that each examines the at least one
risk attribute associated with the financial risk instrument
application and assigns the application to one of a predetermined
set of risk classes; a fusion engine that compares the risk classes
assigned by each of the plurality of decision engines and fuses the
assigned risk classes into an aggregated result representative of
the risk of the financial risk instrument application, wherein the
fusion engine comprises a first multi-classifier fusion module that
uses an associative function to fuse the assigned risk classes into
a first aggregated result and a second multi-classifier fusion that
uses a non-associative function to fuse the assigned risk classes
into a second aggregated result; and a comparison engine that
selects between the first aggregated result generated from the
first multi-classifier fusion module and the second aggregated
result generated from the second multi-classifier fusion module,
the selected result being representative of an underwriting
decision for the financial risk instrument application.
34. A computer-implemented process for underwriting a financial
risk instrument application represented by at least one risk
attribute, comprising: examining the at least one risk attribute
associated with the financial risk instrument application with a
plurality of decision engines; using each decision engine to assign
the application to one of a set of predetermined risk classes;
fusing the assigned risk classes into an aggregated result
representative of the risk of the financial risk instrument
application, wherein the fusing comprises: applying an associative
function to fuse the assigned risk classes into a first aggregated
result; and applying a non-associative function to fuse the
assigned risk classes into a second aggregated result; selecting
between the first aggregated result and the second aggregated
result; and generating an underwriting decision for the financial
risk instrument application according to the selected aggregated
result.
35. A computer-readable medium storing computer instructions for
instructing a computer system to underwrite a financial risk
instrument application represented by at least one risk attribute,
the instructions comprising: examining the at least one risk
attribute associated with the financial risk instrument application
with a plurality of decision engines; using each decision engine to
assign the application to a one of a set of predetermined risk
classes; fusing the assigned risk classes into an aggregated result
representative of the risk of the financial risk instrument
application, wherein the fusing comprises: applying an associative
function to fuse the assigned risk classes into a first aggregated
result; and applying a non-associative function to fuse the
assigned risk classes into a second aggregated result; selecting
one of the first aggregated result and the second aggregated
result; and generating an underwriting decision for the financial
risk instrument application according to the selected aggregated
result.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation-in-part of U.S. patent
application Ser. No. 10/425,721, entitled "System And Process For A
Fusion Classification For Insurance Underwriting Suitable For Use
By An Automated System", filed Apr. 30, 2003.
BACKGROUND OF THE INVENTION
[0002] The present invention relates to a system, process and
computer program product for risk categorization of financial risk
instrument applications, and more particularly to a system, process
and computer program product for providing expert assistance to the
underwriting of such financial risk instrument applications based
on a fusion classification.
[0003] Classification is the process of assigning an input pattern
to one of a predefined set of classes. Classification problems
exist in many real-world applications, such as medical diagnosis,
machine fault diagnosis, handwriting character recognition,
fingerprint recognition, and credit scoring, to name a few. Broadly
speaking, classification problems can be categorized into two
types: dichotomous classification, and polychotomous
classification. Dichotomous classification deals with two-class
classification problems, while polychotomous classification deals
with classification problems that have more than two classes.
[0004] Classification consists of developing a functional
relationship between the input features and the target classes.
Accurately estimating such a relationship is key to the success of
a classifier. Instrument underwriting of financial risk instruments
such as financial credit or loan applications is another area where
these classification problems exist. The underwriting process of a
financial risk instrument application may consists of assigning a
given instrument application, described by its financials, credit
rating, corporate structure, market and other key input, to one of
several risk categories (also referred to as risk or rate classes).
A trained individual or individuals traditionally perform financial
risk instrument underwriting. A given application for a financial
risk instrument (also referred to as an "instrument application")
may be compared against a variety of underwriting rules/standards
set by a financial company. Using the underwriting rules/standards
enables the instrument application to be classified into one of
several risk categories available for a type of coverage requested
by an applicant. The risk categories can affect the payment
structure (in terms of amount and timing) paid by the applicant,
e.g., the higher the risk category, the higher the overall payment.
A decision to accept or reject the application for the instrument
may also be part of this risk classification, as risks above a
certain tolerance level set by the financial company may simply be
rejected.
[0005] One problem associated with this approach in underwriting an
instrument application is that there are a large number of features
(financials, credit rating, corporate structure, market) and
rules/standards that the underwriters have to take into account, as
well as several risk categories (or rate classes). With the large
number of features, rules/standards and risk categories, it is very
difficult and time consuming to go over all of the information
necessary to make a decision and furthermore, the results are often
inadequate in consistency and reliability. The inadequacy of this
process becomes more apparent as the complexity of instrument
applications increases.
[0006] Another problem with this underwriting process is that the
underwriting standards typically do not cover all possible cases
and variations of an application for an instrument. The
underwriting standards may even be self-contradictory or ambiguous,
leading to an uncertain application of the standards. As a result,
the subjective judgment of the underwriter will likely play a role
in the process. Variation in factors such as underwriter training
and experience, and a multitude of other effects can cause
different underwriters to issue different, inconsistent decisions.
Sometimes these decisions can be in disagreement with the
established underwriting standards of the issuing company, while
sometimes they can fall into a "gray area" not explicitly covered
by the underwriting standards.
[0007] Further, there may be an occasion in which an underwriter's
decision could still be considered correct, even if it disagrees
with the written underwriting standards. This situation can be
caused when the underwriter uses his or her own experience to
determine whether the underwriting standards should be adjusted.
Different underwriters may make different determinations about when
these adjustments are allowed, as they might apply stricter or more
liberal interpretations of the underwriting standards. Thus, the
judgment of experienced underwriters may be in conflict with the
desire to consistently apply the underwriting standards.
SUMMARY OF THE INVENTION
[0008] In one embodiment of the invention, there is a system for
underwriting a financial risk instrument application represented by
at least one risk attribute. In this embodiment, there is a
plurality of decision engines that each examines the at least one
risk attribute associated with the financial risk instrument
application and assigns the application to one of a predetermined
set of risk classes. A fusion engine compares the risk classes
assigned by each of the plurality of decision engines and fuses the
assigned risk classes into an aggregated result representative of
the risk of the financial risk instrument application. The fusion
engine comprises a first multi-classifier fusion module that uses
an associative function to fuse the assigned risk classes into a
first aggregated result and a second multi-classifier fusion that
uses a non-associative function to fuse the assigned risk classes
into a second aggregated result. The system also comprises a
production decision engine that assigns the financial risk
instrument application to one of a predetermined set of risk
classes according to the at least one risk attribute associated
with the application and generates a production result
representative of the risk of the application. A comparison engine
selects one of the first aggregated result generated from the first
multi-classifier fusion module and the second aggregated result
generated from the second multi-classifier fusion module and
compares with the production result generated from the production
decision engine. The comparison engine generates an underwriting
decision for the financial risk instrument application according to
the comparison.
[0009] In a second embodiment, there is a system for underwriting a
financial risk instrument application represented by at least one
risk attribute. In this embodiment, there is a plurality of
decision engines that each examines the at least one risk attribute
associated with the financial risk instrument application and
assigns a preference from a set of predetermined risk classes for
the application. The preference of risk classes provides a
conviction in applicability of each risk class assigned to the
financial risk instrument application. A fusion engine compares the
preferences of risk classes generated by each of the plurality of
decision engines and fuses the preferences of risk classes into an
aggregated result representative of the risk of the financial risk
instrument application. The fusion engine comprises a first
multi-classifier fusion module that uses an associative function to
fuse the preferences of risk classes into a first aggregated result
and a second multi-classifier fusion that uses a non-associative
function to fuse the preferences of risk classes into a second
aggregated result. A production decision engine assigns the
financial risk instrument application to a preference from a set of
predetermined risk classes according to the at least one risk
attribute associated with the application and generates a
production result representative of the risk of the application. A
comparison engine selects one of the first aggregated result
generated from the first multi-classifier fusion module and the
second aggregated result generated from the second multi-classifier
fusion module and compares with the production result generated
from the production decision engine. The comparison engine
generates an underwriting decision for the financial risk
instrument application according to the comparison.
[0010] In another embodiment, there is a computer-implemented
process and computer readable medium for underwriting a financial
risk instrument application represented by at least one risk
attribute. In this embodiment, the at least one risk attribute
associated with the financial risk instrument application is
examined with a plurality of decision engines. Each decision engine
is then used to assign the application to one of a set of
predetermined risk classes. The assigned risk classes are fused
into an aggregated result representative of the risk of the
financial risk instrument application. The fusing comprises
applying an associative function to fuse the assigned risk classes
into a first aggregated result and applying a non-associative
function to fuse the assigned risk classes into a second aggregated
result. Then one of the first aggregated result and the second
aggregated result is selected. The selected aggregated result is
then compared with a production result generated from a production
decision engine. An underwriting decision for the financial risk
instrument application is generated according to the comparison of
the selected aggregated result with the production result.
[0011] In still another embodiment, there is a computer-implemented
process and computer readable medium for underwriting a financial
risk instrument application represented by at least one risk
attribute. In this embodiment, the at least one risk attribute
associated with the financial risk instrument application is
examined with a plurality of decision engines. Each decision engine
is used to assign a preference from a set of predetermined risk
classes for the application. The preference of risk classes
provides a conviction in applicability of each risk class assigned
to the financial risk instrument application. The preferences of
risk classes are fused into an aggregated result representative of
the risk of the financial risk instrument application. The fusing
comprises applying an associative function to fuse the preferences
of risk classes into a first aggregated result and applying a
non-associative function to fuse the preferences of risk classes
into a second aggregated result. Then one of the first aggregated
result and the second aggregated result is selected. The selected
aggregated result is compared with a production result generated
from a production decision engine. An underwriting decision for the
financial risk instrument application is generated according to the
comparison of the selected aggregated result with the production
result.
[0012] In yet another embodiment of the invention, there is a
system for underwriting a financial risk instrument application
represented by at least one risk attribute. In this embodiment,
there is a plurality of decision engines that each examines the at
least one risk attribute associated with the financial risk
instrument application and assigns the application to one of a
predetermined set of risk classes. A fusion engine compares the
risk classes assigned by each of the plurality of decision engines
and fuses the assigned risk classes into an aggregated result
representative of the risk of the financial risk instrument
application. The fusion engine comprises a first multi-classifier
fusion module that uses an associative function to fuse the
assigned risk classes into a first aggregated result and a second
multi-classifier fusion that uses a non-associative function to
fuse the assigned risk classes into a second aggregated result. A
comparison engine selects between the first aggregated result
generated from the first multi-classifier fusion module and the
second aggregated result generated from the second multi-classifier
fusion module. The selected result is representative of an
underwriting decision for the financial risk instrument
application.
[0013] In a sixth embodiment, there is a computer-implemented
process and computer readable medium for underwriting a financial
risk instrument application represented by at least one risk
attribute. In this embodiment, the at least one risk attribute
associated with the financial risk instrument application is
examined with a plurality of decision engines. Each decision engine
is then used to assign the application to one of a set of
predetermined risk classes. The assigned risk classes are fused
into an aggregated result representative of the risk of the
financial risk instrument application. The fusing comprises
applying an associative function to fuse the assigned risk classes
into a first aggregated result and applying a non-associative
function to fuse the assigned risk classes into a second aggregated
result. Then one of the first aggregated result and the second
aggregated result is selected. An underwriting decision for the
financial risk instrument application is generated according to the
selected aggregated result.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] FIG. 1 illustrates the architecture of a quality assurance
system based on the fusion of multiple classifiers according to an
embodiment of the invention.
[0015] FIG. 2 illustrates a table of an outer product using the
function T(x,y) according to an embodiment of the invention.
[0016] FIG. 3 illustrates the disjointed risk classes within the
universe of risk classes according to an embodiment of the
invention.
[0017] FIG. 4 illustrates the results of the intersections of the
risk classes and the universe according to an embodiment of the
invention.
[0018] FIGS. 5-9 illustrate the results of T-norm operators
according to an embodiment of the invention.
[0019] FIGS. 10-14 illustrate the normalized results of T-norm
operators according to an embodiment of the invention.
[0020] FIG. 15 illustrates a summary of the fusion of two
classifiers according to an embodiment of the invention.
[0021] FIG. 16 illustrates a penalty matrix for a particular fusion
module according to an embodiment of the invention.
[0022] FIG. 17 illustrates a summary of the fusion of two
classifiers with disagreement according to an embodiment of the
invention.
[0023] FIG. 18 illustrates a summary of the fusion of two
classifiers with agreement and discounting according to an
embodiment of the invention.
[0024] FIGS. 19-23 illustrate the results of T-norm operators
according to an embodiment of the invention.
[0025] FIGS. 24-28 illustrate the normalized results of T-norm
operators according to an embodiment of the invention.
[0026] FIG. 29 illustrates a Dempster-Schaefer penalty matrix
according to an embodiment of the invention.
[0027] FIG. 30 illustrates a comparison matrix according to an
embodiment of the invention.
[0028] FIG. 31 illustrates fusion as a function of a confidence
threshold for Type G cases according to an embodiment of the
invention.
[0029] FIG. 32 illustrates fusion as a function of a confidence
threshold for Type H cases according to an embodiment of the
invention.
[0030] FIG. 33 illustrates a Venn diagram for fusion for Type G
cases according to an embodiment of the invention.
[0031] FIG. 34 illustrates a Venn diagram for fusion for Type H
cases according to an embodiment of the invention.
[0032] FIG. 35 schematically illustrates the classes of fusion
aggregation for the multi-classifier fusion modules shown in FIG.
1.
[0033] FIG. 36 is an example illustrating the operation of a
multi-classifier fusion module in FIG. 1 that uses a
non-associative function.
[0034] FIG. 37 is a flowchart illustrating the operation of the
system shown in FIG. 1.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
[0035] A system, process and computer program product for
underwriting of financial risk instrument applications that is
suitable for use by a computer rather than by human intervention is
described. The system, process and computer program product make
use of existing risk assignments made by human underwriters to
categorize new applications in terms of the risk involved. The term
"financial risk instrument" application generally refers to
financial credit (or loan) applications. It will be recognized,
that the principles disclosed herein may extend beyond the realm of
financial credit underwriting and that it may be applied to any
risk classification process where there is a determination of the
proper payments to cover a given risk. Therefore, the ultimate
domain of this invention may be considered risk classification. As
such, it can be applied as well to risk categorization of service
agreements (e.g., service contracts for aircraft engines or power
system installations or water purification installations).
Therefore, as used herein, the term "financial risk instrument"
refers to financial credit (or loan) applications and/or
applications for service agreements. "Instrument application" is
used as a short phrase to invoke the same concept.
[0036] One technical effect of the invention is to provide an
automated process for consistent and accurate underwriting
decisions for instrument applications. Another technical effect of
the invention is to provide accurate decision support to
underwriters when complexity or inadequate information preclude the
making of a final decision automatically. Various aspects and
components of this system, process and computer product are
described below.
[0037] An aspect of the invention provides a system, process and
computer product for fusing a collection of classifiers used for an
automated financial risk instrument underwriting system and/or its
quality assurance. While the design method is demonstrated for
quality assurance of automated financial risk instrument
underwriting, it is broadly applicable to diverse decision-making
applications in business, commercial, and manufacturing processes.
A process of fusing the outputs of a collection of classifiers is
provided. The fusion can compensate for the potential correlation
among the classifiers. The reliability of each classifier can be
represented by a static or dynamic discounting factor, which will
reflect the expected accuracy of the classifier. A static
discounting factor represents a prior expectation about the
classifier's reliability, e.g., it might be based on the average
past accuracy of the model. A dynamic discounting represents a
conditional assessment of the classifier's reliability, e.g.,
whenever a classifier bases its output on an insufficient number of
points, the result is not reliable. Hence, this factor could be
determined from the post-processing stage in each model. The fusion
of the data will typically result in some amount of consensus and
some amount of conflict among the classifiers. The consensus will
be measured and used to estimate a degree of confidence in the
fused decisions. A particular fusion aggregation approach may be
too strict to gain a high consensus with limited cases available.
In order to overcome this potential drawback, a second fusion
aggregation with fewer classifiers or a different mode of
aggregation is used to provide greater coverage. The details of the
second fusion depend on the context of the particular instrument
application being underwritten. A comparison engine, with context
knowledge, can provide for a selection of the fusion approach or
provide a decision by a combination procedure.
[0038] According to an embodiment of the invention, a fusion module
(also referred to as a fusion engine) combines the outputs of
several classifiers (also referred to as decision engines) to
determine the correct rate class (also referred to as a risk class)
for an instrument application. Using a fusion module with several
decision engines may enable a classification to be assigned with a
higher degree of confidence than is possible using any single
model. According to an embodiment of the invention, a fusion module
function may be part of a quality assurance ("QA") process to test
and monitor a production decision engine ("PDE") that makes the
risk class assignment in real-time. For example, at periodic
intervals, e.g., every week, the fusion module and its components
may review the decisions made by the PDE during the previous week.
The output of this review will be an assessment of the PDE
performance over that week, as well as the identification of cases
with different level of decision quality.
[0039] The fusion module may permit the identification of the best
cases of application classification, e.g., those with
high-confidence, high-consensus decisions. These best cases in turn
may be likely candidates to be added to the set of test cases used
to tune/re-train the PDE. Further, the fusion module may permit the
identification of the worst cases of application classification,
e.g., those with low-confidence, low-consensus decisions. These
worst cases may be likely candidates to be selected for a review by
an auditing staff and/or by senior underwriters.
[0040] The fusion module may also permit the identification of
unusual cases of application classification, e.g., those with
unknown confidence in their decisions, for which the models in the
fusion module could not make any strong commitment or avoided the
decision by routing the instrument application to a human
underwriter. These cases may be candidates for a blind review by
senior underwriters. In addition, a fusion module may also permit
an assessment of the performance of the PDE, by monitoring the PDE
accuracy and variability over time, such as monitoring the
statistics of low, borderline and high quality cases as well as the
occurrence of unusual cases. These statistics can be used as
indicators for risk management.
[0041] According to an embodiment of the invention, the fusion
module may leverage the fact that except for the unusual situation
where all the components (e.g., models) contain the same
information (e.g., an extreme case of positive correlation), each
component should provide additional information. This information
may either corroborate or refute the output of the other modules,
thereby supporting either a measure of consensus, or a measure of
conflict. These measures may define a confidence in the result of
the fusion. In general, the fusion of the components' decisions may
provide a more accurate assessment than the decision of each
individual component.
[0042] According to an embodiment of the invention, the fusion
module may leverage the fact that except for the unusual situation
where all the components (e.g., models) contain the same
information (e.g., an extreme case of positive correlation), each
component should provide additional information. If this
information refutes the output of the other modules, thereby
supporting a measure of conflict, the fusion module provides
decision support information to the human underwriter on the nature
of the conflicts that need to be resolved.
[0043] The fusion module is described in relation to various types
of decision engines, including a case-based decision engine, a
dominance-based decision engine, a multi-variate adaptive
regression splines engine, and a neural network decision engine
respectively. However, the fusion module may use any type of
decision engine. According to an embodiment of the invention, the
fusion module will support a quality assurance process for a
production decision engine. However, it is understood that the
fusion module could be used for a quality assurance process for any
other decision making process, including a human underwriter.
[0044] According to an embodiment of the invention, a general
method for the fusion process, which can be used with decision
engines that may exhibit any kind of (positive, neutral, or
negative) correlation with each other, may be based on the concept
of triangular norms ("T-norm"), a multi-valued logic generalization
of the Boolean intersection operator. The fusion of multiple
decisions, produced by multiple sources, regarding objects (e.g.,
classes) defined in a common framework (e.g., the universe of
discourse) consists of determining the underlying of degree of
consensus for each object (e.g., class) under consideration, i.e.,
the intersections of their decisions. With the intersections of
multiple decisions, possible correlation among the sources needs to
be taken into account to avoid under-estimates or over-estimates.
This is done by the proper selection of a T-norm operator.
[0045] According to an embodiment of the invention, each model is
assumed to be solving the same classification problem. Therefore,
the output of each classifier is a weight assignment that
represents the degree to which a given class is selected. The set
of all possible classes, referred to as U, represents the common
universe of all answers that can be considered by the classifiers.
The assignment of weights to this universe represents the
classifier's ignorance (i.e., lack of commitment to a specific
decision). This is a discounting mechanism that can be used to
represent the classifier's reliability.
[0046] According to an embodiment of the invention, the outputs of
the classifiers may be combined by selecting the generalized
intersection operator (e.g., the T-norm) that better represents the
possible correlation between the classifiers. With this operator,
the assignments of the classifiers are intersected and a derived
measure of consensus is computed. This fusion may be performed in
an associative manner, e.g., the output of the fusion of the first
two classifiers is combined with the output of the third
classifier, and so on, until all available classifiers have been
considered and a non-associative manner.
[0047] Thus, according to an embodiment of the invention, a fusion
module only considers weight assignments made either to disjoint
subsets that contain a singleton (e.g., a risk class) or to the
entire universe of classes U (e.g., the entire set of risk
classes), as will be described in greater detail below. Once
compensation has been made for correlation and fusion has been
performed, the degree of confidence C is computed among the
classifiers and used to qualify the decision obtained from the
fusion. Further, the confidence measure and the agreement or
disagreement of the fusion module's decision is used with the
production engine's decision to assess the quality of the
production engine. As a by-product, the application cases may be
labeled in terms of the decision confidence. Thus, cases with low,
high, or unknown confidence may be used in different ways to
maintain and update the production engine.
[0048] Triangular norms (T-norms) and Triangular conorms
(T-conorms) are the most general families of binary functions that
satisfy the requirements of the conjunction and disjunction
operators, respectively: T-norms T(x,y) and T-conorms S(x,y) are
two-place functions that map the unit square into the unit
interval, i.e., T(x,y): [0,1].times.[0,1].fwdar- w.[0,1] and
S(x,y): [0,1].times.[0,1].fwdarw.[0,1]. They are monotonic
commutative and associative functions. Their corresponding boundary
conditions, i.e., the evaluation of the T-norms and T-conorms at
the extremes of the [0,1] interval, satisfy the truth tables of the
logical AND and OR operators. They are related by the DeMorgan
duality, which states that if N(x) is a negation operator, then the
T-conorm S(x,y,) can be defined as S(x,y)=N(T(N(x), N(y))).
[0049] As described in Bonissone and Decker (1986) the contents of
which are incorporated by reference in their entirety, six
parameterized families of T-norms and their dual T-conorms may be
used. Of the six parameterized families, one family was selected
due to its complete coverage of the T-norm space and its numerical
stability. This family has a parameter p. By selecting different
values of p, T-norms with different properties can be instantiated,
and thus may be used in the fusion of possibly correlated
classifiers.
[0050] Various articles discuss the fusion and the different
features associated therewith, include proofs as to the development
of algorithms associated with the present invention. Chibelushi et
al. (Chibelushi, C. C., Deravi, F., and Mason, J. S. D., "Adaptive
Classifier Integration for Robust Pattern Recognition," IEEE
Transactions on Systems, Man, and Cybernetics, vol. 29, no. 6,
1999, the contents of which are incorporated herein by reference)
describe a linear combination method for combining the outputs of
multiple classifiers used in speaker identification
applications.
[0051] Fairhurst and Rahman (Fairhurst, M. C., and Rahman, A. F.
R., "Enhancing consensus in multi expert decision fusion," IEE
Proc.-Vis. Image Signal Process, vol. 147, no. 1, 2000, the
contents of which are incorporated herein by reference) describe
ENCORE, a multi-classifier fusion system for enhancing the
performance of individual classifiers for pattern recognition
tasks, specifically, the task of hand written digit recognition.
Kuncheva and Jain (Kuncheva, L. I., and Jain, L. C., "Designing
Classifier Fusion Systems by Genetic Algorithms," IEEE Transactions
on Evolutionary Computation, vol. 4, no. 4, 2000, the contents of
which are incorporated herein by reference) describe a genetic
algorithm approach to the design of fusion of multiple
classifiers.
[0052] Xu et al. (Xu, L., Krzyzak, A., and Suen, C. Y., "Methods of
Combining Multiple Classifiers and Their Applications to
Handwriting Recognition," IEEE Transactions on Systems, Man, and
Cybernetics, vol. 22, no. 3, 1992, the contents of which are
incorporated herein by reference) describe several standard
approaches for classifier decision fusion, including the
Dempster-Shafer approach, and demonstrate fusion for handwritten
character recognition.
[0053] Arthur Dempster (A. P. Dempster, "Upper and lower
probabilities induced by a multivalued mapping," Annals of
Mathematical Statistics, 38:325-339, 1967, the contents of which
are incorporated herein by reference) describes a calculus based on
lower and upper probability bounds. Dempster's rule of combination
describes the pooling of sources under the assumption of evidential
independence. Glenn Shafer (G. Shafer, "A Mathematical Theory of
Evidence", Princeton University Press, Princeton, N.J., 1976, the
contents of which are incorporated herein by reference) describes
the same calculus discovered by Dempster, but starting from a set
of super-additive belief functions that are essentially lower
bounds. Shafer derives the same rule of combination as Dempster.
Enrique Ruspini (E. Ruspini, "Epistemic logic, probability, and the
calculus of evidence. Proc. Tenth Intern. Joint Conf. on Artificial
Intelligence, Milan, Italy, 1987, the contents of which are
incorporated herein by reference) goes on to describe a
possible-world semantics for Dempster-Shafer theory.
[0054] B. Schweizer and A. Sklar (B. Schweizer and A. Sklar,
"Associative Functions and Abstract Semi-Groups", Publicationes
Mathematicae Debrecen, 10:69-81, 1963, the contents of which are
incorporated herein by reference) describe a parametric family of
triangular T-norm functions that generalize the concept of
intersection in multiple-valued logics. Piero Bonissone and Keith.
Decker (P. P. Bonissone and K. Decker, "Selecting Uncertainty
Calculi and Granularity: An Experiment in Trading-off Precision and
Complexity" in Kanal and Lemmer (editors) Uncertainty in Artificial
Intelligence, pages 217-247, North-Holland, 1986, the contents of
which are incorporated herein by reference) describe an experiment
based on Schweizer and Sklar's parameterized T-norms. They show how
five triangular norms can be used to represent an infinite number
of T-norms for some practical values of information granularity.
Piero Bonissone (P. P. Bonissone, "Summarizing and Propagating
Uncertain Information with Triangular Norms", International Journal
of Approximate Reasoning, 1(1):71-101, January 1987, the contents
of which are incorporated herein by reference) also describes the
use of Triangular norms in dealing with uncertainty in expert
system, Specifically he shows the use Triangular norms to aggregate
the uncertainty in the left-hand side of production rules and to
propagate it through the firing and chaining of production
rules.
[0055] FIG. 1 illustrates the architecture of a quality assurance
system based on the fusion of multiple classifiers (decision
engines) according to an embodiment of the invention for
underwriting an instrument application that is represented by at
least one risk attribute. A risk attribute as used in this
invention is a contributing factor to the overall risk of the
instrument application that must be evaluated to provide input to
the assessment the overall risk. An illustrative, but
non-exhaustive list of risk attributes includes items such as
financial statements, credit ratings, corporate structure, market
and other key input. For the service agreement scenario, one type
of risk attribute could be the percentage of time that equipment is
run at peak power. One of ordinary skill in the art will recognize
that there are other possible risk attributes for the service
agreement scenario.
[0056] System 100, as illustrated in FIG. 1, includes a number of
quality assurance decision engines 110 that comprise a case-based
reasoning decision engine 112, a MARS decision engine 114, a neural
network decision engine 116, and a dominance-based decision engine
118. The case-based reasoning decision engine 112, MARS decision
engine 114, neural network decision engine 116, and dominance-based
decision engine 118 use classifiers based on a case-based reasoning
model, MARS model, neural network model and dominance-based model,
respectively. U.S. patent application Ser. Nos. 10/170,471 and
10/171,190 provide a more detailed description of the case-based
reasoning model classifier; U.S. patent application Ser. No.
10/425,722 provides a more detailed description of the MARS model
classifier; U.S. patent application Ser. No. 10/425,610 provides a
more detailed description of the neural network model classifier;
and U.S. patent application Ser. No. 10/425,610 provides a more
detailed description of the dominance-based model classifier It is
understood, however, that other types of quality assurance decision
engines 110 could be used in addition to and/or as substitutes for
those listed in the embodiment of the invention illustrated in FIG.
1.
[0057] In this embodiment for FIG. 1, each of the decision engines
120 examines the at least one risk attribute associated with the
financial risk instrument application and assigns the application
to one of a predetermined set of risk classes. Risk classes are
categorized by the letters A, B, C, D, E and represent the
financial risk of the application, in which A represents the lowest
risk and E the highest risk. These letter categories can be thought
of as representing very low risk, moderately low risk, moderate
risk, moderately high risk, high risk, respectively, for example;
very high risk would be considered for decline. One of skill in the
art will recognize that there are other ways of labeling and
designating risk classes.
[0058] Post processing modules 122, 124, 126, and 128 receive the
outputs from the various quality assurance decision engines 120 and
perform processing on the outputs. The results of the
post-processing are input into multi-classifier fusion engine or
module 129 that compares the risk classes assigned by each of the
decision engines 110 and fuses the assigned risk classes into an
aggregated result representative of the risk of the financial risk
instrument application. The multi-classifier fusion engine 129
comprises multi-classifier fusion modules 130 and 131.
Multi-classifier fusion engine 130 uses an associative function to
fuse the assigned risk classes into a first aggregated result and
multi-classifier fusion engine 131 uses a non-associative function
to fuse the assigned risk classes into a second aggregated result.
The first and second aggregated results are representative of a
fusion risk class decision 135 and 136, respectively.
[0059] The multi-classifier fusion engine 129 also comprises a
fusion confidence estimation engine that estimates a degree of
confidence in the aggregated result of the fusion engine. In
particular, the fusion confidence estimation engine estimates a
degree of confidence in the first aggregated result generated from
the multi-classifier fusion module 130 and a degree of confidence
in the second aggregated result generated from the multi-classifier
fusion module 131. The confidence measures estimated by the fusion
estimation engine are shown in FIG. 1 as confidence measures 140
and 141.
[0060] A comparison module or engine 150 receives the fusion risk
class decision 135 and 136 and confidence measures 140 and 141 from
the multi-classifier fusion engine 129. The comparison engine 150
uses the confidence measures 140 and 141 to select between the risk
class decisions 135 and 136 generated from multi-classifier fusion
modules 130 and 131. The comparison engine 150 may also receives
input from a production decision engine 145 that includes a
production risk class decision 147 and a production confidence
measure 149. In this embodiment, the production decision engine 145
is a fuzzy logic rule-based automatic decision engine, however, it
is within the scope of the invention to use a human as the decision
engine. The comparison engine 150 then compares the production
result generated from the production decision engine 145 with the
risk class decision and confidence measure selected from either the
multi-classifier fusion modules 130 and 131. After a comparison has
been made, the comparison engine 150 outputs a compared risk class
decision 151 and a compared confidence measure 153. The compared
risk class decision 151 and compared confidence measure 153 are
representative of an underwriting decision for the financial risk
instrument.
[0061] An evaluation module 155 may evaluate the case confidence
and consensus regarding the compared risk class 151 and the
compared confidence measure 153. Those cases evaluated as "worst
cases" are stored in case database 160 for standard underwriting,
and may be candidates for auditing. Those cases evaluated as
"unusual cases" are stored in case database 165, and may be
candidates for standard underwriting. Those cases evaluated as
"best cases" are stored in case database 170, and may be candidates
for using with the test sets. An outlier detector and filter 180
may ensure that any new additions to the best-case database 170
will be consistent, with the existing cases, preventing logical
outliers from being used.
[0062] As the details of the present invention are explained, one
of ordinary skill in the art will recognize that there are other
possible implementations and therefore the invention is not meant
to be limited to the particular configuration shown in FIG. 1. For
example, it is possible to implement the invention without
employing the production decision engine 145. In this case, the
risk class and confidence measure that the comparison engine 150
selects in its choice from the ones generated from the
multi-classifier fusion modules 130 and 131 will be representative
of the underwriting decision for the instrument application.
[0063] Below is a more detailed description of the quality
assurance run-time decision engines 110, production decision engine
145, multi-classifier fusion modules 130 and 131 and the comparison
module 150 and the functionality performed by each respective
module. Other aspects of System 100 of FIG. 1 are described in U.S.
patent application Ser. No. 10/425,721 with regard to insurance
underwriting, but may be extended to underwriting of an instrument
application.
[0064] Each decision engine 110 generates an output vector I=[I(1),
I(2), . . . I(N+1)] where I(i).di-elect cons.[0,M], where M is a
large real value and N is the number of risk classes. In the
embodiment of the invention illustrated in FIG. 1, each vector I is
identified by a superscript associated with the quality assurance
decision module 120 that generates the vector. Therefore, IC is
generated by case-based reasoning decision engine 112, IM is
generated by MARS decision engine 114, IN neural network decision
engine 116, and ID is generated by dominance-based decision engine
118. Further, each entry I(i), for i=1, . . . , N, can be
considered as the (un-normalized) degree to which the case could be
classified in risk class i. The last element, I(N+1) indicates the
degree to which the case cannot be decided and the entire universe
of risk classes is selected.
[0065] For illustration purposes, assume that five risk classes are
used, i.e., N=5, namely:
Risk Class={A, B, C,D, E, No Decision (Send to UW)}
[0066] By way of this example, assume that the output of the first
decision engine (CBE) is: IC=[0.3, 5.4, 0.3, 0, 0, 0]. This
indicates that the second risk class (e.g., B) is strongly
supported by the decision engine. Normalizing IC to see the support
as a percentage of the overall weights, .sup.C=[0.05, 0.9, 0.05, 0,
0,0], shows that 90% of the weights is assigned to the second risk
class.
[0067] Further, to represent partial ignorance, i.e., cases in
which the decision engine does not have enough information to make
a more specific risk classification, discounting may be used.
According to an embodiment of the invention, discounting may
involve the assignment of some weight to the last element,
corresponding to the universe U=(No Decision: Send to UW). For
example, the previous assignment of I.sup.C could be changed such
that I.sup.C=[0.3, 1.4, 0.3, 0, 0, 4], and its normalized
assignment would be .sup.C=[0.05, 0.23, 0.05, 0, 0, 0.67]. This
example shows how 67% of the weights have now been assigned to the
universe of discourse U (the entire set of risk classes). This
feature allows a representation of the lack of commitment by
individual modules. According to an embodiment of the invention, if
it is necessary to discount a source because it is not believed to
be credible, competent, or reliable enough in generating the
correct decision, a portion of the weight is transferred to the
universe of discourse (e.g., "any of the above categories"). The
determination of the discount may be derived from meta-knowledge,
as opposed to object-knowledge. Object knowledge is the level at
which each decision engine is functioning, e.g., mapping input
vectors into decision bins. Meta-knowledge is reasoning about the
decision engine's performance over time. Discounting could be
static or dynamic. Static discounting may be used a priori to
reflect historical (accuracy) performance of each decision engine.
Dynamic discounting may be determined by evaluating a set of rules,
whose Left Hand Side ("LHS") defines a situation, characterized by
a conjunct of conditions, and whose Right Hand Side ("RHS") defines
the amount by which to discount whichever output is generated by
the classifier. According to an embodiment of the invention,
postprocessing may be used to detect lack of confidence in a
source. When this happens, all the weights may be allocated to the
universe of discourse, i.e., refrain from making any decision.
[0068] According to an embodiment of the invention, each decision
engine will independently perform a post-processing step. For
purposes of illustration, the post processing used for the neural
network decision engine will be described. According to an
embodiment of the invention, to further improve the classification
performance of a neural network module, some post-processing
techniques may be applied to the outputs of the individual
networks, prior to the fusion process. For example, if the
distribution of the outputs did not meet certain pre-defined
criteria, no decision needs to be made by the decision engine.
Rather, the case will be completely discounted by allocating all of
the weights to the entire universe of discourse U. The rationale
for this particular example is that if a correct decision cannot be
made, it would be better not to make any decision rather than
making a wrong decision. Considering the outputs as discrete
membership grades for all risk classes, the four features that
characterize the membership grades may be defined as follows, where
N is the number of risk classes and I the membership function,
i.e., the output of the decision engine.
[0069] 1. Cardinality 1 C = 1 N I ( i )
[0070] 2. Entropy 2 E = 1 E max 1 N I ( i ) .times. log ( I ( i ) )
, where E max = - log ( 1 / N )
[0071] 3. Difference between the highest and the second highest
values of outputs.
1 D = I.sub.max1 - I.sub.max2
[0072] 4. Separation between the rank orders of the highest and the
second highest values of outputs
2 S = RankOrder(I.sub.max1) - RankOrder(I.sub.max2)
[0073] With the features defined for characterizing the network
outputs, the following two-step criteria may be used to identify
the cases with weak decisions:
[0074] Step 1: C<.tau..sub.1 OR C>.tau..sub.2 OR
E>.tau..sub.3
[0075] Step 2: D<.tau..sub.4 AND S.ltoreq.1
[0076] where .tau..sub.1, .tau..sub.2, .tau..sub.3, and .tau..sub.4
are the thresholds. The value of the thresholds is typically
dataset dependent. However, in some embodiments, the value of the
thresholds may be independent of the dataset. In the present
example related to a neural network decision engine (which in turn
is described in greater detail below), the value of the thresholds
may be first empirically estimated and then fine-tuned by a global
optimizer, such as an evolutionary algorithm. As part of this
example, the final numbers are shown below in Table 1. Other
optimization methods may also be used to obtain the thresholds.
3TABLE 1 Type G Type H Thresholds Application Application
.tau..sub.1 0.50 0.30 .tau..sub.2 2.00 1.75 .tau..sub.3 0.92 0.84
.tau..sub.4 0.10 0.21
[0077] Thus, post-processing may be used to identify those cases
for which the module's output is likely to be unreliable. According
to an embodiment of the invention, rather than rejecting such
cases, the model assignment of normalized weights to risk classes
may be discounted by assigning some or all of those weights to the
universe of discourse U.
[0078] As described previously, the fusion modules 130 and 131 may
perform the step of determining a combined decision via either the
associative or non-associative fusion of the decision engine
models' outputs. Associative fusion means that given three or more
decision engines, any two of the engines may be fused, and then
fusing the results with the third classifier, and so on, regardless
of the order. On the other hand, non-associative fusion means the
fusion cannot be done in a piece-wise manner because the order of
the operation is critical. For example, consider the simple
averaging operator acting on three values 1, 3 and 5. To get the
average, all the values are summed and divided by the number of
values, giving 9/3=3 as the average. The non-associativity can be
seen by averaging the values 1 and 3 to obtain 4/2=2 and then
forming the average of the result with 5 to obtain (2+5)/2=3.5.
This can be compared to averaging 3 and 5 to obtain 4 and then
averaging the result with 1 to obtain 2.5. Thus the averaging
operator is not associative.
[0079] Below is a description of the associative fusing operations
performed by the multi-classifier fusion module 130.
[0080] By way of example of determining a combined decision, define
m decision engine S.sub.1, . . . S.sub.m, such that the output of
decision engine S.sub.j is the vector I.sup.j showing the
normalized decision of such decision engine to the N risk classes.
Recall the last (N+1).sup.th element represents the decision
engine's lack of commitment, i.e., I.sup.j=[I.sup.j(1), I.sup.j(2),
. . . , I.sup.j(N+1)], where: 3 I j ( i ) [ 0 , 1 ] and i = 1 N + 1
I j ( i ) = 1
[0081] The un-normalized fusion of the outputs of two decision
engines S1 and S2 is further defined as:
F(I.sup.1,I.sup.2)=Outerproduct(I.sup.1,I.sup.2,T)=A
[0082] where the outer-product is a well-defined mathematical
operation, which in this case takes as arguments the two
N-dimensional vectors I.sup.1 and I.sup.2 and generates as output
the N.times.N dimensional array A. Each element A(i,j) is the
result of applying the operator T to the corresponding vector
elements, namely I.sup.1(i) and I.sup.2(j), e.g.,
A(i,j)=T[I.sup.1(i), I.sup.2(j)]
[0083] As illustrated in FIG. 2, matrix 200 illustrates classes 202
and values 204 for vector I.sup.1 and classes 206 and values 208
for vector I.sup.2. Intersection 210 illustrates one intersection
between the vector I.sup.1 and vector I.sup.2. Other intersections
and representations may also be used.
[0084] The operator T(x,y) may be referred to as a Triangular Norm.
Triangular Norms (also referred to as "T-norms") are general
families of binary functions that satisfy the requirements of the
intersection operators. T-norms are functions that map the unit
square into the unit interval, i.e., T:
[0,1].times.[0,1].fwdarw.[0,1]. T-norms are monotonic, commutative
and associative. Their corresponding boundary conditions, i.e., the
evaluation of the T-norms at the extremes of the [0,1] interval,
satisfy the truth tables of the logical AND operator.
[0085] As there appear to be an infinite number of T-norms, the
five most representative T-norms for some practical values of
information granularity may be selected. According to an embodiment
of the invention, the five T-norms selected are:
4 T-Norm Correlation Type T.sub.1(x, y) '2 max(0, x + y - 1)
Extreme case of negative correlation T.sub.1,5(x, y) '2 max(0,
x.sup.0.5 + y.sup.0.5 - 1).sup.2 Partial case of negative
correlation T.sub.2(x, y) '2 x * y No correlation T.sub.2.5(x, y)
'2 (x.sup.-1 + y.sup.-1 - 1).sup.-1 Partial case of positive
correlation T.sub.3(x, y) = min(x, y) Extreme case of positive
correlation
[0086] The selection of the best T-norm to be used as an
intersection operation in the fusion of the decision engines may
depend on the potential correlation among the engines to be fused.
For example, T3 (the minimum operator) may be used when one
decision engine subsumes the other one (e.g., extreme case of
positive correlation). T2 may be selected when the decision engines
are uncorrelated (e.g., similar to the evidential independence in
Dempster-Shafer). T1 may be used if the decision engines are
mutually exclusive (e.g., extreme case of negative correlation).
The operators T.sub.1.5 and T.sub.2.5 may be selected when the
decision engines show intermediate stages of negative or positive
correlation, respectively. Of course, it will be understood by one
of ordinary skill in the art that other T-norms may also be used.
However, for the purposes of the present invention, these five
T-norms provide a good representation of the infinite number of
functions that satisfy the T-norm properties.
[0087] Because the T-norms are associative, so is the fusion
operator, i.e.,
F(I.sup.1,F(I.sup.2, I.sup.3))=F(F(I.sup.1, I.sup.2),I.sup.3)
[0088] Each element A(i,j) represents the fused assignment of the
two decision engines to the intersection of risk classes r.sub.i
and r.sub.j. FIG. 3 illustrates that each risk class is disjointed
and that U 300, is the universe of all (risk) classes. In this
example, risk classes r.sub.1 302, r.sub.2 304 to r.sub.n 306 are
shown. Given that the risk classes are disjoint, there are five
possible situations:
[0089] (a) When i=j and i<(N+1) then
r.sub.i.andgate.r.sub.j=r.sub.j.an- dgate.r.sub.i=r.sub.i
[0090] (b) When i=j and i=(N+1) then r.sub.i.andgate.r.sub.j=U (the
universe of risk classes)
[0091] (c) When i.noteq.j and i<(N+1) and j<(N+1) then
r.sub.i.andgate.r.sub.j=.phi. (the empty set)
[0092] (d) When i.noteq.j and i=(N+1) then
U.andgate.r.sub.j=r.sub.j
[0093] (e) When i.noteq.j and j=(N+1) then
r.sub.i.andgate.U=r.sub.i
[0094] FIG. 4 depicts a chart 400 that illustrates the result of
the intersections of the risk classes and the universe U, according
to an embodiment of the invention. The chart demonstrates the
intersection according to those situations set forth above, such
that when situation (a) occurs, the results are tabulated in the
main diagonal identified as 410 in FIG. 4. Further, when situation
(b) occurs, the results are tabulated in the appropriate areas
identified as 420 in FIG. 4. When situation (c) occurs, the results
are tabulated in the appropriate areas identified as 430, while
when situations (d) or (e) occur, the results are tabulated in the
appropriate areas identified as 440 in FIG. 4. By way of example,
when one application is rated r1 in the first instance and r2 in
the second instance, the intersection may be tabulated at 450,
where the column for r1 and the row for r2 intersect. In this
example, the intersection of r1 and r2 is the empty set .phi.. The
decisions for each risk class can be gathered by adding up all the
weights assigned to them. According to the four possible situations
described above, weights may be assigned to a specific risk class
only in situation a) and d), as illustrated in FIG. 4. Thus, there
will be:
Weight (r.sub.i)=A(i,i)+A(i,N+1)+A(N+1,i)
Weight (U)=A(N+1,N+1)
[0095] To illustrate the fusion operator based on T-norms, an
example will now be described. Assume that
[0096] I.sup.1=[0.8, 0.15, 0.05, 0, 0, 0] and I.sup.2=[0.9, 0.05,
0.05, 0, 0, 0]
[0097] This indicates that both decision engines are showing a
strong preference for the first risk class (e.g., "A") as they have
assigned them 0.8 and 0.9, respectively. Fusing these decision
engines using each of the five T-norm operators defined above will
generate the corresponding matrices A that are shown in the tables
in FIGS. 5-9, such that FIG. 5 illustrates an extreme positive
correlation, FIG. 6 illustrates a partial positive correlation,
FIG. 7 illustrates no correlation, FIG. 8 illustrates a partial
negative correlation and FIG. 9 illustrates an extreme negative
correlation. If the results are normalized so that the sum of the
entries is equal to one, the matrices are generated, as shown in
the tables in FIGS. 10-14 in a manner corresponding to the
un-normalized results. During the process, the un-normalized
matrices A (FIGS. 5-9) may be used to preserve the associative
property. At the end, the normalized matrices are used (FIGS.
10-14). Using the expressions for weights of a risk class, the
final weights for the N risk classes and the universe U from FIGS.
10-14 can be computed. An illustration of the computation of the
final weights is illustrated in the chart of FIG. 15. Chart 1500
illustrates the five classes 1510, the five T-norms 1520, and the
fused intersection results 1530.
[0098] According to an embodiment of the invention, the confidence
in the fusion may be calculated by defining a measure of the
scattering around the main diagonal. The more the weights are
assigned to elements outside the main diagonal, the less is the
measure of the consensus among the classifiers. This concept may be
represented by defining a penalty matrix P=[P(i,j)], of the form: 4
P ( i , j ) = { max ( 0 , ( 1 - W * i - j ) ) d for 1 i N and 1 j N
1 for i = ( N + 1 ) or j = ( N + 1 )
[0099] This function rewards the presence of weights on the main
diagonal, indicating agreement between the two decision engines,
and penalizes the presence of elements off the main diagonal,
indicating conflict. The conflict increases in magnitude as the
distance from the main diagonal increases. For example, for W=0.2
and d=5 we have the penalty matrix set forth in FIG. 16. Matrix
1600 intersects the column classes 1610 with the row classes 1620
to determine the appropriate penalty.
[0100] Other functions penalizing elements off the main diagonal,
such as any suitable non-linear function of the distance from the
main diagonal, i.e., the absolute value .vertline.i-j.vertline.,
could also be used. The penalty function is used because the
conflict may be gradual, as the (risk) classes have an ordering.
Therefore, the penalty function captures the fact that the
discrepancy between risk classes r.sub.1 and r.sub.2 is smaller
than then the discrepancy between r.sub.1 and r.sub.3 The shape of
the penalty matrix P in FIG. 16 captures this concept, as P1600
shows that the confidence decreases non-linearly with the distance
from the main diagonal. A measure of the normalized confidence is
the sum of element-wise products between and P 1600, e.g.: 5 C ^ =
Normalized Confidence ( A ^ , P ) = i = 1 N + 1 j = 1 N + 1 A ^ ( i
, j ) * P ( i , j )
[0101] where is the normalized fusion matrix. The results of the
fusion of decision engines S1 and S2, using each of the five
T-norms with the associated normalized confidence measure 1540, are
shown in FIG. 15.
[0102] In a situation in which there is a discrepancy between the
two decision engines, this fact may be captured by the confidence
measure. For instance, consider a situation different from the
assignment illustrated in FIGS. 5-14, in which the decision engines
agreed to select the first risk class. Now for example, assume that
the two decision engines are showing strong preferences for
different risk classes, the first decision engine is selecting the
second risk class, while the second decision engine is favoring the
first class:
[0103] I.sup.1=[0.15, 0.85, 0.05, 0, 0, 0] and I.sup.2=[0.9, 0.05,
0.05, 0, 0, 0]
[0104] The results of their fusion are summarized in the table of
FIG. 17, where the chart 1700 illustrates the risk classes 1710,
the T-norms 1720 and the fused intersection results 1730. None of
the risk classes have a high weight and the normalized confidence
1740 has dropped.
[0105] According to an embodiment of the invention, it may be
desirable to be able to discount the one of the decision engine, to
reflect our lack of confidence in its reliability. For example, the
second decision engine (S2) in the first example (in which the
decision engines seemed to agree on selecting the first risk class)
may be discounted:
[0106] I.sup.1=[0.8, 0.15, 0.05, 0, 0, 0] and I.sup.2=[0.9, 0.05,
0.05, 0, 0, 0]
[0107] This discounting is represented by allocating some of the
decision engine's weight, in this instance 0.3, to the universe of
discourse U, (U=No decision: Sent_to_UW):
[0108] I.sup.1=[0.8, 0.15, 0.05, 0, 0, 0] and I.sup.2=[0.6, 0.05,
0.05, 0, 0, 0.3]
[0109] The results of the fusion of I.sup.1 and I.sup.2 are
summarized in FIG. 18 below. Summarization chart 1800 illustrates
the classes 1810, T-norms 1820, the fused intersection results 1830
and the confidence measure 1840. The risk classes have a slightly
lower weight (for T3, T2.5, T2), but the normalized confidence is
higher than with respect to FIG. 17, as there is less conflict.
Fusion matrices A are shown in the tables of FIGS. 19-23, while the
tables of FIGS. 24-28 illustrate matrices . According to an
embodiment of the invention, a fusion rule based on Dempster-Shafer
corresponds to the selection of:
[0110] a) T-norm operator T(x,y)=x*y; and
[0111] b) Penalty function using W=1 (or alternatively
d=.infin.)
[0112] Constraint b) implies the penalty matrix P 2900 illustrated
in FIG. 29. Therefore, the two additional constraints a) and b)
required by Dempster-Shafer theory (also referred to as "DS") imply
that the decision engines to be fused must be uncorrelated (e.g.,
evidentially independent) and that there is no ordering over the
classes, and any kind of disagreement (e.g., weights assigned to
elements off the main diagonal) can only contribute to a measure of
conflict and not, at least to a partial degree, to a measure of
confidence. In DS, the measure of conflict K is the sum of weights
assigned to the empty set. This corresponds to the elements with a
0 in the penalty matrix P 2900 illustrated in FIG. 29.
[0113] According to an embodiment of the invention, the normalized
confidence C described above may be used as a measure of
confidence, i.e.: 6 C ^ = Normalized Confidence ( A ^ , P ) = i = 1
N + 1 j = 1 N + 1 A ^ ( i , j ) * P ( i , j )
[0114] The confidence factor may be interpreted as the weighted
cardinality of the normalized assignments around the main diagonal,
after all the classifiers have been fused. In the case of DS, the
measure of confidence is the complement (to one) of the measure of
conflict K, i.e.: =1-K, where K is the sum of weights assigned to
the empty set.
[0115] An additional feature of the present invention is the
identification of cases that are candidates for a test set,
auditing, or standard reference decision process via the comparison
module. As illustrated in FIG. 1, the comparison engine 150 has
four inputs. These inputs include the decision of the production
engine 145, which according to an embodiment of the invention, is
one of five possible risk classes or a no-decision (e.g., "send the
case to a human underwriter"), i.e.:
D(PDE)=r1 and r1.di-elect cons.{A, B, C, D, E, Sent_to_UW}
[0116] As mentioned above, additional input to the comparison
engine 150 comprises the decision of the multi-classifier fusion
modules 130 and 131, which according to an embodiment of the
invention, is also one of five possible risk classes or a
no-decision (e.g., "send the case to a human underwriter"),
i.e.:
D(FUS)=r2 and r2.di-elect cons.{A, B, CD, E, Sent_to_UW}
[0117] An additional input may comprise the degree of confidence in
the production engine decision. A detailed description of the
computation of the confidence measure is described in the U.S.
patent application Ser. Nos. 10/173,000 and 10/171,575, entitled "A
Process/System for Rule-Based Insurance Underwriting Suitable for
Use by an Automated System". This measure may be equated to the
degree of intersection of the soft constraints used by a production
decision engine ("PDE"). This measure may indicate if a case had
all its constraints fully satisfied (and thus C(PDE)=1) or whether
at least one constraint was only partially satisfied (and therefore
C(PDE)<1).
[0118] An additional input may comprise the degree of confidence in
the fusion process. The normalized confidence measure is C(FUS).
According to an embodiment of the invention, the first test
performed is to compare the two decisions, i.e., D(PDE) and D(FUS).
FIG. 30 illustrates all the possible comparisons between the
decision of the production engine and the multi-classifier fusion
module 130. Comparison matrix 3000 illustrates the D(PDE) classes
3010 and the D(FUS) classes 3020. From the table it can be seen
that label I shows that D(PDE)=D(FUS) and they both indicate the
same, specific risk class. Further, label II shows that the fusion
module made no automated decision and suggested to send the
application to a human underwriter, i.e. D(FUS)=No Decision. Label
III shows that D(PDE).noteq.D(FUS) and that both D(PDE) and D(FUS)
indicate a specific, distinct risk class. In addition, label IV
shows that D(PDE).noteq.D(FUS), and in particular, that the PDE
made no automated decision and suggested to send the application to
a human underwriter, while the Fusion module selected a specific
risk class. Label V shows that D(PDE)=(FUS) and that both D(PDE)
and D(FUS) agree not to make any decision.
[0119] A second test may be done by using this information in
conjunction with the measures of confidence C(PDE) and C(FUS)
associated with the two decisions. With this information, the
performance of the decision engine may be assessed over time by
monitoring the time statistics of these labels, and the frequencies
of cases with a low degree of confidence. According to an
embodiment of the invention, a stable or increasing number of label
I's would be an indicator of good, stable operations. An increase
in the number of label II's would be an indicator that the fusion
module (with its models) needs to be retrained. These cases might
be shown to a team of senior underwriters for a standard reference
decision. An increase in the frequency of label III's or of cases
with low confidence could be a leading indicator of increased
classification risk and might warrant further scrutiny (e.g.,
auditing, retraining of the fusion models, re-tuning of the
production engine). An increase in label IV's may demonstrate that
either the production engine needs re-tuning and/or the fusion
modules needs retraining. An increase in label V's may demonstrate
an increase in unusual, more complex cases, possibly requiring the
scrutiny of senior underwriters. Thus, the candidates for the
auditing process will be the ones exhibiting a low degree of
confidence (C(FUS)<T1), regardless of their agreement with the
PDE and the ones for which the Fusion and the Production engine
disagree, i.e., the ones labeled C.
[0120] The candidates for the standard reference decision process
are the cases for which the multi-classifier fusion module 130
shows no decisions (labeled II or V). The candidates to augment the
test set may be selected among the cases for which the fusion
module and the production engine agree (label I). These cases may
be filtered to remove the cases in which the production engine was
of borderline quality (C(PDE)<T2) and the cases in which the
confidence measure of the fusion was below complete certainty
(C(FUS)<T1). Thresholds T1 and T2, may be data dependent and
must be obtained empirically. By way of example, T1=0.15 and T2=1.
Table 2 below summarizes the conditions and the quality assurance
actions required, according to an embodiment of the invention.
Dashes ("-`) in the entries of the table may indicate that the
result of the confidence measures are not material to the action
taken and/or to the label applied.
5TABLE 2 Decision Confidence Label from Measures FIG 30 C(PDE)
C(FUS) ACTION I .gtoreq.T2 .gtoreq.T1 Candidate to be added to data
set for tuning of PLE II -- -- Candidate for Stand Ref Dec.
Process. After enough cases are collected, re- tune the classifiers
III -- -- Candidate for Auditing IV -- -- Candidate for Stand Ref
Dec. Process. After enough cases are collected, re- tune the
classifiers V -- -- Candidate for Stand Ref Dec. Process. After
enough cases are collected, re- tune the classifiers -- -- <T3
Candidate for Auditing
[0121] According to an embodiment of the invention, the
multi-classifier fusion module 130 may be implemented using
software code on a processor. By way of an example of the results
of an implementation of the present invention, a fusion module was
tested against a case base containing a total of 2,879 cases. After
removing 173 UW cases, the remaining 2,706 cases were segmented
into 831 Type H applications, with three risk classes, and 1,875
Type G applications, with five risk-classes. These cases were then
used to test the fusion process. Because the cases for which the
production engine had made no decision were removed, use of a
comparison matrix similar to the one of Table 1400 will only have
labels I, II, III. The fusion was performed using the T-norm
T2(x,y)=x*y.
[0122] FIG. 31 illustrates the effect of changing the threshold T1
on the measure of confidence , were 0.ltoreq..ltoreq.1. Table 3100
displays decisions 3110, confidence thresholds 3120 and the case
distributions 3130 based on the confidence threshold 3120. Each
column shows the number of cases whose measure of confidence
is.gtoreq.T1. As the threshold is raised, the number of "No Fusion
Decision" increases. A "No Fusion Decision" occurs when the results
of the fusion are deemed too weak to be used. When the threshold T1
is 0, no case is rejected on the basis of the measure of conflict.
This leaves 36 cases for which no decision could be made. As the
threshold is increased, decisions with a high degree of conflict
are rejected, and the number of "No Fusion Decisions"
increases.
[0123] "Agreements" occur when the fused decision agrees with the
production decision engine 145 and with the Standard Reference
Decision (SRD). "False Positives" occur when the fused decision
disagrees with the production decision engine 145, which in turn is
correct since the production decision engine agrees with the
Standard Reference Decision ("SRD"). "False Negatives" occur when
the fused decision agrees with the production decision engine 145,
but both the fusion decision and the production decision engine are
wrong, as they disagree with the SRD. "Corrections" occur when the
fused decision agrees with the SRD and disagrees with the
production decision engine. Finally, "Complete Disagreement" occurs
when the fused decision disagrees with the production decision
engine, and both the fused decision and the production decision
engine 145 disagree with the SRD. Further, similar results were
obtained for Type H applications, and these results are illustrated
in FIG. 32, with table 3200 displaying decisions 3210, confidence
thresholds 3220 and the case distributions 3230 based on the
confidence thresholds 3220.
[0124] FIG. 33 illustrates a Venn diagram 3300 illustrating the
situation for the threshold T1=0.15 (i.e., for C.gtoreq.0.15) for
the Type G applications, while FIG. 34 illustrates a Venn diagram
3400 illustrating the situation for the threshold T1=0.15 (i.e.,
for C.gtoreq.0.15). In the case of the Type G applications (for
T1=0.15) the following labels result:
[0125] A: 1,588+27=1,615 (86.13%) in which 3310 D(FUS)=D(PDE);
(e.g., agreements 3310 and false negative 3320)
[0126] B: =36 (1.92%) in which the fusion did not make any decision
(from =0);
[0127] C1: 212-36=176 (9.39%) in which the fusion was too
conflictive (<0.15); and
[0128] C2: 22+25+1=48 (2.56%) in which D(FUS).noteq.D(PDE) (e.g.,
false positive 3330, corrections 3340 and complete disagreements
3350).
[0129] In the case of the Type H applications (for T1=0.15), the
following labels result:
[0130] A: 729+15=744 cases (89.5%) in which D(FUS)=D(PDE); (e.g.,
agreements 3410 and false negatives 3420);
[0131] B: =37 cases (4.5%) in which the fusion did not make any
decision (from =0);
[0132] C1: 68-37=31 cases (3.7%) in which the fusion was too
conflictive (<0.15); and
[0133] C2: 16+3=19 cases (2.3%) in which D(FUS).noteq.D(PDE) (e.g.,
false positives 3430, corrections 3440 and complete disagreements
3450).
[0134] According to the present example, since there is no SRD in
production, there can only be reliance on the degree of conflict
and the agreement between the fused decision and the production
decision engine 145. If the disagreement between production
decision engine and FUS (e.g., subset C2) is used, it can be
observed that the number of cases in which the fusion will disagree
with the production decision engine, and make a classification, is
48/1875 (2.56%) for Type G applications and 19/831 (2.3%) for Type
H applications . This may be considered a manageable percentage of
cases to audit. Further, this sample of cases may be augmented by
additional cases sampled from subsets C1.
[0135] A further analysis of set C2 in the case of Type G
applications shows that out of 48 cases, the fusion module called
22 of them correctly and 26 of them incorrectly. From the 26
incorrectly called cases, 14 cases were borderline cases according
to the production decision engine. This illustrates that the
problematic cases may be correctly identified and are good
candidates for an audit.
[0136] A further analysis of set C2 in the case of Type H
applications shows that out of 19 cases, the multi-classifier
fusion module 130 incorrectly called 16. Of these 16 cases, 6 cases
were borderline cases, i.e., the production decision engine 145
only had partial degree of satisfaction of the intersection of all
the constraints e.g., C(PDE)<0.9. Furthermore, 11 cases had a
conflict measure <0.4. If the union of these two subsets (e.g.,
the borderline cases and the conflict measure cases) is taken, the
results are 13 cases that are either borderline (from the PDE) or
have low confidence in the fusion, and the remaining 3 cases were
ones that the CBE could not classify (i.e., it could not find
enough similar cases). This again demonstrates that the problematic
cases may be generally correctly identified and are worth
auditing.
[0137] The set B (4.5%) illustrates a lack of commitment and is a
candidate for a review to assign an SRD. The set A may be a
starting point to identify the cases that could go to the test set.
However, set A may need further filtering by removing all cases
that were borderline according to the PDE (i.e., C(PDE)<T2), as
well as removing those cases whose fusion confidence was too low
(i.e., C(FUS)<1). Again T2 will be determined empirically, from
the data.
[0138] Various aspects of the multi-classifier fusion module 131
will now be discussed in greater detail below. Instead of using an
associative function such as a T-Norm operator as with the
multi-classifier fusion module 130, the multi-classifier fusion
module 131 employs convex combination and averaging operators such
as a geometric average, an arithmetic average, a majority vote,
etc. FIG. 35 schematically illustrates the classes of fusion
aggregation generated from multi-classifier fusion modules 130 and
131. In FIG. 35, the intersection operators used in obtaining the
intersection of sources is represented by reference numeral 3510,
with some explicit examples of such fusion operators given by
reference numeral 3570. These are the operators used by the
multi-classifier fusion module 130. As the number of sources or as
the complexity of the problem increase, it is more difficult to end
up with non-empty intersections. For instance, it would be enough
to have one source disagreeing with the remaining (n-1 sources) to
experience a total conflict. Therefore it is necessary to also
consider compensating tradeoff operators, which are situated
between the intersection and union operators and represented by
reference numeral 3550. Explicit examples of such fusion operators
are shown in FIG. 35 by reference numeral 3590.
[0139] This invention has recognized that sometimes there may be a
situation where there is a lack of consensus for a large number of
cases for n decision engines and therefore no decision is possible
and there is good evidence that one or more classifiers may not
apply for a sub-set of applications. This invention has overcome
this potential drawback by using the multi-classifier fusion module
131 in addition to the multi-classifier fusion module 130. The
multi-classifier fusion module 131 is operative with m decision
engines, where m may be a sub-set of decision engines used by the
multi-classifier fusion module 130. Furthermore, multi-classifier
fusion module 131 is extended to allow the aggregation of decision
engines beyond the intersection operation that uses the T-Norm
approach such as the multi-classifier fusion module 130. In
particular, multi-classifier fusion module 131 may aggregate the
decision engines according to convex combinations, such as a
geometric average, an arithmetic average, a majority vote, etc.
which are shown in FIG. 35 by reference numerals 3550 and 3590. The
relationships among the possible aggregation approaches are
illustrated in FIG. 35.
[0140] FIG. 36 is an example illustrating the operation of the
multi-classifier fusion module 131. In this example, FIG. 36 shows
that there are three decision engines 110 referenced as S1, S2, and
S3 that have ordered a set of four alternatives risk classes, A, B,
C, D from the set {A,B,C,D} and that decision engine 145 had an
initial ordering of {A,D,B,D}. The initial ordering from the
decision engine 145 needs to be reconciled with the three decision
engines S1, S2, and S3. This ordering implies that the decision
engine 145 initially prefers A over D over B over C. These
preferences may be stated strictly in an ordinal manner as just
described or may also be expressed as a cardinal number that adds
additional information about the preferences. The preferences from
the decision engines S1, S2, and S3 are as follows:
S1.fwdarw.{D,A,B,C}
S2.fwdarw.{B,A,C,D}
S3.fwdarw.{C,A,B,D}
[0141] The multi-classifier fusion module 131 takes these
individual ordered sets for each of the decision engines S1, S2 and
S3 and produces a fused ordered set according to the type of convex
combination and averaging operator selected.
[0142] One convex combination and averaging operator that the
multi-classifier fusion module 131 may use is a majority vote
operator that employs an average majority-voting rule to assign for
each pair-wise ordering whether or not a majority voted for that
ordering. Referring to FIG. 36, both S1 and S3 rank alternative A
higher than B so that the fusion operator would favor alternative A
with respect to alternative B. More generally, the multi-classifier
fusion module 131 would produce values that populate a preference
matrix with entry (i,j) indicating the preference relation between
alternative i and j. For one form of the majority-voting rule used
as the fusion operator (referred to as operator 1), the preference
matrix for the present example would look as follows:
6 Operator 1 Alternative j Alternative i A B C D A 0 1 1 1 B 0 0 1
1 C 0 0 0 1 D 0 0 0 0
[0143] where an entry (i,j) is set to 1 if a majority of the
decision engines prefer alternative i to alternative j. Using this
matrix, the following global ordering G is obtained as the result
of the fusion of the orderings expressed by the individual decision
engines, which is the output generated from multi-classifier fusion
module 131:
G.fwdarw.{A,B,C,D}
[0144] When the comparison module 150 compares the fused result to
the production decision engine 145 ordering of {A,D,B,C}, it will
see that the decision engine's choice of alternative A as being the
best alternative is in agreement with the decisions of the decision
engines using the majority rule.
[0145] The above example describes one instance of a convex
combination and averaging operator that can be used by the
multi-classifier fusion module 131, and one of ordinary skill in
the art will recognize that similar examples can be provided for
other types of convex combination and averaging operators. For
instance, with the same example as above, one could use a fusion
operator which produces a more informative preference matrix by
using not just 0s and 1s as entries from a majority ruling but a
score that indicates the fraction of experts who agree on a
particular pair-wise ordering. The matrix for this type of fusion
operator will look as follows:
7 Alternative i A B C D A 0 2/3 2/3 2/3 B 1/3 0 2/3 2/3 C 1/3 1/3 0
2/3 D 1/3 1/3 1/3 0
[0146] Another fusion operator available to multi-classifier fusion
module 131 is described. Assume, for example, that there are four
decision engines and four risk classes, A, B, C, D and that the
decision engines provide the values summarized below.
8 Weight Classifier A B C D w.sub.1 S1 0.8 0.1 0.05 0.05 w.sub.2 S2
0.7 0 0.2 0.1 w.sub.3 S3 0 0 0.5 0.5 w.sub.4 S4 0.9 0.1 0 0
[0147] Each decision engine produces a normalized preference vector
described by the values in each risk class which is shown in the
above rows. Also there is a weight that can be assigned to each
decision engine depending on their experience or familiarity with
the particular commercial segment considered, etc. (for a human
underwriter) or the past performance of a particular automated
decision engine for this particular segment.
[0148] In order to assess the overall preference for a risk class,
Cj, the following fusion operation (a weighted average) for each
class is performed
<Cj>=.SIGMA..sub.iw.sub.ia.sub.i/.SIGMA..sub.iw.sub.i.
[0149] The sum is over the decision engines and the a.sub.i are the
values for each decision engine under rate class Cj. For
simplicity, it is assumed that the weights for all the decision
engines are the same. This gives the following fused preference
vector for the decision engine. [A, C, D, B]:=>[0.6, 0.1875,
0.1625, 0.05]=[<C1>, <C2>, <C3>, <C4>].
Associated with this fused decision engine preference, a measure of
confidence is also provided by using an entropy measure.
Confidence=1+.SIGMA.i
[<Ci>Log(<Ci>)+(1-<Ci>)Log(1-<C- i>)]
[0150] where the maximum confidence is 1 and the minimum confidence
is 0. When this is applied to the fused preference vector above for
this operator, the result obtained is Confidence=0.22.
[0151] FIG. 37 is a flowchart illustrating the operation of the
system shown in FIG. 1. In FIG. 37, the results (e.g.,
I.sub.C-I.sub.D) from the decision engines are inputted to the
multi-classifier fusion engine at 3700. The multi-classifier fusion
modules fuse the decision engine results at 3710. In particular,
multi-classifier fusion module 130 fuses the decision engine
results using an associative function such as a triangular-norm
operation, while multi-classifier fusion module 131 fuses the
decision engine results using a convex combination and averaging
operator. In addition, the multi-classifier fusion modules generate
confidence measures at 3720. The comparison engine receives the
fusion results and confidence measures at 3730. In addition, the
comparison engine receives input from the production decision
engine at 3740. The comparison engine chooses between the inputs
from the two fusion modules at 3750 based on which result has the
highest confidence. The comparison engine then compares the
resultant choice with the production decision engine input at 3760.
The output of this comparison is a risk class decision and a
confidence measure that is then used to assign to one of several
databases at 3770 for future use.
[0152] The foregoing flow charts, block diagrams and screen shots
of this disclosure show the functionality and operation of the
system shown in FIG. 1. In this regard, each block/component
represents a module, segment, or portion of code, which comprises
one or more executable instructions for implementing the specified
logical function(s). It should also be noted that in some
alternative implementations, the functions noted in the blocks may
occur out of the order noted in the figures or, for example, may in
fact be executed substantially concurrently or in the reverse
order, depending upon the functionality involved. Also, one of
ordinary skill in the art will recognize that additional blocks may
be added.
[0153] The various embodiments described above comprise an ordered
listing of executable instructions for implementing logical
functions. The ordered listing can be embodied in any
computer-readable medium for use by or in connection with a
computer-based system that can retrieve the instructions and
execute them. In the context of this application, the
computer-readable medium can be any means that can contain, store,
communicate, propagate, transmit or transport the instructions. The
computer readable medium can be an electronic, a magnetic, an
optical, an electromagnetic, or an infrared system, apparatus, or
device. An illustrative, but non-exhaustive list of
computer-readable mediums can include an electrical connection
(electronic) having one or more wires, a portable computer diskette
(magnetic), a random access memory (RAM) (magnetic), a read-only
memory (ROM) (magnetic), an erasable programmable read-only memory
(EPROM or Flash memory) (magnetic), an optical fiber (optical), and
a portable compact disc read-only memory (CDROM) (optical).
[0154] It is apparent that there has been provided in accordance
with this invention, a system, process and computer program product
for fusion classification for risk categorization in underwriting a
financial risk instrument. While the invention has been
particularly shown and described in conjunction with a preferred
embodiment thereof, it will be appreciated that variations and
modifications can be effected by a person of ordinary skill in the
art without and departing from the scope of the invention.
* * * * *