U.S. patent application number 10/867175 was filed with the patent office on 2004-12-23 for method and apparatus for evaluating television program recommenders.
Invention is credited to Lee, Kwok Pun, Schaffer, J. David.
Application Number | 20040261107 10/867175 |
Document ID | / |
Family ID | 23988838 |
Filed Date | 2004-12-23 |
United States Patent
Application |
20040261107 |
Kind Code |
A1 |
Lee, Kwok Pun ; et
al. |
December 23, 2004 |
Method and apparatus for evaluating television program
recommenders
Abstract
A method and apparatus for validating recommendations generated
by a television program recommender uses programmed viewing agents,
in which a viewing agent is programmed with a set of rules that
characterize the viewing preferences of a modeled viewer. During a
training phase, the programmed rules of a viewing agent are applied
to a set of training programs to obtain an agent viewing history,
which is processed by a profiler to derive an agent profile
containing a set of inferred rules. During an evaluation phase, the
programmed rules of the viewing agent are applied to test programs
to obtain an agent evaluation viewing set. In parallel, the
television program recommender generates a set of program
recommendations by applying the agent profile to the test programs.
The agent evaluation viewing set is then compared with the program
recommendations.
Inventors: |
Lee, Kwok Pun; (Yorktown
Heights, NY) ; Schaffer, J. David; (Wappingers Falls,
NY) |
Correspondence
Address: |
PHILIPS INTELLECTUAL PROPERTY & STANDARDS
P.O. BOX 3001
BRIARCLIFF MANOR
NY
10510
US
|
Family ID: |
23988838 |
Appl. No.: |
10/867175 |
Filed: |
June 14, 2004 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10867175 |
Jun 14, 2004 |
|
|
|
09500306 |
Feb 8, 2000 |
|
|
|
6766525 |
|
|
|
|
Current U.S.
Class: |
725/46 ;
348/E7.061 |
Current CPC
Class: |
H04N 21/4755 20130101;
H04N 21/4532 20130101; H04N 21/4668 20130101; H04N 21/454 20130101;
H04N 21/466 20130101; H04N 7/163 20130101; H04N 21/4667
20130101 |
Class at
Publication: |
725/046 |
International
Class: |
G06F 003/00; H04N
005/445; G06F 013/00 |
Claims
What is claimed is:
1. A method for validating program recommendations produced by a
program recommender, comprising the steps of: generating a viewing
agent using one or more programmed rules that characterize viewing
preferences; applying said programmed rules to a set of training
programs to obtain an agent viewing history; processing said agent
viewing history with a profiler to derive an agent profile
containing a set of inferred rules; applying said programmed rules
to a set of test programs to obtain an agent evaluation viewing
set; generating said program recommendations produced by said
program recommender by applying said agent profile to said test
programs; and comparing said agent evaluation viewing set with said
program recommendations.
2. The method of claim 1, wherein said viewing agent is selected to
have viewing habits that model the viewing habits of a type of
viewer.
3. The method of claim 1, wherein said agent viewing history
contains an indication of whether said viewing agent liked or
disliked each program in a set of training programs.
4. The method of claim 1, wherein said agent evaluation viewing set
contains an indication of whether said viewing agent liked or
disliked each program in a set of test programs.
5. The method of claim 1, wherein said viewing preferences
characterize programs by their attributes that are liked or
disliked by said viewing agent.
6. The method of claim 1, wherein said profiler is a component of
said program recommender.
7. The method of claim 1, wherein said inferred rules mimic said
programmed preferences of the viewing agent.
8. The method of claim 1, wherein said viewing preferences include
one or more random programs.
9. The method of claim 1, wherein said viewing preferences change
over time.
10. The method of claim 1, wherein said comparing step further
comprises the step of comparing calculated precision and recall
values to a predefined level of accuracy.
11. A method for validating program recommendations produced by a
program recommender using a viewing agent having programmed viewing
preferences, said method comprising the steps of: processing a
viewing history of said viewing agent using a profiler to generate
an agent profile, said agent profile containing a set of inferred
rules that characterize said programmed viewing preferences;
applying said programmed viewing preferences to a set of test
programs to obtain an agent evaluation viewing set; generating said
program recommendations produced by said program recommender by
applying said agent profile to said test programs; and comparing
said agent evaluation viewing set with said program
recommendations.
12. The method of claim 11, wherein said viewing agent is selected
to have viewing habits that model the viewing habits of a type of
viewer.
13. The method of claim 11, wherein said agent viewing history
contains an indication of whether said viewing agent liked or
disliked each program in a set of training programs.
14. The method of claim 11, wherein said agent evaluation viewing
set contains an indication of whether said viewing agent liked or
disliked each program in a set of test programs.
15. The method of claim 11, wherein said viewing preferences
characterize programs by their attributes that are liked or
disliked by said viewing agent.
16. The method of claim 11, wherein said profiler is a component of
said program recommender.
17. The method of claim 11, wherein said inferred rules mimic said
programmed preferences of the viewing agent.
18. The method of claim 11, wherein said viewing preferences
include one or more random programs.
19. The method of claim 11, wherein said viewing preferences change
over time.
20. The method of claim 11, wherein said comparing step further
comprises the step of comparing calculated precision and recall
values to a predefined level of accuracy.
21. A method for determining the required size of a viewing history
for a program recommender to provide a given level of accuracy,
said method comprising the steps of: generating a plurality of
viewer agents of varying complexity; generating viewing histories
of varying size for each of said viewing agents; determining the
precision for each viewing history size and viewing agent;
determining the recall for each viewing history size and viewing
agent; and determining a required size for said viewing history
such that said precision and recall values exceed a predefined
threshold.
22. The method of claim 21, wherein said precision is determined
from the true positives (TP) and false positives (FP) as follows: 4
Precision = TP CT = TP TP + FP
23. The method of claim 21, wherein said precision is determined
from the true positives (TP) and false negatives (FN) as follows: 5
Recall = TP RT = TP TP + FN
24. A method for determining the required size of a viewing history
for a program recommender to provide a given level of accuracy for
a user, said method comprising the steps of: generating a viewing
agent using one or more programmed rules that characterize viewing
preferences; generating a plurality of viewing histories of varying
sizes for said viewing agent; determining the precision for each
viewing history; determining the recall for each viewing history;
and determining a required size for said viewing history such that
said precision and recall values exceed a predefined threshold.
25. A method for determining the required size of a viewing history
for a program recommender to provide a given level of accuracy,
said method comprising the steps of: generating a plurality of
viewer agents with varying program preferences; generating viewing
histories of varying size for each of said viewing agents;
determining the precision for each viewing history size and viewing
agent; determining the recall for each viewing history size and
viewing agent; and determining a required size for said viewing
history such that said precision and recall values exceed a
predefined threshold.
26. A system for validating program recommendations produced by a
program recommender, comprising: a memory for storing computer
readable code; and a processor operatively coupled to said memory,
said processor configured to: generate a viewing agent using one or
more programmed rules that characterize viewing preferences; apply
said programmed rules to a set of training programs to obtain an
agent viewing history; process said agent viewing history with a
profiler to derive an agent profile containing a set of inferred
rules; apply said programmed rules to a set of test programs to
obtain an agent evaluation viewing set; generate said program
recommendations produced by said program recommender by applying
said agent profile to said test programs; and compare said agent
evaluation viewing set with said program recommendations.
27. A system for validating program recommendations produced by a
program recommender using a viewing agent having programmed viewing
preferences, comprising: a memory for storing computer readable
code; and a processor operatively coupled to said memory, said
processor configured to: process a viewing history of said viewing
agent using a profiler to generate an agent profile, said agent
profile containing a set of inferred rules that characterize said
programmed viewing preferences; apply said programmed viewing
preferences to a set of test programs to obtain an agent evaluation
viewing set; generate said program recommendations produced by said
program recommender by applying said agent profile to said test
programs; and compare said agent evaluation viewing set with said
program recommendations.
28. A system for determining the required size of a viewing history
for a program recommender to provide a given level of accuracy,
comprising: a memory for storing computer readable code; and a
processor operatively coupled to said memory, said processor
configured to: generate a plurality of viewer agents of varying
complexity; generate viewing histories of varying size for each of
said viewing agents; determine the precision for each viewing
history size and viewing agent; determine the recall for each
viewing history size and viewing agent; and determine a required
size for said viewing history such that said precision and recall
values exceed a predefined threshold.
29. A system for determining the required size of a viewing history
for a program recommender to provide a given level of accuracy for
a user, comprising: a memory for storing computer readable code;
and a processor operatively coupled to said memory, said processor
configured to: generate a viewing agent using one or more
programmed rules that characterize viewing preferences; generate a
plurality of viewing histories of varying sizes for said viewing
agent; determine the precision for each viewing history; determine
the recall for each viewing history; and determine a required size
for said viewing history such that said precision and recall values
exceed a predefined threshold.
30. A system for determining the required size of a viewing history
for a program recommender to provide a given level of accuracy,
comprising: a memory for storing computer readable code; and a
processor operatively coupled to said memory, said processor
configured to: generate a plurality of viewer agents with varying
program preferences; generate viewing histories of varying size for
each of said viewing agents; determine the precision for each
viewing history size and viewing agent; determine the recall for
each viewing history size and viewing agent; and determine a
required size for said viewing history such that said precision and
recall values exceed a predefined threshold.
Description
FIELD OF THE INVENTION
[0001] The present invention relates to methods and apparatus for
recommending television programming, and more particularly, to
techniques for evaluating television program recommenders.
BACKGROUND OF THE INVENTION
[0002] As the number of channels available to television viewers
has increased, along with the diversity of the programming content
available on such channels, it has become increasingly challenging
for television viewers to identify television programs of interest.
Historically, television viewers identified television programs of
interest by analyzing printed television program guides. Typically,
such printed television program guides contained grids listing the
available television programs by time and date, channel and title.
As the number of television programs has increased, it has become
increasingly difficult to effectively identify desirable television
programs using such printed guides.
[0003] More recently, television program guides have become
available in an electronic format, often referred to as electronic
program guides (EPGs). Like printed television program guides, EPGs
contain grids listing the available television programs by time and
date, channel and title. Some EPGs, however, allow television
viewers to sort or search the available television programs in
accordance with personalized preferences. In addition, EPGs allow
for on-screen presentation of the available television
programs.
[0004] While EPGs allow viewers to identify desirable programs more
efficiently than conventional printed guides, they suffer from a
number of limitations, which if overcome, could further enhance the
ability of viewers to identify desirable programs. For example,
many viewers have a particular preference towards, or bias against,
certain categories of programming, such as action-based programs or
sports programming. Thus, the viewer preferences can be applied to
the EPG to obtain a set of recommended programs that may be of
interest to a particular viewer.
[0005] Thus, a number of tools have been proposed or suggested for
recommending television programming. The Tivo.TM. system, for
example, commercially available from Tivo, Inc., of Sunnyvale,
Calif., allows viewers to rate shows using a "Thumbs Up and Thumbs
Down" feature and thereby indicate programs that the viewer likes
and dislikes, respectively. Thereafter, the TiVo receiver matches
the recorded viewer preferences with received program data, such as
an EPG, to make recommendations tailored to each viewer.
[0006] There is currently no way, however, to validate the
recommendations generated by such tools for recommending television
programming, short of testing the tools with human volunteers. In
addition, there is no way to determine when the recommendations
have achieved a given level of accuracy. A need therefore exists
for a method and apparatus for validating recommendations generated
by a television program recommender. A further need exists for a
method and apparatus for determining when the size of the viewing
history (record of shows watched/not watched) is sufficient to
provide a given level of accuracy.
SUMMARY OF THE INVENTION
[0007] Generally, a method and apparatus are disclosed for
evaluating the effectiveness of a television program recommender by
evaluating program recommendations generated by the television
program recommender for one or more programmed viewing agents. A
viewing agent is programmed with a set of rules that characterize
the viewing preferences of a modeled viewer. Viewing agents of
varying complexity and having varying program preferences can be
constructed by defining various rules that characterize program
attributes, such as genre, actors and program duration.
[0008] During a training phase of the present invention, a viewing
agent containing the programmed rules is applied to a set of
training programs from an electronic program guide (EPG) to
algorithmically obtain an agent viewing history, indicating whether
the viewing agent would have liked or disliked each training
program. The generated agent viewing history is then processed by
the profiler portion of the television program recommender being
evaluated. The profiler derives an agent profile containing a set
of inferred rules that attempt to mimic the programmed preferences
of the viewing agent. Thus, the profiler attempts to derive the
viewing habits of the viewing agent based on the set of programs
that the viewing agent liked or disliked.
[0009] During an evaluation phase of the present invention, the
programmed rules of the viewing agent are applied to test data from
an electronic program guide (EPG) to obtain an agent evaluation
viewing set. In parallel, the television program recommender
generates a set of program recommendations by applying the agent
profile generated during the training phase to the test data from
the electronic program guide (EPG). The present invention can then
compare the agent evaluation viewing set (generated from the
programmed rules) with a set of program recommendations produced by
the television program recommender being evaluated (generated from
the inferred rules). In this manner, the effectiveness of the
television program recommender can be evaluated.
[0010] According to another aspect of the invention, a method and
apparatus are disclosed for determining the required size of the
viewing history to provide a given level of accuracy. Generally,
the size of the required viewing history can be obtained by
utilizing viewing histories of varying sizes for the same viewing
agent. Thereafter, the size of the viewing history required to
exceed a predefined accuracy threshold can be obtained.
[0011] The viewing agents can be programmed to introduce one or
more random shows into the viewing history, allowing the television
program recommender validator to determine how the program
recommender processes such noise. In a further variation, the
viewing agents can be programmed to gradually change the programmed
viewer preferences over time, allowing the television program
recommender validator to determine how the program recommender
responds to such changes.
[0012] A more complete understanding of the present invention, as
well as further features and advantages of the present invention,
will be obtained by reference to the following detailed description
and drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0013] FIG. 1A illustrates a television program recommender
validator in accordance with a training phase of the present
invention;
[0014] FIG. 1B illustrates a television program recommender
validator in accordance with an evaluation phase of the present
invention;
[0015] FIG. 2 is a schematic block diagram of the television
program recommender validator of FIG. 1B;
[0016] FIG. 3 is a flow chart describing the validation process
(training phase) of FIG. 2 embodying principles of the present
invention;
[0017] FIG. 4 is a flow chart describing the validation process
(evaluation phase) of FIG. 2 embodying principles of the present
invention;
[0018] FIG. 5 is a flow chart describing the sensitivity analysis
process of FIG. 2 embodying principles of the present
invention;
[0019] FIG. 6A illustrates a confusion matrix that analyzes whether
a viewing agent would have liked a given show according to the
programmed preferences and whether the program recommender actually
recommended the show;
[0020] FIG. 6B illustrates a plot of the precision and recall
values of the training data as a function of the number of shows
viewed (size of viewing history); and
[0021] FIG. 6C illustrates a plot of the number of shows viewed
varying as a function of the complexity of the viewing agent.
DETAILED DESCRIPTION
[0022] FIGS. 1A and 1B illustrate a television program recommender
validator 200, discussed further below in conjunction with FIG. 2,
in accordance with a training phase and evaluation phase,
respectively, of the present invention. According to one aspect of
the present invention, the television program recommender validator
200 evaluates the effectiveness of a television program recommender
160 by evaluating program recommendations generated by the
television program recommender 160 for one or more programmed
viewing agents 120 (or algorithmic viewers). As used herein, a
viewing agent 120 is any programmed viewer whose viewing habits are
predetermined. Each viewing agent 120 is programmed with a set of
rules that categorize the viewing preferences of the modeled
viewer.
[0023] Viewing agents 120 of varying complexity and having varying
program preferences can be constructed in accordance with the
present invention by defining various rules that characterize
programs by their attributes. For example, a viewing agent 120 may
be programmed with a strong preference for programs that belong to
one or more particular genres, such as the "action" genre. In
further variations, viewing agents 120 may be programmed with a
preference for shows containing one or more identified actors and
having a given duration.
[0024] For example, if a viewing agent 120 is programmed with a
strong preference for programs that belong to the action genre, all
action shows should be identified by the program recommender 160.
The programs identified directly from the programmed rules of the
viewing agent 120 in accordance with the present invention can be
compared with the program recommendations 130 produced by the
television program recommender 160 utilizing inferred rules
contained in an agent profile 140. In this manner, the
effectiveness of the television program recommender 160 can be
evaluated.
[0025] According to another aspect of the invention, the television
program recommender validator 200 estimates the size of the viewing
history needed for the television program recommender 160 to
provide results with a predefined reliability. Generally, the size
of the required viewing history can be obtained by utilizing
viewing histories of varying sizes for the same viewing agent 120.
Thereafter, the size of the viewing history required to exceed a
predefined accuracy can be obtained.
[0026] The present invention can evaluate any television program
recommender 160, such as the television program recommenders 160
described in U.S. patent application Ser. No. ______, filed ______,
entitled "Method and Apparatus for Recommending Television
Programming Using Decision Trees," (Attorney Docket No. 700772) and
U.S. patent application Ser. No. ______, filed ______, entitled
"Bayesian TV Show Recommender," (Attorney Docket No. 700690), or
the Tivo.TM. system, commercially available from Tivo, Inc., of
Sunnyvale, Calif. It is noted that the profiler 130 is generally a
component of the television program recommender 160.
Training Phase
[0027] FIG. 1A illustrates a training phase of the present
invention. The present invention applies the viewing agent 120
containing the programmed rules to a set of training programs from
an electronic program guide (EPG) 110 to algorithmically obtain an
agent viewing history 125. The agent viewing history 125 indicates
whether the viewing agent 120 would have liked or disliked each of
the training programs 110.
[0028] As shown in FIG. 1A, the profiler portion 130 of the
television program recommender 160 (FIG. 1B) being evaluated then
processes the agent viewing history 125 to derive an agent profile
140 containing a set of inferred rules that mimic the programmed
preferences of the viewing agent 120. Thus, the profiler 130
attempts to derive the viewing habits of the viewing agent 120
based on the set of programs that the viewing agent 120 liked or
disliked. Generally, more accurate program recommenders 160 will
generate sets of inferred rules that more closely resemble the
programmed rules of the viewing agent 120.
Evaluation Phase
[0029] FIG. 1B illustrates an evaluation phase of the present
invention. Generally, the evaluation phase applies the programmed
rules of the viewing agent 120 to a set of test data 150 from an
electronic program guide (EPG) to obtain an agent evaluation
viewing set 170. The television program recommender validator 200
then compares the agent evaluation viewing set 170 with a set of
program recommendations 180 produced by the television program
recommender 160 being evaluated. The television program recommender
160 generates the program recommendations 180 by applying the agent
profile 140 generated during the training phase to the test data
150 from an electronic program guide (EPG).
[0030] FIG. 2 is a schematic block diagram of the television
program recommender validator 200. The television program
recommender validator 200 may be embodied as any computing device,
such as a personal computer or workstation, that contains a
processor 210, such as a central processing unit (CPU), and memory
220, such as RAM and ROM. As shown in FIG. 2, the program
recommender validator 200 contains a validation process (training
phase) 300, discussed further below in conjunction with FIG. 3, a
validation process (evaluation phase) 400, discussed further below
in conjunction with FIG. 4, and a sensitivity analysis process 500,
discussed further below in conjunction with FIG. 5.
[0031] Generally, the validation process (training phase) 300
implements the training phase, discussed above, to generate the
agent profile 140. The validation process (evaluation phase) 400
implements the evaluation phase, discussed above, to compare the
agent evaluation viewing set 170 generated directly from the
programmed rules for the viewing agent 120 with a set of program
recommendations 180 produced by the television program recommender
160 being evaluated. The sensitivity analysis process 500 estimates
the size of the viewing history needed for the television program
recommender 160 to provide results with a predefined
reliability.
[0032] FIG. 3 is a flow chart describing the validation process
(training phase) 300 embodying principles of the present invention.
The validation process (training phase) 300 implements the training
phase, discussed above, to generate the agent profile 140. As shown
in FIG. 3, the validation process (training phase) 300 initially
constructs a viewing agent 120 by defining various rules that
characterize programs by their attributes as being liked or
disliked during step 310. The test data from the electronic program
guide (EPG) 110 is then applied to the programmed rules of the
viewing agent 120 during step 320 to generate the agent viewing
history 125.
[0033] Finally, the profiler 130 is applied to the agent viewing
history 125 during step 330 to identify the inferred rules that
characterize the agent viewing history 125. Program control then
terminates.
[0034] FIG. 4 is a flow chart describing the validation process
(evaluation phase) 400 embodying principles of the present
invention. The validation process (evaluation phase) 400 implements
the evaluation phase, discussed above, to compare the agent
evaluation viewing set 170 generated directly from the programmed
rules for the viewing agent 120 with a set of program
recommendations 180 produced by the television program recommender
160 being evaluated.
[0035] As shown in FIG. 4, the validation process (evaluation
phase) 400 initially applies the test data from the EPG 150 to the
programmed rules of the viewing agent 120 during step 410 to
generate the agent evaluation viewing set 170 indicating programs
that the programmed viewing agent 120 would watch. Thereafter, the
validation process (evaluation phase) 400 applies the test data
from the EPG 150 to the TV recommender 160, using the agent profile
140 generated during the training phase, to generate a set of
program recommendations 180 during step 420.
[0036] During step 430, the validation process (evaluation phase)
400 then compares the set of programs 170 identified during step
410 from the program rules with the set of program recommendations
180 identified by the program recommender 160 during step 420 from
the inferred rules.
[0037] Finally, the effectiveness of the television program
recommender 160 can be evaluated during step 440, based on the
comparison. For example, the effectiveness of the television
program recommender 160 can be evaluated using well-known pattern
recognition techniques, such as the mean square error, precision
and recall (discussed below) or receiver operator characteristic
curves. Program control then terminates.
[0038] FIG. 5 is a flow chart describing the sensitivity analysis
process 500 embodying principles of the present invention.
Generally, the sensitivity analysis process 500 estimates the size
of the viewing history needed for the television program
recommender 160 to provide results with a predefined reliability or
accuracy.
[0039] As shown in FIG. 5, the sensitivity analysis process 300
initially generates viewer agents of varying complexity during step
510. For each viewing agent 120, the sensitivity analysis process
500 then generates viewing histories of varying size during step
520. The validation process (evaluation) 400 is then executed
during step 525 for each viewing history size and viewing agent 120
to compare the agent evaluation viewing set 170 generated directly
from the programmed rules for the viewing agent 120 with a set of
program recommendations 180 produced by the television program
recommender 160 being evaluated.
[0040] Thereafter, for each viewing history size and viewing agent
120, the sensitivity analysis process 500 computes the precision of
the recommendations during step 530 as follows: 1 Precision = TP CT
= TP TP + FP ,
[0041] where TP indicates the true positives, FP indicates the
false positives, and the column total, CT, is equal to the TP plus
the FP.
[0042] In addition, for each viewing history size and viewing agent
120, the sensitivity analysis process 500 computes the recall of
the training data during step 540 as follows: 2 Recall = TP RT = TP
TP + FN ,
[0043] where FN indicates the false negatives and the row total,
RT, is equal to TP plus FN.
[0044] The concepts and calculations for precision and recall are
discussed further below in conjunction with FIG. 6A. The required
size of the viewing history for the precision and recall to exceed
a predefined threshold is then determined during step 550 and the
size of viewing history is plotted as a function of the complexity
of the viewing agent 120 during step 560. Program control then
terminates.
Sensitivity Analysis
[0045] FIG. 6A illustrates a well-known confusion matrix 600 that
analyzes whether the viewing agent 120 would have liked a given
show according to the programmed preferences (satisfied rules in
profile), and whether the program recommender 160 actually
recommended the show. The confusion matrix 600 indicates the true
positives (TP), false positives (FP), true negatives (TN) and false
negatives (FN). The confusion matrix 600 also indicates the row
total (RT=TP+FN) and the column total (CT=TP+FP).
[0046] In accordance with well-known pattern recognition
techniques, such as those described in Boreczky and Rowe,
"Comparison of Video Shot Boundary Detection Techniques," SPIE,
Vol. 2670, 170-79 (1996), incorporated by reference herein, the
precision and recall of the training set can be evaluated according
to the following expressions: 3 Precision = TP CT = TP TP + FP
Recall = TP RT = TP TP + FN
[0047] FIG. 6B illustrates a plot 650 of the precision and recall
values as a function of the number of shows viewed (size of the
viewing history). The precision and recall values vary between
values of zero and one, and generally increase as the size of the
viewing history increases. To obtain a desired degree of accuracy
in accordance with the present invention, the size of the viewing
history can be adjusted such that both the precision and recall
values exceed a predefined threshold.
[0048] FIG. 6C illustrates a plot 680 of the size of the viewing
history needed to achieve a given level of accuracy or reliability,
as a function of the complexity of the viewing agent 120. Clearly,
the number of shows that must be included in the viewing history
must be increased to achieve a given accuracy level as the
complexity of the viewing agent 120 increases.
[0049] In one variation, the viewing agents 120 can be programmed
to introduce one or more random shows into the viewing history 125
and thereby determine how the program recommender 160 processes
such noise. In a further variation, the viewing agents 120 can be
programmed to gradually change the programmed viewer preferences
over time, and thereby determine how the program recommender 160
responds to such changes.
[0050] It is to be understood that the embodiments and variations
shown and described herein are merely illustrative of the
principles of this invention and that various modifications may be
implemented by those skilled in the art without departing from the
scope and spirit of the invention.
* * * * *