U.S. patent application number 12/606400 was filed with the patent office on 2011-04-28 for index-based technique friendly ctr prediction and advertisement selection.
This patent application is currently assigned to Yahoo! Inc.. Invention is credited to Deepak K. Agarwal, Marcus Fontoura, Joaquin Arturo Delgado Rodriguez.
Application Number | 20110099059 12/606400 |
Document ID | / |
Family ID | 43899188 |
Filed Date | 2011-04-28 |
United States Patent
Application |
20110099059 |
Kind Code |
A1 |
Agarwal; Deepak K. ; et
al. |
April 28, 2011 |
INDEX-BASED TECHNIQUE FRIENDLY CTR PREDICTION AND ADVERTISEMENT
SELECTION
Abstract
Methods and systems are provided for click through rate
prediction and advertisement selection in online advertising.
Methods are provided in which output information from a
feature-based machine learning model is utilized. The output
information includes predicted click through rate information. The
output information is used to form a matrix. The matrix is modeled
using a latent variable model. Machine learning techniques can be
used in determining values for unfilled cells of one or more model
matrices. The latent variable model can be used in determining
predicted click through rate information, and in advertisement
selection in connection with serving opportunities.
Inventors: |
Agarwal; Deepak K.;
(Sunnyvale, CA) ; Rodriguez; Joaquin Arturo Delgado;
(Santa Clara, CA) ; Fontoura; Marcus; (Mountain
View, CA) |
Assignee: |
Yahoo! Inc.
Sunnyvale
CA
|
Family ID: |
43899188 |
Appl. No.: |
12/606400 |
Filed: |
October 27, 2009 |
Current U.S.
Class: |
705/14.43 ;
705/14.52; 705/14.54; 706/12; 707/E17.017; 707/E17.044 |
Current CPC
Class: |
G06N 20/00 20190101;
G06Q 30/0256 20130101; G06Q 30/0202 20130101; G06Q 30/0244
20130101; G06Q 30/02 20130101; G06Q 30/0204 20130101; G06Q 30/0254
20130101 |
Class at
Publication: |
705/14.43 ;
705/14.52; 705/14.54; 706/12; 707/E17.044; 707/E17.017 |
International
Class: |
G06Q 30/00 20060101
G06Q030/00; G06Q 10/00 20060101 G06Q010/00; G06F 17/30 20060101
G06F017/30; G06F 15/18 20060101 G06F015/18 |
Claims
1. A method comprising: using one or more computers, obtaining a
matrix of information, the matrix of information comprising output
information from a machine learning model; wherein input
information to the machine learning model comprises advertisement
information, serving opportunity information, and historical click
through rate information; and wherein the output information
comprises predicted click through rate information relating to
predicted click through rates associated with each of a plurality
of advertising instances, each of the plurality of advertising
instances comprising an advertisement and a serving opportunity;
using one or more computers, modeling the matrix of information
using a latent variable model, comprising approximately factorizing
the matrix of information into two submatrices, the two submatrices
being associated with each other by one or more latent variables;
and using one or more computers, using the latent variable model,
determining a set of information, the set of information comprising
predicted click through rate information for each of one or more
advertising instances, each of the one or more advertising
instances comprising an advertisement and a serving
opportunity.
2. The method of claim 1, comprising: using the set of information
in determining a top set of advertisements in connection with a
particular serving opportunity, and serving a subset of the top set
of advertisements in connection with the particular serving
opportunity.
3. The method of claim 1, comprising using a machine learning
technique in approximating values for a portion of the cells of the
matrix of information.
4. The method of claim 1, comprising using a feature-based
logistical regression technique.
5. The method of claim 1, comprising: modeling the matrix of
information using a latent variable model, wherein the two
submatrices comprise a query-associated matrix and an
advertisement-associated matrix; and for at least a particular
query and a particular advertisement, performing an inner product
matrix multiplication operation on the two submatrices in
determining an associated predicted click through rate, wherein the
operation is dependent upon features of the particular query and
the particular advertisement.
6. The method of claim 1, comprising determining the set of
information, wherein each of the one or more advertising instances
comprises an advertisement and a serving opportunity, and wherein
the serving opportunity comprises a keyword query.
7. The method of claim 1, comprising determining the set of
information, wherein each of the one or more advertising instances
comprises an advertisement and a serving opportunity, and wherein
the serving opportunity comprises a Web page.
8. The method of claim 1, comprising determining the set of
information, wherein the predicted click through rate information
for each of the one or more advertising instances comprises a score
that is associated with a predicted click through rate.
9. The method of claim 1, comprising determining the set of
information, wherein the predicted click through rate information
comprises estimated predicted click through rate information.
10. The method of claim 1, comprising obtaining the matrix of
information, wherein input to the machine learning model comprises
user information.
11. The method of claim 1, comprising determining the set of
information, wherein each of the one or more advertising instances
comprises an advertisement and a serving opportunity, and wherein
the serving opportunity comprises user information associated with
a user associated with the serving opportunity.
12. The method of claim 1, comprising modeling the matrix of
information using a latent variable model, wherein the one or more
latent variables are content-based.
13. The method of claim 1, comprising modeling the matrix of
information using a latent variable model, wherein the one or more
latent variables are topical.
14. A system comprising: one or more server computers coupled to
the Internet, and one or more databases coupled to the one or more
servers; wherein the one or more server computers are for:
obtaining a matrix of information, the matrix of information
comprising output information from a machine learning model;
wherein input information to the machine learning model includes
advertisement information, serving opportunity information, and
historical click through rate information; and wherein the output
information comprises predicted click through rate information
relating to predicted click through rates associated with each of a
plurality of advertising instances, each of the plurality of
advertising instances comprising an advertisement and a serving
opportunity; modeling the matrix of information using a latent
variable model, comprising approximately factorizing the matrix of
information into two submatrices, the two submatrices being
associated with each other by one or more latent variables; and
using the latent variable model, determining, and storing in at
least one of the one or more databases, a set of information, the
set of information comprising predicted click through rate
information for each of one or more advertising instances, each of
the one or more advertising instances comprising an advertisement
and a serving opportunity.
15. The system of claim 14, comprising: using the determined set of
information in determining a top set of advertisements in
connection with a particular serving opportunity, and serving a
subset of the top set of advertisements in connection with the
particular serving opportunity.
16. The system of claim 14, comprising using a machine learning
technique in approximating values for a portion of the cells of the
matrix of information.
17. The system of claim 14, comprising: modeling the matrix of
information using a latent variable model, wherein the two
submatrices comprise a query-associated matrix and an
advertisement-associated matrix; and for at least a particular
query and a particular advertisement, performing an inner product
matrix multiplication operation on the two submatrices in
determining an associated predicted click through rate, wherein the
operation is dependent upon features of the particular query and
the particular advertisement.
18. The system of claim 14, wherein each of the one or more
advertising instances comprises an advertisement and a serving
opportunity, and wherein the serving opportunity comprises a
keyword query.
19. The system of claim 14, wherein each of the one or more
advertising instances comprises an advertisement and a serving
opportunity, and wherein the serving opportunity comprises a Web
page.
20. A computer readable medium or media containing instructions for
executing a method, the method comprising: using one or more
computers, obtaining a matrix of information, the matrix of
information comprising output information from a machine learning
model; wherein input information to the machine learning model
includes advertisement information, serving opportunity
information, and historical click through rate information; and
wherein the output information comprises predicted click through
rate information relating to predicted click through rates
associated with each of a plurality of advertising instances, each
of the plurality of advertising instances comprising an
advertisement and a serving opportunity; using one or more
computers, modeling the matrix of information using a latent
variable model, comprising approximately factorizing the matrix of
information into two submatrices, the two submatrices being
associated with each other by one or more latent variables; using
one or more computers, using the latent variable model, determining
a set of information, the set of information comprising predicted
click through rate information for each of one or more advertising
instances, each of the one or more advertising instances comprising
an advertisement and a serving opportunity; and using one or more
computers, using the set of information in determining a top set of
advertisements in connection with a particular serving opportunity.
Description
BACKGROUND
[0001] Modern large-scale information retrieval techniques often
use index-based techniques. Indexing and indexing-based techniques
can allow offline processing that can greatly speed information
retrieval. This can be of great value in applications such as
online advertising or search, where results or advertisements may
need to be determined in just a small fraction of a second. In some
techniques, particular elements or items of a match, such as
content of a Web page or content of advertising, are indexed.
Matrices can be formed using indexed information. Indexing
techniques can then be applied, such as techniques that utilize
inner product matrix multiplication to compute or help compute, for
example, associations, matches, strengths of associations or
matches, or corresponding scores. Such techniques may be used, for
example, in determining a top set of advertisements that match a
particular serving opportunity, such as advertisements, including
creatives, that best match a keyword query, Web site content, using
or other targeting information, etc.
[0002] Performance parameters associated with advertising and other
content, such as click through rate, are critical for many uses and
applications. Furthermore, predicted click through rates for
particular situations that arise in real time can be critical for
optimal matching. For example, in advertising applications,
predicted performance of an advertisement in a given context, such
as predicted click through rate, is often used in determining a
good or optimal set of advertisements to serve in relation to a
given serving opportunity. For instance, in sponsored search, the
rank of an advertisement may be determined by an associated bid
amount multiplied by an associated determined predicted click
through rate (perhaps in combination with other parameters or
factors, etc).
[0003] Machine learning techniques can be of great value in
information retrieval, including determining top matches. For
example, in advertising, machine learning techniques are often used
in determining a top set of matches between advertisements and a
serving opportunity. For example, machine learning can be used to
determine strength of matching between new sets of items, based on
features, such as content-based features, relating to such items.
Training sets of collected historical information can be used to
train a machine learning model, which model can then be applied to
new situations and sets or combinations of items. For example,
machine learning techniques can be used to determine predicted
click through rates associated with matches. However, use of
existing machine learning models on new situations can be
computationally intensive and time-consuming.
[0004] There is a need for faster and more efficient methods and
systems for determining predicted click through rate and top sets
of matched items such as advertisements.
SUMMARY
[0005] Some embodiments of the invention provide methods and
systems for click through rate prediction and advertisement
selection in online advertising. Methods are provided in which
output information from a feature-based machine learning model is
utilized. The output information includes predicted click through
rate information. A matrix of information is formed using the
output information. The matrix is modeled using a latent variable
model. Machine learning techniques, which can include feature-based
regression techniques, can be used in determining values for
unfilled cells of one or more matrices of the model. The latent
variable model can be used in determining predicted click through
rate information, and in advertisement selection, in connection
with serving opportunities.
BRIEF DESCRIPTION OF THE DRAWINGS
[0006] FIG. 1 is a distributed computer system according to one
embodiment of the invention;
[0007] FIG. 2 is a flow diagram of a method according to one
embodiment of the invention;
[0008] FIG. 3 is a flow diagram of a method according to one
embodiment of the invention; and
[0009] FIG. 4 is a conceptual block diagram illustrating one
embodiment of the invention.
[0010] While the invention is described with reference to the above
drawings, the drawings are intended to be illustrative, and the
invention contemplates other embodiments within the spirit of the
invention.
DETAILED DESCRIPTION
[0011] Herein, embodiments of the invention are described primarily
with regard to click through rate prediction. However, the
invention also contemplates other performance measures that may be
associated with advertising, such as conversion rates, etc. Also,
herein, the term "click though rate information" broadly includes
not only actual predicted click through rates, but also information
that can be associated with predicted click through rates, such as
scores that correlate with click through rates. Additionally,
"click through rates", as the term is used herein, includes
approximated click through rates or ranges of click through
rates.
[0012] Herein, various forms of advertising are contemplated,
including video and audio advertising, mobile advertising, etc.
Furthermore, herein, the term "advertisements" includes creatives
associated with advertisements.
[0013] Although latent variable models according to embodiments of
the invention are generally described utilizing approximate
factorization into two submatrices, embodiments are contemplated
that utilize more than two submatrices.
[0014] Online advertising often involves selecting a top set of
advertisements from a large inventory of advertisements, to serve
in association with a particular serving opportunity. Often a set
of advertisements must be selected extremely quickly, such as in a
matter of milliseconds, based on parameters associated with the
serving opportunity. For example, in a sponsored search setting,
one or more advertisements may need to be selected and served
immediately following a user-entered keyword search query. In
non-sponsored search advertising, one or more advertisements may
need to be selected based on parameters including, for example, a
Web page being visited by a user when the advertisements are to be
served. Parameters can also include features, including
content-based features, associated with queries, Web pages,
etc.
[0015] Of course, many additional types of parameters, including
targeting parameters, can be associated with a serving opportunity,
in both sponsored search and non-sponsored search settings. For
example, user-associated parameters can include information
relating to the particular user to be served the advertisements,
the location of the user (geotargeting), past online behavior of
the user, demographics or groups to which the user belongs, etc.
Many other types of parameters, including targeting parameters, may
also be considered, as known in the art.
[0016] Prediction of click through rate associated with serving of
advertisements is critical for many purposes, since it is often
associated with revenue. For example, in some formulations used in
sponsored search, advertisements are selected or ranked for serving
based on a formula which may be, or include factors associated
with, a predicted click through rate multiplied by an associated
advertiser bid amount.
[0017] While advertisements may need to be selected in a matter of
milliseconds, optimal selection can be very computationally
complex, which presents substantial challenges. Offline analysis
and calculations, where such can be applied, are very useful in
coping with these challenges and providing scalability.
[0018] Many parameters and features associated with advertising
relate to content, including words or word-based content features.
For example, words or word-based features in a query or on a Web
page may be compared with words or word-based features in an
advertisement, potentially including weights, measures of
prominence or importance of words or topics, etc. Moreover, more
sophisticated word-based analysis can be performed and utilized,
such as topic determination, determinations of importance of
particular words or topics, weighting of words or topics, etc.
[0019] To facilitate and speed online advertisement selection,
indices and index-based techniques can be used. For example,
indices can be generated of features, such as content-based and
word-based features, associated with queries, Web pages, etc.
Furthermore, historical click through rate information can be
obtained associated with particular serving
opportunity-advertisement instances. This information can be used
as input and training information to a feature-based machine
learning model to output predicted click through rates associated
with new or unobserved instances and combinations.
[0020] Output from such a machine learning model includes click
through rate information. For example, the output can include click
through rate-associated scores, associated with serving
opportunity-advertisement instances, and in association with
features, such as content-based and word-based features, of
advertisements as well as elements of serving opportunities, such
as a query or Web page.
[0021] It would be advantageous to utilize, in an efficient and
scalable way, the output from such a machine learning model,
including predicted click through rate information, in predicting
click through rates associated with new or unobserved serving
opportunity-advertisement instances or combinations.
[0022] In some embodiments of the invention, output from such a
machine learning model is put into a matrix of information. As just
one simple example, the matrix can include query-associated rows
and advertisement-associated columns, including content
feature-based information as well as predicted click through rate
information in association with features. This matrix can then be
approximately factorized using a latent variable (or factor) model,
such as a probabilistic latent variable model. For example, the
matrix can be approximately factorized into a query-associated
submatrix and an advertisement-associated submatrix. Another
example could include one or more other serving
opportunity-associated submatrices (such as a Web page associated
submatrix) and one or more advertisement-associated
submatrices.
[0023] Since historical advertising and click through information
are often sparse, the output as initially formed into the matrix of
information may have many cells without values. In some embodiments
of the invention, machine learning techniques are utilized to
predict, determine, or estimate values for unfilled cells in the
initially formed matrix. In other embodiments, machine learning
techniques are utilized in or after approximate factorization of
the matrix, to determine values for and fill in initially unfilled
cells in the submatrices.
[0024] Latent variable models generally can include modeling a
many-dimensional initial matrix as an approximate factorization, or
approximate decomposition, into two or more fewer-dimensional
matrices. The latent, or unobserved, variable or variables may be a
dimension or dimensions in each of the two matrices. The latent
variable or variables may not be an explicit dimension or
dimensions of the initial matrix, but the initial matrix may
contain information which may implicitly, or by inference or other
manipulation or determination, allow information to be obtained
regarding the latent variable or variables.
[0025] Use of a latent variable model can allow simplification of
an initial many-dimensional matrix of information, such as a matrix
including serving opportunity information, advertisement
information, and predicted click through rate information, into an
approximate factorization of two or more lower-dimensional
matrices, each having one or more latent, or unobserved, variables
(or factors) as one or more dimensions. In some embodiments of the
invention, submatrices are associated by one or more such latent
variables, such as one or more content-based or topical latent
variables.
[0026] For a new serving opportunity-advertisement instance,
features of the particular serving opportunity-advertisement
instance can be utilized in performing an appropriate inner product
matrix multiplication with regard to the two submatrices, to arrive
at a predicted click through rate (or a score relating to a
predicted click through rate) associated with the instance based on
features determined or known to be associated with the
instance.
[0027] As such, in embodiments of the invention, output of a
machine learning model, which includes not only serving opportunity
and advertisement information, but also predicted click through
rate information, is put into a matrix format and then modeled
using a latent variable model. Values can be determined for
unfilled matrix cells using a machine learning technique.
[0028] In some embodiments, a regression-based machine learning
technique is utilized to determine and fill in unfilled matrix cell
values. For example, in some embodiments, a feature-based
logistical regression model is utilized. The feature-based
regression model can be used to predict or estimate values for
latent variables (or factors) associated with the latent variable
model utilized to model the matrix. These values can then be used
in determining associated predicted click through rate information,
which in turn can be used in tilling in the unfilled cells.
Regression-based models, including regression-based latent variable
models, can provide an effective way to manage sparse data
situations, and both cold and warm start scenarios.
[0029] Some embodiments of the invention include determining a top
set of advertisements for serving, or for potential serving, in
connection with a particular serving opportunity. This can be
described as determining or retrieving the "top k" advertisements
in connection with the serving opportunity.
[0030] In some embodiments, in connection with a particular serving
opportunity, using the latent variable model as described above,
predicted click through rate information (such as scores that
correlate with predicted click through rates) is generated for many
potentially servable advertisements in inventory. Using this
information, a top k set of advertisements can be selected. For
example, in some embodiments, advertisements may be considered,
selected or ranked in connection with a serving opportunity based
on a formula such as predicted click through rate multiplied by bid
amount, applied for each of many advertisements. The predicted
click through rates can be supplied using a latent variable model
technique according to embodiments of the invention.
[0031] In some embodiments, once a top k set of advertisements is
determined in this way, additional machine learning techniques can
be used to further refine or re-rank this set, before determining
an actual set of advertisements to serve.
[0032] As such, some embodiments of the invention include
application of a latent variable modeling technique or algorithm to
model output from a machine learning model, where the output
includes predicted click through rate information. This can be used
in providing a scalable algorithm for rapid determination of
predicted click through rates for serving opportunity-advertisement
instances. Such determined information can in turn be utilized, for
example, in determining a top set of advertisements for serving in
connection with a particular serving opportunity. In some
embodiments, a feature-based regression model can be utilized to
till in unfilled matrix cells in connection with the latent
variable modeling.
[0033] FIG. 1 is a distributed computer system 100 according to one
embodiment of the invention. The system 100 includes user computers
104, advertiser computers 106 and server computers 108, all coupled
or couplable to the Internet 102. Although the Internet 102 is
depicted, the invention contemplates other embodiments in which the
Internet 102 is not included, as well as embodiments in which other
networks are included in addition to the Internet 102, including
one or more wireless networks, WANs, LANs, telephone, cell phone,
or other data networks, etc. The invention further contemplates
embodiments in which user computers or other computers may be or
include wireless, portable, or handheld devices such as cell
phones, PDAs, etc.
[0034] Each of the one or more computers 104, 106, 108 may be
distributed, and can include various hardware, software,
applications, programs and tools. Depicted computers may also
include a hard drive, monitor, keyboard, pointing or selecting
device, etc. The computers may use an operating system such as
Windows by Microsoft, etc. Each computer may include a central
processing unit (CPU), a data storage device, and various amounts
of memory including RAM and ROM. Depicted computers may also
include various programming, applications, and software to enable
searching, display or presentation of search results, and
advertising, such as banner or graphical advertising as well as
keyword searching and advertising in a sponsored search
context.
[0035] As depicted, each of the server computers 108 includes one
or more CPUs 110 and a data storage device 112. The data storage
device 112 includes one or more databases 116 and a CTR prediction
and advertisement selection program 114. The one or more databases
116 may be coupled to the one or more server computers 108, which
may include being part of the one or more server computers 108.
[0036] The CTR prediction and advertisement management program 114
is intended to broadly include all programming, applications,
software and other tools necessary to implement or facilitate
methods and systems according to embodiments of the invention. The
elements of the program 114 may exist on uric computer or device,
or may be distributed among multiple computers or devices.
[0037] FIG. 2 is a flow diagram of a method 200 or algorithm
according to one embodiment of the invention. At step 202, using
one or more computers, a matrix of information is obtained, the
matrix of information including output information from a machine
learning model. Input information to the machine learning model
includes advertisement information, serving opportunity
information, and historical click through rate information. The
output information comprises predicted click through rate
information relating to predicted click through rates associated
with each of multiple advertising instances, each of the multiple
advertising instances including an advertisement and a serving
opportunity.
[0038] At step 204, using one or more computers, the matrix of
information is modeled using a latent variable model, including
approximately factorizing the matrix of information into two
submatrices, the two submatrices being associated with each other
by one or more latent variables.
[0039] At step 206, using one or more computers, using the latent
variable model, a set of information is determined, the set of
information including predicted click through rate information for
each of one or more advertising instances, each of the one or more
advertising instances including an advertisement and a serving
opportunity.
[0040] FIG. 3 is a flow diagram of a method 300 according to one
embodiment of the invention. Steps 302, 304 and 306 are similar to
steps 202, 204 and 206 as depicted in FIG. 2.
[0041] At step 308, using one or more computers, the set of
information is used in determining a top set of advertisements in
connection with a particular serving opportunity.
[0042] FIG. 4 is a conceptual block diagram 400 illustrating one
embodiment of the invention. As depicted, historical advertising
and click through rate information 402 is stored in a database
404.
[0043] The information 402 is among input information for a machine
learning model 408. Output information 410 from the machine
learning model includes predicted click though rate information
associated with multiple instances, each instance including an
advertisement and a serving opportunity.
[0044] The output information 410 is organized as a matrix 412. The
matrix may include many cells without values. In some embodiments,
values for such unfilled cells are determined or approximated using
a machine learning technique. In some embodiments, a
regression-based method is utilized.
[0045] Using a latent variable model technique, the matrix 412 is
approximately factorized 416 into to two submatrices. The
submatrices include a query-associated matrix 418 and an
advertisement-associated matrix 420.
[0046] In some embodiments, values for such unfilled cells are
determined or approximated using a machine learning technique. In
some embodiments, a regression-based method is utilized.
[0047] For a particular instance, including a query and an
advertisement, based on determined associated features, appropriate
inner product matrix multiplication 422 of the two submatrices 418,
420 is performed.
[0048] In some embodiments, the matrix multiplication 422 results
in a score 424 associated with the instance, which score 424
correlates with a predicted click through rate 426 associated with
the instance.
[0049] At step 428, a predicted click through rate is determined
for multiple advertisements.
[0050] At step 430, a top k set of advertisements is determined
based at least in part on determined predicted click through rates
associated with advertisements.
[0051] In some embodiments, in performing inner product matrix
multiplication, a Weak And, or "WAND", technique may be utilized.
The WAND technique can reduce the computational complexity and time
that would be required by use of a "brute force", or unsimplified,
method. WAND techniques that can be utilized in embodiments of the
invention are described in the following published article, which
is hereby incorporated herein by reference in its entirety: Andrei
Z. Broder, David Carmel, Michael Herscovici, Aya Soffer, and Jason
Zien. Efficient query evaluation using a two-level retrieval
process. In CIKM'03, Nov. 3-8, 2003.
[0052] In some embodiments, at runtime, in connection with a
particular query (and perhaps other serving opportunity elements
such as an associated publisher Web page, user, etc.), predicted
click through rate information is determined with regard to many
advertisements. The predicted click through rate information can
then be used in determining a top set of advertisements for serving
in connection with the associated serving opportunity.
[0053] For example, in some embodiments, further machine learning
techniques may be used to re-rank, in a more granular or further
refined manner, the determined top set of advertisements. This
re-ranked set, or a sub-set thereof, may be utilized to serve in
connection with the serving opportunity. As such, the latent
variable model technique, as described, can be used in top k
retrieval of advertisements for serving in connection with a
serving opportunity.
[0054] The foregoing description is intended merely to be
illustrative, and other embodiments are contemplated within the
spirit of the invention.
* * * * *