U.S. patent application number 16/508600 was filed with the patent office on 2020-03-05 for system and method for recommending a grammar for a message campaign used by a message optimization system.
The applicant listed for this patent is Persado Intellectual Property Limited. Invention is credited to Assaf Baciu, Rui Miguel Forte, Guy Stephane Krief, Avishalom Shalit.
Application Number | 20200074496 16/508600 |
Document ID | / |
Family ID | 48579463 |
Filed Date | 2020-03-05 |
![](/patent/app/20200074496/US20200074496A1-20200305-D00000.png)
![](/patent/app/20200074496/US20200074496A1-20200305-D00001.png)
![](/patent/app/20200074496/US20200074496A1-20200305-D00002.png)
![](/patent/app/20200074496/US20200074496A1-20200305-D00003.png)
![](/patent/app/20200074496/US20200074496A1-20200305-D00004.png)
![](/patent/app/20200074496/US20200074496A1-20200305-D00005.png)
![](/patent/app/20200074496/US20200074496A1-20200305-D00006.png)
United States Patent
Application |
20200074496 |
Kind Code |
A1 |
Forte; Rui Miguel ; et
al. |
March 5, 2020 |
SYSTEM AND METHOD FOR RECOMMENDING A GRAMMAR FOR A MESSAGE CAMPAIGN
USED BY A MESSAGE OPTIMIZATION SYSTEM
Abstract
A system and method is provided for recommending a grammar for a
message campaign used by a message optimization system. A user
specifies parameters for a new campaign, from which a set of
statistical design budgets is calculated. The user selects a
grammar structure, recommended based on the statistical design
budgets, for the campaign. The n-most relevant past campaigns are
identified. Semantic tags, associated with each previously used
value from the n-most relevant past campaigns and each of a
plurality of untested values, are identified and ranked based on
past performance. The previously used values are ordered by ranked
tag group and then within each tag group, while the untested values
are ordered by ranked tag group and then randomly within the tag
group. Recommended values are selected from the ranked list of
previously used values and untested values depending on the degree
of exploration/conservatism indicated by the user.
Inventors: |
Forte; Rui Miguel; (Athens,
GR) ; Shalit; Avishalom; (Orpington, GB) ;
Krief; Guy Stephane; (Papagou, GR) ; Baciu;
Assaf; (London, GB) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Persado Intellectual Property Limited |
London |
|
GB |
|
|
Family ID: |
48579463 |
Appl. No.: |
16/508600 |
Filed: |
July 11, 2019 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13474695 |
May 17, 2012 |
10395270 |
|
|
16508600 |
|
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06Q 30/0269 20130101;
G06Q 30/0261 20130101; G06Q 30/0241 20130101; G06Q 30/0276
20130101 |
International
Class: |
G06Q 30/02 20060101
G06Q030/02 |
Claims
1. A method performed by a computer system for recommending a
grammar for a message campaign used by a message optimization
system, the method comprising: providing a user interface that
enables a campaign manager to specify one or more parameters for a
new campaign; recommending at least one grammar structure from one
or more past campaigns that are based on the campaign parameters or
from a default grammar in the event that none of the past campaigns
has a grammar that complies with the campaign parameters, the
grammar structure specifying a plurality of message component
types; providing a user interface that enables a campaign manager
to select one of the recommended grammar structures for the new
campaign; for each message component type in the selected grammar
structure, generating a ranked list of previously-used values for
the component type in the one or more past campaigns, wherein the
previously-used values are each associated with a semantic tag and
generating the ranked list comprises: identifying the semantic tags
associated with the previously-used values in the one or more past
campaigns, wherein each semantic tag identifies the semantic
meaning of the associated value, creating a list of the
previously-used values in the one or more past campaigns grouped by
semantic tag, ranking groups of semantic tags based on performance
in the one or more past campaigns of the previously-used values
within a tag group versus other tag groups, and ordering the
previously-used values first by their ranked tag group and second,
within each tag group, by the number of times an individual value
has been identified as the winning value in the one or more past
campaigns; for each message component type, selecting a plurality
of values to recommend testing based at least in part on the ranked
list of previously-used values; enabling the campaign manager to
reject one or more of the recommended values; in response to the
campaign manager rejecting one or more of the recommended values,
providing alternate recommended values for the rejected values; and
generating variations of a message to test based on the grammar
structure and values accepted by the campaign manager.
2. The method of claim 1, wherein the recommended values are
selected in part from the ranked list of previously-used values and
in part from a ranked list of untested values.
3. The method of claim 2, wherein a plurality of untested values is
grouped by semantic tag and ranked according to their semantic tag,
wherein within a tag group an untested value is randomly
ranked.
4. The method of claim 2, wherein the percentage of recommended
untested values depends on a degree of exploration/conservatism
indicated by the campaign manager.
5. The method of claim 1, wherein the recommended values are also
selected in part from a group of untested values having an
associated untested/unranked semantic tag.
6. The method of claim 5, wherein each untested value in the group
of untested values is associated with a confidence level based on
the global ranking of the untested value across all campaigns.
7. The method of claim 1, wherein one or more parameters in the
user interface is associated with a drop-down menu that lists the
options available to the campaign manager for the parameter.
8. The method of claim 1, wherein the parameters comprise campaign
duration, audience size and characteristics, expected response
rate, effect size, constraints, and objectives of the campaign.
9. The method of claim 8, wherein the campaign manager selects the
objectives of the campaign via a drop-down menu, from a sliding
scale, or by inputting a value.
10. The method of claim 1, wherein if the campaign manager does not
specify certain parameters, default values, empirically determined
based on past campaigns, are used.
11. The method of claim 1, further comprising evaluating the
recommended grammar based on various computed metrics.
12. The method of claim 1, wherein providing alternate recommended
values for the rejected values comprises choosing values from the
next best performing tag group or another candidate value
associated with the same semantic tag as the previously rejected
value.
13. A non-transitory computer-readable medium comprising code that,
when executed by a computer system, enables the computer system to
perform the following method for recommending a grammar for a
message campaign used by a message optimization system, the method
comprising: enabling a campaign manager to specify one or more
parameters for a new campaign; recommending at least one grammar
structure from one or more past campaigns that are based on the
campaign parameters or from a default grammar in the event that
none of the past campaigns has a grammar that complies with the
campaign parameters, the grammar structure specifying a plurality
of message component types; enabling a campaign manager to select
one of the recommended grammar structures for the new campaign; for
each message component type in the selected grammar structure,
generating a ranked list of previously-used values for the
component type in the one or more past campaigns, wherein the
previously-used values are each associated with a semantic tag and
generating the ranked list comprises: identifying the semantic tags
associated with the previously-used values in the one or more past
campaigns, wherein each semantic tag identifies the semantic
meaning of the associated value, creating a list of the
previously-used values in the one or more past campaigns grouped by
semantic tag, ranking groups of semantic tags based on performance
in the one or more past campaigns of the previously-used values
within a tag group versus other tag groups, and ordering the
previously-used values first by their ranked tag group and second,
within each tag group, by the number of times an individual value
has been identified as the winning value in the one or more past
campaigns; for each message component type, selecting a plurality
of values to recommend testing based at least in part on the ranked
list of previously-used values; enabling the campaign manager to
reject one or more of the recommended values; in response to the
campaign manager rejecting one or more of the recommended values,
providing alternate recommended values for the rejected values; and
generating variations of a message to test based on the grammar
structure and values accepted by the campaign manager.
14. The non-transitory computer-readable medium of claim 13,
wherein the recommended values are selected in part from the ranked
list of previously-used values and in part from a ranked list of
untested values.
15. The non-transitory computer-readable medium of claim 14,
wherein a plurality of untested values is grouped by semantic tag
and ranked according to their semantic tag, wherein within a tag
group an untested value is randomly ranked.
16. The non-transitory computer-readable medium of claim 14,
wherein the percentage of recommended untested values depends on a
degree of exploration/conservatism indicated by the campaign
manager.
17. The non-transitory computer-readable medium of claim 13,
wherein the recommended values are also selected in part from a
group of untested values having an associated untested/unranked
semantic tag.
18. The non-transitory computer-readable medium of claim 17,
wherein each untested value in the group of untested values is
associated with a confidence level based on the global ranking of
the untested value across all campaigns.
19. The non-transitory computer-readable medium of claim 13,
wherein enabling a campaign manager to specify parameters for a new
campaign comprises providing a user interface wherein the campaign
manager is prompted to enter parameters for the campaign.
20. The non-transitory computer-readable medium of claim 19,
wherein one or more parameters in the user interface is associated
with a drop-down menu that lists the options available to the
campaign manager for the parameter.
21. The non-transitory computer-readable medium of claim 13,
wherein the parameters comprise campaign duration, audience size
and characteristics, expected response rate, effect size,
constraints, and objectives of the campaign.
22. The non-transitory computer-readable medium of claim 21,
wherein the campaign manager selects the objectives of the campaign
via a drop-down menu, from a sliding scale, or by inputting a
value.
23. The non-transitory computer-readable medium of claim 13,
wherein if the campaign manager does not specify certain
parameters, default values, empirically determined based on past
campaigns, are used.
24. The non-transitory computer-readable medium of claim 13,
further comprising evaluating the recommended grammar based on
various computed metrics.
25. The non-transitory computer-readable medium of claim 13,
wherein providing alternate recommended values for the rejected
values comprises choosing values from the next best performing tag
group or another candidate value associated with the same semantic
tag as the previously rejected value.
26. A computer system for recommending a grammar for a message
campaign used by a message optimization system, the system
comprising: a processor; a memory coupled to the processor, wherein
the memory stores instructions that, when executed by the
processor, causes the system to perform the operations of: enabling
a campaign manager to specify one or more parameters for a new
campaign; recommending at least one grammar structure from one or
more past campaigns that are based on the campaign parameters or
from a default grammar in the event that none of the past campaigns
has a grammar that complies with the campaign parameters, the
grammar structure specifying a plurality of message component
types; enabling a campaign manager to select one of the recommended
grammar structures for the new campaign; for each message component
type in the selected grammar structure, generating a ranked list of
previously-used values for the component type in the one or more
past campaigns, wherein the previously-used values are each
associated with a semantic tag and generating the ranked list
comprises: identifying the semantic tags associated with the
previously-used values in the one or more past campaigns, wherein
each semantic tag identifies the semantic meaning of the associated
value, creating a list of the previously-used values in the one or
more past campaigns grouped by semantic tag, ranking groups of
semantic tags based on performance in the one or more past
campaigns of previously-used values within a tag group versus other
tag groups, and ordering the previously-used values first by their
ranked tag group and second, within each tag group, by the number
of times an individual value has been identified as the winning
value in the one or more past campaigns; for each message component
type, selecting a plurality of values to recommend testing based at
least in part on the ranked list of previously-used values;
enabling the campaign manager to reject one or more of the
recommended values; in response to the campaign manager rejecting
one or more of the recommended values, providing alternate
recommended values for the rejected values; and generating
variations of a message to test based on the grammar structure and
values accepted by the campaign manager.
Description
RELATED APPLICATIONS
[0001] This application is a continuation of U.S. patent
application Ser. No. 13/474,695 titled "System and Method for
Recommending a Grammar for a Message Campaign Used by a Message
Optimization System," which was filed on May 17, 2012, the contents
of which are incorporated by reference as if fully disclosed
herein.
BACKGROUND OF THE INVENTION
1. Field of the Invention
[0002] This invention relates generally to a message optimization
system and, more particularly, to a system and method for
recommending a grammar for a message campaign used by a message
optimization system.
2. Description of the Background Art
[0003] Commercial advertising has undergone a significant shift in
the past decade. Traditional media advertising, taking the form of
newspapers, magazines, television commercials, radio advertising,
outdoor advertising, and direct mail, etc., has been decreasing as
the primary method of reaching an audience, especially as related
to certain target demographics or types of products. New media
advertising, in the form of Popup, Flash, banner, Popunder,
advergaming, email advertising, mobile advertising, etc., has been
increasing in prominence.
[0004] One characteristic of new media advertising is the need to
capture an audience's (viewers, readers, or listeners) attention
with limited text. For example, with a banner or text message, the
sponsor of the advertising message may only have a finite number of
characters to persuade its audience to act by clicking on a link,
texting back a message, etc. As a result, companies are
increasingly interested in how to optimize their message, and the
components in the message, to increase the message's response rate.
International Publication Number WO 2011/076318 A1 discloses a
system and method for optimizing a message and is incorporated by
reference herein in its entirety. In this system, the message is
divided into components and multiple values are tested for each
component to determine the best response rates.
[0005] The best message optimization system, however, is only as
good as its starting values. In the past, a user, or campaign
manager, would manually choose the starting values. However, an
inexperienced campaign manager may not know what values to start
with. Therefore, there is a need for a system and method for
recommending an initial grammar for a message campaign used by a
message optimization system based on the data from previous
campaigns.
SUMMARY OF THE INVENTION
[0006] The present invention is directed to a system and method for
recommending a grammar for a message campaign used by a message
optimization system. The method includes enabling a campaign
manager to specify one or more parameters for a new campaign. A set
of statistical design budgets is calculated based on one or more of
the specified parameters. In one embodiment, the statistical design
budgets are calculated based on audience size, effect size, and
expected response rate.
[0007] At least one grammar structure is recommended based on the
set of statistical design budgets, where the grammar structure
specifies a plurality of message component types. A campaign
manager is then able to select one of the recommended grammar
structures for the new campaign. The n-most relevant past campaigns
are identified based on the specified parameters. For each message
component type in the selected grammar structure, a ranked list of
previously used values for the component type in the n-most
relevant past campaigns is generated, where the values are ranked
at least in part based on performance in the n-most relevant past
campaigns. For each message component type, a plurality of values
is selected to recommend based at least in part on the ranked
list.
[0008] The campaign manager is able to reject one or more of the
recommended values. In response to the campaign manager rejecting
one or more of the recommended values, alternate recommended values
are provided for the rejected values. In certain embodiments,
providing alternate recommended values for the rejected values
comprises choosing values from the next best performing tag or
another candidate value associated with the same tag as the
previously rejected value. In one embodiment, the method further
comprises evaluating the recommended grammar based on various
computed metrics.
[0009] In one embodiment, the previously used values are each
associated with a semantic tag. Generating the ranked list includes
identifying the tags associated with the previously used values in
the n-most relevant campaigns. The tags are then ranked based on
performance of tagged values in the n-most relevant campaigns. The
previously used values are ordered first by ranked tag group and
second, within each tag group, by the number of time an individual
value has been identified as the winning value in the n-most
relevant campaigns.
[0010] In one embodiment, the recommend values are selected in part
from the ranked list of previously used values and in part from a
ranked list of untested values. A plurality of untested values may
be grouped by semantic tag and ranked according to their semantic
tag, where within a tag group an untested value is randomly ranked.
The percentage of recommended untested values may depend on a
degree of exploration/conservatism indicated by the campaign
manager.
[0011] In one embodiment, enabling a campaign manager to specify
parameters for a new campaign includes providing a user interface
wherein the campaign manager is prompted to enter parameters for
the campaign. One or more parameters in the user interface may be
associated with a drop down menu that lists the options available
to the campaign manager for the parameter. The parameters may
comprise campaign duration, audience size and characteristics,
expected response rate, effect size, constraints, and objectives of
the campaign. The campaign manager may select the objectives of the
campaign via a drop down menu, from a sliding scale, or by
inputting a value. In certain embodiments, if the campaign manager
does not specify certain parameters, default values, empirically
determined based on past campaigns, are used.
[0012] The present invention provides many unique advantages. For
example, multiple campaign managers are able to centrally and
efficiently access the consolidated, distilled information from
each of their individual campaigns. This allows novice campaign
managers to become productive very quickly and allows individual
campaign managers to manage a greater number of campaigns at the
same time. For example, whereas previously a campaign manager would
need at least 2 to 3 hours to run a single campaign, a campaign
manager using the present system is able to process campaigns
orders of magnitude faster.
BRIEF DESCRIPTION OF THE DRAWINGS
[0013] FIGS. 1a-1b are a flowchart that illustrates a method for
recommending a grammar according to one embodiment of the
invention.
[0014] FIG. 2 is a flowchart that illustrates a method for
recommending a grammar structure according to one embodiment of the
invention.
[0015] FIG. 3 is a flowchart that illustrates a method for
generating a ranked list for previously tested values according to
one embodiment of the invention.
[0016] FIG. 4 is a flowchart that illustrates a method for
generating a ranked list for previously untested values according
to one embodiment of the invention.
[0017] FIG. 5 is a block diagram of an exemplary software
architecture for a recommendation system according to one
embodiment of the invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] The present invention provides a system and method for
recommending a grammar for a message campaign used by a message
optimization system. The message optimization system uses the
grammar to automatically generate variations of the message and
test at least some of the variations.
[0019] As used herein, a "component" is a functional part of a
message. Components may be tangible, such as an introduction,
benefit, or a call to action, or they may be intangible, such as
the level of formality, the verb tense, or the type of persuasion.
Components are filled with "values," which may be categorized by
families. For example, values filling the introduction component
may be from the greetings category (e.g., "Congratulations!",
"Hello!", etc.) or they may be from the urgency category (e.g.,
"Only Today!", "Act Now!", etc.) or from other categories. The type
of components and the values tested for each component are the
"grammar" of the message. The invention as described herein applies
to context-free grammars and grammars that are not
context-free.
[0020] Shown below is a simple example grammar named "sentence",
consisting of four components: "intro", "benefit", "product", and
call to action or "cta". Next to each listed component are values
that may fill the component. For example, the introduction
component may be filled with "Congratulations!" or "Great
news!".
TABLE-US-00001 sentence -> [intro] [benefit] [product] [cta]
intro -> Congratulations! | Great news! benefit -> We have a
great offer for you! | Take advantage of this unique opportunity!
product -> Top up your phone with 5$ and get 2$ on us! | For
every 5$ top-up we'll give you 2$ extra! cta -> Act now and top
up! | Top-up your phone today and win! | Don't waste time, top-up
today!
[0021] A "campaign" is the process for testing grammar structures
and values for the structures to determine which structures and
values receive the best response rate, where the "response rate" is
the acceptance of a call to action (e.g., calling a number, sending
an sms to a number, clicking on a weblink, clicking on a mobile
weblink, proceeding to purchase, etc.). The "expected response
rate" is the rate the campaign manager expects to get from the
target audience for the campaign (e.g., 3%, 5%, etc.). The "effect
size" is the percentage difference in the response rate that can be
created by varying the values of the components. A "campaign
manager" is the user of the message optimization system.
[0022] FIGS. 1a-1b show the preferred method for recommending a
grammar. The campaign manager specifies one or more parameters for
a campaign (step 110). The recommendation system generates a user
interface that enables a campaign manager to enter parameters for a
campaign. In one embodiment, a plurality of configurable parameters
is each associated with a drop down menu that lists the options
available to the campaign manager for the parameter. Examples of
parameters include country, client, language, campaign duration,
product, type of offer, promotion channel, activation/participation
channel, whether the campaign is a continuation or repetition of a
previous campaign, audience size and characteristics (e.g.,
gender), expected response rate, effect size, pValues, constraints
(e.g., message length constraints, mandatory components, etc.), and
grammar quality, etc. In a preferred embodiment, the campaign
manager selects the objectives of the campaign (i.e., the degree of
conservatism/exploration or tested versus untested values). The
campaign manager may select from a drop down menu, select from a
sliding scale, or use other means to indicate the campaign
objectives. In certain embodiments where the campaign manager does
not specify certain parameters, the system uses default values that
are empirically determined based on past campaigns. For example,
parameters n or k, discussed below, may be empirically determined
and set as system-wide parameters.
[0023] The system calculates a set of "statistical design budgets"
based on audience size, effect size, and expected response rate
(step 115). The statistical design budget for a campaign is the
number of components in a message and the number of values to test
for each component. Different variations can have the same or
almost the same statistical design budget. For example, a grammar
with three message components and four values for each component
will generate the same number of message variations as message with
four components and three values to test for each. The number of
values associated with each component can vary within a grammar.
For example, there may be five values for one component and two
values for another.
[0024] If the effect size is small (i.e., the percentage difference
between the response rate of two values is small), then the
audience size must be large in order to return statistically
significant data. For example, if Component Value A has a response
rate of 3.0% and Component Value B has a response rate of 3.3%, the
difference in absolute terms is 0.3% or an effect size of 10%.
[0025] A person skilled in the art would understand that an effect
size of 0-5% is negligible, such that it would be very hard to
differentiate between variations in response rates and inherent
noise. An effect size of 5-10% is generally considered very small
and would require a very large sample in order to establish the
statistical significance. An effect size of 10-15% is considered
small and would require a relatively large sample, but not as large
as the sample required for an effect size of 5-10%. An effect size
of 15-30% is generally considered "medium" and is the focal point
of interest in most optimization scenarios. An effect size of 30%
or greater is large and usually observed when using extreme
component values or with very small response rates. In the above
example, the system would need a very larger audience size in order
to produce useful data.
[0026] The expected response rate is the response rate (from the
target audience) that the campaign manager expects to get for the
campaign (e.g. 3%, 5%, etc.). In one embodiment, a campaign manager
can specify the effect size and response rate in step 110. If the
campaign manager does not specify the effect size and the response
rate, the system may use default values that are empirically
determined based on past campaigns.
[0027] Returning again to FIG. 1, the system recommends one or more
grammar structures (i.e., the type of components and the initial
organization of the components within the grammar) based on the set
of statistical design budgets (step 120). In other words, the
system attempts to recommend a grammar structure that fits within
the set of statistical design budgets. In the example grammar
above, the grammar structure is: sentence->[intro] [benefit]
[product] [cta]. In recommending a grammar structure, the system
may consider appropriate components to break the message up based
on established practices in similar campaigns or commonly recurring
components across a plurality of campaigns. If the campaign manager
rejects all of the recommended grammar structures, the system
recommends additional grammar structures.
[0028] If the campaign manager accepts one of the recommended
grammar structures (step 125), then, for each component type in the
grammar structure, the system generates a ranked list of previously
used values in the n-most relevant past campaigns (step 130). In
the preferred embodiment, values are ranked by semantic meaning
(i.e., tag applied by the campaign manager) and, within a semantic
grouping, by performance (i.e., response rate). The n-most relevant
campaigns are determined using the parameters entered by the
campaign manager in step 110.
[0029] For each component in the selected grammar structure, the
system selects a plurality of values to recommend based at least in
part on the ranked list generated for that component type (step
135). Other factors that the system may consider in recommending
candidate values are values that satisfy any constraints on
category or length (i.e., values that do not satisfy
length/category strength are filtered out), winners from grammars
in similar past campaigns, values that have performed well in
general, or values from lists of untested values. The system may
also consider synonyms or semantically related values to other well
performing candidate values from past campaigns. Values across
languages may be chosen based on their translation. The system may
also select values from both a ranked list of previously used
values and in part from a ranked list of untested values based on
the degree of conservatism/exploration indicated by the campaign
manager. The number of values recommended for each component is
based on the parameters of the statistical design budget for the
selected grammar. The recommended grammar is then presented to the
campaign manager (step 140).
[0030] In one embodiment, the system computes various metrics for
the recommended grammar to enable the campaign manager to evaluate
the recommended grammar (step 145). Factors affecting the
evaluation of the recommended grammar may include the degree of
exploration of the grammar (i.e., how many new/untested component
values are being tried versus how many are tried and tested), the
"structural spread" of the grammar (i.e., whether the grammar
branches out across a wide variety of syntactic constructs or
whether it narrowly explores a small area), how much of the design
budget is being spent (i.e., how many messages does the
experimental design predict we need to send in order to fully test
the grammar), how is the distribution across message components
(i.e., are some components being tested across a wide range of
candidate values and others only a few), amount of wasted text
space (i.e., whether short components are being compared with long
components where the long component values are reserving
unnecessary space), how close we are to the optimal grammar (i.e.,
sometimes the campaign manager will choose a suboptimal candidate
value as the top values from the top performing tags may be
overused), degree of repeatability (i.e., how similar is the
recommended grammar to the grammars that have recently been used on
this audience), and degree of novel investment (i.e., whether the
system is testing a large number of values in components that
generally have fewer combinations or vice versa). The metrics
applied to the recommended grammars need to be interpreted in light
of the parameters of a particular campaign and the objectives of
the campaign manager.
[0031] The campaign manager is able to reject recommended component
values (step 150). For example, there may be legal or branding
issues with certain expressions that the campaign manager wishes to
avoid. If the campaign manager rejects one or more of the component
values (step 155), the system provides alternate values (step 160).
For values chosen from a ranked list, the system selects the next
best value(s) that satisfies any specified constraints (e.g.,
semantic tag constraints, size constraints, etc.). The campaign
manager may choose values from the next best performing tag or
another candidate value associated with the same tag as the
previously rejected value. The system then computes the various
metrics for the recommended grammar with the alternate values
according to step 145. If, however, the campaign manager does not
reject any component values, the system "commits" the grammar for
processing (for use in the campaign) (step 165).
[0032] In certain embodiments, data from past campaigns is stored
in one or more databases. The data may include the grammar that was
used, parameters related to the campaign (e.g., country, client,
language, campaign duration, product, type of offer, promotion
channel, activation/participation channel, whether the campaign is
a continuation or repetition of a previous campaign, audience size
and characteristics, expected response rate, effect size,
constraints, grammar quality, objectives of the campaign, etc.),
the response rate, the corresponding pValue for the response rate,
etc. The response rates, pValues, and other data may be
automatically entered by a message optimization system when a
campaign is run. Typically, a campaign manager manually tags values
with semantic tags.
[0033] FIG. 2 illustrates a process for recommending a grammar
structure according to step 120 in FIG. 1. The system first
determines whether at least k of the n-most relevant past campaigns
have a grammar within the statistical design budget (step 210). If
not, the system selects a grammar structure from a default template
grammar that complies with the statistical design budget (step
230). If it does, then the system determines whether k is greater
than the value 1. If not, the system selects a grammar structure
from the one campaign with a grammar within the statistical design
budget (step 240). If it is, the system presents grammar structures
for each of the k relevant campaigns to the campaign manager and
lets the campaign manager select from the presented grammar
structures (step 250).
[0034] FIG. 3 illustrates a method for generating a ranked list for
previously tested values according to step 130 in FIG. 1. As a
ranked list is generated for each component type in the grammar
structure selected by the campaign manager, steps 320-350 of the
method are performed for each component type in the grammar
structure. The system identifies the n-most relevant campaigns
based on parameters entered by the campaign manager (step 310).
From the grammars used in such campaigns, the system identifies the
various values tested in such campaigns for the relevant component
type (step 320). In the preferred embodiment, only campaigns from
the selected set that include the component type are considered.
All of the identified component values are then grouped by the
semantic tags associated with their values (step 330). The tags in
step 330 are ranked based on past performance, in terms of response
rate, of the values within a tag group, from the highest to the
lowest, versus other tag groups (step 340). In the preferred
embodiment, the past performance data is limited to data from the
n-most relevant campaigns. The tag rankings reflect the performance
of the tag group for the relevant component type in the n-most
relevant campaigns. A ranked list of all component values (for the
applicable component type) from the n-most relevant past campaigns
is then created, wherein component values are ordered first by tag
(in order of tag rankings), and then, within each tag group, by the
number of times the individual value has been identified as the
winning value in the n-most relevant campaigns for the relevant
component type (step 350). The values that do not satisfy length
constraints or other constraints are filtered out.
[0035] In one embodiment, the tags and component values are ranked
as follows:
TABLE-US-00002 Total Tag 1 Tag 2 Tag 3 . . . Tag t Tag 1 m(1, 1)
m(1, 2) m(1, 3) . . . m(1, t) m(1, Total) Tag 2 m(2, 1) m(2, 2)
m(2, 3) . . . m(2, t) m(2, Total) Tag 3 m(3, 1) m(3, 2) m(3, 3) . .
. m(3, t) m(3, Total) . . . . . . . . . . . . . . . . . . Tag t
m(t, 1) m(t, 2) m(t, 3) . . . m(t, t) m(t, Total)
[0036] For every cell value m(i, j), where 0<i.ltoreq.t,
0<j.ltoreq.t, and t is the number of tags in the system,
information about Tag i competing against Tag j is computed as
follows:
[0037] 1. Compute the number of times a component text that is
tagged with Tag i is chosen over a component text that is tagged
with Tag j; compute the number of times a component text that is
tagged with Tag j is chosen over a component text that is tagged
with Tag i; compute the number of times the results show that the
performance of Tag i and Tag j are not statistically different.
[0038] 2. Using the above values, compute the ratio of wins to
losses, ratio of wins to total, and take the weighted average for
the aggregate score for each cell value.
[0039] For every cell value m(i, Total), a summary of the
performance of Tag i across all tags is computed by aggregating all
the cell scores in the row for that tag. The number may be zero as
the two tags may have never competed against each other.
[0040] The above matrix may be computed each time the system
recommends a new grammar because the campaign manager may give
different parameters and the system may select a different set of
n-most relevant campaigns.
[0041] The system ranks the tags from the highest Total to the
lowest Total score. For each component type, a list is then created
of all the component values grouped by tag and ordered first by tag
and then within each tag by the number of times the individual
component value has been identified as a winning value in any past
grammar in the set of n-most relevant campaigns. From this list,
the component values are selected to fill the grammar structure
based on the design budget (e.g., if the design budget calls for
four introduction component values, the system selects the first
component value that comes from the top four component tag groups,
filtering out inappropriate values based on constraints).
[0042] In one embodiment, if two tag groups are equal in
performance (or are untested relative to each other), the ranking
between the two tag groups is randomly determined. In another
embodiment, the ranking is based on a predefined default. In still
another embodiment, for breaking ties between tag groups, the
system will consider a global ranked list that takes into account
ranking information of all campaigns and not just the n-most
relevant campaigns.
[0043] Depending on the ratio of conservatism to exploration
indicated by the campaign manager, the system may add some untested
values into the campaign. The more conservative the campaign, the
more values are selected from the ranked list of tested values from
previous, relevant campaigns, but the more exploratory the
campaign, the fewer tested values are selected and the more
untested, or little-tested values, are selected.
[0044] FIG. 4 illustrates a method for generating a ranked list for
previously untested values according to step 130 in FIG. 1. As is
the case with the method of FIG. 3, this method is performed for
each component type in the grammar structure selected by the
campaign manager. The system retrieves untested values for the
relevant component type from a database, where the untested values
are each associated with tags in the ranked list of tags from step
340 (step 410). All of the untested values in step 410 are grouped
by the semantic tag associated with each value (step 420). The tags
are ranked based on past performance of the tag group for
previously tested values (step 430). In other words, the tag
rankings from step 340 in FIG. 3 are used. For the applicable
component type, a ranked list of all untested component values
grouped by tag is created (step 440). Component values are ordered
first by tag, and then, within each tag group, untested components
values are randomly ranked. Values exceeding length or violating
other constraints are filtered out. In certain embodiments, an
untested value that is not associated with a tag in the ranked list
of tags from step 340 may be used. These untested values having
untested/unranked tags would preferably be grouped together
separately and used by a campaign manager who desires a greater
degree of exploration or would like to test certain untested tags.
In certain embodiments, the system may assign a confidence level to
the untested value having a corresponding untested/unranked tag
based on the tag's global ranking, which is its ranking across all
campaigns and not just the n-most relevant campaigns.
[0045] From the lists generated according to the methods of FIGS. 3
and 4, the system recommends values for each of the component types
in the grammar structure. The number of values recommended for each
component type is based on the statistical design budget. In one
embodiment, the mix of previously-tested values versus untested
values in the recommendation depends on the level of exploration
indicated by the campaign manager. In certain embodiments, the
ratio of conservatism to exploration is determined by a sliding
scale adjusted by the campaign manager in the user interface. In
other embodiments, it is determined by a percentage inputted by the
campaign manager. In still other embodiments, the system uses a
default percentage mix (e.g., 25% untested values, 75%
previously-tested values).
[0046] FIG. 5 illustrates an exemplary system architecture for a
Recommendation System 500. As a person skilled in the art would
understand, the system architecture may be constructed in any
number of ways, having more or less modules and different
interconnectivity, within the scope of the present invention. The
methods of FIGS. 1-4 may be implemented in other systems, and the
invention is not limited to system 500.
[0047] Client Applications 505 provide a user interface via which
users (e.g., campaign managers) can enter parameters for a new
campaign and review and/or modify a recommended grammar. The Client
Applications 505 may be run on any number of systems and may
connect to the Recommendation System 500 through any number of
channels. For example, the Client Applications 505 may include web,
desktop, or mobile applications.
[0048] The Recommendation System 500 has a User Interface (UI)
Generation Module 510, a Recommendation Engine 520, a Ranking
Module 530, a Relevant Campaign Retrieval Module 540, a Statistical
Design Module 550, a Tagging Module 560, a Grammar Evaluation
Module 570, and a Database Interface 580.
[0049] The UI Generation Module 510 provides a user interface
between the client applications 505 and the Recommendation Engine
520. The Recommendation Engine 520 recommends an initial grammar
for the message campaign used by the message optimization system.
The Ranking Module 530 ranks a plurality of semantic tags based on
the past performance of the previously used values associated with
each semantic tag. The Ranking Module 530 then orders the
previously used values first by ranked tag group and then within
each tag group. The plurality of untested values is grouped by
semantic tag, but then randomly ordered within the tag group.
[0050] The Relevant Campaign Retrieval Module 540 retrieves the
relevant past campaigns given the set of user specified input
parameters. The Statistical Design Module 550 produces a set of
statistical design budget options. The Tagging Module 560 interacts
with the UI Generation Module 510 to allow the campaign manager to
tag the components with semantic tags. The Grammar Evaluation
Module 570 computes metrics for a recommended grammar given the set
of input parameters. The Database Interface 580 interfaces with one
or more databases 590, which functions to store past campaign data
(e.g., the grammar for each campaign, the response rate for each
tested value, the pValue indicating the statistical significance of
the difference between tested values for a component, etc.).
[0051] The methods described with respect to FIGS. 1-4 are embodied
in software and performed by a computer system executing the
software. A person skilled in the art would understand that a
computer system has a memory or other physical, computer-readable
storage medium for storing software instructions and one or more
processors for executing the software instructions.
[0052] As will be understood by those familiar with the art, the
invention may be embodied in other specific forms without departing
from the spirit or essential characteristics thereof. Accordingly,
the above disclosure of the present invention is intended to be
illustrative and not limiting of the invention.
* * * * *