U.S. patent application number 15/648446 was filed with the patent office on 2018-01-04 for content recommendation based on collections of entities.
The applicant listed for this patent is VCVC III LLC. Invention is credited to Krzysztof Koperski, Jisheng Liang, Neil Roseman.
Application Number | 20180004843 15/648446 |
Document ID | / |
Family ID | 44912663 |
Filed Date | 2018-01-04 |
United States Patent
Application |
20180004843 |
Kind Code |
A1 |
Koperski; Krzysztof ; et
al. |
January 4, 2018 |
CONTENT RECOMMENDATION BASED ON COLLECTIONS OF ENTITIES
Abstract
Techniques for content recommendation are described. Some
embodiments provide a content recommendation system ("CRS")
configured to recommend content items that are related to a
collection of entities. A content item may be considered related to
a collection of entities based on various factors, including
whether and how often the article references or otherwise covers
the entities of the collection, the size of the article, other
entities that are covered by the article but that are not in the
collection, article recency, or article credibility. Recommending
content items may also or instead include determining entities that
are related to a collection. An entity can be considered related to
a collection based on various factors, such as whether the entity
is of the same or similar type to entities of the collection, or
whether the entity appears in some article in a relationship with
one or more entities of the collection.
Inventors: |
Koperski; Krzysztof;
(Seattle, WA) ; Liang; Jisheng; (Bellevue, WA)
; Roseman; Neil; (Seattle, WA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
VCVC III LLC |
Seattle |
WA |
US |
|
|
Family ID: |
44912663 |
Appl. No.: |
15/648446 |
Filed: |
July 12, 2017 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13038192 |
Mar 1, 2011 |
9710556 |
|
|
15648446 |
|
|
|
|
61309318 |
Mar 1, 2010 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 16/9535 20190101;
G06F 16/951 20190101 |
International
Class: |
G06F 17/30 20060101
G06F017/30 |
Claims
1.-36. (canceled)
37. A method in a content recommendation computer system for
determining and presenting collections of entities and related
content, comprising: receiving via a user interface control of the
content recommendation computer system, user identification of a
plurality of named entities; creating or modifying an entity
collection to include the user identified plurality of named
entities, the entity collection presentable on a display of the
content recommendation computer system and sharable with other
users on separate displays of other computer systems, the entity
collection including multiple named entities each referenced by one
or more of multiple documents indexed by the content recommendation
system, the created or modified entity collection being separate
and distinct from the multiple documents indexed by the content
recommendation system; automatically processing the created or
modified entity collection to automatically determine one or more
additional named entities not already included in the entity
collection that are related to the multiple named entities of the
entity collection by determining whether the one or more additional
named entities share semantic characteristics with one or more of
the multiple named entities or appear in semantic relationships
with one or more of the multiple named entities in one or more of
multiple documents indexed by the content recommendation system;
and presenting via a user interface control the determined one or
more additional named entities for possible inclusion in the entity
collection.
38. The method of claim 37 wherein the user interface control for
receiving the user identification of the plurality of named
entities is separate from the user interface control for presenting
the determined one or more additional named entities for possible
inclusion in the entity collection.
39. The method of claim 37 wherein the automatically processing the
created or modified entity collection to automatically determine
one or more additional named entities not already included in the
entity collection that are related to the multiple named entities
of the entity collection further comprises: automatically
processing the created or modified entity collection to
automatically determine one or more additional named entities not
already included in the entity collection that are related to the
multiple named entities of the entity collection by determining
whether the one or more additional named entities appear in
semantic context with one or more of the multiple named entities in
one or more of multiple documents indexed by the content
recommendation system, appearing in context determined as appearing
within a specified number of surrounding paragraphs, sentences,
words, and/or characters.
40. The method of claim 37, further comprising: automatically
processing the created or modified entity collection to
automatically recommend one or more documents indexed by the
content recommendation system that are related to at least one of
the multiple named entities of the entity collection; and
presenting on the computer system the determined one or more
documents as related documents to the entity collection.
41. The method of claim 40 wherein the automatically processing the
created or modified entity collection to automatically recommend
one or more documents indexed by the content recommendation system
that are related to at least one of the multiple named entities of
the entity collection is determined based at least in part upon one
or more of: whether and how often an indexed article refers to one
or more of the multiple named entities, recency of an indexed
article, or a measure of credibility of an indexed article.
42. The method of claim 40 automatically processing the created or
modified entity collection to automatically recommend one or more
documents indexed by the content recommendation system that are
related to at least one of the multiple named entities of the
entity collection includes recommending one or more of: an article,
a news story, a Web page, or a blog post.
43. The method of claim 40 further comprising: ordering the
determined one or more recommended documents before presenting the
one or more recommended documents based at least in part upon
document recency and/or document relevance.
44. The method of claim 40, further comprising: ordering the
determined one or more recommended documents before presenting the
one or more recommended documents based at least in part on source
credibility, wherein source credibility is determined by
automatically determining a credibility score based upon properties
of a source of the content item.
45. The method of claim 37 wherein the plurality of named entities
and the multiple named entities are each electronically represented
in the content recommendation system in a data repository.
46. The method of claim 37 wherein the user interface control for
receiving user identification of the plurality of named entities is
a control on a website page
47. The method of claim 37 wherein user indication is provided by
specifying a collection name and an indication of the plurality of
the named entities.
48. The method of claim 37 wherein the automatically processing the
created or modified entity collection to automatically recommend
one or more documents indexed by the content recommendation system
that are related to at least one of the multiple named entities of
the entity collection by determining whether the one or more
additional named entities share semantic characteristics with one
or more of the multiple named entities is determined based at least
in part upon whether the one or more additional named entities
share a facet value with at least one of the multiple named
entities of the entity collection.
49. The method of claim 37, further comprising: ordering the
determined one or more additional named entities before presenting
the one or more additional named entities via the user interface
control based at least in part upon a frequency each additional
named entity appears in the indexed documents and based at least in
part upon whether the determined one or more additional named
entities share a facet value with at least one of the multiple
named entities of the entity collection.
50. The method of claim 37, further comprising: receiving an
indication of at least one of the one or more additional named
entities for inclusion in the entity collection; and modifying the
entity collection to include the indicated one or more additional
named entities.
51. The method of claim 37 wherein the user interface for receiving
user identification of a plurality of named entities is provided
via a code module configured to perform the method in a third-party
application context.
52. The method of claim 37 wherein automatically processing the
created or modified entity collection to automatically determine
one or more additional named entities not already included in the
entity collection that are related to the multiple named entities
of the entity collection includes determining one or more
additional named entities that each appear in a subject-verb-object
relationship described in an document indexed by the content
recommendation system and that includes one of the multiple named
entities.
53. A non-transitory computer-readable medium whose contents enable
a computing system to recommend content, by performing a method
comprising: receiving via a first user interface control
indications of a plurality of named entities: creating or modifying
an entity collection, the entity collection including indications
of multiple named entities including the user identified plurality
of named entities, each of the multiple named entities in the
entity collection referenced by one or more of multiple documents
indexed by the content recommendation system, the created or
modified entity collection being separate and distinct from the
multiple documents indexed by the content recommendation system;
automatically processing the created or modified entity collection
to automatically determine one or more additional named entities
not already included in the entity collection that are related to
the multiple named entities of the entity collection by determining
whether the one or more additional named entities share
characteristics with one or more of the multiple named entities or
appear in semantic relationships with one or more of the multiple
named entities in one or more of multiple documents indexed by the
content recommendation system; presenting via a second user
interface control the determined one or more additional named
entities for possible inclusion in the entity collection;
automatically processing the created or modified entity collection
to automatically recommend one or more documents indexed by the
content recommendation system that are related to at least one of
the multiple named entities of the entity collection; and
presenting via a third user interface control the automatically
recommended one or more documents indexed by the content
recommendation system that are related to at least one of the
multiple named entities of the entity collection.
54. A computing system configured to recommend content, comprising:
a memory; and a user interface code module stored on the memory
that is configured, when executed, to: receive via a first user
interface control indications of a plurality of named entities:
create or modify an entity collection, the entity collection
including indications of multiple named entities including the user
identified plurality of named entities, each of the multiple named
entities in the entity collection referenced by one or more of
multiple documents indexed by the content recommendation system,
the created or modified entity collection being separate and
distinct from the multiple documents indexed by the content
recommendation system; automatically process the created or
modified entity collection to automatically determine one or more
additional named entities not already included in the entity
collection that are related to the multiple named entities of the
entity collection by determining whether the one or more additional
named entities share characteristics with one or more of the
multiple named entities or appear in semantic relationships with
one or more of the multiple named entities in one or more of
multiple documents indexed by the content recommendation system;
present via a second user interface control the determined one or
more additional named entities for possible inclusion in the entity
collection; automatically process the created or modified entity
collection to automatically recommend one or more documents indexed
by the content recommendation system that are related to at least
one of the multiple named entities of the entity collection; and
present via a third user interface control the automatically
recommended one or more documents indexed by the content
recommendation system that are related to at least one of the
multiple named entities of the entity collection.
55. The computing system of claim 54 wherein the module is
configured to automatically process the created or modified entity
collection to automatically recommend at least one of an article,
image, advertisement, audio, video, and/or Web page.
56. The computing system of claim 54 wherein the module is
configured to present the determined one or more additional named
entities for possible inclusion in the entity collection and the
automatically recommended one or more documents indexed by the
content recommendation system to at least one of a desktop
computing system, a personal digital assistant, a smart phone, a
mobile application, a laptop computer, and/or a third-party
application.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. patent
application Ser. No. 13/038,192 filed on Mar. 1, 2011, which claims
priority from U.S. Provisional Patent Application No. 61/309,318
filed on Mar. 1, 2010, the contents of which applications are
hereby incorporated herein by reference in their entireties.
TECHNICAL FIELD
[0002] The present disclosure relates to methods, techniques, and
systems for recommending content and, in particular, to methods,
techniques, and systems for recommending content items that are
related to a collection of entities.
BRIEF DESCRIPTION OF THE DRAWINGS
[0003] FIG. 1 illustrates an example screen display provided by an
example embodiment of a content recommendation system.
[0004] FIG. 2 illustrates an example block diagram of an example
embodiment of a content recommendation system.
[0005] FIGS. 3A-3I illustrate example screen displays provided by
an example embodiment of a content recommendation system.
[0006] FIGS. 4A-4D illustrate additional example screen displays
provided by an example embodiment of a content recommendation
system.
[0007] FIG. 5 is an example block diagram of an example computing
system for implementing a content recommendation system according
to an example embodiment.
[0008] FIG. 6 is an example flow diagram of a content
recommendation process performed by an example content
recommendation system according to an example embodiment.
[0009] FIG. 7 is an example flow diagram of an article
recommendation process performed by an example content
recommendation system according to an example embodiment.
[0010] FIG. 8 is an example flow diagram of an entity
recommendation process performed by an example content
recommendation system according to an example embodiment.
[0011] FIG. 9 is an example flow diagram of a query generation
process performed by an example content recommendation system
according to an example embodiment.
[0012] FIG. 10 is an example flow diagram of an article scoring
process performed by an example content recommendation system
according to an example embodiment.
[0013] FIGS. 11A-11E illustrate example screen displays generated
by queries executed for content recommendation provided by an
example embodiment of a content recommendation system.
DETAILED DESCRIPTION
[0014] Embodiments described herein provide enhanced computer- and
network-based methods and systems for recommending content. Example
embodiments provide a content recommendation system ("CRS")
configured to recommend content items that are related to a
collection of entities. A CRS may be used, for example, by a user
to locate or follow pertinent content from a large corpus of
documents, such as from news articles available via a wide-area or
local area networked environment, to a set of entities of personal
interest. Content items may include articles and information about
or contained in articles or other documents. A typical article
comprises text, audio, image, and/or video data that provides
information about one or more entities and relationships between
those entities. Information about or contained in articles thus
includes entities and relationships referenced or otherwise covered
by an article. Entities include people, places (e.g., locations),
organizations (e.g., political parties, corporations, groups),
events, concepts, products, substances, and the like. Table 3,
below, includes a list of example entity types. Fewer or more can
be made available. Information about or contained in articles may
also or instead include various types of meta-information, such as
article titles, authors, publication dates, summaries, quotations,
and the like.
[0015] In one embodiment, a user can interact with the CRS to
define a collection of entities of interest. The CRS then
recommends content items that are related to the entities in the
user's collection. Recommending content items may include
determining one or more articles, such as news stories, Web pages,
Blog posts, or the like, that are related to the collection. An
article can be considered related to a collection based on various
factors, including whether and how often the article references or
otherwise covers the entities of the collection, the size of the
article, other entities that are covered by the article but that
are not in the collection, article recency, article credibility,
and the like. Recommending content items may also or instead
include determining one or more entities that are related to the
collection. An entity can be considered related to a collection
based on various factors, such as whether the entity is of the same
or similar type to entities of the collection, whether the entity
appears in some article in a relationship with one or more entities
of the collection, and the like. Specific example techniques for
determining related articles and entities are discussed, for
example, with respect to FIGS. 7-11, below.
[0016] FIG. 1 illustrates an example screen display provided by an
example embodiment of a content recommendation system. In
particular, FIG. 1 illustrates a Web browser 100 that displays a
screen 102 (e.g., a Web page) that is being used by a user to
interact with the content recommendation system. The screen 102
displays information about and related to a collection named
"Collection1," including a collection contents section 104, a
related entities section 106, and a related articles section 108.
The collection contents section 104 indicates that Collection1
includes two entities: Roger Moore and Sean Connery. The related
articles section 108 includes references to multiple articles that
are related to either or both of the entities of Collection1. For
example, article reference 110 includes a title, a snippet from the
article, and indications of at least some of the entities (e.g.,
Richard Demarco, Edinburgh, The Scotsman, Joseph Beuys, Sean
Connery, Scotland, etc.) referenced in the article. The related
entities section 106 includes references to multiple entities that
are related to the entities in Collection1, including James Bond,
George Lazenby, Pierce Brosnan, Timothy Dalton, Daniel Craig, and
the like.
[0017] Some of the examples discussed herein focus primarily on
collections that include indications of one or more singular
entities (e.g., the actor Sean Connery). In other embodiments, a
collection can include indications that reference multiple
entities, such as by including names that ambiguously reference
multiple entities (e.g., Jaguar the animal or Jaguar the car);
facets (e.g., more finely granular characteristics of entities such
as categories, types, and/or characteristics, such as actor,
politician, nation, drug, automobile, and the like); keyterms
(e.g., one or more words or phrases, such as "bond actors," "007,"
and the like); and the like. Table 4, below, includes a list of
example facets for the various entity types. Fewer or more can be
made available. The techniques described herein are equally
applicable to collections that include such indications of multiple
and/or ambiguous entities.
[0018] FIG. 2 illustrates an example block diagram of an example
embodiment of a content recommendation system. In particular, FIG.
2 illustrates a content recommendation system ("CRS") 200 that
includes a content ingester 211, an entity and relationship
identifier 212, an entity recommender 213, an article recommender
214, and a data store 217. The data store 217 includes a document
store 217a, an entity store 217b, a collection store 217c, and a
relationship index 217d. The content ingester 211 receives and
indexes content from various content sources 255, including sources
such as Web sites, Blogs, news feeds, video feeds, and the like.
The content ingester 211 may also receive content from non-public
or semi-public sources, including subscription-based information
services, access-controlled social networks, and the like. The
content ingester 211 provides content information, including data
included within content items (e.g., text, images, video) and
meta-data about content items (e.g., author, title, date, source),
to the entity and relationship identifier 212. The content
information may be provided directly (as illustrated) and/or via
some intermediary, such as the data store 217.
[0019] The entity and relationship identifier 212 receives content
information from the content ingester 211 and identifies entities
and relationships that are referenced therein. Various automatic
and semi-automatic techniques are contemplated for identifying
entities within content items. In one embodiment, the identifier
212 uses natural language processing techniques, such as parts of
speech tagging and relationship searching, to identify sentence
components such as subjects, verbs, and objects, and to
disambiguate and identify entities. Example relationship searching
technology, which uses natural language processing to determine
relationships between subjects and objects in ingested content, is
described in detail in U.S. patent application Ser. No. 11/012,089,
filed Dec. 13, 2004, and entitled "METHOD AND SYSTEM FOR EXTENDING
KEYWORD SEARCHING FOR SYNTACTICALLY AND SEMANTICALLY ANNOTATED
DATA," and entity recognition and disambiguation technology is
described in detail in U.S. patent application Ser. No. 12/288,158,
filed Oct. 15, 2008, and entitled "NLP-BASED ENTITY RECOGNITION AND
DISAMBIGUATION," both of which are incorporated herein by reference
in their entirety. The use of relationship searching, enables the
CRS 200 to establish second order (or greater order) relationships
between entities and to store such information.
[0020] For example, given a sentence such as "Sean Connery starred
in Goldfinger," the identifier 212 may identify "Sean Connery" as
the sentence subject, "starred" as the sentence verb, and
"Goldfinger" as the sentence object. The identified subjects and
objects are then added as disambiguated entities to the entity
store 217b. In the above example, "Sean Connery" and "Goldfinger"
would be added to the entity store 217b. The identified verbs can
then be used to define relationships between the identified
entities. These defined relationships (e.g., stored as
subject-verb-object or SAO triplets, or otherwise) are then stored
in the relationship index 217d. In the above example, a
representation of the fact that the actor Sean Connery starred in
the film Goldfinger would be added to the relationship index 217d.
In some embodiments, the process of identifying entities may be at
least in part manual. For example, entities may be provisionally
identified by the identifier 212, and then submitted to humans for
editing, finalization, review, and/or approval.
[0021] The entity and relationship identifier 212 may determine
various kinds of information about entities and relationships. In
one embodiment, the identifier 212 determines facets, which include
finely grained characteristics of entities, such as entity types,
roles, qualities, functions, and the like. For example, the entity
Sean Connery may have various associated facets, including that of
actor, producer, knight, and Scotsman. The facet information for
entities may be also stored in the entity store 217b.
[0022] A user 202 creates a collection by specifying or otherwise
identifying one or more entities. In some embodiments, the user 202
interacts with a Web site or other type of user interface to create
a collection by providing a collection name and indicating one or
more entities (or other things such as facets or keywords) to be
included in the collection. Collections specified by the user 202
are stored in the collection store 217c. Techniques for collection
creation and collection management are discussed further with
reference to FIGS. 3A-3I, below.
[0023] Given a collection of entities, the article recommender 214
determines, with reference to information contained in the data
store 217, one or more articles that are related to the collection.
As discussed above, determining the one or more articles may
include finding one or more articles that cover or otherwise
reference entities from the collection. The article recommender 214
then provides the determined articles to the user 202, such as via
a Web site, mobile device, or other user interface. Techniques for
article recommendation are discussed further with reference to
FIGS. 6, 7, 9, and 10, below.
[0024] Given a collection of entities, the entity recommender 213
determines, with reference to information contained in the data
store 217, one or more entities that are related to the collection.
As discussed above, determining the one or more entities may
include finding one or more entities that are related to the
entities of the collection, based on factors such as whether the
determined entities share characteristics (e.g. facets) with the
entities of the collection, appear in relationships with entities
of the collection, and/or appear in context with entities of the
collection. Techniques for entity recommendation are discussed
further with reference to FIGS. 6, 8, 9, and 11, below.
[0025] The described techniques herein are not limited to the
specific architecture shown in FIG. 2. For example, in some
embodiments, content ingestion and entity and relationship
identification may be performed by another (possibly external or
remote) system or component. In other embodiments, the CRS 200 may
not interact directly with users, but rather provide user interface
components (e.g., collection widgets) that may be embedded or
otherwise incorporated in third-party applications or systems, such
as Web sites, smart phones, desktop systems, and the like.
[0026] In addition, although the described techniques for content
recommendation are illustrated primarily with respect to article
recommendation and entity recommendation, other types of content
are contemplated. For example, other embodiments may utilize at
least some of the described techniques to perform or facilitate the
recommendation of other types of content, including advertisements,
audio (e.g., music), video, images, and the like.
[0027] FIGS. 3A-3I illustrate example screen displays provided by
an example embodiment of a content recommendation system. More
specifically, FIGS. 3A-3I depict a running example, in which a user
interacts with a Web site operated by the content recommendation
system to create a collection of entities and then obtain
information about articles and entities that are related to the
collection.
[0028] FIG. 3A shows a home page provided by the content
recommendation system. In particular, FIG. 3A illustrates a Web
browser 300 displaying a screen 301 provided to a user by a content
recommendation system. The screen 301 is a home or main page
provided by the content recommendation system, such as may be
provided to the user when he or she first accesses the content
recommendation system. Various examples of different user
interfaces for accessing a content recommendation system are
described in detail in U.S. patent application Ser. No. 12/288,349,
filed Oct. 16, 2008, and entitled "NLP-BASED CONTENT RECOMMENDER,"
which is incorporated herein by reference in its entirety The
screen 301 includes a search tool 302, a collections information
area 303, and an articles area 304. As will be described further
below, the user can use the search tool 302 to initiate searches
for information about content (e.g., articles, entities) known to
the content recommendation system. The collections information area
303 provides information about collections that are associated with
(e.g., created by) the user. At the point in time the current
example is illustrated, the user has not yet defined any
collections. The articles area 304 provides information about
articles known to the content recommendation system, possibly
arranged by categories, such as popular, politics, business,
entertainment, sports, technology, and the like.
[0029] FIG. 3B illustrates initiation of a search via the content
recommendation system. In particular, FIG. 3B shows the Web browser
300 displaying screen 301 during the initiation of a search by the
user. As the user enters text into the search tool 302, a drop down
menu 310 is generated and displayed. The drop down menu 310
displays multiple indications of entities that match a string
(e.g., "sean con") entered by the user, such that the user can
select one of the entities as the subject of the search. In the
present example, the user selects an entity named "Sean
Connery."
[0030] FIG. 3C shows an entity information page provided as a
result of the search initiated in FIG. 3B. In particular, FIG. 3C
illustrates the Web browser 300 displaying a screen 320 provided in
response to a search for the entity named "Sean Connery." The
screen 320 is an entity information page that includes an entity
relationship graph 321, a related articles area 322, a trend area
323, and a collection creation control 324 (labeled "Follow"). The
relationship graph 321 graphically presents relationships between
Sean Connery and one or more other entities. The related articles
area 322 displays indications of articles that reference Sean
Connery. The trend area 323 provides an indication of how
frequently Sean Connery as been mentioned over a particular time
period (e.g., 30 days). In the present example, the user selects
control 324 (e.g., the "Follow" link) to initiate the creation of a
new collection.
[0031] FIG. 3D depicts collection creation. In particular, FIG. 3D
illustrates screen 320 with a new collection dialog 330 displayed
in response to selection of the collection creation control 324.
Using the new collection dialog 330, the user can name a new
collection (e.g., "Collection1") that includes the entity Sean
Connery.
[0032] FIG. 3E illustrates a response to the creation of a new
collection. In particular, FIG. 3E illustrates screen 320 with a
message 340 that informs the user that the entity Sean Connery is
part of a collection named Collection1. The message 340 includes a
control (e.g., a link) 341 that can be selected by the user to
access a collection information page. In the present example, the
user selects control 341.
[0033] FIG. 3F illustrates a collection information page. In
particular, FIG. 3F illustrates a screen 350 that is a collection
information page for the collection named Collection1, created as
described above. The screen 350 includes a collection membership
area 351, a related entity area 352, a related articles area 353,
and an add entity control 354. The collection membership area 351
displays information, such as names and facets, about the entities
of the current collection. In the present example, the collection
membership area 351 indicates that Collection1 includes a single
entity named "Sean Connery," who is an actor (e.g., has the facet:
actor). The related entity area 352 displays information (e.g.,
names and facets) about entities that are related to the entities
of Collection1. The related articles area 352 displays information
(e.g., titles, text snippets, entities referenced) about articles
that are related to the entities of Collection1. The entity control
area 354 can be selected by the user to add another entity to the
current collection, as will be described below in FIG. 3G.
[0034] FIG. 3G illustrates the addition of an entity to an existing
collection. In particular, FIG. 3G illustrates the collection
information page of screen 350 during the addition of an entity to
Collection1. Here, the user has selected the add entity control 354
(FIG. 3F) and in response, an entity addition dialog 360 is
displayed. The entity addition dialog 360 includes a search tool
with a drop down menu that is populated with entities matching a
string (e.g., "Roger Mo") entered by the user. In the present
example, the user selects the entity named "Roger Moore."
[0035] FIG. 3H shows a response to the addition of an entity to a
collection. In particular FIG. 3H illustrates the screen 350 and
collection information area 351 introduced in FIG. 3F. In the
present example, collection information area 351 has been modified
to include the entity named "Roger Moore," reflecting the addition
to Collection1 performed as described with reference to FIG.
3G.
[0036] FIG. 3I shows an alternative mechanism for adding an entity
to a collection. In particular, FIG. 3I illustrates an entity
addition dialog 370 that is configured to present multiple entities
that are related to the entities of Collection1. The related
multiple entities are presented as suggested additions that can be
selected by the user without the need to type the name of a
particular entity. Such a user interface control can be provided in
addition to, or instead of, the entity addition dialog 360
described with reference to FIG. 3G.
[0037] FIGS. 4A-4D illustrate additional example screen displays
provided by an example embodiment of a content recommendation
system. More specifically, FIGS. 4A-4D depict various additional or
alternative Web-based user interface techniques for creating,
utilizing, and managing collections.
[0038] FIG. 4A shows a widget gallery provided by a content
recommendation system. In particular, FIG. 4A illustrates a Web
browser 400 displaying a screen 401 provided to a user by a content
recommendation system. The screen 401 is a widget gallery provided
by the content recommendation system. Screen 401 includes a control
(e.g., a link) 402 that can be selected by a user to create a
collections widget. A collections widget is a code module that can
be included in an application (e.g., a 3rd party application) and
that is configured to provide content recommendations based on an
associated collection.
[0039] FIG. 4B illustrates the creation of a collections widget. In
particular, FIG. 4B illustrates a screen 410 displayed in response
to a user selection of control 402. The screen 410 includes a
collections widget creation tool 411 that includes a search tool
and drop down menu that can be selected by the user to specify one
or more entities as members of a new collection. In the present
example, the user is creating a collections widget about soccer
players, including Lionel Messi and Diego Maradona.
[0040] FIG. 4C shows a collections widget preview. In particular,
FIG. 4C illustrates a screen 420 that is a collections widget
preview page displayed in response to the creation of a collections
widget as described with reference to FIG. 4B. The screen 420
includes a widget preview area 421 and a widget code area 422. The
preview area 421 displays a preview of the created widget, as it
would appear if it were embedded in a third-party application, such
as a Web page, blog, or the like. The code area 422 provides
instructions (e.g., JavaScript code) that can be copied by a user
into some other application. The provided instructions are
configured, when executed, to display the collections widget
previewed in the preview area 421.
[0041] FIG. 4D shows a collections gallery. In particular, FIG. 4D
illustrates a screen 430 that includes a collections gallery 431
that displays information about collections created by various
users of the content recommendation system. In this manner, users
of the content recommendation system can share their own
collections and/or learn about collections created by other users.
In the illustrated embodiment, a collection created by a user is by
default public and thus accessible by all other users of the
content recommendation system. Other embodiments provide various
types of access control. In one embodiment, a user can indicate or
define specific users and/or groups of users that can access some
or all of the user's collections. In addition, a user can specify
default access control policies that are to be applied to their
collections, such as private, public, group accessible, and the
like.
[0042] Although the user interface techniques of FIGS. 3A-3I and
4A-4D have been described primarily with reference to Web-based
technologies, the described techniques are equally applicable in
other contexts. For example, a collections widget may be
implemented as one or more code objects (e.g., a Java object) that
can be embedded within a client-side desktop or mobile application,
and configured to interact with the content recommendation system
to obtain and display content recommendations based on an
associated collection.
[0043] FIG. 5 is an example block diagram of an example computing
system for implementing a content recommendation system according
to an example embodiment. In particular, FIG. 5 shows a computing
system 500 that may be utilized to implement a content
recommendation system 510.
[0044] Note that one or more general purpose or special purpose
computing systems/devices may be used to implement the content
recommendation system 510. In addition, the computing system 500
may comprise one or more distinct computing systems/devices and may
span distributed locations. Furthermore, each block shown may
represent one or more such blocks as appropriate to a specific
embodiment or may be combined with other blocks. Also, the content
recommendation system 510 may be implemented in software, hardware,
firmware, or in some combination to achieve the capabilities
described herein.
[0045] In the embodiment shown, computing system 500 comprises a
computer memory ("memory") 501, a display 502, one or more Central
Processing Units ("CPU") 504, Input/Output devices 504 (e.g.,
keyboard, mouse, CRT or LCD display, and the like), other
computer-readable media 505, and network connections 506. The
content recommendation system 510 is shown residing in memory 501.
In other embodiments, some portion of the contents, some or all of
the components of the content recommendation system 510 may be
stored on and/or transmitted over the other computer-readable media
505. The components of the content recommendation system 510
preferably execute on one or more CPUs 503 and recommend content
based on entity collections, as described herein. Other code or
programs 530 (e.g., an administrative interface, a Web server, and
the like) and potentially other data repositories, such as data
repository 520, also reside in the memory 501, and preferably
execute on one or more CPUs 503. Of note, one or more of the
components in FIG. 5 may not be present in any specific
implementation. For example, some embodiments may not provide other
computer readable media 505 or a display 502.
[0046] In a typical embodiment, the content recommendation system
510 includes a content ingester 511, an entity and relationship
identifier 512, an entity recommender 513, an article recommender
514, a user interface manager 515, a recommender application
program interface ("API") 516, and a data store 517. The content
ingester 511, entity and relationship identifier 512, user
interface manager 515, and recommender API 516 are drawn in dashed
lines to emphasize that in other embodiments, functions performed
by one or more of these components may be performed externally to
the content recommendation system 510.
[0047] The content ingester 511 performs functions such as those
described with reference to the content ingester 211 of FIG. 2. The
ingester 511 obtains content items, such as Web pages, Blog
postings, videos, audio files, and the like from various content
sources 555 via network 550, and stores information about the
obtained content items in the data store 517, for use by other
components, such as the entity and relationship identifier 512, the
entity recommender 513, and/or the article recommender 514.
[0048] The entity and relationship identifier 512 performs
functions such as those described with reference to the entity and
relationship identifier 212 of FIG. 2. The identifier 512 processes
the information about content items stored in the data store 517 in
order to identify entities and relationships between them.
Identified entities and relationships are stored in the data store
517, for use by other components, such as the recommenders 513 and
514.
[0049] The UI manager 515 provides a view and a controller that
facilitate user interaction with the content recommendation system
510 and its various components. For example, the UI manager 515 may
provide interactive access to the content recommendation system
510, such that users can create collections, manage/edit
collections, share collections, obtain recommendations based on
collections, and the like. In some embodiments, access to the
functionality of the UI manager 515 may be provided via a Web
server, possibly executing as one of the other programs 530. In
such embodiments, a user operating a Web browser executing on one
of the client devices 560 can interact with the content
recommendation system 510 via the UI manager 515.
[0050] The entity recommender 513 and article recommender 514
perform functions such as those described with reference to the
entity recommender 213 and article recommender 214, respectively,
of FIG. 2. The recommenders 513 and 514 receive, possibly via the
UI manager 515, an indication of a collection (e.g., stored in the
data store 517). In response, the recommenders 513 and 514
respectively determine one or more content items (e.g., entities
and/or articles) that are related to entities of the indicated
collection. The determined content items can be cached (e.g.,
stored in the data store 517) for later use and/or provided
directly in response to a received request, such as may be
generated by the UI manager 515.
[0051] In one embodiment, the recommenders 513 and 514 operate
synchronously in an on-demand manner, in that they perform their
functions in response to a received request, such as in response to
a user interface event processed by the UI manager 515. In another
embodiment, the recommenders 513 and 514 operate asynchronously, in
that they automatically determine related content items for one or
more collections. For example, the recommenders 513 and 515 may
automatically execute from time to time (e.g., once per hour, once
per day) in order to generate bulk content recommendations for some
or all collections stored in the data store 517. The recommenders
513 and 514 may execute upon the occurrence of other types of
conditions, such as when a new collection is created, when new
content items are indexed, and the like.
[0052] The API 516 provides programmatic access to one or more
functions of the content recommendation system 510. For example,
the API 516 may provide a programmatic interface to one or more
functions of the content recommendation system 510 that may be
invoked by one of the other programs 530 or some other module. In
this manner, the API 516 facilitates the development of third-party
software, such as user interfaces, plug-ins, news feeds, adapters
(e.g., for integrating functions of the content recommendation
system 510 into Web applications), and the like. In addition, the
API 516 may be in at least some embodiments invoked or otherwise
accessed via remote entities, such as one of the third-party
applications 565, to access various functions of the content
recommendation system 510. For example, a third-party application
may create a collection and/or receive recommendations based on a
collection from the content recommendation system 510 via the API
516. The API 516 may also be configured to provide collections
widgets (e.g., code modules) that can be integrated into
third-party applications and that are configured to interact with
the content recommendation system 510 to make at least some of the
described functionality available within the context of other
applications.
[0053] The data store 517 is used by the other modules of the
content recommendation system 510 to store and/or communicate
information. As discussed above, components 511-516 use the data
store 517 to record various types of information, including
content, information about stored content including entities and
relationships, information about collections, user information, and
the like. Although the components 511-516 are described as
communicating primarily through the data store 517, other
communication mechanisms are contemplated, including message
passing, function calls, pipes, sockets, shared memory, and the
like.
[0054] The content recommendation system 510 interacts via the
network 550 with content sources 555, third-party applications 565,
and client computing devices 560. The network 550 may be any
combination of media (e.g., twisted pair, coaxial, fiber optic,
radio frequency), hardware (e.g., routers, switches, repeaters,
transceivers), and protocols (e.g., TCP/IP, UDP, Ethernet, Wi-Fi,
WiMAX) that facilitate communication between remotely situated
humans and/or devices. The client computing devices 560 include
desktop computing systems, notebook computers, mobile phones, smart
phones, personal digital assistants, tablet computers, and the
like.
[0055] Other or additional functions and/or data are contemplated.
For example, in some embodiments, the content recommendation system
510 includes additional content recommendation components that are
specialized to other types of content, such as for video,
quotations, images, audio, advertisements, product information, and
the like.
[0056] In an example embodiment, components/modules of the content
recommendation system 510 are implemented using standard
programming techniques. For example, the content recommendation
system 510 may be implemented as a "native" executable running on
the CPU 503, along with one or more static or dynamic libraries. In
other embodiments, the content recommendation system 510 may be
implemented as instructions processed by a virtual machine that
executes as one of the other programs 530. In general, a range of
programming languages known in the art may be employed for
implementing such example embodiments, including representative
implementations of various programming language paradigms,
including but not limited to, object-oriented (e.g., Java, C++, C#,
Visual Basic.NET, Smalltalk, and the like), functional (e.g., ML,
Lisp, Scheme, and the like), procedural (e.g., C, Pascal, Ada,
Modula, and the like), scripting (e.g., Perl, Ruby, Python,
JavaScript, VBScript, and the like), and declarative (e.g., SQL,
Prolog, and the like).
[0057] The embodiments described above may also use either
well-known or proprietary synchronous or asynchronous client-server
computing techniques. Also, the various components may be
implemented using more monolithic programming techniques, for
example, as an executable running on a single CPU computer system,
or alternatively decomposed using a variety of structuring
techniques known in the art, including but not limited to,
multiprogramming, multithreading, client-server, or peer-to-peer,
running on one or more computer systems each having one or more
CPUs. Some embodiments may execute concurrently and asynchronously,
and communicate using message passing techniques. Equivalent
synchronous embodiments are also supported. Also, other functions
could be implemented and/or performed by each component/module, and
in different orders, and by different components/modules, yet still
achieve the described functions.
[0058] In addition, programming interfaces to the data stored as
part of the content recommendation system 510, such as in the data
store 517, can be available by standard mechanisms such as through
C, C++, C#, and Java APIs; libraries for accessing files,
databases, or other data repositories; through scripting languages
such as XML; or through Web servers, FTP servers, or other types of
servers providing access to stored data. The data store 517 may be
implemented as one or more database systems, file systems, or any
other technique for storing such information, or any combination of
the above, including implementations using distributed computing
techniques.
[0059] Different configurations and locations of programs and data
are contemplated for use with techniques of described herein. A
variety of distributed computing techniques are appropriate for
implementing the components of the illustrated embodiments in a
distributed manner including but not limited to TCP/IP sockets,
RPC, RMI, HTTP, Web Services (XML-RPC, JAX-RPC, SOAP, and the
like). Other variations are possible. Also, other functionality
could be provided by each component/module, or existing
functionality could be distributed amongst the components/modules
in different ways, yet still achieve the functions described
herein.
[0060] Furthermore, in some embodiments, some or all of the
components of the content recommendation system 510 may be
implemented or provided in other manners, such as at least
partially in firmware and/or hardware, including, but not limited
to one or more application-specific integrated circuits ("ASICs"),
standard integrated circuits, controllers executing appropriate
instructions, and including microcontrollers and/or embedded
controllers, field-programmable gate arrays ("FPGAs"), complex
programmable logic devices ("CPLDs"), and the like. Some or all of
the system components and/or data structures may also be stored as
contents (e.g., as executable or other machine-readable software
instructions or structured data) on a computer-readable medium
(e.g., as a hard disk; a memory; a computer network or cellular
wireless network or other data transmission medium; or a portable
media article to be read by an appropriate drive or via an
appropriate connection, such as a DVD or flash memory device) so as
to enable or configure the computer-readable medium and/or one or
more associated computing systems or devices to execute or
otherwise use or provide the contents to perform at least some of
the described techniques. Some or all of the system components and
data structures may also be stored as data signals (e.g., by being
encoded as part of a carrier wave or included as part of an analog
or digital propagated signal) on a variety of computer-readable
transmission mediums, which are then transmitted, including across
wireless-based and wired/cable-based mediums, and may take a
variety of forms (e.g., as part of a single or multiplexed analog
signal, or as multiple discrete digital packets or frames). Some or
all of the system components and data structures may also be stored
in one or more non-transitory computer-readable storage mediums.
Such computer program products may also take other forms in other
embodiments. Accordingly, embodiments of this disclosure may be
practiced with other computer system configurations.
[0061] FIG. 6 is an example flow diagram of a content
recommendation process performed by an example content
recommendation system according to an example embodiment. In
particular, FIG. 6 illustrates a process that may be implemented
by, for example, one or more elements of the content recommendation
system 200, such as the entity recommender 213 and/or the article
recommender 214, as described with reference to FIG. 2. The process
recommends content items based on entities of a collection.
[0062] The illustrated process begins at block 602, where it
defines a collection of entities. Defining a collection of entities
may include receiving indications of one or more entities, such as
may be provided by an interactive user interface operated by a
user. An indication of an entity may be or include a unique
identifier of an entity. In other cases, the indication of the
entity may be ambiguous, in that it may refer to more than one
entity. In still other cases, a class or group of entities may be
specified by a facet. For example, the facet "Actor" could be used
to indicate all entities that are known to be actors. In other
cases, one or more keyterms that include one or more words or
phrases may be used to indicate one or more entities. For example,
the keyterms "soccer player" may be used to indicate all entities
that have or are associated with textual descriptions that match,
or approximately match, the keyterms "soccer player." Defining a
collection may further include storing received entity indications
in a data structure or other data arrangement, such as a file,
database, or the like.
[0063] At block 604, the process determines content items, such as
articles or entities, related to entities of the collection. In one
embodiment, determining related content items includes locating
content items that cover, include, or otherwise reference one or
more entities of the collection. For example, determining related
articles may include locating articles that each reference at least
one of the entities of the collection. In another embodiment,
determining related content items includes determining content
items that are in a relationship with one or more of the entities
of the collection. For example, determining related entities may
include determining one or more entities that are each in a
relationship with at least one of the entities of the collection.
In a further embodiment, determining related content items includes
locating content items that have characteristics that match, or are
in some way similar to, characteristics of one or more entities of
the collection. For example, determining related entities may
include determining entities that share at least one facet with at
least one of the entities of the collection. Determining related
content items may also include ordering and/or ranking the content
items based on various factors, such as relevance, credibility,
recency, and the like. Determining related articles and entities
are discussed further with respect to FIGS. 7 and 8,
respectively.
[0064] At block 606, the process recommends the determined content
items. Recommending the determined content items may include
providing (e.g., displaying, presenting, transmitting, storing,
forwarding) indications of the content items, such as via a user
interface screen, as described above with reference to FIGS. 1 and
3A-3I. In some embodiments, indications of the determined content
items may be provided indirectly, such as via a user interface
widget or other component that is incorporated into a third-party
application. In other embodiments, recommending the determined
content items includes storing the content items (or indications
thereof) in a data store for later consumption or use by another
component.
[0065] Some embodiments perform one or more operations/aspects in
addition to, or instead of, the ones described with reference to
the process of FIG. 6. For example, in one embodiment, the process
also provides user interface widgets configured to provide content
recommendations in the context of a third-party application, such
as those described with reference to FIGS. 4A-4D.
[0066] FIG. 7 is an example flow diagram of an article
recommendation process performed by an example content
recommendation system according to an example embodiment. In
particular, FIG. 7 illustrates a process that may be implemented
by, for example, one or more elements of the content recommendation
system 200, such as the article recommender 214, as described with
reference to FIG. 2. The process recommends articles based on
entities of a collection.
[0067] The illustrated process begins at block 702, where it
receives an indication of a collection including entities E1, E2, .
. . , E3 and number of articles to recommend (N).
[0068] At block 704, the process initializes an article set A to be
the empty set (A={ }). The process also initializes an entity set E
to include all of the entities of the collection (E={E1, E2, . . .
, E3}). The process further initializes a threshold T=N/|E|, or in
other words initializes T to be the quotient of the number of
articles to recommend (N) over the cardinality of the set E (|E|).
The process attempts to find, for each entity, at least T articles
that reference that entity.
[0069] In blocks 706-714, the process performs a loop in which it
retrieves articles that are related to one or more of the entities
in the set E, and removes entities from E when sufficient articles
covering the removed entities have been retrieved, until E is
empty, or some other stopping condition occurs.
[0070] At block 706, the process determines an ordered list of
relevant articles that each reference at least one entity in E. To
determine the ordered list of articles, the process performs a
query that is a Boolean OR of all of the entities in the set E. For
example, if the set E includes entities E1, E2, and E3, the
resulting query would be id(E1) OR id(E2) OR id(E3), where the
operator id(X) returns a unique identifier for an entity X. If any
of the elements of the set E refer to two or more entities (e.g.,
are ambiguous), facets, or keyterms, the resulting query will
incorporate such elements, such as by including the keyterms,
facets, or other disambiguating information, as discussed further
with respect to FIG. 9, below.
[0071] The determined list of articles is ordered based on an
information retrieval weighting that is based at least in part on
one or more of a credibility score, a recency score, a term
frequency, total number of terms in the article, length of the
article, and presence of terms in the title of the article. An
example process for scoring an article for purposes of ordering the
list of articles is described with reference to FIG. 10, below.
[0072] At block 708, the process retrieves and adds the top N
articles of the determined list of articles to the resultant set A.
In some embodiments, determining the list of articles (as in block
706) is computationally less expensive than actually retrieving the
articles. For example, determining the list of articles may be
performed by one or more database queries that operate against
indexed article information, and thus may execute relatively
quickly as compared to the cost of actually retrieving article
content, which is typically stored external to the database and
thus may have a correspondingly larger access and retrieval time.
Thus, as illustrated here, the process only retrieves the top N
articles indicated in the ordered list of articles, and adds them
to the set A, thus adding the N most relevant articles (with
respect to the query of block 706) to the set A. In one example
embodiment, N is 10. Other example embodiments may incorporate a
different number of desired articles.
[0073] At block 710, the process determines whether there are not
enough articles (the number of articles in the determined list of
articles is less than N), and if so, continues at block 716,
otherwise at block 712. With this check, the process performs an
early exit from the loop of blocks 706-714 in cases when the total
number of articles determined in block 706 is less than or equal to
the number of articles to recommend (N). It is appropriate to exit
the loop of blocks 706-714 in this case because performing
additional queries will not result in additional articles being
returned. For example, if 10 articles have been requested (N=10)
and the query of block 706 only returned four articles, the process
would exit, because a subsequent query would not result in any
articles other than those four articles being returned.
[0074] At block 712, the process, for each entity in the set E,
counts the number of articles in the set A that reference the
entity. If the count is greater than the threshold T, the process
removes the entity from set E. In other words, when an entity has
been sufficiently covered by articles in the resultant set A, the
entity will be removed from the set E, thereby reducing the size of
the set E and moving the process one step closer to its stopping
condition of an empty set E. Removing entities from set E will
cause the performance of a different query on subsequent iterations
of the loop of blocks 706-714, typically resulting in the
determination of a list of articles that is different from previous
query results in membership and/or order (due to the information
retrieval weighting being different due to a different query).
[0075] At block 714, the process determines whether the set E is
empty, and if so, continues at block 716, otherwise continues the
loop of blocks 706-714. When the set of entities (E) is empty, then
a sufficient number of articles covering each of the entities of
the collection have been retrieved, and the loop terminates. In
some embodiments, an additional stopping condition is included to
limit the maximum number of iterations of the loop of blocks
706-714. For example, the test at block 714 may instead be to
determine whether the set E is empty or whether the loop of blocks
706-714 has iterated a maximum number of times (e.g., four times).
Such a test bounds the running time of the process in cases of
large collections (e.g., where the set E has many elements).
[0076] At block 716, the process sorts the resultant article set A
based on article relevance. Article relevance may be based on
various factors, including article recency (e.g., publishing date
of each article) and entity coverage (e.g., the number of entities
of the collection referenced in each article).
[0077] After block 716, the process returns an indication of the
ordered set of articles A. Returning the set A may include
returning a data structure (e.g., an XML record) that includes
references to each of the articles in set A. In other embodiments,
the details of the set A are stored in a database record (or other
persistent data store), and an identifier for the database record
is returned.
[0078] Some embodiments perform one or more operations/aspects in
addition to, or instead of, the ones described with reference to
the process of FIG. 7. For example, although the process is
described iteratively here, in other embodiments, the process is
implemented recursively. Also, instead of consecutively performing
separate, different queries, other embodiments may perform multiple
different queries in parallel. In addition, other embodiments may
perform queries that differ from those discussed above, including
queries that connect one or more of the entities using different
operators (e.g., logical AND, NOT, etc.). Techniques for article
recommendation used in a specific example embodiment are described
further below.
[0079] FIG. 8 is an example flow diagram of an entity
recommendation process performed by an example content
recommendation system according to an example embodiment. In
particular, FIG. 8 illustrates a process that may be implemented
by, for example, one or more elements of the content recommendation
system 200, such as the entity recommender 213, as described with
reference to FIG. 2. The process recommends entities based on
entities of a collection.
[0080] The illustrated process begins at block 802, where it
receives an indication of a collection including multiple
entities.
[0081] At block 804, the process determines a first set of entities
that are each related to one of the multiple entities. Determining
the first set of entities includes searching for entities that
appear in relationships expressed in content items such as
articles. As discussed above, the content recommendation system may
include an entity and relationship data store that represents
relationships between entities. This data store can be searched for
all entities that are related to at least one of the entities of
the collection. In one embodiment, the process initiates multiple
queries, where each query is directed to finding entities related
to one of the entities of the collection. More specifically, each
query asks for the set of entities that are one of the subject or
object of a subject-verb-object relation in which the one entity of
the collection appears as the other of the subject or object.
Typically, the process executes the multiple queries in parallel,
in order to efficiently locate a broad set of entities related to
the entities of the collection. For example, given a collection
including Sean Connery and Roger Moore, the process would issue two
queries: a first one requesting entities in relationships with Sean
Connery and a second one requesting entities in relationships with
Roger Moore. Examples of these queries and corresponding example
search results using an example embodiment are described with
reference to FIGS. 11A and 11B, below.
[0082] At block 806, the process determines shared characteristics
of the first set of entities, including common facets and common
keyterms. Determining the shared characteristics of the first set
of entities includes iterating over the relationships returned by
the multiple queries performed at block 804, and counting the
related entities (not the collection entities) occurring in the
relationships, facets assigned to the each of the entities of the
relationships, and keyterms occurring in the relationships. For
example, given a collection that includes Roger Moore and Sean
Connery, particular entities may commonly appear in relationships
with those entities, such as James Bond. Also, facets that could be
common to the example collection include actor and knight.
[0083] Determining the shared characteristics (e.g., facets) may
also include ranking the shared characteristics. For example,
facets may be ranked by frequency counts, such that more commonly
appearing facets occur prior to less commonly appearing facets. At
the end of block 806, the process has determined a ranked list of
the main facets shared by the entities in the collection. For
example, given a collection including Roger Moore and Sean Connery,
shared facets could include actor and knight.
[0084] In addition, keyterms may be ranked based on measures of
document frequency, such as based on the product of term document
frequency and inverse document frequency, where term document
frequency is the number of times a term appears in a given
document, and inverse document frequency is the logarithm of the
quotient of the total number of documents and the number of
documents containing the term. In this manner, the keyterms that
are most relevant and specific to the given collection can be
identified. For example, given a collection including Roger Moore
and Sean Connery, example keyterms could include "007" and "bond
actor."
[0085] At block 808, the process determines a second set of
entities that each appear in an article in a relationship with one
of the shared/common facets identified in block 806 and in context
with one of the common keyterms identified in block 806. This block
results in finding other entities that are potentially similar to
the first set, through their common facets and/or keywords.
Determining the second set of entities includes performing a query
that searches for entities that are in a relationship with an
entity having at least one of the common facets (e.g., find
entities that are in a relationship with an actor), the
relationship appearing in an article in context with at least one
of the keyterms (e.g., the keyterm "007"), where the common facets
and keyterms are those identified at block 806, above. One entity
appears in context with another entity or keyterm if it appears in
an article within a predefined window of text (e.g., 50 characters,
three sentences, same paragraph) around the other entity. For
example, given a collection including Roger Moore and Sean Connery,
the process finds a second set of entities that are in a
relationship with a common facet, such as actor, the relationships
appearing in an article in context with a common keyterm, such as
"007." Such entities could include, for example, James Bond, Daniel
Craig, Pierce Brosnan, and the like. After performing the above
queries, the process iterates through the returned relationships
and collects entities and their frequency counts. An example of the
type of query performed at block 808 and corresponding example
search results using an example embodiment are described with
reference to FIG. 11C, below. Additional examples of the kinds of
queries performed at block 808 are described further below.
[0086] At block 810, the process determines a third set of entities
that each appear in an article in a relationship with one of the
multiple entities and in context with another of the multiple
entities. Determining the third set of entities includes performing
multiple queries to capture inter connections between the entities
in the collection. Performing the multiple queries includes, for
each entity of the collection, performing a query that searches for
entities that are in a relationship with the entity of the
collection, the relationship appearing in an article in context
with one of the other entities of the collection. Again, these
multiple queries can be performed in parallel. For example, given a
collection including Roger Moore and Sean Connery, the process here
performs two queries: first, a query that searches for entities
related to Roger Moore and appearing in context with Sean Connery,
and second, a query that searches for entities related to Sean
Connery and appearing in context with Roger Moore. Examples of
these queries and corresponding example search results using an
example embodiment are described with reference to FIGS. 11D and
11E, below. After performing the multiple queries, the process
again iterates through the returned relationships and collects
entities and their frequency counts.
[0087] At block 812, the process orders a combination of the first,
second, and third sets. In some embodiments, ordering the
combination of the three sets is based on entity frequency counts
and whether entities of the combination share one or more of the
common facets identified in block 806. For example, entities of the
combination may be ordered by their frequency score (e.g., more
frequently occurring entities occurring earlier in the combination)
multiplied by a facet score, which is a measure of how many
entities in the original collection also have a particular facet.
For example, given a collection containing Roger Moore and Sean
Connery, which both have the actor facet, the process may find two
related entities, Daniel Craig and James Bond. As between Daniel
Craig and James Bond, Daniel Craig would have a higher facet score
than James Bond, because Daniel Craig also has the actor facet.
[0088] After block 812, the process returns the ordered combination
of entities. Returning the ordered combination may include
returning a data structure (e.g., an XML record) that includes
references to each entity in the combination. In other embodiments,
the details of the ordered combination are stored in a database
record (or other persistent data store), and an identifier for the
database record is returned.
[0089] Some embodiments perform one or more operations/aspects in
addition to, or instead of, the ones described with reference to
the process of FIG. 8. For example, in one embodiment, the process
constrains its execution time by considering a predetermined number
of entities, facets, keyterms, and the like. For example, at block
806, the process may elect to iterate over a predetermined number
of relationships (e.g., the top 10) returned by the multiple
queries returned at block 804, where the relationships returned by
the multiple queries are ranked in order of relevance. As another
example, at block 808 the process may generate a query that
includes only a predetermined number of the most common facets
and/or keyterms. Techniques for entity recommendation used in a
specific example embodiment are described further below.
[0090] FIG. 9 is an example flow diagram of a query generation
process performed by an example content recommendation system
according to an example embodiment. In particular, FIG. 9
illustrates a process that may be implemented by, for example, one
or more elements of the content recommendation system 200, as
described with reference to FIG. 2. The process generates a query
or a portion of a query that can be used to determine (e.g.,
search) for entities, relationships, and/or articles, as described
with reference to FIGS. 7 and/or 8, above.
[0091] The illustrated process begins at block 902, where it
receives an indication of an element of a collection. As noted, in
some embodiments, collections include various types of indications
of singular or plural entities. Thus, the collection element may be
or include a name of one or more entities (e.g., the string "Sean
Connery," "James Bond," "Michael Jackson"), a facet of one or more
entities (e.g., actor, politician, book, film), or a keyterm (e.g.,
one or more words describing one or more entities, including
"soccer player," "good actor," and "prescription drugs").
[0092] At block 904, the process determines whether the element
refers to a known and disambiguated entity ("KDE"), and if so,
continues at block 906, otherwise at block 908. The element refers
to a KDE if it refers to an entity that has been uniquely
identified and/or indexed by the content recommendation system. If
this is the case, then the entity is associated with a unique
identifier (e.g., in the entity data store 217b) that can be used
within a query.
[0093] At block 906, the process constructs a query that includes a
unique identifier associated with the entity. As noted, a KDE is
associated with a unique identifier that can be used in a query.
Thus, constructing the query includes obtaining (e.g., looking up
in the entity data store 217b) the unique identifier and generating
a query (or portion thereof) that includes the identifier. Then,
the process continues at block 920.
[0094] At block 908, the process determines whether the element
refers to a known but not-disambiguated entity ("KNDE"), and if so,
continues at block 910, otherwise at block 912. The element refers
to a KNDE if it refers to one or more entities that are known to
the content recommendation system but have not yet been indexed or
otherwise assigned a unique identifier.
[0095] At block 910, the process constructs a query that includes
the name of the entity and optionally disambiguating terms.
Constructing the query includes determining whether the entity
needs to be disambiguated. In some cases, such as when the entity
name is at least substantially unique (e.g., as determined based on
inverse document frequency, where names with a higher inverse
document frequency are typically more unique), there is no need to
disambiguate the entity, and using the entity name alone may
suffice. Other cases where disambiguation may not be needed include
entities that are emerging popular entities that are currently
widely covered in the news and thus typically do not need
disambiguation. Emerging popular entities can be determined with
reference to various sources, such as Wikipedia page view counts,
Wikipedia in-bound link counts, popular queries on major search
engines, trending topics on social networks (e.g., Twitter), and
the like. If the entity does not need to be disambiguated, the
constructed query will typically include just the entity name.
[0096] Cases where the entity needs to be disambiguated may include
entity names that refer to multiple entities, entity names that are
common names (e.g., "Will Smith"), entities that are not emerging
popular entities, or the like. When the entity needs to be
disambiguated, the constructed query includes the entity name and
one or more disambiguating terms. The disambiguating terms may be
based on the entity's facets and/or related entities. For example,
to query Will Smith the football player (as opposed to Will Smith
the actor), a constructed query could include the string "Will
Smith" and various keyterms that express facets (e.g., "football
player") and/or entities (e.g., "New Orleans Saints," "Super Bowl")
that are associated with and/or related to the entity Will Smith
the football player. After constructing the query, the process
continues at block 920.
[0097] At block 912, the process determines whether the element is
a keyterm, and if so, continues at block 914, otherwise at block
916. At block 914, the process constructs a query that includes all
or some of the keyterm. In some embodiments, variations and/or
stems of the keyterms may be included in the query. Then, the
process continues at block 920.
[0098] At block 916, the process determines whether the element is
a facet, and if so, continues at block 918, otherwise at block 922.
In some embodiments, a facet may be identified by particular
syntax, such as square brackets (e.g., [actor], [football_player],
[athlete]). At block 918, the process constructs a query that
includes a facet token. A facet token identifies the facet such
that articles that cover or otherwise refer to the facet can be
efficiently obtained. In some embodiments, articles are annotated
with one or more facet tokens prior to indexing, such that articles
can searched based on facet coverage. The one or more facet tokens
that annotate an article may be limited to a predetermined number
of facets, such as those associated with the primary (e.g., most
relevant or important) entities referenced in an article. In this
manner, the content recommendation system is configured to return
articles that are substantially related to queried facet. Then, the
process continues at block 920.
[0099] At block 922, the process provides an indication of an
unknown element type. At this point, the process has been unable to
match any of the known element types, and returns an error or other
indication (e.g., raises an exception) that the collection element
is not of a recognized type.
[0100] At block 920, the process provides the constructed query.
Providing the constructed query includes returning all or part of a
query string that can be used by another process (e.g., the article
recommendation process of FIG. 7) to search for content items such
as articles or entities.
[0101] After blocks 920 and 922, the process returns. Some
embodiments perform one or more operations/aspects in addition to,
or instead of, the ones described with reference to the process of
FIG. 9. Also, in some embodiments, not all of the discussed
collection element types are supported or required in every
embodiment. For example, the process may handle only known
disambiguated entities (KDEs) and known but not-disambiguated
entities (KNDEs).
[0102] FIG. 10 is an example flow diagram of an article scoring
process performed by an example content recommendation system
according to an example embodiment. In particular, FIG. 10
illustrates a process that may be implemented by, for example, one
or more elements of the content recommendation system 200, as
described with reference to FIG. 2. The process scores an article
(e.g., determines a weight or rank for an article) for purposes of
ordering multiple articles during article recommendation, as
described with reference to FIG. 7, above.
[0103] The illustrated process begins at block 1002, where it
receives an indication of an article and a query.
[0104] At block 1004, the process determines a relevance score
based on one or more query terms. Determining the relevance score
includes calculating a score for the article with respect the
query, based on one or more information retrieval relevance
metrics. One such metric is the product of term document frequency
and inverse document frequency, where term document frequency is
the number of times a term from the query appears in the article,
and inverse document frequency is the logarithm of the quotient of
the total number of documents in a corpus (e.g., the document store
217a) of the content recommendation system and the number of
documents in the corpus that contain the term.
[0105] At block 1006, the process determines a credibility score
for the article. In some embodiments, the article credibility score
can be determined based on properties of the source of the article.
For example, if the source of the article is a Website that
receives a high volume of traffic and/or is ranked highly by one or
more search engines (e.g., via Google's PageRank or similar
metrics), the article may be considered more credible than an
article having a source Website with minimal traffic or low search
engine rank.
[0106] At block 1008, the process determines a recency score for
the article. In one embodiment, the recency score is based on the
difference between the article date (e.g., publication date, last
edit date) and the date of article ingestion by the content
recommendation system, such that articles having a smaller such
difference will have a greater recency score. In other embodiments,
recency score may be based on individual factors, such as ingestion
date or publication date, taken alone.
[0107] At block 1010, the process determines an overall score based
on the relevance score, credibility score, and recency score. In
one embodiment, the overall score is computed by multiplying the
relevance score, credibility score, and recency score together,
along with appropriate scaling or normalization factors.
[0108] After block 1010, the process returns. Some embodiments
perform one or more operations/aspects in addition to, or instead
of, the ones described with reference to the process of FIG. 10.
For example, in some embodiments, the process does not compute its
own article relevance score. Instead, such a score is computed by
an underlying document search and retrieval system, and the process
instead calculates an article boosting score based on the product
of the credibility score and recency score. The document boosting
score is then passed along to the underlying document search and
retrieval system for purposes of ranking search results. One such
embodiment utilizes the Apache Lucene document search
library/system, as described in more detail below.
[0109] FIGS. 11A-11E illustrate example screen displays generated
by queries executed for content recommendation provided by an
example embodiment of a content recommendation system. In
particular, FIGS. 11A-11E illustrate search results provided by an
example content recommendation system in response to received
entity relationship queries, such as may be generated by the entity
recommendation process of FIG. 8, above. For illustrative purposes,
the entities and relationships of FIGS. 11A-11E are presented and
described in the context of a human-readable user interface,
whereas an automatic process such as that of FIG. 8 would
programmatically access the same or similar information via an
application program interface or other mechanism.
[0110] FIG. 11A shows a screen 1100 (e.g., a Web page) that
displays search results returned in response to a first search for
all entities related to a specified entity. The screen 1100
includes a search tool used to specify a query 1101 and search
results 1102. The search results 1102 include multiple relationship
records, e.g., records 1106, that each describe one relationship.
In this case, the query 1101 is of the form "Sean Connery" <
> * < > *, which requests all relationships that include
the entity Sean Connery, using a relationship search query
language, such as "EQL" provided by Evri Inc. In response, the
content recommendation system has returned multiple relationship
records, including records 1106. Each of the records includes
indicators of a subject 1103 (e.g., Chris Matthews), action 1104
(e.g., acts like), and object 1105 (e.g., Sean Connery), as well as
an expansion control 1107. In response to a user selection of the
expansion control 1107, the screen 1100 provides a context snippet
1108, which presents textual context taken from the article (or
other content) in which the displayed relationship originally
appeared. In some embodiments, the entity recommendation process of
FIG. 8 makes queries of this sort at block 804.
[0111] FIG. 11B shows search results 1112 returned in response to a
second search for all entities related to a specified entity. The
example of FIG. 11B is similar to that of FIG. 11A, except that in
FIG. 11B, the user has entered a query 1111 of the form "Roger
Moore" < > * < > *. In response, the content
recommendation system has returned search results 1112. In some
embodiments, the entity recommendation process of FIG. 8 makes
queries of this sort at block 804.
[0112] FIG. 11C shows search results 1122 returned in response to a
search for entities in relationships with an entity having a
specified facet, where the relationships appear in context with one
or more specified keyterms. In particular, a user has entered query
1121 ("[actor]< > * < > * context contains "007" or
"Bond actor"), which requests all entities in relationships with an
entity that has the facet actor, where the relationships appear in
context with either the keyterm "007" or the keyterm "Bond actor."
In response, the content recommendation system has returned the
search results 1122. One example relationship record 1123 has been
expanded to show a context snippet 1124 illustrating the presence
of both keyterms "007" and "Bond actor" in the article that
describes the corresponding relationship. Context is typically
defined according to the query language, for example, as some
number of surrounding paragraphs, sentences, words, characters,
etc. In some embodiments, the entity recommendation process of FIG.
8 makes queries of this sort at block 808.
[0113] FIG. 11D shows search results 1132 returned in response to a
first search for entities in relationships with a specified entity,
where the relationships appear in context with one or more
specified entities. In particular, a user has entered query 1131
("Sean Connery" < > * < > * context contains "Roger
Moore"), which requests all entities in relationships with the
entity Sean Connery, where the relationships appear in context with
the entity "Roger Moore." In response, the content recommendation
system has returned the search results 1132 including relationship
records 1133 and 1135. Relationship record 1133 has been expanded
to show a context snippet 1134 illustrating the presence of the
entity "Roger Moore" in the article that describes the
corresponding relationship. In some embodiments, the entity
recommendation process of FIG. 8 makes queries of this sort at
block 810.
[0114] FIG. 11E shows search results 1142 returned in response to a
second search for entities in relationships with a specified
entity, where the relationships appear in context with one or more
specified entities. In particular, a user has entered query 1141
("Roger Moore" < > * < > * context contains "Sean
Connery"), which requests all entities in relationships with the
entity Roger Moore, where the relationships appear in context with
the entity "Sean Connery." In response, the content recommendation
system has returned the search results 1142 including relationship
record 1143. Note that relationship record 1143 describes the same
relationship as relationship record 1135 (FIG. 11D). Typically,
during the entity recommendation process, the content
recommendation system will remove duplicate relationships and/or
entities. For example, prior to providing a set of recommended
entities, the content recommendation system may remove or coalesce
duplicate entities. In some embodiments, the entity recommendation
process of FIG. 8 makes queries of this sort at block 810.
[0115] In the following, additional example techniques for article
recommendation, entity recommendation, and query construction are
discussed.
Article Recommendation in an Example Embodiment
[0116] The following describes an approach to article
recommendation used by one example embodiment. Similar techniques
could be applied to other media, including videos, images,
advertisements, and the like.
[0117] 1. Article Retrieval and Ranking
[0118] For the underlying information retrieval capability, the
content recommendation system of this embodiment uses Lucene, which
is an example of a Vector Space Model (VSM) based information
retrieval system. Lucene retrieves and ranks documents in an index
by determining how relevant a given document is to a user's query.
Additional details regarding Lucene's ranking are available at:
http://lucene.apache.org/java/2_1_0/scoring.html and
http://lucene.apache.org/java/2_1_0/api/org/apache/lucene/search/Similari-
ty.html.
[0119] In general, the more times a query term appears in a
document relative to the number of times the term appears in all
the documents in the collection, the more relevant that document is
to the query. This is accomplished by using the TF-IDF ("term
frequency-inverse term frequency") weighting in the scoring of
documents.
[0120] In addition, Lucene allows influencing of search results by
a process known as "boosting." The content recommendation system
applies document level boosting--while indexing, the function
document.setBoost( ) is invoked before a document is added to the
index (such as may be represented by document store 217a).
[0121] The boost value for each document is computed as a
combination of the document's source credibility and date.
[0122] The content recommendation system prefers to retrieve
documents from credible sources and documents that are recent. In
order to do so, the document recommendation system assigns to each
document source a credibility score (e.g., 1 to 5), which is
translated into a credibility weight. Document credibility can be
assigned manually or computed based on some properties of a source
(e.g., Google PageRank of the source Web page, Internet traffic to
the source Web page, etc.). Credibility weight is multiplied by
recency weight to form document boost.
[0123] Recency weight is based on the difference between the date
of a document and the date when ingestion started. For example:
TABLE-US-00001 TABLE 1 1. int dateDifference = docDate -
ingestionStartDate; 2. if ( dateDifference < -185 )
dateDifference = -66; 3. // Dateboost for docs older than 185 days
is identical 4. else if ( dateDifference < -35 ) 5.
dateDifference = -35 + ( ( dateDifference + 35 ) / 5 ) ; 6. //
Dateboost for docs older than 35 days has 7. // a smaller gradient
8. dateBoost = 1.1 {circumflex over ( )} dateDifference 9.
documentBoost = credibilityBoost * dateBoost
[0124] 2. Article Recommendation
[0125] The goal of article recommendation is to return a set of
articles that cover the set of entities in a collection. The
content recommendation system prefers articles that: cover multiple
entities than just about a single entity; provide more recent
information or news about the entities; and/or are relevant to the
entities (e.g., the articles are about one or more of the entities,
instead of simply mentioning them in passing).
[0126] Also, the content recommendation system prefers that most of
the entities are covered by the returned articles, instead of only
the dominant entities being covered by the returned articles. To do
that, the content recommendation system issues multiple queries.
Each time when constructing a new query, the content recommendation
system removes the entities that have been well covered by articles
from previous queries. This recursive algorithm is described as
below.
[0127] Initialization:
[0128] Set threshold T=number of requested articles
(nReqArticles)/collection size.
[0129] Set maximum number of recursions=N (currently set to 4).
[0130] Start with an empty article set A, and an entity set S that
contains all the entities in the given collection (E1, E2, . . . ,
Ek).
[0131] Step 1:
[0132] Given the entity set S={E1, E2, . . . , Ek}, issue a query
that is a boolean OR of all the entities: e.g., query(E1) OR
query(E2) OR . . . OR query(Ek). See below for a description of how
queries are constructed for each entity according to one
embodiment.
[0133] The query will return a list of articles. The order of
retrieved articles is based on information retrieval weighting,
described above, taking into account the credibility score, the
recency score, term frequency, number of terms in the article,
length of the article, and presence of terms in the title of the
article.
[0134] Then place the first nReqArticles articles to set A. The
content recommendation system does not retrieve all articles
returned by the query because the retrieval of an article is
computationally expensive.
[0135] If a total number of articles returned for the query is
smaller or equal to nReqArticles, jump to Step 3, because shorter
queries will not return additional articles.
[0136] Step 2:
[0137] For each entity in set S, count total number of articles in
set A that include the entity; and if the count exceeds the
threshold T, remove the entity from the set S.
[0138] If the entity set S is not empty AND number of recursions is
less than threshold N, then go to Step 1, otherwise continue to
Step 3.
[0139] Step 3:
[0140] Sort the articles in set A based on a measure that combines
recency of each article (e.g., publishing date) and number of
collection entities covered in the article. Return the sorted
article list.
[0141] As an example of the operation of the above process, if a
user wants to retrieve 6 articles for a collection that includes
Roger Moore and Sean Connery the content recommendation system
submits a query: did123 AND did456, where did123 is a unique id for
Roger Moore, and did456 is a unique id for Sean Connery. If the
underlying information retrieval system (e.g., Lucene) returns
total of 20 articles the content recommendation system retrieves
detailed information about the first 6 articles. It may be found
that 4 articles are about Sean Connery, 1 article is about Roger
Moore, and 1 article is about both of them. As the number of
articles for Sean Connery exceeds the threshold T, he is removed
from the set S and a second query is executed: did123. The content
recommendation system then combines results of the second query
with the results of the first query, taking care that it does not
add articles present in results for both queries twice.
Entity Recommendation in an Example Embodiment
[0142] The following describes an approach to recommending other
entities that might be of interest to the user, as used by one
example embodiment. The goal is to suggest other entities that are
either related to or similar to the entities in the collection. The
content recommendation system attempts to understand the user's
interest based on the existing entities in a collection.
Furthermore, the entity suggestion should adapt to user actions,
such as addition/deletion of entities. The content recommendation
system prefers entities that are related to multiple entities in
the collection and/or entities of the same type as the ones in the
collection.
[0143] Process Summary
[0144] 1) Run independent queries for each entity.
[0145] 2) From search results, determine characteristics of the
collection, e.g., main facets, common keyterms, and the like.
[0146] 3) Run secondary queries to find other entities that are
similar.
[0147] 4) Collect top entities from all query results, put them in
a bucket, and count them.
[0148] 5) Give more weight to entities that match the main facets
in the collection.
[0149] Process Details:
[0150] Step 1:
[0151] Run parallel relationship search queries using each entity
in the collection:
E 1 <> * <> * E 2 <> * <> * ##EQU00001## Ek
<> * <> * ##EQU00001.2##
[0152] The relationship query examples are shown in a syntax
defined according to the relationship query language (EQL) of
Evri.TM. Corporation. Other queries as defined by other languages
may be similarly incorporated. Again, depending on the collection
element type (e.g., a KDE or KNDE), the content recommendation
system either uses the entity's ID or name (plus disambiguating
terms, if necessary) when constructing the query. A discussion of
how queries may be constructed for each entity is described further
below.
[0153] Step 2:
[0154] Iterate through the top N relationship search results from
each query, and collect the following pieces of information:
[0155] 1) Facet counts--frequency count of facets assigned to each
query entity in the returned relationships. The facet refers to the
type or role that an entity has in a given context. For example,
Arnold Schwarzenegger could be referred to as a politician, an
actor, a producer, or a body builder, etc. The facet counts allow
the content recommendation system to identify the main facets
shared by the entities in the given collection.
[0156] 2) Related entity counts--collect other entities
co-occurring in each returned relationship, and count their
occurrences.
[0157] 3) Keyterm counts--count frequency of keyterms (e.g., words
and phrases) that occur in the returned relationships.
[0158] As a result of this step, the content recommendation system
has a list of main facets shared by entities in the collection. The
facets are ranked by their frequency counts.
[0159] Then, the keyterms are ranked by a measure of
(term-frequency * IDF), where IDF is inverse-document frequency.
IDF is a measure of the general importance of the term (obtained by
dividing the number of all documents by the number of documents
containing the term, and then taking the logarithm of that
quotient). The goal is to identify the keyterms that are most
relevant and specific to the given collection.
[0160] Step 3:
[0161] Take the most common facets shared by the entities in the
collection, as well as the most common keyterms, in order to find
other entities that are potentially similar. The content
recommendation system constructs the following query:
TABLE-US-00002 [Facet1] OR [Facet2] <>* <>* CONTEXT
CONTAINS Term1 OR Term2 OR ...... OR TermN
[0162] The "CONTEXT CONTAINS" parameter indicates the terms must
appear within a small window of sentences around the relationship.
In one embodiment, the default window size is 3 sentences. Again,
the content recommendation system iterates through the returned
relationships and collects occurring entities and their frequency
counts.
[0163] Step 4:
[0164] Run parallel joint queries to capture the inter-connection
between entities in the collection:
E 1 <> * <> * CONTEXT CONTAINS E 2 OR E 3 OR Ek
##EQU00002## E 2 <> * <> * CONTEXT CONTAINS E 1 OR E 3
OR Ek ##EQU00002.2## ##EQU00002.3## Ek <> * <> *
CONTEXT CONTAINS E 1 OR E 2 OR Ek - 1 ##EQU00002.4##
[0165] Again, the content recommendation system collects and counts
frequency of other entities occurring in each returned
relationship.
[0166] Step 5:
[0167] Given all the entities collected from Step 2-4, the content
recommendation system places all of them in a bucket, and
accumulates a total frequency count for each entity. Then, for each
entity, the content recommendation system checks whether it shares
the same facet with the ones in the top facet list.
[0168] If yes, the entity weight is boosted by: frequency * facet
score, where the facet score is defined as number of entities in
the collection that have this facet.
[0169] Finally, the content recommendation system sorts the
entities by their weights, and returns a ranked list of suggested
entities
[0170] As one example, the entities in a first collection are Roger
Moore and Sean Connery. The top entities suggested to the user by
the above process are Pierce Brosnan, Timothy Dalton, and Daniel
Craig, all of whom have played the character (entity) James
Bond.
[0171] As another example, the entities in a second collection are
Sean Connery and one of his Bond movies, "Dr. No". Under the above
process, the user will be provided some of the other Bond movies
played by Connery, including "From Russia with Love," "Gold Finger"
and "You Only Live Twice." Once the user selects one of the movies
to add to the collection, more similar movies will show up on the
top of the suggestion list, such as "Thunderball."
[0172] Extensions and Variations
[0173] As discussed, each user-created collection represents
certain information interest or topic that users want to keep
track. The content recommendation system returns the latest news
that is relevant to the collection.
[0174] In addition to the described entity-centric profile pages,
profile pages may be created that are centered around topics,
stories, or events. Those topics will be represented as
collections.
[0175] The definition of collection may be extended to include not
only named entities, but also concepts, key terms and facets. Users
will be able to add concepts (e.g., Global Warming, financial
crisis) and/or arbitrary key terms (words or phrases) to be
included in a collection. The concepts/key-terms are handled
similarly as entities. The above algorithms for recommending
articles and related entities/concepts still apply. Allowing
key-terms or concepts would let users define information request
more precisely. For example, suppose a user is interested in
following any engagement between China and Russia in terms of oil,
a collection could be created including entities China and Russia,
as well as a keyterm "oil."
[0176] In addition, users can add one or more facets to a
collection. This is useful when user is interested in a certain
category of entities, instead of specific entities. For example,
suppose a user wants to keep track of college football Heisman
Trophy candidates and winners, a collection could be created as
(Heisman Trophy, [College_Football_Player]). When searching for
articles, the content recommendation system returns articles that
contain the term "Heisman Trophy" and mentions of any college
football players. When recommending related entities, the content
recommendation system may restrict the suggested entities to belong
to the given facets. In this case, the facets in a collection are
treated as filter.
[0177] The handling of such extensions for the purpose of query
construction is discussed further below.
Query Construction in an Example Embodiment
[0178] In one example embodiment, a collection may comprise of any
combination of: entities (KDE or KNDE), keyterms, or facets. The
following describes on approach to query building for each type of
collection members.
[0179] 1. KDEs ("Known and Disambiguated Entities")
[0180] KDEs are associated with their unique ID in the index.
Therefore, during query time, KDEs can be searched by their ID. For
example, the ID for Sean Connery is 136965. The actual query being
issued for retrieving documents is "did_136965"
[0181] 2. KNDEs ("Known but Not-disambiguated Entities")
[0182] KNDEs are entities that are included in the Entity Store,
but they have not been disambiguated/identified in the index.
During query time, they can only be searched by their name
string.
[0183] For entities that have ambiguous name, such as those that
share a same name with other entities, context constraints can be
included in the query in order to perform disambiguation. The
disambiguating queries are in the form of: [0184] Name (Entity) AND
Disambiguating Terms
[0185] In other words, given information known about the KNDE in
the entity store (e.g., properties of the entity), a query can be
constructed that will return relevant results about the entity.
[0186] First, it is determined whether to apply disambiguating
terms based on the uniqueness of an entity's name. If the name is
substantially unique, then no disambiguation is needed. The
uniqueness is determined based on at least some of the following
factors:
[0187] 1) IDF ("inverse document frequency") weight of the name,
computed from a large corpus; the higher the IDF weight, the more
unique the name.
[0188] 2) If the entity is a person name, determining whether it
has a common family name or given name. The common family and given
names are readily available from external sources.
[0189] 3) The total number of entities in the entity store that
share the same name
[0190] Example pseudo-code for determining whether an entity needs
disambiguation follows:
TABLE-US-00003 TABLE 2 1. needDisambiguation(Entity): 2. If
isCommonPersonName(Entity) = true, return true; 3. Else if
IDF-weight(Entity) > 11, return false; 4. Else if
IDF-weight(Entity) > 8.5 AND 5.
Number_of_entities_with_same_name(Entity) < 3, 6. return false;
7. Else if is_Emerging_Popular_Entity(entity) = true, 8. return
false; 9. Else, return true;
[0191] The goal of the function is_Emerging_Popular_Entity(entity)
is to catch entities that are popular or suddenly emerging (e.g.,
Susan Boyle, balloon boy) all over the news. Emerging popular
entities typically do not need disambiguation.
[0192] Popularity of entities may be determined based on external
sources, such as Wikipedia page view counts (Wikipedia traffic
statistics on hourly basis are publicly available from sources such
as http://stats.grok.se), Wikipedia in-bound link counts (i.e., how
many pages link to the given page), popular queries from major
search engines, trending topics from sources like Twitter, or the
like.
[0193] The function is_Emerging_Popular_Entity(Entity) would return
true if a) its popularity measure exceeds a pre-determined
threshold; and b) the entity has highest popularity score compared
to other entities that have the same name. For example, Will Smith
the football player might be popular due to his team is playing the
2010 Super Bowl, but Will Smith the actor still has higher overall
popularity score.
[0194] Next, for entities that need disambiguation, disambiguation
terms are selected based on the given entity's facet. Suppose the
KNDE is Will Smith, the American football player of the NFL's New
Orleans Saints. It is known that football players are usually
associated with related facets (e.g., Football teams, Football
coaches, Football players) and/or keywords (e.g., NFL, Super bowl,
quarterback, tackle, linebacker, touchdown).
[0195] The above related facets and keyterms can be termed "facet
profiles." Facet profiles can be generated automatically off-line,
using the KDEs that have been tagged in the index. For example, the
entities that have been tagged as football player may be analyzed
to determine which facets and keyterms are mostly correlated to
football players. Of course, facet profiles can also be generated
manually by a human, or by a combination of manual and automated
processes.
[0196] In the entity store, each KNDE is associated with a number
of static properties that are extracted from structured or
semi-structured sources (e.g., Wikipedia). Some of the properties
are links to other entities (e.g., for a football player, the list
of teams he has played for, the college he went to, birthplace,
spouse, etc.). The content recommendation system processes the
linked entities, and selects the ones that match the related facets
above. For example, for Will Smith, the content recommendation
system would select "New Orleans Saints" the football team as a
related entity.
[0197] As the result, the disambiguation terms include
facet-related keywords, plus linked entities. For example, the
query for Will Smith could be:
TABLE-US-00004 "Will Smith" AND ("New Orleans Saints" OR "NFL" OR
"Super bowl" OR "Defensive end" OR "tackle")
[0198] 3. Keyterms
[0199] If a collection entry is a keyterm (word or phrase), the
content recommendation system uses the term as keyword query for
retrieving documents.
[0200] 4. Facets
[0201] When a collection entry is a facet, such as a football
player, one approach is to use the facet directly in the query. In
the Evri Query Language (EQL), the syntax for searching for a facet
is using square bracket, such as [Football_Team]. In the index,
tagged/disambiguated entities are annotated with their facets.
Thus, a facet query like [Football_Team] would return articles that
contain mentions of any football teams.
[0202] This approach may have some drawbacks. The ranking of the
retrieved articles is based on factors such as TF-IDF weighting and
article recency. Therefore, this approach may result in less
relevant articles due to:
[0203] (a) Articles containing many mentions of different entities
with the given facet. For example, for the query [Football_Team],
articles that cover many games and teams would get ranked higher,
but those articles are usually less interesting.
[0204] b) Articles that are very recent but only mention the facet
in passing.
[0205] c) For news articles and blogs on the Web, the facet appears
in news headlines or other boiler-plate part of the web page that
are not related to the main article.
[0206] To address this issue, the content recommendation system may
restrict the facets to the main entities in a given article. In one
approach, during indexing of each article, the content
recommendation system first identifies the top K entities in the
article, then collects all the facets assigned to the top entities,
and then annotates the article with those facets using special
tokens (e.g., "facet.football.team").
[0207] Later at search time, given a facet Football_Team, the
content recommendation system translates it into the query term
"facet.football.team." By doing this, the content recommendation
system obtains articles that tend to be about the facet.
[0208] The top entities in an article are ranked and selected by an
importance measure (e.g., how important an entity is with respect
to the main topic of the article), which is computed based on:
total number of occurrences of each entity in the article, giving
boost to entities appearing in the article title, and giving
penalty to entities that are place names or publisher names.
Example Entity Types
[0209] The following Table defines several example entity types in
an example embodiment. Other embodiments may incorporate different
types.
TABLE-US-00005 TABLE 3 Person Organization Location Concept Event
Product Condition Organism Substance
Example Facets
[0210] The following Table defines several example facets in an
example embodiment. Other embodiments may incorporate different
facets.
TABLE-US-00006 TABLE 4 PERSON actor Evri/Person/Entertainment/Actor
PERSON animator Evri/Person/Entertainment/Animator PERSON
cinematographer Evri/Person/Entertainment/Cinematographer PERSON
comedian Evri/Person/Entertainment/Comedian PERSON fashion_designer
Evri/Person/Entertainment/Fashion_Designer PERSON musician
Evri/Person/Entertainment/Musician PERSON composer
Evri/Person/Entertainment/Musician/Composer PERSON producer
Evri/Person/Entertainment/Producer PERSON director
Evri/Person/Entertainment/Director PERSON radio_personality
Evri/Person/Entertainment/Radio_Personality PERSON
television_personality
Evri/Person/Entertainment/Television_Personality PERSON author
Evri/Person/Entertainment/Author PERSON model
Evri/Person/Entertainment/Model PERSON screenwriter
Evri/Person/Entertainment/Screenwriter PERSON playwright
Evri/Person/Entertainment/Playwright PERSON conductor
Evri/Person/Entertainment/Conductor PRODUCT film
Evri/Product/Entertainment/Movie PRODUCT television_show
Evri/Product/Entertainment/Television_Show PRODUCT album
Evri/Product/Entertainment/Album PRODUCT musical
Evri/Product/Entertainment/Musical PRODUCT book
Evri/Product/Entertainment/Book PRODUCT newspaper
Evri/Product/Publication PERSON politician
Evri/Person/Politics/Politician PERSON cabinet_member
Evri/Person/Politics/Cabinet_Member PERSON government_person
Evri/Person/Politics/Government_Person PERSON
political_party_leader Evri/Person/Politics/Political_Party_Leader
PERSON judge Evri/Person/Politics/Judge PERSON country_leader
Evri/Person/Politics/Politician/World_Leader PERSON
joint_chiefs_of_staff
Evri/Person/Politics/Politician/Joint_Chiefs_of_Staff PERSON
white_house_staff Evri/Person/Politics/White_House_Staff PERSON
activist Evri/Person/Politics/Activist PERSON lobbyist
Evri/Person/Politics/Lobbyist PERSON ambassador
Evri/Person/Politics/Ambassador PERSON analyst Evri/Person/Analyst
PERSON journalist Evri/Person/Journalist PERSON blogger
Evri/Person/Blogger ORGANIZATION band
Evri/Organization/Entertainment/Band ORGANIZATION political_party
Evri/Organization/Politics/Political_Party ORGANIZATION
advocacy_group Evri/Organization/Politics/Advocacy_Group EVENT
film_award_ceremony Evri/Event/Entertainment/Film_Award_Ceremony
EVENT music_award_ceremony
Evri/Event/Entertainment/Music_Award_Ceremony EVENT
television_award_ceremony
Evri/Event/Entertainment/Television_Award_Ceremony EVENT court_case
Evri/Event/Politics/Court_Case ORGANIZATION television_network
Evri/Organization/Entertainment/Company/Television_Network
ORGANIZATION music_production_company
Evri/Organization/Entertainment/Company/Music_Production_Company
ORGANIZATION film_production_company
Evri/Organization/Entertainment/Company/Film_Production_Company
LOCATION congressional_district
Evri/Location/Politics/Congressional_District LOCATION
military_base Evri/Location/Politics/Military_Base ORGANIZATION
congressional_committee
Evri/Organization/Politics/Congressional_Committee ORGANIZATION
international_organization
Evri/Organization/Politics/International_Organization ORGANIZATION
government_agency Evri/Organization/Politics/Government_Agency
ORGANIZATION armed_force Evri/Organization/Politics/Armed_Force
ORGANIZATION terrorist_organization
Evri/Organization/Politics/Terrorist_Organization ORGANIZATION
us_court Evri/Organization/Politics/US_Court ORGANIZATION
cabinet_department Evri/Organization/Politics/Cabinet_Department
LOCATION continent Evri/Location/Continent LOCATION
geographic_region Evri/Location/Geographic_Region LOCATION country
Evri/Location/Country LOCATION province Evri/Location/Province
LOCATION state Evri/Location/State LOCATION city Evri/Location/City
LOCATION us_city Evri/Location/City LOCATION neighborhood
Evri/Location/Neighborhood LOCATION building
Evri/Location/Structure/Building LOCATION island
Evri/Location/Island LOCATION mountain Evri/Location/Mountain
LOCATION body_of_water Evri/Location/Body_of_Water ORGANIZATION
media_companyEvri/Organization/Entertainment/Company/Media_Company
ORGANIZATION haute_couture_house
Evri/Organization/Entertainment/Company/Haute_Couture_House
ORGANIZATION publishing_company
Evri/Organization/Entertainment/Company/Publishing_Company
ORGANIZATION entertainment_company
Evri/Organization/Entertainment/Company CONCEPT fictional_character
Evri/Concept/Entertainment/Fictional_Character PERSON
military_leader Evri/Person/Politics/Military_Leader PERSON
military_person Evri/Person/Politics/Military_Person EVENT
military_conflict Evri/Event/Politics/Military_Conflict PERSON
terrorist Evri/Person/Politics/Terrorist PERSON criminal
Evri/Person/Criminal PERSON explorer Evri/Person/Explorer PERSON
inventor Evri/Person/Technology/Inventor PERSON lawyer
Evri/Person/Lawyer PERSON artist Evri/Person/Artist PERSON painter
Evri/Person/Artist/Painter PERSON revolutionary
Evri/Person/Revolutionary PERSON spiritual_leader
Evri/Person/Spiritual_Leader PERSON philosopher
Evri/Person/Philosopher PERSON anthropologist
Evri/Person/Anthropologist PERSON architect Evri/Person/Architect
PERSON historian Evri/Person/Historian PERSON editor
Evri/Person/Editor PERSON astronaut Evri/Person/Astronaut PERSON
photographer Evri/Person/Photographer PERSON scientist
Evri/Person/Technology/Scientist PERSON economist
Evri/Person/Economist PERSON technology_person
Evri/Person/Technology/Technology_Person PERSON business_person
Evri/Person/Business/Business_Person PERSON stock_trader
Evri/Person/Business/Business_Person/Stock_Trader PERSON first_lady
Evri/Person/Politics/First_Lady ORGANIZATION us_state_legislature
Evri/Organization/Politics/Legislative_Body/State_Legislature
ORGANIZATION legislative_body
Evri/Organization/Politics/Legislative_Body ORGANIZATION
executive_body Evri/Organization/Politics/Executive_Body PERSON
team_owner Evri/Person/Sports/Team_Owner PERSON sports_announcer
Evri/Person/Sports/Sports_Announcer PERSON sports_executive
Evri/Person/Sports/Sports_Executive PERSON olympic_medalist
Evri/Person/Sports/Olympic_Medalist PERSON athlete
Evri/Person/Sports/Athlete PERSON coach Evri/Person/Sports/Coach
PERSON sports_official Evri/Person/Sports/Sports_Official PERSON
motorcycle_driver Evri/Person/Sports/Athlete/Motorcycle_Rider
PERSON race_car_driver Evri/Person/Sports/Athlete/Race_car_Driver
ORGANIZATION auto_racing_team
Evri/Organization/Sports/Auto_Racing_Team PERSON baseball_player
Evri/Person/Sports/Athlete/Baseball_Player ORGANIZATION
baseball_team Evri/Organization/Sports/Baseball_Team PERSON
basketball_player Evri/Person/Sports/Athlete/Basketball_Player
ORGANIZATION basketball_team
Evri/Organization/Sports/Basketball_Team PERSON football_player
Evri/Person/Sports/Athlete/Football_Player ORGANIZATION
football_team Evri/Organization/Sports/Football_Team PERSON
hockey_player Evri/Person/Sports/Athlete/Hockey_Player ORGANIZATION
hockey_team Evri/Organization/Sports/Hockey_Team PERSON
soccer_player Evri/Person/Sports/Athlete/Soccer_Player ORGANIZATION
soccer_team Evri/Organization/Sports/Soccer_Team ORGANIZATION
sports_league Evri/Organization/Sports/Sports_League PERSON
cricketer Evri/Person/Sports/Athlete/Cricketer ORGANIZATION
cricket_team Evri/Organization/Sports/Cricket_Team PERSON cyclist
Evri/Person/Sports/Athlete/Cyclist ORGANIZATION cycling_team
Evri/Organization/Sports/Cycling_Team PERSON volleyball_player
Evri/Person/Sports/Athlete/Volleyball_Player ORGANIZATION
volleyball_team Evri/Organization/Sports/Volleyball_Team PERSON
rugby_player Evri/Person/Sports/Athlete/Rugby_Player ORGANIZATION
rugby_team Evri/Organization/Sports/Rugby_Team PERSON boxer
Evri/Person/Sports/Athlete/Boxer PERSON diver
Evri/Person/Sports/Athlete/Diver PERSON golfer
Evri/Person/Sports/Athlete/Golfer PERSON gymnast
Evri/Person/Sports/Athlete/Gymnast PERSON figure_skater
Evri/Person/Sports/Athlete/Figure_Skater PERSON horse_racing_jockey
Evri/Person/Sports/Athlete/Horse_Racing_Jockey PERSON
lacrosse_player Evri/Person/Sports/Athlete/Lacrosse_Player
ORGANIZATION lacrosse_team Evri/Organization/Sports/Lacrosse_Team
PERSON rower Evri/Person/Sports/Athlete/Rower PERSON swimmer
Evri/Person/Sports/Athlete/Swimmer PERSON tennis_player
Evri/Person/Sports/Athlete/Tennis_Player PERSON
track_and_field_athlete
Evri/Person/Sports/Athlete/Track_and_Field_Athlete PERSON wrestler
Evri/Person/Sports/Athlete/Wrestler PERSON triathlete
Evri/Person/Sports/Athlete/Triathlete EVENT sports_competition
Evri/Event/Sports/Sports_Event/Sporting_Competition EVENT
sports_event Evri/Event/Sports/Sports_Event EVENT olympic_sport
Evri/Event/Sports/Olympic_Sports EVENT election
Evri/Event/Politics/Election LOCATION sports_venue
Evri/Location/Sports/Sports_Venue ORGANIZATION sports_division
Evri/Organization/Sports/Sports_Division ORGANIZATION
sports_event_promotion_company
Evri/Organization/Sports/Sports_Event_Promotion_Company
ORGANIZATION sports_organization
Evri/Organization/Sports/Sports_Organization ORGANIZATION company
Evri/Organization/Business/Company ORGANIZATION news_agency
Evri/Organization/Business/Company/News_Agency PRODUCT cell_phone
Evri/Product/Technology/Cell_Phone PRODUCT computer
Evri/Product/Technology/Computer PRODUCT software
Evri/Product/Technology/Software PRODUCT video_game
Evri/Product/Technology/Software/Video_Game PRODUCT
video_game_console Evri/Product/Technology/Video_Game_Console
PRODUCT media_player Evri/Product/Technology/Media_player
ORGANIZATION website Evri/Organization/Technology/Website
ORGANIZATION technology_company
Evri/Organization/Technology/Company PRODUCT magazine
Evri/Product/Publication ORGANIZATION financial_services_company
Evri/Organization/Business/Company/Financial_Services_Company
ORGANIZATION radio_network
Evri/Organization/Entertainment/Company/Radio_Network ORGANIZATION
futures_exchange Evri/Organization/Business/Futures_Exchange
ORGANIZATION stock_exchange
Evri/Organization/Business/Stock_Exchange ORGANIZATION
government_sponsored_enterprise
Evri/Organization/Politics/Government_Sponsored_Enterprise
ORGANIZATION political_organization
Evri/Organization/Politics/Political_organization ORGANIZATION
labor_union Evri/Organization/Politics/Labor_Union ORGANIZATION
nonprofit_corporation
Evri/Organization/Business/Company/Nonprofit_Corporation
ORGANIZATION nonprofit_organization
Evri/Organization/Nonprofit_Organization ORGANIZATION
national_laboratory Evri/Organization/Politics/National_Laboratory
ORGANIZATION unified_combatant_commands
Evri/Organization/Politics/Unified_Combatant_Commands ORGANIZATION
research_institute Evri/Organization/Research_Institute CONCEPT
stock_market_index Evri/Concept/Business/Stock_Market_Index PERSON
business_executive
Evri/Person/Business/Business_Person/Business_Executive PERSON
corporate_director
Evri/Person/Business/Business_Person/Corporate_Director PERSON
banker Evri/Person/Business/Business_Person/Banker PERSON publisher
Evri/Person/Business/Business_Person/Publisher PERSON us_politician
Evri/Person/Politics/U.S._Politician PERSON nobel_laureate
Evri/Person/Nobel_Laureate PERSON chemist Evri/Person/Chemist
PERSON physicist Evri/Person/Physicist ORGANIZATION
business_organization
Evri/Organization/Business/Business_Organization ORGANIZATION
consumer_organization
Evri/Organization/Business/Consumer_Organization ORGANIZATION
professional_association
Evri/Organization/Business/Professional_Association PERSON investor
Evri/Person/Business/Business_Person/Investor PERSON financier
Evri/Person/Business/Business_Person/Financier PERSON money_manager
Evri/Person/Business/Business_Person/Money_Manager ORGANIZATION
aerospace_company
Evri/Organization/Business/Company/Aerospace_Company ORGANIZATION
advertising_agency
Evri/Organization/Business/Company/Advertising_Company ORGANIZATION
agriculture_company
Evri/Organization/Business/Company/Agriculture_Company ORGANIZATION
airline Evri/Organization/Business/Company/Airline ORGANIZATION
architecture_firm
Evri/Organization/Business/Company/Architecture_Firm ORGANIZATION
automotive_company
Evri/Organization/Business/Company/Automotive_Company ORGANIZATION
chemical_company
Evri/Organization/Business/Company/Chemical_Company ORGANIZATION
clothing_company
Evri/Organization/Business/Company/Clothing_Company ORGANIZATION
consulting_company
Evri/Organization/Business/Company/Consulting_Company ORGANIZATION
cosmetics_company
Evri/Organization/Business/Company/Cosmetics_Company ORGANIZATION
defense_company Evri/Organization/Business/Company/Defense_Company
ORGANIZATION distribution_company
Evri/Organization/Business/Company/Distribution_Company
ORGANIZATION gaming_company
Evri/Organization/Business/Company/Gaming_Company ORGANIZATION
electronics_company
Evri/Organization/Business/Company/Electronics_Company ORGANIZATION
energy_company Evri/Organization/Business/Company/Energy_Company
ORGANIZATION hospitality_company
Evri/Organization/Business/Company/Hospitality_Company ORGANIZATION
insurance_company
Evri/Organization/Business/Company/Insurance_Company ORGANIZATION
law_firm Evri/Organization/Business/Company/Law_Firm ORGANIZATION
manufacturing_company
Evri/Organization/Business/Company/Manufacturing_Company
ORGANIZATION mining_company
Evri/Organization/Business/Company/Mining_Company ORGANIZATION
pharmaceutical_company
Evri/Organization/Business/Company/Pharmaceutical_Company
ORGANIZATION railway_company
Evri/Organization/Business/Company/Railway ORGANIZATION
real_estate_company
Evri/Organization/Business/Company/Real_Estate_Company ORGANIZATION
retailer Evri/Organization/Business/Company/Retailer ORGANIZATION
shipping_company
Evri/Organization/Business/Company/Shipping_Company ORGANIZATION
software_company
Evri/Organization/Technology/Company/Software_Company ORGANIZATION
steel_company Evri/Organization/Business/Company/Steel_Company
ORGANIZATION telecommunications_company
Evri/Organization/Business/Company/Telecommunications_Company
ORGANIZATION utilities_company
Evri/Organization/Business/Company/Utilities_Company ORGANIZATION
wholesaler Evri/Organization/Business/Company/Wholesaler
ORGANIZATION television_production_company
Evri/Organization/Entertainment/Company/Television_Production_Company
ORGANIZATION food_company
Evri/Organization/Business/Company/Food_Company ORGANIZATION
beverage_company
Evri/Organization/Business/Company/Food_Company/Beverage_Company
ORGANIZATION restaurant
Evri/Organization/Business/Company/Food_Company/Restaurant
ORGANIZATION winery
Evri/Organization/Business/Company/Food_Company/Beverage_Company
EVENT film_festival Evri/Event/Entertainment/Film_Festival
ORGANIZATION film_festival Evri/Event/Entertainment/Film_Festival
PRODUCT anime Evri/Product/Entertainment/Anime PRODUCT aircraft
Evri/Product/Aircraft PRODUCT military_aircraft
Evri/Product/Aircraft/Military_Aircraft PRODUCT vehicle
Evri/Product/Vehicle PRODUCT ballet
Evri/Product/Entertainment/Ballet PRODUCT opera
Evri/Product/Entertainment/Opera PRODUCT painting
Evri/Product/Entertainment/Painting PRODUCT song
Evri/Product/Entertainment/Single EVENT technology_conference
Evri/Event/Technology/Technology_Conference CONCEPT legislation
Evri/Concept/Politics/Legislation CONCEPT treaty
Evri/Concept/Politics/Treaty ORGANIZATION trade_association
Evri/Organization/Business/Trade_Association ORGANIZATION
technology_organization
Evri/Organization/Technology/Technology_Organization ORGANIZATION
educational_institution Evri/Organization/Educational_Institution
LOCATION museum Evri/Location/Structure/Building/Museum LOCATION
religious_building
Evri/Location/Structure/Building/Religious_Building PERSON
astronomer Evri/Person/Astronomer PERSON mathematician
Evri/Person/Mathematician PERSON academic Evri/Person/Academic
PERSON dancer Evri/Person/Entertainment/Dancer PRODUCT play
Evri/Product/Entertainment/Play LOCATION botanical_garden
Evri/Location/Botanical_Garden LOCATION hospital
Evri/Location/Health/Hospital PERSON psychiatrist
Evri/Person/Health/Psychiatrist PERSON physician
Evri/Person/Health/Physician PERSON nurse Evri/Person/Health/Nurse
ORGANIZATION journalism_organization
Evri/Organization/Journalism_Organization ORGANIZATION
healthcare_company
Evri/Organization/Business/Company/Healthcare_Company ORGANIZATION
religious_organization Evri/Organization/Religious_Organization
PERSON biologist Evri/Person/Scientist/Biologist PERSON biochemist
Evri/Person/Scientist/Biochemist PERSON botanist
Evri/Person/Scientist/Botanist PERSON poet
Evri/Person/Entertainment/Author/Poet PERSON curler
Evri/Person/Sports/Athlete/Curler PERSON biathlete
Evri/Person/Sports/Athlete/Biathlete PERSON alpine_skier
Evri/Person/Sports/Athlete/Alpine_Skier PERSON cross-country_skier
Evri/Person/Sports/Athlete/Cross-country_Skier PERSON
freestyle_skier Evri/Person/Sports/Athlete/Freestyle_Skier PERSON
luger Evri/Person/Sports/Athlete/Luger PERSON nordic_combined_skier
Evri/Person/Sports/Athlete/Nordic_Combined_Skier PERSON
speed_skater Evri/Person/Sports/Athlete/Speed_Skater PERSON
skeleton_racer Evri/Person/Sports/Athlete/Skeleton_Racer PERSON
ski_jumper Evri/Person/Sports/Athlete/Ski_Jumper PERSON snowboarder
Evri/Person/Sports/Athlete/Snowboarder PERSON bobsledder
Evri/Person/Sports/Athlete/Bobsledder PERSON bodybuilder
Evri/Person/Sports/Athlete/Bodybuilder PERSON equestrian
Evri/Person/Sports/Athlete/Equestrian PERSON fencer
Evri/Person/Sports/Athlete/Fencer PERSON hurler
Evri/Person/Sports/Athlete/Hurler PERSON martial_artist
Evri/Person/Sports/Athlete/Martial_Artist PERSON canoer
Evri/Person/Sports/Athlete/Canoer LOCATION music_venue
Evri/Location/Entertainment/Music_Venue LOCATION aquarium
Evri/Location/Aquarium LOCATION cemetery Evri/Location/Cemetery
LOCATION national_park Evri/Location/National_Park LOCATION volcano
Evri/Location/Volcano LOCATION zoo Evri/Location/Zoo LOCATION
structure Evri/Location/Structure LOCATION airport
Evri/Location/Structure/Airport LOCATION bridge
Evri/Location/Structure/Bridge LOCATION hotel
Evri/Location/Structure/Hotel LOCATION palace
Evri/Location/Structure/Palace LOCATION monument
Evri/Location/Structure/Monument LOCATION street
Evri/Location/Street LOCATION amusement_park
Evri/Location/Amusement_Park LOCATION unitary_authority
Evri/Location/Unitary_Authority PRODUCT drug_brand
Evri/Product/Health/Drug_Brand PRODUCT weapon Evri/Product/Weapon
PRODUCT missile_system Evri/Product/Weapon/Missile_System PRODUCT
firearm Evri/Product/Weapon/Firearm PRODUCT artillery
Evri/Product/Weapon/Artillery PRODUCT anti-aircraft_weapon
Evri/Product/Weapon/Anti-aircraft_Weapon PRODUCT anti-tank_weapon
Evri/Product/Weapon/Anti-tank_Weapon PRODUCT biological_weapon
Evri/Product/Weapon/Biological_Weapon PRODUCT chemical_weapon
Evri/Product/Weapon/Chemical_Weapon CHEMICAL chemical_weapon
Evri/Product/Weapon/Chemical_Weapon SUBSTANCE chemical_weapon
Evri/Product/Weapon/Chemical_Weapon PRODUCT explosive
Evri/Product/Weapon/Explosive PRODUCT weapons_launcher
Evri/Product/Weapon/Weapons_Launcher PERSON chess_player
Evri/Person/Chess_Player PERSON sculptor
Evri/Person/Artist/Sculptor PRODUCT game Evri/Product/Game
ORGANIZATION theater_company
Evri/Organization/Entertainment/Company/Theater_Company PERSON
badminton_player Evri/Person/Sports/Athlete/Badminton_Player
PRODUCT naval_ship Evri/Product/Watercraft/Naval_Ship PRODUCT
battleship Evri/Product/Watercraft/Naval_Ship/Battleship PRODUCT
cruiser Evri/Product/Watercraft/Naval_Ship/Cruiser PRODUCT
aircraft_carrier
Evri/Product/Watercraft/Naval_Ship/Aircraft_Carrier PRODUCT
destroyer Evri/Product/Watercraft/Naval_Ship/Destroyer PRODUCT
frigate Evri/Product/Watercraft/Naval_Ship/Frigate PRODUCT
submarine Evri/Product/Watercraft/Naval_Ship/Submarine PRODUCT
cruise_ship Evri/Product/Watercraft/Cruise_Ship PRODUCT yacht
Evri/Product/Watercraft/Yacht PRODUCT ocean_liner
Evri/Product/Watercraft/Ocean_Liner LOCATION county
Evri/Location/County PRODUCT symphony
Evri/Product/Entertainment/Symphony ORGANIZATION television_station
Evri/Organization/Entertainment/Company/Television_Station
ORGANIZATION radio_station
Evri/Organization/Entertainment/Company/Radio_Station CONCEPT
constitutional_amendment
Evri/Concept/Politics/Constitutional_Amendment PERSON
australian_rules_footballer
Evri/Person/Sports/Athlete/Australian_Rules_Footballer ORGANIZATION
australian_rules_football_team
Evri/Organization/Sports/Australian_Rules_Football_Team
ORGANIZATION criminal_organization
Evri/Organization/Criminal_Organization PERSON poker_player
Evri/Person/Poker_Player PERSON bowler
Evri/Person/Sports/Athlete/Bowler PERSON yacht_racer
Evri/Person/Sports/Athlete/Yacht_Racer PERSON water_polo player
Evri/Person/Sports/Athlete/Water_Polo_Player PERSON
field_hockey_player Evri/Person/Sports/Athlete/Field_Hockey_Player
PERSON skateboarder Evri/Person/Sports/Athlete/Skateboarder PERSON
polo_player Evri/Person/Sports/Athlete/Polo_Player PERSON
gaelic_footballer Evri/Person/Sports/Athlete/Gaelic_Footballer
PRODUCT programming_language
Evri/Product/Technology/Programming_Language PERSON engineer
Evri/Person/Technology/Engineer EVENT cybercrime
Evri/Event/Technology/Cybercrime EVENT criminal_act
Evri/Event/Criminal_Act PERSON critic Evri/Person/Critic PERSON
pool_player Evri/Person/Pool_Player PERSON snooker_player
Evri/Person/Snooker_Player PERSON competitive_eater
Evri/Person/Competitive_Eater PRODUCT data_storage_medium
Evri/Product/Technology/Data_Storage_Medium PRODUCT
data_storage_device Evri/Product/Technology/Data_Storage_Device
PERSON mountain_climber Evri/Person/Mountain_Climber PERSON aviator
Evri/Person/Aviator ORGANIZATION cooperative
Evri/Organization/Cooperative CONCEPT copyright_license
Evri/Concept/Copyright_License EVENT observance
Evri/Event/Observance PERSON outdoor_sportsperson
Evri/Person/Sports/Outdoor_Sportsperson PERSON rodeo_performer
Evri/Person/Sports/Rodeo_Performer PERSON sports_shooter
Evri/Person/Sports/Athlete/Sports_Shooter CONCEPT award
Evri/Concept/Award CONCEPT entertainment_series
Evri/Concept/Entertainment/Entertainment_Series PERSON chef
Evri/Person/Chef PERSON cartoonist
Evri/Person/Entertainment/Cartoonist PERSON comics_creator
Evri/Person/Entertainment/Comics_Creator PERSON nobility
Evri/Person/Nobility PERSON porn_star Evri/Person/Porn_Star PERSON
archaeologist Evri/Person/Scientist/Archaeologist PERSON
paleontologist Evri/Person/Scientist/Paleontologist PERSON
victim_of_crime Evri/Person/Victim_of_Crime LOCATION region
Evri/Location/Region PERSON linguist Evri/Person/Linguist PERSON
librarian Evri/Person/Librarian PERSON bridge_player
Evri/Person/Bridge_Player PERSON choreographer
Evri/Person/Entertainment/Choreographer PRODUCT camera
Evri/Product/Technology/Camera PRODUCT publication
Evri/Product/Publication PRODUCT comic
Evri/Product/Entertainment/Comic PRODUCT short_story
Evri/Product/Entertainment/Short_Story ORGANIZATION
irregular_military_organization
Evri/Organization/Irregular_Military_Organization SUBSTANCE
chemical_element Evri/Substance/Chemical_Element SUBSTANCE alkaloid
Evri/Substance/Organic_Compound/Alkaloid SUBSTANCE glycoside
Evri/Substance/Glycoside SUBSTANCE amino_acid
Evri/Substance/Amino_Acid SUBSTANCE protein Evri/Substance/Protein
SUBSTANCE enzyme Evri/Substance/Enzyme SUBSTANCE hormone
Evri/Substance/Hormone SUBSTANCE hydrocarbon
Evri/Substance/Organic_Compound/Hydrocarbon SUBSTANCE
inorganic_compound Evri/Substance/Inorganic_Compound SUBSTANCE
lipid Evri/Substance/Organic_Compound/Lipid SUBSTANCE steroid
Evri/Substance/Organic_Compound/Lipid/Steroid SUBSTANCE molecule
Evri/Substance/Molecule SUBSTANCE polymer
Evri/Substance/Molecule/Polymer SUBSTANCE terpene
Evri/Substance/Organic Compound/Terpene SUBSTANCE toxin
Evri/Substance/Toxin SUBSTANCE antibiotic
Evri/Substance/Health/Antibiotic SUBSTANCE antioxidant
Evri/Substance/Health/Antioxidant SUBSTANCE anti-inflammatory
Evri/Substance/Health/Anti-inflammatory SUBSTANCE
antiasthmatic_drug Evri/Substance/Health/Antiasthmatic_drug
SUBSTANCE anticonvulsant Evri/Substance/Health/Anticonvulsant
SUBSTANCE antihistamine Evri/Substance/Health/Antihistamine
SUBSTANCE antihypertensive Evri/Substance/Health/Antihypertensive
SUBSTANCE antiviral Evri/Substance/Health/Antiviral SUBSTANCE
painkiller Evri/Substance/Health/Painkiller SUBSTANCE Painkiller
Evri/Substance/Health/Painkiller SUBSTANCE anesthetic
Evri/Substance/Health/Anesthetic SUBSTANCE antibody
Evri/Substance/Antibody SUBSTANCE chemotherapeutic_drug
Evri/Substance/Health/Chemotherapeutic SUBSTANCE anti-diabetic_drug
Evri/Substance/Health/Anti-diabetic SUBSTANCE antianginal_drug
Evri/Substance/Health/Antianginal SUBSTANCE muscle_relaxant
Evri/Substance/Health/Muscle_relaxant SUBSTANCE hypolipidemic_drug
Evri/Substance/Health/Hypolipidemic_Drug SUBSTANCE
psychoactive_drug Evri/Substance/Health/Psychoactive_Drug SUBSTANCE
vaccine Evri/Substance/Health/Vaccine SUBSTANCE
gastrointestinal_drug Evri/Substance/Health/Gastrointestinal_Drug
SUBSTANCE erectile_dysfunction_drug
Evri/Substance/Health/Erectile_Dysfunction_Drug SUBSTANCE
organometallic_compound
Evri/Substance/Organic_Compound/Organometallic_Compound SUBSTANCE
phenol Evri/Substance/Organic_Compound/Phenol SUBSTANCE ketone
Evri/Substance/Organic_Compound/Ketone SUBSTANCE amide
Evri/Substance/Organic_Compound/Amide
SUBSTANCE ester Evri/Substance/Organic_Compound/Ester SUBSTANCE
ether Evri/Substance/Organic_Compound/Ether SUBSTANCE
heterocyclic_compound
Evri/Substance/Organic_Compound/Heterocyclic_Compound SUBSTANCE
organic_compound Evri/Substance/Organic_Compound SUBSTANCE
carbohydrate Evri/Substance/Organic_Compound/Carbohydrate SUBSTANCE
peptide Evri/Substance/Organic_Compound/Peptide SUBSTANCE
organohalide Evri/Substance/Organic_Compound/Organohalide SUBSTANCE
organosulfur_compound
Evri/Substance/Organic_Compound/Organosulfur_Compound SUBSTANCE
aromatic_compound Evri/Substance/Organic_Compound/Aromatic_Compound
SUBSTANCE carboxylic_acid
Evri/Substance/Organic_Compound/Carboxylic_Acid SUBSTANCE
nucleic_acid Evri/Substance/Nucleic_Acid SUBSTANCE ion
Evri/Substance/Ion ORGANISM cyanobacterium
Evri/Organism/Health/Cyanobacterium ORGANISM
gram-positive_bacterium
Evri/Organism/Health/Gram-positive_Bacterium ORGANISM
gram-negative_bacterium
Evri/Organism/Health/Gram-negative_Bacterium ORGANISM
acid-fast_bacterium Evri/Organism/Health/Acid-fast_Bacterium
ORGANISM dna_virus Evri/Organism/Health/DNA_Virus ORGANISM
rna_virus Evri/Organism/Health/RNA_Virus CONDITION symptom
Evri/Condition/Health/Symptom CONDITION injury
Evri/Condition/Health/Injury CONDITION inflammation
Evri/Condition/Health/Inflammation CONDITION disease
Evri/Condition/Health/Disease CONDITION cancer
Evri/Condition/Health/Disease/Cancer ORGANISM medicinal_plant
Evri/Organism/Health/Medicinal_Plant ORGANISM poisonous_plant
Evri/Organism/Poisonous_Plant ORGANISM herb Evri/Organism/Herb
CONCEPT medical_procedure Evri/Concept/Health/Medical_Procedure
ORGANISM bacterium Evri/Organism/Health/Bacterium ORGANISM virus
Evri/Organism/Health/Virus ORGANISM horse Evri/Organism/Horse
PERSON fugitive Evri/Person/Fugitive ORGANIZATION military_unit
Evri/Organization/Politics/Military_Unit ORGANIZATION
law_enforcement_agency
Evri/Organization/Politics/Law_Enforcement_Agency LOCATION
golf_course Evri/Location/Golf_Course PERSON law_enforcement_agent
Evri/Person/Politics/Law_Enforcement_Agent PERSON magician
Evri/Person/Entertainment/Magician LOCATION educational_institution
Evri/Organization/Educational_Institution CONCEPT social_program
Evri/Concept/Politics/Social_Program EVENT international_conference
Evri/Event/Politics/International_Conference
[0211] All of the above U.S. patents, U.S. patent application
publications, U.S. patent applications, foreign patents, foreign
patent applications and non-patent publications referred to in this
specification and/or listed in the Application Data Sheet,
including but not limited to U.S. patent application Ser. No.
13/038,192, entitled "CONTENT RECOMMENDATION BASED ON COLLECTIONS
OF ENTITIES," filed Mar. 1, 2011; and U.S. Provisional Patent
Application No. 61/309,318, entitled "CONTENT RECOMMENDATION BASED
ON COLLECTIONS OF ENTITIES," filed Mar. 1, 2010, are incorporated
herein by reference, in their entireties.
[0212] From the foregoing it will be appreciated that, although
specific embodiments have been described herein for purposes of
illustration, various modifications may be made without deviating
from the spirit and scope of this disclosure. For example, the
methods, techniques, and systems for content recommendation are
applicable to other architectures. For example, instead of
utilizing a Vector Space Model of document indexing, systems that
are programmed to perform natural language processing (e.g., parts
of speech tagging) can be employed. Also, the methods, techniques,
and systems discussed herein are applicable to differing query
languages, protocols, communication media (optical, wireless,
cable, etc.) and devices (such as wireless handsets, mobile
communications devices, electronic organizers, personal digital
assistants, portable email machines, game machines, pagers,
navigation devices such as GPS receivers, etc.).
* * * * *
References