U.S. patent application number 12/944842 was filed with the patent office on 2011-08-04 for method for human ranking of search results.
This patent application is currently assigned to ADVANCED RESEARCH LLC. Invention is credited to Bryant Christopher Lee.
Application Number | 20110191327 12/944842 |
Document ID | / |
Family ID | 44342515 |
Filed Date | 2011-08-04 |
United States Patent
Application |
20110191327 |
Kind Code |
A1 |
Lee; Bryant Christopher |
August 4, 2011 |
Method for Human Ranking of Search Results
Abstract
A search engine may perform a search for a user search query. A
rankings of result items may be returned as a result of the search.
The user may enter new suggested rankings for some of the items.
The new suggested rankings may change the rankings of result items
when other users search for the same search query at a later
time.
Inventors: |
Lee; Bryant Christopher;
(Cambridge, MA) |
Assignee: |
ADVANCED RESEARCH LLC
Bethesda
MD
|
Family ID: |
44342515 |
Appl. No.: |
12/944842 |
Filed: |
November 12, 2010 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61299994 |
Jan 31, 2010 |
|
|
|
61302117 |
Feb 6, 2010 |
|
|
|
Current U.S.
Class: |
707/723 ;
707/E17.014 |
Current CPC
Class: |
G06F 16/00 20190101 |
Class at
Publication: |
707/723 ;
707/E17.014 |
International
Class: |
G06F 17/30 20060101
G06F017/30 |
Claims
1. A method for human ranking of search engine results, the method
performed by a computer system having access to a database, the
method comprising: receiving at the computer system a search query
from a user; wherein the search query is provided by the user
either at the computer system or at a client computing device
connected to the computer system over a network; performing a
search over a set of data in response to the search query, the set
of data being at the computer system or accessible to the computer
system; generating a rankings of a set of result items returned
from the search over the set of data; adjusting the rankings of the
set of result items by a method of using human suggested rankings
comprising: retrieving from the database a previously suggested
rankings of result items that were provided by a other users;
wherein the previously suggested rankings of result items contains
suggested rankings for zero, one, or more of the result items in
the set of result items; wherein the previously suggested rankings
of result items in the database were provided by the other users,
the other users having conducted a search using the same search
query earlier in time; wherein the previously suggested rankings of
result items in the database were provided by the other users by
entering suggested rankings of result items, the result items
having been displayed in a prior results document that was returned
after the other users conducted a search using the same search
query earlier in time; adjusting the rankings of the set of result
items based on the retrieved previously suggested rankings of
result items that were provided by the other users to obtain an
adjusted rankings of the set of result items; displaying the
adjusted rankings of the set of result items in a results document,
wherein the user may enter suggested rankings for zero, one, or
more result items in the results document, such suggested rankings
to be stored in the database to be available for potential later
use in adjusting the rankings of a later set of result items
returned in response to a later search queries of a later
users.
2. The method of claim 1, wherein adjusting the rankings of the set
of result items based on the retrieved previously suggested
rankings of result items from the other users to obtain an adjusted
rankings of the set of result items is performed by a method
comprising: re-ranking one or more result items in the set of
result items to the last user-suggested ranking the one or more
result items received from the other users, the other users having
conducted a search using the same search query earlier in time.
3. The method of claim 1, wherein adjusting the rankings of the set
of result items based on the retrieved previously suggested
rankings of result items from the other users to obtain an adjusted
rankings of the set of result items is performed by a method
comprising: re-ranking one or more result items in the set of
result items to an average user-suggested ranking the one or more
result items have received from the other users, the other users
having conducted a search using the same search query earlier in
time.
4. The method of claim 1, wherein adjusting the rankings of the set
of result items based on the retrieved previously suggested
rankings of result items from the other users to obtain an adjusted
rankings of the set of result items is performed by a method
comprising: assigning two weights, the first weight being
determined based on user-suggested rankings one or more result
items in the set of result items have received from the other
users, the other users having conducted a search using the same
search query earlier in time, and the second weight being
determined by the content, link structure, relevancy, quality, or
other substantive qualities of one or more result items in the set
of result items, and the two weights being used in combination to
adjust the rankings of the one or more result items in the set of
result items.
5. The method of claim 1, wherein the results document includes a
discussion component for user discussion or a link to a discussion
component for user discussion, the discussion component for user
discussion comprising a wiki, a forum, a bulletin board, a chat
room, or other interface component configured for user
discussion.
6. The method of claim 1, wherein the results document includes a
version control component that displays information about one or
more previous versions of the results document or a link to a
version control component that displays information about one or
more previous versions of the results document; wherein each of the
one or more previous versions of the results document display a
past adjusted rankings of the set of result items, the past
adjusted rankings of the set of result items being generated at the
current time or having been generated at a previous time by a
method of using human suggested rankings comprising: retrieving
from the database a previously suggested rankings of result items
that were provided by a other users; wherein the previously
suggested rankings of result items contains suggested rankings for
zero, one, or more of the result items in the set of result items;
wherein the previously suggested rankings of result items in the
database were provided by the other users, the other users having
conducted a search using the same search query earlier in time;
wherein the previously suggested rankings of result items in the
database were provided by the other users by entering suggested
rankings of result items, the result items having been displayed in
a prior results document that was returned after the other users
conducted a search using the same search query earlier in time;
wherein the other users entered the suggested rankings of result
items prior to a specific time, the specific time being associated
with the one or more previous versions of the results document;
adjusting the rankings of the set of result items based on the
retrieved previously suggested rankings of result items that were
provided by the other users to obtain an adjusted rankings of the
set of result items.
7. The method of claim 6, wherein the user may choose to view a
differences between two or more previous versions of the results
document.
8. The method of claim 1, wherein the results document includes
user interface elements next to one or more result items; wherein
the user may enter suggested rankings for the one or more result
items via the user interface elements.
9. The method of claim 1, wherein the user may enter suggested
rankings for one or more result items listed on the results
document by selecting and moving the one or more result items to a
higher or lower ranking.
10. The method of claim 1, wherein the user may only enter
suggested rankings for the result items in the results document if
the user has editing permissions.
11. The method of claim 1, wherein the user must pass a test to
show he or she is a human and not a computer program before the
user can enter suggested rankings for the result items in the
results document.
12. The method of claim 1, wherein the user may only enter
suggested rankings for the result items in the results document a
limited number of times over a limited time period.
13. The method of claim 1, wherein the search query received from
the user is one search query from a set of merged search queries;
wherein adjusting the rankings of the set of result items by a
method of using human suggested rankings is instead performed by a
method comprising: retrieving from the database a previously
suggested rankings of result items that were provided by a other
users; wherein the previously suggested rankings of result items
contains suggested rankings for zero, one, or more of the result
items in the set of result items; wherein the previously suggested
rankings of result items in the database were provided by the other
users, the other users having conducted a search using any search
query from the set of merged search queries earlier in time;
wherein the previously suggested rankings of result items in the
database were provided by the other users by entering suggested
rankings of result items, the result items having been displayed in
a prior results document that was returned after the other users
conducted a search using any search query from the set of merged
search queries earlier in time; adjusting the rankings of the set
of result items based on the retrieved previously suggested
rankings of result items that were provided by the other users to
obtain an adjusted rankings of the set of result items.
14. The method of claim 13, wherein the results document includes a
merge component that displays information about one or more sets of
merged search queries or a link to a merge component that displays
information about one or more sets of merged search queries.
15. The method of claim 13, wherein the results document includes
user interface elements that may be used to add one or more search
queries to a set of merged search queries or to remove one or more
search queries from a set of merged search queries.
16. The method of claim 1, wherein the user and the other users are
divided into one or more classes; wherein the user is in a class;
wherein adjusting the rankings of the set of result items by a
method of using human suggested rankings is instead performed by a
method comprising: retrieving from the database a previously
suggested rankings of result items that were provided by a other
users; wherein the previously suggested rankings of result items
contains suggested rankings for zero, one, or more of the result
items in the set of result items; wherein the previously suggested
rankings of result items in the database were provided by the other
users, the other users having conducted a search using the same
search query earlier in time, the other users being in the same
class as the user; wherein the previously suggested rankings of
result items in the database were provided by the other users by
entering suggested rankings of result items, the result items
having been displayed in a prior results document that was returned
after the other users conducted a search using the same search
query earlier in time; adjusting the rankings of the set of result
items based on the retrieved previously suggested rankings of
result items that were provided by the other users to obtain an
adjusted rankings of the set of result items.
17. A system including at least one computer system having access
to a database, the system configured to: receive at the computer
system a search query from a user; wherein the search query is
provided by the user either at the computer system or at a client
computing device connected to the computer system over a network;
perform a search over a set of data in response to the search
query, the set of data being at the computer system or accessible
to the computer system; generate a rankings of a set of result
items returned from the search over the set of data; adjust the
rankings of the set of result items by a method of using human
suggested rankings comprising: retrieving from the database a
previously suggested rankings of result items that were provided by
a other users; wherein the previously suggested rankings of result
items contains suggested rankings for zero, one, or more of the
result items in the set of result items; wherein the previously
suggested rankings of result items in the database were provided by
the other users, the other users having conducted a search using
the same search query earlier in time; wherein the previously
suggested rankings of result items in the database were provided by
the other users by entering suggested rankings of result items, the
result items having been displayed in a prior results document that
was returned after the other users conducted a search using the
same search query earlier in time; adjusting the rankings of the
set of result items based on the retrieved previously suggested
rankings of result items that were provided by the other users to
obtain an adjusted rankings of the set of result items; display the
adjusted rankings of the set of result items in a results document,
wherein the user may enter suggested rankings for zero, one, or
more result items in the results document, such suggested rankings
to be stored in the database for potential later use in adjusting
the rankings of a later set of result items returned in response to
a later search queries of a later users.
18. A computer program product, comprising a computer usable medium
having a computer readable program code embodied therein, said
computer readable program code adapted to be executed to implement
a method of human ranking of search engine results, said method
using a computer system having access to a database, said method
comprising: receiving at the computer system a search query from a
user; wherein the search query is provided by the user either at
the computer system or at a client computing device connected to
the computer system over a network; performing a search over a set
of data in response to the search query, the set of data being at
the computer system or accessible to the computer system;
generating a rankings of a set of result items returned from the
search over the set of data; adjusting the rankings of the set of
result items by a method of using human suggested rankings
comprising: retrieving from the database a previously suggested
rankings of result items that were provided by a other users;
wherein the previously suggested rankings of result items contains
suggested rankings for zero, one, or more of the result items in
the set of result items; wherein the previously suggested rankings
of result items in the database were provided by the other users,
the other users having conducted a search using the same search
query earlier in time; wherein the previously suggested rankings of
result items in the database were provided by the other users by
entering suggested rankings of result items, the result items
having been displayed in a prior results document that was returned
after the other users conducted a search using the same search
query earlier in time; adjusting the rankings of the set of result
items based on the retrieved previously suggested rankings of
result items that were provided by the other users to obtain an
adjusted rankings of the set of result items; displaying the
adjusted rankings of the set of result items in a results document,
wherein the user may enter suggested rankings for zero, one, or
more result items in the results document, such suggested rankings
to be stored in the database for potential later use in adjusting
the rankings of a later set of result items returned in response to
a later search queries of a later users.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority from U.S. provisional
patent application Ser. No. 61/299,994 filed Jan. 31, 2010 and U.S.
provisional patent application Ser. No. 61/302,117 filed Feb. 6,
2010, which are both incorporated herein by reference.
FIELD OF THE INVENTION
[0002] The present invention relates generally to search engines,
and more particularly, to methods of ranking search results and to
user interfaces that are used to display search results.
BACKGROUND
[0003] Search engines assist users in finding relevant documents or
other informational content from among a larger corpus. A common
application of search engines is to assist users in finding
information, documents, and files that are located on a network. A
common application of search engines is to assist users in finding
information, documents, and files on the World Wide Web ("Web"),
which comprises a massive network of Web documents. The user
submits a word or phrase he or she is interested in, called a
search query, and the search engine seeks to provide a list of high
quality Web documents that are relevant to that query. The list of
Web documents is usually ranked with higher-quality and more
relevant documents appearing first.
[0004] The ranking is conventionally done either by examination of
the contents of the Web documents or by examining the link
structure of the Web documents. For example, a search engine may
examine the words on a web document and assign an importance
ranking according to how similar or related the words on the web
document are to the search query. For example, a search engine may
examine the link structure of a web document and assign an
importance ranking according to which other documents link to the
document and which other documents the document links to. For
example, a search engine may assign an importance ranking to a
document based on the user's search history. For example, a search
engine may assign an importance ranking to a document based on
whether the user has previously navigated to and made a purchase
from that document.
[0005] One problem with search engines is that the methods of
ranking documents often do not produce rankings that users believe
to be an accurate reflection of relevance or quality. Often, a Web
document that users believe to be the highest quality and most
relevant is not listed first or second or anywhere near the top of
the results list. A second problem is that users who notice this
problem do not have a means to fix it so that future users who
search for the same query can receive better search results.
SUMMARY OF THE INVENTION
[0006] One aspect of the invention is directed to a method that
includes receiving at a computer system a search query from a user.
The method includes performing a search over a set of data in
response to the search query. The method includes generating a
rankings of a set of result items from the search over the set of
data. The method includes adjusting the rankings of the set of
result items by a method of using human suggested rankings
comprising: retrieving from a database previously suggested
rankings of result items that were provided by other users; and
adjusting the rankings of the set of result items based on the
retrieved previously suggested rankings of result items that were
provided by the other users to obtain an adjusted rankings of the
set of result items. The method includes displaying the adjusted
rankings of the set of result items in a results document, wherein
the user may enter suggested rankings for zero, one, or more result
items in the results document, such suggested rankings to be stored
in the database to be available for potential later use.
[0007] One aspect of the invention is directed to a method that
includes receiving at a computer system a search query from a user.
The method includes performing a search over a set of data in
response to the search query. The method includes generating a
rankings of a set of result items from the search over the set of
data. The method includes adjusting the rankings of the set of
result items by a method of using human suggested rankings
comprising: retrieving from a database previously suggested
rankings of result items that were provided by other users; the
previously suggested rankings of result items containing suggested
rankings for some of the result items in the set of result items;
the other users having conducted a search using the same search
query earlier in time; the other users having entered the suggested
rankings of result items after being displayed with the results of
their search using the same search query earlier in time; and
adjusting the rankings of the set of result items based on the
retrieved previously suggested rankings of result items that were
provided by the other users to obtain an adjusted rankings of the
set of result items. The method includes displaying the adjusted
rankings of the set of result items in a results document, wherein
the user may enter suggested rankings for zero, one, or more result
items in the results document, such suggested rankings to be stored
in the database to be available for potential later use in
adjusting the rankings of a later set of result items returned in
response to a later search queries of a later users.
[0008] One aspect of the invention is directed to adjusting the
rankings of the set of result items based on the retrieved
previously suggested rankings of result items from the other users
to obtain an adjusted rankings of the set of result items by a
method comprising: re-ranking one or more result items in the set
of result items to the last user-suggested ranking the one or more
result items received from the other users, the other users having
conducted a search using the same search query earlier in time.
[0009] One aspect of the invention is directed to adjusting the
rankings of the set of result items based on the retrieved
previously suggested rankings of result items from the other users
to obtain an adjusted rankings of the set of result items by a
method comprising: re-ranking one or more result items in the set
of result items to an average user-suggested ranking the one or
more result items have received from the other users, the other
users having conducted a search using the same search query earlier
in time.
[0010] One aspect of the invention is directed to adjusting the
rankings of the set of result items based on the retrieved
previously suggested rankings of result items from the other users
to obtain an adjusted rankings of the set of result items by a
method comprising: assigning two weights, the first weight being
determined based on user-suggested rankings one or more result
items in the set of result items have received from the other
users, the other users having conducted a search using the same
search query earlier in time, and the second weight being
determined by the content, link structure, relevancy, quality, or
other substantive qualities of one or more result items in the set
of result items, and the two weights being used in combination to
adjust the rankings of the one or more result items in the set of
result items.
[0011] The invention improves on the prior art in at least the
following ways: it may improve the quality of rankings of the set
of result items returned as a result of a user search so that more
relevant result items are ranked higher and less relevant result
items are ranked lower; and it allows users to improve the quality
of rankings of the set of result items for later users.
BRIEF DESCRIPTION OF THE DRAWINGS
[0012] FIG. 1 is a diagram illustrating an exemplary environment in
which one embodiment of the present invention may operate;
[0013] FIG. 2 is a flow diagram illustrating operations that may be
performed for human ranking of search results in one embodiment of
the present invention;
[0014] FIG. 3 is a diagram illustrating an exemplary results
document that could be produced in accordance with one embodiment
of the present invention.
DETAILED DESCRIPTION
[0015] FIG. 1 is a block diagram illustrating an exemplary network
environment for implementation of one embodiment of the present
invention. The network environment 100 may include multiple clients
110 connected to one or more servers 120 via a network 140. Network
140 may include a local area network (LAN), a wide area network
(WAN), a telephone network, such as the Public Switched Telephone
Network (PSTN), an intranet, the Internet, or a combination of
networks. Two clients 110 and one server 120 have been illustrated
for simplicity, though in practice there may be more or fewer
clients and servers.
[0016] Clients 110 may be personal computers, digital assistants,
personal digital assistants, cellular phones, mobile phones, smart
phones, pagers, digital tablets, laptop computers, Internet
appliances, and other processor-based devices. Clients may be
client computing devices. Client computing devices may be personal
computers, digital assistants, personal digital assistants,
cellular phones, mobile phones, smart phones, pagers, digital
tablets, laptop computers, Internet appliances, and other
processor-based devices. Servers 120 may include entities that
process, search, and/or maintain documents or files in a manner
consistent with the principles of the invention. Server 120 may be
implemented as a number of networked server devices, though it is
illustrated as a single entity.
[0017] Server 120 may include a search engine 125 that can receive
search queries from clients 110. In one embodiment, the search
engine 125 may function as a query-based web page search engine.
The search engine may cause the output, such as the display, of
search results on the client 110. The search results may be in the
form a results document. The results document may contain a set of
result items.
[0018] In one embodiment, the search engine 125 processes search
queries for documents located on the network. It may process
queries by searching the content database 127 in which documents
and/or index information about documents on the network are stored.
It may return a set of documents and/or index information (called
the set of result items) and may rank this set of result items
according to relevancy and quality.
[0019] In one embodiment, the search engine then accesses the user
rankings database 128 to access previous user-suggested rankings of
result items for the search query. A user-suggested ranking of a
result item is a ranking of the result item that was suggested by a
user. The search engine may adjust the results rankings based on
these suggested rankings.
[0020] Those skilled in the art will appreciate that the documents
and/or index information about documents in the content database
127 and the user-suggested ranks in the user rankings database 128
may be stored in a single database or in more or fewer databases.
Alternatively, they may be stored in a networked system of
databases. It is for simplicity of illustration that these elements
are described as residing in two different databases. The term
database as used in this patent may refer to a computer database or
any data storage or network of data storage.
[0021] FIG. 2 is a flow chart illustrating an example method 200
that may be performed by search engine 125 consistent with aspects
of the invention. Search engine 125 may receive a search query from
a user (act 201). The search query may be received at the server
120. The user may provide the search query either at the server 120
or at a client 110 that is connected to the server 120 over the
network 140. In response, the search engine may perform a search
over a set of data, where the set of data is at the computer system
or accessible to the computer system (act 202). The set of data
that is searched over may be the content database 127. The search
engine may generate a rankings of a set of result items returned
from the search over the set of data (act 203). The step of
generating a rankings of a set of result items may be performed by
any conventional or nonconventional method of generating a rankings
of a set of result items such as but not limited to: analysis of
the content of the document and other documents or analysis of the
link structure of the document and other documents. The search
engine may adjust the rankings of the set of result items by a
method of using human suggested rankings (act 204). Act 204 may
include retrieving from the user rankings database 128 a previously
suggested rankings of result items that were provided by a other
users. The previously suggested rankings of result items contains
suggested rankings for zero, one, or more of the result items in
the set of result items. The previously suggested rankings of
result items in the user rankings database 128 may have been
provided by the other users, the other users having conducted a
search using the same search query earlier in time. The previously
suggested rankings of result items in the database may have been
provided by the other users by entering suggested rankings of
result items, the result items having been displayed in a prior
results document that was returned after the other users conducted
a search using the same search query earlier in time. Act 204 may
include adjusting the rankings of the set of result items based on
the retrieved previously suggested rankings of result items that
were provided by the other users to obtain an adjusted rankings of
the set of result items. The search engine may display the adjusted
rankings of the set of result items in a results document (act
205). The user may enter suggested rankings for zero, one, or more
result items in the results document (act 206). Such suggested
rankings are stored in the user rankings database 128 to be
available for potential later use in adjusting the rankings of a
later set of result items returned in response to a later search
queries of a later users.
[0022] A suggested ranking for a result item in the user rankings
database 128 may be a database entry with information regarding:
the user who made the user-suggested ranking, the suggested ranking
of the result item, the name or identifying information of the
result item, the search query used by the user who made the
user-suggested ranking to obtain the result item, the date and time
that the suggested ranking was entered, and other information. A
suggested ranking of a result item may be a suggestion that the
result item be ranked at a certain ranking. For example, a
suggested ranking for website A may be a suggestion that the
website A be ranked third in a set of result items returned when a
user conducts a search using search query B. A suggested ranking is
not merely an indication that a user recommended or did not
recommend a website; is not merely an indication that a user voted
up or voted down a website; is not merely a rating of a website on
a scale for example from 1 to 5. Rather a suggested ranking
indicates that a user made a suggestion that the result item be
ranked at a certain ranking in the set of result items.
[0023] In this patent, the term "same search query" is defined as a
search query that is similar or substantially the same. Two search
queries need not be exactly identical to be the "same search
query." For example, a search query of "Michael Jordan" may be the
"same search query" as "the Michael Jordan," "person Michael
Jordan," "Michael Jordan basketball," "Michael J basketball
player," and so on in some embodiments of the invention.
[0024] In one embodiment, the step in act 204 of adjusting the
rankings of the set of result items based on the retrieved
previously suggested rankings of result items that were provided by
the other users to obtain an adjusted rankings of the set of result
items may be accomplished by a method comprising: re-ranking one or
more result items in the set of result items to the last
user-suggested ranking the one or more result items received from
the other users, the other users having conducted a search using
the same search query earlier in time. Additional procedures may be
used in this adjustment method as follows: When a result item is
re-ranked, it may displace a result item that was ranked at that
location in the original ranking, the original ranking being the
ranking of the result items before the adjustment; said result item
is moved down one rank if the re-ranked result item was ranked
below it in the original ranking, and said result item is moved up
one rank if the re-ranked result item was above it in the original
ranking; this movement of said first result item may displace a
second result item; the second result item is moved down one rank
if the first result item was moved down, and the second result item
is moved up if the first result item was moved up; the process is
repeated for a third, fourth, fifth, and so on result items as
necessary until no result items need to be re-ranked. Other
procedures may be used to deal with the situation in which a result
item that is re-ranked displaces a result item that was ranked at
that location in the original ranking. For example, the result item
that was ranked at that location in the original ranking could be
deleted from the set of result items.
[0025] In one embodiment, the step in act 204 of adjusting the
rankings of the set of result items based on the retrieved
previously suggested rankings of result items that were provided by
the other users to obtain an adjusted rankings of the set of result
items may be accomplished by a method comprising: re-ranking one or
more result items in the set of result items to an average
user-suggested ranking the one or more result items have received
from the other users, the other users having conducted a search
using the same search query earlier in time. The average may be a
mean, median, mode, weighted average, or other kind of average.
Additional procedures may be used in this adjustment method as
follows: When a result item is re-ranked, it may displace a result
item that was ranked at that location in the original ranking, the
original ranking being the ranking of the result items before the
adjustment; said result item is moved down one rank if the
re-ranked result item was ranked below it in the original ranking,
and said result item is moved up one rank if the re-ranked result
item was above it in the original ranking; this movement of said
first result item may displace a second result item; the second
result item is moved down one rank if the first result item was
moved down, and the second result item is moved up if the first
result item was moved up; the process is repeated for a third,
fourth, fifth, and so on result items as necessary until no result
items need to be re-ranked. Other procedures may be used to deal
with the situation in which a result item that is re-ranked
displaces a result item that was ranked at that location in the
original ranking. For example, the result item that was ranked at
that location in the original ranking could be deleted from the set
of result items.
[0026] In one embodiment, the step in act 204 of adjusting the
rankings of the set of result items based on the retrieved
previously suggested rankings of result items that were provided by
the other users to obtain an adjusted rankings of the set of result
items may be accomplished by a method comprising: assigning two
weights, the first weight being determined based on user-suggested
rankings one or more result items in the set of result items have
received from the other users, the other users having conducted a
search using the same search query earlier in time, and the second
weight being determined by the content, link structure, relevancy,
quality, or other substantive qualities of one or more result items
in the set of result items, and the two weights being used in
combination to adjust the rankings of the one or more result items
in the set of result items. Additional procedures may be used in
this adjustment method as follows: When a result item is re-ranked,
it may displace a result item that was ranked at that location in
the original ranking, the original ranking being the ranking of the
result items before the adjustment; said result item is moved down
one rank if the re-ranked result item was ranked below it in the
original ranking, and said result item is moved up one rank if the
re-ranked result item was above it in the original ranking; this
movement of said first result item may displace a second result
item; the second result item is moved down one rank if the first
result item was moved down, and the second result item is moved up
if the first result item was moved up; the process is repeated for
a third, fourth, fifth, and so on result items as necessary until
no result items need to be re-ranked. Other procedures may be used
to deal with the situation in which a result item that is re-ranked
displaces a result item that was ranked at that location in the
original ranking. For example, the result item that was ranked at
that location in the original ranking could be deleted from the set
of result items.
[0027] In one embodiment, when a user suggests a new ranking for an
item, the item is re-ranked to the suggested ranking, the other
items are displaced accordingly, and the rankings of all or less
than all of the items in the search results list are stored in the
user rankings database. In this embodiment, adjusting the rankings
of the search items according to user-suggested rankings may be
accomplished by retrieving the rankings for all or less than all of
the items from the user rankings database.
[0028] Those skilled in the art will recognize that adjusting the
rankings of the search items according to user suggested rankings
(act 204) may be performed according to many different methods and
processes. The foregoing description of preferred embodiments of
act 204 provides illustration and description, but is not intended
to be exhaustive or to limit the implementation of act 204 to the
precise forms disclosed. Modifications and variations are possible
in light of the above teachings or may be acquired from practice of
the invention. For example, although many of the operations
described above were described in a particular order, many of the
operations are amenable to being performed simultaneously or in
different orders to still achieve the same or equivalent
results.
[0029] In one embodiment, the user may not suggest rankings of
items unless the user has a user account and is logged in to the
user account. In one embodiment, the user may not suggest rankings
of items unless the user has editing permissions. Editing
permissions may be given to all or less than all of the users of
the invention. Editing permissions may be given or taken away from
a user at different times. Some users may receive editing
permissions when they create a user account. Other users may not
receive editing permissions when they create a user account. There
may be different levels of editing permissions.
[0030] In one embodiment, the user must pass a test to show he or
she is a human and not a computer program before the user can enter
suggested rankings for the result items in the results
document.
[0031] In one embodiment, the user may only enter suggested
rankings for the result items in the results document a limited
number of times over a limited time period.
[0032] In one embodiment, the user and the other users are divided
into one or more classes. The user and the other users may be in
multiple classes at once. The user may be in a class or more than
one class. In this embodiment, suggested rankings entered by a user
are only used to adjust rankings made in response to another user's
search query if the two users are of the same class, for example
both users being of the class "interested in sports." It may be
appreciated that users may be assigned to classes in many ways and
not just according to interest area. Other kinds of classes may be:
users who are female, users who are over the age of 25, users who
have a college education, users who work in the computer software
industry, and so on. It may be appreciated that classes may be
automatically assigned, assigned by running a computer program,
assigned by manual intervention, or specified by the user. In this
embodiment where the user and the other users are divided into one
or more classes, act 204 may be performed by a method comprising:
retrieving from the database a previously suggested rankings of
result items that were provided by a other users; wherein the
previously suggested rankings of result items contains suggested
rankings for zero, one, or more of the result items in the set of
result items; wherein the previously suggested rankings of result
items in the database were provided by the other users, the other
users having conducted a search using the same search query earlier
in time, the other users being in the same class as the user;
wherein the previously suggested rankings of result items in the
database were provided by the other users by entering suggested
rankings of result items, the result items having been displayed in
a prior results document that was returned after the other users
conducted a search using the same search query earlier in time;
adjusting the rankings of the set of result items based on the
retrieved previously suggested rankings of result items that were
provided by the other users to obtain an adjusted rankings of the
set of result items.
[0033] FIG. 3 is a diagram illustrating an exemplary results
document 300 displaying the ranked results returned in response to
a search query. The result items on results document 300 are
exemplary result items returned by search engine 125 for the
exemplary search "Michael Jordan." In one embodiment, each result
item may be listed with a title 301, a summary of relevant content
302, and URL 303. The title 301 may be linked to the document. In
other embodiments, the result items may be listed with more, less,
or different information.
[0034] In one embodiment, the user may enter suggested rankings for
one or more result items listed on the results document by
selecting and moving the one or more result items to a higher or
lower ranking. In one embodiment, this may be accomplished by
clicking on the title 301, summary of relevant content 302, or URL
303 to select the result item and then moving the result item. In
one embodiment, some or all result items may be accompanied by a
move button 304 such that the user may click the move button and
then drag the item to a new location in the list that corresponds
to a new ranking. Selecting and moving the one or more result items
to a higher or lower ranking can be accomplished in many ways,
using many different kinds of user interface, and the above
examples of ways to select and move a result item are exemplary and
not limiting. The new user-suggested ranking may be stored as a new
user-suggested ranking in the user rankings database 128.
[0035] In one embodiment, the results document includes user
interface elements next to one or more result items. The user may
enter suggested rankings for the one or more result items via the
user interface elements. In one embodiment, some or all result
items may be accompanied by a text input box where the user may
enter a new suggested ranking. The new user-suggested ranking may
be stored as a new user-suggested ranking in the user rankings
database 128.
[0036] Those skilled in the art will recognize that many different
kinds of user interface actions may be used to allow the user to
enter a new user-suggested ranking and cause that ranking to be
stored in the user rankings database 128. The foregoing description
of preferred embodiments provides illustration and description, but
is not intended to be exhaustive or to limit implementation to the
precise forms disclosed. Modifications and variations are possible
in light of the above teachings or may be acquired from practice of
the invention.
[0037] In one embodiment, a merge component 310 is included on the
results document. In one embodiment, the search query received from
the user is one search query from a set of merged search queries,
and adjusting the rankings of the set of result items by a method
of using human suggested rankings is performed by a method
comprising: retrieving from the database a previously suggested
rankings of result items that were provided by a other users;
wherein the previously suggested rankings of result items contains
suggested rankings for zero, one, or more of the result items in
the set of result items; wherein the previously suggested rankings
of result items in the database were provided by the other users,
the other users having conducted a search using any search query
from the set of merged search queries earlier in time; wherein the
previously suggested rankings of result items in the database were
provided by the other users by entering suggested rankings of
result items, the result items having been displayed in a prior
results document that was returned after the other users conducted
a search using any search query from the set of merged search
queries earlier in time; adjusting the rankings of the set of
result items based on the retrieved previously suggested rankings
of result items that were provided by the other users to obtain an
adjusted rankings of the set of result items.
[0038] The results document may include a merge component 310 that
displays information about one or more sets of merged search
queries or a link to a merge component 310 that displays
information about one or more sets of merged search queries. The
results document may include user interface elements that may be
used to add one or more search queries to a set of merged search
queries or to remove one or more search queries from a set of
merged search queries. In one embodiment of the merge component
310, a text input field 311 allows entering a search query and a
merge button 312 causes that search query to be added to the set of
merged search queries 313. The set of merged search queries may
include the search query submitted by the user to generate the
current results document. Each search query in the set of merged
search queries may return the same results document, and new user
suggested rankings of items on the results document may be used to
adjust the rankings of items when searching for any of the merged
search queries. An unmerge button 314 next to each currently merged
search query may, when clicked, cause the search query to be
removed from the set of merged search queries.
[0039] In one embodiment, the results document includes a version
control component 320 that displays information about one or more
previous versions of the results document or a link to a version
control component 320 that displays information about one or more
previous versions of the results document. Each of the one or more
previous versions of the results document may display a past
adjusted rankings of the set of result items, the past adjusted
rankings of the set of result items being generated at the current
time or having been generated at a previous time by a method of
using human suggested rankings comprising: retrieving from the
database a previously suggested rankings of result items that were
provided by a other users; wherein the previously suggested
rankings of result items contains suggested rankings for zero, one,
or more of the result items in the set of result items; wherein the
previously suggested rankings of result items in the database were
provided by the other users, the other users having conducted a
search using the same search query earlier in time; wherein the
previously suggested rankings of result items in the database were
provided by the other users by entering suggested rankings of
result items, the result items having been displayed in a prior
results document that was returned after the other users conducted
a search using the same search query earlier in time; wherein the
other users entered the suggested rankings of result items prior to
a specific time, the specific time being associated with the one or
more previous versions of the results document; adjusting the
rankings of the set of result items based on the retrieved
previously suggested rankings of result items that were provided by
the other users to obtain an adjusted rankings of the set of result
items.
[0040] In one embodiment of the version control component 320, the
component may include information about the current version of the
results document in a current version display 321. The component
may include date information 322, time information 323, and user
information 324 for previous versions of the results document, a
previous version of the results document being a document
displaying the ranking of the set of result items adjusted
according to the user-suggested rankings of items provided by users
up to a specific time that is earlier than the current time, the
specific time being associated with the previous version of the
results document. The date information 322 and time information 323
may reflect the specific earlier time that is associated with the
previous version of the results document. The user information 324
may reflect the last user to enter a user suggested ranking before
or at that specific time. Each of the current and previous version
listings may include a first checkbox 325 and a second checkbox 326
allowing the selection of two previous versions of the results
document simultaneously. The user may choose to view a differences
between two or more previous versions of the results document. A
display of the differences between the two selected previous
versions may appear when the user clicks the compare button 327.
The show button 328 may cause a previous version to be displayed,
the user choosing which previous version to display using the first
checkbox 325 or another interface element.
[0041] In one embodiment, the results document includes a
discussion component 330 for user discussion or a link to a
discussion component 330 for user discussion, the discussion
component for user discussion comprising a wiki, a forum, a
bulletin board, a chat room, or other interface component
configured for user discussion.
CONCLUSION
[0042] By adjusting search result rankings with user-suggested
rankings, users may receive rankings of result items that are more
accurate with respect to the rankings that users believe the result
items should have.
[0043] The foregoing description of preferred embodiments of the
present invention provides illustration and description, but is not
intended to be exhaustive or to limit the invention to the precise
forms disclosed. Modifications and variations are possible in light
of the above teachings or may be acquired from practice of the
invention. For example, although many of the operations described
above were described in a particular order, many of the operations
are amenable to being performed simultaneously or in different
orders to still achieve the same or equivalent results.
* * * * *