U.S. patent application number 13/219835 was filed with the patent office on 2012-08-23 for automatic event recognition and cross-user photo clustering.
This patent application is currently assigned to GOOGLE INC.. Invention is credited to Matthew S. Steiner.
Application Number | 20120213404 13/219835 |
Document ID | / |
Family ID | 45894647 |
Filed Date | 2012-08-23 |
United States Patent
Application |
20120213404 |
Kind Code |
A1 |
Steiner; Matthew S. |
August 23, 2012 |
AUTOMATIC EVENT RECOGNITION AND CROSS-USER PHOTO CLUSTERING
Abstract
Methods, systems, and apparatus, including computer programs
encoded on a computer storage medium, for automatic event
recognition and photo clustering. In one aspect, methods include
receiving, from a first user, first image data corresponding to a
first image, receiving, from a second user, second image data
corresponding to a second image, comparing the first image data and
the second image data, and determining that the first image and the
second image correspond to a coincident event based on the
comparing.
Inventors: |
Steiner; Matthew S.;
(Mountain View, CA) |
Assignee: |
GOOGLE INC.
Mountain View
CA
|
Family ID: |
45894647 |
Appl. No.: |
13/219835 |
Filed: |
August 29, 2011 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
61444520 |
Feb 18, 2011 |
|
|
|
Current U.S.
Class: |
382/103 ;
382/218 |
Current CPC
Class: |
G06F 16/5838 20190101;
G06K 9/00677 20130101; G06K 9/6267 20130101; G06K 9/64 20130101;
G06F 16/5866 20190101 |
Class at
Publication: |
382/103 ;
382/218 |
International
Class: |
G06K 9/00 20060101
G06K009/00; G06K 9/68 20060101 G06K009/68 |
Claims
1. A system comprising: a data processing apparatus; and a computer
storage medium encoded with a computer program, the program
comprising instructions that when executed by the data processing
apparatus cause the data processing apparatus to perform operations
comprising: receiving, from a first user, first image data
corresponding to a first image; receiving, from a second user,
second image data corresponding to a second image; comparing the
first image data and the second image data; and determining that
the first image and the second image each correspond to an event
based on the comparing, the event being coincident with respect to
the first image and the second image.
2. The system of claim 1, the operations further comprising
providing an indication to the first user, the indication
indicating that the second user has an image corresponding to the
coincident event.
3. The system of claim 2, the operations further comprising
receiving input from the first user, the input indicating a request
to add the second image to a collection of images associated with
the coincident event.
4. The system of claim 3, the operations further comprising
providing the request to the second user.
5. The system of claim 4, the operations further comprising
receiving input from the second user, the input indicating a
response to the request.
6. The system of claim 1, wherein the second user is a member of a
social network of the first user.
7. The system of claim 1, wherein the second user is a person
determined to be in the first image.
8. The system of claim 1, wherein the second user is a member of a
social network of a person determined to be in the first image.
9. The system of claim 1, wherein the first image data comprises a
first timestamp and the second image data comprises a second
timestamp.
10. The system of claim 9, wherein comparing the first image data
and the second image data comprises: determining a difference
between the first timestamp and the second timestamp; and
determining that the difference is less than a threshold
difference.
11. The system of claim 1, wherein the first image data comprises
first geo-location data and the second image data comprises second
geo-location data.
12. The system of claim 11, wherein comparing the first image data
and the second image data comprises: determining a distance between
the first geo-location and the second geo-location; and determining
that the distance is less than a threshold distance.
13. The system of claim 1, wherein the first image data comprises
first check-in data and the second image data comprises second
check-in data.
14. The system of claim 13, wherein comparing the first image data
and the second image data comprises: determining a first location
specified by the first check-in data; determining a second location
specified by the second check-in data; and determining that the
first and second locations refer to a same geographical
location.
15. The system of claim 1, wherein the first image data and the
second image data each comprise at least one of timestamp data,
geo-location data, check-in data, and content data.
16. The system of claim 15, wherein comparing the first image data
and the second image data comprises: processing the content data of
the first image data and the second image data; and identifying a
landmark as being present in both the first image and the second
image.
17. The system of claim 15, wherein comparing the first image data
and the second image data comprises: processing the content data of
the first image data and the second image data; and identifying a
human face as being present in both the first image and the second
image.
18. The system of claim 1, wherein the first image data and the
second image data each comprise ancillary data, the ancillary data
comprising data added to the first image data and the second image
data after generation of the first image data and the second image
data.
19. The system of claim 18, wherein the ancillary data comprises
one or more tags indicating a presence of one or more particular
persons in each of the first image and the second image.
20. The system of claim 1, the operations further comprising:
generating a confidence score based on the comparing; and
determining that the confidence score exceeds a threshold
confidence score, wherein determining that the first image and the
second image correspond to a coincident event is based on the
confidence score exceeding the threshold confidence score.
21. A computer-readable medium coupled to one or more processors
having instructions stored thereon which, when executed by the one
or more processors, cause the one or more processors to perform
operations comprising: receiving, from a first user, first image
data corresponding to a first image; receiving, from a second user,
second image data corresponding to a second image; comparing the
first image data and the second image data; and determining that
the first image and the second image each correspond to an event
based on the comparing, the event being coincident with respect to
the first image and the second image.
22. A computer-implemented method, comprising: receiving, from a
first user, first image data corresponding to a first image;
receiving, from a second user, second image data corresponding to a
second image; comparing the first image data and the second image
data; and determining that the first image and the second image
each correspond to an event based on the comparing, the event being
coincident with respect to the first image and the second image.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Prov. Pat. App.
No. 61/444,520, filed Feb. 18, 2011, the disclosure of which is
expressly incorporated herein by reference in its entirety.
TECHNICAL FIELD
[0002] This specification generally relates to data processing.
BACKGROUND
[0003] People take photographs (photos) to document events and to
keep memories. People often share the photos with friends and
family. In recent years, digital photography has become more
mainstream. Using digital photography, a photographer can capture a
photograph and store the photograph as a digital file. The digital
file can be stored to computer-readable memory, can be copied and
can be electronically distributed. The Internet has made the
sharing of photos much easier. People can email photos to friends,
or post them on websites for others to view. Social networking
websites are also used to share photos with friends and
acquaintances.
SUMMARY
[0004] Implementations of the present disclosure are generally
directed to event recognition and cross-user photo clustering. In
some implementations, first image data corresponding to a first
image and second image data corresponding to a second image are
compared to determine whether the first image and the second image
correspond to a coincident event. The first image and the second
image include respective digital photographs (photos) taken of an
event by one or more users. In some implementations, two or more of
timestamp data, geo-location data, content data and ancillary data
are compared between the first image and the second image. In some
implementations, if the first image and the second image correspond
to a coincident event, a first user associated with the first image
and a second user associated with the second image are notified.
The first image and the second image can be published to an event
page.
[0005] In general, innovative aspects of the subject matter
described in this specification may be embodied in methods that
include the actions of receiving, from a first user, first image
data corresponding to a first image, receiving, from a second user,
second image data corresponding to a second image, comparing the
first image data and the second image data, and determining that
the first image and the second image each correspond to an event
based on the comparing, the event being coincident with respect to
the first image and the second image.
[0006] These and other implementations may each optionally include
one or more of the following features. For instance, actions
further include providing an indication to the first user, the
indication indicating that the second user has an image
corresponding to the coincident event; actions further include
receiving input from the first user, the input indicating a request
to add the second image to a collection of images associated with
the coincident event; actions further include providing the request
to the second user; actions further include receiving input from
the second user, the input indicating a response to the request;
the second user is a member of a social network of the first user;
the second user is a person determined to be in the first of image;
the second user is a member of a social network of a person
determined to be in the first image; the first image data includes
a first timestamp and the second image data includes a second
timestamp; comparing the first image data and the second image data
includes: determining a difference between the first timestamp and
the second timestamp, and determining that the difference is less
than a threshold difference; the first image data includes first
geo-location data and the second image data includes second
geo-location data; comparing the first image data and the second
image data includes: determining a distance between the first
geo-location and the second geo-location, and determining that the
distance is less than a threshold distance; the first image data
includes first check-in data and the second image data includes
second check-in data; comparing the first image data and the second
image data includes: determining a first location specified by the
first check-in data, determining a second location specified by the
second check-in data, and determining that the first and second
locations refer to a same geographical location; the first image
data and the second image data each include at least one of
timestamp data, geo-location data, check-in data and content data;
comparing the first image data and the second image data includes:
processing the content data of the first image data and the second
image data, and identifying a landmark as being present in both the
first image and the second image; comparing the first image data
and the second image data includes: processing the content data of
the first image data and the second image data, and identifying a
human face as being present in both the first image and the second
image; the first image data and the second image data each include
ancillary data, the ancillary data including data added to the
first image data and the second image data after generation of the
first image data and the second image data; the ancillary data
includes one or more tags indicating a presence of one or more
particular persons in each of the first image and the second image;
and actions further include: generating a confidence score based on
the comparing, and determining that the confidence score exceeds a
threshold confidence score, wherein determining that the first
image and the second image correspond to a coincident event is
based on the confidence score exceeding the threshold confidence
score.
[0007] The details of one or more implementations of the subject
matter described in this specification are set forth in the
accompanying drawings and the description below. Other potential
features, aspects, and advantages of the subject matter will become
apparent from the description, the drawings, and the claims.
BRIEF DESCRIPTION OF THE DRAWINGS
[0008] FIG. 1 depicts an example system that can be used in
accordance with implementations of the present disclosure.
[0009] FIG. 2 depicts an example environment for event recognition
and photo clustering.
[0010] FIG. 3 is a flowchart of an example process for event
recognition and photo clustering.
[0011] FIG. 4 is a swim lane diagram of an example process for
event recognition and photo clustering.
[0012] FIG. 5 depicts an example social graph.
[0013] Like reference numbers represent corresponding parts
throughout.
DETAILED DESCRIPTION
[0014] The present disclosure describes systems and techniques for
recognizing the occurrence of an event from digital images provided
as digital photographs (photos). Photos corresponding to the event
can be clustered across multiple users. In particular, different
users can take photos of a coincident event. For example, a first
user can take photos of an event and a second user can take photos
of the same event. The users can each upload their respective
photos to a server or a system of servers for publication to a
viewing audience. For example, each user can upload their
respective photos to a social networking website for publication to
other users of the social networking website. Data corresponding to
each of the photos can be processed to determine that the photos of
the different users correspond the same event. The data can include
timestamp data, data regarding people identified within the photos,
data regarding landmarks identified within the photos, and/or
geographic location (geo-location) data indicating where the photos
were taken. The data can also include other image analysis data,
such as license plates, street signs, store fronts, bar codes, and
other data.
[0015] In some implementations, photos of the same event are
clustered across the multiple users. The clustering can include
adding photos published by the second user to an event webpage
created by the first user, and/or creating an event webpage or
webpages designated for the event and uploading photos of the first
and second users to the event webpage. In some implementations,
user permission is requested before uploading their respective
photos to the event webpage. In some implementations, the event
data is displayed on an application on a computing device, rather
than a webpage.
[0016] In some implementations, event recognition and photo
clustering can be achieved based on one or more social graphs
associated with a user of a social networking service. For example,
a particular user of the social networking service can publish
photos using the social networking service. Other users of the
social networking service can also publish photos using the social
networking service. Photos published by the other users can be
reviewed in view of and can be selectively clustered with photos of
the particular user. The review and clustering can be based on a
social relationship between the particular user and the other users
within the social networking service. The other users' photos can
be searched to find more photos of the coincident event.
[0017] By way of non-limiting example, Alice and Bob can attend the
wedding of Claire and David. In this example, the wedding of Claire
and David is defined as an event. Alice and Bob can both take
photos of the event and upload the photos for publication using a
social networking service. Data underlying the photos can be
processed such that the photos uploaded by Alice and Bob are
recognized as being photos of the same event. For example, the data
underlying the photos can indicate that at least a subset of
Alice's photos and of Bob's photos each include Claire and David,
and that the photos were both taken within a threshold timeframe of
one another. In response to recognizing that the photos correspond
to the same event, a request can be issued to Alice and Bob. The
request can request whether Alice and Bob each want to include
their photos in an event webpage created for event. The request can
also include a request for a proposed event title (e.g., "Claire
and David's wedding"). Alice and Bob can individually approve the
inclusion of their photos in the event webpage.
[0018] Using Alice as an example user, photos taken by other users
that are socially connected to Alice in the social networking
service can be reviewed to determine whether photos uploaded by the
other users also correspond to the event. This can also be done in
view of Bob's social connections, as well as the social connections
of each of Claire and David. In this manner, photos of a coincident
event can be aggregated from different users, such that more photos
of the event can be shared with the family and friends, beyond only
the photos taken by a particular user. Photos of users indirectly
connected to Alice (e.g., a connection of a connection, a
connection of a connection of a connection, etc.) can also be
reviewed to determine whether the photos correspond to the
event.
[0019] FIG. 1 depicts an example system 100 that can be used in
accordance with implementations of the present disclosure. The
example system 100 includes a computing device 102 associated with
a user 104, a computing device 106 associated with a user 108, a
network 110, and a computing system 114. The computing device 102,
the computing device 106 and the computing system 114 can
communicate with each other through the network 110. The computing
system 114 can include one or more computing devices 116 (e.g., one
or more servers) and one or more computer-readable storage devices
118 (e.g., one or more databases).
[0020] Each computing device 102, 106 can represent various forms
of processing devices. Example processing devices can include a
desktop computer, a laptop computer, a handheld computer, a
personal digital assistant (PDA), a cellular telephone, a network
appliance, a camera, a smart phone, an enhanced general packet
radio service (EGPRS) mobile phone, a media player, a navigation
device, an email device, a game console, or a combination of any
these data processing devices or other data processing devices. The
computing devices 102, 106, 116 may be provided access to and/or
receive application software executed and/or stored on any of the
other computing systems 102, 106, 116. The computing device 116 can
represent various forms of servers including, but not limited to a
web server, an application server, a proxy server, a network
server, or a server farm.
[0021] In some implementations, the computing devices may
communicate wirelessly through a communication interface (not
shown), which may include digital signal processing circuitry where
necessary. The communication interface may provide for
communications under various modes or protocols, such as Global
System for Mobile communication (GSM) voice calls, Short Message
Service (SMS), Enhanced Messaging Service (EMS), or Multimedia
Messaging Service (MMS) messaging, Code Division Multiple Access
(CDMA), Time Division Multiple Access (TDMA), Personal Digital
Cellular (PDC), Wideband Code Division Multiple Access (WCDMA),
CDMA2000, or General Packet Radio System (GPRS), among others. For
example, the communication may occur through a radio-frequency
transceiver (not shown). In addition, short-range communication may
occur, such as using a Bluetooth, WiFi, or other such
transceiver.
[0022] In some implementations, the system 100 can be a distributed
client/server system that spans one or more networks such as
network 110. The network 110 can be a large computer network, such
as a local area network (LAN), wide area network (WAN), the
Internet, a cellular network, or a combination thereof connecting
any number of mobile clients, fixed clients, and servers. In some
implementations, each client (e.g., computing devices 102, 106) can
communicate with servers (e.g., computing device 116) via a virtual
private network (VPN), Secure Shell (SSH) tunnel, or other secure
network connection. In some implementations, the network 110 may
further include a corporate network (e.g., intranet) and one or
more wireless access points.
[0023] FIG. 2 depicts an example environment 200 for event
recognition and photo clustering. The example environment 200
includes a photo clustering system 202. The photo clustering system
202 can be implemented, for example, using one or more computing
systems (e.g., the computing system 114 of FIG. 1). In some
implementations, the photo clustering system 202 can be provided in
the context of a social networking service. The social networking
service can include a webpage or a profile page (e.g., user pages
204, 206) for respective users of the social networking service
(e.g., users 104, 108 of FIG. 1). The user pages 204, 206 can be
used to display information about the users 104, 108 that is
posted, or otherwise published by the users 104, 108 using the
social networking service. The user pages 204, 206 can also include
photos, videos, or other media posted by the users 104, 108. The
social networking service can also associate the user pages 204,
206 with other user pages, as described in further detail
below.
[0024] By way of non-limiting example, each user 104, 108 takes
photos of an event 220 using, for example, a digital camera or a
smart phone. The users 104, 108 can transfer the digital photos to
user devices 102, 106. That is, the users 104, 108 can each
transfer files corresponding to their respective digital photos to
computer-readable memory of the computing devices 102, 106. Using
the respective computing devices 102, 106, the users 104, 108 can
each upload respective files 208, 210 corresponding to the digital
photos taken of the event 220 to their respective user pages 204,
206. In this manner, data stored in the files 208, 210 can be
processed to publish the photos to the respective user pages 204,
206. The users 104, 108 can share their photos with their
acquaintances through the social networking service. The users 104,
108, however, may not be aware of each other's photos of the
event.
[0025] The photo clustering system 202 can compare data associated
with the photos and stored in the files 208, 210 received from the
users 104, 108 on the user pages 204, 206. In some implementations,
the files 208, 210 include data corresponding to the photos taken
by the users 104, 108. The data can include image content data,
timestamp data, geo-location data, and/or check-in data. The
timestamp data can correspond to when a particular photo was taken
and the geo-location data and check-in data can correspond to where
a particular photo was taken. The timestamp, geo-location, and
check-in data can be associated with the particular photo as
metadata by, for example, a smart phone that stores with the photo
the time and location of the smart phone when the photo is taken.
Digital cameras can also store time and location information with
photos. The photo clustering system 202 can compare the data of the
photos. Further, the system 202 can also compare the contents of
photos. For example, the users 104, 108 or other users of the
social networking service can tag the photos, providing identities
of people in the photos. The system 202 can also use facial
recognition techniques to identify people in photos. The system 202
can also detect recognizable landmarks in the photos, such as
famous buildings or other unique backgrounds.
[0026] The photos can also include check-in data, which can be data
provided by the user that the user was at a certain location at a
certain time or for a certain duration of time, including when the
photo was taken. For example, the users 104, 108, can check in at a
specific location, such as a restaurant, indicating that the users
104, 108 were at the location at a specific time. Additionally, the
check-in data can be provided automatically to the user. For
example, a smart phone can be configured to automatically check in
a user when the user is in certain locations. The check-in data can
also refer to events in addition to, or instead of, locations. For
example, a user can check in to a birthday party of a friend of the
user.
[0027] The system 202 can compare photos at a time to detect
coincident elements between the photos. When the system 202 detects
coincident elements, the system 202 can determine that the photos
208, 210 are of the same event. In some implementations, the system
202 determines the photos are of the same event if the data
associated with the photos indicates that the photos were taken at
approximately the same time and near the same location. The system
202 can use the timestamp data to determine the time each photo was
taken. The system 202 can include a threshold amount of time that
is used to determine whether the photos are deemed to have been
taken at approximately the same time. The system 202 can determine
that photos taken within the threshold amount of time (e.g., X
minutes, where X can be any positive number) of each other are
close enough in time that the photos might be of the same event.
For example, a difference between a first timestamp associated with
a first photo and a second timestamp associated with a second photo
can be determined. If the difference is less than or equal to the
threshold amount of time, the first and second photos can be deemed
to have been taken at approximately the same time. If the
difference is greater than the threshold amount of time, the first
and second photos can be deemed to have not been taken at
approximately the same time.
[0028] With regard to whether the photos were taken at the same or
approximately the same location, the system 202 can determine
whether at least a particular person is in each of the photos,
whether a particular landmark is in each of the photos, and/or
whether respective geo-locations of the photos correspond to one
another. For example, if two photos each include a particular
person and were taken at the same or approximately the same time
(discussed above), the photos can be deemed to have been taken at
the same or approximately the same location. As another example, if
a particular landmark (e.g., a building, a statue, a monument) is
present in each of the photos, the photos can be deemed to have
been taken at the same or approximately the same location.
[0029] As another example, geo-location data of the photos can
indicate that the photos were taken at the same or approximately
the same location. With regard to geo-location data, the system 202
can use a threshold distance of the respective geo-locations of the
photos. If the distance between the geo-locations is less than the
threshold distance, the photos are considered as having been taken
at the same or approximately the same location. For example, a
difference between a first geo-location associated with a first
photo and a second geo-location associated with a second photo can
be determined. If the difference is less than or equal to the
threshold distance, the first and second photos can be deemed to
have been taken at approximately the same location. If the
difference is greater than the threshold distance, the first and
second photos can be deemed to have not been taken at the same
location.
[0030] As another example, check-in data can indicate that the
photos were taken at the same or approximately the same location or
event. For example, if two users check in to the same restaurant
and each user takes a photo, the photos can be determined to have
been taken at the same location. Or the two users can check in to a
same event, such as the same birthday party. If the check-in data
for two photos indicate the photos were taken at the same birthday
party, the photos can be determined to have been taken at the same
event.
[0031] The system 202 can also determine that a set of photos from
one user are of one event. The system 202 can use information
provided by the user, for example, the user can label a set of
photos with an event title. Additionally, the system 202 can
determine that two of the photos provided by one user are of the
same event using the techniques described herein. The system 202
may also determine that photos received from one user and that were
taken within a threshold time of one another are of the same event.
The system 202 can use this information to determine that sets of
photos provided by two users are of the same event. For example,
the user 104 can upload a set of ten photos for publication to the
user page 204. The user 108 can upload a set of five photos for
publication to the user page 206. The photo clustering system 202
can process the photos and determine that the ten photos uploaded
by the user 104 are of a coincident event. Similarly, the system
202 can process the photos and determine that the five photos
uploaded by the user 108 are also of a coincident event. The photo
clustering system 202 may then determine that at least one photo of
the set uploaded by the user 104 is of a coincident event as at
least one photo of the set uploaded by the user 108. For example,
the photo clustering system can compare timestamp data,
geo-location data, check-in data, and/or photo content, as
discussed herein. The photo clustering system 202 can determine
that all ten photos uploaded by the user 104 and all five photos
uploaded by the user 108 are of the same event.
[0032] The photo clustering system 202 can also generate a
confidence score as to whether photos are of a coincident event.
The confidence score can be based on factors including a difference
between the timestamps of the photos, a difference between the
geo-locations of the photos, a confidence measure corresponding to
whether a person detected in each of the photos is the same person,
a confidence measure corresponding to whether a landmark detected
in each of the photos is the same landmark, and/or a confidence
measure corresponding to whether two check-in locations/events are
the same location/event. The confidence score can also be based on
how many elements of information the system determines to be the
same or sufficiently similar between the photos. For example, if
the timestamps of the two photos are within the threshold time, the
same person is detected in each of the photos, and the same
landmark is detected in each of the photos, a first confidence
score can be provided. If the timestamps of the two photos are
within the threshold time, but the same person is not detected in
each of the photos, and/or the same landmark is not detected in
each of the photos, a second confidence score can be provided, the
second confidence score being less than the first confidence score.
In some implementations, the confidence score can be lower when
elements of the photos are different. For example, although photos
may be taken at a similar time and each includes a person that is
deemed to be the same person in both photos (e.g., based on facial
recognition techniques and/or tagging), but the difference between
the geo-locations of the photos is greater than the threshold
distance, a relatively low confidence score can be provided.
[0033] The photo clustering system 202 can cluster the photos that
are determined to be of the same event. In some implementations, an
event page 222 can be generated. The event page 22 can include a
webpage corresponding to the coincident event in the photos, and
the photos can be published to the event page (e.g., upon approval
of the owner of the photos). For example, the event page 222 can be
a webpage provided by the social networking service, and can
display information about the coincident event including the photos
provided by the users 104, 108. The event page 222 can include
privacy settings that allow only some users of the social
networking service to view the page. For example, viewing of the
event page 222 can be limited only to the social networking
contacts of the users whose photos are included on the event page
222. Other privacy settings can also be used.
[0034] In some implementations, the event page 222 can be generated
by a user. For example, a user (e.g., the user 104) can generate
the event page 222 using the social networking service and can
publish the photos to the event page 222. The photo clustering
system 202 can determine that photos posted by another user (e.g.,
the user 108) of the social networking service correspond to the
event associated with the event page 222. The other user (e.g., the
user 108) can be notified of the existence of the event page 222,
as discussed herein, and can provide permission to add their photos
to the event page. In this manner, the photos of both users are
published through the event page 222. In some implementations, the
photo clustering system 202 can determine photos that are private
to another user (e.g., the user 108) correspond to the event of the
event page 222. The system can notify the user 108 and ask if the
user 108 wants to publish and include the photos through the event
page 222.
[0035] The users 104, 108 can each receive notifications requesting
permission to add their photos to an event page 222. The
notifications can include information about the other users who
have added photos to the event page 222. Consequently, users can
determine whether they want to share their photos with the users
who have photos published to the event page 222. In some
implementations, the users 104, 108 can upload photos and other
media directly to an event page. Further, notifications can be used
to confirm that the photos determined to be of a coincident event
were, in fact, taken at the coincident event. For example, a user
may receive a notification that an event page 222 has been created
and that the user's photos may correspond to the event associated
with the event page 222. The user can provide feedback indicating
that the user's photos indeed do not correspond to the event
associated with the event page 222. In this manner, the photo
clustering system 202 can process the feedback to further improve
the accuracy of coincident event detection.
[0036] In some implementations, the photo clustering system 202 can
cluster photos based on different sets of similar elements. For
example, if two photos do not both have timestamp data, but contain
both a same landmark and a same person, the two photos can still be
clustered as corresponding to a same event. Additionally, if two
photos have check-in data corresponding to a same event, the two
photos can be clustered as corresponding to a same event. In some
implementations, the confidence score can be weighted depending on
which elements are found to be similar.
[0037] In some implementations, photos with similar elements that
do not include a similar time element (e.g., a similar timestamp or
check-in data corresponding to a similar time) can be clustered
together as corresponding to each other. For example, photos that
include the Eiffel Tower can be clustered together regardless of
time. These clustered photos can be further clustered as events if
a subset of the clustered photos include data corresponding to a
similar time that the subset of photos was taken.
[0038] FIG. 3 is a flowchart of an example process 300 for event
recognition and photo clustering. In some implementations, actions
represented in the example process 300 may be performed by a
computing system (e.g., the computing system 114 of FIG. 1).
[0039] A first photo file is received (302). The first photo file
can be received by the computing system 114 of FIG. 1, for example.
The first photo file can include data such as content data,
timestamp data and geo-location data. The first photo file can also
include ancillary data that can include tags or labels of people,
objects, and/or locations in the photo and/or a time and date of
the photo, including captions and comments. A second photo file is
received (304). The second photo file can be received by the
computing system 114 of FIG. 1, for example. The second photo file
can include data such as content data, timestamp data and
geo-location data. The second photo file can also include ancillary
data that can include tags or labels of people, objects, and/or
locations in the photo and/or a time and date of the photo,
including captions and comments.
[0040] Corresponding data between the first and second photo files
is compared (306). For example, the computing system 114 of FIG. 1
compares the corresponding data between the first and second photo
files. The data first and second photo files can be compared on a
data type-by-data type basis. For example, a first timestamp of the
first photo file can be compared to a second timestamp of the
second photo file. As another example, first geo-location data of
the first photo file can be compared to second geo-location data of
the second photo file. As another example, first check-in data of
the first photo file can be compared to second check-in data of the
second photo file. The data type-by-data type comparison can be
performed for all of the data fields common to both the first and
second photo files. If the data between the first and second photo
files does not correspond, the first and second photos are deemed
not to be of the same event. If the data between the first and
second photo files corresponds, the first and second photos are
deemed to be of the same event. The first and second photos files
can be identified as corresponding to a coincident event (310). For
example, event data can be added to each of the first and second
photo files, the event data indicating that the first and second
photo files correspond to a coincident event. In this manner, if
the first photo file and a third photo file are deemed to
correspond to a coincident event, the second photo file and the
third photo file can automatically be deemed to correspond to a
coincident event without processing of their respective data.
[0041] In some implementations, if the data of two or more common
data fields between the first and second photo files correspond the
first and second photo files are deemed to be of the same event.
For example, if the timestamp data indicates that the first and
second photos were taken at approximately the same time and the
geo-location data indicates that the first and second photos were
taken at approximately the same location, the first and second
photo files are deemed to be of the same event. As another example,
if the timestamp data indicates that the first and second photos
were taken at approximately the same time and the content data
indicates that the first and second photos include coincident
content (e.g., at least one person common to both the first and
second photos, at least one landmark common to both the first and
second photos), the first and second photo files are deemed to be
of the same event. As another example, if the content data
indicates that the first and second photos include coincident
content (e.g., at least one person common to both the first and
second photos, at least one landmark common to both the first and
second photos) and the geo-location data indicates that the first
and second photos were taken at approximately the same location,
the first and second photo files are deemed to be of the same
event. Other combinations of correspondence between the data of the
first and second photo files can also indicate that the first and
second photo files are deemed to be of the same event.
[0042] In some implementations, facial recognition techniques can
be applied to the content data of the first and second photo files
to determine whether a particular person is in both the first and
second photos. The identity of the person can remain anonymous. In
some implementations, user-supplied information regarding
identities of people in photos can be used to determine whether the
same person is in both pictures. For example, one or more tags
associated with the first and second photos can be compared to
determine whether one or more particular people are in the first
and second photos.
[0043] FIG. 4 is a swim lane diagram 400 of an example process for
event recognition and photo clustering. In some implementations,
operations represented in the swim lane diagram 400 may be
performed by one or more computing devices of a system such as the
system 100 of FIG. 1.
[0044] A first set of photos of an event is generated (402). For
example, a first user (e.g., user 104) takes photos of an event
(e.g., event 220) using a digital camera that generates the first
set of photos. A second set of photos of the event is generated
(404). For example, a second user (e.g., user 108) takes photos of
the same event (e.g., event 220) using a digital camera that
generates the second set of photos. As the first and second users
take photos of the same event, the photos are generated at a
similar time (i.e., within a threshold time of one another) and in
a similar location (i.e., within a threshold distance of one
another).
[0045] The first set of photos are uploaded (406). The first user
can upload the first set of photos to a computer system (e.g., the
computing system 114 of FIG. 1) using a client device (e.g., the
computing device 102 of FIG. 1). The first set of photos is
received (408). For example, the computing system 114 of FIG. 1
receives the first set of photos. The computing system can be
associated with a social networking service, and the first user can
upload the first set of photos to a corresponding user page of the
social networking service. Alternatively or additionally, the
computing system can be associated with a photo sharing service
and/or a data storage service, to which the first user can upload
the first set of photos.
[0046] An event page is generated (410). The event page can include
information about the event as well as the first set of photos. The
information about the event can be received from the first user.
Additionally, information about the event can be determined from
the first set of photos. For example, the time and location of the
event can be determined from data included with the photos of the
first set of photos. The data can include a timestamp of when each
photo was taken and a geo-location of where each photo was taken.
The time and location can correspond to the time and location of
the event. The first user can also provide information about the
event such as a title of the event. The first user can also
designate which photos were taken at the event.
[0047] The second set of photos is uploaded (412). The second user
can upload the second set of photos to a computer system (e.g., the
computing system 114 of FIG. 1) using a client device (e.g., the
computing device 106 of FIG. 1). The second set of photos is
received (414). For example, the computing system 114 of FIG. 1
receives the second set of photos. The computing system can be
associated with the social networking service, and the second user
can upload the second set of photos to a corresponding user page of
the social networking service. Alternatively or additionally, the
computing system can be associated with the photo sharing service
and/or the data storage service, to which the second user can
upload the second set of photos. The second user can also designate
and provide information about the uploaded photos. For example, the
second user can designate that certain photos are photos of the
event.
[0048] Photos of the first set of photos and photos of the second
set of photos are determined to be of a coincident event (416).
Data associated with the photos received from the first user and
the second user is compared to determine whether the photos are of
the same event. The data can be compared using the techniques
described herein. If the photos are determined to be photos of the
same event, notifications are sent to the first and second users
regarding combining the photos (418).
[0049] The first and second users can receive the notifications
alerting them of the other user's photos of the event. The first
and second users can respond to the permission requests (420, 422).
If the first and second users both allow their respective photos to
be combined on the same event page, the photos are clustered, or
otherwise combined to the event page (424). If the first and second
users both create event pages of the same event, the system can
request permission to merge the two pages into one event page.
Alternatively, the first or second user can decline the inclusion
of their photos on the same event page as the other user. The
system can provide separate event pages for each of the users, or
no event page at all, depending on their responses to the
permission requests. In addition, the system can have additional
privacy settings, for example, users can prevent the system from
notifying other users about their photos without their
permission.
[0050] FIG. 5 depicts an example social graph 500. The example
social graph 500 corresponds to a user ("Alice") identified using a
node 502. The social graph 500 can be determined based on Alice's
use of a computer-implemented social networking service. For
example, Alice can generate a profile within the social networking
service and can digitally associate the profile with profiles of
other users of the social networking service. Alice can upload
photos that can be published using the social networking service.
In the example social graph 500 of FIG. 5, other users of the
social networking service include user ("Bob") identified by a node
504, user ("Claire") identified by a node 506, user ("David")
identified by a node 508, and user ("Zach") identified by a node
513. Bob and Claire are both contacts of Alice within the social
networking service, as indicated by edges 509, 511, respectively.
Claire is a contact for Bob, as indicated by edge 517. David is a
contact of Claire (edge 519), and Zach is a contact of David (edge
515). For example, Alice previously approved Claire and David as
contacts in the social networking service, such that information
and/or photos provided by Alice may be automatically shared with
Claire and David. Likewise, previous approval can have occurred for
the other contacts of the other users.
[0051] In the example social graph 500 of FIG. 5, David is not a
contact of Alice within the social networking service. Instead,
David may be another user of the social networking service that has
limited access to the information and/or posts provided by Alice.
Zach is a contact of David, as indicated by the edge 515, but is
not a contact of Alice, Bob, or Claire.
[0052] In the example social graph of FIG. 5, Alice has uploaded
photos of an event. In this example, the event can be Claire and
David's wedding. The system can create an event page 510. In some
implementations, the event page 510 may include a privacy setting,
set by Alice as the first user who uploaded photos of the event,
which enables any user of the social networking service to view and
comment on the event page 510. In some implementations, Alice is
able to establish a privacy setting of an event page such that only
contacts of Alice within the social networking service, or a subset
of contacts of Alice within the social networking service are able
to view and comment on the event page.
[0053] The system can find Bob's photos of the event by searching
through Alice's social contacts. The system can determine that
Bob's photos, or a subset of Bob's photos corresponds to the event.
The system can notify Alice that Bob has also uploaded photos of
the event. The system can also notify Bob that Alice has created an
event page for the photos of the event. If both Alice and Bob
approve, the system can include both Alice and Bob's photos on the
event page 510.
[0054] The system can also find Zach's photos of the event, even
though neither Alice nor Bob are contacts of Zach. The photos can
include pictures of Claire and David, as the event photographed is
their wedding. The system can then search through Claire and
David's contacts to determine if any of the photos they uploaded
are also of the event. The system can therefore find Zach's photos
of the event, and notify Zach of the event page 510. The system can
also notify Alice to request permission to include Zach's photos in
the event page. In some implementations, the system can request
permission from both Alice and Bob. Alternative permission and
privacy settings can be used for the event pages.
[0055] Implementations of the present disclosure and all of the
functional operations provided herein can be realized in digital
electronic circuitry, or in computer software, firmware, or
hardware, including the structures disclosed in this specification
and their structural equivalents, or in combinations of one or more
of them. Implementations of the present disclosure can be realized
as one or more computer program products, i.e., one or more modules
of computer program instructions encoded on a computer readable
medium for execution by, or to control the operation of, data
processing apparatus. The computer readable medium can be a
machine-readable storage device, a machine-readable storage
substrate, a memory device, a composition of matter effecting a
machine-readable propagated signal, or a combination of one or more
of them. The term "data processing apparatus" encompasses all
apparatus, devices, and machines for processing data, including by
way of example a programmable processor, a computer, or multiple
processors or computers. The apparatus can include, in addition to
hardware, code that creates an execution environment for the
computer program in question, e.g., code that constitutes processor
firmware, a protocol stack, a database management system, an
operating system, or a combination of one or more of them.
[0056] A computer program (also known as a program, software,
software application, script, or code) can be written in any form
of programming language, including compiled or interpreted
languages, and it can be deployed in any form, including as a stand
alone program or as a module, component, subroutine, or other unit
suitable for use in a computing environment. A computer program
does not necessarily correspond to a file in a file system. A
program can be stored in a portion of a file that holds other
programs or data (e.g., one or more scripts stored in a markup
language document), in a single file dedicated to the program in
question, or in multiple coordinated files (e.g., files that store
one or more modules, sub programs, or portions of code). A computer
program can be deployed to be executed on one computer or on
multiple computers that are located at one site or distributed
across multiple sites and interconnected by a communication
network.
[0057] The processes and logic flows described in this disclose can
be performed by one or more programmable processors executing one
or more computer programs to perform functions by operating on
input data and generating output. The processes and logic flows can
also be performed by, and apparatus can also be implemented as,
special purpose logic circuitry, e.g., an FPGA (field programmable
gate array) or an ASIC (application specific integrated
circuit).
[0058] Processors suitable for the execution of a computer program
include, by way of example, both general and special purpose
microprocessors, and any one or more processors of any kind of
digital computer. Generally, a processor will receive instructions
and data from a read only memory or a random access memory or both.
Elements of a computer can include a processor for performing
instructions and one or more memory devices for storing
instructions and data. Generally, a computer will also include, or
be operatively coupled to receive data from or transfer data to, or
both, one or more mass storage devices for storing data, e.g.,
magnetic, magneto optical disks, or optical disks. However, a
computer need not have such devices. Moreover, a computer can be
embedded in another device, e.g., a mobile telephone, a personal
digital assistant (PDA), a mobile audio player, a Global
Positioning System (GPS) receiver, to name just a few. Computer
readable media suitable for storing computer program instructions
and data include all forms of non volatile memory, media and memory
devices, including by way of example semiconductor memory devices,
e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,
e.g., internal hard disks or removable disks; magneto optical
disks; and CD ROM and DVD-ROM disks. The processor and the memory
can be supplemented by, or incorporated in, special purpose logic
circuitry.
[0059] To provide for interaction with a user, implementations of
the present disclosure can be implemented on a computer having a
display device, e.g., a CRT (cathode ray tube) or LCD (liquid
crystal display) monitor, for displaying information to the user
and a keyboard and a pointing device, e.g., a mouse or a trackball,
by which the user can provide input to the computer. Other kinds of
devices can be used to provide for interaction with a user as well;
for example, feedback provided to the user can be any form of
sensory feedback, e.g., visual feedback, auditory feedback, or
tactile feedback; and input from the user can be received in any
form, including acoustic, speech, or tactile input.
[0060] The computing system can include clients and servers. A
client and server are generally remote from each other and
typically interact through a communication network. The
relationship of client and server arises by virtue of computer
programs running on the respective computers and having a
client-server relationship to each other.
[0061] While this disclosure includes some specifics, these should
not be construed as limitations on the scope of the disclosure or
of what may be claimed, but rather as descriptions of features of
example implementations of the disclosure. Certain features that
are described in this disclosure in the context of separate
implementations can also be provided in combination in a single
implementation. Conversely, various features that are described in
the context of a single implementation can also be provided in
multiple implementations separately or in any suitable
subcombination. Moreover, although features may be described above
as acting in certain combinations and even initially claimed as
such, one or more features from a claimed combination can in some
cases be excised from the combination, and the claimed combination
may be directed to a subcombination or variation of a
subcombination.
[0062] Similarly, while operations are depicted in the drawings in
a particular order, this should not be understood as requiring that
such operations be performed in the particular order shown or in
sequential order, or that all illustrated operations be performed,
to achieve desirable results. In certain circumstances,
multitasking and parallel processing may be advantageous. Moreover,
the separation of various system components in the implementations
described above should not be understood as requiring such
separation in all implementations, and it should be understood that
the described program components and systems can generally be
integrated together in a single software product or packaged into
multiple software products.
[0063] A number of implementations have been described.
Nevertheless, it will be understood that various modifications may
be made without departing from the spirit and scope of the
disclosure. For example, various forms of the flows shown above may
be used, with steps re-ordered, added, or removed. Accordingly,
other implementations are within the scope of the following
claims.
* * * * *