U.S. patent application number 12/037347 was filed with the patent office on 2009-08-27 for system for logging life experiences using geographic cues.
This patent application is currently assigned to Microsoft Corporation. Invention is credited to Longhao Wang, Xing Xie, Ruochi Zhang, Yu Zheng.
Application Number | 20090216435 12/037347 |
Document ID | / |
Family ID | 40999107 |
Filed Date | 2009-08-27 |
United States Patent
Application |
20090216435 |
Kind Code |
A1 |
Zheng; Yu ; et al. |
August 27, 2009 |
SYSTEM FOR LOGGING LIFE EXPERIENCES USING GEOGRAPHIC CUES
Abstract
A system for logging life experiences using geographic cues. The
system variously provides a comprehensive life-logging tool for
recording each life event; a vacation album for revisiting and
reliving vacation routes and associated photos; an information
service for finding popular routes and locations; a statistical
tool for analyzing metrics of a person's life; and a personal
website service for sharing personal information. In one
implementation, the system receives a user's GPS log files and
multimedia content at a website. The system segments the GPS log
files into geographic routes corresponding to user trips, and tags
the multimedia content with geographic cues from the GPS log files.
Then, the system indexes the geographic routes so that users can
retrieve the geographic routes by browsing or by search techniques.
The system displays animations of selected routes on a map, and
displays the multimedia content at corresponding locations along
the map route, as the route is replayed. The system also provides
browsing and spatial and temporal techniques to search a person's
travels and can provide graphical displays of the person's activity
statistics.
Inventors: |
Zheng; Yu; (Beijing, CN)
; Wang; Longhao; (Beijing, CN) ; Xie; Xing;
(Beijing, CN) ; Zhang; Ruochi; (Beijing,
CN) |
Correspondence
Address: |
LEE & HAYES, PLLC
601 W. RIVERSIDE AVENUE, SUITE 1400
SPOKANE
WA
99201
US
|
Assignee: |
Microsoft Corporation
Redmond
WA
|
Family ID: |
40999107 |
Appl. No.: |
12/037347 |
Filed: |
February 26, 2008 |
Current U.S.
Class: |
701/533 ;
707/999.003; 707/999.004; 707/999.1; 707/E17.018 |
Current CPC
Class: |
H04N 21/4126 20130101;
H04N 21/4345 20130101; G06Q 10/10 20130101; G06T 19/003 20130101;
H04N 21/4524 20130101; G11B 27/322 20130101; G11B 27/105 20130101;
G06F 16/9537 20190101; G09B 29/007 20130101; H04N 2201/3253
20130101; G08G 1/096838 20130101; Y10S 707/92 20130101 |
Class at
Publication: |
701/209 ; 707/3;
707/4; 701/202; 707/100; 707/E17.018 |
International
Class: |
G01C 21/34 20060101
G01C021/34; G06F 7/06 20060101 G06F007/06; G06F 17/30 20060101
G06F017/30 |
Claims
1. A method, comprising: receiving at a website a global
positioning system (GPS) log file from a user; receiving multimedia
content from the user; segmenting the GPS log file into geographic
routes corresponding to user trips; associating the multimedia
content to geographic routes by tagging locations to the multimedia
content; indexing the geographic routes; retrieving the geographic
routes by browsing or searching; and displaying at the website an
animation of one of the user trips on a displayed map, wherein the
animation displays the multimedia content at geographic locations
on the displayed map corresponding to the geographic cues
associated with the multimedia content.
2. The method as recited in claim 1, further comprising mining the
GPS data uploaded by the user to derive a personal knowledge
database corresponding to the user.
3. The method as recited in claim 2, further comprising mining the
GPS log file uploaded from the user and the personal knowledge
database of the user to derive a public knowledge data.
4. The method as recited in claim 1, wherein tagging the multimedia
content with geographic cues from the GPS log file enables a search
for a particular multimedia content to return a corresponding
geographic route.
5. The method as recited in claim 4, further comprising receiving a
search criterion from the user, wherein the search criterion
retrieves geographic routes that meet the search criterion from the
spatial-temporal index; and wherein the geographic routes are
listed as search results on the website.
6. The method as recited in claim 5, wherein the search criterion
comprises one or both of: a spatial search criterion created via
the user selecting a geographic area on the map in order to
retrieve search results comprising geographic routes made as
different user trips within the selected geographic area; or a
temporal search criterion created via the user selecting a time
interval in order to retrieve search results comprising geographic
routes made as different user trips within the selected time
interval.
7. The method as recited in claim 3, further comprising
recommending geographic routes and local features, wherein when a
user in a given location requests a recommendation for a geographic
route or a local feature, popular geographic routes and popular
local features are retrieved from the public knowledge database and
displayed to the user; and wherein the public knowledge database is
mined from personal preferences of individual users in the personal
knowledge databases, the personal knowledge databases derived in
turn from the GPS log files uploaded by the individual users.
8. The method as recited in claim 7, further comprising interfacing
with mobile devices, wherein when a mobile device in a given
location requests a recommendation for a geographic route or a
local feature, popular geographic routes and popular local features
are retrieved from the public knowledge database and displayed to
the mobile device.
9. The method as recited in claim 1, further comprising displaying
ancillary information calculated in relation to the user trip or
from a public knowledge database to correspond with times and
places along the user trip being displayed as a geographic route on
the website; and wherein the ancillary information includes one of
a weather report, a temperature, a distance of the user trip, a
duration of the user trip, a start and an ending time of the user
trip, a number of multimedia images and videos associated with the
user trip, a number of segments of the user trip, modes of
transportation used during the user trip, a latitude and longitude
of a location on the user trip, an altitude of a location on the
user trip, news associated with a time or a location of the user
trip, various map views associated with locations along the user
trip, or advertisements associated with locations along the user
trip.
10. The method as recited in claim 2, further comprising deriving
statistics from a user's activity indicated by the personal
knowledge database derived from the user's GPS log files; creating
a visual depiction of the statistics; and displaying the visual
depiction of the statistics to the user on the website or an a
mobile device of the user.
11. The method as recited in claim 10, further comprising
displaying statistics of the user's activities, including one of:
an activity level for each day of the week; a ratio of
transportation modes used over a time interval; a pattern of
arrivals to or departures from a routine user location.
12. A system, comprising: a web user interface for uploading a
user's GPS log files and multimedia contents to a service; a data
processor to segment the GPS log files into geographic routes
corresponding to user trips; the data processor to associate the
multimedia contents to geographic routes by tagging locations to
the multimedia contents; an indexing engine to index the geographic
routes; a route searching engine to retrieve the geographic routes
by browsing or searching; a database to store GPS information and
multimedia information of the user; and a visualization engine to
display an animation of one of the geographic routes on a map
displayed on the web user interface and to display multimedia
contents associated with times and locations along the geographic
route.
13. The system as recited in claim 12, further comprising a route
searching engine to receive a search criterion from the user,
wherein the route searching engine uses the search criterion to
retrieve geographic routes that meet the search criterion from the
spatial-temporal index; and wherein the geographic routes are
listed as search results on the web user interface.
14. The system as recited in claim 13, wherein the search criterion
comprises one or both of: a spatial search criterion created via
the user selecting a geographic area on the map in order to
retrieve search results comprising geographic routes made as
different user trips within the selected geographic area; or a
temporal search criterion created via the user selecting a time
interval in order to retrieve search results comprising geographic
routes made as different user trips within the selected time
interval.
15. The system as recited in claim 12, further comprising a
recommendation engine for requesting geographic routes and local
features, wherein when a user in a given location requests a
recommendation for a geographic route or a local feature, then
popular geographic routes and popular local features are retrieved
from a public knowledge database; and wherein the public knowledge
database is mined from personal preferences of individual users
derived from the GPS log files uploaded by the individual
users.
16. The system as recited in claim 15, further comprising a mobile
devices interface, wherein when a mobile device in a given location
requests a recommendation for a geographic route or a local
feature, popular geographic routes and popular local features are
retrieved from the public knowledge database and displayed to the
mobile device.
17. The system as recited in claim 12, wherein the visualization
engine displays ancillary information calculated in relation to the
user trip or from a public knowledge database to correspond with
times and places along the user trip being displayed as a
geographic route on the website; and wherein the ancillary
information includes one of a weather report, a temperature, a
distance of the user trip, a duration of the user trip, a start and
an ending time of the user trip, a number of multimedia images and
videos associated with the user trip, a number of segments of the
user trip, modes of transportation used during the user trip, a
latitude and longitude of a location on the user trip, an altitude
of a location on the user trip, news associated with a time or a
location of the user trip, various map views associated with
locations along the user trip, or advertisements associated with
locations along the user trip.
18. The system as recited in claim 12, wherein the visualization
engine derives statistics from a user's activity indicated by the
user's uploaded GPS log files, creates a visual depiction of the
statistics, and displays the visual depiction of the statistics to
the user on the website or an a mobile device of the user.
19. The system as recited in claim 18, wherein the visualization
engine displays statistics of the user's activities, including one
of: an activity level for each day of the week; a ratio of
transportation modes used over a time interval; or a pattern of
arrivals to or departures from a routine user location.
20. A system, comprising: means for segmenting a GPS log file
received from a user into geographic routes; means for tagging
multimedia contents received from the user with geographic
locations from the GPS log file to associate the multimedia
contents with the geographic routes; means for indexing the
geographic routes; and means for displaying an animation of one of
the user trips on a map, wherein the animation displays the
multimedia content at geographic locations on the map corresponding
to the geographic cues associated with the multimedia content.
Description
RELATED APPLICATIONS
[0001] This patent application is related to U.S. patent
application Ser. No. (Attorney docket No. 322852.01) to Wang et
al., entitled, "A Flexible Spatio-Temporal Indexing Scheme for
Large-Scale GPS Track Retrieval," filed concurrently herewith, and
incorporated herein by reference; and to U.S. patent application
Ser. No. (Attorney docket No. 322848.01) to Zheng et al., entitled,
"Learning Transportation Mode from Raw GPS Data," filed
concurrently herewith, and incorporated herein by reference.
BACKGROUND
[0002] With decreasing prices and increasing accuracy in
pinpointing locations, Global Positioning System (GPS) devices such
as GPS phones have become prevalent. As never before, voluminous
GPS log data are accumulated continuously and unobtrusively. These
large volumes of GPS data have given rise to a generation of
conventional applications on the Internet ("the web"). Such
applications allow users to upload, share, and browse GPS track
logs. So far, however, GPS data have been utilized directly in
relatively raw form without conventional applications providing
much insight into the full potential GPS data can provide for
tracking a person's experiences. Moreover, existing search methods
that use tags, such as activity tags and region tags, do not
generally satisfy users' needs to put the GPS data to full personal
use.
[0003] Managing and understanding the collected GPS data are two
important issues that could increase the value of the GPS data and
the applications that use them. On one hand, indexing the
ever-increasing GPS data could enable effective retrieval for users
trying to find GPS data of interest. By understanding a user's GPS
data, an application is more likely to provide novel services that
will stimulate users to passionately contribute their GPS data.
Thus far, however, GPS data are conventionally used in relatively
raw form without much understanding of their full potential.
SUMMARY
[0004] This disclosure describes a system for logging life
experiences using geographic cues. The system variously provides a
comprehensive life-logging tool for recording each life event; a
vacation album for revisiting and reliving vacation routes and
associated photos; an information service for finding popular
routes and locations; a statistical tool for analyzing metrics of a
person's life; and a personal website service for sharing personal
information. In one implementation, the system receives a user's
GPS log files and multimedia content at a website. The system
segments the GPS log files into geographic routes corresponding to
user trips, and tags the multimedia content with geographic cues
from the GPS log files. Then, the system indexes the geographic
routes so that users can retrieve the geographic routes by browsing
or by search techniques. The system displays animations of selected
routes on a map, and displays the multimedia content at
corresponding locations along the map route, as the route is
replayed. The system also provides browsing and spatial and
temporal techniques to search a person's travels and can provide
graphical displays of the person's activity statistics.
[0005] This summary is provided to introduce the subject matter of
a system for logging life experiences using geographic cues, which
is further described below in the Detailed Description. This
summary is not intended to identify essential features of the
claimed subject matter, nor is it intended for use in determining
the scope of the claimed subject matter.
BRIEF DESCRIPTION OF THE DRAWINGS
[0006] FIG. 1 is a diagram of an exemplary system for logging life
experiences using geographic cues.
[0007] FIG. 2 is a block diagram of an exemplary life-logging
framework.
[0008] FIG. 3 is a screen shot of an exemplary web user interface
of a life-logging framework.
[0009] FIG. 4 is a diagram of an exemplary spatial search user
interface.
[0010] FIG. 5 is a diagram of an exemplary temporal search user
interface
[0011] FIG. 6 is a diagram of exemplary statistical activity
information derived from a user's GPS log files.
[0012] FIG. 7 is a diagram of an exemplary ratio of a user's
transportation modes derived from the user's GPS log files.
[0013] FIG. 8 is a diagram of exemplary statistical arrival and
departure patterns derived from the user's GPS log files.
[0014] FIG. 9 is a flow diagram of an exemplary method of logging
life experiences using geographic cues.
DETAILED DESCRIPTION
[0015] Overview
[0016] This disclosure describes systems and methods for logging
life experiences using geographic cues. An exemplary system
provides effective visualization, organization, effective mining,
and fast retrieval of GPS log data for both personal and public
use. The system not only provides a powerful platform for enabling
users to effectively manage their personal GPS data but also helps
users access and understand other users' experiences via exemplary
processing of raw GPS data.
[0017] In one implementation, an exemplary system provides a
website at which users may upload their GPS logs and associated
multimedia content, such as digital photographs, videos, and audio
clips, etc. The system determines specific GPS "trip" or "journey"
tracks from the GPS data, and tags the multimedia content with
corresponding GPS location coordinates, indexing GPS trajectories
uploaded by users based at least in part on user behavior in
uploading GPS data. The terms GPS track, GPS trajectory, trip,
route, and journey are used somewhat interchangeably herein.
[0018] In a database for each user, the system processes the GPS
data and related multimedia content into a spatial-temporal index.
Users can voluntarily enable public access of their personal GPS
and multimedia data, which is processed into a public database of
useful, favorite places and travel routes.
[0019] An exemplary user interface (UI) visualizes the GPS data
over digital maps and displays associated past events actively via
the multimedia content, e.g., as an animation. In one
implementation, the system includes a calendar-based browsing
method and search methods to navigate the user's GPS log data and
corresponding multimedia data. By browsing, users can view a day's
worth of data over maps by clicking the date in a calendar. By
searching, users can retrieve the data over a longer term by
selecting a spatial area of the map and/or by selecting a time
interval as the query. A results list of trips with embedded
multimedia data are generated by both techniques. Given a results
list, users can sort the results according to different features,
e.g., the start/end time, average rate of speed, length of a trip,
etc. Moreover, when using the search techniques, users can present
the search results by day, and then rank them according to the
features mentioned above.
[0020] By clicking a trip in the results list, users can view the
trip's detailed information including the start/end time, length,
duration, average rate, number of images photographed along the
route, etc. Then users can play the trip in animation. An icon
representing the user moves along the selected trip route on the
map and actively displays/plays the content of the multimedia data
at the places where the videos, photos, audio clips, etc., were
acquired. Meanwhile, the exact time and name of each location the
user traversed is simultaneously displayed during the process of
animation, along with ancillary information, such as the weather at
that time, news events that were occurring at the same time,
etc.
[0021] In one implementation, the system interprets aspects of a
GPS trajectory, including inferring its transportation mode, i.e.,
which segments of the GPS trajectory represent car travel, a bus
ride, a train ride, a boat ride, bicycling, walking, running,
etc.
[0022] In addition, given a GPS track log and associated multimedia
data that users have created, the system helps users to visualize
their personal events on web maps and to understand their life
patterns, for example, to obtain such useful and interesting
information as: [0023] statistical information about a user's
outdoor movements; [0024] ratios between different transportation
modes during a journey or during a time interval; [0025] statistics
on the time of day that one leaves a workplace, etc.
[0026] Thus, in one implementation, the exemplary system provides a
GPS log-driven application over web maps. Given a GPS track log and
associated multimedia data acquired by users, the system assists
the users to visualize their personal events on web maps and to
understand their life patterns. By optionally publishing their GPS
tracks and associated multimedia content, users can share their
life experiences with others and absorb rich knowledge from others'
GPS tracks.
[0027] The exemplary system thus has many uses, including: 1) as a
comprehensive life-logging tool for recording each life event; 2)
as a vacation album for revisiting and reliving vacation routes and
associated photos, sounds, and multimedia content; 3) as an
information service for finding current popular shopping spots,
acquiring entertainment recommendations, avoiding traffic
congestion, and finding scenic map routes; 4) as a statistical tool
for analyzing metrics of one's own life and forming a
self-improvement plan; and 5) as a personal website service for
sharing personal information with the public and building a social
community, akin to MYSPACE.COM, YOUTUBE.COM, FACEBOOK.COM, etc.
[0028] Exemplary System
[0029] FIG. 1 shows an exemplary system 100 for logging life
experiences using geographic cues. The components and layout of the
exemplary system 100 are just one example for the sake of
description. Other components and layouts are possible for the
exemplary system 100. A user's phone 102 and computing device 104
are communicatively coupled with an exemplary service 106, e.g.,
via the Internet 108. Numerous other users, such as user "N" 110
are also connected to the exemplary service 106.
[0030] The service 106 enables each user 102, 104, 110 to upload
GPS data and multimedia files to a life-logging framework 112,
which in turn can be accessed and queried by user phones 102 and
user computing devices 104, 106. In one implementation, the
life-logging framework 112 includes UI's for computing devices 104,
such as desktop, notebook, and mobile computers, and for mobile
phones 102 and other mobile communication devices. Through the
UI's, users may upload GPS and multimedia content, and in turn,
download animated GPS trajectories of their own and others' trips,
including the multimedia content and other useful information keyed
to the particular route that the GPS trajectory depicts over a
geographic map 114, for example an aerial or bird's-eye view
digital map.
[0031] FIG. 2 shows the exemplary life-logging framework 112 of
FIG. 1, and associated entities, in greater detail. The components
and layout of the exemplary life-logging framework 112 are just one
example for the sake of description. Other components and layouts
are possible for the exemplary life-logging framework 112. In the
illustrated implementation, the life-logging framework 112 includes
the web UI 202 and the mobile devices UI 204 introduced above,
through which users 104 may upload their GPS log files 206 and
their multimedia files 208, such as video clips 210, images 212,
and audio clips 214.
[0032] The exemplary life-logging framework 112 also includes a
data-preprocessing engine 216, for parsing the GPS/multimedia data
218; a GPS data mining engine 220, personal knowledge databases
222, a public knowledge database 224, an indexing engine 226,
spatial-temporal indexes 228, a personal archiving engine 230, a
visualization engine 232, a route searching engine 234, and a
recommendations engine 236.
[0033] Operation of the Exemplary Life-Logging Framework
[0034] When the user's phone 102 or other mobile communication
device is GPS enabled, GPS data 206 from the phone 102 can be
uploaded to the service 106, e.g., either directly or via a
download of the GPS data 206 to the user's computing device 104,
and in turn from the computing device 104 to the service 106. The
user 104 may also upload multimedia files 208 to the service 106
via either the web UI 202 or the mobile devices UI 204.
[0035] After users 104 upload their GPS log data 206 with
corresponding multimedia data 208 to the service 106, the system
segments the GPS log data 206 into several trips if the time
interval between two consecutive GPS points exceeds a time
threshold. Then the pre-processing engine 216 projects the
multimedia data 208 to their associated corresponding geographic
locations. Meanwhile, the indexing engine 226 builds a spatial and
temporal index 228 for the user 104 over the GPS data 206 so that
the user 104 can organize their data more effectively and
efficiently.
[0036] In general, the GPS data mining engine 220 and the indexing
engine 226 of the life-logging framework 112 mine, index, and/or
cross-correlate the GPS data 206 and the uploaded multimedia
content 208 to create an indexed spatial-temporal index 228. In
turn, a personal knowledge database 222 of GPS trajectories is
created as well as related multimedia content 208 for the user 104.
A public knowledge database 224 of GPS trajectories, related
multimedia content 208, and other useful information can be created
for other users of the service 106.
[0037] After the data pre-processing engine 216 parses the received
files 206, 208, it tags each multimedia file 208 with the
corresponding GPS coordinates, e.g., of the geographic location
where the multimedia files 208 were acquired, so that the tagged
GPS/multimedia data 218 are ready for creative and effective
browsing by users 104. In one implementation, based on user
behavior of uploading their GPS trajectories, the indexing engine
226 builds a spatial-temporal index 228 for the user 104 over the
parsed GPS data 218 for rapidly retrieving GPS tracks over maps
114. That is, presented with a search query consisting of a spatial
range selected over a map 114 and/or presented with a temporal
interval of interest, the route searching engine 234 retrieves all
GPS tracks across the spatial range and/or temporal interval. Such
searches are further described in relation to FIGS. 4 and 5.
[0038] For personal use, the personal archiving engine 230 assists
each user 104 to archive his/her own historical data 222 from which
the GPS data mining engine 220 can mine many types of information,
such as personal transportation routes and routines, significant
places, life patterns, etc. These types of information are
processed by the visualization engine 232, which drives and
animates the web UI 202, and the mobile devices UI 204. The
personal knowledge 222 is further leveraged to help users summarize
their own experiences and preferences, for example travel and
sports events, and thereby acquire healthy habits for daily life.
From the public data 224, the route searching engine 234 and the
recommendations engine 236 can learn classic sports routes, popular
travel routes, popular places, and traffic conditions of various
different routes at different times. The recommendations engine 236
presents the mined knowledge 224 to users 104 via the Web UI 202 or
the mobile devices UI 204 when users 104 need suggestions.
[0039] In one implementation, the route searching engine 234
employs a spatio-temporal search function powered by a flexible
indexing scheme based on user behavior of uploading GPS tracks 302.
For example, users 104 are more likely to upload GPS data 206 of
the recent past than of the distant past. Hence, in one
implementation, the life-logging framework 112 leverages a B+ tree
to index frequently updated groups and utilizes a sorted dynamic
array for rarely updated groups. Once the update frequency of a
group drops below a threshold, the indexing engine 226 may convert
the particular index from a B+ tree format to a sorted dynamic
array format. The skewed nature of accumulated GPS tracks 302 is
taken into account, so that compared with conventional
spatio-temporal indexing techniques, the exemplary spatial-temporal
indexes 228 require less index space and less update cost while
keeping satisfactory retrieval performance. The exemplary indexing
techniques are described in detail in the above-cited U.S. patent
application Ser. No. (Attorney docket No. 322852.01) to Wang et
al., entitled, "A Flexible Spatio-Temporal Indexing Scheme for
Large-Scale GPS Track Retrieval," which is incorporated herein by
reference.
[0040] In one implementation, the exemplary life-logging framework
112 employs supervised learning to automatically learn the
transportation modes of a given GPS track 302, including walking,
taking a bus, riding a bike, driving a car, etc., from the raw GPS
data 206. When a GPS log file 206 is uploaded, the data
pre-processing engine 216 divides the GPS track 302 into trips and
then partitions each trip into segments. Then, features from each
segment are extracted and sent to an inference model. Two different
inference models are considered when learning a user's
transportation mode. In one model, the segments of GPS tracks 302
are regarded as independent instances and handled as a normal
classification problem with general classifiers. After the
inference, post-processing is performed to improve the prediction
accuracy by taking the transition probability between different
transportation modes into account. In the other inference model, a
conditional random field (CRF) technique is leveraged to perform
the inference. Since the conditional probabilities between
different transportation modes are considered in the CRF's
graphical model, post-processing is not performed with this
model.
[0041] Advantages of the exemplary automatic techniques for
learning transportation modes of a user 104 from raw GPS data 206
are: 1) the exemplary techniques can infer compound trips, which
include more than one kind of transportation mode; 2) the exemplary
techniques are independent of other information from maps and other
sensors; and 3) the transportation mode inference model learned
from the data set of some users 104 can be applied to infer GPS
data of other users. Automatically learning the transportation
modes of a user 104 from raw GPS data 206 and forming
transportation mode inference models is further described in the
above-cited U.S. patent application Ser. No. (Attorney docket No.
322848.01) to Zheng et al., entitled, "Learning Transportation Mode
from Raw GPS Data," which is incorporated herein by reference. Once
the segments of a GPS track 302 are associated with various
transportation modes, the web UI 202 or the mobile devices UI 204
can display an icon of the current transportation mode (car icon,
train icon, etc.) as the user 104 traverses the GPS track 302 that
is on display.
[0042] Exemplary User Interfaces
[0043] FIG. 3 shows an exemplary web UI 202. Compared to
conventional text-based techniques for presenting GPS data, the
exemplary web UI 202 provides a more creative, concise, and
explicit approach for expressing user experiences. Hence, users 104
can more accurately and colorfully connect to their personal trips
and past events, and also obtain more information from other
people's experience when they browse a GPS track 302. The exemplary
web UI 202 includes various options, including an upload option 304
for uploading GPS log files 206 and multimedia content 208, a
search option 306 for specifying a spatial and/or temporal search
query, a navigation option 308 for browsing for past trips via
calendar based browsing or map-based browsing, and a "what's hot"
option 310 for accessing the recommendations engine 236 in order to
find a popular or suggested travel route, commercial establishment,
or popular location.
[0044] When the navigation option 308 of the exemplary web UI 202
is selected, a calendar-based browsing technique enables navigating
the user's GPS tracks 302 and corresponding multimedia data 208,
i.e., from the personal knowledge database 222 and the
spatial-temporal indexes 228. In navigation (or browsing) mode, the
user 104 can view their data one day at a time over maps by
clicking a date 312 in the calendar 314. A results list 316 of
trips is generated and displayed, each trip including embedded
multimedia data 208 that is activated for display when an icon 320
representing the user 104 arrives at the place in the displayed
route that is associated with the particular multimedia
content.
[0045] Given the results list 316, users 104 can sort the trips
according to different features, e.g., the start/end time, average
rate, or the length of a trip. Then, by clicking a trip in the
results list 316, users 104 can view the trip's detailed
information 318 including the start/end time, length, duration,
average rate, number of images taken along the trip's route, etc.
Then the user 104 can play the trip in animation. An icon 320
representing the user 104 moves along the selected trip route 302
and shows the content of multimedia data 208 at each place where a
respective multimedia content 208 was acquired. The exact time and
name of each location along the route 302 can also be
simultaneously displayed during the animation.
[0046] FIG. 4 shows a search option 306 of the exemplary web UI
202. In one implementation, a user 104 may search for (past or
public) GPS tracks 302 by spatial area 402, by time period, or by
both of these criteria at once in a spatial-temporal Boolean
combination.
[0047] Like the browse/navigation option 308 described above with
respect to FIG. 3, the spatial search option 306 of FIG. 4
generates a results list 316 of trips that is displayed in list
form (316) and/or as visualized routes, e.g., routes 1-7, on the
displayed map 114. The user 104 can arbitrarily scale the range of
the selected spatial area 402 over the map 114. As shown in FIG. 4,
the route searching engine 234 will retrieve the GPS tracks not
only within the selected spatial area 402 but also those GPS tracks
intersecting but not lying fully inside the spatial area 402. For
example, in FIG. 4, GPS tracks 1, 3, 4, and 6 are retrieved for the
results list 316 by the search function because they are fully
contained by the selected spatial area 402; GPS tracks 2, 5, and 7
are also retrieved for the results list 316 by the search function
because they intersect the selected spatial area 402 even though
these GPS tracks are not fully contained by the selected spatial
area 402. GPS track 8, on the other hand, is not captured for the
results list 316 because track 8 is not contained by the selected
spatial area 402 and does not intersect the selected spatial area
402.
[0048] Each trip includes embedded multimedia data 208 that is
activated for display when an icon 320 representing the user 104
arrives at the place in the displayed route that is associated with
the particular multimedia content. In the search options 306, the
user 104 can increase the number of GPS tracks 302 to be returned
in the results list 316 by increasing the spatial area selected
over the map 114 and/or by selecting a larger time interval for the
query. Moreover, when using the search options 306, the user 104
can present the search results by day, and then rank or sort these
results according to features mentioned above: i.e., the start/end
time, average rate, length of a trip, etc.
[0049] FIG. 5 shows a temporal search option 502 of the exemplary
web UI 202. In one implementation, the user 104 may search for GPS
tracks 302 by time interval. Any GPS tracks during the designated
time interval, such as the illustrated tracks 1 and 2, are
displayed in a results list 316 and/or as routes visualized on the
map 114. Each route includes embedded multimedia data 208 that is
activated for display when an icon 320 representing the user 104
arrives at the place in the displayed route that is associated with
the particular multimedia content. When using the temporal search
options 502, the user 104 can present the search results by day,
and then rank or sort these results according to features mentioned
above: i.e., the start/end time, average rate, length of a trip,
etc.
[0050] FIG. 6 shows exemplary visualizations of information from
the personal knowledge database 222. As a user 104 logs life events
using GPS geographic cues via the life-logging framework 112, the
visualization engine 232 can produce graphic compilations of useful
data to display on the web UI 202 to help the user 104 visualize
his/her life. In FIG. 6, the visualization engine 232 shows mean
distance of outdoor movement across days of the week, and mean
duration of outdoor movement across days of the week. In one
implementation, the life-logging framework 112 includes logic
and/or a statistics engine to draw suggestive conclusion to display
with the compilations of life event data. Thus, in FIG. 6, both the
graph of the mean distance of outdoor movement across days of the
week and the mean duration of outdoor movement across days of the
week suggest that the user 104 is more active on the weekends than
during the work-week, and may suggest an activity norm for
comparison drawn from the public knowledge database 224.
[0051] FIG. 7 shows another exemplary visualization of information
from the personal knowledge database 222. As the user 104 logs life
events using GPS geographic cues via the life-logging framework
112, the visualization engine 232 may produce a graphic compilation
showing a ratio of transportation modes used over an interval of
time. The life-logging framework 112 may include logic and/or a
statistics engine to draw a suggestive conclusion, for example,
that the user 104 should consider driving their car less and
walking more for health or taking the bus to help the
environment.
[0052] FIG. 8 shows another exemplary visualization of information
from the personal knowledge database 222. As the user 104 logs life
events using GPS geographic cues via the life-logging framework
112, the visualization engine 232 may produce a graphic compilation
showing a graph of the time of day that the user 104, for example,
arrives home from work or school (or leaves work or school). The
life-logging framework 112 may include logic and/or a statistics
engine to draw a suggestive conclusion that the user 104 arrives
home later than usual during a certain time of the year, e.g., in a
December pre-holiday period, and should consider maintaining a life
balance, including maintaining enough sleep, during such times.
[0053] Exemplary Methods
[0054] FIG. 9 shows an exemplary method 900 of logging life
experiences using geographic cues. In the flow diagram, the
operations are summarized in individual blocks. The exemplary
method 900 may be performed by combinations of hardware, software,
firmware, etc., for example, by components of the exemplary
life-logging framework 112.
[0055] At block 902, GPS log files and multimedia content are
received from a user. In one implementation, the GPS files and
multimedia content are received at a service via a website. The
user can upload GPS information and multimedia content, such as
digital images, videos, and audio clips, via computer or mobile
communication device, such as a cell phone.
[0056] At block 904, the GPS log files are segmented into
geographic routes. Various schemes and algorithms can be employed
to partition GPS log files into separate geographic routes, or
trips. For example, a marked break in geographic continuity between
two temporally successive GPS points might suggest the end of one
route and the beginning of another--as when the user turns off the
GPS aware device between locations. Sometimes a logical
circumstance can be imposed on otherwise contiguous GPS locations
to delineate separate GPS tracks. For example, sometimes the method
900 can be programmed to break a geographic route into two routes
at a logical transition, e.g., when the user transitions from
land-based travel and boards an ocean-going ship. In general,
geographic routes can be distinguished from each other by passage
of a certain time interval that surpasses a threshold between GPS
points.
[0057] At block 906, the multimedia contents are associated with
geographic routes by tagging locations to each piece of multimedia
content. After the data pre-processing and parsing the received
files, each multimedia file is tagged with corresponding GPS
coordinates, e.g., of the geographic location where the multimedia
files were acquired, or to which they are otherwise associated, so
that the tagged GPS/multimedia data are ready for creative and
effective browsing or searching.
[0058] At block 908, the geographic routes are indexed. Based on
user's uploading of their GPS data, the technique builds a
spatial-temporal index for the user over the parsed GPS data for
rapidly retrieving GPS tracks over maps.
[0059] At block 910, the geographic routes are retrieved by
browsing or searching. The user can browse, e.g., by calendar day,
or submit a search query consisting of a spatial range selected
over a map and/or a temporal interval of interest. The technique
retrieves all GPS tracks across the spatial range and/or temporal
interval. Alternatively, the user performs an image/multimedia
search or otherwise finds a stored piece of multimedia content.
Finding the multimedia content then returns any GPS track(s)
associated with the multimedia content.
[0060] At block 912, an animation of a geographic route is
displayed along with the multimedia content at corresponding
locations along the geographic route. That is, the user can
play/relive the route or trip forward and backwards over the map.
An icon represents the user and/or the icon represents the current
mode of transportation for a given segment of a journey.
[0061] When a piece of multimedia content has been linked to a
location or a time along the route, then at that part of the GPS
track being displayed in animation, the multimedia content is shown
or played. Other information related to a person's personal life
patterns can also be derived from the uploaded GPS log files and
visualized for the user on a computer display or mobile phone. For
example, the method 900 can derive common user routines or favorite
routes and locations.
[0062] The method 900 can also show graphic summaries of activity
levels per day, week, or month, etc., and can graph movement
patterns and related statistics. The method 900 can also analyze
and present statistics about a user's transportation modes in
graphical form. In one implementation, the method 900 analyzes a
person's personal patterns and makes suggestions based on
pre-programming or based on comparison with norms derived from a
public database of other users' optionally shared life
patterns.
[0063] Conclusion
[0064] Although exemplary systems and methods have been described
in language specific to structural features and/or methodological
acts, it is to be understood that the subject matter defined in the
appended claims is not necessarily limited to the specific features
or acts described. Rather, the specific features and acts are
disclosed as exemplary forms of implementing the claimed methods,
devices, systems, etc.
* * * * *