U.S. patent application number 16/712824 was filed with the patent office on 2020-07-23 for generating composite images for display on a mobile device based on ground truth image rendering.
The applicant listed for this patent is Uber Technologies, Inc.. Invention is credited to Gregory Mark Mahowald, Aaron Matthew Rogan, Taehun Yoon.
Application Number | 20200232803 16/712824 |
Document ID | / |
Family ID | 71608322 |
Filed Date | 2020-07-23 |
United States Patent
Application |
20200232803 |
Kind Code |
A1 |
Rogan; Aaron Matthew ; et
al. |
July 23, 2020 |
GENERATING COMPOSITE IMAGES FOR DISPLAY ON A MOBILE DEVICE BASED ON
GROUND TRUTH IMAGE RENDERING
Abstract
Systems and methods are disclosed herein for monitoring a
location of a vehicle approaching a pickup location associated with
a transportation service and generating composite images for
display on a driver client device in response to determining that
the vehicle is within a threshold distance of a pickup location.
The systems and methods use sensor data from the driver client
device and a device localization process to monitor the location of
the driver client device located in the vehicle. In response to
determining that the driver client device (and thus, the vehicle)
is within a threshold distance of the pickup location the service
generates one or more composite images by overlaying a stored image
of the pickup location with one or more persistent augmented
reality elements associated with the pickup location and/or the
transportation service.
Inventors: |
Rogan; Aaron Matthew;
(Westminster, CO) ; Yoon; Taehun; (Superior,
CO) ; Mahowald; Gregory Mark; (Broomfield,
CO) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Uber Technologies, Inc. |
San Francisco |
CA |
US |
|
|
Family ID: |
71608322 |
Appl. No.: |
16/712824 |
Filed: |
December 12, 2019 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
62812098 |
Feb 28, 2019 |
|
|
|
62795988 |
Jan 23, 2019 |
|
|
|
62812101 |
Feb 28, 2019 |
|
|
|
62801010 |
Feb 4, 2019 |
|
|
|
62801012 |
Feb 4, 2019 |
|
|
|
62802145 |
Feb 6, 2019 |
|
|
|
62812107 |
Feb 28, 2019 |
|
|
|
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06K 9/6215 20130101;
H04L 67/18 20130101; H04W 4/029 20180201; G01C 21/3438 20130101;
G01C 21/3644 20130101; H04W 4/021 20130101; G01S 19/48 20130101;
G06F 16/587 20190101; G06F 3/016 20130101; G01C 21/343 20130101;
G06F 16/5866 20190101; G06F 16/7837 20190101; G06T 19/006 20130101;
H04B 1/3827 20130101; G06K 9/00671 20130101; G06Q 10/02 20130101;
G06T 2200/24 20130101; G06K 9/6201 20130101; G06K 9/00791 20130101;
H04N 5/23222 20130101; G06K 9/00624 20130101; G06Q 50/30 20130101;
H04W 4/185 20130101; G06T 19/20 20130101; G06T 2219/024 20130101;
G06Q 10/047 20130101 |
International
Class: |
G01C 21/34 20060101
G01C021/34; G06F 16/587 20060101 G06F016/587; G06K 9/00 20060101
G06K009/00; G06Q 10/04 20060101 G06Q010/04; G06Q 50/30 20060101
G06Q050/30; G06Q 10/02 20060101 G06Q010/02; H04W 4/021 20060101
H04W004/021; H04W 4/029 20060101 H04W004/029 |
Claims
1. A computer-implemented method for generating images associated
with a transportation service, the method comprising: receiving
confirmation from a client device of a driver that the driver will
navigate to a pickup location; in response to receiving the
confirmation, monitoring a location of the client device to
determine whether the client device is within a threshold distance
of the pickup location; determining, based on the monitoring, that
the client device is located with the threshold distance of the
pickup location; in response to determining that the client device
is located within the threshold distance of the pickup location,
selecting, based in part on a direction that the client device is
facing, an image of one or more candidate images that each
correspond to the pickup location; and instructing the client
device to generate for display the image.
2. The method of claim 1, further comprising: generating persistent
augmented reality content for display on the client device, the
persistent augmented reality content associated with the pickup
location; and instructing the client device to display the
persistent augmented reality content as an overlay on top of the
image.
3. The method of claim 1, wherein determining that the client
device is within the threshold distance of the pickup location
comprises: responsive to an occurrence of a condition, instructing
the client device to capture an image as the client device
approaches the pickup location; comparing a rendering of the
captured image to the plurality of candidate images; and
determining, based on the comparing, that the captured image
matches at least one of the candidate images.
4. The method of claim 3, further comprising determining the
occurrence of the condition by detecting that a global positioning
system (GPS) trace of the client device has crossed within the
threshold distance from the pickup location, wherein the candidate
images include metadata indicating that the candidate images each
correspond to a location that is within the threshold distance of
the pickup location.
5. The method of claim 2, further comprising: identifying, in the
image, a visual indicator of the pickup location; generating
persistent augmented reality content identifying the visual
indicator; and placing the augmented reality content on a portion
of the image where the visual indicator is located.
6. The method of claim 2, further comprising: identifying, in the
image, a visual indicator of a rider associated with the pickup
location; generating persistent augmented reality content
identifying the rider; and placing the augmented reality content on
a portion of the image where the rider is located.
7. The method of claim 1, further comprising: detecting, based in
part on data obtained from a sensor of the client device, that a
vehicle associated with the client device is stopped; and
triggering display of the image in response to detecting that the
vehicle is stopped.
8. A non-transitory computer-readable storage medium storing
computer program instructions executable by a processor to perform
operations comprising: receiving confirmation from a client device
of a driver that the driver will navigate to a pickup location; in
response to receiving the confirmation, monitoring a location of
the client device to determine whether the client device is within
a threshold distance of the pickup location; determining, based on
the monitoring, that the client device is located with the
threshold distance of the pickup location; in response to
determining that the client device is located within the threshold
distance of the pickup location, selecting, based in part on a
direction that the client device is facing, an image of one or more
candidate images that each correspond to the pickup location; and
instructing the client device to generate for display the
image.
9. The non-transitory computer-readable storage medium of claim 8,
wherein the operations further comprise: generating persistent
augmented reality content for display on the client device, the
persistent augmented reality content associated with the pickup
location; and instructing the client device to display the
persistent augmented reality content as an overlay on top of the
image.
10. The non-transitory computer-readable storage medium of claim 8,
wherein determining that the client device is within the threshold
distance of the pickup location comprises: responsive to an
occurrence of a condition, instructing the client device to capture
an image as the client device approaches the pickup location;
comparing a rendering of the captured image to the plurality of
candidate images; and determining, based on the comparing, that the
captured image matches at least one of the candidate images.
11. The non-transitory computer-readable storage medium of claim
10, wherein the operations further comprise determining the
occurrence of the condition by detecting that a global positioning
system (GPS) trace of the client device has crossed within the
threshold distance from the pickup location, wherein the candidate
images include metadata indicating that the candidate images each
correspond to a location that is within the threshold distance of
the pickup location.
12. The non-transitory computer-readable storage medium of claim 9,
wherein the operations further comprise: identifying, in the image,
a visual indicator of the pickup location; generating persistent
augmented reality content identifying the visual indicator; and
placing the augmented reality content on a portion of the image
where the visual indicator is located.
13. The non-transitory computer-readable storage medium of claim 9,
wherein the operations further comprise: identifying, in the image,
a visual indicator of a rider associated with the pickup location;
generating persistent augmented reality content identifying the
rider; and placing the augmented reality content on a portion of
the image where the rider is located.
14. The non-transitory computer-readable storage medium of claim 8,
wherein the operations further comprise: detecting, based in part
on data obtained from a sensor of the client device, that a vehicle
associated with the client device is stopped; and triggering
display of the image in response to detecting that the vehicle is
stopped.
15. A system comprising: a processor for executing computer program
instructions; and a non-transitory computer-readable storage medium
storing computer program instructions executable by the processor
to perform operations for estimating a location of a client device,
the operations comprising: receiving confirmation from a client
device of a driver that the driver will navigate to a pickup
location; in response to receiving the confirmation, monitoring a
location of the client device to determine whether the client
device is within a threshold distance of the pickup location;
determining, based on the monitoring, that the client device is
located with the threshold distance of the pickup location; in
response to determining that the client device is located within
the threshold distance of the pickup location, selecting, based in
part on a direction that the client device is facing, an image of
one or more candidate images that each correspond to the pickup
location; and instructing the client device to generate for display
the image.
16. The system of claim 15, wherein the operations further
comprise: generating persistent augmented reality content for
display on the client device, the persistent augmented reality
content associated with the pickup location; and instructing the
client device to display the persistent augmented reality content
as an overlay on top of the image.
17. The system of claim 15, wherein determining that the client
device is within the threshold distance of the pickup location
comprises: responsive to the occurrence of a condition, instructing
the client device to capture an image as the client device
approaches the pickup location; comparing a rendering of the
captured image to the plurality of candidate images; and
determining, based on the comparing, that the captured image
matches at least one of the candidate images.
18. The system of claim 64, wherein the operations further
comprise: identifying, in the image, a visual indicator of the
pickup location; generating persistent augmented reality content
identifying the visual indicator; and placing the augmented reality
content on a portion of the image where the visual indicator is
located.
19. The system of claim 16, wherein the operations further
comprise: identifying, in the image, a visual indicator of a rider
associated with the pickup location; generating persistent
augmented reality content identifying the rider; and placing the
augmented reality content on a portion of the image where the rider
is located.
20. The system of claim 15, wherein the operations further
comprise: detecting, based in part on data obtained from a sensor
of the client device, that a vehicle associated with the client
device is stopped; and triggering display of the image in response
to detecting that the vehicle is stopped.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional
Application No. 62/812,098, filed Feb. 28, 2019, U.S. Provisional
Application No. 62/795,988, filed Jan. 23, 2019, U.S. Provisional
Application No. 62/812,101, filed Feb. 28, 2019, U.S. Provisional
Application No. 62/801,010, filed Feb. 4, 2019, U.S. Provisional
Application No. 62/801,012, filed Feb. 4, 2019, U.S. Provisional
Application No. 62/802,145, filed Feb. 6, 2019, and U.S.
Provisional Application No. 62/812,107, filed Feb. 28, 2019, which
are incorporated by reference in their entirety.
TECHNICAL FIELD
[0002] The present disclosure relates to location determination
with limited or no reliance on global positioning system (GPS)
signals, and in particular to determining a location estimate for a
client device and generating composite images for display on the
client device based on the determined location.
BACKGROUND
[0003] Many systems may use global positioning system (GPS)
coordinates to estimate the position of persons carrying client
devices. For example, a person (referred to herein as a "rider")
carrying a client device may wish to arrange for transportation
from his or her present location to another location, and may
execute an application (e.g., a transportation service and/or
ridesharing application) on his or her client device to obtain
transportation from a transportation provider (referred to herein
as a "driver"). Existing systems in this scenario match the rider
with a driver, and instruct the driver to travel to a location of
the rider for pickup. However, the GPS coordinates used to estimate
the position of the rider may be unknown or inaccurate due to
interference or reception problems, such as those caused by tall
buildings that distort satellite signals, for example. This
erroneous GPS data (e.g., inaccurate or missing data) may cause
inconveniences for the driver when attempting to locate the rider
for pickup, and/or designating a pickup location altogether.
Similarly, this scenario may cause inconveniences for the rider
while waiting for the driver to travel to the pickup location given
the potentially inaccurate data and cause the rider to consider
alternate applications that offer rideshare services. Furthermore,
it may be difficult for the driver to identify the rider's actual
location, particularly in congested areas with heavy foot traffic,
which may lead to inefficiencies in the pickup process, cause
frustration to the parties, and delay the transportation
service.
[0004] Existing systems seek to solve the technical problem of
locating a rider and identifying the rider to the driver by
prompting riders to manually confirm their locations within the
application. However, these systems can be cumbersome for riders
that are unfamiliar with their surroundings or unsure of their
exact locations. Additionally, many of these systems do not aide
the driver in identifying the rider's true location as the driver
approaches the pickup location if the GPS traces from the rider's
client device, or the rider's confirmed location, are inaccurate.
The technical problem of identifying an accurate location of a
rider to a driver approaching a pickup location is not addressed by
existing systems.
SUMMARY
[0005] Systems and methods are disclosed herein for determining a
location of a vehicle and generating for display on a driver client
device composite images including augmented reality ("AR") elements
associated with a transportation service. To this end, a service
(e.g., that connects a rider with a driver in the context of a
transportation application) receives a request from a user of the
service for transportation to a destination location. In response
to receiving the request, the service matches the requesting user
with one of a plurality of available drivers based in part on the
respective locations of the rider and driver. If the matched driver
accepts an invitation from the service to fulfill the request, the
service monitors the location of the driver's vehicle as the driver
approaches a pickup location for the transportation service.
[0006] In one embodiment, the service initializes the determination
of the vehicle location based on global positioning system (GPS)
data of the driver client device, and continues to use GPS traces
as a proxy for the driver's location until the driver is in the
vicinity of the pickup location. In response to determining that
the vehicle is within a first threshold distance of the pickup
location, to counter the possibility that the driver's GPS traces
are inaccurate (e.g., due to being within an urban canyon), the
service uses a device localization process to further monitor the
vehicle location by comparing renderings of images captured by the
driver client device to candidate images corresponding to the
pickup location. In response to the captured image matching at
least one of the candidate images, the service determines that the
vehicle is within a second threshold distance of the pickup
location and generates for display on the driver client device a
composite image including a base image of the pickup location and a
persistent AR overlay including one or more AR elements associated
with the pickup location and/or the service request.
BRIEF DESCRIPTION OF THE DRAWINGS
[0007] FIG. 1 is a block diagram illustrating a location estimation
system, according to one embodiment.
[0008] FIG. 2 is an illustration of GPS traces in a region where
GPS signals are inaccurate according to one embodiment.
[0009] FIG. 3 is an illustration of a manner in which a vehicle
location is initialized and then updated, according to one
embodiment.
[0010] FIG. 4 is an illustration of a composite image identifying
rider and pickup locations on a driver client device, according to
one embodiment.
[0011] FIG. 5 is an illustrative flowchart of a process for
generating composite images for display on a client device based on
ground truth image rendering, according to one embodiment.
[0012] FIG. 6 is a block diagram that illustrates a computer
system, according to one embodiment.
[0013] The figures depict various embodiments for purposes of
illustration only. One skilled in the art will readily recognize
from the following discussion that alternative embodiments of the
structures and methods illustrated herein may be employed without
departing from the principles of the invention described
herein.
DETAILED DESCRIPTION
System Environment
[0014] FIG. 1 is a block diagram illustration a location estimation
system, according to one embodiment. System 100 includes vehicle
101, which contains or is carrying client device 110. The
functionality of client device 110 is described in further detail
with respect to FIG. 6 below. In some embodiments, client device
110 is integrated into vehicle 101 as a component of vehicle 101.
Client device 110 executes an application, such as a transportation
service and/or ridesharing application where a rider may request a
ride from the rider's current location to a desired destination,
and where the rider may be connected to a driver who also uses the
application, where the driver will provide the ride. While the
driver travels toward a rider, and while the driver is transporting
a rider to a destination, a map may be viewed by the driver or the
rider via the application where an indicator of the rider's
position is displayed. In an exemplary embodiment, client device
110 is mounted on a dashboard or windshield of vehicle 101 and has
a forward-facing camera that faces the road. While this exemplary
embodiment is referred to throughout, in some embodiments, the
application instead commands images to be captured from a
stand-alone camera (e.g., embedded in a device that is affixed to a
dashboard or windshield of vehicle 101).
[0015] In an embodiment, client device 110 automatically captures
one or more images based on commands received from the application.
For example, client device 110 captures a frame upon a certain
condition being satisfied (e.g., a certain distance has been
traveled, or a certain amount of time has passed, from a reference
point, or a certain threshold distance of a pickup location has
been reached). Times at which images are automatically captured by
client device 110 will be described in further detail below with
reference to FIGS. 3-5. Automatic capturing of one or more images
may be an opt-in feature, where the application by default does not
automatically capture images using a camera of client device 110,
and where the application has a setting that, if selected by a user
of client device 110 (e.g., a driver of vehicle 101), enables the
application to automatically capture the images. While accurate
pinpointing of a driver's location using the systems and methods
described herein may rely on opting in, the location of the driver
may be determined based on GPS traces of client device 110 (even if
inaccurate) should a driver of vehicle 101 not opt in.
[0016] In some embodiments, client device 110 transmits the
image(s) to location determination service 130 over network 120,
where location determination service 130 receives the image(s) and
compares them to known images, stored at image rendering database
132, to determine the location of client device 110. The location
determination service 130 compares the determined location to a
pickup location associated with a service request, and responsive
to detecting that client device 110 is within a second threshold
distance of the pickup location, instructs image generation service
134 to generate a composite image of the pickup location for
display on client device 110.
[0017] In some embodiments, the functionality of location
determination service 130, image rendering database 132, and/or
image generation service 134 is located within client device 110,
and thus need not be accessed by network 120, as depicted.
Functionality of location determination service 130 and image
generation service 134 may be integrated as one or more modules of
the application. Image rendering database 132 may be accessed by
location determination service 130 directly, or over network 120.
Location determination service 130 and image generation service 134
may be one or more modules of an application, such as a ridesharing
application, or may be components of a transportation service
generally, such as a ridesharing service. In some embodiments where
location determination service 130 and image generation service 134
are modules of the application, some or all of the contents of
image rendering database 132 are transmitted to the client device
for performing localization at the client device. The functionality
of location determination service 130 and image generation service
134 will be described in further detail below with respect to FIGS.
2-6.
Identifying Regions Prone to Erroneous GPS Readings
[0018] FIG. 2 is an illustration of GPS traces in a region where
GPS signals are inaccurate according to one embodiment. Region 200
includes GPS traces 202 of a client device (e.g., client device
110) as derived from a GPS sensor of client device 110. As an
illustrative example, the GPS traces 202 were derived from client
device 110 while vehicle 101 was on a road. Because of the
existence of tall buildings within region 200, the GPS signals used
to derive the GPS traces are distorted and provide inaccurate GPS
traces. This is evidenced by the GPS traces being at locations that
are not on a road. GPS traces may similarly be unreliable for a
client device associated with a user of a ridesharing application
who is located at, or traveling to, a pickup location.
[0019] Region 200 is exemplary of a location known to location
determination service 130 to have or cause erroneous GPS data. The
identification of various regions, like region 200, which are
associated with erroneous GPS data may be performed automatically
by location determination server 130, or may be made based on
manual feedback (e.g., performed in advance of executing process
500). For example, location determination service 130 may detect
that users of a ridesharing application in a given location set a
pickup pin at a location different from their GPS traces at a
frequency that exceeds an implementer-defined threshold, and may
determine therefrom that GPS data derived from client devices
within that region are likely erroneous. As another example,
location determination service 130 may detect that GPS traces of
users (e.g., drivers) of a ridesharing application are, at a
frequency above a threshold, in areas inaccessible to drivers, such
as within buildings or parks that do not have road access, and may
determine therefrom that GPS data derived from client devices
within that region are likely erroneous. As yet another example,
location determination service 130 may receive feedback from users
that their client devices are determining erroneous locations based
on GPS sensors of those client devices, and may determine therefrom
that GPS data derived from client devices within that region are
likely erroneous.
Exemplary Initialization using GPS and Transition to Image-Based
Localization
[0020] FIG. 3 is an example of a vicinity encompassing a pickup
location, according to one embodiment. In the embodiment shown in
FIG. 3, a rider has requested (e.g., using the above-described
application) a transportation service to a destination location and
a pickup location 300 has been designated as the starting point for
the transportation service. In one embodiment, location
determination service 130 selects a pickup location for the
requesting user and provides the selected pickup location for
display on a rider client device. Alternatively, location
determination service 130 selects one or more candidate pickup
locations within a threshold distance of a current location of a
rider client device and provides the candidate pickup options for
display on the rider client device. In response to receiving a
selection of candidate pickup location from the rider client
device, location determination service 130 sets the selected
location as the pickup location. In still other embodiments,
location determination service 130 may detect that the requesting
user set a pickup pin via the application and may assign the
associated location of the pickup pin as the pickup location.
[0021] In response to receiving the service request, location
determination service 130 matches the requesting user with one of a
plurality of available drivers. In one embodiment, selection of an
available driver is based in part on current locations of driver
and rider client devices as determined automatically using global
positioning system (GPS) data received from the driver and rider
client devices. Location determination service 130 further provides
an invitation message to the matched driver inviting the driver to
fulfill the service request, and if the driver accepts via input to
driver client device 110, location determination service 130
assigns the selected driver to the transportation service.
[0022] In response to assigning the selected driver to the
transportation service, location determination service 130
initiates monitoring of location data of the driver's vehicle (such
as vehicle 101) as vehicle 101 travels from its current location to
the pickup location 300. In one embodiment, location determination
service 130 periodically retrieves GPS traces acquired using a GPS
sensor of client device 110 and compares the retrieved GPS traces
to a GPS identifier of the pickup location 300 to determine a
position of vehicle 101 relative to the pickup location 300.
[0023] Location determination service 130 continues to monitor the
location of vehicle 101 using received GPS traces to detect a
scenario where vehicle 101 reaches a position (e.g., a position "A"
in FIG. 3) within a first threshold distance 310 of the pickup
location 300. Detection that vehicle 101 has reached the first
threshold distance 310 causes location determination service 130 to
trigger a device localization process to continue monitoring the
location of vehicle 101 as vehicle 101 approaches the pickup
location.
[0024] In response to determining that vehicle 101 has reached the
first threshold distance 310 (e.g., is at position A), the
application instructs client device 110 to capture an image that
will be used by location determination service 130 to determine the
location of vehicle 101 without further use of a GPS sensor of
client device 110. The application causes client device 110 to
transmit to location determination service 130 a rendering of the
captured image. Location determination service 130 identifies a
subset of entries of image rendering database 132 that correspond
to a vicinity of the first threshold distance 310 and compares the
rendering to renderings of each entry in the subset. For example,
keypoints (i.e., spatial locations in the rendering that define
areas of interest) of the received rendering may be extracted and
compared to keypoints of candidate renderings to determine whether
a threshold amount of keypoints match. In some embodiments, in
order to perform this determination, location determination service
130 determines that the received rendering does not completely
match any entry of the entries. For example, when comparing
two-dimensional renderings, location determination service 130 may
determine that not all keypoints of the received rendering match
any candidate rendering. When comparing three-dimensional
renderings, location determination service 130 may determine that
the keypoints of the image do not match all keypoints of any
perspective of any candidate rendering.
[0025] Location determination service 130 may first perform the
matching coarsely by leveraging GPS to reduce the search space. By
using a large radius around a query image GPS position, location
determination service 130 could produce candidate images or 3D
sections of the scene against which to match the captured image.
Location determination service 130 may further filter candidate
images or 3D sections by using the heading direction of the query
image or 3D scene coordinates to more readily align them to a "base
map" that stitches together known renderings into a model of the
world (e.g., a map of a 2D or 3D model of known renderings).
Additional techniques such as vocabulary trees, a bag-of-words
model, or machine learning can be used to quickly retrieve a
matching set of images or 3D content.
[0026] In one embodiment, location determination service 130
performs alignment of 3D models to determine a matching rendering.
3D-3D alignment may be performed in various ways. For example,
traditional approaches use an iterative closest point (ICP)
algorithm. The alignment may be seeded using machine-learning
models that semantically segment the 3D scene. The segmentation can
be used to create a coarse alignment between similar semantic
structures, such as, car-to-car, light post-to-light post, and so
forth. After generating the coarse alignment, location
determination service 130 may revert to traditional ICP to perform
the final precision alignment in an accelerated fashion.
[0027] In response to determining that the received rendering does
not completely match any entry of the entries, location
determination service 130 determines that a percentage of
characteristics of the received rendering match characteristics of
the given entry, and determines whether the percentage exceeds a
threshold. In response to determining that the percentage exceeds
the threshold, location determination service 130 determines that
the received rendering matches the given entry based on the partial
match. Likewise, in response to determining that the percentage
does not exceed the threshold, location determination service 130
determines that the received rendering does not match the given
entry notwithstanding the partial match.
[0028] In response to finding a matching rendering, location
determination service 130 determines the location of vehicle 101 at
a location indicated in the entry that includes the matching
rendering (i.e., at a location within the first threshold distance
310). Location determination service 130 continues to monitor the
location of vehicle 101 by instructing client device 110 to capture
one or more frames upon a certain condition being satisfied (e.g.,
a certain distance has been traveled or a certain amount of time
has passed from a reference point or since a previous location of
client device 110 was determined).
[0029] Location determination service 130 continues to use
image-based localization to monitor the location of vehicle 101 to
detect a scenario where vehicle 101 reaches a position (e.g., a
position "B" in FIG. 3) within a second threshold distance of the
pickup location 300. For example, location determination service
130 compares images captured by client device 110 to stored
renderings that each correspond to the pickup location 300
(referred to herein as "candidate images"). If the captured image
matches at least one of the candidate images, location
determination service 130 determines that vehicle 101 is within the
second threshold distance 320 of the pickup location 300.
[0030] While the embodiment discussed above utilizes GPS traces to
monitor the location of client device 110 until client device 110
reaches the first threshold distance 310, in other embodiments,
location determination service 130 performs device localization
before the first threshold distance 310 is reached. For example,
responsive to detecting that vehicle 101 has entered a region where
GPS signals are known to be inaccurate, such as region 200,
location determination service 130 triggers device localization
regardless of whether vehicle 101 has reached position A.
[0031] Responsive to determining that vehicle 101 is within a
second threshold distance 320 of the pickup location 300, location
determination service 130 instructs image generation service 134 to
generate a composite image of the pickup location 300 for display
on client device 110. Alternatively, for safety reasons, triggering
display of the composite image may be based on determining that
vehicle 101 has been stopped for at least a threshold period of
time. For example, location determination service 130 may monitor
sensor data of client device 110 (e.g., data obtained from an
accelerometer of client device 110) and may send the image
generation instruction to image generation service 134 in response
to determining that vehicle 101 is within the second threshold
distance 320 and has been stopped for at least a threshold period
of time (e.g., 10 seconds). The threshold period of time may be
greater than a time period during which vehicle 101 might be
stopped at a stop sign or as a result of sudden braking such that
determining that vehicle 101 has been stopped for at least the
threshold period of time indicates that vehicle 101 is likely
stopped at a traffic signal, at the pickup location, or has
otherwise pulled over and can safely view the composite image. In
another embodiment, if location determination service 130 detects
renewed movement of client device 110 (and thus, vehicle 101) prior
to the driver confirming through client device 110 that the pickup
has occurred and the requested service has started, location
determination service 130 instructs client device 110 to cease
display of the composite image. Responsive to detecting a next stop
of vehicle 101 for over the threshold period of time, location
determination service 130 to resume display of the composite image.
In still another embodiment, display of the image is based on user
input through client device 110. For example, the driver might
request, through the application, an AR option for identifying the
rider associated with the service request. Still further, location
determination service 130 might provide for display on client
device 110 a smaller image of a map of the vicinity of vehicle 101
while vehicle 101 is in motion and may trigger display of the
composite image in response to determining that vehicle 101 has
been stopped for at least the threshold period of time.
[0032] In one embodiment, the composite image includes an image
corresponding to the pickup location (the "base image") overlaid by
one or more persistent augmented reality ("AR") elements associated
with the pickup location and/or the service request. The base image
may be selected by image generation service 134 based on a line of
sight of a camera sensor of client device 110 and/or based a
position and angle of client device 110 (e.g., as measured by an
inertial measuring unit (IMU) of client device 110). In one
embodiment, the base image is the frame captured by client device
110 that matches at least one of the candidate images of the pickup
location retrieved from image rendering database 132.
Alternatively, image generation service 134 selects a candidate as
the base image.
[0033] Image generation service 134 generates an overlay of one or
more static or dynamic computer-generated elements by identifying,
in the base image, at least one visual indicator of the pickup
location and/or the rider associated with the service request and
places the AR elements on a portion of the base image where the
visual indicator is located. For example, in one embodiment, the
visual indicator corresponds to the pickup location and the AR
elements include an identification of the pickup location, such as
a highlighted portion of a road or sidewalk, flag or other
graphical marker, text (e.g., text reading "The pickup location is
20 meters ahead," or "Stop here"), and the like. Similarly, if the
rider has opted-in to sharing the rider's location and/or
photograph with the driver, image generation service 134 determines
whether the base image includes an identification of the rider
(e.g., if the base image is an image captured by client device 110)
using stored data associated with the rider (e.g., the rider's
photograph) and/or location data associated with a rider client
device. As will be discussed below with respect to FIGS. 4 and 6,
if image generation service 134 identifies the rider and/or rider
client device in the base image, image generation service 134 may
generate one or more AR elements associated with the rider, such as
an arrow or other graphical marker pointing to the rider, text
identifying the rider, and the like. Image generation service 314
then superimposes the AR elements on a position of the base image
where the visual indicator is located to generate the composite
image. The composite image is then provided for display on client
device 110. In some embodiments, image generation service 134
generates the composite image in response to location determination
service 130 determining that vehicle 101 is within the second
threshold distance 320 of the pickup location 300, but does not
provide the composite image for display on client device 110 until
location determination service 130 determines that vehicle 101 is
stopped.
[0034] While the embodiment described above generates a composite
image responsive to determining that vehicle 101 is within a second
threshold distance of the pickup location, in some embodiments,
updated composite images may be generated responsive to determining
that vehicle 101 has reached one or more subsequent thresholds.
Image generation service 134 may continuously generate and provide
to client device 110 updated composite images as vehicle 101 moves
toward the pickup location such that the images are displayed in a
video-like fashion on client device 110.
Exemplary Display of Augmented Reality Elements on Driver
Device
[0035] FIG. 4 is an illustration of a composite image 402 displayed
on client device 110. In one embodiment, client device 110 is
mounted (e.g., on the dashboard or windshield) of vehicle 101.
Alternatively, client device 110 may be associated with a heads-up
display unit that displays AR elements in the driver's line of
sight.
[0036] In response to determining that vehicle 101 is within the
second threshold distance of the pickup location, location
determination service 130 instructs image generation service 134 to
generate a composite image of the vicinity of the pickup location.
The details of the device localization process are described above
with respect to FIG. 3, the details of which apply fully hereto. In
response to receiving the image generation instruction, image
generation service 134 selects a base image of the pickup location
for display on client device 110 and generates one or more
persistent static and/or dynamic AR elements associated with the
pickup location and/or the service request.
[0037] In one embodiment, image generation service 134 generates
the AR elements using the camera direction and tilt of client
device 110. For example, if location determination service 130
determines that the pickup location is within a line of sight of a
camera sensor of client device 110, image generation service 134
generates one or more AR elements to be superimposed on the base
image at positions of the base image corresponding to visual
indicators of the pickup location and/or the requesting rider. In
the displayed composite image 402, client device 110 displays an
image of a road approaching a pickup location. The base image is
overlaid with graphical and textual elements identifying the visual
indicators (e.g., the rider and the pickup location). For example,
composite image 402 includes an arrow 404 pointing to the pickup
location and a text box 406 including the rider's name and rating,
and the distance from client device 110 to the rider's position.
Responsive to determining that the rider has moved (e.g., based on
data received from the rider client device), image generation
service 134 generates an updated composite image for display on
client device 110 showing an updated rider location. In this way,
real-time or near real-time location changes of the rider may be
shown with AR overlays on client device 110. In various
embodiments, the text box 406 includes different information about
the rider based on rider-specified settings. Additionally or
alternatively, the AR elements may include graphical and/or textual
elements identifying the pickup location, such as a highlighted
portion of the road, sidewalk, and/or other element of the base
image associated with the pickup location (e.g., a pole, a sign, a
tree, and the like).
Exemplary Composite Image Generation for Display on Driver
Device
[0038] FIG. 5 is an illustrative flowchart of a process for
generating a composite image of a pickup location for display on a
client device (e.g., client device 110 within vehicle 101), in
accordance with some embodiments of the disclosure. A service,
(e.g., a service that connects a rider with a driver in the context
of a transportation application) receives a request for a ride from
a user client device. Upon receiving the request, location
determination service 130 identifies a location of the user and
assigns a driver to service the service request based in part on a
comparison of the user's location and the locations of a plurality
of candidate drivers. Process 500 begins by the service receiving
502 a confirmation from a client device (e.g., client device 110
within vehicle 101) of the assigned driver that the driver will
navigate to a pickup location associated with the service
request.
[0039] As the driver travels to the designated pickup location,
location determination service 130 monitors 504 the location of
client device 110 determine when client device 110 is within first
and second threshold distances of the pickup location. The details
of the device monitoring process are described above with respect
to FIG. 3, the details of which apply fully hereto. For example, in
some embodiments, location determination service 130 initially
monitors the location of client device 110 using GPS sensor data of
client device 110, and, responsive to determining that client
device 110 (and thus, vehicle 101) is within a first threshold
distance of the pickup location, triggers a device localization
process to further monitor the location of vehicle 101.
[0040] Location determination service 130 continues to monitor the
location of client device 110 as vehicle 101 approaches the pickup
location to determine 506 whether client device 110 is within a
second threshold distance of the pickup location. Renderings of
images captured by client device 110 are compared to a plurality of
candidate images corresponding to the pickup location. If the
rendering of the captured image does not match at least one of the
candidate images, location determination service 130 continues to
monitor the location of client device 110 until location
determination service 130 determines that an image captured by
client device 110 matches a candidate image. In response to
determining that the captured rendering matches at least one
candidate image, location determination service 130 determines that
the location of vehicle 101 is the location associated with the
matching rendering and that vehicle 101 is within a second
threshold distance of the pickup location.
[0041] In response to determining that client device 110 is located
within the second threshold distance of the pickup location (and,
optionally, in response to determining that vehicle 101 has been
stopped for at least a threshold period of time, e.g., based on an
accelerometer of client device 110), location determination service
130 instructs image generation service 134 to generate a composite
image of the pickup location for display on client device 110.
Process 500 continues with image generation service 134 selecting
508 an image of a vicinity of the pickup location (the "base
image"). In one embodiment, image generation service 134 selects as
the base image the frame captured by client device 110 that matches
at least one candidate image. Alternatively, the matching candidate
image is selected as the base image. In still other embodiments,
image generation service 134 selects a different image of the
pickup location stored in image rendering database 132 based in
part on a direction that client device 110 is facing and/or a
portion of the pickup location that is within a line of sight of a
camera sensor of client device 110.
[0042] Image generation service 134 generates the composite image
by identifying, in the base image, at least one visual indicator of
the pickup location and/or the requesting rider, generating at
least one persistent AR element associated with the visual
indicator, and superimposing the generated element on a portion of
the base image where the visual indicator is located. In one
embodiment, image generation service 134 determines whether the
rider associated with the service request has consented to sharing
the rider's location and/or photograph with the driver. If image
generation service 134 determines that the rider has consented to
such data sharing and if image generation service 134 determines
that the rider is visible in the base image (e.g., in instances
where the base image is an image captured by client device 110),
image generation service 134 generates one or more AR elements
identifying the rider for display on client device 110.
Additionally or alternatively, image generation service 134
generates one or more graphical and/or textual AR elements
identifying the pickup location, as discussed above with respect to
FIGS. 3 and 4. Image generation service 134 overlays the one or
more AR elements onto the base image to generate the composite
image and instructs client device 110 to generate for display the
composite image.
Computing Hardware
[0043] The entities shown in FIG. 1 are implemented using one or
more computers. FIG. 6 is a block diagram that illustrates a
computer system 600 for acting as a client 110 or location
determination service 130, according to one embodiment. Illustrated
are at least one processor 602 coupled to a chipset 604. Also
coupled to the chipset 604 are a memory 606, a storage device 608,
a keyboard 610, a graphics adapter 612, a pointing device 614, and
a network adapter 616. A display 618 is coupled to the graphics
adapter 612. In one embodiment, the functionality of the chipset
604 is provided by a memory controller hub 620 and an I/O
controller hub 622. In another embodiment, the memory 606 is
coupled directly to the processor 602 instead of the chipset
604.
[0044] The storage device 608 is any non-transitory
computer-readable storage medium, such as a hard drive, compact
disk read-only memory (CD-ROM), DVD, or a solid-state memory
device. The memory 606 holds instructions and data used by the
processor 602. The pointing device 614 may be a mouse, track ball,
or other type of pointing device, and is used in combination with
the keyboard 610 to input data into the computer system 600. The
graphics adapter 612 displays images and other information on the
display 618. The network adapter 616 couples the computer system
600 to the network 120.
[0045] As is known in the art, a computer 600 can have different
and/or other components than those shown in FIG. 6. In addition,
the computer 600 can lack certain illustrated components. For
example, the computer acting as the location determination service
130 can be formed of multiple blade servers linked together into
one or more distributed systems and lack components such as
keyboards and displays. Moreover, the storage device 608 can be
local and/or remote from the computer 600 (such as embodied within
a storage area network (SAN)).
Additional Considerations
[0046] The foregoing description described one embodiment of the
invention in which a central server including location
determination service 130, image rendering database 132, and image
generation service 134 monitors the location of a client device
(such as client device 110) and generates at least one composite
image for display on the client device. In other embodiments, some
or all of the functions described from the perspective of the
central server are performed on the client device. For example,
location determination service may be a module installed in a
transportation application executing on the client device.
Additionally, while the embodiment described herein generates
composite images comprising a base image overlaid with one or more
computer-generated AR elements, in other embodiments, the AR
elements are overlaid on a video feed captured by a camera of
client device 110. Still further, while the described embodiment
describes a transportation service provider as a driver of a
vehicle, in other embodiments, the transportation service provider
is an autonomous vehicle that transports the rider to the
destination location.
[0047] The foregoing description has been presented for the purpose
of illustration; it is not intended to be exhaustive or to limit
the invention to the precise forms disclosed. Persons skilled in
the relevant art can appreciate that many modifications and
variations are possible in light of the above disclosure.
[0048] Some portions of this description describe embodiments in
terms of algorithms and symbolic representations of operations on
information. These algorithmic descriptions and representations are
commonly used by those skilled in the data processing arts to
convey the substance of their work effectively to others skilled in
the art. These operations while described functionally
computationally or logically are understood to be implemented by
computer programs or equivalent electrical circuits microcode or
the like. Furthermore, it has also proven convenient at times to
refer to these arrangements of operations as modules without loss
of generality. The described operations and their associated
modules may be embodied in software firmware hardware or any
combinations thereof.
[0049] Any of the steps operations or processes described herein
may be performed or implemented with one or more hardware or
software modules alone or in combination with other devices. In one
embodiment a software module is implemented with a computer program
product comprising a computer-readable medium containing computer
program code which can be executed by a computer processor for
performing any or all of the steps operations or processes
described.
[0050] Embodiments may also relate to an apparatus for performing
the operations herein. This apparatus may be specially constructed
for the required purposes and/or it may comprise a general-purpose
computing device selectively activated or reconfigured by a
computer program stored in the computer. Such a computer program
may be stored in a non-transitory tangible computer readable
storage medium or any type of media suitable for storing electronic
instructions which may be coupled to a computer system bus.
Furthermore, any computing systems referred to in the specification
may include a single processor or may be architectures employing
multiple processor designs for increased computing capability.
[0051] Embodiments may also relate to a product that is produced by
a computing process described herein. Such a product may comprise
information resulting from a computing process where the
information is stored on a non-transitory tangible computer
readable storage medium and may include any embodiment of a
computer program product or other data combination described
herein.
[0052] Finally, the language used in the specification has been
principally selected for readability and instructional purposes,
and it may not have been selected to delineate or circumscribe the
inventive subject matter. It is therefore intended that the scope
of the invention be limited not by this detailed description but
rather by any claims that issue on an application based hereon.
Accordingly, the disclosure of the embodiments of the invention is
intended to be illustrative but not limiting of the scope of the
invention which is set forth in the following claims.
* * * * *