U.S. patent application number 17/582236 was filed with the patent office on 2022-05-19 for v2x with 5g/6g image exchange and ai-based viewpoint fusion.
The applicant listed for this patent is R. Kemp Massengill, David E. Newman. Invention is credited to R. Kemp Massengill, David E. Newman.
Application Number | 20220157168 17/582236 |
Document ID | / |
Family ID | |
Filed Date | 2022-05-19 |
United States Patent
Application |
20220157168 |
Kind Code |
A1 |
Newman; David E. ; et
al. |
May 19, 2022 |
V2X with 5G/6G Image Exchange and AI-Based Viewpoint Fusion
Abstract
Autonomous vehicles are required to communicate with each other
in 5G or 6G, to avoid hazards, mitigate collisions, and facilitate
the flow of traffic. However, for cooperative action, each vehicle
must determine the wireless address and position of other vehicles
in proximity, so that they can communicate directly with each
other. It is not sufficient to know the wireless address alone; the
wireless address must be associated with an actual vehicle in view.
Methods disclosed herein enable vehicles to simultaneously acquire
360-degree images of other vehicles in traffic, and transmit those
images wirelessly along with their wireless addresses. The various
images are then "fused" by identifying objects that are viewed from
at least two directions, and calculating their positions by
triangulation. The resulting traffic map, or a listing of the
vehicle positions, is then broadcast along with the wireless
addresses of the vehicles The vehicles can then determine which
wireless address belongs to which of the vehicles in proximity, and
can thereby cooperate with each other to avoid accidents and
facilitate the flow of traffic.
Inventors: |
Newman; David E.; (Palos
Verdes, CA) ; Massengill; R. Kemp; (Palos Verdes,
CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Newman; David E.
Massengill; R. Kemp |
Palos Verdes
Palos Verdes |
CA
CA |
US
US |
|
|
Appl. No.: |
17/582236 |
Filed: |
January 24, 2022 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
63260814 |
Sep 1, 2021 |
|
|
|
63243437 |
Sep 13, 2021 |
|
|
|
63245227 |
Sep 17, 2021 |
|
|
|
63246000 |
Sep 20, 2021 |
|
|
|
63256042 |
Oct 15, 2021 |
|
|
|
63271335 |
Oct 25, 2021 |
|
|
|
63272352 |
Oct 27, 2021 |
|
|
|
63287428 |
Dec 8, 2021 |
|
|
|
63288237 |
Dec 10, 2021 |
|
|
|
63288807 |
Dec 13, 2021 |
|
|
|
63290731 |
Dec 17, 2021 |
|
|
|
International
Class: |
G08G 1/137 20060101
G08G001/137; H04W 4/46 20060101 H04W004/46; G08G 1/04 20060101
G08G001/04; G08G 1/056 20060101 G08G001/056; G08G 1/16 20060101
G08G001/16; G08G 1/017 20060101 G08G001/017; G08G 1/09 20060101
G08G001/09; G08G 1/01 20060101 G08G001/01 |
Claims
1. A method for a first vehicle to communicate with a second
vehicle, the second vehicle proximate to a third vehicle, the
method comprising: a. broadcasting a planning message specifying a
particular time; b. at the particular time, acquiring a first image
depicting the second vehicle and the third vehicle; c. receiving,
from the second vehicle, an imaging message comprising a second
image, the second image acquired by the second vehicle at the
particular time, the second image depicting the first vehicle and
the third vehicle; and d. determining, according to the first image
and the second image, a coordinate listing comprising a position of
the first vehicle, a position of the second vehicle, and a position
of the third vehicle.
2. The method of claim 1, wherein the planning message and the
imaging message are transmitted according to 5G or 6G
technology.
3. The method of claim 1, wherein the second image further includes
an indication of a direction of travel of the second vehicle.
4. The method of claim 1, further comprising: a. determining, from
the imaging message, a wireless address of the second vehicle; and
b. adding, to the coordinate listing, the wireless address of the
second vehicle and a wireless address of the first vehicle.
5. The method of claim 1, further comprising: a. measuring a
distance from the first vehicle to either the second vehicle or the
third vehicle; and b. determining the coordinate listing according
to the distance.
6. The method of claim 1, further comprising: a. providing,
according to the coordinate listing, a traffic map comprising a
two-dimensional image indicating the position of the first vehicle,
the position of the second vehicle, and the position of the third
vehicle; and b. indicating, on the traffic map, a wireless address
of the first vehicle.
7. The method of claim 1, wherein the imaging message further
indicates at least one of a vehicle type, a color, or a lane
position of the second vehicle.
8. The method of claim 1, wherein the coordinate listing further
indicates at least one of a vehicle type, a color, or a lane
position of the first vehicle.
9. The method of claim 1, further comprising broadcasting the
coordinate listing.
10. The method of claim 1, further comprising: a. determining that
a traffic collision with the second vehicle is imminent; b.
determining, according to the coordinate listing, which wireless
address corresponds to the second vehicle; and c. transmitting, to
the second vehicle, an emergency message.
11. The method of claim 1, wherein the coordinate listing includes
a fourth vehicle which is not depicted in the first image.
12. The method of claim 1, further comprising: a. acquiring a
plurality of images of vehicles in traffic; b. providing the
plurality of images to a computer containing an artificial
intelligence model; and c. determining, according to the artificial
intelligence model, a predicted coordinate listing comprising
predicted positions of the vehicles.
13. The method of claim 12, further comprising: a. acquiring a
further image of further vehicles in traffic; b. receiving at least
one message from at least one proximate vehicle, the at least one
message comprising an additional image of the vehicles in traffic;
c. providing the further image and the additional image as input to
an algorithm based at least in part on the artificial intelligence
model; and d. determining, as output from the algorithm, an updated
coordinate listing comprising predicted positions of the further
vehicles.
14. Non-transitory computer-readable media in a second vehicle, the
second vehicle in traffic, the traffic comprising a first vehicle
and at least one other vehicle, the media containing instructions
that when implemented by a computing environment cause a method to
be performed, the method comprising: a. receiving, from the first
vehicle, a planning message specifying a time; b. acquiring, at the
specified time, an image comprising the first vehicle and the at
least one other vehicle; c. transmitting, to the first vehicle, an
imaging message comprising the image; and d. receiving, from the
first vehicle, a coordinate listing or a traffic map comprising
positions of the first vehicle, the second vehicle, and the at
least one other vehicle.
15. The media of claim 14, the method further comprising: a.
determining, for each of the first, second, and third vehicles, a
vehicle type or a vehicle color; and b. transmitting, to the first
vehicle, a message comprising the determined vehicle types or
vehicle colors.
16. The media of claim 14, the method further comprising
transmitting, to the first vehicle, a wireless address of the
second vehicle.
17. The media of claim 16, wherein: a. the coordinate listing or
the traffic map further indicates, in association with the position
of the second vehicle, the wireless address of the second vehicle;
and b. the coordinate listing or the traffic map further indicates,
in association with the position of the first vehicle, a wireless
address of the first vehicle.
18. A computer containing an artificial intelligence structure
comprising; a. one or more inputs, each input comprising an image
of traffic, the traffic comprising a plurality of vehicles; b. one
or more internal functions, each internal function operably linked
to one or more of the inputs; and c. an output operably linked to
the one or more of the internal functions, the output comprising a
prediction of a two-dimensional position of each vehicle of the
plurality.
19. The computer of claim 18, the artificial intelligence structure
further comprising one or more adjustable variables associated with
the one or more internal functions, the one or more adjustable
variables adjusted by supervised learning according to a plurality
of individually recorded inputs.
20. The computer of claim 18, further comprising an algorithm,
based at least in part on the artificial intelligence structure,
the algorithm configured to take, as input, one or more images of
further vehicles in traffic, and to provide, as output, a
two-dimensional position of each of the further vehicles.
Description
PRIORITY CLAIMS AND RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional
Patent Application Ser. No. 63/260,814, entitled "Localization and
Identification of Vehicles in Traffic by 5G Messaging", filed Sep.
1, 2021, and U.S. Provisional Patent Application Ser. No.
63/243,437, entitled "V2X Messaging in 5G with Simultaneous GPS
Reception", filed Sep. 13, 2021, and U.S. Provisional Patent
Application Ser. No. 63/245,227, entitled "V2X with 5G Image
Exchange and AI-Based Viewpoint Fusion", filed Sep. 17, 2021, and
U.S. Provisional Patent Application Ser. No. 63/246,000, entitled
"V2X Connectivity Matrix with 5G Sidelink", filed Sep. 20, 2021,
and U.S. Provisional Patent Application Ser. No. 63/256,042,
entitled "Hailing Procedure for V2R, V2V and V2X Initial Contact in
5G", filed Oct. 15, 2021, and U.S. Provisional Patent Application
Ser. No. 63/271,335, entitled "Semaphore Messages for Rapid 5G and
6G Network Selection", filed Oct. 25, 2021, and U.S. Provisional
Patent Application Ser. No. 63/272,352, entitled "Sidelink V2V,
V2X, and Low-Complexity IoT Communications in 5G and 6G", filed
Oct. 27, 2021, and U.S. Provisional Patent Application Ser. No.
63/287,428, entitled "V2X and Vehicle Localization by Local Map
Exchange in 5G/6G", filed Dec. 8, 2021, and U.S. Provisional Patent
Application Ser. No. 63/288,237, entitled "V2X with 5G/6G Image
Exchange and AI-Based Viewpoint Fusion", filed Dec. 10, 2021, and
U.S. Provisional Patent Application Ser. No. 63/288,807, entitled
"V2X Messaging in 5G/6G with Simultaneous GPS Reception", filed
Dec. 13, 2021, and U.S. Provisional Patent Application Ser. No.
63/290,731, entitled "Vehicle Connectivity, V2X Communication, and
5G/6G Sidelink Messaging", filed Dec. 17, 2021, all of which are
hereby incorporated by reference in their entireties.
FIELD OF THE INVENTION
[0002] The invention relates to systems and methods for short-range
locating and identification of vehicles in traffic.
BACKGROUND OF THE INVENTION
[0003] Autonomously operated vehicles are expected to cooperate
with each other to avoid traffic hazards and facilitate the flow of
traffic generally. Such cooperation relies on knowing the locations
of other vehicles in proximity and, if wirelessly connected,
identifying their access codes.
[0004] What is needed is means for vehicles in traffic to determine
the locations and, if connected, the wireless addresses of other
proximate vehicles.
[0005] This Background is provided to introduce a brief context for
the Summary and Detailed Description that follow. This Background
is not intended to be an aid in determining the scope of the
claimed subject matter nor be viewed as limiting the claimed
subject matter to implementations that solve any or all of the
disadvantages or problems presented above.
SUMMARY OF THE INVENTION
[0006] In a first aspect, there is a method for a first vehicle to
communicate with a second vehicle, the second vehicle proximate to
a third vehicle, the method comprising: broadcasting a planning
message specifying a particular time; at the particular time,
acquiring a first image depicting the second vehicle and the third
vehicle; receiving, from the second vehicle, an imaging message
comprising a second image, the second image acquired by the second
vehicle at the particular time, the second image depicting the
first vehicle and the third vehicle; and determining, according to
the first image and the second image, a coordinate listing
comprising a position of the first vehicle, a position of the
second vehicle, and a position of the third vehicle.
[0007] In another aspect, there is non-transitory computer-readable
media in a second vehicle, the second vehicle in traffic, the
traffic comprising a first vehicle and at least one other vehicle,
the media containing instructions that when implemented by a
computing environment cause a method to be performed, the method
comprising: receiving, from the first vehicle, a planning message
specifying a time; acquiring, at the specified time, an image
comprising the first vehicle and the at least one other vehicle;
transmitting, to the first vehicle, an imaging message comprising
the image; and receiving, from the first vehicle, a coordinate
listing or a traffic map comprising positions of the first vehicle,
the second vehicle, and the at least one other vehicle.
[0008] In another aspect, there is a computer containing an
artificial intelligence structure comprising; one or more inputs,
each input comprising an image of traffic, the traffic comprising a
plurality of vehicles; one or more internal functions, each
internal function operably linked to one or more of the inputs; and
an output operably linked to the one or more of the internal
functions, the output comprising a prediction of a two-dimensional
position of each vehicle of the plurality.
[0009] This Summary is provided to introduce a selection of
concepts in a simplified form. The concepts are further described
in the Detailed Description section. Elements or steps other than
those described in this Summary are possible, and no element or
step is necessarily required. This Summary is not intended to
identify key features or essential features of the claimed subject
matter, nor is it intended for use as an aid in determining the
scope of the claimed subject matter. The claimed subject matter is
not limited to implementations that solve any or all disadvantages
noted in any part of this disclosure.
[0010] These and other embodiments are described in further detail
with reference to the figures and accompanying detailed description
as provided below.
BRIEF DESCRIPTION OF THE DRAWINGS
[0011] FIG. 1A is a schematic sketch of an exemplary embodiment of
a traffic scenario, according to some embodiments.
[0012] FIG. 1B is a schematic sketch of an exemplary embodiment of
an image transmitted by a vehicle in traffic, according to some
embodiments.
[0013] FIG. 1C is a schematic sketch of an exemplary embodiment of
another image transmitted by a vehicle in traffic, according to
some embodiments.
[0014] FIG. 1D is a schematic sketch of an exemplary embodiment of
a traffic map derived by viewpoint fusion, according to some
embodiments.
[0015] FIG. 2 is a sequence chart showing an exemplary embodiment
of a procedure for determining a traffic map derived by viewpoint
fusion, according to some embodiments.
[0016] FIG. 3 is a flowchart showing an exemplary embodiment of a
procedure for determining a traffic map derived by viewpoint
fusion, according to some embodiments.
[0017] FIG. 4A is a schematic showing an exemplary embodiment of a
planning message, according to some embodiments.
[0018] FIG. 4B is a schematic showing an exemplary embodiment of an
imaging message, according to some embodiments.
[0019] FIG. 4C is a schematic showing an exemplary embodiment of a
mapping message, according to some embodiments.
[0020] FIG. 5A is a schematic showing an exemplary embodiment of an
artificial intelligence structure, according to some
embodiments.
[0021] FIG. 5B is a flowchart showing an exemplary embodiment of a
procedure for developing an AI model, according to some
embodiments.
[0022] FIG. 6A is a schematic showing an exemplary embodiment of
input parameters for an artificial intelligence model, according to
some embodiments.
[0023] FIG. 6B is a flowchart showing an exemplary embodiment of a
procedure for using an AI-derived algorithm, according to some
embodiments.
[0024] Like reference numerals refer to like elements
throughout.
DETAILED DESCRIPTION
[0025] Disclosed herein are procedures enabling autonomous and
semi-autonomous vehicles in traffic to determine the locations of
other vehicles, and of landmarks, by exchange of image data.
Systems and methods disclosed herein (the "systems" and "methods",
also occasionally termed "embodiments" or "arrangements", generally
according to present principles) can provide urgently needed
wireless communication protocols to provide situational awareness
in traffic, localize both wirelessly connected and non-connected
vehicles, and communicate specifically (unicast) with selected
vehicles. With such capabilities, vehicles can reduce traffic
fatalities, facilitate traffic flow, and provide V2V and V2X
communication options appropriate for 5G and 6G technologies,
according to some embodiments.
[0026] Autonomous and semi-autonomous vehicles are potentially able
to provide greatly increased vehicle safety and traffic efficiency
by detecting other vehicles, determining their locations, and
making wireless contact with each connected vehicle. In addition,
vehicles may detect non-vehicle entities such as pedestrians,
obstructions, traffic lights, and signage, along with items related
to communications such as 5G/6G access points. Traffic awareness
and V2X connectivity are essential in detecting hazards, planning
hazard avoidance strategies, and communicating with the other
vehicles to cooperatively react to an imminent collision. By
exchanging messages at electronic speeds, computer-operated
vehicles can coordinate their actions, cooperatively adjust their
speed and direction, and thereby avoid almost all types of highway
accidents, saving countless lives.
[0027] As used herein, a device or entity "knows" something if the
device or entity has the relevant information. An "autonomous"
vehicle is a vehicle operated by a processor, with little or no
human control most of the time. A "semi-autonomous" vehicle is a
vehicle at least partially operated by a processor, or which can be
fully operated by a processor temporarily, such as during emergency
intervention. A wireless message is "unicast" if it is addressed
specifically to a particular recipient, and "broadcast" if it is
transmitted without specifying a recipient. A wireless entity can
transmit "specifically" if the intended recipient is spatially
localized and identified by the transmitting entity, as opposed to
transmitting blindly to a wireless address. "V2V" means
vehicle-to-vehicle messaging. "V2X" means vehicle-to-everything
messaging. A "vehicle" is to be construed broadly, including any
mobile conveyance such as cars, trucks, busses, motorcycles,
scooters, and the like. "Wireless entities" are systems or devices
capable of wireless communication such as connected vehicles,
pedestrians with smart phones, roadside access points or base
stations, and so forth. "Items" or "objects" in images include
things that can appear in images. A "traffic map" is a map showing
vehicles and other items in a two-dimensional distribution
including wireless addresses of the items when known, whereas a
"coordinate listing" is a list of location coordinates of the
items, their wireless addresses if known, and other information,
associated with the items. Items in a traffic map or coordinate
listing may include items viewed or visible by at least one
vehicle. Such items may include vehicles, persons, roadside objects
such as buildings or signs or distinctive vegetation, or distant
objects that can help align images taken from different angles and
positions. An image "includes" or "comprises" an object if the
image depicts the object. "View" and "visible" are to be construed
broadly, including detectable by sensors or instruments, and/or
imaged (or able to be imaged) from a viewpoint, in visible or
infrared light or other imaging medium. "MAC" (medium access
control) and "RNTI" (radio network temporary identification) are
wireless address codes. A "demodulation reference" is a series of
modulated resource elements configured to exhibit levels (such as
amplitude or phase levels) of a modulation scheme, as opposed to
data. "AI" (artificial intelligence) is computer-based
decision-making, as used herein, usually involving a large number
of interacting factors in a complex problem.
[0028] Traffic "situation awareness" includes determining which
vehicles are present, measuring their locations or distances
relative to the measuring entity or other point, and determining
their status. "Status" in this context includes whether a vehicle
is autonomous, semi-autonomous, or human-controlled, wirelessly
connected or not, and the wireless address if known. For detecting
other vehicles, many autonomous vehicles include cameras (visible
and infrared) and sensors (such as lidar, radar, and the like).
However, these imagers and sensors cannot detect a hidden vehicle
such as a car obscured by an intervening truck, among many other
traffic situations. The traffic map (an image showing positions of
vehicles, usually annotated with their wireless addresses), or the
corresponding coordinate listing (a list of coordinates of the
vehicles and wireless addresses), may reveal items that are hidden
to one vehicle but visible to another vehicle.
[0029] "Localization" means determining the locations or positions
or two-dimensional coordinates of vehicles in traffic, in some
coordinate system. If the distances are not known, the coordinates
may have an arbitrary distance scale. A traffic map with an
arbitrary distance scale is still useful to the vehicles by
indicating which of the vehicles has which wireless address. For
example, the traffic map can inform a particular vehicle that
another vehicle directly in front is autonomous and has a certain
wireless address, so that the particular vehicle can then
communicate with the vehicle in front for cooperation in hazard
avoidance. Localization using satellite-based navigation systems
such as GPS generally cannot reliably resolve closely-spaced
vehicles such as vehicles in adjacent lanes, because the motion of
the vehicles can distort the satellite signals. In addition, GPS
depends on correlating signals from multiple satellites, and the
distance traveled by the vehicle between acquisitions from the
various satellites can result in further uncertainty. In addition,
satellite coverage is spotty in many urban environments, steep
canyons, and many other places where traffic situation awareness is
needed. Localization using narrow directed beams of radio waves is
also unlikely to provide the necessary reliability because large
phased-array antennas are needed for degree-level spatial
resolution. In addition, high-frequency beams are notoriously prone
to reflections from nearby conducting objects, such as vehicles in
dense traffic, further confusing the single-vehicle selectivity. In
addition, the beam necessarily continues propagating beyond the
intended recipient and can be received by other entities in line.
Therefore, except for the simplest idealized cases, situation
awareness of dense traffic at freeway speeds remains an unsolved
problem, absent the systems and methods disclosed below.
[0030] As detailed in the examples, vehicles can obtain real-time
traffic awareness by exchanging images from their various
viewpoints, and then combining the images (that is, perform
"viewpoint fusion") to generate a traffic map of the vehicles
and/or a coordinate listing of the two-dimensional coordinates of
the vehicle positions. The vehicles can exchange their wireless
addresses at the same time, so that the traffic map or coordinate
listing can associate each vehicle's wireless address with its
position. A "planning entity" is a vehicle or other wireless entity
that initiates the procedure by broadcasting a "planning message",
requesting that other vehicles take an image, such as a 360-degree
image, of surrounding traffic at a particular time. The
"participating vehicles" acquire their images of traffic from their
own viewpoints at the specified time, and then transmit the image
in an "imaging message" to the planning entity for analysis. The
planning entity then combines or "fuses" the images with its own
image, also acquired at the specified time, and thereby produces a
coordinate listing of the vehicle coordinates. The planning entity
can also make a traffic map depicting the vehicles in relative
positions from each other. The coordinate listing and traffic map
may have an arbitrary distance scale, or they may have a calibrated
distance scale according to a measured distance between, for
example, the planning entity and one of the vehicles. The planning
entity can then broadcast the coordinate listing and/or the traffic
map in a "mapping message". The other vehicles can receive the
mapping message, determine the wireless address of a selected
vehicle, and communicate specifically with that vehicle.
[0031] To determine the overall distance scale of the traffic map
or listing, the planning entity can measure the distance to at
least one vehicle using a sensor, such as radar or lidar, and
calibrate all the other distances to that measurement. Optionally,
the participating vehicles can measure distances to some of the
items in their view, and can include those distances in their
imaging messages. The planning entity may be configured to include,
in the coordinate listing or map, the distance values so
determined. In addition, many coordinate values may be
over-determined by multiple vehicle viewpoints, and the planning
entity may resolve disagreements between corresponding distance
measurements by averaging or least-squares analysis or maximum
likelihood fitting or other analysis calculation suitable for
combining multiple measurements.
[0032] The planning entity can include its wireless address in the
planning message, and the participating vehicles can include their
wireless addresses in their imaging messages. After receiving one
or more imaging messages from the participating vehicles, the
planning entity can perform viewpoint fusion by identifying objects
that appear in two or more of the images The planning entity can
then, by triangulation, determine the two-dimensional coordinates
of each visible item. (However, if the item and the two viewpoints
happen to lie on a straight line, a third non-collinear viewpoint
may be needed to disambiguate the location.) The traffic map or
listing may include the coordinates of each of the participating
vehicles and their wireless addresses, so that the vehicles can
then use the traffic map to determine which wireless address
belongs to which vehicle in their view. The traffic map or listing
can also include non-participating vehicles that appear in the
images. The traffic map or listing can thereby provide the
participating vehicles, and other entities receiving the traffic
map, full traffic situation awareness. In addition, by providing
the wireless addresses of each vehicle if known, the traffic map or
listing can enable the vehicles (and wireless fixed assets) to
specifically communicate with each other, and thereby cooperate to
avoid traffic hazards and improve traffic management.
[0033] The planning and imaging messages may also include
subsidiary information, such as the GPS coordinates of the
transmitting vehicle if known, as well as the transmitting vehicle
type (sedan, bus, truck, etc.), the color, and/or which lane that
the transmitting vehicle is in, among other parameters which may
simplify the viewpoint fusion and resolve ambiguities. If the GPS
data is included, preferably the GPS acquisition time and the speed
of the vehicle are also indicated, so that the planning entity can
correct for the displacement of the vehicle between the time of GPS
acquisition and image acquisition.
[0034] In viewpoint fusion, the planning entity may analyze
multiple images of the same, or nearly the same, vehicles from
different viewpoints. The planning entity can calculate a mutually
consistent location for each of the items in view as well as the
locations of the vehicles acquiring the images. However, each of
those overlapping location determinations is likely to be slightly
different, due to measurement uncertainties and limited resolution
achievable. The planning entity can combine the various
measurements with a fitting function, such as maximum-likelihood,
or least-squares, or mean with outlier-rejection, or other type of
function to determine a best-fit location for each item in the
traffic map or listing. In this way, items that are viewed by
multiple participating vehicles may be located with higher
precision than achievable by each of the observers individually. In
addition, the locations of the vehicles acquiring the images can be
determined as part of the self-consistent spatial distribution
solution.
[0035] The vehicles may transmit their messages according to 5G or
6G sidelink specifications. If an access point or base station is
within range, the communication may be according to "sidelink
mode-1" in which the base station sets the timing and manages the
vehicle messages. If no network interface is available, the
vehicles themselves may set up the timing and bandwidths of a
"sidelink mode-2" local network. Alternatively, the vehicles may
transmit their messages at-will or according to another wireless
technology such as Wi-Fi.
[0036] The planning entity may use artificial intelligence or
machine learning in several steps of this procedure. AI may be
useful for determining the type of each object that is viewed from
different directions. AI may also be useful for determining the
self-consistent distribution of item locations based on the images,
Finally, AI may assist in combining redundant position
determinations by, for example, recognizing and rejecting
outliers.
[0037] Some vehicles in traffic may not be visible to other
vehicles, due to an intervening item such as a truck. However, a
vehicle that is hidden from a first vehicle may be visible to
another vehicle. In that case, the hidden vehicle can be placed on
the traffic map based on the images that include it. Then, upon
receiving the traffic map, the first vehicle may discover the
existence and location of that hidden vehicle. Revealing hidden
vehicles or pedestrians could be life-saving. For example, if the
first vehicle detects an imminent collision, it may ask the truck
to change lanes. But if there is a hidden vehicle beside the truck,
then the truck would refuse to change lanes, and precious time
would be lost while the first vehicle searches for another
mitigation. If, however, the first vehicle knows that the hidden
vehicle is present, the first vehicle can select a different
avoidance strategy without delay.
[0038] Optionally, each participating entity can mark, on its
traffic image, its direction of travel, for example by placing an
icon on the image corresponding to the front of the vehicle or to a
line passing centrally and longitudinally through the vehicle. If
the road is straight, then each of the participating vehicles
generally share the same travel direction, aside from lane changes
and the like, and can use the road direction as a common angle
reference. Alternatively, a vehicle may have a compass such as an
electronic compass that indicates what direction is north, which
can be the angle reference. Each entity can add an icon or other
indication to its image, indicating the travel direction (which is
the same as the road direction unless changing lanes), or north or
other specified geographic direction. Alternatively, a code may be
included with the imaging message indicating which column of pixels
in the image is aligned with the travel direction or other
geographical direction, so that the planning entity can align the
images from multiple participating vehicles accordingly. Although
images with random orientations can be fused by rotating them until
they come into alignment according to the objects appearing in both
images, image fusion is much simpler if the images share a common
orientation.
[0039] In some embodiments, the planning entity may include
software configured to recognize vehicles, or other objects, viewed
from different directions. For example, a pickup truck looks very
different when viewed from the side or front or back. Image
processing software may assign a model shape, such as a generic
three-dimensional pickup truck shape, to such an item in the image,
and may thereby recognize the vehicle from different directions in
the other images.
[0040] In summary, the planning entity may transmit a planning
message to other participating vehicles, specifying a time to
synchronize their image acquisition. The participating vehicles
(and other entities in range) may then acquire their traffic images
at that time, and transmit each image in an imaging message to the
planning entity. The imaging message may also include the
participating vehicle's wireless address code and other identifying
information. The planning entity can then combine the various
images, and optionally its own image of the traffic, and thereby
prepare a traffic map or listing of the positions of the various
vehicles and entities in view. In addition, the traffic map or
listing may include the wireless addresses of the vehicles, if
known. The planning entity can then broadcast a mapping message
that includes the traffic map or coordinate listing. Each of the
participating vehicles, and other entities, can receive the mapping
message, identify its own position in the traffic map or listing
according to its wireless address, and thereby determine which of
the vehicles in its view are associated with which wireless
addresses. The entities can then communicate specifically with the
other vehicles, and thereby cooperate in avoiding hazards and
managing the flow of traffic.
[0041] The following examples illustrate a process for acquiring
and combining traffic images from multiple viewpoints.
[0042] FIG. 1A is a schematic sketch of an exemplary embodiment of
a traffic scenario, according to some embodiments. As depicted in
this non-limiting example, a 4-lane highway 100 includes a first
vehicle 101 rendered as a car, a second vehicle 102 rendered as
another car, a third vehicle 103 rendered as a bus, a fourth
vehicle 104 rendered as a motorcycle, a fifth vehicle 105 rendered
as a pickup truck, and a roadside object 106 rendered as an antenna
of an access point or base station of a network. The first and
second vehicles 101, 102 are shown in bold because they are
participating in a plan to acquire and transmit images of the
surrounding traffic. The other items are present but not
participating.
[0043] In the depicted example, the first vehicle 101 broadcasts a
wireless planning message requesting that other vehicles in range
acquire images, such as 360-degree images, of the traffic in view,
and that the image acquisition be at a particular time, such as 100
milliseconds after the planning message. The planning message may
also request that the images be transmitted back to the first
vehicle 101 using a predetermined format. The first vehicle 101 can
receive the imaging message and perform viewpoint fusion by
correlating and triangulating items visible in both images.
[0044] FIG. 1B is a schematic sketch of an exemplary embodiment of
an image transmitted by a vehicle in traffic, according to some
embodiments. As depicted in this non-limiting example, the image
110 shows various objects from the viewpoint of the first vehicle
101, including the back of the bus 113, the back of the
motorcyclist 114, the back of the pickup truck 115, and the back of
the second vehicle 112 which is partially obscured by the pickup
truck 115 from the first vehicle's viewpoint. The first vehicle 101
may acquire this image 110 from its viewpoint at the specified
time.
[0045] FIG. 1C is a schematic sketch of an exemplary embodiment of
another image transmitted by a vehicle in traffic, according to
some embodiments. As depicted in this non-limiting example, the
image 120 shows various objects from the viewpoint of the second
vehicle 102, including the front of the bus 123, the front of the
motorcyclist 124, the front of the pickup truck 125, and the front
of the first vehicle 121, and the roadside access point 126. The
second vehicle 102 may acquire this image 120 from its viewpoint at
the specified time, and then may broadcast the image 120 (or a
compactified version) to the first vehicle, and the other entities
can receive it as well if they have receivers. As can be seen, the
items in the image 120 are shifted laterally and reversed in order
relative to the image 110 from the first vehicle's viewpoint, due
to the different positions of the first and second vehicles 101,
102. The lateral shifts of items that appear in multiple images
thereby reveal the locations of the items. A coordinate listing of
positions, and a two-dimensional traffic map, may be prepared from
the images 110, 120 using viewpoint fusion, in which each object
that appears in both images can be correlated with a location.
Without a distance calibration, the object locations can be
arranged in the correct order but the overall scale may remain
arbitrary. But if the first vehicle (or another entity) measures
one or more distances to the imaged objects, using radar or lidar
for example, then the scale of the distances between all of the
objects in the traffic map and the coordinate listing can be
adjusted accordingly.
[0046] FIG. 1D is a schematic sketch of an exemplary embodiment of
a traffic map derived by viewpoint fusion, according to some
embodiments. As depicted in this non-limiting example, the traffic
map 130 includes the 4-lane highway 130, an icon or box
representing the location of the first vehicle 131, another icon or
box representing the location of the second vehicle 132, and
further icons or boxes representing the locations of the third
vehicle 133, the fourth vehicle 134, the fifth vehicle 135, and the
roadside object 136. Each of the boxes is in a position, relative
to the others, as determined form the images. The traffic map 130,
or a coordinate listing with the same information in numerical
form, may be broadcast by the first vehicle, so that the second and
other vehicles can determine which vehicles are in proximity and
their wireless addresses if known.
[0047] The boxes for the third and fourth vehicles 133, 134 are
shown dashed to indicate that those vehicles are present but did
not respond to wireless messages and are presumed to not be in
wireless communication. The first and second vehicle boxes 131, 132
are shown bold because they participated by providing images from
which the traffic map 130 was derived. The box representing the
fifth and sixth items 136, 136 are shown solid but not bolded
because they are in radio contact, but did not provide images.
Wireless addresses are provided in three of the boxes 131, 132,
136. The first vehicle 101 included its wireless address in the
planning message, and the second vehicle 102 included its wireless
addresses in the imaging message. The roadside access point 106
provided its wireless address in a separate message. The fifth
vehicle 105 with a blank box 135 is wirelessly connected but not
participating; hence there is not yet enough information to
correlate its wireless address with its physical location. After
receiving the traffic map (or listing), the fifth vehicle 105
noticed that the box 135 representing its location is blank, and
then may transmit a location message indicating that it is the
vehicle represented by the box 135, while providing its wireless
address, so that each of the vehicles may then update their traffic
maps or listings to include that address.
[0048] FIG. 2 is a sequence chart showing an exemplary embodiment
of a procedure for determining a traffic map or coordinate listing,
derived by viewpoint fusion, according to some embodiments. A
sequence chart is a chart showing actions of various entities
versus time, like an oscilloscope trace or logic analyzer display.
In this non-limiting example, actions of three vehicle, vehicle-1,
-2, and -3 and an access point are shown as boxes along horizontal
lines, representing time. Vertical lines demark certain times.
Waiting intervals are shown as double-ended arrows. Options are
shown dashed.
[0049] Initially, vehicle-1 broadcasts a planning message 201
suggesting that all vehicles in range acquire images of traffic at
a particular specified time 202. Three vehicles do so (one image
acquisition is labeled as 203). The vehicles then wait a random LBT
(listen-before-talk) interval 204 (arrow here and elsewhere) before
transmitting their imaging messages. After each LBT interval, the
vehicles determine whether another vehicle is transmitting, and if
so, the ready vehicle waits another random interval after the
current transmission is finished. In this case, vehicle-1 happens
to have selected the shortest random LBT interval 204. Vehicle-1
then (detecting no interfering transmissions) broadcasts its
imaging message 205 as shown.
[0050] Vehicles-2 and -3 expire their waiting intervals and are
then ready to transmit. However, they detect the vehicle-limaging
message 205 when their LBT intervals expire, and therefore they
abort their own transmissions as indicated by slashed boxes (one
labeled as 206). After the vehicle-1 imaging message is done,
vehicles-2 and -3 again delay random LBT intervals. Vehicle-2 wins
this time, and broadcasts its imaging message 207. After message
207 is done, vehicle-3 waits another LBT interval and broadcasts
its imaging message 208. The imaging messages 205, 207, and 208 may
include additional information, such as the wireless address of the
transmitting entity, measured distances between the transmitting
entity and various items in view, a time offset at which the image
was acquired if not at the specified time 202, the velocity and/or
acceleration of the transmitting entity, whether the transmitting
entity is a vehicle or pedestrian or fixed asset or whatever,
whether the transmitting entity is computer-controlled or
human-driver-operated, GPS coordinates, lane occupation, and other
optional information, according to some embodiments. (In another
embodiment, the vehicles may have been assigned different frequency
bands, and may be able to transmit their imaging messages
simultaneously on different frequencies, thereby avoiding message
interference and saving time.)
[0051] In some embodiments, various entities may analyze the
imaging messages, identify items in the images, correlate multiple
views of the items, and thereby perform viewpoint fusion to develop
a traffic map or listing. In the depicted case, the entity that
initially broadcast the planning message 201, which is vehicle-1,
performs the analyses at 209, and then broadcasts the resulting
traffic map or listing 210 indicating the two-dimensional locations
of items seen in the imaging messages. The traffic map or listing
may include item coordinates relative to vehicle-1 or other
coordinate origin. The other vehicles can determine distances from
their own viewpoint by subtraction, shifting the origin from
vehicle-1 to, say, vehicle-2 or vehicle-3. In addition, each item
listed in the traffic map may show, or refer to, additional
information about that item, such as its wireless address, an
indication of the class of item (vehicle, roadside object, etc.),
the type of vehicle (car, semi, bus, pickup, motorcycle, etc.), and
other information about each item in the traffic map or
listing.
[0052] In some embodiments, other entities, besides the initial
planning entity, may process the images and generate their own
traffic map or listing. For example, vehicle-2 may perform the
analyses and transmit its version of the traffic map 211 (in dash)
instead of vehicle-1, or at a separate time, or on a separate
frequency. Alternatively, the access point 106 may receive the
imaging messages 205, 207, and 208, and may prepare its own version
of the traffic map, and broadcast it at 212. Since the computing
power of an access point is generally superior to that of most
vehicles, the access point may be able to prepare the traffic map
sooner than the vehicles, or with better accuracy, or other
advantages. Whether one or more of the vehicles prepares the
traffic map, or the access point or other entity does so, may
depend on factors such as whether an access point is available
within range of the low-power transmissions of the vehicles. In
some embodiments, the planning entity determines the traffic map by
default, but may withhold its map message if another entity, such
as the access point, transmits its map first. Optionally, the
various entities may indicate, in broadcast messages for example,
which entities will prepare traffic maps.
[0053] FIG. 3 is a flowchart showing an exemplary embodiment of a
procedure for determining a traffic map or coordinate listing,
derived by viewpoint fusion, according to some embodiments. As
depicted in this non-limiting example, at 301 a first vehicle (or
other wirelessly connected entity) broadcasts a planning message
specifying a time. At 302, at the specified time, each vehicle in
range acquires an image of the surrounding traffic. The images may
be 360-degree images. The images may include an overlay icon
indicating the travel direction, or north, or the road direction,
or other direction. The images may include an overlay or annotation
indicating the measured distance to objects in the images. At 303,
each participating entity waits a randomly-selected delay and, if
no other transmissions are detected, broadcasts its imaging message
including its wireless address.
[0054] Transmitting large images wirelessly tends to occupy
significant time, and in the envisioned application, multiple
vehicles may be waiting to transmit their image data. Therefore,
the imaging messages may be encoded for speed and compactness. For
example, pixels may be merged since high resolution is usually not
necessary for determining vehicle locations. The image pixels may
be rendered in black-gray-white instead of full color. The images
are expected to provide enough shape and texture information to
enable each object to be identified in multiple viewpoints, and
sufficient spatial information to resolve adjacent vehicles
reliably. As a further compactification, the image may be divided
into sections of different spatial resolution, with higher pixel
resolution in the sections that include vehicles, and low
resolution in sections showing the sky. Further image
compactification may be arranged using methods known to those with
skill in the digital imaging arts.
[0055] At 304, the first vehicle (and optionally other entities)
receives the imaging messages from the participating vehicles. The
first vehicle analyzes the received images along with its own image
acquired at the specified time, and identifies objects that appear
in more than one of the images. At 305, the first vehicle
calculates the locations of those objects in a two-dimensional
coordinate system, thereby determining a listing of coordinates
and, optionally, a traffic map displaying those objects. The
coordinate axes may be geographical (such as latitude and longitude
lines) or local (such as parallel and perpendicular to the
roadway), for example. When the vehicle locations are dependent on
multiple viewpoints with unknown measurement errors, the first
vehicle may perform a fitting routine to determine each vehicle
position using, for example, a least-squares or maximum-likelihood
or other best-fit location for each vehicle. The first vehicle may
select objects to include in the traffic map according to
relevance. For example, the traffic map may include vehicles and
pedestrians but not trees or buildings (although those fixed
objects may be useful in analyzing the various viewpoints). In
addition, AI may be useful in each of those tasks.
[0056] At 306, the first vehicle determines which one, of the
objects in the traffic map or coordinate listing, is associated
with a known wireless address. The first vehicle then specifies the
associated wireless address along with the location of the object
in the map and listing. For example, the first vehicle may
determine an object's wireless address from an imaging message
transmitted by the object, and may include, in the map data
associated with the object, the wireless address of the object. The
mapping message may include a coordinate listing of two-dimensional
coordinates of the objects in the map, relative to a common origin
such as the location of the first vehicle, and using a common
coordinate system such as the direction of the roadway. The first
vehicle may also include, in the mapping message, in a section
providing data about each object in the map, a code indicating the
type of object, such as car, bus, truck, pedestrian, motorcycle,
bicycle, fixed asset, and so forth.
[0057] At 307, the first vehicle broadcasts a mapping message,
including the local traffic map or coordinate listing, to other
proximate vehicles. The mapping message may be a two-dimensional
image such as an overhead view of the local traffic. Alternatively,
and more compactly, the mapping message may be formatted as a list
of coordinates specifying the location of each (relevant) object in
view. Each object may be a vehicle or pedestrian or other entity,
especially wireless or mobile objects, in the images. If the
distance scale is known, the coordinates may be in meters relative
to an origin, such as an origin at the first vehicle or other
relevant point. If the distance scale is not known, the map and
listing may provide relative locations in an arbitrary scale, such
that the vehicles involved can determine which other vehicles are
around them even if the separation distance is not specified. It is
generally sufficient for the first vehicle to measure one
separation distance, using radar or lidar for example, and then to
calibrate all the other distances and coordinates accordingly. In
the coordinate system, one of the axes (say the X axis) may be
parallel to the roadway (at the position of the first vehicle) and
the other coordinate (Y) may be perpendicular to the roadway. Other
configurations of the map data are possible and foreseen.
[0058] At 308, each vehicle, and other wireless entities in range,
can receive the mapping message and determine where the other
vehicles are located. Vehicles that are hidden to some of the
participating vehicles, but are in view of other participating
vehicles, can be revealed in the map or listing. The wireless
addresses of each participating, or otherwise known, entity may be
included in the map or listing, so that the entities can
communicate specifically with each other. For example, a vehicle
detecting an imminent collision with a vehicle in front may
determine, from the map or listing, the wireless address of that
vehicle in front, and may transmit an emergency message
specifically to the other vehicle instructing it to take immediate
evasive action. Absent the procedures disclosed, the vehicles would
not know how to contact a specific other vehicle, and cooperative
collision avoidance would be vastly more difficult.
[0059] The following examples illustrate possible message formats
for the planning, imaging, and mapping messages.
[0060] FIG. 4A is a schematic showing an exemplary embodiment of a
planning message, according to some embodiments. As depicted in
this non-limiting example, a planning message 400 may be
transmitted by a vehicle or other entity, to cause other vehicles
to record images of surrounding traffic and send them back to the
planning entity. The planning message 400 may include a number of
fields (five shown, all optional), each field providing information
about a planned synchronous image acquisition for traffic mapping.
A first field 401 may include a demodulation reference to assist
receivers in demodulating and interpreting the rest of the message.
A second field 402 may indicate the message type, which in this
case is a planning message for image acquisition. A third field 403
may indicate a time at which the synchronous images are to be
acquired. In some embodiments, the planned imaging time may be a
universal time based on an external precision time base such as
GPS. In other embodiments, the planned imaging time may be
specified relative to the start or end of the planning message,
such as 1 or 10 or 100 milliseconds after the start or end of the
planning message. (For this application, it is not necessary to
synchronize the image acquisition times precisely.) A fourth field
404 may contain the wireless address of the planning entity. A
fifth field 405 may include one or more flags (single-bit or
two-bit indicators of various options, for example). The planning
message may be transmitted by a vehicle or other entity that plans
to perform the viewpoint fusion after receiving the imaging
messages.
[0061] FIG. 4B is a schematic showing an exemplary embodiment of an
imaging message, according to some embodiments. As depicted in this
non-limiting example, a vehicle receiving a planning message may
acquire an image of surrounding traffic at the specified time, and
transmit the image back to the planning entity. The imaging message
410 may include fields, all of which are optional. A first field
411 may be a demodulation reference to assist the receiver in
demodulating the rest of the message. A second field 412 may
specify the type as an imaging message. The third field 413 may
specify the wireless address, such as a MAC address or a
pre-assigned RNTI code or other identification code, of the entity
transmitting the imaging message 410. A fourth field 414 may
specify the format of the data to follow, such as specifying
whether high-resolution and low-resolution sections are included,
pixel sizes and distributions, degrees covered, and other data
about the image (this field may not be necessary if the format is
specified in the planning message). A fifth field 415 may include
the image data, which may be compactified for faster transmission,
while still providing sufficiently detailed resolution of the
imaged objects that they can be identified in multiple views and
accurately positioned in the traffic map. A sixth field 416 may
include additional data, such as the measured distances to one or
more objects in the image if known, or extra information about the
transmitting entity such as its vehicle type or color, for example.
Many autonomous and semi-autonomous vehicles include lidar or radar
or sonar or parallax or other types of sensors that measure the
distance to other objects in view, and that distance data may be
included to help the planning entity set the distance scale of the
map, as well as to constrain the object positions in the traffic
map.
[0062] FIG. 4C is a schematic showing an exemplary embodiment of a
mapping message, according to some embodiments. As depicted in this
non-limiting example, a mapping message 420 may include four
fields, all optional. The first field 421 may be a demodulation
reference, the second field 422 may indicate that the message type
is mapping, the third field 423 may specify the format of the data
to follow, and the fourth field 424 may include a listing of the
objects visible in the images, including the two-dimensional
position of each object relative to an origin such as the planning
entity position, along with the wireless address of each object (if
known) and its type of object (car, truck, bus, pedestrian,
motorcycle, etc.).
[0063] Vehicles receiving the mapping message may determine which
of the vehicles in surrounding traffic have which wireless
addresses, and may therefore contact one of the other vehicles
selectively and communicate specifically with the selected vehicle.
Absent this ability, cooperation for collision avoidance and
traffic management would be far more difficult.
[0064] The systems and methods further include artificial
intelligence (AI) models with machine learning (ML), specifically
to assist in the viewpoint fusion process, as detailed in the
following examples.
[0065] An embodiment of AI for viewpoint fusion pertains to object
recognition in images. A pickup truck, for example, looks very
different when viewed from the front, side, and back. Other items,
such as two white cars of a particular make and model, may look
quite similar. Distinguishing similar-looking items while
recognizing an item viewed from different directions, is a complex
problem. However, it is generally not possible to do viewpoint
fusion until specific items are identified in at least two of the
images (or, if present in only one image, the distance may be
measured and provided along with the image). Therefore, the systems
and methods include an AI model configured to take, as input,
multiple images of a set of vehicles viewed from various
directions, and to produce, as output, an indication of which items
in the images correspond to each other. For example, the AI model
may include a database of vehicle prototypes, each vehicle
prototype including a three-dimensional shape with characteristic
features such as windows of a particular shape, bumpers of a
particular design, trim, lights, etc. By comparing the imaged items
to the vehicle prototypes, as the prototypes would appear from
various angles, the AI model may determine whether each item in
each image corresponds to the same prototype, and therefore to the
same physical object.
[0066] In addition, the AI model may be configured to distinguish
two similar-looking items according to subtle differences in
detail, such as a mud splash appearing on one of the images but not
the other, a difference in the number of occupants, or feature
differences such as the hubcap design or a bumper sticker or the
license plate code (if visible), for example. To do so, the AI
model may be configured to start with the closest relevant
prototype, then populate the prototype with features visible in one
of the images, and determine whether those features are present in
the other images. If so, they are probably all the same physical
object. If they differ in even one discrete feature that should be
visible based on the viewpoints, the images probably depict two
different, but similar, objects.
[0067] The AI model may be based on an AI structure such as a
neural net, in a computer such as a supercomputer. The AI structure
may have multiple inputs such as the images acquired at multiple
viewpoints, and multiple outputs such as a coordinate listing
indicating locations of each item appearing in more than one image,
and a number of internal functions that include adjustable
variables. The internal functions may take input data, such as the
images, and may process the images by digitization, feature
parsing, statistical correlation, and other image processing
techniques to extract features. The internal functions may pass
processed data to other internal functions, which may pass further
processed data to the output. The output may include a listing of
image items identified as vehicles or other items, and their
positions.
[0068] The output coordinate listing may then be compared to the
actual distribution of vehicles, as determined by an overhead
photograph or by human analysts for example, thereby determining
the accuracy of the AI prediction. The adjustable variables may
then be varied, singly or in groups, to improve the accuracy of the
image-object assignments and location determinations. For example,
one variable may be a weighting factor or threshold, determining
how similar (or different) two images can be while depicting the
same object. Such a weighting factor may play an important role in
separating two different similarly-shaped objects, versus a single
object viewed from two different directions. Many other variables
in the internal functions may have more obscure meanings or none at
all. Nevertheless, if the AI structure has a sufficient number of
layers and adjustable variables, when "trained" using a
sufficiently large number of traffic image examples, the AI model
is (usually) able to discern the vehicles, and identify them among
the various images, and predict their relative positions, with
satisfactory accuracy for most applications.
[0069] Due to the potentially large number of inputs and adjustable
variables in the model, and the very large amount of training data
likely needed for convergence of the model, the AI structure is
preferably prepared in a supercomputer. The supercomputer may be a
classical semiconductor-based computer, with sufficient speed and
thread count and processor count to perform the model training in a
feasible amount of time. Alternatively, the supercomputer may be a
quantum computer having "qbits" or quantum bits as its working
elements. Quantum computers may provide especial advantages to
solving AI models because they can very rapidly explore a complex
terrain of values, such as the highly interrelated effects of the
various inputs on the output results. Therefore, the systems and
methods include a quantum computer programmed to include an AI
structure and trained on images acquired by each of several
vehicles in traffic, and to prepare a comprehensive map of the
vehicles, and optionally fixed items, detected by the vehicles.
[0070] FIG. 5A is a schematic showing an exemplary embodiment of an
artificial intelligence structure, according to some embodiments.
As depicted in this non-limiting example, an AI structure 500, such
as a neural net or other type of AI structure, may include inputs
501 depicted as squares, internal functions 503, 505 depicted as
circles, and an output 507 depicted as a triangle. Directed links
502, 504 convey data from the inputs 501 to a first layer of
internal functions 503 and to a second layer of internal functions
505 and to the output 507. In the context of viewpoint fusion of
traffic images, the inputs 501 may include the images 508 acquired
simultaneously by multiple vehicles in traffic, GPS data if known
of the various vehicles 509, and additional factors 510 such as the
make and model and color of the vehicles, what lane they are in,
and other features as reported by the participating vehicles.
[0071] The internal functions 503, 505 perform mathematical and
logical operations on the data provided to each internal function
by the links 502, 504 leading to each internal function, such as
weighted averaging, nonlinear operations such as range compression,
logical operation such as selecting one of the links based on the
data from another link, among many other possible operations. The
internal functions include adjustable variables that can be
adjusted or "trained" to solve a particular problem, such as image
viewpoint fusion to generate a two-dimensional map. The links 502,
504 may also include operations such as weighting, in some
embodiments. Some embodiments include feedback and more complex
topologies, as indicated by the dashed arrow 514. The results of
the last layer 505 of internal functions is then conveyed 506 to
the output 507, which may calculate an average, or a maximum, or
other combination of the processed data provided to it by the last
set of links 506. Although the diagram shows links connecting just
a few of the functions in each layer, in an actual AI structure
each function of one layer may link to all of the functions of the
next layer. Only two layers are shown, but larger numbers of layers
of internal functions are generally required to obtain satisfactory
predictions.
[0072] The output 507 is a traffic map or coordinate listing. Also
shown is an extra input 511 which is the "ground truth", the actual
distribution of vehicles in traffic, as determined by an overhead
camera or other means for determining the two-dimensional
distribution of the vehicles independently of the input images.
This extra input 511 is not provided to the structure 500, but
rather is used as a training tool by comparing 513 the actual
coordinate listing 512 with the output 507 of predicted locations,
and thereby determining an accuracy of the output 507.
[0073] The AI structure 500 can be "tuned" or "trained", thereby
forming an AI model of the traffic imaging application, by
adjusting the internal variables until the output coordinate
listing is sufficiently accurate. For example, the variables can be
adjusted singly or in groups, in various directions and by various
amounts, to determine whether any of those variations results in
better agreement between the output 507 and the actual traffic
distribution 512. If so, the variables can be adjusted further in
the same manner, and if not, the variables can be reversed or
adjusted in some other way. In millions or billions (or trillions)
of iterative variations, the predictive accuracy may incrementally
improve, and the computer may retain the best-performing variable
set so far obtained. The computer may start multiple separate
explorations of different configurations of the variables,
following each separately to search for the best prediction
accuracy. By repeating this process for many traffic scenarios, if
the AI structure 500 is sufficiently broad and versatile, the
output 507 may successively approach the actual distributions 512,
in most situations. The AI model is then "trained".
[0074] Typically AI structures are large and unwieldy, impractical
for field use by traveling vehicles. Therefore, a fieldable
algorithm may be developed from the AI model. Typically, some or
many of the links and internal functions may have little effect on
the output. Therefore, a simpler and more compact version of the AI
model may be developed by "pruning" or deleting the unproductive
links, inputs, and internal functions, and freezing the variables
at the best values, thereby producing an AI-based algorithm that
can be downloaded to vehicles for traffic situation awareness in
the field. Alternatively, the algorithm may be a different type of
calculation tool, such as a computer program, an interpolatable
tabulation of values, a graphical analysis tool, or other means for
determining the coordinate listing from the images and the other
input data.
[0075] FIG. 5B is a flowchart showing an exemplary embodiment of a
procedure for developing an AI model, according to some
embodiments. As depicted in this non-limiting example, at 551 an AI
structure is developed or acquired, such as a software package in a
supercomputer, capable of being applied to the viewpoint fusion
problem. specifically, the AI structure may be configured to
compare images, perform geometric operations, recognize objects,
predict location coordinates of the objects, and compare the
predictions to an independently provided coordinate listing for
training.
[0076] At 552, a large number of traffic scenarios are imaged from
multiple viewpoints, and the images and other data are fed as
inputs to the AI structure. Variables in the AI structure may then
be tuned or adjusted at 553 to obtain sufficiently accurate
predictions of the traffic distribution under a wide range of
conditions. Then at 554, when the AI model is sufficiently reliable
for field applications, an algorithm can be derived from it and
downloaded to vehicles, such as autonomous or semi-autonomous
vehicles. The vehicles can then acquire and share images in
traffic, apply those images to the algorithm, obtain a traffic map
or coordinate listing from the algorithm, and thereby gain traffic
situation awareness. The vehicles can then cooperate in avoiding
collisions and regulating the flow of traffic. Optionally, the
vehicles may record their image data and the predictions for
further refinement of the model.
[0077] FIG. 6A is a schematic showing an exemplary embodiment of
input parameters for an artificial intelligence model, according to
some embodiments. As depicted in this non-limiting example, the
input parameters 600 may include images 601 acquired simultaneously
by multiple vehicles in traffic, each image covering a large angle
such as 360 degrees around the vehicle. The inputs may also include
GPS data 602 such as the approximate latitude and longitude of each
vehicle if known, along with the speed of the vehicle and the time
between the image acquisition and the GPS acquisition. Further
inputs may include descriptions 603 of the surrounding vehicles and
of the vehicle taking the image, such as the type of vehicle
(sedan, SUV, limo, delivery van, etc), color, special features such
as visible damage or cargo, which lane of a multilane road the
vehicle is in, for example. The inputs 601, 602, 603 are then
processed by the AI model to produce the predicted traffic map or,
more compactly, the coordinate listing 605 indicating the positions
of the vehicles relative to one of them. The positions may have an
arbitrary length scale, giving only the relative orientations of
each vehicle from each other vehicle in the images. Alternatively,
the length scale may be determined from one or more distance
measurements between two of the objects using radar or lidar, for
example, in which case the positions can be specified in meters or
other units. The coordinate listing 605 may also include the
wireless addresses 604 of the participating vehicles, so that the
vehicles can contact each other specifically, and thereby cooperate
in avoiding hazards.
[0078] FIG. 6B is a flowchart showing an exemplary embodiment of a
procedure for using an AI-derived algorithm, according to some
embodiments. As depicted in this non-limiting example, at 651 a
first vehicle transmits a planning message to the other vehicles on
a common channel, the planning message configured to request that
the vehicles acquire images of traffic at a particular time, such
as one second after the planning message. The planning message may
also specify a format for the vehicles to transmit their images to
the first vehicle. The planning message may also include a wireless
address of the first vehicle.
[0079] At 652, the first vehicle and the other vehicles acquire
their images, each image showing various other vehicles and perhaps
fixed items in proximity. The vehicles then transmit imaging
messages to the first vehicle, including the images and the
wireless addresses of the transmitting vehicles. At 653, the first
vehicle performs viewpoint fusion by identifying objects that
appear in multiple images, calculating the position of those
objects by triangulation, and thereby combining the image data into
a traffic map or coordinate listing of position coordinates. In
addition, the traffic map or coordinate listing may include the
wireless addresses of the vehicles that have supplied them. Further
description such as vehicle type may also be provided. The first
vehicle then broadcasts the traffic map as coordinate listing of
the vehicles in proximate traffic, and optionally other entitles
such as roadside wireless entities, in a mapping message. In
addition, the first vehicle, or another entity, may generate a
traffic map as an image, optionally annotated with the wireless
addresses of the entities.
[0080] At 654, the vehicles have received the position data and
wireless addresses of the other vehicles, as well as position data
for the non-communicating vehicles. The vehicles can then
communicate specifically with each other, to avoid accidents and
manage the flow of traffic.
[0081] The wireless embodiments of this disclosure may be aptly
suited for cloud backup protection, according to some embodiments.
Furthermore, the cloud backup can be provided cyber-security, such
as blockchain, to lock or protect data, thereby preventing
malevolent actors from making changes. The cyber-security may
thereby avoid changes that, in some applications, could result in
hazards including lethal hazards, such as in applications related
to traffic safety, electric grid management, law enforcement, or
national security.
[0082] In some embodiments, non-transitory computer-readable media
may include instructions that, when executed by a computing
environment, cause a method to be performed, the method according
to the principles disclosed herein. In some embodiments, the
instructions (such as software or firmware) may be upgradable or
updatable, to provide additional capabilities and/or to fix errors
and/or to remove security vulnerabilities, among many other reasons
for updating software. In some embodiments, the updates may be
provided monthly, quarterly, annually, every 2 or 3 or 4 years, or
upon other interval, or at the convenience of the owner, for
example. In some embodiments, the updates (especially updates
providing added capabilities) may be provided on a fee basis. The
intent of the updates may be to cause the updated software to
perform better than previously, and to thereby provide additional
user satisfaction.
[0083] The system and method may be fully implemented in any number
of computing devices. Typically, instructions are laid out on
computer readable media, generally non-transitory, and these
instructions are sufficient to allow a processor in the computing
device to implement the method of the invention. The computer
readable medium may be a hard drive or solid state storage having
instructions that, when run, or sooner, are loaded into random
access memory. Inputs to the application, e.g., from the plurality
of users or from any one user, may be by any number of appropriate
computer input devices. For example, users may employ vehicular
controls, as well as a keyboard, mouse, touchscreen, joystick,
trackpad, other pointing device, or any other such computer input
device to input data relevant to the calculations. Data may also be
input by way of one or more sensors on the vehicle, an inserted
memory chip, hard drive, flash drives, flash memory, optical media,
magnetic media, or any other type of file-storing medium. The
outputs may be delivered to a user by way of signals transmitted to
vehicle steering and throttle controls, a video graphics card or
integrated graphics chipset coupled to a display that maybe seen by
a user. Given this teaching, any number of other tangible outputs
will also be understood to be contemplated by the invention. For
example, outputs may be stored on a memory chip, hard drive, flash
drives, flash memory, optical media, magnetic media, or any other
type of output. It should also be noted that the invention may be
implemented on any number of different types of computing devices,
e.g., embedded systems and processors, personal computers, laptop
computers, notebook computers, net book computers, handheld
computers, personal digital assistants, mobile phones, smart
phones, tablet computers, and also on devices specifically designed
for these purpose. In one implementation, a user of a smart phone
or WiFi-connected device downloads a copy of the application to
their device from a server using a wireless Internet connection. An
appropriate authentication procedure and secure transaction process
may provide for payment to be made to the seller. The application
may download over the mobile connection, or over the WiFi or other
wireless network connection. The application may then be run by the
user. Such a networked system may provide a suitable computing
environment for an implementation in which a plurality of users
provide separate inputs to the system and method. In the below
system where vehicle controls are contemplated, the plural inputs
may allow plural users to input relevant data at the same time.
[0084] It is to be understood that the foregoing description is not
a definition of the invention but is a description of one or more
preferred exemplary embodiments of the invention. The invention is
not limited to the particular embodiments(s) disclosed herein, but
rather is defined solely by the claims below. Furthermore, the
statements contained in the foregoing description relate to
particular embodiments and are not to be construed as limitations
on the scope of the invention or on the definition of terms used in
the claims, except where a term or phrase is expressly defined
above. Various other embodiments and various changes and
modifications to the disclosed embodiment(s) will become apparent
to those skilled in the art. For example, the specific combination
and order of steps is just one possibility, as the present method
may include a combination of steps that has fewer, greater, or
different steps than that shown here. All such other embodiments,
changes, and modifications are intended to come within the scope of
the appended claims.
[0085] As used in this specification and claims, the terms "for
example", "e.g.", "for instance", "such as", and "like" and the
terms "comprising", "having", "including", and their other verb
forms, when used in conjunction with a listing of one or more
components or other items, are each to be construed as open-ended,
meaning that the listing is not to be considered as excluding other
additional components or items. Other terms are to be construed
using their broadest reasonable meaning unless they are used in a
context that requires a different interpretation.
* * * * *