U.S. patent application number 14/261182 was filed with the patent office on 2014-09-11 for method and device for controlling and monitoring the surrounding areas of an unmanned aerial vehicle (uav).
This patent application is currently assigned to Prox Dynamics AS. The applicant listed for this patent is Prox Dynamics AS. Invention is credited to Petter Muren, Dag Henning Paulsen.
Application Number | 20140257596 14/261182 |
Document ID | / |
Family ID | 47722224 |
Filed Date | 2014-09-11 |
United States Patent
Application |
20140257596 |
Kind Code |
A1 |
Paulsen; Dag Henning ; et
al. |
September 11, 2014 |
METHOD AND DEVICE FOR CONTROLLING AND MONITORING THE SURROUNDING
AREAS OF AN UNMANNED AERIAL VEHICLE (UAV)
Abstract
The embodiments herein disclose a method and a remote control
for controlling and monitoring surrounding areas of an Unmanned
Aerial Vehicle (UAV) by an operator with a remote control
comprising a flight display. The embodiments disclose a surprising
advantageously effect for the purposes of controlling and
monitoring the surroundings of a UAV by combining the image
captured by a UAV camera with a transparently overlaid positional
and navigation map providing a perception enabling the operator to
have a complete overall view of the situation, utilizing a common
screen section of a flight display and thereby not having to shift
eye view. Viewing images and positional and navigation information
in the way described has shown not to interfere with each other,
rather on the contrary.
Inventors: |
Paulsen; Dag Henning;
(Voyenenga, NO) ; Muren; Petter; (Slependen,
NO) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Prox Dynamics AS |
Hvalstad |
|
NO |
|
|
Assignee: |
Prox Dynamics AS
Hvalstad
NO
|
Family ID: |
47722224 |
Appl. No.: |
14/261182 |
Filed: |
April 24, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13749918 |
Jan 25, 2013 |
8744647 |
|
|
14261182 |
|
|
|
|
Current U.S.
Class: |
701/2 |
Current CPC
Class: |
B64C 19/00 20130101;
G05D 1/0038 20130101; G05D 1/0044 20130101 |
Class at
Publication: |
701/2 |
International
Class: |
B64C 19/00 20060101
B64C019/00 |
Foreign Application Data
Date |
Code |
Application Number |
Mar 22, 2012 |
NO |
20120341 |
Claims
1. A method for controlling, and monitoring surrounding areas of,
an Unmanned Aerial Vehicle (UAV) by an Operator with a remote
control, comprising: providing a flight display, and a UAV equipped
with a camera and a positioning device providing an image of
surrounding areas and positioning data of an operating field of the
UAV, wirelessly transmitted to a remote control; displaying the
image captured by the camera on the flight display; providing a
transparent layer upon the image comprising a cardinal-oriented map
of a range of the operating field including at least a cardinal
point, a UAV symbol oriented on the flight display relative to the
cardinal point corresponding to a real compass direction of the UAV
provided by the positioning device; and a waypoint symbol
positioned relative to the UAV symbol corresponding in distance and
direction to a real geographical relation between the UAV and the
waypoint.
2. The method according to claim 1, wherein the cardinal-oriented
map further comprises an Operator position symbol positioned
relative to the UAV symbol and the waypoint symbol corresponding in
distance and direction to a real geographical relation between the
Operator and the UAV and the waypoint, respectively.
3. The method according to claim 1, further comprising displaying
one or more rings indicating a scale of the cardinal-oriented
map.
4. The method according to claim 3, wherein the one or more rings
are centered around the Operator position symbol which coincides
with a center of the flight display.
5. The method according to claim 3, wherein the one or more rings
are marked with a radius stating the scale of the cardinal-oriented
map.
6. The method according to claim 4, further comprising when the UAV
passes a predefined distance from the Operator position, then
switching the scale of the cardinal-oriented map, and changing the
one or more rings in terms of size and scale indication.
7. The method according to claim 3, further comprising when the UAV
passes a predefined distance to the waypoint position, then
switching the scale of the cardinal-oriented map and coinciding a
center of the one or more rings with the waypoint symbol.
8. The method according to claim 1, wherein the image is a video
image or one or more snapshots.
9. A remote control adjusted to control, and monitor surrounding
areas of, an Unmanned Aerial Vehicle (UAV) by an Operator, the
remote control comprising: a flight display in communication with
the remote control, the remote control being in communication with
an UAV that is equipped with a camera and a positioning device
providing an image, live or still a series of snapshots, of the
surrounding areas and positioning data of the operating field of
the UAV, wirelessly transmitted to the remote control, the flight
display being adjusted to display the image captured by the camera
on the flight display; and provide a transparent layer upon the
image comprising a cardinal-oriented map of a range of the
operating field including at least a cardinal point, a UAV symbol
oriented on the flight display relative to the cardinal point
corresponding to a real compass direction of the UAV provided by
the positioning device, and a waypoint symbol positioned relative
to the UAV symbol corresponding in distance and direction to a real
geographical relation between the UAV and the waypoint.
Description
CROSS-REFERENCE TO RELATED APPLICATION
[0001] This application is a continuation to of U.S. patent
application Ser. No. 13/749918, filed Jan. 25, 2013, entitled
METHOD AND DEVICE FOR CONTROLLING AND MONITORING THE SURROUNDING
AREAS OF AN UNMANNED AERIAL VEHICLE (UAV) that claims priority to
Norwegian Patent No. 20120341 filed Mar. 22, 2012 entitled METHOD
AND DEVICE FOR CONTROLLING AND MONITORING THE SURROUNDING AREAS OF
AN UNMANNED AERIAL VEHICLE (UAV) the entirety of both of which are
incorporated herein by reference.
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT
[0002] n/a
FIELD OF THE INVENTION
[0003] The embodiments herein relate to controlling and monitoring
the surrounding areas of an Unmanned Aerial Vehicle (UAV).
BACKGROUND OF THE INVENTION
[0004] Modern warfare and law enforcement are characterized by an
increasing need for up-to-date situational awareness. To track
down, or to protect against, criminals, paramilitary forces or
terrorists, law enforcement personnel and soldiers often have an
immediate need for information about what is around the next corner
or over the hill.
[0005] Hostile forces frequently hide themselves from view or
exploit the local terrain to gain tactical advantage or escape from
pursuers. In the presence of hostile forces, a simple brick wall,
barbed wire fence, a body of water, buildings or even a large open
area can be an insurmountable obstacle when time is of the essence
and tactical resources are unavailable. An active or undetected
threat can make the situation dangerous.
[0006] Visible indications, noises or predictable actions can
reveal friendly forces and put them at risk. Stealth and surprise,
however, are important elements that can give a tactical advantage.
An UAV is an aircraft with no pilot on board (Also referred to
herein as NUAV, where N is an abbreviation for Nano). UAVs can be
remotely controlled (e.g. flown by a pilot/operator at a remote
ground control station using a controller) or it can fly
autonomously based on pre-programmed flight plans or more complex
dynamic automation systems. An UAV may also be referred to as a
drone. UAVs equipped with video cameras and positioning devices
transmit live pictures and positioning information to the operator
of the UAV and allow their operator to perform surveillance tasks
and gather information from a safe position without exposing
themselves.
[0007] An UAV may be operated and controlled using a remote
control. Traditionally an UAV is operated by controlling such as
e.g. a joystick. The joystick is configured to communicate with the
UAV, i.e. to send control signal to the UAV using any suitable
communication technology. Most joysticks are two-dimensional,
having two axes of movement (similar to a mouse), but one and
three-dimensional joysticks do exist. A joystick is generally
configured so that moving the stick left or right signals movement
of the UAV along the X axis, and moving it forward (up) or back
(down) signals movement of the UAV along the Y axis. In joysticks
that are configured for three-dimensional movement, twisting the
stick left (counter-clockwise) or right (clockwise) signals
movement of the UAV along the Z axis. In some types of remote
controls one or more of these stick movements can be replaced by
moving the remote control itself. These three axes--X, Y, and
Z--are, in relation to an aircraft, roll, pitch, and yaw.
[0008] In addition to a joystick, the remote control configured to
operate and control the UAV is usually also supplemented with a
Graphical User Interface (GUI). A GUI may be a flight display
providing e.g. different dynamic navigation information like speed,
direction, waypoints etc. If the UAV is equipped with a camera
streaming video or snapshots to the UAV base, these videos or
snapshots should also be displayed through the GUI. Traditionally,
video and snapshots are displayed on a screen separate from the
navigation display, or in a separate part of the GUI display.
UAV-kits adjusted for operations in the field have only limited
space, since it should comprise both joystick, GUI and power supply
as well as the UAVs themselves, preferably compactly stacked so
that the personnel easily can carry it in the field during
operation. It is therefore a need for a space efficient way of
displaying both navigation information and images captured by the
UAV without compromising usability and perception.
SUMMARY OF THE INVENTION
[0009] The embodiments herein disclose a a remote control
comprising a flight display and a method for controlling and
monitoring surrounding areas of an UAV with the remote control.
[0010] In one aspect of the embodiments herein, a method is
disclosed for controlling, and monitoring surrounding areas of, an
Unmanned Aerial Vehicle (UAV) by an Operator with a remote control
comprising a flight display, wherein the UAV is equipped with a
camera and a positioning device providing an image (live, still or
as a series of snapshots) of the surrounding areas and positioning
data of the operating field of the UAV, wirelessly transmitted to
the remote control, comprising the steps of displaying the image
captured by the camera on the flight display, and providing a
transparent layer upon the image comprising a cardinal-oriented,
preferably north-oriented map of a range of the operating field
including at least a cardinal point, preferably north point symbol
(north used in this description could be replaced by any cardinal
or intercardinal reference), a UAV symbol oriented on the flight
display relative to the north point symbol corresponding to the
real compass direction of the UAV provided by the positioning
device, and a waypoint symbol positioned relative to the UAV symbol
corresponding in distance and direction to the real geographical
relation between the UAV and the waypoint.
[0011] In another aspect of the embodiments herein, a remote
control is disclosed adjusted to control, and monitor surrounding
areas of, an Unmanned Aerial Vehicle (UAV) by an operator
comprising a flight display, wherein the UAV is equipped with a
camera and a positioning device providing an image of the
surrounding areas and positioning data of the operating field of
the UAV, wirelessly transmitted to the remote control,
characterized in that the flight display is adjusted to displaying
the image captured by the camera on the flight display, and
providing a transparent layer upon the image comprising a
north-oriented map of a range of the operating field including at
least a north point symbol, a UAV symbol oriented on the flight
display relative to the north point symbol corresponding to the
real compass direction of the UAV provided by the positioning
device, and a waypoint symbol positioned relative to the UAV symbol
corresponding in distance and direction to the real geographical
relation between the UAV and the waypoint.
[0012] Embodiments herein afford many advantages, of which a
non-exhaustive list of examples follows:
[0013] An advantage of the embodiments herein is that they provide
a space efficient remote control for controlling and monitoring
surroundings of an UAV.
[0014] Another advantage of the embodiments herein is that they do
not compromise usability of the remote control and perception of
the information provided by the remote control.
[0015] Furthermore, the embodiments herein provide the advantage of
enabling the operator of the remote control to have a complete
overall view of the situation, utilizing a common screen section of
a flight display and thereby not having to shift eye view. Viewing
images and positional and navigation information does not interfere
with each other, rather on the contrary.
[0016] The embodiments herein are not limited to the features and
advantages mentioned above. A person skilled in the art will
recognize additional features and advantages upon reading the
following detailed description.
BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The following detailed description of the embodiments herein
is accompanied by drawings in order to make it more readily
understandable. In the drawings:
[0018] FIG. 1 shows an example of a flight display in a UAV remote
control with navigation and positional information transparently
overlaid upon an image captured by an UAV camera;
[0019] FIG. 2 shows an example of an extract of the elements of the
navigation and positional information in overlay mode;
[0020] FIGS. 3 and 4 shows examples of different range rings of the
navigation and positional information;
[0021] FIG. 5 shows an example of an extract of the elements of the
navigation and positional information in expand mode;
[0022] FIG. 6 shows an example of an extract of the elements of the
navigation and positional information in basic mode; and
[0023] FIG. 7 is a flow chart illustrating an example of the method
according to the present invention.
[0024] The drawings are not necessarily to scale and the dimensions
of certain features may have been exaggerated for the sake of
clarity. Symbols, video and pictures will usually be in color.
Emphasis is placed upon illustrating the principle of the
embodiments herein.
DETAILED DESCRIPTION OF THE INVENTION
[0025] In the following, the embodiments herein will be discussed
and example embodiments described by referring to the accompanying
drawings.
[0026] The embodiments herein provides a surprising advantageously
aspect of the gestalt effect for the purposes of controlling and
monitoring the surroundings of a UAV by combining the image
captured by a UAV camera with a transparently overlaid positional
and navigation map providing a perception enabling the operator to
have a complete overall view of the situation, utilizing a common
screen section of a flight display and thereby not having to shift
eye view. Viewing images and positional and navigation information
in the way described in the following has shown not interfere with
each other, rather on the contrary. The gestalt effect is the
form-generating capability of our senses, particularly with respect
to the visual recognition of figures and whole forms instead of
just a collection of simple lines and curves.
[0027] The UAV is operated and controlled using a remote control.
The UAV and the remote control transmit and receive signals to/from
each other, using any suitable wireless communication technology.
Movements of the remote control made by e.g. an operator or
activation of buttons on the remote control correspond to movements
of the UAV. The UAV comprises a camera for providing images of the
surrounding areas in which the UAV is located. The images taken by
the camera is transmitted to and displayed on a flight display. In
some embodiments, the flight display is comprised in the remote
control, e.g. on the front side of the remote control, or it may be
provided as a separate unit connected to the remote control. The
UAV further comprises positioning devices configured to obtain
information related to the position of the UAV in operation and to
provide the positional and navigational information to the remote
control.
[0028] FIG. 1 shows an example of a flight display. The illustrated
flight display provides an operator with sensor information (i.e.
motion imagery and snapshots), system information, and graphical
cues designed to enhance the operator's situational and positional
awareness. As shown, there are three distinct areas of the flight
display which is used to display relevant information during
flight: [0029] Centre screen--Imagery (live video and Snapshot),
and position awareness cues. [0030] Right bar--UAV status and
information. [0031] Left bar--Base station and Operator status and
NUAV/Payload commands.
[0032] The centre screen symbols and display modes are in the
following described in further details.
[0033] The centre screen display combines imagery and different
levels of positional and navigation information, transparently
overlaid on the imaginary.
[0034] The center screen can be switched either manually by
activation of e.g. a button on the remote control or automatically
between the following modes: [0035] Overlay mode [0036] Expand mode
[0037] Basic mode
[0038] The imagery may comprise either real time motion (video)
imagery or a snapshot depending on selected payload mode. The image
is captured by the camera mounted on the UAV, and wirelessly
streamed to the remote control equipment on a Digital Data Link
(DDL).
[0039] The presentation of the positional and navigation
information is adjusted to a typical scenario where the UAV starts
from the Operator's position heading towards a target point or a
waypoint (WP). The information comprises a simplified
north-oriented map comprising minimal information to allow the
operator to maintain relative positional awareness of its own
position, UAV position and position of selected objects. In an
example embodiment, there are three modes available to the operator
through the flight menu as described below. The imagery information
in the figures has been removed for clarity in all examples.
[0040] The Overlay mode is the default mode, and provides the
operator with the highest level of information. FIG. 2 shows the
elements of the Overlay mode, and table 1 describes each of
them.
TABLE-US-00001 TABLE 1 Element Description Operator (user) The
Operator (i.e. normally the user) position is in the position
centre of the screen, even when the user relocates. UAV position
UAV position in relation to the user. In the and heading example,
the UAV is about 50 meters NNE of the user. The direction of the
UAV indicates the NUAV heading (in this case the heading is NNE).
Range ring The range ring provides the scale of the map overlay.
The overlay will scale automatically in order to keep all
geographical reference objects (i.e. the Operator, the UAV, and the
selected WP/RP) within the display area. Time of day Time of day in
UTC. Current UAV MGRS (Military Grid Reference System) of position
current UAV position. Continuously updated by downlinked
positioning device information, such as GPS information, from the
UAV. Waypoint Position of selected WP. The user might select a WP
position from the Flight menu in order for it to be displayed. A
line connecting the UAV and the WP is displayed to ease navigation
when flying towards the WP. Waypoint Name of the selected WP.
Bearing (in MILS) and information distance from the NUAV to the
selected WP. Only displayed if a WP is selected. Reference point
Position of selected RP. The user must select a RP position from
the Flight menu in order for it to be displayed. Reference point
Name of the selected RP. Bearing (in MILS) information and distance
from the selected RP to the UAV. Only displayed if a RP is
selected.
[0041] The map will automatically scale to keep all geographically
referenced objects (i.e. Operator, UAV, and selected WP/RP) within
the display area. The scale of the map is indicated by range rings
with an outer radius of 100 m, 200 m, 300 m, 400 m, 600 m, 800 m or
1000 m. The radius of the outer ring is marked on the ring.
[0042] If the selected WP or RP is the outermost object on the map,
the scale will remain constant to keep these within the display
area. The Operator may have the possibility to deselect objects,
and deselecting the outermost object will rescale the map to
encompass the next outermost object.
[0043] If the UAV is the outermost object, the map will rescale to
the minimum radius required to keep the UAV within the display
area. As the distance from the Operator to the UAV is increased the
map is zoomed out in steps, and the corresponding range ring is
displayed. The opposite will happen if the UAV is flown towards the
Operator thus reducing the distance.
[0044] FIGS. 3 and 4 shows examples of different range rings. The
300 m and 600 m, and the 400 m and 800 m range rings are almost
identical except for the numeric radius marker. In order to
differentiate between them a smaller UAV symbol might be used when
the radius is more than 400 m (i.e. 600 m and above) as illustrated
in FIG. 4.
[0045] The 1000 m range ring in this example is smaller and is
using a thicker outline than the other range rings. This is to
highlight to the Operator that the outermost object is close to the
maximum range of the DDL. It also indicates that the map is at the
smallest scale, and objects outside approximately 1200 m will not
be displayed.
[0046] FIG. 5 shows an example of the Expand mode. In this mode,
the display is centred on the selected WP and expanded to cover a
radius of 20 meters from the WP. This allows the Operator to
accurately maneuver the UAV in relation to the WP at longer ranges
using the screen only.
[0047] The UAV must be within a pre-defined distance (in this
example, approximately 20 meters) of the selected WP in order to
switch to Expand mode. If the mode is selected when the UAV is
outside the predefined distance, the mode will be pending, i.e. it
will automatically enter Expand mode when the UAV is maneuvered to
within the predefined distance to the WP. The system will
automatically transfer back to Overlay mode if the UAV is
maneuvered outside the Expand mode coverage area.
[0048] The information provided in Expand mode is similar to
Overlay mode, except for the elements described in table 2.
TABLE-US-00002 TABLE 2 Element Description Operator (user) The
Operator position is only displayed if position it is within the
Expand mode coverage area. Range ring Only a 20 meter range ring is
displayed. If the UAV is flown outside approximately 20 meters from
the WP, the system transfers back to Overlay mode. Waypoint
Position of selected WP. position Always centred on the screen.
Refpoint Position of selected RP. Only displayed if position within
the Expand mode coverage area.
[0049] FIG. 6 shows an example of Basic mode where the map overlay
is removed from the Centre screen and replaced by symbols to
indicate UAV heading.
[0050] FIG. 7 is a flow chart illustrating one embodiment of the
basic steps of the method according to the invention. The
embodiment includes respectively receiving captured image and
positioning data from the UAV, displaying the image on the flight
display and providing a transparent layer of a map based on the
positioning data upon the image.
[0051] The above description discloses different example
embodiments for illustrative purposes. A person skilled in the art
would realize a variety of different combinations of symbols,
symbol designs all being within the scope of the embodiments.
[0052] It should be emphasized that the term "comprises/comprising"
when used in this specification is taken to specify the presence of
stated features, integers, steps or components, but does not
preclude the presence or addition of one or more other features,
integers, steps, components or groups thereof. It should also be
noted that the words "a" or "an" preceding an element do not
exclude the presence of a plurality of such elements.
[0053] It should also be emphasized that the steps of the methods
defined in the appended claims may, without departing from the
embodiments herein, be performed in another order than the order in
which they appear in the claims.
* * * * *