U.S. patent application number 16/352994 was filed with the patent office on 2020-09-17 for method and system for providing object detection warning.
This patent application is currently assigned to Caterpillar Inc.. The applicant listed for this patent is Caterpillar Inc.. Invention is credited to Kristen Ann Gerstner, Renita Ryan Metcalf, Shailesh Babu Ramineni, Darrell Richards, Sarah Elizabeth Schonert, Micheal David Valerio, Brad Robert Van De Veer, Qi Xing.
Application Number | 20200290637 16/352994 |
Document ID | / |
Family ID | 1000003944307 |
Filed Date | 2020-09-17 |
![](/patent/app/20200290637/US20200290637A1-20200917-D00000.png)
![](/patent/app/20200290637/US20200290637A1-20200917-D00001.png)
![](/patent/app/20200290637/US20200290637A1-20200917-D00002.png)
![](/patent/app/20200290637/US20200290637A1-20200917-D00003.png)
![](/patent/app/20200290637/US20200290637A1-20200917-D00004.png)
![](/patent/app/20200290637/US20200290637A1-20200917-D00005.png)
![](/patent/app/20200290637/US20200290637A1-20200917-D00006.png)
![](/patent/app/20200290637/US20200290637A1-20200917-D00007.png)
United States Patent
Application |
20200290637 |
Kind Code |
A1 |
Schonert; Sarah Elizabeth ;
et al. |
September 17, 2020 |
METHOD AND SYSTEM FOR PROVIDING OBJECT DETECTION WARNING
Abstract
A method of providing object detection warning to an operator of
a machine is provided. The machine includes object detection
modules, a display unit and a controller communicably coupled to
the object detection modules and the display unit. The method
detects at least one object being located within a predefined
distance range from the machine and determines a distance and
position of the detected at least one object relative to the
machine. The method further displays a threat warning widget on the
display unit. The widget includes a plurality of threat level
indicators, each of the threat level indicators being
representative of a warning level as a function of distance and
position of the object relative to the machine. Furthermore, the
method activates at least one threat level indicator based on the
determined distance and position of the detected at least one
object relative to the machine.
Inventors: |
Schonert; Sarah Elizabeth;
(Dunlap, IL) ; Metcalf; Renita Ryan; (Peoria,
IL) ; Richards; Darrell; (Bloomington, IL) ;
Gerstner; Kristen Ann; (Peoria, IL) ; Valerio;
Micheal David; (Morton, IL) ; Ramineni; Shailesh
Babu; (Peoria, IL) ; Van De Veer; Brad Robert;
(Washington, IL) ; Xing; Qi; (Peoria, IL) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Caterpillar Inc. |
Deerfield |
IL |
US |
|
|
Assignee: |
Caterpillar Inc.
Deerfield
IL
|
Family ID: |
1000003944307 |
Appl. No.: |
16/352994 |
Filed: |
March 14, 2019 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06K 9/00805 20130101;
B60W 50/14 20130101; B60W 30/0956 20130101; B60W 2050/146
20130101 |
International
Class: |
B60W 50/14 20060101
B60W050/14; G06K 9/00 20060101 G06K009/00; B60W 30/095 20060101
B60W030/095 |
Claims
1. A method of providing object detection warning to an operator of
a machine, the machine including one or more object detection
sensors, a display unit and a controller communicably coupled to
the one or more object detection modules and the display unit, and
one or more machine cameras positioned on the machine, the method
comprising: detecting, by the controller using the one or more
object detection sensors, at least one object being located within
a predefined distance range from the machine; determining, by the
controller using the one or more object detection sensors, a
distance and position of the detected at least one object relative
to the machine; displaying, by the controller on the display unit,
a threat warning widget including a plurality of threat level
indicators, each of the threat level indicators being
representative of a warning level as a function of distance and
position of the object relative to the machine, and a plurality of
threat sectors corresponding to a field of view of each of the
machine cameras; activating, by the controller, at least one threat
level indicator based on the determined distance and position of
the detected at least one object relative to the machine;
identifying, by the controller, one of the machine cameras being
displayed on the display unit along with the threat warning widget;
and highlighting, by the controller on the display unit, one of the
threat sectors on the threat warning widget, based on the
identified machine camera being displayed on the display unit.
2. (canceled)
3. The method of claim 1, wherein the threat warning widget
includes a plurality of threat sectors corresponding to a capturing
range of each of the one or more object detection sensors, each of
the threat sectors further including a plurality of threat level
indicators.
4. The method of claim 3 further comprising: detecting, by the
controller using the one or more object detection sensors, the at
least one object being positioned within one of the plurality of
threat sectors; and activating, by the controller, one of the
threat level indicators of the detected threat sector displayed on
the display unit.
5. The method of claim 1, wherein each of the plurality of threat
level indicators correspond to a predefined range of distances from
the machine.
6. The method of claim 1, wherein the plurality of threat level
indicators includes a first indicator representing a low
criticality warning level, a second indicator representing a medium
criticality warning level and a third indicator representing high
criticality warning level.
7. The method of claim 1, wherein the plurality of threat level
indicators includes at least one threat level indicator blinking at
a frequency.
8. An object detection warning system for a machine, the system
comprising: one or more object detection sensors, positioned on the
machine, for detecting at least one object being located within a
predefined distance range from the machine; one or more machine
cameras positioned on the machine; a display unit for displaying a
threat warning widget including a plurality of threat level
indicators, each of the threat level indicators being
representative of a warning level as a function of distance and
position of the object relative to the machine, and a plurality of
threat sectors corresponding to a field of view of each of the
machine cameras; a controller communicably coupled to the one or
more object detection modules sensors and the display unit, the
controller being configured to: determine a distance and position
of the detected at least one object relative to the machine;
control the display of the threat warning widget on the display
unit by activating at least one threat level indicator of the
plurality of threat level indicators based on the determined
distance and position of the detected at least one object relative
to the machine; identifying, by the controller, one of the machine
cameras being displayed on the display unit along with the threat
warning widget; and highlighting, by the controller on the display
unit, one of the threat sectors on the threat warning widget, based
on the identified machine camera being displayed on the display
unit.
9. (canceled)
10. The object detection warning system of claim 8, wherein the
threat warning widget includes a plurality of threat sectors
corresponding to a capturing range of each of the one or more
object detection sensors, each of the threat sectors further
including a plurality of threat level indicators.
11. The object detection warning system of claim 10, wherein the
controller is further configured to: detect the at least one object
being positioned within one of the plurality of threat sectors; and
activate one of the threat level indicators of the detected threat
sector of the threat warning widget displayed on the display
unit.
12. The object detection warning system of claim 8, wherein each of
the plurality of threat level indicators correspond to a predefined
range of distances from the machine.
13. The object detection warning system of claim 8, wherein the
plurality of threat level indicators displayed on the display unit,
includes a first indicator representing a low criticality warning
level, a second indicator representing a medium criticality warning
level and a third indicator representing high criticality warning
level.
14. The object detection warning system of claim 8, wherein the
plurality of threat level indicators includes at least one threat
level indicator blinking at a predefined frequency.
15. A machine comprising: an object detection warning system
including: one or more object detection sensors positioned on the
machine for detecting at least one object being located within a
predefined distance range from the machine, one or more machine
cameras positioned on the machine, each of the one or more object
detection modules sensors and the machine cameras defining a
plurality of threat zones associated with the machine; a display
unit for displaying a threat warning widget including a plurality
of threat sectors for each of the plurality of threat zones, each
of the threat sectors including a plurality of threat level
indicators being representative of warning levels as function of
distance and position of the object relative to the machine; and a
controller communicably coupled to the one or more object detection
modules sensors and the display unit, the controller being
configured to: determine a distance of the detected at least one
object relative to the machine; detect the at least one object
being positioned within a threat zone; control the threat warning
widget displayed on the display unit by activating at least one
threat level indicator of one of the threat sectors based on the
detected threat zone and the distance of the detected at least one
object relative to the machine; identify one of the machine cameras
being displayed on the display unit along with the threat warning
widget; and highlight one of the threat sectors on the threat
warning widget, based on the identified machine camera being
displayed on the display unit.
16. (canceled)
17. The machine of claim 15, wherein each of the plurality of
threat level indicators correspond to a predefined range of
distances from the machine.
18. The machine of claim 15, wherein the plurality of threat level
indicators displayed on the display unit, includes a first
indicator representing a low criticality warning level, a second
indicator representing a medium criticality warning level and a
third indicator representing high criticality warning level.
19. The machine of claim 15, wherein the plurality of threat level
indicators includes at least one threat level indicator blinking at
a predefined frequency.
Description
TECHNICAL FIELD
[0001] The present disclosure relates generally to object detection
systems and methods. More particularly, the present disclosure
relates to a method and system for providing object detection
warning to an operator of a machine operating at a worksite.
BACKGROUND
[0002] Large machines, such as, wheel loaders, off-highway haul
trucks, excavators, motor graders, and other types of earth-moving
machines are used to perform a variety of tasks that often involve
moving intermittently between and stopping at certain locations
within a worksite. Operators of such machines need to be aware of
the machine surroundings to perform smooth operations at the
worksite. However, the operator can directly view only a portion of
surroundings of the machines as there exist a number of blind spots
around the machine at the worksite, that may not be directly
visible to the operator during operation of the machine. Therefore,
accurate operation of the machine becomes a challenge for the
operator.
[0003] U.S. Pat. No. 9,139,133 (hereinafter referred to as the '133
patent) relates to a vehicle collision warning system and method
that alerts a driver of an impending collision by displaying
enhanced video of an upcoming road segment, where the enhanced
video includes an in-path object that is highlighted so that it
stands apart. The system receives video of an upcoming road segment
from a forward-looking vision system, obtain certain collision data
for an in-path object, and then use the video and the collision
data to provide enhanced video to a visual display that is seen by
the driver. The video may include an in-path object that is
highlighted according to the collision data (e.g., highlighted with
a color-coded scheme where the in-path object changes color based
on time-to-collision data) so that it stands apart from the rest of
the upcoming road segment.
SUMMARY OF THE INVENTION
[0004] In one aspect, a method of providing object detection
warning to an operator of a machine is provided. The machine
includes one or more object detection modules, a display unit and a
controller communicably coupled to the one or more object detection
modules and the display unit. The method detects at least one
object being located within a predefined distance range from the
machine and determines a distance and position of the detected at
least one object relative to the machine. The method further
displays a threat warning widget on the display unit. The widget
includes a plurality of threat level indicators, each of the threat
level indicators being representative of a warning level as a
function of distance and position of the object relative to the
machine. Furthermore, the method activates at least one threat
level indicator based on the determined distance and position of
the detected at least one object relative to the machine.
[0005] In another aspect, an object detection warning system is
provided. The system includes one or more object detection modules,
positioned on the machine, for detecting at least one object being
located within a predefined distance range from the machine. The
system further includes a display unit for displaying a threat
warning widget including a plurality of threat level indicators,
each of the threat level indicators being representative of a
warning level as a function of distance and position of the object
relative to the machine. Furthermore, the system includes a
controller communicably coupled to the one or more object detection
modules and the display unit. The controller is configured to
determine a distance and position of the detected at least one
object relative to the machine and control the display of the
threat warning widget on the display unit by activating at least
one threat level indicator of the plurality of threat level
indicators based on the determined distance and position of the
detected at least one object relative to the machine.
[0006] In a yet another aspect, a machine having the object
detection warning system is provided. The object detection warning
system includes one or more object detection modules positioned on
the machine for detecting at least one object being located within
a predefined distance range from the machine and one or machine
cameras positioned on the machine. Each of the one or more object
detection modules and the machine cameras define a plurality of
threat zones corresponding to a respective capturing range. The
system further includes a display unit for displaying a threat
warning widget including a plurality of threat sectors for each of
the plurality of threat zones, each of the threat sectors including
a plurality of threat level indicators being representative of
warning levels as function of distance and position of the object
relative to the machine. Furthermore, the system includes a
controller communicably coupled to the one or more object detection
modules and the display unit. The controller is configured to
determine a distance of the detected at least one object relative
to the machine and detect the at least one object being positioned
within a threat zone. Further, the controller is configured to
control the threat warning widget displayed on the display unit by
activating at least one threat level indicator of one of the threat
sectors based on the detected threat zone and the distance of the
detected at least one object relative to the machine.
BRIEF DESCRIPTION OF THE DRAWINGS
[0007] FIG. 1 illustrates an exemplary machine including an object
detection warning system, in accordance with an embodiment of the
present disclosure;
[0008] FIG. 2 illustrates a top view of the exemplary machine of
FIG. 1;
[0009] FIG. 3 illustrates a block diagram of the object detection
warning system, in accordance with an embodiment of the present
disclosure;
[0010] FIG. 4 illustrates a top view of the exemplary machine with
associated threat zones, in accordance with an embodiment of the
present disclosure;
[0011] FIG. 5 illustrates an exemplary operator interface, in
accordance with an embodiment of the present disclosure;
[0012] FIG. 6 illustrates a method flowchart for providing object
detection warning, in accordance with an embodiment of the present
disclosure; and
[0013] FIG. 7 illustrates an operator interface displaying the
object detection warning, in accordance with the embodiment of the
present disclosure.
DETAILED DESCRIPTION
[0014] Reference will now be made in detail to specific embodiments
or features of the present disclosure, examples of which are
illustrated in the accompanying drawings. Generally, corresponding
reference numbers will be used throughout the drawings to refer to
the same or corresponding parts. Also, wherever possible, the same
reference numbers will be used throughout the drawings to refer to
the same, or the like parts.
[0015] FIG. 1 illustrates an exemplary machine 100 operating at a
worksite 102. For example, the machine 100 may be embodied as a
loader, a compactor, a dozer, an excavator, or any other similar
machine. The machine 100 may be an earth moving machine, a
construction machine, a mining machine, an agricultural machine, a
forest machine, etc. In the illustrated embodiment, the machine 100
is depicted as a wheel loader. Further, the machine 100 may be a
manned machine or an unmanned machine. In some embodiments, the
machine 100 may be a machine having various levels of autonomy,
such as a fully-autonomous machine, a semi-autonomous machine, a
remotely operated machine, or a remotely supervised machine.
[0016] In an embodiment of the present disclosure, the machine 100
includes a first portion 104 and a second portion 106 capable of
articulation and oscillation with respect to one another. To this
end, the machine 100 includes a two-part frame system or a split
chassis system 108 that supports both the first machine part 104
and the second machine part 106 atop respective sets of ground
engaging members 110. The ground engaging members 110 are
configured to propel the machine 100 at the worksite 102. As shown
in FIG. 1, the ground engaging members 110 include a set of drive
wheels having a pair of front wheels 110-1 and a set of rear wheels
110-2. Although the ground engaging members 110 are shown to be
drive wheels, it may be contemplated that other types of ground
engaging members, such as endless tracks, may also be
implemented.
[0017] The first portion 104 of the machine 100 supports an engine
compartment 112 to house a power unit 114 and an operator cab 116.
The power unit 114 includes an engine, hereinafter referred to as
the engine 114, that may be based on one of the commonly applied
power-generation units, such as an internal combustion engine (ICE)
having a V-type configuration engine, an in-line, configuration
engine, or an engine with different configurations, as is
conventionally known. However, aspects of the present disclosure,
need not be limited to a particular type of engine 114.
[0018] Further, the operator cab 116 may be an enclosure that may
include one or more of electronic panels, displays, buttons,
joysticks and various other physically actuable entities. It may
further be contemplated that the operator cab 116 may be remotely
located in case of a remotely operated machine 100.
[0019] The second portion 106 of the machine 100 includes a work
tool 118 configured to perform a designated work, such as modifying
a work surface at the worksite 102. For example, the work tool 118
may be a bucket, grapple, grader, and so on, that may be powered by
the power unit 114.
[0020] The machine 100 includes an object detection system 119
configured to monitor the surroundings of the machine 100 and
detect objects in proximity of the machine 100. The object
detection system 119 may include one or more machine cameras 120
and one or more object detection modules 122, positioned on the
machine 100. The machine cameras 120 are positioned on one or more
sides of the machine 100 and are configured to capture images of
the surroundings of the machine 100, such as to cover the blind
zones and low visibility zones in the surroundings of the machine
100. In the illustrated example, the machine 100 includes four
machine cameras 120-1, 120-2, 120-3 and 120-4, positioned on front,
rear, left and right side (shown in FIG. 2) of the machine 100
respectively. It may be contemplated that the sides of the machine
100 are defined as per the direction of travel of the machine 100.
Each of the machine cameras 120 has a respective field of view for
capturing images of the surroundings of the machine 100. For
example, the machine camera 120-1 is configured to capture the
front region of the surrounding of the machine 100, whereas the
rear machine camera 120-2 is configured to capture the rear side of
the surrounding of the machine 100. Similarly, the left and right
machine cameras 120-3, 120-4 are configured to capture left and
right side of the surrounding the machine 100.
[0021] The object detection system 119 further includes the object
detection modules 122 positioned on one or more sides of the
machine 100 and configured to detect an object in proximity to the
machine 100. In an exemplary embodiment, the machine 100 includes
four object detection modules 122-1, 122-2, 122-3 and 122-4
positioned in conjunction to the machine cameras 120-1, 120-2,
120-3 and 120-4, respectively.
[0022] The object detection modules 122 may include radars, well
known in the art, having their respective capturing ranges and
configured to detect objects within a predefined range of distances
falling in their capturing ranges. The object detection modules 122
may include near range, medium range, and/or far range radars
configured to detect objects at predefined range of distances. For
example, a near range may correspond to a distance range of 1 meter
to 8 meters, whereas the medium range may correspond to a distance
range of 8 meters to 18 meters and the far range may correspond to
18 meters to 30 meters distance range from the machine 100. It may
be contemplated that the values of the ranges of distances may vary
to achieve similar results, without deviating from the scope of the
claimed subject matter. It may also be contemplated that other
typical obstacle detection sensors, such as sonar systems, laser
systems, optical systems, infrared systems, etc., may also be
included without deviating from the scope of the claimed subject
matter.
[0023] Although the machine cameras 120 and the object detection
modules 122 are shown to be positioned at four sides of the machine
100 and in conjunction to one another, it may be contemplated that
the number and positions of the machine cameras 120 and the object
detection modules 122 may be varied to achieve similar results
without deviating from the scope of the claimed subject matter. In
some alternative examples, the machine cameras 120 and the object
detection modules 122 may be positioned independent of one another.
In some other examples, the machine cameras 120 and the object
detection modules 122 may be combined into a single unit without
deviating from the scope of the claimed subject matter.
[0024] In an embodiment of the present disclosure, the machine 100
includes an object detection warning system 300 for providing a
warning of an object detected in proximity of the machine 100, to
an operator stationed within the operator cab 116. The object
detection warning system 300 and its operation is further described
in greater detail with reference to FIGS. 3 through 6.
[0025] The object detection warning system 300, hereinafter
interchangeably referred to as the system 300, may include suitable
logic, circuitry, and/or interfaces that are configured to provide
warning of an object detected near the machine 100. A person having
ordinary skill in the art would appreciate that the scope of the
disclosure is not limited to include the system 300 provided
onboard the machine 100. In some embodiments, the system 300, or
portions of the system 300, may be implemented remotely with
respect to the machine 100, without departing from the scope of the
disclosure.
[0026] Referring to FIG. 3, the object detection warning system 300
includes a memory device 302, one or more output devices 304, the
object detection system 119 having the one or more machine cameras
120 and the object detection modules 122, and a controller 306. In
an exemplary embodiment of the present disclosure, the one or more
components of the system 300 may be configured to communicate via a
communication network (not shown). Additionally, or alternatively,
the system 300 may include fewer or additional elements, not
described herein for the sake of brevity of the disclosure.
[0027] The memory device 302 is configured to store a set of
instructions that are executable by the controller 306 to perform
predetermined operations associated with the machine 100. The
memory device 302 may include, but are not limited to, a
Random-Access Memory (RAM), a Read Only Memory (ROM), a Hard Disk
Drive (HDD), and a Secure Digital (SD) card.
[0028] The one or more output devices 304 are configured to provide
output to the operator of the machine 100 in the form of audio,
video, textual and/or any other form, based on instructions
received from the controller 306. The one or more output devices
304 may be positioned within the operator cab 116 of the machine
100 and/or may be remotely located with respect to the machine 100
and the worksite 102. The one or more output devices 304 includes a
display unit 308 for providing visual indications to the operator
related to various operations of the machine 100. The display unit
308 may embody, for example, a monitor, a Liquid Crystal Display
(LCD) screen, a plasma screen, a screen of a handheld device, or
any other device capable of providing visual information to the
operator of the machine 100. It may be contemplated that the output
devices 304 may further include other types of output devices, such
as audio output device and/or tactile output device to provide an
additional audio and/or tactile indications to the operator in
combination with the visual indications, regarding various
operations of the machine 100.
[0029] The controller 306 is communicably coupled to the memory
device 302, the one or more output devices 304, the one or more
machine cameras 120 and the object detection modules 122. The
controller 306 is configured to execute the instruction stored in
the memory device 302 to perform one or more predetermined
operations associated with the machine 100. The controller 306 may
be implemented using one or more controller technologies, such as
Application Specific Integrated Circuit (ASIC), Reduced Instruction
Set Computing (RISC) technology, Complex Instruction Set Computing
(CISC) technology, etc. According to an embodiment of the present
disclosure, the controller 306 may be implemented as an Electronic
Control Module (ECM) of the machine 100.
[0030] In an embodiment of the present disclosure, the controller
306 is configured to identify a number of threat zones associated
with the machine 100. In one example, the threat zones may be based
on the capturing ranges of the object detection modules 122 and/or
the field of views of the machine cameras 120 that may cover one or
more blind zones or low visibility zones around the machine 100.
The capturing ranges of each of the object detection modules 122 as
well as the field of views of each of the machine cameras 120 may
be stored in the memory device 302, and the controller 306 may
identify the threat zones based on the stored capturing range and
field of view information. In one example, the controller 306
identifies four threat zones, such as a front threat zone 310, a
rear threat zone 312, a left threat zone 314 and a right threat
zone 316, as shown in FIG. 4. Accordingly, the front threat zone
310 corresponds to the capturing range of the object detection
module 122-1 and the field of view of the machine camera 120-1
positioned on the front of the machine 100, and the rear threat
zone 312 corresponds to the capturing range of the object detection
module 122-2 and the field of view of the machine camera 120-2
positioned on the rear of the machine 100. Similarly, the left
threat zone 314 and the right threat zone 316 correspond to the
capturing ranges of the object detection modules 122-3, 122-4 and
the field of view of the machine cameras 120-3, 120-4 positioned on
the left and right side of the machine 100, respectively. Although,
the illustrated threat zones 310, 312, 314, 316, are described to
be based on both the capturing ranges of the object detection
modules 122 and the field of views of the machine cameras 120, it
may be contemplated that in some embodiments where the machine
cameras 120 and the object detection modules 122 are independent of
one another, some threat zones may be based only on the field of
view of machine cameras 120 or only on the capturing ranges of the
object detection modules 122, while some of the threat zones may be
based on both.
[0031] Referring to FIG. 5, the controller 306 is configured to
display an operator interface 400, on the display unit 308, to
provide various information to the operator about the operations of
the machine 100. In an embodiment of the present disclosure, the
operator interface 400 may include a first display section 402
configured to display a camera view 403 captured by one of the
machine cameras 120 and a second display section 404 configured to
display a threat warning widget 406 for providing an object
detection warning to the operator of the machine 100. In one
example, the first display section 402 may be displayed as a larger
primary section whereas the second display section 404 may be
displayed as a smaller auxiliary section on the operator interface
400. In other embodiments, the threat warning widget 406 may be
superimposed on the camera view 403 displayed on the first display
section 402. It may be further contemplated that the layout of the
operator interface 400, the camera view 403 and the threat warning
widget 406 is merely exemplary and may be varied without deviating
from the scope of the claimed subject matter.
[0032] The controller 306 is configured to receive the images
captured by each of the machine cameras 120 for displaying the
camera view 403 on the display unit 308. In some embodiments, the
controller 306 may be configured to display a camera view according
to the movement or gear selection of the machine 100. For example,
the controller 306 may display the camera view 403 as captured by
the rear machine camera 120-2 on the first display section 402 of
the operator interface 400, when the machine 100 is in reverse gear
or is detected to be moving in reverse direction. Alternatively, or
additionally, the controller 306 may switch the camera view 403
displayed on the display unit 308 in response to the operator's
selection of one of the machine cameras 120.
[0033] The controller 306 is further configured to communicate with
the one or more machine cameras 120 and the one or more object
detection modules 122 to control display of the threat warning
widget 406 on the display unit 308.
[0034] In an embodiment of the present disclosure, the threat
warning widget 406 includes a number of threat sectors
corresponding to and representing each of the identified threat
zones 310, 312, 314 and 316. For example, the threat warning widget
406 includes a first threat sector 408 representing the front
threat zone 310, a second threat sector 410 representing the rear
threat zone 312, a third threat sector 412 representing the left
threat zone 314 and a fourth threat sector 414 representing the
right threat zone 316. As shown in FIG. 5, the threat sectors 408,
410, 412, 414 are displayed around a machine representation 416 in
the center of the widget 406. In one example, the threat sectors
408, 410, 412, 414 are shown to have an arcuate shape, representing
the semi-circular capturing ranges of the object detection modules
122 and the field of view of each of the machine cameras 120.
However, it may be contemplated that the number, shape and
positioning of these threat sectors on the widget 406, are merely
exemplary and may be varied to achieve similar results without
deviating from the claimed subject matter. It may also be
contemplated that the number of threat sectors displayed on the
widget 406 may be varied according to the number of object
detection modules 122 and the machine cameras 120 positioned on the
machine 100. Similar to the threat zones, it may be contemplated
that in some embodiments where the machine cameras 120 and the
object detection modules 122 are independent of one another, some
threat sectors on the widget 406 may be based only on the field of
view of machine cameras 120 while some of the threat sectors may be
based only on the capturing ranges of the object detection modules
122. For example, when the machine 100 includes only two object
detection modules 122 positioned on the front and back side of the
machine 100, and four machine cameras 122, then the threat sectors
408, 410 may be based on the capturing range of the object
detection modules 122-1, 122-2 as well as the field of view of the
machine cameras 120-1, 120-2, whereas the threat sectors 412, 414
are based only on the field of view of the machine cameras 120-3,
120-4 respectively.
[0035] Each of the threat sectors 408, 410, 412, 414 includes a
number of threat level indicators to indicate a level or
criticality of threat warning, as a function of distance as well as
position of a detected object relative to the machine 100. For
example, each of the threat sectors 408, 410, 412 and 414 includes
a first threat level indicator 418 indicating a low criticality
warning level, a second threat level indicator 420 indicating a
medium criticality warning level and a third threat level indicator
422 indicating a high criticality warning level associated with the
object detection near the machine 100. Although there are three
threat level indicators described herein, it may be contemplated
that the number of threat level indicators displayed on the widget
406 is also exemplary and may be varied without deviating from the
scope of the claimed subject matter.
[0036] Each of the threat level indicators 418, 420 and 422 are
calibrated based on predefined distance ranges with respect to the
machine 100. For example, a first distance range between 18 meters
to 30 meters from the machine 100 may be used to calibrate the
first threat level indicator 418, which indicates a low criticality
warning level. Similarly, a second distance range between 8meters
to 18 meters from the machine 100 may be used to calibrate the
second threat level indicator 420, which indicates medium
criticality warning level. Further, a third distance range between
1 meter to 8 meters from the machine 100 may be used to calibrate
the third threat level indicator 422, which indicates the high
criticality warning level. The criticality levels associated with
these warnings may indicate how much distance and/or time is left
for the operator of the machine 100 to take an action in order to
avoid collision with the detected object. It may be contemplated
that the predefined distance ranges may be varied, such as based on
the positioning (for example, height and angle) of the object
detection modules 122 and the machine cameras 120 on the machine
100, without deviating from the scope of the claimed subject
matter.
[0037] In an exemplary embodiment, the threat level indicators 418,
420, 422 may be in the form of color bands based on the level of
criticality associated with the warning associated with them. For
example, the first threat level indicator 418 may be a yellow
colored band to visually indicate a low criticality warning level,
the second threat level indicator 420 may be an orange colored band
to visually indicate medium criticality warning level and the third
threat level indicator 422 may be a red colored band to visually
indicate high criticality warning level associated with the object
detected in proximity of the machine 100. It may be contemplated
that the threat level indicators 418, 420, 422 may be represented
in any other manner and/or using any other combination of
color-coded representations, to achieve similar results, without
deviating from the scope of the claimed subject matter.
[0038] In an embodiment of the present disclosure, each of the
threat level indicators 418, 420, 422 are configured to be
activated by the controller 306, as and when an object is detected
within the corresponding distance range (for the threat level
indicators 418, 420 and 422) from the machine 100. For example, as
soon as an object is detected at a distance within the distance
range of first threat level indicator 418, i.e., less than 30
meters and more than 18 meters from the machine 100, the yellow
colored band of the first threat level indicator 418 is activated
by the controller 306, thereby indicating to the operator of the
machine 100, a low criticality warning level associated with the
detected object. Similarly, when the object comes closer to the
machine 100, and is at a distance less than 8 meters and greater
than or equal to 1 meter, the third threat level indicator 422 is
activated by the controller 306 to indicate a high criticality
warning level associated with the detected object. Furthermore, in
an exemplary embodiment, the third threat level indicator 422 may
be further configured to blink at a predefined frequency (for
example, 10 times per minute) when the detected object comes even
closer to the machine 100 such that a distance between the object
and the machine 100 is less than the third distance range
corresponding to the third threat level indicator 422. Therefore,
the blinking third threat level indicator 422 may indicate to the
operator that an urgent and immediate action, such as stopping the
machine 100 movement, is required to be taken to avoid collision
with the detected object.
[0039] In an embodiment, the controller 306 is configured to
control the display of the widget 406 according to a position of
the detected object relative to the machine 100. The controller 306
is configured to determine the position of the detected object
relative to the machine 100, based on the information received from
one of the object detection modules 122 that detects the object in
proximity of the machine 100. The position of the object relative
to the machine 100 indicates whether the object is detected on the
left or right or rear or front of the machine 100.
[0040] Furthermore, based on the detected position of the object,
the controller 306 is configured to identify one of the threat
zones 310, 312, 314, 316, within which the object is positioned,
and accordingly identify the relevant threat sectors 408, 410, 412,
414 and the respective threat level indicators 418, 420, 422, to be
activated on the widget 406. For example, when the object detection
module 122-3, positioned on the left side of the machine 100,
detects an object at a distance of 15 meters from the machine 100,
the controller 306 identifies and activates the corresponding
threat level indicator of a threat sector, i.e., the second threat
level indicator 420-3 (in orange color) of the third threat sector
412 on the left, on the widget 406 for providing the object
detection warning to the operator of the machine 100.
[0041] In an embodiment of the present disclosure, the controller
306 is further configured to provide a visual indication on the
widget 406, along with the object detection warning, for indicating
an active machine camera 120 being displayed on the first display
section 402 of the user interface 400. As shown in FIG. 5, the
controller 306 is configured to outline and/or highlight (as shown
by 424) one of the threat sectors 408, 410, 412, 414 to indicate
the machine camera being displayed as the camera view 403. In some
examples, the controller 306 may highlight and/or outline one of
the threat sectors 408, 410, 412, 414 in a fourth color, such as
blue, which is not otherwise included anywhere in the widget 406.
In one example, the controller 306 is configured to identify the
object detection module 122 being positioned in conjunction to the
active machine camera 120, and accordingly highlight and/or outline
one of the threat sectors 408, 410, 412, 414 displayed on the
widget 406, corresponding to the determined object detection module
122. For example, if the rear machine camera 120-2 is being
displayed on the first display section 402 of the operator
interface 400, then the controller 306 identifies the second object
detection module 122-2 positioned on the rear side of the machine
100 and accordingly highlights (as shown by 424) the second threat
sector 410 on the widget 406. Alternatively, where the machine
cameras 120 are positioned independent of the object detection
modules 122, the controller 306 may highlight one the threat
sectors on the widget 406, based only on the identified machine
camera 120. In this case, even when the object detection is not
available in any one or more of the threat zones around the machine
100, the widget 406 may provide a visual indication of one of the
machine cameras 120 being viewed on the display unit 308 by
highlighting one of the threat sectors on the widget 406.
[0042] Therefore, during operation, the threat warning widget 406
of the present disclosure, provides a dual indication to the
operator, i.e., of an object detection warning (indicating how
close the object is from the machine 100 and on which side of the
machine 100 is the object positioned) as well as an indication of
the active machine camera 120 being viewed on the display unit 308.
In cases where there are multiple objects detected in proximity of
the machine 100, all the corresponding threat sectors 408, 410,
412, 414 and their threat level indicators 418, 420, 422 may be
activated by the controller 306, in a similar manner as described
above.
Industrial Applicability
[0043] The object detection warning system 300, in accordance with
the embodiments of the present disclosure, provides information
about an object detected in proximity of the machine 100. The
object detection warning system 300 displays a threat warning
widget 406 on a display unit 308 associated with the machine 100,
where the widget 406 provides a visual indication of the object
detection warning along with an indication of which machine camera
120 is being viewed on the display unit 308 alongside of the widget
406.
[0044] FIG. 6 illustrates an exemplary method for providing an
object detection warning to an operator of the machine 100. FIG. 7
illustrates an exemplary widget 406' displayed on the display unit
308 for providing the object detection warning, when an object 702
is detected in proximity of the machine 100.
[0045] Referring to FIGS. 6 and 7, during operation of the machine
100 at the worksite 102, at step 602, the controller 306 detects,
using the one or more object detection modules 122, that the object
702 is positioned in proximity of the machine 100, i.e., within a
predefined distance range from the machine 100. For example, one or
more object detection modules 122-1, 122-2, 122-3 and 122-4
positioned on the machine 100 are configured to detect the object
702 within a predefined range of distances and provide position
information about the detected object 702 relative to the machine
100 to the controller 306.
[0046] At step 604, the controller 306 determines a position and a
distance of the object 702 relative to the machine 100. For
example, the distance of the object 702 determines how far is the
object 702 from the machine 100, and the position of the object 702
determines which side (such as front, rear, left or right side) of
the machine 100 is the object 702 detected. The controller 306 may
identify the position of the object 702, based on the positioning
of the respective object detection module 122 that detected the
object 702. Therefore, when the object detection module 122-3
positioned on the left of the machine 100 detects the object 702,
the controller 306 automatically knows the position of the object
702 to be on the left side of the machine 100. Further, the
controller 306 identifies one of the threat zones 308, 312, 314,
316 in which the object 702 is detected, based on the determined
position of the object 702 relative to the machine 100.
[0047] At step 606, the controller 306 displays the threat warning
widget 406' on the display unit 308, for providing object detection
warning to the operator of the machine 100. In an embodiment of the
present disclosure, the threat warning widget 406' includes a
number of threat sectors 408, 410, 412, 414 and a number of threat
level indicators 418, 420, 422 that represent warning levels as
function of position and distance of the object 702 relative to the
machine 100. As illustrated, each of the threat sectors 408, 410,
412, 414 include their respective threat level indicators 418, 420,
422 displayed on the widget 406'. Each of the threat sectors 408,
410, 412, 414 indicate the position of the detected object 702
relative to the machine 100, and the threat level indicators 418,
420, 422, within each of the threat sectors 408, 410, 412, 414,
indicate the distance of the detected object 702 relative to the
machine 100. For example, the threat level indicators 418-3, 420-3,
422-3 positioned within the third threat sector 412 are indicative
of distance of the object 702 detected on the left side of the
machine 100. Similarly, the threat level indicators 418-2, 420-2,
422-2 positioned within the second threat sector 410 are indicative
of distance of an object detected on the rear side of the machine
100.
[0048] In an exemplary embodiment, by default, all the threat
sectors 408, 410, 412, 414' and their corresponding threat level
indicators 418, 420, 422 remain deactivated on the widget 406'. In
one example, all the threat sectors 408', 410, 412, 414 and their
respective threat level indicators 418, 420, 422 may be displayed
in one color, such as grey color to indicate the deactivated state.
At step 608, the controller 306 activates one of the threat level
indicators 418, 420, 422 of one of the threat sectors 408, 410,
412, 414 based on the detected position and distance of the object
702 relative to the machine 100. For example, as shown in FIG. 7,
when the object 702 is detected on the left side of the machine 100
at a distance of 7 meters, the controller 306 activates the third
threat level indicator 422-3, by highlighting it in red color, on
the third threat sector 412 displayed on the widget 406', thereby
indicating to the operator that the object 702 is located on the
left side of the machine 100 and requires an immediate action by
the operator, such as stopping the machine 100 movement, to avoid
collision.
[0049] At step 610, the controller 306 further identifies one of
the machine cameras 120 being displayed as camera view 403' on the
operator interface 400' along with the threat warning widget 406'.
The controller 306 identifies the corresponding object detection
module 122 positioned in conjunction to the identified machine
camera 120 and/or the field of view of the identified machine
camera 120 being displayed. At step 612, based on the identified
machine camera 120 being viewed as the camera view 403', the
controller 306 highlights and/or outlines the corresponding one of
the threat sectors 408, 410, 412, 414 on the widget 406', thereby
indicating to the operator which machine camera 120 is being viewed
on the operator interface 400'. For example, if the machine camera
120-2, positioned on the rear side of the machine 100, is active
and displayed on the operator interface 400', then the controller
306 accordingly highlights or outlines (as shown by highlight 424')
the second threat sector 410 on the widget 406'.
[0050] The threat warning widget 406, 406' of the present
disclosure provides two types of indications to the operator of the
machine 100. Firstly, the object detection warning, whenever the
object 702 is detected anywhere in proximity of the machine 100,
both in terms of distance and position of the detected object 702
relative to the machine 100. Secondly, the widget 406, 406' also
makes the operator aware of the machine camera 120 being displayed
on the display unit 308. Therefore, when the object 702 is detected
on a side which is not being currently displayed on the display
unit 308, the operator will know from the widget 406, 406' and
accordingly switch to view the relevant machine camera to view the
detected object 702 on the display unit 308 and take appropriate
actions. In cases where there are multiple objects detected in
proximity of the machine 100, all the corresponding threat sectors
408', 410', 412', 414' and threat level indicators 418', 420', 422'
will be activated in a manner described above.
[0051] It will be apparent to those skilled in the art that various
modifications and variations can be made to the system of the
present disclosure without departing from the scope of the
disclosure. Other embodiments will be apparent to those skilled in
the art from consideration of the specification and practice of the
system disclosed herein. It is intended that the specification and
examples be considered as exemplary only, with a true scope of the
disclosure being indicated by the following claims and their
equivalent.
* * * * *