U.S. patent application number 13/190688 was filed with the patent office on 2013-01-31 for apparatus and method of displaying hardware status using augmented reality.
This patent application is currently assigned to AMERICAN POWER CONVERSION CORPORATION. The applicant listed for this patent is Michael L. Whelihan. Invention is credited to Michael L. Whelihan.
Application Number | 20130026220 13/190688 |
Document ID | / |
Family ID | 47596407 |
Filed Date | 2013-01-31 |
United States Patent
Application |
20130026220 |
Kind Code |
A1 |
Whelihan; Michael L. |
January 31, 2013 |
APPARATUS AND METHOD OF DISPLAYING HARDWARE STATUS USING AUGMENTED
REALITY
Abstract
A method of accessing equipment status information in a data
center includes detecting, by a mobile computing device, a
fiduciary marker within at least one image captured by an image
capture device, decoding the equipment identification data from the
fiduciary marker, providing the equipment identification data to a
data center management system for the data center through a
communications network, receiving status information for the piece
of equipment from the data center management system through the
communication network, and displaying the status information on a
display screen of the mobile computing device. The fiduciary marker
is configured to contain encoded equipment identification data
associated with a piece of equipment.
Inventors: |
Whelihan; Michael L.;
(Newport, RI) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Whelihan; Michael L. |
Newport |
RI |
US |
|
|
Assignee: |
AMERICAN POWER CONVERSION
CORPORATION
West Kingston
RI
|
Family ID: |
47596407 |
Appl. No.: |
13/190688 |
Filed: |
July 26, 2011 |
Current U.S.
Class: |
235/375 |
Current CPC
Class: |
G06F 3/011 20130101;
G06F 16/9554 20190101; G06K 9/00671 20130101; G06F 3/04815
20130101; G06T 7/73 20170101; G06Q 10/087 20130101; G06K 2009/3225
20130101; G06T 19/006 20130101 |
Class at
Publication: |
235/375 |
International
Class: |
G06F 17/30 20060101
G06F017/30 |
Claims
1. A method of accessing equipment status information in a data
center, the method comprising: detecting within the data center, by
a mobile computing device, a fiduciary marker within at least one
image captured by an image capture device coupled to the mobile
computing device, the fiduciary marker configured to contain
encoded equipment identification data associated with a piece of
equipment; decoding, by the mobile computing device, the equipment
identification data from the fiduciary marker; providing, by the
mobile computing device, the equipment identification data to a
data center management system for the data center through a
communications network; receiving, by the mobile computing device,
status information comprised of at least one of an alert message,
an alarm message, a sensor value, and a group name for the piece of
equipment from the data center management system through the
communication network; and displaying the status information on a
display screen of the mobile computing device.
2. The method of claim 1, wherein the fiduciary marker includes a
two-dimensional barcode.
3. The method of claim 1, wherein displaying the status information
includes superimposing the status information upon the at least one
image.
4. The method of claim 1, further comprising sending, by the mobile
computing device and through the communication network, a control
command to the piece of equipment through the data center
management system.
5. The method of claim 1, wherein the equipment identification data
includes an Internet Protocol (IP) address of the piece of
equipment.
6. The method of claim 1, wherein the equipment identification data
includes a serial number of the piece of equipment.
7. The method of claim 1, wherein the equipment identification data
includes a Media Access Control (MAC) address of the piece of
equipment.
8. The method of claim 1, wherein the equipment identification data
includes a hostname associated with the piece of equipment.
9. The method of claim 1, wherein the equipment identification data
includes a model number of the piece of equipment.
10. The method of claim 1, wherein the equipment identification
data includes a location of the piece of equipment.
11. The method of claim 1, further comprising selecting the status
information via a GUI and receiving and displaying additional
information of at least one of an alert message, an alarm message,
a sensor value, and a group name.
12. The method of claim 1, wherein the at least one image includes
a live video feed.
13. An apparatus for accessing equipment status information in a
data center, the apparatus comprising: a camera; and a mobile
computing device having a processor, a display screen and a
wireless communication unit, the mobile computing device coupled to
the camera and configured to receive at least one image captured by
the camera; the processor configured to: detect a fiduciary marker
within the at least one image, the fiduciary marker configured to
contain encoded equipment identification data associated with a
piece of equipment in the data center; decode the equipment
identification data from the fiduciary marker; send the equipment
identification data to a data center management system for the data
center using the wireless communication unit; receive status
information comprised of at least one of an alert message, an alarm
message, a sensor value, and a group name for the piece of
equipment from the data center management system using the wireless
communication unit; and display the status information on the
display screen.
14. The apparatus of claim 13, wherein the fiduciary marker
includes a two-dimensional barcode.
15. The apparatus of claim 13, wherein the processor is further
configured to display the status information superimposed upon the
at least one image.
16. The apparatus of claim 13, wherein the processor is further
configured to send a control command to the piece of equipment
through the data center management system using the wireless
communication unit.
17. The apparatus of claim 13, wherein the equipment identification
data includes at least one of an Internet Protocol (IP) address of
the piece of equipment, a serial number of the piece of equipment,
a Media Access Control (MAC) address of the piece of equipment, a
hostname associated with the piece of equipment, a model number of
the piece of equipment, and a location of the piece of
equipment.
18. The apparatus of claim 13, wherein the processor is further
configured to send a request for additional information pertaining
to the status information, which includes at least one of an alert
message, an alarm message, a sensor value, and a group name.
19. A computer readable medium comprising computer-executable
instructions that when executed on a processor perform the acts of:
detecting, within a data center, a fiduciary marker within at least
one image captured by an image capture device coupled to the
processor, the fiduciary marker configured to contain encoded
equipment identification data associated with a piece of equipment;
decoding the equipment identification data from the fiduciary
marker; providing the equipment identification data to a data
center management system for the data center through a
communications network; receiving status information comprised of
at least one of an alert message, an alarm message, a sensor value,
and a group name for the piece of equipment from the data center
management system through the communication network; and displaying
the status information on a display screen coupled to the
processor.
20. The computer readable medium of claim 19, wherein the fiduciary
marker includes a two-dimensional barcode having hardware
identification data encoded therein.
Description
BACKGROUND
[0001] Centralized data centers are facilities for maintaining
information technology (IT) infrastructures, which include
computers, communications, and other electronic equipment. The IT
infrastructure may also include power supplies for powering the
equipment, environmental controls for cooling the equipment and
power supplies, and security systems for protecting the equipment.
Specialized systems called data center managers or data center
management systems are used by data center operations personnel to
centrally--and to some extent, automatically--monitor and control
the data center infrastructure in real-time. A central server
communicates with the various other devices over a communications
network to collect operational data, issue commands, and manage
inventory, among other functions.
[0002] Operations personnel typically interact with the data center
manager using a client device, such as a personal computer
connected to the data center manager. The client runs software that
is configured to provide a user interface to the data center
manager. Typically, the client user interface enables the user to
monitor the status of many pieces of equipment within the data
center. However, this requires that each piece of equipment be
named or otherwise identified in a human-readable manner that
allows the user to distinguish one piece of equipment from another
and know where each piece of equipment is physically located within
the data center.
SUMMARY
[0003] According to one embodiment, a method of accessing equipment
status information in a data center includes detecting, by a mobile
computing device, a fiduciary marker within at least one image
captured by an image capture device, decoding the equipment
identification data from the fiduciary marker, providing the
equipment identification data to a data center management system
for the data center through a communications network, receiving
status information for the piece of equipment from the data center
management system through the communication network, and displaying
the status information on a display screen of the mobile computing
device. The fiduciary marker is configured to contain encoded
equipment identification data associated with a piece of equipment.
In another embodiment, the fiduciary marker may include a
two-dimensional barcode.
[0004] In another embodiment, displaying the status information may
include superimposing the status information upon the at least one
image.
[0005] In another embodiment, the method may include sending, by
the mobile computing device and through the communication network,
a control command to the piece of equipment through the data center
management system.
[0006] In another embodiment, the equipment identification data may
include an Internet Protocol (IP) address of the piece of
equipment, a serial number of the piece of equipment, a Media
Access Control (MAC) address of the piece of equipment, a hostname
associated with the piece of equipment, a model number of the piece
of equipment, and/or a location of the piece of equipment.
[0007] In another embodiment, the status information may include an
alert message, an alarm message, a sensor value, and/or a group
name.
[0008] In another embodiment, the image may be a live video feed or
a series of images.
[0009] According to one embodiment, an apparatus for accessing
equipment status information in a data center includes a camera and
a mobile computing device. The mobile computing device has a
processor, a display screen and a wireless communication unit. The
mobile computing device is coupled to the camera and configured to
receive at least one image captured by the camera. The processor is
configured to detect a fiduciary marker within the at least one
image. The fiduciary marker is configured to contain encoded
equipment identification data associated with a piece of equipment
in the data center. The processor is further configured to decode
the equipment identification data from the fiduciary marker, send
the equipment identification data to a data center management
system for the data center using the wireless communication unit,
receive status information for the piece of equipment from the data
center management system using the wireless communication unit, and
display the status information on the display screen. In another
embodiment, the fiduciary marker may include a two-dimensional
barcode.
[0010] In another embodiment, the processor may be configured to
display the status information superimposed upon the at least one
image. In yet another embodiment, the processor may be configured
to send a control command to the piece of equipment through the
data center management system using the wireless communication
unit.
[0011] In another embodiment, the equipment identification data may
include an Internet Protocol (IP) address of the piece of
equipment, a serial number of the piece of equipment, a Media
Access Control (MAC) address of the piece of equipment, a hostname
associated with the piece of equipment, a model number of the piece
of equipment, and.or a location of the piece of equipment.
[0012] In another embodiment, the status information may include an
alert message, an alarm message, a sensor value, and/or a group
name.
[0013] According to one embodiment, a computer readable medium
includes computer-executable instructions that when executed on a
processor perform the acts of detecting, within a data center, a
fiduciary marker within at least one image captured by an image
capture device coupled to the processor, decoding the equipment
identification data from the fiduciary marker, providing the
equipment identification data to a data center management system
for the data center through a communications network, receiving
status information for the piece of equipment from the data center
management system through the communication network, and displaying
the status information on a display screen coupled to the
processor. In one embodiment, the fiduciary marker may be
configured to contain encoded equipment identification data
associated with a piece of equipment.
[0014] According to another embodiment, the fiduciary marker may
include a two-dimensional barcode having hardware identification
data encoded therein.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings are not intended to be drawn to
scale. In the drawings, each identical or nearly identical
component that is illustrated in various figures is represented by
a like numeral. For purposes of clarity, not every component may be
labeled in every drawing. In the drawings:
[0016] FIG. 1 is a functional block diagram of an exemplary
hardware device management system in accordance with one
embodiment;
[0017] FIG. 2 illustrates an exemplary image displayed on a mobile
computing device of one embodiment;
[0018] FIG. 3A illustrates another exemplary image displayed on the
mobile computing device of FIG. 2;
[0019] FIG. 3B illustrates yet another exemplary image displayed on
the mobile computing device of FIG. 2; and
[0020] FIG. 4 is a flow diagram of a process of visualizing
hardware status in accordance with one embodiment.
DETAILED DESCRIPTION
[0021] Embodiments of the invention are not limited in their
application to the details of construction and the arrangement of
components set forth in the following description or illustrated in
the drawings. Embodiments of the invention are capable of other
embodiments and of being practiced or of being carried out in
various ways. Also, the phraseology and terminology used herein is
for the purpose of description and should not be regarded as
limiting. The use of "including," "comprising," or "having,"
"containing," "involving," and variations thereof herein, is meant
to encompass the items listed thereafter and equivalents thereof as
well as additional items.
[0022] Various embodiments of the present invention are directed to
methods and apparatuses for monitoring and/or controlling pieces of
equipment, such as hardware devices in a data center, using a
mobile computing device, such as a smart phone with a built-in
camera. In at least one embodiment, a one- or two-dimensional
barcode is affixed on or near each piece of hardware. The barcode
includes encoded identifying information for the associated
hardware, such as an Internet Protocol (IP) address, serial number,
Media Access Control (MAC) address, hostname, model number,
location, or other identifying information. Using the camera of the
mobile computing device, a user captures an image of the barcode.
The mobile computing device detects the barcode within the image,
decodes the identifying information in the barcode, and sends the
identifying information to a data center management system (e.g.,
wirelessly). In response, the data center management system returns
to the mobile computing device status information for the
corresponding hardware device. The status information is then
displayed on the screen of the mobile computing device, for
example, in an augmented reality scheme where the status is
superimposed over the image as captured by the camera. Thus, a user
can walk from one hardware device to another with, for example, a
smart phone in hand, and retrieve the status of any piece of
equipment simply by capturing images of the barcode for each
device.
[0023] Augmented reality techniques for providing a computer user
or television viewer with an indirect view of a real-world
environment whose elements are augmented by computer-generated
sensor input, such as sound, graphics, or tactile feedback may be
used in some embodiments. In one example, a user of a video-enabled
device can view, either locally or remotely, the environment as
captured by a camera. The video display may include supplemental
text or graphics that relate to one or more objects within the
user's view but do not exist physically in the environment. The
artificial "first down" line that appears in American football
telecasts is one example of augmented reality, since the line as it
appears on television does not actually exist on the football
field.
[0024] Two-dimensional barcodes may be used in some embodiments.
One example of a two-dimensional barcode is the Quick Response (QR)
code developed by Denso Wave Inc. of Chita-gun, Aichi, Japan. QR
codes are an improvement over conventional one-dimensional bar
codes because more data can be encoded in the pattern over a
two-dimensional surface. Among other fields, QR codes are widely
used in industrial management, such as for asset identification and
inventory management. Typically, a fiduciary marker, or fiducial,
having a two-dimensional barcode is photographed by a camera, and
the data in the barcode is then extracted from the image by a
compatible device. Several existing open-source toolkits for
augmented reality and analyzing fiduciary markers, such as
ARToolKit, ARToolKitPlus, and ZXing, may be used in conjunction
with various embodiments. Fiduciary markers are not limited to
two-dimensional barcodes.
[0025] FIG. 1 is a functional block diagram of an exemplary
hardware device management system 100 according to one embodiment.
One or more hardware devices 110 within a data center are
interconnected through a network 120, such as an Ethernet network,
to a data center manager 130. A wireless router 140 is connected to
the network 120. The hardware devices 110 may include, for example,
devices that can be remotely monitored and/or controlled over the
network 120, such as servers, uninterruptible power supplies,
routers, cooling systems, lighting, or other types of devices
typically found in a data center environment. It should be
understood that hardware devices 110 may include virtual devices,
such as hardware emulated by software running on another computer,
or logical groups of devices, such as server farms or disk
arrays.
[0026] In one example, the data center manager 130 is a computer
running software configured to control, monitor, and manage each of
the hardware devices 110. Such software may, for example, perform
one or more of the following monitoring and automation functions:
alarming and notification, control, status, visualization,
configuration, reporting and analytics. One exemplary data center
manager 130 includes InfraStruxure Central sold by American Power
Conversion Corporation of West Kingston, R.I. It should be
understood that the data center manager 130 may perform other
functions, such as data collection/gathering, resource
planning/allocation, and/or implementation (e.g., change tracking,
inventory tracking, dependency analysis, prediction and modeling),
and that other systems may be used to perform one or more of these
functions.
[0027] An optional client 135 communicates with and serves as a
front-end user interface for the data center manager 130. For
example, the client 135 may be a conventional personal computer
(PC) connected to the network 120. The client 135 may run software
(e.g., a web server or custom application) that is configured to
provide a user with a graphical user interface (GUI) that enables
the user to view alarms, warnings, and other messages pertaining
one or more of the hardware devices 110. In another example, the
client 135 may enable the user to control one or more of the
hardware devices 110 through the data center manager 130.
[0028] Although the conventional client 135 is useful, it typically
requires the user to travel to a certain place where the client is
physically installed, thus limiting its usefulness in large or
distributed data center environments where the client 135 may be
far away from some of the hardware devices 110. For example, if the
user of the data center manager 130 wants to check the status of
one of the hardware devices 110, he or she may have to walk to the
hardware device 110, retrieve an identification number from the
device, and then return to the client 135 to search for the status
information using the identification number. In some circumstances,
the hardware devices 110 may not be adequately or properly labeled
with identification information, making it more difficult for the
user to check on such devices using the data center manager 130. In
another example, the client 135 may provide status information for
dozens, if not hundreds, of hardware devices 110, many of which may
be similar in function and/or location. This makes it more
difficult for the user to quickly locate status information for one
hardware device 110 of interest.
[0029] According to one embodiment, a fiduciary marker 112, such as
a one- or two-dimensional barcode, is affixed on or near each
hardware device 110. In one embodiment, the fiduciary marker 112 is
a QR code, although any type of fiduciary marker may be used. The
fiduciary marker 112 is generated in advance (e.g., when the
hardware device 110 is installed or inventoried) and includes
encoded identifying information for the associated hardware, such
as an Internet Protocol (IP) address, serial number, Media Access
Control (MAC) address, hostname, model number, location, or other
identifying information. Each fiduciary marker 112 may be unique so
that no two are the same within the data center and each represents
a different hardware device 110. The fiduciary marker 112 may be
affixed, for example, to the faceplate of the hardware device 110,
on an equipment rack containing the hardware device, or in another
suitable location where the marker can be readily observed by a
person standing near the device. The fiduciary marker 112 does not
need to be affixed directly to the hardware device 110. In some
embodiments, the fiduciary marker 112 may be included on a tag
attached to the hardware device 110 or on mounting hardware near
the device.
[0030] FIG. 2 illustrates a mobile computing device 150 with a
built-in digital camera (not shown) and display 152, such as an
iPhone by Apple Inc. of Cupertino, Calif. According to one
embodiment, a user carrying the mobile computing device 150 may
approach the hardware device 110 of FIG. 1 and capture an image of
the fiduciary marker 112 with the camera. The captured image,
including the fiduciary marker 112 and any other objects in the
field of view of the camera, may be displayed on the display 152.
For example, multiple hardware devices and/or multiple markers may
appear within the captured image. One example image, indicated at
210, is shown in FIG. 2. In addition to the fiduciary marker 112
(in this example, a QR code), the example image 210 includes an
equipment rack 220 housing one hardware device 110. The image may
be, for example, a snapshot, a series of snapshots, or a live video
feed.
[0031] In one embodiment, the mobile computing device 150 includes
software for scanning and decoding the fiduciary marker 112 using
conventional techniques. For QR codes, such software is widely
available for several mobile operating system platforms, including,
for example, iOS by Apple Inc., Blackberry.RTM. OS by Research in
Motion of Waterloo, Ontario, Canada, and the open-source Android
operating system developed in part by Google Inc. of Mountain View,
Calif.
[0032] The scanning/decoding software is configured to detect at
least one fiduciary marker 112 anywhere within the captured image
and to decode identifying information encoded within the marker(s).
The identifying information may contain, for example, an IP address
of the hardware device 110 (e.g., 10.218.10.125). Subsequent to
decoding the identifying information, the identifying information
is sent wirelessly to the data center manager 130 through the
wireless router 140 shown in FIG. 1 and indicated at 160.
Communications between the mobile computing device 150 and the
wireless router 140 may be, for example, performed using a
Wi-Fi.TM., 3G, 4G or other wireless data communications standard.
In another embodiment, the captured image is also sent to the data
center manager 130. The data center manager 130 may maintain a
database of images associated with various pieces of equipment in
the data center. The images may be retrieved from the database to
provide a visual reference for the equipment.
[0033] Once the identifying information is received by the data
center manager 130, status information relating the corresponding
hardware device 110 is returned to the mobile computing device 150
over the wireless connection 160. Referring now to FIG. 3A, in one
embodiment, the status information is displayed on the screen 152,
as indicated at 210. The status 230 may include, for example, an
indication that there is an alarm active for the hardware device
110, identifying information for the hardware device (such as name,
address, model number, and/or version number) or other status
information. In another embodiment, the user may select the status
230 (e.g., by touching it or clicking on it) to display more
detailed information.
[0034] In one embodiment, the mobile computing device 150 includes
client software for interacting with the data center manager 130.
The client software enables, among other things, the user to log
into the data center manager 130 with user credentials (e.g.,
username and password) and authenticate the mobile computing device
150 for security purposes. The client software may optionally
include a simplified interface, as compared to the software running
on the client 135, to accommodate the limited display 152 size and
processing power of the mobile computing device 150. For example,
the interface may be text-only and/or limited to several lines of
information. In particular, the simplified interface may be
configured to display status information for one piece of equipment
at a time. In another example, the interface may include graphics
and/or buttons for enabling the user to view and/or interact with
the interface.
[0035] In another embodiment, as shown in FIG. 3B, the status
information may be superimposed over the captured image. The
exemplary captured image 210 includes the fiduciary marker 112 and
any other objects within the field of view of the camera, such as
one hardware device 110 mounted in the equipment rack 220 with a
fiduciary marker 112 affixed to the front panel.
[0036] The technique described above is a form of augmented
reality, in which the user can physically move within the data
center environment from one hardware device 110 to another, with
camera of the mobile computing device 150 pointed at the respective
fiduciary markers 112. As each fiduciary marker 112 comes into view
of the camera, the status of the corresponding device, indicated at
210 in FIGS. 3A and 3B, is displayed on the screen 152 near the
device itself. Although the status information is not physically
present at or near the hardware device 110, the mobile computing
device 150 enables the user to see the status information through
the mobile computing device as if the status information were
physically present before the user.
[0037] FIG. 4 is a flow diagram of a process 400 of visualizing
equipment status information in accordance with one embodiment.
Process 400 begins at block 402. At block 404, a fiduciary marker
is detected within an image. The image may be captured, for
example, by a camera in a smart phone or other mobile computing
device. For instance, a user of the smart phone may approach a
piece of equipment in a data center environment and take a
photograph or live video of the piece of equipment and/or the
surrounding environment. If a fiduciary marker, such as a
two-dimensional barcode (e.g., a QR code) appears within the image,
the marker is detected by the process 400.
[0038] At block 406, once the fiduciary marker is detected, the
data encoded in the marker is decoded. The data may include
identification information for uniquely identifying the piece of
equipment associated with the fiduciary marker. For example, the
identification information may include an IP address assigned to
the piece of equipment. At block 408, once the data is decoded, the
data is sent to a data center manager or data center management
system. The data may be sent, for example, over a wireless
communication network, a wired communication network, or a
combination of both.
[0039] The data center manager may process the data and return
status information for the piece of equipment to the process 400.
Thus, at block 410, the status information is received from the
data center manager. Once the status information is received, at
block 412, the status information is displayed, for example, on a
display of the smart phone. In one embodiment, the status
information is superimposed upon the image or video. The process
400 may return to block 404 and repeat each time a new image is
captured or when the image within the video changes. For example,
as the user moves about the data center and aims the camera at
different fiduciary markers, the status information for each
respective piece of equipment will be displayed on the smart
phone.
[0040] In another embodiment, the user may use the smart phone to
send commands to the piece of equipment. For example, once the
piece of equipment is identified using the fiduciary marker, the
smart phone may provide a user interface that enables the user to
send a command, such as to reset an alert or to turn a power outlet
on or off, to the respective piece of equipment. The command may
pass through the data center manager, or be sent directly to the
piece of equipment through the communication network.
[0041] Having thus described several aspects of at least one
embodiment of this invention, it is to be appreciated various
alterations, modifications, and improvements will readily occur to
those skilled in the art. Such alterations, modifications, and
improvements are intended to be part of this disclosure, and are
intended to be within the spirit and scope of the invention. For
example, any type of fiduciary marker may be used in place of a
one- or two-dimensional barcode. In another example, a
positioning-based system (such as a Global Positioning System
[GPS]) may be used instead of or in addition to the fiduciary
marker. In this example, the mobile computing device determines its
position within the data center and provides status information for
one or more pieces of equipment that are nearby (e.g., within
several meters). In another example, the mobile computing device
may be configured to perform functions other than reporting
equipment status, such as providing equipment control. Non-limiting
examples of equipment control include resetting the device,
clearing a fault, alert or warning message, setting, changing or
deleting an alert/alarm/warning threshold value, and turning a
power outlet on or off. Accordingly, the foregoing description and
drawings are by way of example only.
* * * * *