U.S. patent application number 09/945776 was filed with the patent office on 2002-04-18 for system for, and method of, situation-relevant asistance to interaction with the aid of augmented-reality technologies.
Invention is credited to Friedrich, Wolfgang, Wohlgemuth, Wolfgang.
Application Number | 20020046368 09/945776 |
Document ID | / |
Family ID | 27576004 |
Filed Date | 2002-04-18 |
United States Patent
Application |
20020046368 |
Kind Code |
A1 |
Friedrich, Wolfgang ; et
al. |
April 18, 2002 |
System for, and method of, situation-relevant asistance to
interaction with the aid of augmented-reality technologies
Abstract
The invention relates to a system for, and method of,
situation-relevant assistance to interaction with the aid of
augmented-reality technologies. To achieve optimized assistance,
especially when setting up a system, commissioning and right up to
the maintenance of systems and processes controlled by means of
automation technology it is proposed that a specific operating
situation be detected automatically, that the operating situation
be analyzed and that information data relevant for the specific
analyzed operating situation be automatically selected from static
information and be displayed.
Inventors: |
Friedrich, Wolfgang;
(Bubenreuth, DE) ; Wohlgemuth, Wolfgang;
(Erlangen, DE) |
Correspondence
Address: |
BAKER BOTTS L.L.P.
30 ROCKEFELLER PLAZA
NEW YORK
NY
10112
US
|
Family ID: |
27576004 |
Appl. No.: |
09/945776 |
Filed: |
September 4, 2001 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
09945776 |
Sep 4, 2001 |
|
|
|
PCT/DE00/00668 |
Mar 2, 2000 |
|
|
|
Current U.S.
Class: |
714/45 |
Current CPC
Class: |
G05B 2219/32014
20130101; G05B 19/4183 20130101; G05B 2219/35482 20130101; G05B
2219/35494 20130101; G05B 19/41875 20130101; G05B 2219/35495
20130101; Y02P 90/02 20151101; G05B 2219/31027 20130101; G05B
19/409 20130101 |
Class at
Publication: |
714/45 |
International
Class: |
G06F 011/32 |
Foreign Application Data
Date |
Code |
Application Number |
Mar 2, 1999 |
DE |
19909154.4 |
Mar 2, 1999 |
DE |
19909023.8 |
Mar 2, 1999 |
DE |
19909018.1 |
Mar 2, 1999 |
DE |
19909012.2 |
Mar 2, 1999 |
DE |
19909011.4 |
Mar 2, 1999 |
DE |
19909010.6 |
Mar 2, 1999 |
DE |
19909013.0 |
Mar 2, 1999 |
DE |
19909009.2 |
Mar 2, 1999 |
DE |
19909016.5 |
Mar 2, 1999 |
DE |
19909154.4 |
Claims
We claim:
1. A system for documentation processing for situation-relevant
assistance to interaction between a user and an engineering
apparatus particularly for engineering and industrial applications,
comprising storage means for storing documentation data and
comprising acquisition means designed as an image recording device
for acquiring real information comprising access means for
accessing the documentation data and comprising analysis means for
evaluating the real information and for selecting the stored
documentation data as a function of the real information.
2. The system according to claim 1, wherein the documentation data
are static and dynamic information data.
3. The system according to claim 1, wherein the analysis means for
evaluating the real information are provided in such a way that a
deployment context, in particular an object of the documentation
data is determined from the real information, and in that the
system includes visualization means for visualizing the
documentation data.
4. The system according to claim 1, wherein the acquisition means
are user-controlled.
5. The system according to claim 3, wherein the visualization means
are designed as display devices disposed in the vicinity of
eyepieces of data goggles, in that the acquisition means provided
is an image acquisition device disposed on the data goggles, and in
that a microphone disposed on the data goggles is provided to
detect speech commands.
6. A method of documentation processing for situation-relevant
assistance to interaction between a user and an engineering
apparatus particularly for engineering and industrial applications,
wherein documentation data are stored and real information is
acquired by means of acquisition means designed as an image
recording device, wherein the documentation data are accessed in
such a way that the real information is analyzed and the stored
documentation data are selected as a function of the real
information.
7. The system according to claim 6, wherein the documentation data
are static and dynamic information data.
8. The system according to claim 6, wherein a deployment context,
in particular an object of the documentation data is determined
from the real information, and in that the documentation data
determined are visualized via visualization means.
9. The system according to claim 6, wherein the acquisition means
are user-controlled.
10. The system according to claim 8, wherein the visualization
means are designed as display devices disposed in the vicinity of
eyepieces of data goggles, in that the acquisition means provided
is an image acquisition device disposed on the data goggles, and in
that a microphone disposed on the data goggles is provided to
detect speech commands.
Description
BACKGROUND OF THE INVENTION
[0001] The invention relates to a system for, and a method of,
documentation processing, particularly for engineering and
industrial applications.
[0002] Such a system and method are used, for example, in the field
of automation technology, for production machinery and machine
tools, in diagnostic/service support systems, and for complex
components, equipment and systems such as, for example, vehicles
and industrial machinery and plants.
[0003] GB 2 327 289 discloses a job aiding apparatus which, by
means of a display, simultaneously visualizes instructions on the
job to be carried out and information on the results of this job
for the benefit of a conveyor belt operator. The job results are
detected by the operator by means of measuring means or by entering
a specific part number, and the job process is recorded by
cameras.
[0004] The object of the invention to specify a system and a method
which, in specific operational situations, permits rapid and
reliable access to relevant documentation data.
[0005] This object is achieved by a system for documentation
processing for situation-relevant assistance to interaction between
a user and an engineering apparatus, particularly for engineering
and industrial applications, comprising storage means for storing
documentation data and comprising acquisition means designed as an
image recording device for acquiring real information, comprising
access means for accessing the documentation data and comprising
analysis means for evaluating the real information and for
selecting the stored documentation data as a function of the real
information.
[0006] This object is achieved by a system for documentation
processing for situation-relevant assistance to interaction between
a user and an engineering apparatus, particularly for engineering
and industrial applications, comprising storage means for storing
documentation data and comprising acquisition means designed as an
image recording device for acquiring real information, comprising
access means for accessing the documentation data and comprising
analysis means for evaluating the real information and for
selecting the stored documentation data as a function of the real
information.
[0007] The documentation data can, for example, be data compiled
and collected while a plant, an automation technology-controlled
system or a process was set up, and/or documentation data
maintained and, when necessary, updated according to predefinable
criteria during operation of a plant or an automation system. These
documentation data can be stored on storage means which are stored
both locally, i.e. at the site of the specific application, or
alternatively at any other site, for example the site of the
respective manufacturers of the individual plant components. With
the aid of the acquisition means, the real information is acquired,
for example from an image content, and is analyzed via the analysis
means, thereby enabling the real objects to be associated with the
object data stored in the documentation data. On the basis of the
real information data, for example in the form of a detected
object, the additional object data contained in the documentation
data are then selected, particularly in an automatic manner, and
are made available in situ, for example for service purposes. This
enables situation-relevant, rapid access to the data specifically
required.
[0008] Advantageous refinements consist in the documentation data
being static and/or dynamic information data. Examples of such
static information include technical data from manuals, exploded
views, maintenance instructions, etc. Examples of dynamic
information include process values such as temperature, pressure,
signals, etc.
[0009] Rapid, situation-relevant access to the documentation data
is further assisted by the feature that the acquisition means
and/or the visualizing means being designed as data goggles.
[0010] The invention is described and explained below in more
detail with reference to the specific embodiments depicted in the
Figures, in which:
[0011] FIG. 1 shows a block diagram of an exemplary embodiment of a
documentation processing system;
[0012] FIG. 2 shows a block diagram of an exemplary embodiment of a
documentation processing system;
[0013] FIG. 3 shows a further block diagram of an exemplary
embodiment of a documentation processing system; and
[0014] FIG. 4 shows a specific application for situation-relevant
access to documentation data.
[0015] FIG. 1 shows a schematic depiction of a system for
documentation processing making use of augmented-reality
techniques. The system consists of analysis means A to which
acquisition means 2 and visualization means B are connected. Via a
data link D, the analysis means A can be coupled to storage means
1. The storage means 1 contain information Il..In as documentation
data. The acquisition means 2 serve for acquiring real information
R of a process 3, for example an automation system or a system or
process controlled by means of automation technology.
[0016] FIG. 1 constitutes the basic structure of a system for a
situation-relevant documentation processing for engineering and
industrial applications. With the aid of the acquisition means 2,
for example a video camera, real information of the engineering and
industrial application 3 is acquired and is analyzed and evaluated
with the aid of the analysis means A, for example the digital image
processing means. Analysis of the real information R with the aid
of the analysis means A is performed, for example, by individual
objects, i.e. individual components of an automation system or an
industrial application, being detected. Said detection can be
effected, for example, in the form of additional information
applied to the real objects, for example in the form of bar code
stickers, or by comparing the image information with reference data
stored in the analysis means A and/or the documentation data 1.
After a desired object corresponding to the real information R has
been found, the documentation data Il..In are displayed to a user,
either automatically and/or under the user's interactive control,
with the aid of the visualization means B. These documentation data
Il..In form additional information for situation-relevant
assistance with the repair, with maintenance etc. Depending on the
real information the documentation data required in each case are
presented in accordance with the given situation. The documentation
data Il..In can, for this purpose, either be stored in situ, for
example on a data processing device, or at remote locations, being
accessed in this case, for example, via an Internet link.
[0017] FIG. 2 shows a schematic depiction of a documentation
processing system for situation-relevant assistance to interaction
between a user and automation apparatuses Al..An. The user, not
explicitly shown in FIG. 2, is equipped with mobile equipment 4, 6.
The mobile equipment 4, 6 includes data goggles 4 fitted with a
video camera 2 and a microphone 11. The data goggles are linked to
a device for communication without the use of wires, for example a
radio transceiver 6, which can communicate with the automation
system Al..An via a radio interface 15. The automation system
Al..An can be linked, via a data link 14, to an augmented-reality
system 10, hereinafter also abbreviated as AR system. The AR system
includes an information module 1b for storing or accessing
information data, an AR base module 8 and an AR application module
9. The AR system 10 can be linked to the Internet 5 via a data link
13, with optional access to further storage data and documentation
data la via an Internet link 12 shown by way of example.
[0018] The user who is equipped with the data goggles 4 and the
mobile radio transceiver 7 is able to move freely within the plant
Al..An for maintenance and service purposes. For example, if
maintenance of, or repair to, a particular subcomponent of plants
Al..An has to be carried out, appropriate access to the relevant
documentation data 1a, 1b is established with the aid of the camera
2 of the data goggles 4, optionally controlled by speech commands
detected by the microphone 11. To do this, a data link to plant
Al..An or with an appropriate radio transceiver unit is set up via
the radio interface 15, and the data transmitted to the AR system
10. Within the AR system, the data obtained from the user are
analyzed in accordance with the situation, and information data 1a,
1b are accessed automatically or in a manner controlled
interactively by the user. The relevant documentation data 1a, 1b
obtained are transmitted via the data links 14, 15 to the
transceiver 6, with the overall result that an analysis is carried
out on the basis of the operational situation detected, said
analysis forming the basis for the selection of data from the
available static information. This results in a situationally
appropriate, object-oriented or component-oriented selection of
relevant knowledge from the most up-to-date data sources 1a, 1b.
Information is displayed with the aid of the visualization
component used in each case, for example a handheld PC or data
goggles. AR-based technologies are referred to. The operator in
situ is therefore provided only with the information he needs. This
information is always up-to-date. The service technician therefore
does not suffer from information overload from a "100-page manual",
for example.
[0019] FIG. 3 shows a further specific application of a
documentation processing system for service and maintenance. The
system consists of an augmented-reality system 10 which comprises
an information module 1b for storing information data, an AR base
system 8 and an AR application module 9. The AR system 10 can be
linked to the Internet 5 via connecting lines 13, 18. Thence a link
is possible, via an exemplary data link 12, to a remote PC 16 with
a remote expert 22. Linkage between the individual modules of the
AR system 10 is effected via links 19, 20, 21. The user
communication between a user 7 and the AR system is effected via
interfaces 8, 23. To this end, the AR system can be linked to a
transceiver which enables bidirectional data communication between
the AR system 10 and the user 7 via data goggles 4, either directly
via the interface 8 or via a radio transceiver 17, located in the
vicinity of the user 7, via an interface 23. The link 23 can be
implemented via a separate data link or via the mains as a
"power-line" modem. As well as a display device disposed in the
vicinity of the eye pieces, the data goggles 4 comprise an image
acquisition device 2 in the form of a camera and a microphone 11.
With the aid of the data goggles 4, the user 7 can move round the
plants Al..An and carry out service or maintenance activities.
[0020] With the aid of the data goggles 4 and the corresponding
radio transceivers, e.g. the radio transceiver 17 worn by personnel
directly on the body, it is possible to achieve preventive
functionality: the initial step is the detection of the respective
operational situation, for example by the camera 2 or via
localization by the personnel 7. On the basis of the operational
situation detected, a selection of data from the plant Al..An
undergoing maintenance is made in the AR system. The fundamental
advantage of the system depicted in FIG. 3 is that this system
assists the cooperation of the individual single functionalities in
an application-relevant manner: i.e. firstly a specific operational
situation is detected automatically, and this operational situation
is then analyzed, the aspects relevant at that point being
determined automatically from the most up-to-date available static
information in conjunction with the dynamic data acquired
instantaneously. As a result, for example, assembly suggestions are
correlated with current process data. As a result, personnel 7 are
provided with a situationally appropriate display of the relevant
information, for example by a superposed visualization of the
respective data in such a way that the real operational situation
in the field of view of the personnel is expanded by the
information acquired. As a result, personnel 7 are very rapidly put
in the position of being able to act, thereby ensuring the
requisite machine operating times.
[0021] Assistance to the maintenance technician 7 in situ can also
be provided via the remote expert 22 and the knowledge 16 available
at the location of the remote expert 22.
[0022] FIG. 4 shows a specific application of situation-relevant
access to documentation data. FIG. 4 shows a first monitor region
B1 which shows a plant component. Shown in the right-hand monitor
region B2 is a user 7 who, for example, is looking at an individual
plant component. The user 7 is equipped with data goggles 4 which
comprise a camera 2 as an acquisition means. Additionally disposed
on the data goggles 4 are a microphone 11 and a loudspeaker 16. The
left-hand monitor region B1 shows a view of conduits which can be
viewed with the data goggles shown in window B2. Marked in the
left-hand monitor region B1 are two points P1, P2 which each
represent two image details viewed with the aid of the data goggles
4. After the first point P1 has been viewed, i.e. after the conduit
disposed at or near point P1 has been viewed, additional
information is visualized for the user 7 in the data goggles 4.
This additional information 11 consists of documentation data
which, regarding the first point P1, include operational
instructions for this pipe section and, regarding point P2,
comprise the installation instruction to be implemented in a second
step. The installation instruction in this case consists of the
user 7 being informed of the torque and the sense of rotation of
the screwed joint of point P2 via visualization of the additional
data 112. The user 7 is therefore very quickly provided with
situationally appropriate instructions for the object being viewed.
If an intelligent tool is used which is able to detect the torque
applied at any given moment, it is also possible for the user to be
told, on the basis of the current torque, to increase or reduce the
torque as required.
[0023] Below, background information is provided to the field of
application of the invention: this involves an application-oriented
requirement analysis and development of AR-based systems to support
operational processes being developed, production and service of
complex engineering products and plants in fabrication and process
technology, and for service support systems as with motor vehicles,
or for maintaining any industrial equipment.
[0024] Augmented reality, AR in brief, is a novel type of
man-machine interaction of major potential for supporting
industrial operational processes. With this technology, the field
of view of the observer is enriched with computer-generated virtual
objects, which means that intuitive use can be made of product or
process information. In addition to the extremely simple
interaction, the deployment of portable computers opens up AR
application fields involving high mobility requirements, for
example if process, measured or simulation data are linked to the
real object.
[0025] The situation of German industry is characterized by
increasing customer requirements in terms of individuality and
quality of products and by the development processes taking
substantially less time. Especially in developing, producing and
servicing complex industrial products and plants it is possible, by
means of innovative solutions to man-machine interaction, both to
achieve jumps in efficiency and productivity and to design the work
so as to enhance competence and training, by the users' need for
knowledge and information being supported in a situationally
appropriate manner on the basis of data available in any case.
[0026] Augmented reality is a technology with numerous innovative
fields of application:
[0027] In development, for example, a "mixed mock-up" approach
based on a mixed-virtual environment can result in a distinct
acceleration of the early phases of development. Compared with
immersive "virtual reality" (VR) solutions, the user is at a
substantial advantage in that the haptic properties can be depicted
faithfully with the aid of a real model, whereas aspects of visual
perception, e.g. for display variants, can be manipulated in a
virtual manner. In addition, there is a major potential for
user-oriented validation of computer-assisted models, e.g. for
component verification or in crash tests.
[0028] In flexible production it is possible, inter alia, to
considerably facilitate the process of setting up machinery for
qualified skilled operators by displaying, e.g. via mobile AR
components, mixed-virtual clamping situations directly in the field
of view. Fabrication planning and fabrication control appropriate
to the skilled worker in the workshop is facilitated if information
regarding the respective order status is perceived directly in situ
in connection with the corresponding products. This also applies to
fitting, with the option of presenting the individual procedural
steps to the fitter in a mixed-virtual manner even in the training
phase. In this connection it is possible, e.g. by comparing real
fitting procedures with results of simulations, to achieve
comprehensive optimizations which both improve the quality of
operation scheduling and simplify and accelerate the fitting
process in the critical start-up phase.
[0029] Finally, regarding service, conventional technologies are by
now barely adequate for supporting and documenting the complex
diagnostic and repair procedures. Since, however, these processes
in many fields are in any case planned on the basis of digital
data, AR technologies provide the option of adopting the
information sources for maintenance purposes and of explaining the
dismantling process to an engineer, e.g. in the data goggles, via
the superposition with real objects. Regarding cooperative
operation, the AR-assisted "remote eye" permits a distributed
problem solution by virtue of a remote expert communicating across
global distances with the member of staff in situ. This case is
particularly relevant for the predominantly medium-sized machine
tool manufacturers. Because of globalization, they are forced to
set up production sites for their customers worldwide. Neither,
however, is the presence of subsidiaries in all the important
markets achievable on economic grounds, nor is it possible to
dispense with the profound knowledge of experienced service staff
of the parent company with respect to the increasingly more complex
plants.
[0030] The special feature of man-machine interaction in augmented
reality is the very simple and intuitive communication with the
computer, supplemented, for example, by multimode interaction
techniques such as speech processing or gesture recognition. The
use of portable computer units in addition enables entirely novel
mobile utilization scenarios, with the option of requesting the
specific data at any time via a wireless network. Novel
visualization techniques permit direct annotation, e.g. of measured
data or simulation data, to the real object or into the real
environment. In conjunction with distributed applications, a number
of users are able to operate in a real environment with the aid of
a shared database (shared augmented environments) or to cooperate
with AR support in different environments.
[0031] Augmented reality has been the subject of intense research
only in the last few years. Consequently, only a few applications
exist, either on the national or the international level, usually
in the form of scientific prototypes in research
establishments.
[0032] U.S.A.: As with many novel technologies, the potential uses
of augmented reality were first tapped in North America. Examples
include cockpit design or maintenance of mechatronic equipment. The
aircraft manufacturer Boeing has already carried out initial field
trials using AR technology in the assembly field. The upshot is
that in this hi-tech area too the U.S.A. occupy a key position,
potentially making them technological leaders.
[0033] Japan: Various AR developments are being pushed in Japan,
e.g. for mixed-virtual building design, telepresence or
"cyber-shopping". The nucleus is formed by the Mixed Reality
Systems Laboratory founded in 1997, which is supported jointly as a
center of competence by science and by commerce and industry.
Particular stimuli in the consumer goods field are likely in the
future from the Japanese home electronics industry.
[0034] Europe: So far, only very few research groups have been
active in Europe in the AR field. One group at the University of
Vienna is working on approaches to mixed-real visualization. The
IGD group, as part of the ACTS project CICC, which has now come to
an end, has developed initial applications for the building
industry and a scientific prototype for staff training in car
manufacturing.
[0035] The invention in particular should be seen in the specific
context of the fields of application "production machinery and
machine tools" (NC-controlled, automation-technology processes) and
"diagnostics/service support systems for complex engineering
components/equipment/systems" (e.g. vehicles, but also industrial
machinery and plants).
[0036] To sum up, the invention therefore relates to a system for,
and method of, situation-relevant assistance to interaction with
the aid of augmented-reality technologies. To achieve optimized
assistance, especially with setting up a system, commissioning and
right up to the maintenance of systems and processes controlled by
means of automation technology it is proposed that a specific
operating situation be detected automatically, that the operating
situation be analyzed and that information data relevant for the
specific analyzed operating situation be automatically selected
from static information and be displayed.
* * * * *