U.S. patent application number 12/555821 was filed with the patent office on 2010-02-04 for automated application interaction using a virtual operator.
Invention is credited to Amir Geva, Eugeniusz Walach.
Application Number | 20100027896 12/555821 |
Document ID | / |
Family ID | 41608433 |
Filed Date | 2010-02-04 |
United States Patent
Application |
20100027896 |
Kind Code |
A1 |
Geva; Amir ; et al. |
February 4, 2010 |
AUTOMATED APPLICATION INTERACTION USING A VIRTUAL OPERATOR
Abstract
A computer-implemented method for automating interaction with a
computer system includes linking a control computer system to an
input interface and to an output interface of a client computer
system, which is operative for producing user interface images on a
display device. The control computer system executes distinct
software modules that include a virtual operator for simulating
actions of a human operator. Execution of the software modules
causes the control computer system to capture an image from the
output interface, and to recognize information in the image. In
response to the information, the virtual operator controls an input
device to automatically execute predetermined operations on the
client computer system via the input interface.
Inventors: |
Geva; Amir; (Yokne'am llit,
IL) ; Walach; Eugeniusz; (Haifa, IL) |
Correspondence
Address: |
IBM CORPORATION, T.J. WATSON RESEARCH CENTER
P.O. BOX 218
YORKTOWN HEIGHTS
NY
10598
US
|
Family ID: |
41608433 |
Appl. No.: |
12/555821 |
Filed: |
September 9, 2009 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
11426949 |
Jun 28, 2006 |
|
|
|
12555821 |
|
|
|
|
Current U.S.
Class: |
382/229 ;
345/156; 358/474; 715/700 |
Current CPC
Class: |
G06K 9/72 20130101; G06Q
10/10 20130101; G06F 21/52 20130101; G06F 3/0481 20130101; G06K
9/00442 20130101; G06F 11/3414 20130101; G06F 21/36 20130101 |
Class at
Publication: |
382/229 ;
715/700; 358/474; 345/156 |
International
Class: |
G06F 3/01 20060101
G06F003/01; G06K 9/72 20060101 G06K009/72 |
Claims
1. A computer-implemented method for automating interaction with a
computer system comprising: linking a control computer system to an
input interface of a client computer system and to an output
interface of the client computer system operative for producing
user interface images on a display device; and executing with the
control computer system distinct software modules that include a
virtual operator for simulating actions of a human operator,
wherein execution of the software modules causes the control
computer system to perform the steps of: capturing an image from
the output interface; recognizing information in the image; and
responsively to the information, controlling an input device by the
virtual operator to thereby automatically execute predetermined
operations on the client computer system via the input
interface.
2. The method according to claim 1, wherein capturing the image
from the output interface comprises receiving the image from an
image scanner.
3. The method according to claim 1, wherein recognizing information
in the image comprises verifying that the recognized information
matches a pattern that is known to the virtual operator.
4. The method according to claim 1, wherein the predetermined
operations comprise entering data.
5. The method according to claim 1, wherein recognizing information
comprises optical character recognition.
6. The method according to claim 1, wherein the client computer
system comprises a plurality of client computers, and the virtual
operator comprises a plurality of instances of the virtual
operator, the method comprising coordinating the plurality of
instances of the virtual operator to concurrently control the
plurality of client computers.
7. The method according to claim 1, wherein recognizing information
in the image comprises providing a verification image comprising
the recognized information to the human operator.
8. The method according to claim 1, wherein the predetermined
operations comprise executing data input scripts comprising
simulated device events.
9. A computer software product for automating interaction with a
computer system, comprising a computer storage medium in which
computer program instructions are stored, wherein the instructions
comprise distinct modules that include a virtual operator for
simulating actions of a human operator, which instructions, when
executed by a control computer linked to an input interface of a
client computer system and to an output interface of the client
computer system operative for producing user interface images on a
display device, cause the control computer to perform the steps of:
capturing an image from the output interface; recognizing
information in the image; and responsively to the information,
controlling an input device by the virtual operator to thereby
automatically execute predetermined operations on the client
computer system via the input interface.
10. The computer software product according to claim 9, wherein
capturing the image from the output interface comprises receiving
the image from an image scanner.
11. The computer software product according to claim 9, wherein
recognizing information in the image comprises verifying that the
recognized information matches a pattern that is known to the
virtual operator.
12. The computer software product according to claim 9, wherein the
predetermined operations comprise entering data.
13. The computer software product according to claim 9, wherein
recognizing information comprises optical character
recognition.
14. The computer software product according to claim 9, wherein the
client computer system comprises a plurality of client computers,
and the virtual operator comprises a plurality of instances of the
virtual operator, the method comprising coordinating the plurality
of instances of the virtual operator to concurrently control the
plurality of client computers.
15. The computer software product according to claim 9, wherein
recognizing information in the image comprises providing a
verification image comprising the recognized information to the
human operator.
16. The computer software product according to claim 9, wherein the
predetermined operations comprise executing data input scripts
comprising simulated device events.
17. A data processing system automating interaction with a computer
system, comprising: a client computer system operative for
producing user interface images on a display device; a control
computer system linked to an input interface and to an output
interface of the client computer system; a processor; and a memory
accessible to the processor storing programs and data objects
therein, wherein execution of the programs causes the processor to
execute with the control computer system distinct software modules
that include a virtual operator for simulating actions of a human
operator, and to perform the steps of: capturing an image from the
output interface; recognizing information in the image; and
responsively to the information, controlling an input device by the
virtual operator to thereby automatically execute predetermined
operations on the client computer system via the input
interface.
18. The data processing system according to claim 17, wherein
capturing the image from the output interface comprises receiving
the image from an image scanner.
19. The data processing system according to claim 17, wherein
recognizing information in the image comprises verifying that the
recognized information matches a pattern that is known to the
virtual operator.
20. The data processing system according to claim 17, wherein the
predetermined operations comprise entering data.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation-in-part of application
Ser. No. 11/426,949, titled "SYSTEM, METHOD AND COMPUTER PROGRAM
PRODUCT FOR PERFORMING INFORMATION TRANSFER USING A VIRTUAL
OPERATOR," filed Jun. 28, 2006, which is incorporated herein by
reference.
FIELD OF THE INVENTION
[0002] The present invention relates generally to software
solutions and specifically to providing automatic interaction among
business software applications using a virtual operator.
BACKGROUND OF THE INVENTION
[0003] Modern organizations typically employ interactive business
software solutions, including multiple operative products that may
support diverse business units. Such products typically include
software applications to provide support for business requirements,
e.g., financial operations, customer relationship management, human
resources, professional services, purchasing, and distribution.
[0004] The software applications may run on computerized systems,
which can include legacy as well as modern systems. Often, at least
a portion of the software and hardware of the computerized systems
are tailored to support customized customer-specific functionality.
Customizations may be applied at the time a new application is
obtained, or as business needs change. Techniques utilized to
enable a given system to be customized range from source code
customization by professional services experts to interactive tool
based approaches that allow end customers to make some changes
themselves. At the end of the day, regardless of whether the
software applications are "out-of-the-box", customized, legacy or
modern, they all need to be interactive to allow the data in each
system to be accessible, shareable, and consistent.
BRIEF SUMMARY
[0005] An embodiment of the present invention provides a
computer-implemented method for automating interaction with a
computer system, which is carried out by linking a control computer
system to an input interface of a client computer system and to an
output interface of the client computer system, which is operative
for producing user interface images on a display device. Distinct
software modules are executed with the control computer system,
including a virtual operator for simulating actions of a human
operator. Execution of the software modules causes the control
computer system to capture an image from the output interface and
to recognize information in the image. In response to the
information, the virtual operator controls an input device to
automatically execute predetermined operations on the client
computer system via the input interface.
[0006] Other embodiments of the invention provide computer software
product and apparatus for carrying out the above-described method.
Still other embodiments of the invention provide techniques for
configuring a computer software product for carrying out the
above-described method cooperatively with computer apparatus.
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
[0007] For a better understanding of the present invention,
reference is made to the detailed description of the invention, by
way of example, which is to be read in conjunction with the
following drawings, wherein like elements are given like reference
numerals, and wherein:
[0008] FIG. 1 is a block diagram of a system for automating
interaction with a computer system, in accordance with a disclosed
embodiment of the invention;
[0009] FIG. 2 shows a user interface screen, which displays a user
interface image, in accordance with a disclosed embodiment of the
invention;
[0010] FIG. 3 is a flow chart of a method for automating
interaction with a computer system, in accordance with a disclosed
embodiment of the invention;
[0011] FIG. 4 is a block diagram of a system for automating
interaction with a computer system, in accordance with alternate
embodiments of the invention; and
[0012] FIG. 5 illustrates scanned and reference images produced by
the embodiment of FIG. 4.
DETAILED DESCRIPTION OF EMBODIMENTS
Overview
[0013] Embodiments of the present invention that are described
hereinbelow provide improved methods, products and systems for
automating interaction with a computer system. When integrating
multiple modern software systems with legacy software systems, for
example, it is a primary goal to maximize the efficiency of the
integration by minimizing the effort necessary for data to flow
among the systems.
[0014] In the past, approaches to the issue have included manual
data updates and queries to the legacy software systems, wherein
human operators have prepared data in the required format, and have
manually keyed the prepared data into the legacy software systems.
This approach has proven to be costly in terms of resources,
suffers from poor throughput, and is typically inaccurate. Another
historical approach has been to replace the legacy software systems
with new computerized systems designed to accurately integrate the
data flow while providing the necessary functionality. This
approach has proven to be quite expensive, wherein the replacement
systems take significant resources to develop, often fail to meet
the objectives of the design, and are typically provided later than
scheduled.
[0015] Yet another approach from the past has been to develop
automated processes to convert data off-line, e.g., the processes
are scheduled to operate during non-production hours, changing data
in a modern format to a format acceptable to the legacy software
systems. One drawback of this historical approach includes a lack
of direct connectivity among the modern and legacy software
systems. For example, verification that a client address stored in
a legacy software system is accurate can take an unacceptably long
time. This approach has also proven to be extremely expensive from
a strategic point of view, as the conversion processes are
typically "hard-wired", or precisely tailored to match the data
structures of the legacy software systems. Constant updates to the
conversion processes are required as the modern software systems
evolve.
[0016] Embodiments of the present invention provide a virtual
operator that emulates actions performed by a human operator using
legacy software. The virtual operator interacts with the legacy
software via an automated software system interfacing between
modern and legacy software systems, or simply among any number of
non-integrated software systems. In the context of the present
patent application and claims, the term "virtual operator" refers
to any software application or process that is capable of
automatically performing actions on computer systems by interaction
with a user interface, thus simulating identical actions performed
by a human operator. In the context of the present patent
application and claims, the term "legacy software system" refers to
any software application that executes on a computer system and
that continues to be used in spite of available newer technology,
typically since the software application still meets the
requirements of at least some of the users of the software
application.
[0017] In embodiments of the invention, a control computer system
is linked to input and output interfaces of a client computer
system that executes legacy computer software. The client computer
system produces user interface images on a display device,
typically in response to data entered via the input interface,
e.g., using a keyboard or mouse. A virtual operator running on the
control computer system simulates actions of a human operator, as
described hereinbelow. The virtual operator captures one of the
images from the output interface, and recognizes information in the
image. Typically, the virtual operator uses an image analyzer to
compare the image with a collection of image templates. In one
example, the image analyzer uses optical character recognition
(OCR) to analyze the image. In another example, complex processes
such as creating hierarchical relationships among areas of the
screen may be used in order to apply disambiguation via semantic
entity identification.
[0018] In some embodiments, the virtual operator verifies that the
recognized information matches a known pattern. An image verifier
is typically used to corroborate the match by extracting some type
of marker from the image, such as text indicative of a screen
function. For example, an image template may be used to make
comparisons using the screen title text to identify the image. In
another example, the image verifier may confirm that the currently
displayed image on the display device of the client computer system
is an address change screen.
[0019] The virtual operator controls the input device,
automatically executing predetermined operations on the client
computer system. The input device is typically used to input entry
data comprising simulated keystrokes, mouse clicks, and other
operator actions according to an input script, as described
hereinbelow. In the present example, entry data is input to the
input interface of the client computer system via the input device
to change a customer address. Alternatively, predetermined
operations may be executed to simulate other human operator
actions, to query legacy software systems, or to insert and update
data, as required. Embodiments of the present invention provide a
single reliable, high-speed interface among multiple software
systems, such as between a modern software system and a legacy
software system.
System Description
[0020] Reference is now made to FIG. 1, which is a block diagram
that schematically illustrates a system 10 for automating
interaction with a computer system, in accordance with a disclosed
embodiment of the invention. System 10 typically comprises a
control computer system 12 and at least one client computer system
14. Control computer system 12 may interface with multiple client
computer systems, which are not shown for the sake of simplicity.
Although portions of system 10 shown in FIG. 1 and other drawing
figures herein are shown as comprising a number of separate
functional blocks, these blocks are not necessarily separate
physical entities, but rather may represent, for example, different
computing tasks or data objects stored in a memory that is
accessible to a processor of control computer system 12.
Embodiments of the present invention enable control computer system
12 to automatically interact with the multiple client computer
systems irrespective of their hardware and software configurations,
as described hereinbelow. Control computer system 12 typically
comprises a general-purpose control computer 16, which is
programmed in software to carry out the functions that are
described herein. Control computer system 12 also comprises an
input device 18 and a storage unit 20, as described
hereinbelow.
[0021] Control computer 16 comprises a processor 22, a capture
interface 24, and a memory 26. Capture interface 24 may comprise a
Bluetooth.RTM. adapter, an Infrared Data Association (IrDA) device,
a cable connection, or any communication interface for capturing
image data that allows control computer 16 to receive output from a
screen, e.g., in the form of an image bitmap. A virtual operator
28, an image analyzer module 30, and an image verifier module 32
are held in memory 26. Virtual operator 28, image analyzer module
30, and image verifier module 32 may be downloaded to control
computer 16 in electronic form, over a network, for example, or
they may alternatively be provided on tangible media, such as
optical, magnetic or electronic memory media. Further
alternatively, at least some of the functions of control computer
16 may be carried out by dedicated electronic logic circuits.
Although the embodiment relates to one particular system for
automating interaction with a computer system, the principles of
automated interaction that are implemented in system 10 may
similarly be applied, mutatis mutandis, in other types of automated
computer interaction systems using other techniques for
automatically inputting entry data to client computer systems.
[0022] Input device 18 may comprise any suitable device or device
simulator that allows control computer 16 to produce a sequence of
device signals such as keyboard keystrokes or mouse clicks for
input to a communications interface, which is typically connected
to another computer system. One example of a suitable input
simulator is the Microsoft Windows.TM. SendInput function for the
Windows user interface, available at the Microsoft Developer
Network website (msdn.microsoft.com). The SendInput function
provides a tool to support simulation of keystrokes, mouse motions,
and button clicks. Storage unit 20 may be any storage device that
is capable of storing images captured by capture interface 24,
image templates, and any other data stored for retrieval by virtual
operator 28.
[0023] Client computer system 14 typically comprises a
general-purpose client computer 34, which is programmed in software
to carry out the functions that are described herein. Client
computer system 14 also comprises a screen 36, as described herein.
Client computer 34 comprises an input interface 38, which may
comprise any suitable communications interface that receives input
signals representative of actions performed using input devices
such as keyboards and computer mice. Input device 18 may provide
simulated input, and as described hereinbelow, software
applications may be used to enable remote control of client
computer system 14 over a network. Thus embodiments of the present
invention provide interoperability between client computer system
14 and input device 18 without necessitating any modifications to
input interface 38. Screen 36 comprises a display device and an
output interface 40, which may comprise a Bluetooth.RTM. adapter,
an Infrared Data Association (IrDA) device, a cable connection, or
any communication interface for outputting image data that allows
client computer 34 to export visual display data, e.g., in the form
of a compressed image.
[0024] System 10 may include devices 42 for producing a software
product on a computer-readable medium, for example a CD or DVD
writer. Devices employing many other technologies are also suitable
for use as devices 42, for example, writers adapted to tapes, hard
disc drives, RAM, ROM, flash memory devices, floppy drives,
programmable read-only memory (PROM), erasable programmable
read-only memory (EPROM), and magneto-optical discs. In all of
these technologies patterns or transitions in the physical state of
the medium are formed therein. These patterns and sequences of
transitions encode instructions or data, e.g., binary numbers that
are acceptable to the processor. The nature of the state
transitions varies with the particular medium, and includes changes
in magnetic domains in magnetic media, formation of pits or bumps
on an optical disk, etc. Once the transitions are established, the
medium may be transferred to another computing device 44, which is
capable of reading the medium, recognizing the transitions, and
decoding them to identify executable instructions or data.
[0025] In an alternative embodiment of the present invention,
control computer system 12 and client computer system 14 may run on
a single computer system 106 as separate processes. Communication
between the systems may comprise events passed between the separate
processes via any suitable application programming interface known
in the art. The present embodiment provides for simplified image
capture and input signal generation due to the processes running on
single computer system 106, while eliminating the need to
physically link multiple computer systems.
[0026] Reference is now made to FIG. 2, which is a schematic,
pictorial illustration of a user interface screen, which displays
user interface images, in accordance with a disclosed embodiment of
the invention. Client computer 34 (FIG. 1) presents a user
interface image 46 having application data fields 48 on screen 36.
For the purposes of illustration, image 46 is described using an
example of an action performed on a software application to
determine policy holder data in association with an insurance
policy maintained in a legacy software system. Image 46 comprises a
screen function label 50, a policy number field 52, a policy holder
name field 54 and data fields 48.
[0027] Client computer system 14 (FIG. 1) is adapted to execute at
least one software application. The software application may be
responsive to input operations, e.g., by signals sent to input
interface 38, as described in more detail hereinbelow. In the
present example, the software application provides storage and
maintenance functionality for policy holder data for an insurance
firm. Typically, the software application generates images such as
image 46 that are then displayed on screen 36 in response to the
actions. For example, a logon session can involve performing
multiple actions in the predetermined order required to enter login
and password information.
[0028] The software application may require additional input
operations that cause client computer 34 (FIG. 1) to display a
different image on screen 36. Virtual operator 28 typically
captures the image from screen 36 or from output interface 40,
using any image capture method known in the art. Virtual operator
28 uses image analyzer module 30 to recognize information in the
image. A pseudocode listing of a sample image analysis that may be
used in the context of the present example, whereby the software
application is running on a mainframe computer, is presented
hereinbelow in Listing 1. Function find_nearest in Listing 1 tries
to locate each application data field based upon relative
coordinates of the data field, for example, policy number field
52.
[0029] Image verifier module 32 (FIG. 1) is used to confirm that
the recognized information matches a pattern that is known to
virtual operator 28, as described hereinabove. A pseudocode listing
of a sample screen recognition routine that may be used in this
context is presented hereinbelow in Listing 2. In the present
example, the image contains a marker describing the function of the
image, in the text of screen function label 50. A pseudocode
listing of a sample screen verification routine that may be used in
this context is presented hereinbelow in Listing 3. In the present
example, screen function label 50 is verified to match expected
screen function label values.
[0030] Typically, the software application provides some images
that allow information entry, whereas other images are displayed
that only allow information to be viewed. Some of the images
provided by the software application may indicate to virtual
operator 28 (FIG. 1) that information entry has been completed. The
input operations required for executing actions for each software
application are known in advance, in the form of the above-noted
input script, which is typically stored in storage unit 20.
Automatic Application Interaction
[0031] Reference is now made to FIG. 3, which is a flow chart that
schematically illustrates a method for automating interaction with
a computer system, in accordance with a disclosed embodiment of the
invention. Shown by way of example, the name of a policy holder at
an insurance firm may need to be changed due to an update in the
marital status of the policy holder. Virtual operation 28 would
therefore need to simulate several actions of a human operator in
order for to automatically interact with a legacy software
application running on client computer system 14 to successfully
negotiate the requirement.
[0032] Control computer system 12 (FIG. 1) is linked to client
computer system 14 in a computer system linking step 56. Typically,
input device 18 is linked to input interface 38, and output
interface 40 is linked to capture interface 24.
[0033] Control computer system 12 (FIG. 1) executes virtual
operation 28 to simulate actions of a human operator, thus
automatically interacting with the software application running on
client computer system 14, in a virtual operator executing step 58.
Virtual operation 28 (FIG. 1) causes processor 22 to perform the
steps listed hereinbelow.
[0034] Capture interface 24 (FIG. 1) captures an image, e.g., image
46 displayed on screen 36 (FIG. 2) in an image capturing step 60.
Additionally or alternatively, image 46 may be captured by capture
interface 24 via output interface 40. In the current example,
output interface 40 is a universal serial bus (USB), and image 46
is captured from the software application.
[0035] As described hereinabove, image analyzer module 30 may be
invoked by virtual operation 28 (FIG. 1) to analyze image 46. Image
analyzer module 30 recognizes information in image 46 in an
information recognition step 62, as described hereinabove in
reference to the pseudocode listing presented in Listing 1. A
simple text layout is easily recognized by image analyzer module
30, while comparing image 46 to image templates stored in storage
unit 20. The example provided by image 46 (FIG. 2), taken from a
sample mainframe computer application, may be analyzed with ease by
image analyzer module 30. However, more complex analyses may be
required if geometric primitives, e.g., rectangles or lines, are
utilized by the software application running on client computer
system 14, as described hereinabove.
[0036] Virtual operation 28 (FIG. 1) typically uses image verifier
module 32 to verify that the recognized information matches a known
pattern in a recognized information pattern matching verification
step 64. In the current example, image verifier module 32 verifies
that the marker describing the function of image 46 matches the
image template used by image analyzer module 30 in information
recognition step 62. Screen function label 50 (FIG. 2), the marker
in the present example, has the value "INQ100", and is verified
against the image template by image verifier module 32, thus
confirming the identification of image 46 as matching the known
pattern of the address change screen of the software application
delineated by the image template.
[0037] Virtual operation 28 responds to the pattern verification by
image verifier module 32, by controlling input device 18 to
automatically execute predetermined operations on client computer
system 14 in a device controlling and operation executing step 66.
These operations may be specified in an input script. A library of
such input scripts may be stored in storage unit 20. A pseudocode
listing of routines to generate data input that may be used in this
context is presented hereinbelow in Listing 4. Virtual operation 28
causes input device 18 to automatically execute predetermined
operations, e.g., inputting entry data to client computer system
14, thereby simulating actions of the human operator. In an
embodiment of the present invention, virtual operation 28 uses
RealVNC.COPYRGT. server and client software applications, available
from RealVNC Ltd, Betjeman House, 104 Hills Road, Cambridge.
Cambridgeshire, CB2 1LQ. UK, to control client computer system 14
over a network. The RealVNC applications utilize remote framebuffer
(also known as Remote Frame Buffer or RFB) functionality, a
protocol which provides remote access to graphical user interfaces
at the framebuffer level, and is available from the RealVNC web
site (realvnc.com). Thus, the RealVNC applications provide the
ability to send the output of input device 18 to input interface 38
and to place device events into the input event queue of the
operating system of client computer system 14. The device events
are then processed on client computer system 14 as though a
physical input device had been used to generate the events.
[0038] Each input script provides entry data for performing actions
that have been precisely tailored to execute operations for a
specific software application on a particular computer system via
input device 18. In the present example, the entry data comprises
keystrokes and mouse clicks input by input device 18 to execute the
operations delineated by an input script. The input script provides
the entry data that is necessary when using the software
application running on client computer system 14 in order to
automatically select policy holder name field 54 and to
automatically revise the policy holder name of the example.
[0039] The method then terminates at a final step 68.
[0040] In tests performed utilizing an embodiment of FIG. 1 of the
above-described automated method, improved results were typically
observed. The tests were performed using a single general-purpose
computer running the Microsoft Windows operating system, wherein
both control computer system 12 and client computer system 14 were
implemented as processes running on the computer. In an exemplary
test, a human operator interacted with multiple screens provided by
a software application used for performance of a beneficiary change
process in an insurance policy management system. Some application
screens needed to be reviewed to verify information visually, and
other application screens were used to apply changes utilizing the
verified information. The human operator typically required
approximately two minutes to complete a specific delineated set of
actions. Application of the automated method resulted in identical
results in an average of ten seconds, for a typical improvement in
efficiency of approximately an order of magnitude.
Alternative Embodiments
[0041] Reference is now made to FIG. 4, which is a block diagram of
an alternative system 70 for automating interaction with a computer
system, in accordance with alternate embodiments of the invention.
Apart from the differences described below, the operation of
alternative system 70 is generally similar to that of system 10
(FIG. 1). As mentioned hereinabove, virtual operation 28 may
interface with multiple client computer systems to automatically
interact with software applications executing on the multiple
client computer systems. In some embodiments, control computer
system 12 utilizes additional software modules, e.g., duplicate
virtual operator, image analyzer, and image verifier modules. In
alternate embodiments, a single instance of each distinct software
module is required to interface with the multiple client computer
systems and multiple software applications running thereon.
[0042] In some embodiments, a client computer system 72 (FIG. 1)
captures images from a scanner 74 linked to a client computer 76,
and the images are typically stored in a database 78, as described
hereinbelow. In an alternative embodiment, images are provided via
any conventional image provider 80. In yet another alternative
embodiment, captured images are received via a computer network 82
after having been sent as image data over the computer network from
a remote computer system.
[0043] In an alternative embodiment, a human operator may be used
to verify simulated actions of virtual operation 28 (FIG. 1) prior
to the entry data being input automatically to the client computer
system. For example, when erroneous actions could have harmful
effects, or significant risk is incurred by applying updates to the
software system, the human operator may perform a review of the
entry data on a control screen 84 and may provide a formal
approval, before the entry data is entered to the software
application, as described hereinbelow.
[0044] For purposes of illustration, in an embodiment of the
present invention, a software application running on client
computer system 72 (FIG. 4) is used to manage back-office services
for an insurance provider. The software application provides
services comprising receiving payments from customers, associating
the payments with insurance policies having outstanding monetary
balances, and selectively applying the payments to the
balances.
[0045] Reference is now made to FIG. 5, which is a schematic,
pictorial illustration of scanned and reference images using the
embodiment of FIG. 4. In the present embodiment, client computer 76
(FIG. 4) scans a check received from a client, and captures a check
image 86 using scanner 74, storing check image 86 in database 78.
Client computer 76 processes check image 86 and identifies an
account number image 88, a bank account owner signature image 90,
and a payment amount image 92. The software application processes
the images, deriving a bank account owner name 94, ("Jane Doe"), an
account number 96, ("1234567890"), and a payment amount 98,
("$122.50"), in the present example.
[0046] The software application searches database 78 (FIG. 4) and
locates an insurance policy holder, ("Jane Doe"), that matches bank
account owner name 94, and provides client computer system 72 with
an insurance policy holder listing image, which is substantially
similar to image 46 (FIG. 2) for display by client computer 76 on
screen 36. The policy holder listing image typically displays the
insurance policy number, the name of the customer, or insurance
policy holder, and any outstanding monetary balance due. The
software application displays check image 86 and the images
identified thereupon, in addition to the processed check data,
e.g., bank account owner name 94, account number 96, and payment
amount 98.
[0047] In the present example, virtual operation 28 (FIG. 4) may
capture the insurance policy holder listing image from screen 36.
Image analyzer module 30 may be used by virtual operation 28 to
recognize check image 86 and the images displayed thereupon,
account number image 88, bank account owner signature image 90, and
payment amount image 92. Virtual operation 28 uses image verifier
module 32 to verify that the images match the processed check data,
typically stored in database 78 in the present example. Image
verifier module 32 matches the images to the processed check data
and in response to the matching, virtual operation 28 controls
input device 18 to automatically input entry data to client
computer system 72 to confirm processing of the check.
[0048] In an alternative embodiment of the present invention, after
virtual operation 28 (FIG. 4) has simulated the actions of a human
operator by confirming that the check should be applied to the
insurance policy holder's balance, an additional step may be taken
for the purpose of safety. The software application provides an
additional verification image 100 to the human operator on control
screen 84. Additional verification image 100 comprises the images
derived from check image 86, the processed check data, an
acceptance button icon 102, and a rejection button icon 104. The
human operator can compare bank account owner signature image 90 to
bank account owner name 94, for example, and if satisfied, accept
the check for payment of the insurance policy by using acceptance
button icon 102. Alternatively, rejection button icon 104 may be
used by the human operator to reject the check for payment of the
balance.
[0049] In the previous description, numerous specific details are
set forth in order to provide a thorough understanding of the
present invention. It will be apparent to one skilled in the art,
however, that the present invention may be practiced without these
specific details. In other instances, well-known circuits, control
logic, and the details of computer program instructions for
conventional algorithms and processes have not been shown in detail
in order not to obscure the present invention unnecessarily.
[0050] As will be appreciated by one skilled in the art, the
present invention may be embodied as a system, method or computer
program product. Accordingly, the present invention may take the
form of an entirely hardware embodiment, an entirely software
embodiment (including firmware, resident software, micro-code,
etc.) or an embodiment combining software and hardware aspects that
may all generally be referred to herein as a "circuit," "module" or
"system." Furthermore, the present invention may take the form of a
computer program product embodied in any tangible medium of
expression having computer usable program code embodied in the
medium.
[0051] Any combination of one or more computer usable or computer
readable medium(s) may be utilized. The computer-usable or
computer-readable medium may be, for example but not limited to, an
electronic, magnetic, optical, electromagnetic, infrared, or
semiconductor system, apparatus, device, or propagation medium.
More specific examples (a non-exhaustive list) of the
computer-readable medium would include the following: an electrical
connection having one or more wires, a portable computer diskette,
a hard disk, a random access memory (RAM), a read-only memory
(ROM), an erasable programmable read-only memory (EPROM or Flash
memory), an optical fiber, a portable compact disc read-only memory
(CDROM), an optical storage device, a transmission media such as
those supporting the Internet or an intranet, or a magnetic storage
device. Note that the computer-usable or computer-readable medium
could even be paper or another suitable medium upon which the
program is printed, as the program can be electronically captured,
via, for instance, optical scanning of the paper or other medium,
then compiled, interpreted, or otherwise processed in a suitable
manner, if necessary, and then stored in a computer memory. In the
context of this document, a computer-usable or computer-readable
medium may be any medium that can contain, store, communicate,
propagate, or transport the program for use by or in connection
with the instruction execution system, apparatus, or device. The
computer-usable medium may include a propagated data signal with
the computer-usable program code embodied therewith, either in
baseband or as part of a carrier wave. The computer usable program
code may be transmitted using any appropriate medium, including but
not limited to wireless, wireline, optical fiber cable, RF,
etc.
[0052] Computer program code for carrying out operations of the
present invention may be written in any combination of one or more
programming languages, including an object oriented programming
language such as Java, Smalltalk, C++ or the like and conventional
procedural programming languages, such as the "C" programming
language or similar programming languages. The program code may
execute entirely on a user's computer, partly on the user's
computer, as a stand-alone software package, partly on the user's
computer and partly on a remote computer or entirely on the remote
computer or server. In the latter scenario, the remote computer may
be connected to the user's computer through any type of network,
including a local area network (LAN) or a wide area network (WAN),
or the connection may be made to an external computer (for example,
through the Internet using an Internet Service Provider).
[0053] Embodiments of the present invention are described below
with reference to flowchart illustrations and/or block diagrams of
methods, apparatus (systems) and computer program products
according to embodiments of the invention. It will be understood
that each block of the flowchart illustrations and/or block
diagrams, and combinations of blocks in the flowchart illustrations
and/or block diagrams, can be implemented by computer program
instructions. These computer program instructions may be provided
to a processor of a general purpose computer, special purpose
computer, or other programmable data processing apparatus to
produce a machine, such that the instructions, which execute via
the processor of the computer or other programmable data processing
apparatus, create means for implementing the functions/acts
specified in the flowchart and/or block diagram block or
blocks.
[0054] These computer program instructions may also be stored in a
computer-readable medium that can direct a computer or other
programmable data processing apparatus to function in a particular
manner, such that the instructions stored in the computer-readable
medium produce an article of manufacture including instruction
means, which implement the function/act specified in the flowchart
and/or block diagram block or blocks.
[0055] The computer program instructions may also be loaded onto a
computer or other programmable data processing apparatus to cause a
series of operational steps to be performed on the computer or
other programmable apparatus to produce a computer implemented
process such that the instructions, which execute on the computer
or other programmable apparatus provide processes for implementing
the functions/acts specified in the flowchart and/or block diagram
block or blocks.
[0056] It will be appreciated by persons skilled in the art that
the present invention is not limited to what has been particularly
shown and described hereinabove. Rather, the scope of the present
invention includes both combinations and sub-combinations of the
various features described hereinabove, as well as variations and
modifications thereof that are not in the prior art, which would
occur to persons skilled in the art upon reading the foregoing
description.
TABLE-US-00001 COMPUTER PROGRAM LISTINGS Listing 1
image=acquire_screen OCR.recognize(image) for (text in set of
phrases) if (text ends_with ":") labels.add(text) else
fields.add(text) end for (field in fields)
label=find_nearest(labels,field) associate_field(label,field) end
Listing 2 Recognize(image) black_white_image=binarize(image) //
convert color characters to b&w binary image cclist =
find_connected_components(black_white_image) // red marks below
for_each component letter=classify(component)
insert_to_spatial_structure(letter) end
compose_phrases_from_spatial_structure end Listing 3
SetField(field_label,value) label=find_closest_match(field_label)
field=retrieve_associated_field(label)
coords=extract_field_coordinates(field) mouse_click(coords)
generate_keystrokes(value) end Listing 4 mouse_click(coords)
input=create MOUSEINPUT structure input.coords=cords
SendInput(input) // WIN32 API function end
generate_keystrokes(value) for_each character in value
generate_keystroke(character) end end generate_keystroke(character)
input=create KEYBDINPUT structure input.character=character
SendInput(input) // WIN32 API function end
* * * * *