U.S. patent application number 13/566243 was filed with the patent office on 2014-01-09 for modeling and evaluating application performance in a new environment.
This patent application is currently assigned to International Business Machines Corporation. The applicant listed for this patent is Rong N. Chang, Chang-Shing Perng, Byung Chul Tak, Chunqiang Tang. Invention is credited to Rong N. Chang, Chang-Shing Perng, Byung Chul Tak, Chunqiang Tang.
Application Number | 20140012562 13/566243 |
Document ID | / |
Family ID | 49879179 |
Filed Date | 2014-01-09 |
United States Patent
Application |
20140012562 |
Kind Code |
A1 |
Chang; Rong N. ; et
al. |
January 9, 2014 |
MODELING AND EVALUATING APPLICATION PERFORMANCE IN A NEW
ENVIRONMENT
Abstract
A method for evaluating the performance of an application when
migrated from a first environment in which the application is
currently executing to a second, different environment includes
generating a virtual application that mimics the resource consuming
behavior of the application, executing the virtual application in
the second environment, and evaluating the performance of the
virtual application in the second environment.
Inventors: |
Chang; Rong N.;
(Pleasantville, NY) ; Perng; Chang-Shing; (Goldens
Bridge, NY) ; Tak; Byung Chul; (Elmsford, NY)
; Tang; Chunqiang; (Ossining, NY) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Chang; Rong N.
Perng; Chang-Shing
Tak; Byung Chul
Tang; Chunqiang |
Pleasantville
Goldens Bridge
Elmsford
Ossining |
NY
NY
NY
NY |
US
US
US
US |
|
|
Assignee: |
International Business Machines
Corporation
Armonk
NY
|
Family ID: |
49879179 |
Appl. No.: |
13/566243 |
Filed: |
August 3, 2012 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
13544232 |
Jul 9, 2012 |
|
|
|
13566243 |
|
|
|
|
Current U.S.
Class: |
703/22 |
Current CPC
Class: |
G06F 2201/815 20130101;
G06F 11/3409 20130101; G06F 11/3461 20130101; G06F 2201/865
20130101 |
Class at
Publication: |
703/22 |
International
Class: |
G06F 9/45 20060101
G06F009/45 |
Claims
1. A system for evaluating a performance of a distributed, complex
multi-tier application when migrated from a first environment in
which the distributed, complex multi-tier application is currently
executing to a second, different environment, the system
comprising: an application structure analyzer for generating a
virtual application that mimics a resource consuming behavior and
an inter-tier behavior of the distributed, complex multi-tier
application; a drone control module for executing the virtual
application in the second environment; and a performance evaluator
for evaluating a performance of the virtual application in the
second environment.
2.-3. (canceled)
4. The system of claim 1, wherein the distributed, complex
multi-tier application is executed by a plurality of servers.
5. The system of claim 4, further comprising: at least one data
collection agent residing on at least one of the plurality of
servers, for collecting data indicative of the resource consuming
behavior.
6. The system of claim 5, wherein the at least one data collection
agent is integrated with a hypervisor of the at least one of the
plurality of servers.
7. The system of claim 5, further comprising: a database for
storing the data.
8. The system of claim 5, wherein the data is collected from the
distributed, complex multi-tier application while the application
is live.
9. The system of claim 5, wherein the data comprises a central
processing unit utilization of the distributed, complex multi-tier
application.
10. The system of claim 5, wherein the data comprises a disk
input/output of the distributed, complex multi-tier
application.
11. The system of claim 5, wherein the data comprises an amount of
memory consumed per thread of the distributed, complex multi-tier
application.
12. The system of claim 5, wherein the data further comprises data
indicative of an internal threading structure of the distributed,
complex multi-tier application.
13. The system of claim 12, wherein the data indicative of an
internal threading structure of the distributed, complex multi-tier
application comprises a number of threads processed by the
distributed, complex multi-tier application.
14. The system of claim 12, wherein the data indicative of an
internal threading structure of the distributed, complex multi-tier
application comprises a change over time to a thread processed by
the distributed, complex multi-tier application.
15. The system of claim 5, wherein the data further comprises data
indicative of a transaction path of the distributed, complex
multi-tier application.
16. The system of claim 1, wherein the resource consuming behavior
is modeled on a per-transaction-step basis.
17. An apparatus comprising a non-transitory computer readable
storage medium containing an executable program for evaluating a
performance of a distributed, complex multi-tier application when
migrated from a first environment in which the distributed, complex
multi-tier application is currently executing to a second,
different environment, where the program performs steps comprising:
generating a virtual application that mimics a resource consuming
behavior and an inter-tier behavior of the distributed, complex
multi-tier application; executing the virtual application in the
second environment; and evaluating a performance of the virtual
application in the second environment.
18. The apparatus of claim 17, wherein the generating comprises:
collecting data indicative of the resource consuming behavior from
the distributed, complex multi-tier application while the
application is live; producing a model of the resource consuming
behavior; and deploying one or more drones in the second
environment, where the one or more drones consume resources
according to the model.
19. The apparatus of claim 17, wherein the virtual application is
executed in the second environment without executing the
distributed, complex multi-tier application in the second
environment.
20. (canceled)
21. The system of claim 4, wherein the plurality of servers
performs presentation, application processing, and data management
for the distributed, complex multi-tier application as logically
separate processes over the plurality of servers.
22. The system of claim 15, wherein the data indicative of a
transaction path comprises end-to-end trails of messages that
travel across multiple servers of the distributed, complex
multi-tier application.
23. The system of claim 22, wherein the end-to-end trails start
with an initial request originating with a first server of the
multiple servers and end with a reply message originating with a
second server of the multiple servers.
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of U.S. patent
application Ser. No. 13/544,232, filed Jul. 9, 2012, which is
herein incorporated by reference in its entirety.
BACKGROUND OF THE INVENTION
[0002] The present invention relates generally to data migration
and relates more specifically to the migration of distributed,
complex multi-tier applications.
[0003] FIG. 1 is a block diagram illustrating an exemplary
distributed, complex multi-tier application ("Target Application")
operating in a first environment ("Environment A"). As illustrated,
a distributed, complex multi-tier application is an application in
which presentation, application processing, and data management are
performed as logically separate processes over multiple cooperating
servers (e.g., "Servers A-D"). These servers might include, for
example, one or more hypertext transfer protocol (HTTP) servers,
application servers, and/or relational database management
systems.
[0004] When a distributed, complex multi-tier application is to be
migrated to a new environment (e.g., a cloud environment), it is
often difficult to estimate the resources necessary to ensure that
the application's performance in the new environment matches its
performance in the old environment. It is also non-trivial to
project the application's performance for workloads that are
heavier than those it has already encountered. For instance, in
FIG. 1, Environment A has access to a first set of resources
including, but not limited to, processing, memory, and hardware
resources. Supported by the first set of resources, the Target
Application may be capable of processing fifty-eight requests per
second. However, it is unknown what the Target Application's
performance might be if the Target Application were to be migrated
to Environment B, which has access to a different, second set of
resources.
[0005] Although the Target Application could be directly installed
in Environment B and evaluated accordingly, this approach is not
ideal for several reasons. For one, installation is complicated by
the subtle interdependencies between the application tiers,
potentially complex configurations, and application specific
treatments. Moreover, it is costly and labor-intensive to migrate
and store all of the data associated with a distributed, complex
multi-tier application. Finally, if the Target Application is to be
migrated to multiple new environments, all of the above
complications will apply in each new environment.
SUMMARY OF THE INVENTION
[0006] A method for evaluating the performance of an application
when migrated from a first environment in which the application is
currently executing to a second, different environment includes
generating a virtual application that mimics the resource consuming
behavior of the target application, executing the virtual
application in the second environment, and evaluating the
performance of the virtual application in the second
environment.
BRIEF DESCRIPTION OF THE DRAWINGS
[0007] So that the manner in which the above recited features of
the present invention can be understood in detail, a more
particular description of the invention may be had by reference to
embodiments, some of which are illustrated in the appended
drawings. It is to be noted, however, that the appended drawings
illustrate only typical embodiments of this invention and are
therefore not to be considered limiting of its scope, for the
invention may admit to other equally effective embodiments.
[0008] FIG. 1 is a block diagram illustrating an exemplary
distributed, complex multi-tier application operating in a first
environment;
[0009] FIG. 2 is a block diagram illustrating one embodiment of a
system for evaluating the performance of a target application in a
new environment, according to the present invention;
[0010] FIG. 3 is a flow diagram illustrating one embodiment of a
method for evaluating the performance of a target application in a
new environment, according to the present invention; and
[0011] FIG. 4 is a high-level block diagram of the performance
evaluation method that is implemented using a general purpose
computing device.
DETAILED DESCRIPTION
[0012] In one embodiment, the invention is a method and apparatus
for evaluating application performance in a new environment.
Embodiments of the invention generate a virtual application that
mimics a target application (e.g., a distributed, complex
multi-tier application) and can be executed in a new environment in
order to estimate the target application's performance in the new
environment. In one embodiment, the virtual application comprises
one or more components or "drones" that are dynamically configured
to consume the same amount of resources as the target application.
Thus, the terms "virtual application" and "drone" or "drones" may
be used interchangeably herein to refer to entities that are
configured to consume a specified amount of resources based on the
resource usage of a target application.
[0013] FIG. 2 is a block diagram illustrating one embodiment of a
system 200 for evaluating the performance of a target application
in a new environment, according to the present invention.
[0014] As illustrated, the system 200 generally comprises a
plurality of components, including: an application structure
analyzer 202, a drone control module 204, a workload modeler 206, a
workload generator 208, and a performance evaluator 210. The
operations of each of these components 202-210 are discussed in
greater detail below. Although each of these components 202-210 is
illustrated as a discrete component, it will be appreciated that
one or more of these components 202-210 may be combined into a
single component that performs multiple tasks. Moreover, any one of
these components 202-210 may be further split into two or more
smaller components that perform a subset of the tasks performed by
the larger component. Any of the components 202-210 may comprise a
hardware processor (e.g., a central processing unit, co-processor,
or specialized processor).
[0015] The system 200 is coupled to a database 212, which stores
data related to a target application 214. In one embodiment, the
target application 214 comprises a distributed, complex multi-tier
application that is to be migrated to a new environment 216. Thus,
the target application 214 comprises a plurality of servers
218.sub.1-218.sub.n (hereinafter collectively referred to as
"servers 218") that cooperate to perform presentation, application
processing, and data management, among potentially other tasks. In
one embodiment, each of the servers 218 further includes a data
collection agent or data collector 220.sub.1-220.sub.n (hereinafter
collectively referred to as "data collectors 220"). In one
embodiment, the data collectors are integrated with the hypervisors
associated with each of the servers 218.
[0016] In operation, the data collectors 220 collect data from the
servers 218. The collected data represents the target application's
resource usage, internal threading structure, and/or transaction
paths. In one embodiment, the collected data specifically includes
one or more of the following: the target application's network
usage, the target application's central processing unit (CPU)
utilization, the target application's disk system calls, the target
application's memory system calls, and the target system's request
log (which may, in turn, include the target application's threading
structure). The collected data is stored in the database 212.
[0017] As discussed above, the database 212 is accessible by the
system 200, and the various components 202-210 of system 200
operate on the collected data in different ways. For instance, the
application structure analyzer 202 uses the data relating to the
target application's resource usage, internal threading structure,
and/or transaction paths to produce a model of the target
application 214. Within the context of the present invention, the
term "resource usage" is understood to refer to an application's
consumption of central processing unit (CPU) cycles, input/output
(I/O) bandwidth, and/or storage space. The term "internal threading
structure" is understood, within the context of the present
invention, to refer to an application's number of active threads,
the roles assigned to each active thread, and/or the patterns in
which the active threads are created and destroyed. The term
"transaction path" is understood, within the context of the present
invention, to refer to the end-to-end trails of user messages that
travel across multiple server processes (i.e., starting at an
initial request and ending at a final reply message).
[0018] The drone control module 204 then generates a "virtual
application" based on this model. The virtual application comprises
a plurality of drones 222.sub.1-222.sub.m (hereinafter collectively
referred to as "drones 222") that mimic the resource consuming
behavior of the target application 214. There need not necessarily
be a one-to-one correspondence between the number of servers 218 in
the target application 214 and the number of drones 222 in the
virtual application. The drone control module 204 deploys and
controls the operation of the drones 222 in the new environment
216.
[0019] In addition, the workload modeler 206 uses data relating to
the target application's workload (e.g., request logs) to produce a
model of the workload. The workload generator 208 then uses the
workload model to generate a "virtual workload" that mimics the
actual workload of the target application 214.
[0020] The performance evaluator 210 collects data from the new
environment 216 relating to the performance statistics of the
virtual application. In particular, the performance statistics
relate to the processing of the virtual workload by the virtual
application. The performance evaluator 210 evaluates this data in
accordance with one or more metrics in order to estimate how the
target application 214 will perform in the new environment 216.
[0021] FIG. 3 is a flow diagram illustrating one embodiment of a
method 300 for evaluating the performance of a target application
in a new environment, according to the present invention. The
method 300 may be performed, for example, by the system 200
illustrated in FIG. 2. As such, reference is made in the discussion
of the method 300 to various elements depicted in FIG. 2. However,
it will be appreciate that the method 300 may also be performed by
systems having alternate configurations.
[0022] The method 300 begins at step 302 and proceeds to step 304,
where data is collected (e.g., by the data collectors 220) from a
live target application executing in a first environment. The
target application is "live" in the sense that it is deployed and
functioning in the first environment. In one embodiment, the target
application is a distributed, complex multi-tier application. In
this embodiment, the target application performs presentation,
application processing, and data management as logically separate
processes over multiple cooperating servers (e.g., HTTP servers,
application servers, and/or relational database management
systems).
[0023] In one embodiment, the collected data includes the target
application's resource usage, internal threading structure, and/or
transaction paths. In one embodiment, the internal threading
structure includes the number of threads and changes to threads
over time as processed by the target application.
[0024] In one embodiment, the transaction paths are constructed
from partial transaction paths reported by the data collectors 220.
For instance, from the data reported by the data collectors 220
associated with three different servers, a complete transaction
path through the three servers may be constructed.
[0025] In one embodiment, the resource usage includes the CPU, disk
input/output (I/O), and memory consumed per thread of the target
application. For instance, the method 200 might count the number of
CPU cycles between read/write system calls (e.g., using a time
stamp counter) and determine whether this count will be stable
across the CPU architecture. Different access types will require
different numbers of CPU cycles. For instance, reading a register,
reaching a level 1 (L1) cache, reaching a level 2 (L2) cache,
reaching a level 3 (L3) cache, and reading a main memory will all
require varying numbers of CPU cycles. In further embodiments, the
method 300 counts the number of CPU cycles between other types of
system calls and interrupts (e.g., user-kernel crossings). Memory
access latency may be a non-negligible factor under these
circumstances.
[0026] The memory footprint and resident set size may also be
relevant in step 304, as they directly relate to buffer cache
behavior. Thus, in one embodiment, break (brk) system calls may be
tracked and/or related to the number of threads and memory size. In
a further embodiment, the number of pages touched by the target
application during particular stages of operation is tracked. For
instance, the number of pages touched during thread context
switches (e.g., per CPU) may be tracked by the method 300.
[0027] In step 306, the resource consumption and inter-tier
behavior of the target application are modeled (e.g., by the
application structure analyzer 202). In one embodiment, the
per-transaction-step resource consumption of the target application
in particular is modeled. The per-transaction-step resource
consumption may include, for example, one or more of: the number of
required CPU cycles, the number of main memory read requests, the
number of main memory write requests, the number of disk read
requests, the number of disk write requests, the amount of data
(e.g., number of bytes) received over a network interface card, or
the amount of data sent over a network interface card associated
with a given transaction step.
[0028] In step 308, the workload of the target application is
modeled (e.g., by the workload modeler 206). In one embodiment,
modeling the workload includes identifying the types of requests
that are processed by the target application and the resource usage
patterns associated with these types of requests.
[0029] In step 310, a virtual application is generated (e.g., by
the drone control module 204). The virtual application mimics the
application structure (e.g., communications between servers and how
the communications travel) and resource usage patterns (e.g., CPU
cycles per memory access) of the target application. Although the
virtual application may produce an exact replay of the target
application's resource access history, this is not always the case.
In alternate embodiments, statistical modeling may be used to
produce a virtual application whose behavior is qualitatively
identical to the target application's behavior. For instance, a
read/write probability of the target application may be translated
in the virtual application into files with the same number and
sizes as in the first environment. As a further example, an I/O
size distribution associated with the target application may be
translated in the virtual application into two distributions.
[0030] In one embodiment, the application structure that is
mimicked includes the threading structure (e.g., number of threads
and changes to threads over time) of the target application. In one
embodiment, the resource usage patterns that are mimicked include
the CPU, disk I/O, and memory consumed per thread of the target
application.
[0031] As discussed above, the virtual application comprises one or
more drones that mimic the target application. For example, each
drone may mimic a particular server that is part of the target
application. In one embodiment, the drones are dynamically
configured to consume the same amount of resources and create the
same number of threads as the servers of the target application.
For instance, in one embodiment, initial system calls and RSS
runtime changes associated with the target application are mimicked
by the virtual application. In a further embodiment, the virtual
application mimics the paging (I/O) behavior of the target
application. For instance, each thread associated with the virtual
application may be configured to touch a number of pages observed
being touched by the live target application during a thread
context switch.
[0032] In step 312, a virtual workload is generated (e.g., by the
workload generator 208). The virtual workload relies on the
workload model generated in step 308 to produce a virtual workload
of varying intensity and request types that realistically mimics
the actual workload of the target application.
[0033] In step 314, the virtual application is executed in a second
environment, different from the first environment. In one
embodiment, the virtual application is executed in accordance with
the virtual workload. Thus, the virtual application will process
synthesized data (the virtual workload) by consuming resources that
are substantially equal to the resources consumed by the target
application when processing real workloads.
[0034] In step 316, data related to execution of the virtual
application in the second environment is collected. In step 318,
the collected data is either evaluated in accordance with a given
metric (e.g., by the performance evaluator 21) or output to a third
party for evaluation. The given metric might include, for example,
a target throughput or target response time (e.g., average response
time) of the target application in the new environment.
[0035] The method 300 ends in step 320.
[0036] In some embodiments, one or more steps of the method 300 are
repeated in order to compensate for variability in hardware and/or
software performance. For instance, when the virtual workload is
generated, the degree of sequentiality of the block layout of the
dummy files (which are used by the drones to emulate the disk
input/output) may vary depending on the conditions of the current
target disks, and the sequentiality of the file blocks can greatly
affect application performance. The disk input/output (I/O)
performance may also vary depending on the physical locations of
the file blocks (e.g., outer rim of the disk versus closer to the
center of the disk) and on the disk's internal cache states (which
are not shown or controllable by user-level applications). Thus,
data relating to the execution of the virtual application in the
second environment may need to be collected and evaluated multiple
times in order to absorb variances such as those discussed.
[0037] The virtual application that is generated and executed
according to the method 300 allows one to analyze the performance
of a target application in a new environment without having to
actually deploy the target application in the new environment.
Thus, a tremendous amount of time and resources may be conserved.
Moreover, the analysis produced according to the method 300 is more
accurate than conventional mathematical modeling approaches that
require modeling of hardware and non-trivial modeling of
hypervisors.
[0038] Furthermore, the method 300 is capable of measuring the
target application's performance in connection with workloads that
were not encountered during the profiling of the target application
(e.g., steps 304-312). If the target environment should change, it
is likely that the workload of the target application will also
change. The method 300 may thus be implemented to measure the
performance of the target application for hypothetical (i.e., not
observed) workloads as well as real (i.e., observed) workloads.
This allows one to test the scalability of the target application
in the new environment.
[0039] Moreover, the method 300 can be implemented to measure the
performance of the target workload when one or more components of
the target application are hypothetically replaced with components
that behave differently and/or have different performance
characteristics. For example, the method 300 may have access to
profiling data for a MYSQL database and a POSTGRESQL database.
Then, even if the method 300 has only profiled and modeled an
application that uses a MYSQL database, a corresponding virtual
application may be constructed that includes a database part that
follows the resource usage patterns of a POSTGRESQL database.
Performance data for this virtual application may help determine
whether the MYSQL database of the target application should be
replaced with another type of database in the new environment.
[0040] FIG. 4 is a high-level block diagram of the performance
evaluation method that is implemented using a general purpose
computing device 400. In one embodiment, a general purpose
computing device 400 comprises a processor 402, a memory 404, a
performance evaluation module 405 and various input/output (I/O)
devices 406 such as a display, a keyboard, a mouse, a stylus, a
wireless network access card, an Ethernet interface, and the like.
In one embodiment, at least one I/O device is a storage device
(e.g., a disk drive, an optical disk drive, a floppy disk drive).
It should be understood that the performance evaluation module 405
can be implemented as a physical device or subsystem that is
coupled to a processor through a communication channel.
[0041] Alternatively, the performance evaluation module 405 can be
represented by one or more software applications (or even a
combination of software and hardware, e.g., using Application
Specific Integrated Circuits (ASIC)), where the software is loaded
from a storage medium (e.g., I/O devices 406) and operated by the
processor 402 in the memory 404 of the general purpose computing
device 400. Thus, in one embodiment, the performance evaluation
module 405 for evaluating application performance in a new
environment, as described herein with reference to the preceding
figures, can be stored on a computer readable storage medium (e.g.,
RAM, magnetic or optical drive or diskette, and the like).
[0042] It should be noted that although not explicitly specified,
one or more steps of the methods described herein may include a
storing, displaying and/or outputting step as required for a
particular application. In other words, any data, records, fields,
and/or intermediate results discussed in the methods can be stored,
displayed, and/or outputted to another device as required for a
particular application. Furthermore, steps or blocks in the
accompanying figures that recite a determining operation or involve
a decision, do not necessarily require that both branches of the
determining operation be practiced. In other words, one of the
branches of the determining operation can be deemed as an optional
step.
[0043] While the foregoing is directed to embodiments of the
present invention, other and further embodiments of the invention
may be devised without departing from the basic scope thereof.
Various embodiments presented herein, or portions thereof, may be
combined to create further embodiments. Furthermore, terms such as
top, side, bottom, front, back, and the like are relative or
positional terms and are used with respect to the exemplary
embodiments illustrated in the figures, and as such these terms may
be interchangeable.
* * * * *