U.S. patent application number 10/177865 was filed with the patent office on 2002-10-31 for adaptive load generation.
Invention is credited to Har'El, Uri, Kinreich, Ilan, Sherman, Yahali.
Application Number | 20020161553 10/177865 |
Document ID | / |
Family ID | 22738188 |
Filed Date | 2002-10-31 |
United States Patent
Application |
20020161553 |
Kind Code |
A1 |
Har'El, Uri ; et
al. |
October 31, 2002 |
Adaptive load generation
Abstract
A method provides for a series of procedural steps in the
determination of the state of operation of a computer web
application responding to requests by a multiplicity of client
machines. The procedural steps are accomplished by use of a
plurality of load generators which apply loading to the web
application simulating the actual loading of any one of the client
machines. One of the load generators is selected to increase its
load by an incremental loading. A further load generator is
employed in the place of a probing client machine for observation
of the performance of the web application. At the probing client
machine, there is an observing of the transactions per second (TPS)
and response time of the web application, the observing taking
place both before and after the application of the load increment
to the web application. Thereupon, by logic analysis of the
presence or absence of an increase in the TPS, as well as a
presence or absence of an increase in the response time, a
determination is made as to whether the web application has
attained its full capacity or whether the load generator has
attained full capacity, the latter case requiring a utilization of
a further one of the load generators for further loading of the web
application.
Inventors: |
Har'El, Uri; (Zur Yigal,
IL) ; Sherman, Yahali; (Tel Aviv, IL) ;
Kinreich, Ilan; (Wayland, MA) |
Correspondence
Address: |
FULBRIGHT & JAWORSKI, LLP
666 FIFTH AVE
NEW YORK
NY
10103-3198
US
|
Family ID: |
22738188 |
Appl. No.: |
10/177865 |
Filed: |
June 20, 2002 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
10177865 |
Jun 20, 2002 |
|
|
|
09199592 |
Nov 25, 1998 |
|
|
|
6434513 |
|
|
|
|
Current U.S.
Class: |
702/186 ;
714/E11.193 |
Current CPC
Class: |
G06F 11/3428 20130101;
G06F 11/3414 20130101; G06F 2201/87 20130101; G06F 2201/875
20130101; G06F 11/3419 20130101; G06F 11/3442 20130101 |
Class at
Publication: |
702/186 |
International
Class: |
G06F 015/00 |
Claims
What is claimed is:
1. A method, employing adaptive load generation, for measuring
performance of a web application which processes requests by client
machines, the method comprising procedural steps of: providing a
set of load generators for generating loads which simulate the
loading on the web application of requests by client machines;
employing a plurality of load generators from said set of load
generators to said web application with an initial amount of
loading; a first step of directing a first of said plurality of
load generators to increase its load by a first load increment;
determining if there has been any increase in transactions per
second (TPS) and response time of said web application resulting
from said first load increment; and applying a logic analysis to a
determination of said determining step wherein an absence of
increase in the TPS in conjunction with the presence of an increase
in the response time is determinative of said web application being
in an operating state of full capacity.
2. A method according to claim 1 wherein said determining step
includes a first step of observing said TPS and said response time
prior to said directing step, and a second step of observing said
TPS and said response time subsequent to said determining step.
3. A method according to claim 2 further comprising steps of
selecting an additional one of said generators to function as a
probing client machine, and performing said first and said second
steps of observing said (TPS) and said response time at said
probing client machine, said plurality of load generators providing
the true number of effective clients loading said web
application.
4. A method according to claim 3 wherein there are a plurality of
client machines accessing said web application, each of said client
machines running respective threads concurrently, and wherein
implementation of the method provides for a plurality of
measurements of web application performance during the running of
the threads of the respective clients.
5. A method according to claim 1 wherein, in said logic analysis,
an increase in the TPS is taken as a determination that said web
application and said first load generator are operating
satisfactorily, the procedure of the method continuing with a
second step of directing one of said load generators of said set of
load generators to increase a loading of said web application by a
second load increment, thereafter there is a repetition of said
steps of determining and applying.
6. A method according to claim 5 wherein, in said second step of
directing, there is a directing of a second load generator of said
plurality of load generators to increase its load.
7. A method according to claim 1 wherein, in said logic analysis,
an absence of an increase in said TPS in conjunction with an
absence of an increase in said response is determinative of said
first load generator having reached an operating state of full
capacity, wherein the procedure of the method continues with:
selecting a second generator of said plurality of generators; and a
second step of directing said second load generator to increase a
loading of said web application by a second load increment,
thereafter, there is a repetition of said steps of determining and
applying.
8. A method according to claim 1 further comprising a step,
subsequent to said applying step, of assessing whether a
performance goal of said web application has been attained,
thereafter, in the event that the performance goal has not been
attained, there is a repetition of said steps of determining and
applying, and in the event that the performance goal has been
attained, there is a terminating of the procedure of the
method.
9. A method according to claim 1 further comprising a step, prior
to said determining step, of assessing whether said web application
has attained a performance goal, the procedure of the method
terminating if the performance goal has been attained, the
procedure of the method continuing with said determining step if
the performance goal has not been attained.
Description
BACKGROUND OF THE INVENTION
[0001] This invention relates to the simulation of loading a web
application and, more particularly, to the simulation of a loading
of a web application by clients accessing the web application.
[0002] Communication of data from a central repository of data such
as an Internet or Intranet to numerous persons referred to as
clients is accomplished by use of a web application interconnected
via a communication system to the clients. The specific resource
requirements upon the web application for processing a client's
request for data depend on the nature of the data requested by the
client, and, therefore, the loading of the web application by the
client varies upon the clients. The total load on the web
application increases with the number of clients accessing the web
application and, with a large number of clients, may exceed the
operating capacity of the web application in terms of meeting
a/many performance goal/s. Such performance goals may be measured
in terms of transactions per second (TPS), average response time,
number of clients being served, CPU utilization or other.
[0003] To provide good service to the clients by the web
application, there is a need to know the capacity of the web
application in terms of the number of clients which can be served
and/or how close the web application is to its capacity limit. To
conduct such an analysis of a web application, equipment known as
load generators have been developed to simulate the load generated
by clients upon the web application. To simulate a large number of
clients, there is employed a load generation system wherein one or
more computers function as load generators to simulate the
multiplicity of clients, such as personal computers, which may
access the web application.
[0004] Presently available web application testing does not
adequately address the foregoing need for knowledge of the web
application capacity.
SUMMARY OF THE INVENTION
[0005] The foregoing need is met and other advantages are provided,
in accordance with the invention, by a procedure of monitoring the
performance of the web application while adapting the load
generation for testing the web application performance. The
invention provides for an increasing of the load upon the web
application, while monitoring the transactions per second (TPS),
this being followed by a further increase in the loading with a
monitoring of the response time. During this procedure, there is a
checking of whether the web application has reached a determined
performance goal. If the performance goal is attained, the
procedure terminates because no further performance is anticipated.
The procedure also involves steps performed to determine whether a
load generator, used in the loading of the web application, has
reached its limit.
[0006] It is advantageous to designate a single client machine, to
be referred to as a probing client, to simulate a single client
obtaining of data from the web application. The monitoring of the
performance characteristics of the web application by the probing
client serves as an accurate measure of the-web application
performance.
[0007] In the simulation of the load imposed by simulated client,
there is a creation of a test script that defines the behavior of
the simulated clients. A user of the load generation system can set
in advance the number of simulated clients for each load generator
in order to determine the load size. The load size and its
distribution on the load generators can be set to different values
for different periods of the test.
[0008] The logic for interpreting the measurements is as follows.
If the load size on a given load generator has increased, and its
TPS remains steady, then there are two possibilities. The first
possibility is that the load generator has reached its full
capacity, and therefore the load increase actually interferes with
the operation of the load generator. As a second possibility, the
application being tested (ABT) has reached its full capacity and
cannot produce more TPS. In order to determine which of these
possibilities has occurred, two indicators are used.
[0009] The first indicator is the measurement of the response time,
as taken from the probing client. If the response time does not
change in value upon an increase in the load, this is understood to
indicate that the ABT has not experienced a change in load size.
Because, otherwise, the TPS would also have increased. Thus, the
load generator is operating at full capacity. If the response time
increases as the load increases, this is understood to mean that
the ABT is unable to process more requests and, therefore, any
further increase in the load size will affect the time, namely the
response time, which is required to complete each request. A
further indicator is found in the act of increasing the load on a
different one of the load generators. If the TPS experiences an
increase, this means that the ABT has successfully processed the
additional request and, therefore, the first of the load generators
has reached its full capacity and, accordingly, is not able to
produce additional effective requests.
[0010] Definitions
[0011] Central Console
[0012] The central console delivers centralized management of all
testing functions including selecting and defining test sessions,
developing test scripts, scheduling test sessions and graphically
and statistically monitoring results.
[0013] The Central Console
[0014] Configures Test Sessions.
[0015] Schedules test session scripts for simulated clients and
probing clients. Monitors the application's performance under the
generated load. Manages the test session as it is running Displays
the performance of the ABT
[0016] ABT (Application Being Tested)
[0017] The web application being tested
[0018] Load Generator
[0019] An entity that generates simulated clients. Load generators
have the task of bombarding the application being tested with HTTP
protocol call requests as defined in the scripts. The number of
simulated clients at any given moment is determined by the
user-defined session.
[0020] Load Generator Machine
[0021] The machine that hosts and runs one or multiple load
generators. Load generator machines are not limited to running a
single load generator.
[0022] Probing Client
[0023] An entity that simulates only one client at any time. A
probing client can run at the same time as loading testing. Probing
clients are used to obtain performance measurements that can be
more exact than a load generator machine running many simulated
clients.
[0024] Probing Client Machine
[0025] A machine that hosts and runs a probing client. Probing
client machines typically only run and host a single probing
client.
[0026] Simulated Clients
[0027] Entities generated by load generators and probing clients.
Each such entity is a simulation of a client accessing the
application being tested (ABT).
[0028] Clients
[0029] Persons or entities which access a website via a browser or
equivalent.
[0030] TPS (Transactions Per Second)
[0031] The number of HTTP transactions (such as a Get or Post)
submitted by a clients to the ABT in a defined time period, divided
by the number of seconds in the time period.
[0032] ART (Application Response Time)
[0033] The time (in seconds) required for the ABT to respond to a
request sent by a client, simulated client or probing client).
BRIEF DESCRIPTION OF THE DRAWING
[0034] The aforementioned aspects and other features of the
invention are explained in the following description, taken in
connection with the accompanying drawing figures wherein:
[0035] FIG. 1 is a stylized view of a test system for conducting
the procedure of the invention; and
[0036] FIG. 2 is a block diagram showing the procedure of the
invention.
[0037] Identically labeled elements appearing in different ones of
the figures refer to the same element but may not be referenced in
the description for all figures.
DETAILED DESCRIPTION OF THE INVENTION
[0038] In FIG. 1, a web application load testing system 10
comprises a plurality of load generators 12 which simulate the
loading of clients, the load generators 12 applying loads to a web
application 14. Loading of the web application 14 is also provided
by a probing client 16. The performance of the web application in
handling requests of the load generators 12 and of the probing
client 16 are evaluated by a Central Console 18. The system 10 is
operated in accordance with the procedure of the invention, as will
be described hereinafter with reference to FIG. 2, to establish the
capacity of the web application to process data requests of
numerous clients serviced via the web.
[0039] In establishing the capacity of the web application, a
question arises in the use of the load generation system 10 as to
how many simulated clients can be run on each of the load
generators 12. By way of example, one may decide, theoretically, to
run 10,000 such clients on a single machine, however the machine
may not perform the task adequately because each client consumes an
amount of system resources, which resources are limited. Even if
one could calculate in advance how many resources each simulated
client would consume, the capacity of the load generator 12 could
not be determined simply by dividing the system resources by the
number of the clients. This is because modem operating systems
allow processes to compete over resources. Accordingly, there is
need to find out how many simulated clients can be run on a given
machine while utilizing its full capacity of resources, without
interference among the responses to the respective clients.
[0040] A further consideration in the implementation of the
invention is the capacity of the web application, namely, how many
clients can access the web application until the web application
reaches a predefined limit or performance goal (in terms of
response time or other performance metrics). The load generation
system 10 addresses these considerations by implementation of the
procedure of the invention wherein the loading of the number of
clients is simulated by the load generators without requiring a
generator to exceed its capacity. In addition, the procedure of the
invention provides for detecting the point wherein the ABT reaches
its full capacity. By use of these procedural steps, in accordance
with the invention, the adaptive load generation frees a user of
the system 10 from the burden of conducting the simulation.
[0041] The adaptive load generation procedure makes use of an
existing web loading program, such as a product known as "WebLoad"
(available from RadView Software, Inc., Lexington, Mass.), for the
stress testing of applications. The WebLoad product employs the
elements shown in FIG. 1, namely, the load generator 12, the
Central Console 18, and the probing client 16. The load generator
12 is a multi threaded application that is able to simulate the
behavior of multiple clients. Each simulated client is implemented
as a separate thread, generating HTTP request according to a
predefined set of instructions called a "test script".
[0042] Several measurements of a web application's performance are
taken while each thread, of each of the respective clients, is
running. Such measurements may include response time and throughput
rate. These measurements are collected from all threads and packed
together to a statistics report. Several other measurements are
calculated based on the work of all threads, such as the number of
successful TPS, and are added to the other measurements in the
statistics report. The compilation of data and generation of such
reports is well known. These reports are sent to the Central
Console 18 every several seconds (as determined by the user),
achieving a real-time reporting of the performance of the web
application.
[0043] The probing client 16 shares the same functionality as one
load generator 12, but implements only one simulated client. The
measurements taken from the probing client 16 are more accurate
than those of the load generators 12 because there is only one
probing client per Probing Client machine, and the resources of the
machine are devoted to this simulated client.
[0044] Also, in each of the load generators 12, the measurements
sent are the averages of the measurements taken from all the
simulated clients running on a Load Generator machine. In the
manner of one of the load generators 12, the machine of the probing
client 16 reports every several seconds to the Central Console 18.
The Central Console 18 acts as a main control point for the load
generators 12 and the probing client 16, wherein it is set forth
how many load generators 12 and how many simulated clients may be
activated in any given time during a test session. The Central
Console 18 is able to start and to stop each of the load generators
12, and to change the number of simulated clients during a running
of the load generators 12. The monitor 18 also collects the
statistics reports sent from the load generators 12 and from the
probing client, and stores the reports in a statistics database for
reporting and for analysis. The ability of the system 10 to change
the load size dynamically during a test session, and also the fact
that the results are received in real time during the test, are an
important feature of the performance measurement procedure with the
adaptive load generation system 10.
[0045] With reference to FIG. 2, the procedure 20 for operation of
the system 10 in accordance with the invention is depicted as a set
of procedural steps represented by blocks in the diagram. The
procedure begins at 22 after which there is initialization of the
load generators and an activating of the probing client, as shown
at block 24. At block 26, a load generator is selected from a list
of available generators 12 (FIG. 1). The TPS generated from the
selected generator, together with the corresponding response time
measured on the probing client 16, are recorded and stored as shown
at block 28.
[0046] Then, at block 30, the load size is increased for the
generator by the generator by a predefined amount. At this point,
at block 32, a decision is made as to whether the performance goal
of the web application has been reached. In the event that the
performance goal has been reached, then it is understood there is
no further performance to be gained from the web application, and
the procedure is stopped at 34. In the event that the performance
goal of the web application has not been reached, the procedure
advances to block 36. Therein, after a delay which allows the
generator 10 to complete the load increase, and the Central Console
to collect a fresh statistics report from the generator and the
probing client, the TPS is compared to the value of TPS which was
stored previously prior to the step of the operation.
[0047] If the TPS has increased, this is an indication that the
increase in load size has increased also the effective load on the
ABT. Therefore, one can continue using the present load generator.
Accordingly, the procedure reverts back to block 26 wherein the
load generator is selected, and thereafter the procedure continues
into block 28. If there has been no increase in the TPS, then the
procedure advances to block 38 which provides for a measurement of
the response time of the web application in response to a request
by the probing client 16.
[0048] If the measurement of response time indicates an increase in
the response time, then the procedure advances to block 40 wherein
it is concluded that the increase in response time is due to the
fact that the ABT can not respond to more requests without
interfering with the processing of requests of other clients
machines. Thus, the ABT is considered to have reached its full
capacity. At this point in the procedure, a further check is made
at block 42 to determine if the performance goal of the web
application has been reached. If the performance goal of the web
application has been reached, then the procedure terminates at 44.
On the other hand, if the performance goal of the web application
has not been reached, then the procedure reverts to block 28 in
which further measurements are employed to verify whether or not
the ABT has, in fact, reached its full capacity.
[0049] In the event that the measurement of response time, at block
38, does not show an increase in the response time, the procedure
advances to block 46 where it is concluded that the assumed
increasing of loading of the web application by the load generator
10 has, in fact, not occurred because the generator is, apparently,
in its state of full capacity. Thus, there has, in fact, been no
effective load change. Accordingly, at block 46, the load size
generated by the load generator 10 is decreased back to the point
where the generator was effective. This generator is then removed
from the list of generators available for incrementing the load
size.
[0050] Thereupon, at block 48, a determination is made as to
whether the list of available load generators 12 is empty. In the
event that the answer is affirmative, that there are no more load
generators available, at block 50, then the procedure terminates at
52 because the maximum amount of loading of the web application 14
by the system 10 has been done. The system 10 has reached its full
ability to generate load, and the test is completed. However, at
block 48, in the event that the list is not empty, there is at
least one more load generator 12 available for loading the web
application 14. Then the procedure reverts back to block 26 wherein
a further one of the load be generators 12 is selected for loading
the web application 14.
[0051] The procedure advances through blocks 28 and 30 for further
loading and measurement of the results of the loading. With respect
to block 40, it is noted that one criteria for determining that the
ABT is in a state of full capacity is that the response time
exceeds three seconds, by way of example. Thus, following block 42,
even though the ABT is considered to be in its full state of
capacity based on the response time of three seconds, one can still
continue with the test to determine the web application performance
for a response time of four seconds, by way of example.
[0052] The foregoing procedure wherein the adaptive load generation
provides for a constant monitoring of a web application during an
increasing of the load size effects not only the web application
but also the load generators and the communication layer that
connects the web application to the load generators. In summarizing
the foregoing procedure, it is noted that the adaptive mode
generation enables a decision to be made when a given load
generator has reached its full capacity, followed by a continuing
of the test procedure with another one of the load generators. The
foregoing monitoring procedure also enables one to determine that
the ABT has reached its full capacity in accordance with a specific
criteria such as TPS, response time or CPU (Central processing
unit) utilization, by way of example. The monitoring information
can be collected from the load generators, the ABT, and also from
the probing claim 16 for which the probing client machine initiates
a single client accessing the web application.
[0053] In view of the foregoing description of the process of the
invention, it is apparent that the adaptive load generation enables
a user of the system of FIG. 1 to determine automatically the
status of the system, and thereby maximize utility of the web
application in responding to requests of client machines.
[0054] It is to be understood that the above described embodiment
of the invention is illustrative only, and that modifications
thereof may occur to those skilled in the art. Accordingly, this
invention is not to be regarded as limited to the embodiment
disclosed herein, but is to be limited only as defined by the
appended claims.
* * * * *