U.S. patent application number 14/305050 was filed with the patent office on 2014-10-02 for virtual machine management method and apparatus.
The applicant listed for this patent is FUJITSU LIMITED. Invention is credited to Tsunehisa Doi.
Application Number | 20140298338 14/305050 |
Document ID | / |
Family ID | 48781195 |
Filed Date | 2014-10-02 |
United States Patent
Application |
20140298338 |
Kind Code |
A1 |
Doi; Tsunehisa |
October 2, 2014 |
VIRTUAL MACHINE MANAGEMENT METHOD AND APPARATUS
Abstract
A non-transitory computer-readable recording medium has a
program stored therein for causing a computer to execute a process.
The process includes estimating a cost of executing a live
migration of a virtual machine, using a count value of an access
counter for counting the number of accesses to a memory allocated
to the virtual machine, a capacity of the memory, and a bandwidth
of data transfer between physical machines relating to the live
migration.
Inventors: |
Doi; Tsunehisa; (Kawasaki,
JP) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
FUJITSU LIMITED |
Kawasaki-shi |
|
JP |
|
|
Family ID: |
48781195 |
Appl. No.: |
14/305050 |
Filed: |
June 16, 2014 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
PCT/JP2012/050273 |
Jan 10, 2012 |
|
|
|
14305050 |
|
|
|
|
Current U.S.
Class: |
718/1 |
Current CPC
Class: |
G06F 9/4856 20130101;
G06F 2009/4557 20130101; G06F 9/5016 20130101; G06F 9/45558
20130101; G06F 9/5088 20130101 |
Class at
Publication: |
718/1 |
International
Class: |
G06F 9/50 20060101
G06F009/50 |
Claims
1. A non-transitory computer-readable recording medium having a
program stored therein for causing a computer to execute a process,
the process comprising: estimating a cost of executing a live
migration of a virtual machine, using a count value of an access
counter for counting the number of accesses to a memory allocated
to the virtual machine, a capacity of the memory, and a bandwidth
of data transfer between physical machines relating to the live
migration.
2. The non-transitory computer-readable recording medium as claimed
in claim 1, wherein the cost is a memory transfer time.
3. The non-transitory computer-readable recording medium as claimed
in claim 1, the process further comprising: excluding the virtual
machine from execution targets of the live migrations, or stopping
execution of the live migration if the live migration of the
virtual machine is being executed, based on a result of comparing
the estimated cost with a predetermined threshold value.
4. The non-transitory computer-readable recording medium as claimed
in claim 1, the process further comprising: determining an
execution plan of the live migrations of a plurality of the virtual
machines, based on the estimated costs; and executing the live
migrations based on the allocation plan.
5. A method of managing a live migration of a virtual machine,
executed by a computer, the method comprising: estimating a cost of
executing the live migration of the virtual machine, using a count
value of an access counter for counting the number of accesses to a
memory allocated to the virtual machine, a capacity of the memory,
and a bandwidth of data transfer between physical machines relating
to the live migration.
6. The method as claimed in claim 5, wherein the cost is a memory
transfer time.
7. The method as claimed in claim 5, further comprising: excluding
the virtual machine from execution targets of the live migrations,
or stopping execution of the live migration if the live migration
of the virtual machine is being executed, based on a result of
comparing the estimated cost with a predetermined threshold
value.
8. The method as claimed in claim 5, further comprising:
determining an execution plan of the live migrations of a plurality
of the virtual machines, based on the estimated costs; and
executing the live migrations based on the allocation plan.
9. An apparatus for managing a live migration of a virtual machine,
comprising: a cost estimation unit configured to estimate a cost of
executing the live migration of the virtual machine, using a count
value of an access counter for counting the number of accesses to a
memory allocated to the virtual machine, a capacity of the memory,
and a bandwidth of data transfer between physical machines relating
to the live migration.
10. The apparatus as claimed in claim 9, wherein the cost is a
memory transfer time.
11. The apparatus as claimed in claim 9, further comprising: an
allocation control unit configured to exclude the virtual machine
from execution targets of the live migrations, or to stop execution
of the live migration if the live migration of the virtual machine
is being executed, based on a result of comparing the estimated
cost with a predetermined threshold value.
12. The apparatus as claimed in claim 9, further comprising: an
allocation planning unit configured to determine an execution plan
of the live migrations of a plurality of the virtual machines,
based on the estimated costs; and a migration execution unit
configured to execute the live migrations based on the allocation
plan.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation application of
International Application PCT/JP2012/050273 filed on Jan. 10, 2012
and designated the U.S., the entire contents of which are
incorporated herein by reference.
FIELD
[0002] The disclosures herein generally relate to a program, a
method, and an apparatus for virtual machine management.
BACKGROUND
[0003] In recent years, virtual machine technology (virtualization
technology) has been actively used for running virtual machines
(VM) on physical machines (PM). By using the virtual machine
technology, multiple virtual machines can run on a single physical
machine. And each of the virtual machines can run an operating
system and/or an application that may be different from those
running on the other virtual machines.
[0004] By introducing the virtual machine technology on a server,
multiple virtualized servers can be aggregated in one physical
server. This aggregation is reasonable, and results in cost
reduction. However, for example, there are cases where one of the
aggregated virtual machines may adversely influence the other
virtual machines when load states of the virtual machines change.
Therefore, live migration technology has been developed in the
virtual machine technology that migrates a virtual machine to
another physical machine without stopping services provided by the
virtual machine.
[0005] An overview of the live migration technology will be
described. Live migration methods can be classified into the
pre-copy method and the post-copy method. When executing a live
migration, switching needs to be executed for a virtual machine to
be migrated, at least for its contents of the physical memory,
storage, and network. The pre-copy method is a method that moves
the state of a CPU after having the content of the physical memory
moved. In contrast to the above, the post-copy method moves the
state of the CPU without moving the content of the physical memory.
In this case, although the content of the physical memory has not
been moved to a destination machine, the address translation table
of the virtual memory system on the destination machine is set to
an empty state. With this setting, when the CPU accesses the memory
on the destination machine, data to be accessed does not exist in
the physical memory at an initial stage. Therefore, a page fault
occurs, which causes required pages loaded from a hard disk to fill
the physical memory.
[0006] Note that a large-capacity memory does not need to be moved
on the hard disk because the large-capacity memory on the hard disk
is shared when using either of the methods.
[0007] There is a conventional technology that determines whether a
live migration is feasible when a live migration is requested, and
stops the live migration if a negative determination is obtained
(see, for example, Patent Document 1). This technology assigns an
attribute of safety level to each VM to prevent a low safety level
VM and a high safety level VM from running on the same computer.
This avoids a circumstance where execution of a high safety level
VM is adversely influenced by a defect of a low safety level
VM.
[0008] Also, there is a technology that determines necessity of a
migration based on load information just before executing the
migration to avoid unnecessary migrations (see, for example, Patent
Document 2). Note that, as for the load information, Patent
Document 2 only discloses CPU usage rates and memory usage rates of
a source server, a destination server, and a virtual machine.
[0009] Also, there is a technology that estimates time required for
movement for a virtual machine to move from a physical machine, on
which the virtual machine is currently running, to another physical
machine (see, for example, Patent Documents 3-4). Patent Documents
3 and 4 disclose that memory transfer time can be estimated using a
memory change rate.
[0010] Also, there is a technology that determines a degree of
difference between two consecutive frames of images using a counter
value that counts the number of write accesses to a frame memory of
a camera (see, for example, Patent Document 5).
[0011] A rate of memory overwrites during a memory transfer in a
live migration (namely, a rate of copy operations that need to be
executed again) is called a "dirty rate" (or "memory change rate").
The dirty rate is a value obtained by dividing an amount of memory
change by an amount of memory transfer. If the dirty rate is too
high, there may be a case where a memory transfer cannot be
completed. To cope with such a case, an implementation may be
adopted that stops the context to forcibly end a memory transfer.
In this case, likelihood is high for service outage time to exceed
an acceptable range. It should be avoided to exceed the limit of
service outage time that is specified in a service level agreement
(SLA) with a customer who uses a service of virtual machines.
Therefore, to avoid such a long service outage, there is a case
where load distribution is secured by stopping a live migration of
a virtual machine to be migrated, and instead, executing a live
migration of another virtual machine.
[0012] Therefore, it is necessary to avoid a state where a context
stop continues for a long time due to a live migration, or stoppage
of a live migration in advance. To achieve these requirements, it
is necessary to keep load of a physical machine at a low level in
usual operations before a live migration, and to obtain a memory
dirty rate so that a memory transfer time can be predicted for
executing a live migration.
[0013] Note that a prediction value T of a memory transfer time
required for a live migration is obtained by the following formula
where M is a memory capacity, tp is a data transmission band width
(transfer throughput) used for a live migration, and r is a dirty
rate (a value obtained by dividing an amount of memory change by an
amount of memory transfer).
T=M/{tp(1-r)} (1)
[0014] Therefore, the memory transfer time for a live migration can
be estimated if the dirty rate in usual operations is obtained.
[0015] However, there is an overhead increase to detect the dirty
rate in general. A method of detecting the dirty rate for a live
migration is as follows. First, an area that has been copied is set
as a write-protected area at a hardware level. Then, a write
request to the write-protected area is trapped at a software level.
The trap triggers a predetermined routine to operate for detecting
the write to the specific area, and for storing information about
the write. This process makes the routine operate every time a
write is trapped, which causes an increase in overhead. Therefore,
if this process is used for detecting the dirty rate in usual
operations, a considerable overhead is inevitably generated. This
makes the load greater for the usual operations of the virtual
machine.
[0016] Therefore, a technology has been needed for quickly
obtaining estimation values of a dirty rate and a memory transfer
time for a pre-copy in usual operation while a live migration is
not being executed.
RELATED-ART DOCUMENTS
Patent Documents
[0017] [Patent Document 1] Japanese Laid-open Patent Publication
No. 2010-238044
[0018] [Patent Document 2] Japanese Laid-open Patent Publication
No. 2011-108014
[0019] [Patent Document 3] Japanese Laid-open Patent Publication
No. 2011-138184
[0020] [Patent Document 4] Japanese Patent Application No.
2010-233020
[0021] [Patent Document 5] Japanese Laid-open Patent Publication
No. 2006-243940
SUMMARY
[0022] According to at least one embodiment of the present
invention, a non-transitory computer-readable recording medium has
a program stored therein for causing a computer to execute a
process. The process includes estimating a cost of executing a live
migration of a virtual machine, using a count value of an access
counter for counting the number of accesses to a memory allocated
to the virtual machine, a capacity of the memory, and a bandwidth
of data transfer between physical machines relating to the live
migration.
[0023] The object and advantages of the embodiment will be realized
and attained by means of the elements and combinations particularly
pointed out in the claims. It is to be understood that both the
foregoing general description and the following detailed
description are exemplary and explanatory and are not restrictive
of the invention as claimed.
BRIEF DESCRIPTION OF DRAWINGS
[0024] FIG. 1 is a schematic view illustrating an example of a live
migration of a virtual machine;
[0025] FIG. 2 is a schematic view illustrating a process of copying
a memory;
[0026] FIG. 3 is a schematic view illustrating an overall
configuration of apparatuses relating to an embodiment of the
present invention;
[0027] FIG. 4 is a schematic view illustrating an embodiment;
[0028] FIG. 5 is a schematic view illustrating an embodiment;
[0029] FIG. 6 is a schematic view illustrating an embodiment;
[0030] FIG. 7 is a schematic view illustrating an embodiment;
[0031] FIG. 8 is a schematic view illustrating a process flow in an
allocation management unit;
[0032] FIG. 9 is a schematic view illustrating a process executed
on a counter in a VM management unit;
[0033] FIG. 10 is a schematic view illustrating an example of a
plan determination and an execution of live migrations;
[0034] FIG. 11 is a schematic view illustrating another example of
a series of steps of a live migration;
[0035] FIG. 12 is a schematic view illustrating a functional block
diagram according to an embodiment; and
[0036] FIG. 13 is a schematic view illustrating a hardware
configuration of a server and a VM management apparatus.
DESCRIPTION OF EMBODIMENTS
[0037] In the following, embodiments of the present invention will
be described with reference to the drawings. It should be noted
that the following embodiments are provided for understanding the
present invention, not for limiting the scope of the present
invention. Also, the multiple embodiments in the following
description are not mutually exclusive. Therefore, it is noted that
any combinations of different embodiments can be realized unless
any contradictions arise. Also, steps of any one of the methods or
programs described in claims may be executed in a different order
of the steps, or multiple steps may be executed concurrently.
Further, it is obvious that these embodiments are included within a
technological scope of the claims.
[0038] A pre-copy method is assumed to be adopted in the
embodiments of the present invention.
[0039] An overview of the pre-copy method will be described in the
following.
[0040] FIG. 1 illustrates an example of a live migration of a
virtual machine B111. The virtual machine B111 runs on the server
110, which is a physical machine, under control of a VM management
unit 115. The virtual machine B111 manages its own virtual memory
space. A physical memory area in the physical memory of the server
110 is referred to as a memory A112 that is included in the virtual
memory managed by the virtual machine B111. A case will be
described where a live migration is executed in this state so that
the virtual machine B111 is migrated to the server 120 and runs as
a virtual machine B'121 on the server 120. The server 120 includes
a VM management unit 125. Also, a physical memory area in the
physical memory of the server 120 is referred to as a memory A'122
that is included in the virtual memory managed by the virtual
machine B'121, which is the area to be copied.
[0041] FIG. 2 illustrates a process of copying a physical memory.
To execute a live migration using the pre-copy method, first, it is
necessary to copy the content of the physical memory A112 managed
by the virtual machine B111 to the physical memory A'122 on the
server 120 (as described above, a virtual memory area that has
swapped out to a hard disk does not need to be copied because both
servers share the hard disk). At time t1, the content of the memory
A112 differs from the content of the memory A'122.
[0042] At time t2, a copy operation starts for copying overall
information 222 of the content of the memory A112 to the memory
A'122. During the copy operation, the content of the memory A112 is
being overwritten.
[0043] At time t3, a copy operation starts for copying data 224
that has been overwritten during the immediately preceding transfer
to the memory A'122. During the copy operation, the content of the
memory A112 is being overwritten.
[0044] At time t4, a copy operation starts for copying data 226
that has been overwritten during the immediately preceding transfer
to the memory A'122. During the copy operation, the content of the
memory A112 is being overwritten.
[0045] At time t5, a copy operation starts for copying data 228
that has been overwritten during the immediately preceding transfer
to the memory A'122. During the copy operation, the content of the
memory A112 is being overwritten.
[0046] The above operations are repeated until an estimated copy
time for remaining data becomes, for example, less than or equal to
one second (or the amount of remaining data to be copied becomes
less than or equal to a predetermined amount). At time tn, if the
estimated copy time for remaining data becomes less than or equal
to one second, the context of the virtual machine B111 is stopped.
Then, the remaining copy part is copied from the memory A112 to the
memory A'122, and the register information of the CPU of the
virtual machine B111 is copied to the virtual machine B'122. Also,
the storage and network are switched.
[0047] After that, the virtual machine B'121 is started. By these
operations, a live migration of the virtual machine B111 is
executed from the server 110 to the server 120.
[0048] As above, multiple copy operations are required for making a
copy without a difference because the memory of a running virtual
machine is kept on changing.
Embodiments
[0049] FIG. 3 illustrates an overall configuration of apparatuses
relating to the embodiments of the present invention. As
illustrated in FIG. 3, a VM management apparatus 310, a server 1, a
server 2, and a server 3 are connected with each other by a network
NW. These are all physical machines. Virtual machines VM1, VM2, and
VM3 run on the server 1. These virtual machines are managed by a VM
management unit 1. The VM management unit 1 holds operational
states of the respective virtual machines, and provides a virtual
machine environment, which includes an external interface, for the
virtual machines. Also, the VM management unit 1 may execute a live
migration in cooperation with the VM management apparatus 310.
Similarly, virtual machines VM4, VM5, and VM6 run on the server 2
managed by a VM management unit 2. The server 3 includes a VM
management unit 3. FIG. 3 illustrates a state where the virtual
machines VM1 and VM2 on the server 1 are being prepared for a live
migration to the server 3.
[0050] The VM management apparatus 310 manages all virtual machines
that exist on the server 1, the server 2, and the server 3. The
allocation management unit 312 makes a plan of dynamic relocation
of the virtual machines based on operational states of the
respective virtual machines, operational states of the respective
physical machines, and information including a command of an
operator, and then, executes the plan with live migrations. An
allocation management unit 312 may also monitor a state of a live
migration to stop the live migration if necessary.
[0051] Note that the physical machines in FIG. 3 do not need to be
contained in different housings. Also, the allocation management
unit 312 may exist in the physical machines where the virtual
machines exist. Also, the physical machines may execute information
transmission with a communication function using a medium other
than the network NW. Alternatively, the network NW may transmits
information in cooperation with the other communication function.
By making the bandwidth greater with multiple paths for information
transmission, time required for a live migration can be
shortened.
[0052] FIG. 4 illustrates an embodiment of the present invention. A
functional block diagram and a hardware block diagram of the server
1 are illustrated in the upper and lower halves of FIG. 4,
respectively. Note that the diagrams are arranged vertically with
upward arrows to designate data flow in FIG. 4.
[0053] The VM management unit manages virtual machines VM1, VM2,
and VM3. The VM management unit includes counters 401, 402, and 403
for counting memory accesses to the virtual machines VM1, VM2, and
VM3, respectively. The server 1 may include a CPU 440, a physical
memory 460, and an access counter 450 as illustrated in the
hardware block diagram. The access counter 450 is implemented in
hardware.
[0054] Here, hardware means hardware including wired logic and/or a
microprogram.
[0055] Although the access counter 450 is provided adjacent to the
physical memory 460 in the hardware block diagram, the access
counter 450 may be positioned at another place such as a portion in
the CPU 440. The VM management unit may periodically read the
access counter 450 using timer interruptions. Based on the value of
the access counter 450, the VM management unit counts and
accumulates the number of accesses to each of the VM1, VM2, and
VM3, and holds the numbers in the counters 401, 402, and 403,
respectively. As a result, the counters 401, 402, and 403 store the
number of accesses to the VM1, VM2, and VM3, respectively. Note
that the access counter 450 may be reset every time a virtual
machine is switched (every time a context is switched).
[0056] Note that information about a specific overwritten location
in the memory cannot be identified because there is only one access
counter 450 in the present embodiment. Therefore, assuming that
memory transfer in a live migration is executed by units of pages,
the dirty rate r of the physical memory managed by the VM1 is
represented by the following formula where a represents a weight, W
represents the capacity of a page, n represents the number of
memory writes per unit time recorded in the counter 401, and tp
represents the transfer amount per unit time (namely,
throughput).
r=.alpha.nW/tp (2)
[0057] Note that the weight .alpha. is provided for taking multiple
writes in the same page into consideration. Namely, if multiple
writes are done on a specific page, only this page needs to be
transferred again. Therefore, the weight .alpha. is less than
one.
[0058] Therefore, by substituting formula (2) for r in formula (1),
expected transfer time T.sub.VM1 of the memory managed by the VM1
is represented by the following formula.
T.sub.VM1=M/{tp(1-.alpha.nW/tp)} (3)
[0059] Note that if the access counter 450 counts the number of
accesses for both writes and reads, and the rate of writes is
assumed to be .beta. for the total number of counted accesses,
expected transfer time T.sub.VM1 of the memory managed by the VM1
is represented by the following formula.
T.sub.VM1=M/{tp(1-.beta..alpha.nW/tp)} (4)
[0060] This transfer time T.sub.VM1 is used as an estimated value
of memory transfer time for a live migration. Transfer time
T.sub.VM1 may be recognized as an estimated cost for a live
migration of the VM1 because it represents a resource consumed as
time.
[0061] The weight a above relates to a degree of dispersion of
memory access. Here, .alpha. and .beta. may be determined
beforehand by statistically measuring memory access while a virtual
machine is running. Also, these values may be determined beforehand
for respective virtual machines. Also, these values may be
determined beforehand for respective applications that run on
virtual machines.
[0062] The counters 401, 402, and 403 accumulate and store the
number of accesses to the VM1, VM2, and VM3, respectively. The
number of accesses per unit time may be calculated by obtaining the
difference of counter values between the start of a unit time and
the end of the unit time. Alternatively, the value of a counter is
read at unit time intervals, and the counter is reset after
respective reads. Contents of the counters may be used at the
allocation management unit 312. Note that operations of the
allocation management unit 312 will be described later using FIGS.
10-11.
[0063] FIG. 5 illustrates another embodiment of the present
invention. The CPU 440 includes multiple performance counters for
measuring various kinds of performance. An access counter 550 may
be used, which is a performance counter for counting write accesses
to the physical memory 460. Other elements in FIG. 5 are the same
as those in FIG. 4, and assigned the same numerical codes. The
access counter 550 may be used in the same way as the access
counter 450 in FIG. 4. The access counter 550 is a hardware
counter; hence load of the CPU 440 is very small when using the
access counter 550. Note that hardware in this specification means
hardware including hard-wired logic and/or a microprogram.
Therefore, during usual operations while a live migration is not
being executed, an estimated value (estimated cost) of memory
transfer time can be obtained by formula (3) described above. In
contrast to this, as described above, overhead increases if the
method is introduced that puts the memory into a write-protected
state to trap and count memory write accesses in software during
usual operations. This overhead makes the load greater for usual
operations. In respect of the overhead, the process of calculating
an estimated value of a memory transfer in the present embodiment
has a great advantage in that the process has virtually no
influence on usual operations.
[0064] In addition, by obtaining an estimated cost of a live
migration for each virtual machine in this way, an execution plan
for live migrations can be determined more precisely. Note that
determination of an execution plan will be concretely described
later using FIGS. 10-11.
[0065] FIG. 6 illustrates yet another embodiment of the present
invention. Note that the same elements as in FIGS. 4-5 are assigned
the same numerical codes. Multiple access counters 650 are memory
access counters that are provided for respective virtual machines,
and are implemented in hardware. Therefore, an access counter only
for one of the VMs, for example, the VM1, exits among multiple
access counters 650. Therefore, the VM management unit reads the
content of an access counter that corresponds to one of the virtual
machines in the present embodiment. Then, the read value is
accumulated in a counter in the VM management unit that corresponds
to one of the virtual machines.
[0066] If the access counter 650 is to count write accesses, an
estimated value of memory transfer time can be calculated using
formula (3). Alternatively, if the access counter 650 is to count
read and write access, an estimated value of memory transfer time
can be calculated using formula (4).
[0067] FIG. 7 further illustrates another embodiment of the present
invention where access counters 751, 752, 753, . . . , and 799 are
provided in the physical memory 460 by units of pages. These access
counters may be provided, for example, for pages that are used for
memory access management in a virtual memory system. Such a page
may have the same unit size as the transfer unit size of a memory
swap between the physical memory 460 and a dynamic storage device
(not shown). Alternatively, the page may have the same unit size as
the memory transfer unit size of a live migration. If the multiple
access counters 751, 752, 753, . . . , and 799 are aggregated to be
treated as one counter, the same process as in the first embodiment
illustrated in FIG. 4 may be executed. Namely, values of the access
counters for respective memory areas are summed as the number of
memory accesses to respective virtual machines, which is then
accumulated in the counter that corresponds to one of the virtual
machines.
[0068] In the present embodiment, the number of memory accesses to
each virtual machine is accumulated in a corresponding counter in
the VM management unit. An estimated value of memory transfer time
can be similarly obtained using formulas (3) or (4).
[0069] Note that the following process may be executed in the
present embodiment as will be described with the VM1 taken as an
example.
[0070] First, the following preconditions are assumed. [0071] (1)
The physical memory 460 has L pages (where L denotes the number of
pages). Instead of counters 401, 402, and 403 that correspond to
respective virtual machines in the VM management unit, L counters
(not shown) are provided in the VM management unit, which
corresponds to the L access counters. The L counters have initial
values of zero.
[0072] Then, the following steps are executed using timer
interruptions. [0073] (A) Read each of the L access counters
periodically using timer interruptions, and accumulate the read
values in the L corresponding counters in the VM management unit.
[0074] (B) Using timer interruptions, read counters in the VM
management unit for pages allocated to the VM1 at unit time
intervals (for example, the unit time used for calculating a dirty
rate), and examine the number of counters whose values are greater
than zero (for example, j counters may be found). Examine the other
virtual machines similarly. Reset the L counters in the VM
management unit to zero.
[0075] In the above process, for example, a memory dirty rate
r.sub.VM1 for the VM1, which is obtained by dividing a memory
change amount by a transfer amount, may be obtained by the
following formula where W represents the capacity of a page, and tp
represents a transfer amount per unit time (namely,
throughput).
r.sub.VM1=jW/tp (5)
[0076] This dirty rate r.sub.VM1 may fluctuate depending on time
when data is obtained. Therefore, obtained values may be
accumulated to obtain a moving average to be used as the dirty
rate. Note that if an architecture is adopted in the physical
memory 460 in which the number of pages of the physical memory
allocated to each virtual machine is dynamically changed, care
should be taken that this dirty rate r.sub.VM1 is an approximate
value.
[0077] Then, an estimated value of memory transfer time can be
calculated from this dirty rate using formula (1).
[0078] One skilled in the art may come up with other modified
examples from the above embodiments.
[0079] In the above embodiments, values of the counters in the VM
management unit may be transmitted to the allocation management
unit 312 in the VM management apparatus 310. The allocation
management unit 312 may calculate an estimated value of memory
transfer time.
[0080] As described above, counters for memory access are used in
the above embodiments. As another modified example, a performance
counter may be used that counts the number of references to a TLB
(Translation Look aside Buffer) (not shown). When a CPU accesses a
memory, a virtual address needs to be translated into a physical
address. Therefore, a virtual memory system has an address
translation table in which a virtual address is associated with a
physical address. A TLB is a cache of such an address translation
table, which can be accessed faster. Therefore, the CPU refers to
the TLB to translate a virtual memory address into a physical
memory address every time memory access is executed. Therefore, the
number of references to the TLB is equivalent to, or highly
correlated with the number of accesses. Therefore, if using a
counter that counts the number of references to the TLB,
substantially the same method can be adopted as in the embodiment
illustrated in FIG. 4 or 5.
[0081] FIG. 8 illustrates a process flow in the allocation
management unit 312.
[0082] At Step 802, the number of memory accesses is obtained for
each VM and accumulated in the VM management unit.
[0083] At Step 804, an estimated value of memory transfer time is
calculated for the virtual machine by formula (1), (3), or (4). The
estimated value of memory transfer time may be called the
"estimated cost" for a migration.
[0084] The estimated cost may be output to a display unit at Step
820. Note that the displaying may be executed with an arbitrary one
of the steps.
[0085] At Step 806, the estimated cost may be compared a
predetermined threshold value. If it is determined "YES", the
process goes forward to Step 808. If "NO", the process goes back to
Step 802.
[0086] At Step 808, if the virtual machine has the estimated cost
that exceeds the threshold value, the virtual machine may be
excluded for a live migration. Alternatively, the virtual machine
may be included in a list of virtual machines that are excluded for
live migrations, which is used later for determining an allocation
plan. In addition, if a live migration is being executed for the
virtual machine, the live migration may be stopped.
[0087] At Step 810, it is determined whether any virtual machines
are left to be processed. If it is determined "NO", all virtual
machines have processed. If it is determined "YES", the process
goes back to Step 802, and a next virtual machine is processed.
Note that Step 820 may display various kinds of output information
including a list of virtual machines excluded for live migrations
and memory dirty rates.
[0088] The process in FIG. 8 may be triggered by a timer
interruption. Or it may be executed by a command from an
operator.
[0089] FIG. 9 is a schematic view illustrating a process executed
on a counter in the VM management unit. This process may be
triggered by a timer interruption.
[0090] Step 902 indicates that an interruption is generated by a
timer that triggers this process.
[0091] At Step 904, the memory access counter is read. As described
above, the memory access counter may take one of the various forms
depending on the architecture of the physical machine.
[0092] At Step 906, a virtual machine is identified that runs at
the very time when the memory access counter is read. It is
identified by the VM management unit.
[0093] At Step 908, the read value is loaded in the counter of the
corresponding virtual machine. The interval between reads of the
access counter can be changed by setting the interval between timer
interruptions. It is desirable to have the interval between timer
interruptions shorter than an interval of virtual machine
switching. By setting the interval in this way, it is possible to
recognize how much the value of the access counter increases while
a virtual machine runs. The increased value may be accumulated in
the corresponding counter in the VM management unit. If it is
possible to reset the access counter, it may be reset when the
virtual machine switches. Alternatively, if the access counter can
be set with a value, the value of the counter in the VM management
unit is set when the virtual machine switches, and the access
counter may continue to count up until it is switched with another
virtual machine. Then, the result of counting up of the access
counter may be overwritten in the counter in the VM management
unit. Also, this process may be executed when the virtual machine
switches.
[0094] Thus, the process triggered by the timer interruption is
completed, and a process before the interruption may be
resumed.
[0095] FIG. 10 illustrates an example of a plan determination and
an execution of live migrations executed by the allocation
management unit 312.
[0096] At Step 1002, it is determined whether a live migration is
required. Factors for the determination include a command of an
operator, a rise of operation rate on a specific physical machine,
a planned rise of operation rate, maintenance of a physical
machine, a version upgrade of the VM management unit, an
aggregation of virtual machines to a physical machine for reducing
power consumption, and a correlation with the operation rate on the
specific physical machine. If the determination result is "NO", the
process ends. If the determination result "YES", the process goes
to Step 1004. In the following description of the process, FIG. 3
and FIG. 12 are also referred to.
[0097] At Step 1004, an allocation plan is determined by an
allocation planning unit 1212 in the allocation management unit 312
(1210). As an example of an allocation plan, virtual machines are
reordered in ascending order of estimated costs (estimated values
of memory transfer time). Then, the allocation plan is determined
so that live migrations will be started with a first virtual
machine in this order. As a destination physical machine of
movement, a physical machine may be selected that has the least
operation rate among physical machines, or it may be predetermined.
Or, it may be specified by an operator.
[0098] At Step 1006, live migrations are executed based on the
determined allocation plan. As described with reference to FIG. 8,
if the estimated cost exceeds the threshold value during execution
of a live migration, the live migration may be stopped.
[0099] FIG. 11 illustrates another example of a series of steps of
a live migration.
[0100] At Step 1102, a request for a live migration is generated,
and this process is started.
[0101] At Step 1104, an access frequency is calculated from the
read value of the memory access counter for each virtual
machine.
[0102] At Step 1106, the dirty rate is estimated from the access
frequency for each of the virtual machines.
[0103] At Step 1108, an estimated value of memory transfer time,
namely, an estimated cost of a live migration is calculated from at
least the memory amount, the dirty rate, and the transfer
bandwidth.
[0104] At Step 1110, the estimated live migration costs of the
virtual machines are sorted in ascending order.
[0105] At Step 1112, virtual machines having lower estimated costs
are tentatively prioritized to be migrated. Other costs are checked
that include correlations with the other virtual machines,
importance of stable operation, and fluctuation factors of the cost
in the future.
[0106] At Step 1114, an allocation plan is created so that a live
migration order is determined considering appropriate costs and
benefits obtained at Step 1112. Note that to select an appropriate
order, an evaluation function may be determined beforehand that
takes various factors at Step 1112 into consideration, and the
allocation plan may be determined that includes an optimal live
migration order based on output results of the evaluation
function.
[0107] At Step 1116, live migrations are executed in order based on
the allocation plan.
[0108] At Step 1118, as already described with reference to FIG. 8,
the live migration may be stopped even if the live migration is
being executed as long as the estimated cost is great.
[0109] Thus, the process of live migrations is executed. Note that
the steps described above may be executed in a different order of
the steps, or multiple steps may be executed concurrently unless
any contradictions arise.
[0110] FIG. 12 illustrates a functional block diagram according to
an embodiment of the present invention.
[0111] An access counter 1202 is an access counter for a
memory.
[0112] A counter accumulation unit 1204 is a set of counters that
accumulates and stores the contents of plural of the access
counters 1202, for example, for respective virtual memories.
[0113] A cost estimation unit 1206 calculates an estimated value of
memory transfer time from a value of the counter accumulation unit
1204, a transfer bandwidth, and a physical memory capacity of a
virtual machine, as an estimation of the cost of a live migration.
The cost estimation unit 1206 may be a part of the allocation
management unit 1220 below.
[0114] A display unit 1208 may display the estimated cost and dirty
rate of each virtual machine, an allocation plan, and a state of
progression of live migrations.
[0115] The allocation management unit 1220 includes the allocation
control unit 1210, the allocation planning unit 1212, and a
migration execution unit 1214. In addition, the cost estimation
unit 1206 may be included as described above.
[0116] The allocation planning unit 1212 may tentatively prioritize
live migrations having smaller estimated costs for migration. Other
costs may be checked that include correlations with the other
virtual machines, importance of stable operation, and fluctuation
factors of the cost in the future. Further, an allocation plan is
created so that live migration order is determined considering
overall costs and benefits. Note that to select an appropriate
order, an evaluation function may be determined beforehand that
takes various factors into consideration, and the allocation plan
may be determined that includes an optimal live migration order
based on output results of the evaluation function.
[0117] The migration execution unit 1214 executes migrations based
on the allocation plan.
[0118] The allocation control unit 1210 manages migrations as a
whole. For example, the allocation control unit 1210 may stop a
live migration even if the live migration is being executed as long
as the estimated cost is great.
[0119] FIG. 13 illustrates a hardware configuration of servers and
the VM management apparatus 310, which are physical machines.
[0120] The physical machine may include a CPU 1302, a ROM 1304, a
drive unit 1306, a RAM 1310, a network controller 1312, and an HDD
1314. These units are connected with each other by a bus 1316.
[0121] The CPU 1302 may include a performance counter 1303. Also,
the drive unit 1306 can read and write to a recording medium 1308
in which programs are stored. A program read from the recording
medium 1308 can be executed by the CPU 1302.
[0122] Note that the program may be stored in the portable
recording medium 1308. The portable recording medium 1308 is one or
more non-transitory storage media having a structure. For example,
the portable recording medium 1308 may be a magnetic storage
medium, an optical disk, an optical-magnetic storage medium, a
non-volatile memory, or the like. A magnetic storage medium may be
an HDD, a flexible disk (FD), a magnetic tape (MT), or the like. An
optical disk may be a DVD (Digital Versatile Disc), a DVD-RAM, a
CD-ROM (Compact Disc-Read Only Memory), a CD-R (Recordable)/RW
(ReWritable), or the like. Also, an optical-magnetic storage medium
may be an MO (Magneto-Optical disk), or the like.
[0123] All examples and conditional language recited herein are
intended for pedagogical purposes to aid the reader in
understanding the invention and the concepts contributed by the
inventor to furthering the art, and are to be construed as being
without limitation to such specifically recited examples and
conditions, nor does the organization of such examples in the
specification relate to a showing of the superiority and
inferiority of the invention. Although the embodiments of the
present invention have been described in detail, it should be
understood that the various changes, substitutions, and alterations
could be made hereto without departing from the spirit and scope of
the invention.
* * * * *