U.S. patent application number 14/053094 was filed with the patent office on 2014-04-17 for inter-memory data transfer control unit.
This patent application is currently assigned to HITACHI, LTD.. The applicant listed for this patent is HITACHI, LTD.. Invention is credited to Masahiro ARAI, Hideaki FUKUDA, Hiroshi HIRAYAMA, Hiroshi KANAYAMA, Masanori TAKADA.
Application Number | 20140104967 14/053094 |
Document ID | / |
Family ID | 43501394 |
Filed Date | 2014-04-17 |
United States Patent
Application |
20140104967 |
Kind Code |
A1 |
ARAI; Masahiro ; et
al. |
April 17, 2014 |
INTER-MEMORY DATA TRANSFER CONTROL UNIT
Abstract
A data transfer device that transfers data to a memory according
to an instruction from a processor via a bus through which a
response indicating completion of data writing in the memory is not
sent back, comprises an inter-memory data transfer control unit
including an operation start trigger receiving unit, a parameter
acquiring unit, a read unit, and a write unit. When the write unit
detects switching of a write destination memory, the write unit
confirms write completion as to the memory by a procedure different
from writing. When a data transfer instructed by the processor is
completed, the write unit confirms write completion as to the write
destination memory at the end of the data transfer by the procedure
different from writing. The inter-memory data transfer control unit
notifies the processor of completion of an inter-memory data
transfer based on the confirmation of the write completion.
Inventors: |
ARAI; Masahiro; (Mashida,
JP) ; HIRAYAMA; Hiroshi; (Yokohama, JP) ;
TAKADA; Masanori; (Yokohama, JP) ; KANAYAMA;
Hiroshi; (Odawara, JP) ; FUKUDA; Hideaki;
(Odawara, JP) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
HITACHI, LTD. |
Tokyo |
|
JP |
|
|
Assignee: |
HITACHI, LTD.
Tokyo
JP
|
Family ID: |
43501394 |
Appl. No.: |
14/053094 |
Filed: |
October 14, 2013 |
Related U.S. Patent Documents
|
|
|
|
|
|
Application
Number |
Filing Date |
Patent Number |
|
|
12747086 |
Mar 30, 2011 |
8572342 |
|
|
PCT/JP2010/003661 |
Jun 1, 2010 |
|
|
|
14053094 |
|
|
|
|
Current U.S.
Class: |
365/189.17 |
Current CPC
Class: |
G06F 13/28 20130101;
G11C 7/10 20130101; G06F 13/4282 20130101; G06F 2213/0026
20130101 |
Class at
Publication: |
365/189.17 |
International
Class: |
G11C 7/10 20060101
G11C007/10 |
Claims
1. An inter-memory data transfer control unit transferring data
from a first memory to a second memory, comprising: an operation
start trigger receiving unit for receiving an operation start
trigger from a processor; a parameter acquiring unit for acquiring
a transfer instruction parameter created by the processor; a read
unit for reading data from the first memory designated by the
transfer instruction parameter; a write unit for writing the data
read by the read unit into the second memory designated by the
transfer instruction parameter, wherein when a data transfer
instructed by the processor is completed, in order to confirm that
the data written into the second memory is completed, the write
unit performs confirmation of write completion as to the second
memory at the end of the data transfer by the procedure different
from writing into the second memory, and the inter-memory data
transfer control unit notifies the processor of completion of an
inter-memory data transfer based on the confirmation of the write
completion.
2. The inter-memory data transfer control unit according to claim
1, wherein in order to confirm that writing into the second memory
is completed, the write unit of the inter-memory data transfer
control unit performs reading from the first memory through the
same bus as used in writing into the second memory as the procedure
different from writing into the second memory.
3. The inter-memory data transfer control unit according to claim
1, wherein in order to confirm that writing into the second memory
is completed, the write unit of the inter-memory data transfer
control unit performs access to a register in a memory controller
through the same bus as used in writing into the second memory as
the procedure different from writing into the second memory.
4. The inter-memory data transfer control unit according to claim
3, wherein the memory controller is enabled to refer to at least
one register provided in the memory controller for at least a
number of times of writing by the memory controller.
5. The inter-memory data transfer control unit according to claim
4, wherein in order to confirm that writing into the second memory
is completed, the inter-memory data transfer control unit performs
reading from the register enabled to referring for the number of
times of writing by the memory controller through the same bus as
used in writing into the second memory as the procedure different
from writing into the second memory, and the inter-memory data
transfer control unit compares a value read from the register with
the number of times of writing by the inter-memory data transfer
control unit and confirms whether the data transfer is normally
performed.
6. The inter-memory data transfer control unit according to claim
1, wherein in order to confirm that writing into the second memory
is completed, the inter-memory data transfer control unit includes
a write completion confirmation request/response processing unit
for processing a write completion confirmation request dedicated to
requesting for securing writing and a write completion confirmation
response for response to the write completion confirmation request
as the procedure different from writing into the second memory, the
inter-memory data transfer control unit confirms completion of the
writing by sending the write completion confirmation request and
receiving the write completion confirmation response.
7. The inter-memory data transfer control unit according to claim
1, wherein the inter-memory data transfer control unit includes a
signal line between the inter-memory data transfer control unit and
a memory controller for confirmation of securing writing and for
response to the confirmation request, in order to confirm that
writing into the second memory is completed, as the procedure
different from writing into the second memory, the inter-memory
data transfer control unit sends a signal for requesting
confirmation of securing writing to the memory controller, the
inter-memory data transfer control unit receives a signal of
completion of securing writing that the memory controller having
received the signal for requesting confirmation of securing writing
sent to a request source as a response to the request for
confirmation of securing writing when the memory controller detects
completion of a requested writing, and the inter-memory data
transfer control unit confirms completion of writing by receiving
the signal of completion of securing writing.
8. The inter-memory data transfer control unit according to claim
1, wherein the inter-memory data transfer control unit detects a
timing of switching of the write destination memory from the first
memory to the second memory by switching of an access destination
memory controller.
9. The inter-memory data transfer control unit according to claim
1, wherein the inter-memory data transfer control unit determines a
timing of switching of the write destination memory from the first
memory to the second memory with reference to an access destination
address of the memory.
10. The inter-memory data transfer control unit according to claim
9, wherein the inter-memory data transfer control unit determines a
timing of switching of the write destination memory from the first
memory to the second memory by the memory controller.
11. The inter-memory data transfer control unit according to claim
9, wherein the inter-memory data transfer control unit determines a
timing of switching of the write destination memory from the first
memory to the second memory by a DIMM of the second memory.
12. The inter-memory data transfer control unit according to claim
9, wherein the inter-memory data transfer control unit determines a
timing of switching of the write destination memory from the first
memory to the second memory by a bank of the memory.
13. The inter-memory data transfer control unit according to claim
1, wherein the transfer instruction parameter includes at least, as
the information on a data transfer instruction, a start address of
a transfer source memory, at least one start address of a transfer
destination memory, necessity information on whether or not check
of completion of writing to each transfer destination memory, and a
transfer data length, when the write unit detects switching of the
write destination memory from the first memory to the second
memory, in order to confirm that writing into the first memory is
completed, the write unit determines whether or not performing
confirmation of write completion as to the first memory based on
the necessity information in the transfer instruction parameter,
when the write unit determines that the confirmation of write
completion is required, the write unit performs confirmation of
write completion as to the first memory by the procedure different
from writing into the second memory, when a data transfer
instructed by the processor is completed, in order to confirm that
writing into the second memory is completed as to the write
destination memory at the end of the data transfer, the write unit
determines whether or not performing confirmation of write
completion as to the write destination memory based on the
necessity information in the transfer instruction parameter, and
when the write unit determines that the confirmation of write
completion is required, the write unit performs confirmation of
write completion as to the write destination memory at the end of
the data transfer by the procedure different from writing into the
second memory.
14. The inter-memory data transfer control unit according to claim
1, wherein the inter-memory data transfer control unit includes two
different-operation start trigger receiving units, if the
inter-memory data transfer control unit is started by receiving the
operation start trigger from one of the operation start trigger
receiving units, when the inter-memory data transfer control unit
detects switching of a write destination memory from the first
memory to the second memory, in order to confirm that writing into
the first memory is completed, the inter-memory data transfer
control unit performs confirmation of write completion as to the
first memory by the procedure different from writing into the
second memory, when a data transfer instructed by the processor is
completed, in order to confirm that writing into the second memory
is completed as to the write destination memory at the end of the
data transfer, the inter-memory data transfer control unit performs
confirmation of write completion as to the write destination memory
at the end of the data transfer by the procedure different from
writing into the second memory, and the inter-memory data transfer
control unit notifies the processor of completion of an
inter-memory data transfer based on the confirmation of write
completion; if the inter-memory data transfer control unit is
started by receiving the operation start trigger from the other one
of the operation start trigger receiving units, the inter-memory
data transfer control unit omits performing confirmation of write
completion as to the write destination memory, and the inter-memory
data transfer control unit notifies the processor of completion of
the inter-memory data transfer.
15. The inter-memory data transfer control unit of claim 1, wherein
the inter-memory data transfer control unit is a direct memory
access.
Description
TECHNICAL FIELD
[0001] The present invention relates to a data transfer device and
a method of controlling the same.
BACKGROUND ART
[0002] In storage devices (disk array systems and storage systems
of this kind) of recent years, a cache memory is installed in a
storage controller in order to increase the speed of reading and
writing data. The cache memory holds a part of data to be written
to a final storage medium such as a hard disk drive (hereinafter
referred to as "HDD") or a solid state disk (hereinafter referred
to as "SSD") or a part of data to be frequently read, thereby,
hiding access latency of the medium and achieving a fast response
to a host.
[0003] For example, when the host writes data to the storage
device, a storage controller for processing the data provided in
the storage device returns a write completion response to the host
at the stage where the data is stored in the cache memory. Since
the cache memory is much faster than a final storage medium, the
host can issue the next I/O request immediately. Meanwhile, the
storage controller operates so as to write the data in the cache
memory to the final storage medium asynchronously with the
operation of the host at a certain timing.
[0004] The storage controller is often configured of multiple
controller units in order to improve fault tolerance. For this
reason, cache memories (hereinafter, simply referred to as
memories) are also arranged at multiple locations in a dispersed
manner. For improvement in processing efficiency and redundancy by
duplication, a data transfer is often executed between the
memories. For such data transfer, a network path using an internal
bus technology is formed between the controller units. For example,
a standard such as PCI Express (registered trademark) is used for
the bus. In addition, a data transfer controller called direct
memory access (DMA) performs a data transfer between the cache
memories according to a command by a processor executing storage
control software.
[0005] In the case of PCI Express, so-called Posted Access is used
in which the write request target does not issue a completion
response (Completion) to the memory. In addition, buffers are
provided on the path from the network path to the memory controller
in the storage controller by reason of data processing by hardware.
Accordingly, a source of write request cannot confirm whether or
not data has passed through all the buffers on the path and has
been written to the memory. For this reason, when data is written
to a memory via a network path or the like, a dummy memory read
request is issued via the same path in a storage device or the like
to cause the buffers in the middle of the path to flush and then to
ensure the writing of data to the memory. In PCI Express, a memory
read request is called Non-Posted Access. The Non-Posted Access
does not goes past Posted Access such as a memory write request and
does not arrive at the memory before the Posted Access on the same
path. Accordingly, if a response to the read request is returned,
it is guaranteed that the write data according to the memory write
request that has been issued in advance has passed through the
buffers and has been written to the final memory.
[0006] Here, multiple network paths are set in some cases in order
to secure a bandwidth because a transfer bandwidth achievable by a
single path is limited. In order to efficiently perform a data
transfer between multiple memories, it is necessary to evenly
utilize the network paths. A technique to achieve load distribution
through the network paths has been discussed heretofore as a method
to eliminate unevenness and then to utilize the paths
uniformly.
[0007] For example, Patent Literature (PTL) 1 discloses a method in
which a hash value of a packet to be transferred is calculated, and
a path is selected in accordance with the hash value. According to
this method, the packet can be sent while the load is distributed
to the multiple paths. Accordingly, the network transfer can be
efficiently performed.
CITATION LIST
Patent Literature
[0008] [PTL 1] [0009] Japanese Patent Application Laid-open
Publication No. 2008-263436
SUMMARY OF INVENTION
Technical Problem
[0010] However, several problems occur when the aforementioned
method is applied to a storage device or a data transfer system of
this kind that performs the inter-memory data transfer between the
controller units coupled to each other via a PCI Express bus or the
like.
[0011] The first problem is a problem with the aforementioned dummy
read. In a PCI Express bus or the like, it is guaranteed that
Non-Posted Access stays behind Posted-Access in the same path.
However, when Non-Posted Access passes through a different path,
Non-Posted Access may arrive at the final memory before Posted
Access that has been sent in advance. Since the purpose of sending
a dummy read is to confirm that the writing is ensured by use of
the characteristic that the dummy read stays behind a write request
issued before the dummy read, the dummy read needs to be issued via
the same path as that of the write request. However, according to
the method in PTL 1, since the distribution of paths is performed
on the basis of the hash value of a packet, there arises a problem
that both of the request packets may not pass through the same
path. The formats of both the packets are obviously different.
Accordingly, the hash values of both the packets are extremely
unlikely to be the same. In the first place, there is provided no
mechanism of issuing a dummy read when a write request is issued.
In addition, PTL 1 indicates that a different identifier, which is
defined in advance, may be used instead of the hash value, as a
modification. However, to realize the method using the identifier
in PTL 1, it is required to prepare a memory table for storing
correspondence between write packets and dummy read packets, and
identification of paths through which each write packet has passed.
The number of data packets to be transferred between the memories
is enormous and amounts to hundred thousands to millions per
second, thus simply searching the memory table will take very long
time. Rather, the above processes will carry a different purpose
for which the load distribution as a problem raised in PTL 1.
[0012] The second problem is that transfer time largely depends on
the degree of concentration of access to a destination memory. The
technique disclosed in PTL 1 performs load distribution on multiple
paths, but the destination of data is a single device. Thus, it is
considered that PTL 1 assumes that a data transfer is performed
only through a network. The assumption in PTL1 is also apparent
from the method disclosed by PTL 1, which performs load
distribution by managing the degree of congestion of the path
itself (use condition of data buffer). On the other hand, in the
case of this storage device, access to a certain memory is not
necessarily one through a network path, and an access by a DMA in
the same storage controller in which the memory is provided also
conflicts with another. Accordingly, load distribution cannot be
always performed significantly even if only the degree of
congestion of the path is managed.
[0013] The third problem is that the degree of congestion of a
memory access is also attributable to a memory allocation method
(memory usage layout) by control software. For example, consider a
case where a memory channel 1 is allocated a control information
area where access is relatively small, and a memory channel 2 is
allocated a large number of cache areas where access is frequently
made. In this case, if packets are passed through the network paths
with an even ratio, the access to the memory channel 2 under
passing through the network paths resultantly gets stack, and the
packets of the subsequent memory channel 1 are thus accumulated.
This problem cannot be solved by the conventional technique of load
distribution on a network as proposed in PTL 1.
[0014] The present invention has been made in view of the
aforementioned problems.
[0015] Specifically, an object of the present invention is to
provide a data transfer device and its controlling method for
ensuring writing to a memory without holding path information and
additionally introducing control software even if the memory is
coupled to multiple paths.
[0016] In addition, another object of the present invention is to
provide a data transfer device and its controlling method for
efficiently transferring data without causing unevenness in the
paths even when access to a specific memory from inside and outside
of a storage controller is concentrated.
[0017] Moreover, still another object of the present invention is
to provide a data transfer device and its controlling method for
performing an efficient transfer while memory usage information of
upper level control software is taken into account.
Solution to Problem
[0018] In the present invention, for solving at least one of the
aforementioned problems, provided is a data transfer device and its
controlling method, the device having a DMA that detects a point of
time when a memory channel (hereinafter, referred to as a memory
plane) of a write destination of data switches to another during
writing of data, and ensures the writing when the memory channel
switches to another during the data writing or every time a series
of transfers ends even in the same memory channel.
[0019] Moreover, in the present invention, for solving at least one
of the aforementioned problems, provided is a data transfer device
and its controlling method, the device having a status management
mechanism for each DMA and path arbiter to refer to transfer status
information of each other, and a determining unit for performing an
efficient transfer on the basis of the information.
[0020] Furthermore, in the present invention, for solving at least
one of the aforementioned problems, provided is a data transfer
device and its controlling method for performing transfer amount
control and a pass selection by utilizing memory layout information
and priority information on each I/O, which is notified by the
control software.
Advantageous Effects of Invention
[0021] According to the present invention, writing to a memory can
be ensured in a network path using an internal bus technology in
which a write response is not performed, without storing a write
request issuance path by hardware for each packet, and without
additionally introducing a control for ensuring writing from the
control software.
[0022] In addition, according to the present invention, when
multiple network paths are present, a path can be selected while a
congestion situation of the destination memory is taken into
account. Thus, an efficient transfer can be performed.
[0023] Moreover, according to the present invention, an efficient
data transfer can be performed by the same hardware in accordance
with the characteristics of the control software operating on the
hardware.
BRIEF DESCRIPTION OF DRAWINGS
[0024] FIG. 1 is a block diagram showing a configuration of a
storage device in an embodiment of the present invention.
[0025] FIG. 2 is an external view showing the configuration of the
storage device in the embodiment of the present invention.
[0026] FIG. 3 is a block diagram showing a configuration of a
storage controller of the storage device in the embodiment of the
present invention.
[0027] FIG. 4 is a block diagram showing an example of an internal
configuration of an ASIC mounted on the storage controller in the
embodiment of the present invention.
[0028] FIG. 5 is a diagram schematically showing a start-up method
of a DMA controller in the embodiment of the present invention.
[0029] FIG. 6 is a diagram showing an example of a format of a
transfer instruction parameter to the DMA controller in the
embodiment of the present invention.
[0030] FIG. 7 is a flowchart showing an operation procedure of a
DMA in the embodiment of the present invention.
[0031] FIG. 8 is a diagram showing an example of a memory address
map in the embodiment of the present invention.
[0032] FIG. 9 is a ladder chart describing an entire operation of
data transfer control between memories in the embodiment of the
present invention.
[0033] FIG. 10 is a diagram showing a configuration example of a
write confirmation request in a modification of the embodiment of
the present invention.
[0034] FIG. 11 is a diagram showing a configuration example of a
write confirmation response in the modification of the embodiment
of the present invention.
[0035] FIG. 12 is a ladder chart showing an operation procedure of
the DMA of the data transfer control between memories in the
modification of the embodiment of the present invention.
[0036] FIG. 13 is a block diagram showing an internal configuration
of an ASIC in a modification of the embodiment of the present
invention.
[0037] FIG. 14 is a block diagram showing an internal configuration
of an ASIC in Example 2 of the present invention.
[0038] FIG. 15 is a diagram showing a configuration example of a
table in the ASIC in Example 2 of the present invention.
[0039] FIG. 16 is a flowchart describing an operation of a DMA in
Example 2 of the present invention.
[0040] FIG. 17 is a diagram showing a configuration example of a
table included in a status block 400 in a modification of Example 2
of the present invention.
[0041] FIG. 18 is a diagram showing an example of a format of a
transfer instruction parameter to a DMA controller in Example 3 of
the present invention.
[0042] FIG. 19 is a diagram showing a configuration example of a
memory layout information setting table in Example 3 of the present
invention.
[0043] FIG. 20 is a diagram schematically showing exchange of data
between a DMA and an internal data transfer path arbiter in Example
3 of the present invention.
[0044] FIG. 21 is a flowchart showing an operation of the internal
data transfer path arbiter in Example 3 of the present
invention.
DESCRIPTION OF EMBODIMENTS
[0045] An embodiment for carrying out the present invention will be
described using Examples 1, 2 and 3. Note that, an embodiment that
can be carried out by partially changing Examples will be described
as a modification in each of Examples.
Example 1
[0046] Example 1 will be described with reference to FIGS. 1 to
9.
[0047] FIG. 1 is a block diagram showing a configuration of a
storage system in this embodiment. A storage device 10 includes
multiple drive chassis 51, 52, . . . , and a storage controller
chassis 50. The storage device 10 is coupled to a management
terminal 31 via a management LAN 30, and setting or the like of the
device can be performed via the management terminal 31. The
management terminal 31 may be included in the storage device
10.
[0048] In addition, the storage device 10 is coupled to host
computers 20A to 20C via a storage network 40. An operating system
and an application operating thereon or a virtual machine is
installed on the host computers 20A-20C, and thus the host
computers 20A-20C issue an I/O to the storage device 10.
[0049] Sets of multiple disk drives D00 to D0N and D10 to D1N are
installed in the drive chassis 51, 52, . . . , respectively. Each
of the disk drives D00 to D1N is an HDD, an SSD or a final storage
medium of this kind.
[0050] The disk drives D00 to D1N are coupled to disk array
controller main packages 60A and 60B via backend mutual coupling
mechanisms 500 provided in the drive chassis 51, 52, . . . ,
respectively, backend coupling cables 510, and I/F cards 90A and
91B provided in the storage controller chassis 50. Each of the
backend mutual coupling mechanisms 500 is a SAS (Serial Attached
SCSI) expander, for example, and each of the backend coupling
cables 510 is a SAS cable, for example. In addition, each of the
I/F cards 90A and 91B is an HBA (Host Bus Adapter) or the like on
which a SAS protocol chip is mounted, for example. With this
coupling, the storage controller can send and receive data to and
from the disks.
[0051] The storage controller chassis 50 is provided with disk
array controller main packages 60A and 60B, MP (micro processor)
packages 70A and 70B having installed thereon processors for
controlling the main packages 60A and 60B, the aforementioned I/F
cards 90A and 90B for coupling with the drives, and I/F cards for
coupling with the host computers 20A to 20C.
[0052] A storage control program operates in each of the MP
packages 70A and 70B. Control lines 71A to 72B from the MP packages
70A and 70B are coupled to the disk array controller main packages
60A and 60B for sending instructions to the main packages 60A and
60B. Meanwhile, the disk array controller main packages 60A and 60B
are coupled to each other via multiple internal data transfer paths
101 and 102 for mutually transferring data.
[0053] Note that, only the two packages, which are A-based and
B-based packages, are shown in this embodiment, but more than two
packages may be provided and may be coupled to each other, or the
storage controller chassis themselves may be coupled to each other
via I/F cards. For example, a part of the I/F cards 80A to 91B may
be configured to be exchangeable with an I/F card for connecting
between storage controller chassis 50. In addition, the
aforementioned mutual couplings may be configured to be usable in
the same manner as the internal data transfer paths 101 and
102.
[0054] FIG. 2 is an external view illustrated to facilitate
understanding of the storage device in the present embodiment. The
storage controller chassis 50 is installed at the lowermost level
of the device, and the drive chassis 51 to 58 are shown on top of
the controller chassis 50. Although not illustrated, cables for
forming the above-described backend coupling cables and the storage
network 40 are wired at the backside of the device.
[0055] FIG. 3 is a block diagram showing the internal
configurations of the disk array controller main packages 60A and
60B as well as the MP packages 70A and 70B for controlling the disk
array controller main packages 60A and 60B. In addition, the
configuration of the I/F cards 80A to 91B connected thereto is
shown in the block diagram.
[0056] Since the A-based and B-based systems are in a mutually
symmetrical structure, hereinafter, the description will be given
using the A-based system as the representative.
[0057] An ASIC (Application Specific Integrated Circuit) 200A and
memories M0, M1, M2 and M3 are mounted on the disk array controller
main package 60A. Various functions are implemented in the ASIC
200A. For example, DMAs (Direct Memory Access) 210A to 212A, which
perform a transfer between memories, are provided in the ASIC 200A.
Each of the memories M0 to M3 is a cache memory of the storage
controller, and a DRAM (Dynamic Random Access Memory) or the like
which is non-volatilized by battery or the like is used. DRAMs are
often implemented as a DIMM (Dual Inline Memory Module). In
addition, the disk array controller main package 60A is coupled to
a group of I/F cards via the ASIC 200A. FE (Front-End) I/F cards
80A and 81A are the I/F cards for coupling to the hosts via the
storage network 40. Each of the FE I/F cards includes a Fibre
Channel protocol chip or the like installed thereon, and multiple
host coupling ports.
[0058] Meanwhile, BE (Back-End) I/F cards 90A and 91A are coupled
to the disks of the drive chassis 51, 52 . . . via the backend
cables 510.
[0059] In the MP package 70A, a processor chip set 75, processors
760 and 761 as well as local memories 770 and 771 are provided.
Programs or the like that operate on these processors are stored in
the local memories 770 and 771. Depending of the type or generation
of the processor, the local memories 770 and 771 may be coupled to
the chip set 75 or directly coupled to the processors 760 and 761.
In addition, the chip set 75 may be embedded in the processors 760
and 761. The processors 760 and 761 are coupled to the ASICs 200A
and 200B of the main packages 60A and 60B via the control lines 71A
and 71B, and thus control the hardware. The control lines 71A and
71B are based on a technology such as a PCI Express, for
example.
[0060] FIG. 4 is a diagram showing a configuration of the internal
function blocks of each of the aforementioned ASICs 200A and 200B.
In each of the ASICs 200A and 200B, there exist: the aforementioned
DMAs 210, 211 and 212; PCI Express cores 221 and 223 which control
PCI Express used as the internal data transfer paths (I-Path);
internal data transfer path arbiters 220 and 222 for the internal
data transfer paths 101 and 102; memory controllers 230, 231, 232
and 233; I/F controller blocks 240, 241, 242 and 243; processor
ports 271 and 272; routing processors (RP) 260 and 261; and a
switch block 250 for mutually coupling the aforementioned blocks
with each other.
[0061] Each of the DMAs 210 to 212 (inter-memory data transfer
control unit) is a transfer control block having a function to
receive an instruction from the processors and then to copy only
data from a certain memory address to another memory address only
for the amount of a specified transfer length of the data. Each of
the DMAs 210 to 212 is configured of: a read unit (RX) 2101, which
reads data from a transfer source memory; a write unit (TX) 2102,
which writes data to a transfer destination memory; and a parameter
fetch unit (PF) 2103, which fetches a transfer parameter and then
analyzes the transfer parameter. An instruction method relating to
a data transfer and a start-up method of a transfer process will be
described later in detail. Further, though there are provided three
DMAs in the present embodiment, the number of the DMAs may be two
or less, or four or more.
[0062] Each of the internal data transfer paths 101 and 102 is a
network for performing a transfer between memories across the
packages, and PCI Express is used for the internal data transfer
paths 101 and 102 as described above. Each of the PCI Express cores
221 and 223 is a control block that performs a basic transmission
and reception operation of PCI Express, more specifically hardware
control, data link control, transaction protocol control and the
like on a PCI Express bus. Each of the internal data transfer path
arbiters 220 and 222 is an intermediate control block that adjusts
network path usage requests from multiple control blocks, then
grants the right of usage in accordance with a predetermined rule,
and allows the usage. For example, when the DMA 210 and DMA 211
concurrently attempt data transmission, the internal data transfer
path arbiters 220 and 222 performs arbitration processing such as
first providing a right of use to the DMA 210 for its transmission
and then providing a right of use to the DMA 211 for its
transmission.
[0063] The memory controllers 230 to 233 are coupled to the
memories M0 to M3 or M4 to M7, respectively. Each of the memory
controllers 230 to 233 is a control block that writes and reads
data into or from the memories M0, M1, M2, and M3, and the memories
M4, M5, M6, and M7, and sends and receives a data packet based on
requests from each of the DMAs 210 to 212 and other control blocks.
The memory controllers 230 to 233 are respectively provided with
arbiters (ARB) for adjusting the access sequence when accesses are
concurrently made from multiple control blocks. The memory
controllers 230 to 233 are also called a memory channel. A unique
memory address space is allocated to the memory M0-M7 coupled to a
corresponding one of the memory controllers 230 to 233.
[0064] The I/F control blocks 240 to 243 are blocks to which the
aforementioned I/F cards 80A-81B and 90A-91B are coupled and
perform data transmission with the I/F cards. The processor ports
271 and 272 are blocks to which the control lines 71A, 72A or the
like are coupled and perform sending and receiving control
information data by the processors 760, 761 and the ASIC 200, and
further perform control of mutual communication by the processors
760A, 761A, 760B and 761B.
[0065] Each of the routing processors 260 and 261 is a control
block that performs processing to distribute a command received
from a protocol chip via the I/F controller blocks 240-243 to one
of the processors 760A to 761B, which is responsible for the
controlling of the command.
[0066] Various control blocks in addition to the aforementioned
blocks may exist in the ASIC 200. For example, a hardware
arithmetic control block such as a data compression or data
encryption block, a dedicated memory access controller from a
processor, or the like is considered as an optional block.
[0067] FIG. 5 is a diagram schematically showing a structure for
the processors 760 and 761 to issue a transfer instruction to the
DMAs 210 to 212 in the ASIC 200 in the present embodiment. Note
that, the procedure to start any of the DMAs from any of the
processors is the same in the description below. Accordingly, a
case where the processor 760 starts the DMA 210 will be hereinafter
described as the representative.
[0068] In a case where the processor 760 attempts to transfer data
at a certain memory address to a different memory address by use of
the DMA 210, the processor 760 builds a parameter 300 (transfer
instruction parameter) on the local memory 770 for issuing a
transfer instruction to the DMA 210. Although the parameter 300 is
configured with a transfer source memory address, a transfer
destination memory address, a transfer length and the like, the
details thereof will be described later. Here, multiple parameters
300 are to be continuously written to the local memory 770.
[0069] The processor 760 continuously writes the number of the
parameters 300 required for the attempted data transfer to the
local memory 770, and sets, as the value in a start trigger
register 2 of the DMA 210, a start address of the local memory 770
at which the top parameter 300 is stored. For example, in FIG. 5,
since four parameters, parameter 0 to parameter 3 from an address
1000h (hex) of the local memory 770 are built, the processor 760
sets the value 1000h in the start trigger register 2 of the DMA
210.
[0070] When the value is set in the register 2, the DMA 210 reads
the parameters 300 from the address indicated by the value of the
local memory 770, then analyzes the parameters 300 and performs the
data transfer instructed by the parameters 300. For example, in
FIG. 5, since the value 1000h (hex) is set in the start trigger
register 2, the DMA 210 operates to read the parameter 300 from the
address 1000h of the local memory 770. The operation of the DMA 210
will be described later in detail.
[0071] FIG. 6 is a diagram showing an exemplary format of the
parameter 300 used by the processor 760 to issue a data transfer
instruction to the DMA 210 in the present embodiment. A parameter
ID field 3001 is a field where a unique number set for each of the
parameters 300 is specified. A transfer source address field 3002
is a field where a start address of a memory that becomes a
transfer source is specified. A transfer destination address field
3003 is a field where a start address of a memory that becomes a
transfer destination is specified. A transfer length field 3004 is
a field where a data length of data to be transferred is specified.
A chain number field 3005 is a field where a grouping of the
parameter 300 is specified. Each value in the chain number field
3005 is specified in the order from the chain number 3005 of "1"
when there exists the other parameter 300 to be grouped, and the
chain number 3005 of "0" is set as the last parameter 300 as
grouped. Namely, the value in the chain number field 3005 is
incremented by 1 and a group of the parameters 300 up to 0 are
considered to be in one data transfer group. (When the chain does
not exist, 0 is set as the chain number 3005.) When the chain
number 3005 other than 0 is set, the DMA 210 does not terminate the
processing at the processing for the parameter 300, but performs a
chain operation to continuously execute data transfers instructed
by the group of subsequent parameters 300, instead of ending the
processing of one parameter 300. A path number field 3006 is a
field where which number of the path among the multiple internal
data transfer paths 101 and 102 is to be used is specified.
Although one path number field 3006 is set in FIG. 6, multiple path
number fields 3006 may be provided so that the internal data
transfer paths 101 and 102 can be specified individually for cases
where data is read, data is written, and the like. Note that, the
configuration of the parameter 300 is only an example, and the
parameter 300 may have another field required for different
control.
[0072] FIG. 7 is a flowchart showing a data transfer operation of
the DMA 210 in the present embodiment. The data transfer operation
of the DMAs of the present embodiment will be described with
reference to the drawing. Here, the data transfer operation
automatically ensures writing during each period in which a write
destination memory plane switches to another and each period in
which a series of transfers ends.
[0073] The DMA 210 starts the operation in accordance with a
start-up instruction from the processor 760. Although the operation
starts when the processor 760 sets a value in the start trigger
register 2 in the present embodiment, the DMA 210 may be configured
to start on the event triggered by signal transmission via a signal
line, a message transmission or the like. When a value is set in
the start trigger register 2, the parameter fetch unit 2103 of the
DMA 210 refers to the address of the local memory 770 indicated by
the set value, then fetches one parameter 300 and analyzes the
setting content (S8000). For example, the start address of the
transfer source memory is acquired from the transfer source address
field 3002, and the start address of the transfer destination
memory is acquired from the transfer source address field 3003. In
addition, the transfer length (total data transfer amount) is
acquired from the transfer length field 3004. Here, when the DAM
210 starts once, fetching of the next parameter 300 is pre-fetched
in the background (S8005).
[0074] The read unit 2101 of the DMA 210 specifies which memory
controller is to be accessed from the transfer source address on
the basis of the result of the analysis performed by the parameter
fetch unit 2103 (S8010). For example, assume that the memory
addresses are allocated in the manner shown in FIG. 8. When the
transfer source address is an address 23000h, since the address is
in the addresses of 20000h-2FFFFh allotted to the memory M1, the
read unit 2101 determines that the memory controller 231 to which
the memory M1 is coupled is to be accessed. Subsequently, the read
unit 2101 issues a memory read request with respect to the memory
controller 231, the request specifying the address to be accessed
and the data length to be read (S8020). If the target memory
controller is in the same ASIC 200, the memory read request is
directly issued to the memory controller, and if the target memory
controller is in the different ASIC 200, the memory read request is
issued to the memory controller via the internal data path transfer
path 101 or 102. The transfer length that the DMA 210 can transfer
at a time is relatively small. For this reason, the data of the
transfer length specified by the parameter 300 are sometimes
transferred over multiple times. If the transfer length specified
by the parameter 300 is larger than the transfer length that the
DMA 210 can transfer at a time, the maximum transfer length that is
transferable by the DMA 210 is specified as the data length. If the
transfer length specified by the parameter 300 is equal to or
smaller than the transfer length that the DMA 210 can transfer at a
time, the transfer length specified by the parameter 300 is
specified in the read request as the data length. For example, if
the transfer length that the DMA 210 can transfer at a time is 2048
bytes, and the transfer length specified by the parameter 300 is
4096 bytes, 2048 bytes is specified in the read request as the data
length to be read. Likewise, if the transfer length that the DMA
210 can transfer at a time is 512 bytes in this case, 512 bytes is
specified in the read request as the data length to be read.
[0075] The memory controllers 230 to 233 read data of the transfer
length specified in the read request from the memories M0 to M7 and
then return a read response (Completion) to the read unit 2101
together with the read data, the data specified by the address
specified in the read request. The read response is sent directly
to the read unit 2101 if the DMA 210 is in the same ASIC 200. If
the DMA 210 is in the different ASIC, the read response is sent to
the read unit 2101 via the internal data transfer path 101 or 102.
The read unit 2101 of the DMA 210 receives the read response with
the data read from the corresponding one of the memory controllers
230 to 233 (S8030) and then stores the received data in a buffer
inside the DMA 210.
[0076] When the data is stored in the buffer inside the DMA 210,
the write unit 2102 of the DMA 210 specifies which one of the
memory controllers 230 to 233 is to be accessed from the transfer
destination address on the basis of the result of the analysis
performed by the parameter fetch unit 2103 (S8040). For example, if
the memory addresses are allocated in the manner shown FIG. 8, and
the transfer destination address is an address 51000h, the write
unit 2102 determines that the memory controller 230 to which the
memory M4 is coupled is to be accessed.
[0077] For the second and subsequent parameters 300, if the memory
controller specified as the aforementioned access destination is
different from the memory controller specified last time in S8040
(S8050: Yes), the write unit 2102 issues a dummy read request to
the memory controller specified last time (S8060). The dummy read
request may be one that stays behind the memory write request that
is issued previously, such as a memory read request whose transfer
length is 0 (in other words, a request to be processed after
writing to the memory for all of the memory write requests that are
issued previously is completed). If the target memory controller to
which the dummy read request is sent is in the same ASIC 200, the
write unit 2102 issues the dummy read request directly to the
memory controller, and if the target memory controller is in the
different ASIC 200, the write unit 2102 issues the dummy read
request to the memory controller via the internal data transfer
path 101 or 102. Here, if a portion (buffer or the like) where data
remains does not exists on the path from the DMA 210 to the memory
controllers 230 to 233 in the same ASIC 200, the following
configuration may be employed. Specifically, the issuance of the
dummy read request may be omitted as long as the memory controller
specified last time is in the same ASIC, even when the memory
controller specified in S8040 this time is different from the
memory controller specified in S8040 last time (S8050: Yes). The
write unit 2102 determines if an error exists when a read response
for the dummy read request is returned (S8070). If there is no
error (S8070: No), the write unit 2102 sends a memory write request
to the memory controller of a new access destination together with
the data stored in the buffer inside the DMA 210 (S8080).
[0078] Note that, if the memory controller of the access
destination is the same as the last time in S8050 (S8050: No), and
also the parameter 300 read in S8000 is the same as the first one,
the processing of S8060 and S8070 does not have to be performed,
and only the processing of S8080 is performed.
[0079] In addition, if there is an error in the memory read
response in S8070 (S8070: Yes), the write unit 2102, the read unit
2101 and the parameter fetch unit 2103 abort all of the data
transfer processes for the parameter 300 and the subsequent
parameters 300 chained to the parameter 300 (in other words, the
series of the parameters 300 whose values are set in the chain
number fields 3005 are continuously incremented from the parameter
300) (S8300) and then notify the processor 760, 761 of the status
of termination due to an error.
[0080] The transfer length that the DMA 210 can transfer at a time
is small in many cases as described above. For this reason, upon
end of a transfer, the write unit 2102 accumulates the transfer
amount until the last time and the transfer amount at this time,
and then determines whether or not the accumulated transfer amount
achieves the transfer length specified by the parameter (S8090). If
the accumulated amount achieves the specified transfer length
(S8090: Yes), the write unit 2102 lastly sends a dummy read request
to the memory controller specified in S8040 and then ensures the
writing of the data to the memory so far (S8100). The write unit
2102 then receives a read response from the memory controller for
the dummy read request and then determines if an error exists
(S8110). If there is an error, the write unit 2102 proceeds to
S8300 and aborts the transfer in the manner described above. The
read unit 2101 and the write unit 2102 notify the parameter fetch
unit 2103 that the transferred amount has achieved the specified
transfer length and wait for the next instruction. The parameter
fetch unit 2103 determines whether or not the next parameter is
chained (S8120). If the next parameter is not chained (S8120: No),
the parameter fetch unit 2103 notifies the processor 760, 761 of
the transfer end status (S8130), and terminates the data transfer
process.
[0081] Here, if the accumulated transfer amount does not achieve
the transfer length specified by the parameter 300 in S8090 (S8090:
No), the write unit 2102 and the read unit 2101 subtract the
transfer amount of this time from the remaining transfer length
(S8200), then shift the transfer source address and the transfer
destination address by the transfer length of this time (S8210) and
then repeat the processing until the accumulated transfer amount
achieves the transfer length specified by the parameter 300 (The
process returns to S8010).
[0082] In addition, if a parameter 300 is chained in S8100 (S8100:
Yes), the data transfer process for the next parameter 300 is
performed subsequently. (The process returns to S8000.)
[0083] The DMA 210 of the present embodiment performs the data
transfer process in the manner described above.
[0084] Here, in the actual control implementation, since the
parameter fetch unit 2103, the read unit 2101 and the write unit
2102 may operate simultaneously in parallel with each other, each
processing (S8000 to S8130) is performed in parallel with each
other in terms of time in some cases. In addition, the three units
which are the parameter fetch unit 2103, the read unit 2101 and the
write unit 2102 are provided and assigned the roles in the manner
described above in the present embodiment. However, another
configuration may be employed as long as the configuration can
fulfill the same functions as the DMA.
[0085] FIG. 9 is a ladder chart showing an entire image of the data
transfer control between the memories including the above described
data transfer control of the DMA shown in FIG. 7 in the present
embodiment. The vertical direction in the ladder chart shows the
time passage, and the horizontal direction in the ladder chart
shows the control between the resources as well as the flow of
data. The resources herein are the local memories 770 and 771, the
processors 760 and 761, the memories M0 to M7 and the DMAs 210 to
213. However, since the same resources perform the same operation,
the reference numerals are omitted. Note that, the memory #x
denotes a transfer source memory, and the memory #0 and the memory
#1 denote transfer destination memories, respectively, and each of
the memories is any one of the memories of the memory controllers
230-233 to which the memories M0 to M7 are coupled, respectively,
and the transfer source and the transfer destination may be
overlapped with another. The entire image of the data transfer
control will be described with reference to the ladder chart.
[0086] The processor 760, 761 generates a group of the parameters
300 by the method shown in FIG. 5 on a local memory on the basis of
the parameter format shown in FIG. 6 (R7000). The processor 760,
761 subsequently sets, in the trigger register 2 of a DMA, an
address of the local memory in which the parameters 300 are
generated, thereby, triggering the start of the DMA (R7100). The
DMA fetches the first parameter 300 in the group of the parameters
300 from the local memory with reference to the trigger register 2
(R7200). In addition, as a background task, the DMA pre-fetches the
next parameter 300 at an appropriate timing (R7210). The DMA
analyzes the parameter 300, then reads data from the memory #n
(R7300 to R73xx) and writes the data to the memory #1 in parallel
with the reading process (R7400 to R74xx) on the basis of the
result of the analysis. When the write destination memory plane
switches from the memory #0 to the memory #1, the DMA issues a
dummy read request to ensure the writing of the data performed so
far (R7500). Then, the DMA subsequently continues the data reading
(R7600 to R76xx) and also continues the writing of the data to the
memory #1 (R7700 to R77xx). When the transfer for the transfer
length specified by the parameter 300 is completed, the DMA issues
a dummy read request to the final memory plane and thereby performs
the processing to ensure the writing (R7800). If the subsequent
parameter 300 is not chained, i.e., the chain number 3005 of the
next parameter 300 is 0, the DMA returns the transfer status to the
processor 760, 761 via the local memory (R7900).
(Effects of Example 1)
[0087] According to Example 1, write requests to the same memory
controller 230-233 are continuously sent without ensuring the
writing, and when a write request is sent to a memory controller
230-233 of a different write destination, a dummy read request can
be sent to the previous memory controller 230-233. Since the
requests to the same memory controller 230-233 are sent via the
same path, the writing of the data for the write request issued
previously can be ensured by sending a dummy read request to the
previous memory controller 230-233 by use of Non-Posted Access,
after the write request sent by use of Posted Access. Accordingly,
even in a case where a protocol not issuing a completion response
of wiring is used in the internal data transfer paths 101 and 102,
the writing to a memory can be ensured without storing the issuance
path of the write request for each packet. In addition, the path
storage region for each packet becomes unnecessary. Thus, it is
possible to achieve a downsizing of the implementation. Moreover,
in this embodiment, each of the DMAs 210 to 212 can automatically
send a dummy read request at the timing when the memories M0 to M7
of the write destinations switch from one to another and when
writing ends. Thus, such an additional the processing as issuing a
dummy read request by control software executed in the management
terminal 31 or the host computer 20 is unnecessary. Accordingly, it
is possible to achieve faster performance as the entire system.
(Modification 1 of Example 1)
[0088] In this embodiment, a dummy read request is issued to a
corresponding one of the memories M0 to M7 in order to ensure
writing. However, the embodiment may be modified so that a read
request can be issued to a register of a corresponding one of the
memory controllers 230 to 233, instead. Here, an assumption is made
that the read request to the register uses Non-Posted Access. In
this modification, no access for ensuring writing is made to the
memories M0 to M7. Thus, it is possible to ensure writing faster,
and there is an additional effect that improves use efficiency of
the memories. Moreover, the aforementioned register of each of the
memory controllers 230-233 may be provided with a function to
record the number of write requests and then to set the number to 0
when the register is read. In this case, each of the DMAs 210 to
212 can determine a normal/abnormal state of the transfer by
comparing the value read from the register with the number of the
issued write requests. Accordingly, there is an additional effect
that can further increase the reliability to ensure writing. The
above register can be provided for each resource accessing the DMAs
and the other memories or for each access area. Further, the memory
controller 230-233 can be provided with a register for holding the
number of read requests and the other statistic information in
addition to the write requests.
(Modification 2 of Example 1)
[0089] In this embodiment, a dummy read request (Non-Posted Access)
is issued to ensure writing to the memory. However, the embodiment
may be modified so that a dedicated command (write ensuring
request) by use of Posted Access, which stays behind the preceding
writing in advance, can be defined and then issued. For example, a
TLP (Transaction Layer Packet) of the format as shown in FIG. 10 is
defined as a write confirmation request. Then, when the write
confirmation request is sent, the memory controller 230-233 flushes
data in the buffer to the memory and then asynchronously returns a
response (write confirmation response) by use of Posted Access as
shown in FIG. 11. The sequence of this case becomes one that is
shown in FIG. 12. In the case of this modification, unlike the case
where a Non-Posted Access is used, each of the DMAs 210 to 212 does
not have to secure the buffer for receiving response data (read
response) and wait. Thereby, there is an additional effect that can
enhance the parallelism of the operations and thus improve the
performance. Here, an undefined request (TLP) is discarded in
normal PCI Express, so that the definition to receive the
aforementioned request and response is implemented in the PCI
Express cores 221 and 223.
(Modification 3 of Example 1)
[0090] In this embodiment, a dummy read request is issued to a
memory to ensure writing thereto. However, a dedicated signal line
for a request to ensure writing may be implemented in the ASIC 200
and provided on the paths of the internal data transfer paths 101
and 102. FIG. 13 shows an implementation example of this case. When
writing needs to be ensured, a corresponding one of the DMAs 210 to
212 instructs a signal controller 280 to send a signal to a
corresponding one of the memory controllers 230 to 233 for which
writing needs to be ensured. The signal includes information such
as the number of transfers for the writing from the one of the DMAs
210 to 212, or the like. Then, the one of the memory controllers
230 to 233 returns an ensuring completion signal when the required
number of transfers arrives, for example. In this manner, the
corresponding one of the DMAs 210 to 212 can confirm, through the
signal controller 280, that the writing is ensured. In the case of
this modification, the request to ensure writing and the response
do not pass through the data transfer paths. Thus, there is no
influence on the data transfer. Accordingly, there is an additional
effect that improves the transfer efficiency.
(Modification 4 of Example 1)
[0091] In this embodiment, writing is secured at the time of
switching of the memory controllers 230-233. However, the writing
may be secured on the unit of DIMM installed beyond the memory
controllers 230-233. This configuration can be realized by
modifying the embodiment so that the address areas are managed not
for the memory controllers 230-233 but for the DIMMs and it is
determined at S8050 in FIG. 7 whether or not the DIMM as an access
destination is different from the previous one. Usually the DIMM is
constructed with multiple memory chips and managed with a unit of
bank. Therefore, similarly, the times of switching the banks in the
DIMM may be the times for securing writing. In this case, the
address areas are managed for the banks.
(Modification 5 of Example 1)
[0092] In this embodiment, the processing to ensure writing is
performed at the timing when the memory planes switch from one to
another. However, the processing to ensure writing may be performed
periodically for each constant number of transfers even before the
memory plane switches to another. In the case of this modification,
writing can be periodically ensured even when data of very long
transfer length is written to a memory, so that sureness of the
writing of data can be improved, and further effect can be obtained
such as facilitating recovery processing on error occurrence
through performing frequent secure processings. In addition, the
processors 760 and 761 may be configured to be capable of
explicitly specifying the presence or absence of the necessity or
the frequency of the processing to ensure writing in the transfer
parameter. Alternatively, it is also possible to provide two
different initiating trigger registers for the DMAs 210-212 and
when initiated by one trigger register the DMAs 210-212 function
with write secure processing, and when initiated by the other
trigger register the DMAs 210-212 function without write secure
processing. In this case, there is an additional effect that
realizes reduction of an unnecessary operation to ensure writing
and thereby improves the processing performance.
(Modification 6 of Example 1)
[0093] In this embodiment, the DMAs 210-212 read the parameters 300
and transfer data from memories to memories as instructed by the
parameters 300. However, it is possible to provide another DMA for
transferring small data written in a buffer or a register to the
memories along with performing write secure processing. In
addition, since the transfer length and the transfer source address
can be known by the size and the address of the buffer or the
register, it is possible to omit creating the parameters 300 for
the DMA and to modify the initiating trigger register so that the
transfer destination address is set therein. Further, the
notification of transfer status to the processors 760, 761 may be
omitted when the data transfer is normally completed. In the
present modification of Example 1, there is an additional effect
that overhead in transferring data by a small size for creating the
parameters and/or for checking the transfer status is reduced, and
thereby the processing performance in data transfer by a small size
can be improved.
[0094] Note that, Modifications 1 to 6 described above may be
combined as appropriate.
Example 2
[0095] Example 2 will be described with reference to FIGS. 14 and
15.
[0096] The basic configuration of a device in Example 2 is the same
as that of the storage device shown in FIG. 1, and the
configuration of the storage controller is the same as the one
shown in FIG. 3. However, the functions included in the ASIC 200
are different. In the description below, the portions different
from Example 1 will be mainly described, and the description of the
same portions will be omitted. Note that, since the ASIC 200A and
the ASIC 200B have the same configuration, the alphabet symbol at
the end of each of the reference numerals is omitted in the
description below.
[0097] FIG. 14 shows a functional block diagram of the ASIC 200 in
the storage controller in Example 2. A DMA status block 400 that
manages statuses of the DMAs 210 to 212 is provided in the ASIC
200. Each of the DMAs 210 to 212 is provided with a status manager
(SM) 2104 that manages the status of the DMA itself and that
acquires information on the other DMAs 210 to 212.
[0098] The DMA status block 400 has a table 401 that shows a
transfer status of each of the DMAs 210 to 212. Details of the
table 401 will be described later with reference to FIG. 15. The
status manager 2104 of each of the DMAs 210 to 212 accesses the DMA
status block 400 via the switch block 250, then acquires a record
of the information on the DMA itself and the information on the
other DMAs 210 to 212, which is utilized for determining on a path
selection.
[0099] FIG. 15 shows an example of the table 401 in the ASIC 200 of
Example 2. The table 401 has the following fields: a number (DMA
number) 4010 of each of the DMAs 210 to 212; a current status 4020;
a transfer source memory controller number 4030; a transfer
destination memory controller number 4040; an in-use internal data
transfer path number 4050; a request type 4060; or the like.
[0100] The DMA number 4010 is the field where the number that
identifies each of the DMAs 210 to 212 in a corresponding one of
the ASICs 200A and 200B is stored. The current status 4020 is the
field where whether the corresponding DMA is in a transfer (BUSY)
state or a stop (IDLE) state is shown. The transfer source memory
controller number 4030 shows the number that identifies one of the
memory controllers 230 to 233 as transfer source, the memory
controller specified by parameter analysis in a corresponding one
of the DMAs 210 to 212. Likewise, the transfer destination memory
controller number 4040 shows the number that identifies one of the
memory controllers 230 to 233 as transfer destination, the memory
controller specified by parameter analysis in a corresponding one
of the DMAs 210 to 212. The internal data transfer path number 4050
shows the number that identifies the internal data transfer path
101 or 102 when data is transferred via one of the internal data
transfer paths 101 and 102. The request type 4060 shows whether the
internal data transfer path is used for reading (RX), writing (TX)
or both (TX/RX). Note that in this embodiment, although the number
of the field indicating the transfer destination memory controller
number 4040 is one, when the DMA has a function of transferring
data from one transfer source to multiple transfer destinations
concurrently, the table 401 is configured to prepare multiple
fields for the transfer destination memory controller numbers
accordingly.
[0101] FIG. 16 shows an exemplary flowchart when the DMA 210 in
this embodiment performs a transfer operation. Note that, each of
the DMAs 211 and 212 performs the same operation as that of the DMA
210. The parameter fetch unit 2103 of the DMA 210 analyzes a
parameter by use of the same method shown in FIG. 7 and then
specifies the memory controllers 230 to 233 of the transfer source
and the transfer destination (S8010 and S8040). As a result of the
analysis, if a corresponding one of the memory controllers 230 to
233 of the transfer source or the transfer destination needs to be
accessed via the internal data transfer path 101 or 102 (in other
words, the memories M4 to M7 of the different package need to be
accessed), the status manager 2104 of the DMA 210 accesses the DMA
status block 400, thereby, determining which one of the internal
data transfer paths 101 and 102 is to be used. Specifically, the
status manager 2104 refers to the table 401 and then counts the
number of records for each request type 4060 for each of the
internal data transfer paths 101 and 102. If the access that uses
the internal data transfer path 101 or 102 is reading (in other
words, only the corresponding one of the memory controllers 230 to
233 of the transfer source corresponds to the memories M4 to M7 of
the different package) (S9020: Yes), the status manager 2104 counts
the number of records each including "RX" in the request type 4060
for each of the internal data transfer paths 101 and 102 (S9030).
Meanwhile, if the access that uses the internal data transfer path
101 or 102 is writing (in other words, only the corresponding one
of the memory controllers 230 to 233 of the transfer destination
corresponds to the memories M4 to M7 of the different package)
(S9040: Yes), the status manager 2104 counts the number of records
each including "TX" in the request type 4060 for each of the
internal data transfer paths 101 and 102 (S9050). If the access
that uses the internal data transfer path 101 or 102 is both
reading and writing (in other words, both of the memory controllers
230 to 233 of the transfer source and the transfer destination
correspond to the memories M4 to M7 of the different package), the
status manager 2104 counts the number of records for each of the
internal data transfer paths 101 and 102 (S9060). The status
manager 2104 selects one of the internal data transfer paths 101
and 102 whose number of records resulting from the aforementioned
counting for the request type 4060 is the lowest (S9070). Then, the
status manager 2104 updates the information on the record
corresponding to the DMA 210 itself such as the memory controllers
230 to 233 to be used, the internal data transfer path number to be
used and the like. For example, in a case where the number 4010 of
the DMA 210 itself is "0," and data is to be transferred from the
memory controller 0 of the same package as the DMA 210 to the
memory controller 5 of the different package by use of the internal
data transfer path 0 (101), the status manager 2104 sets "BUSY" in
the status 4020, "0" in the transfer source memory controller
number 4030, "5" in the transfer destination memory controller
number 4040, "0" in the internal data path number 4050, and "TX" in
the request type 4060 for the record having "0" in the DMA number
4010.
[0102] Upon completion of the transfer of the data in accordance
with the parameter 300 (S8020 to S8030 or S8050 to S8110 in FIG.
7), the status manager 2104 of the DMA 210 again updates the status
of the table 401 (S9090). Specifically, the status manager 2104
sets "IDLE" in the status 4020 corresponding to the DMA 210 itself,
and clears the transfer source memory controller number 4030, the
transfer destination memory controller number 4040, the internal
data transfer path number 4050 and the request type 4060.
(Effects of Example 2)
[0103] According to Example 2, in a case where the multiple
internal data transfer paths 101 and 102 exist, one of the internal
data transfer paths 101 and 102 to be used for a data transfer can
be selected while the use conditions of the internal data transfer
paths 101 and 102 by the other resources (DMAs) are taken into
account. Thus, an efficient transfer can be achieved. Here, as a
conventional method that is used frequently, there is a method in
which available transfer buffer spaces of the network paths are
checked first for a path selection. However, with this method, when
the multiple DMAs 210 to 212 check the available buffer spaces at
slightly different times, all of the DMAs 210 to 212 may possibly
select the same path. Accordingly, an efficient transfer cannot be
performed in this case. In Example 2, each of the DMAs 210 to 212
that has referred to the table 401 updates the table 401 with the
result of the selection. Accordingly, the next one of the DMAs 210
to 212 can determine which one of the internal data transfer paths
101 and 102 is to be used for a data transfer, while taking the
result of the selection made by the previous one of the DMAs 210 to
212 into account. Thus, it is unlikely for all of the DMAs 210 to
212 to select the same one of the internal data transfer paths 101
and 102, thereby, allowing a more efficient path selection.
(Modification 1 of Example 2)
[0104] In Example 2, each of the DMAs 210 to 212 makes a
determination on path selection with reference to the information
of the other DMAs 210 to 212. However, the determination logic may
be provided in the status block 400, and then, the DMAs 210 to 212
may be configured to send an inquiry to the status block 400. In
this case, a path selector is provided in the status block 400, and
each of the DMAs 210 to 212 sends a request including the memory
controllers 230 to 233 of the transfer source and the transfer
destination to the path selector. Then, the path selector performs
the processing of S9000 to S9070 in FIG. 16 in accordance with the
request and then responds with the selected internal data transfer
path 101 or 102. In the case of present Modification 1, the path
selection logic can be located at one location in a concentrated
manner, so that there is a possibility to improve the
implementation efficiency. In addition, there is an additional
effect that facilitates prevention of information mismatch due to
data writing during data reading or the like because the status
block 400 can adjust the update timing of the records in the table
401.
(Modification 2 of Example 2)
[0105] In Example 2, the determination for a path selection is made
with reference to the selection statuses of the internal data
transfer paths 101 and 102 by each of the DMAs 210 to 212. However,
the determination may be made while available statuses of buffers
of the internal data transfer paths 101 and 102, the number of
remaining transfer credits of PCI Express, or the like is taken
into account in addition to aforementioned selection statuses. For
example, such a determination can be made that the internal data
transfer path 1 (102) can be selected when usage of the internal
data transfer path 0 (101) is low, but the buffer is full. As an
example, the aforementioned modification can be implemented by
adding a function to monitor the amount of buffer used in each of
the internal data transfer paths 101 and 102 to a corresponding one
of the internal data transfer path arbiters 220 and 222, or adding
a function to monitor the number of remaining transfer credits to
each of the PCI Express cores 221 and 223. In Modification 2, the
status block 400 is configured to store therein a table 402 as
shown in FIG. 17, the table 402 storing therein the number of
remaining transfer credits or the like. In the example of FIG. 17,
more detailed credits of the buffers such as headers, data and the
like are stored in the table 402. Specifically, the following
credits are stored: Posted Header buffer CREDITS (PH_CREDITS);
Posted Data buffer CREDITS (PD_CREDITS); Non-Posted Header buffer
CREDITS (NPH_CREDITS); Non-Posted Data buffer CREDITS
((NPD_CREDITS); Completion Header buffer CREDITS (CPLH_CREDITS);
and Completion Data buffer CREDITS (CPLD_CREDITS). In the case of
the Posted-Access such as memory write requests, the reference can
be made by comparing the number of remaining PH_CREDITS or
PD_CREDITS, while in the case of the Non Posted-Access such as
memory read requests, the reference can be made by comparing the
number of remaining NPH_CREDITS or NPD_CREDITS. In the case of this
modification, there is an additional effect that makes it possible
to perform a path selection with a higher accuracy because the
congestion situation of each of the internal data transfer paths
101, 102 can be taken into account.
(Modification 3 of Example 2)
[0106] In Example 2, the determination for a path selection is made
with reference to the selection statuses of the internal data
transfer paths 101 and 102 by each of the DMAs 210 to 212. However,
the determination may be made while the use conditions regarding
the transfer source memory controller number 4030 and the transfer
destination memory controller number 4040 of the table 401 are
taken into account in addition to the aforementioned selection
statuses. Assume that an access to the memory controller 231 is
made through the internal data transfer paths 101, 102. For
example, in a case where the number of the DMAs 210 to 212 in the
internal data transfer path 0 (101) is small, but all of the access
destinations are the same memory controller 230, the buffer of the
memory controller 230 may possibly become full, and thus the
subsequent packets get stuck in the internal data transfer path 0.
In this respect, a case is conceivable where the internal data
transfer path 1 (102) is selected even if the number of the DMAs
210 to 212 that select the internal data transfer path 1 (102) is
large. This Modification 3 can be realized by the configuration in
that the status manager 2104 performs count, for each of the
internal data transfer paths 101 and 102, in S9030, S9050 and S9060
only for the DMA whose transfer destination memory controller is
the controller 231 for example. In the case of Modification 3,
there is an additional effect that makes it possible to perform a
path selection with high accuracy because the congestion situation
of the memory controller of the access destination can be taken
into account in addition to the congestion situation of each of the
internal data transfer paths 101, 102.
(Modification 4 of Example 2)
[0107] In Example 2, the determination for a path selection is made
with reference to the selection statuses of the internal data
transfer paths 101 and 102 by each of the DMAs 210 to 212. However,
in addition to this, more detailed information such as the transfer
length, the number of transfer blocks or the number of remaining
transfer blocks may be added in the table 401 and then used for
making the determination. In the case of this modification, the
accuracy for the path selection can be increased, and there is an
additional effect that enables a more efficient determination.
[0108] Note that, Modifications 1 to 4 of Example 2 described above
may be combined as appropriate.
Example 3
[0109] Example 3 will be described with reference to FIGS. 18 to
20.
[0110] The basic configuration of a device in Example 3 is the
storage device shown in FIG. 1, and the configuration of the
storage controller is the same as the one shown in FIG. 3. However,
the format of the parameter 300 used by the processors 760 and 761
to issue a transfer instruction to the DMAs 210 to 212 is
different. In addition, Example 3 is different in that the
processors 760 and 761 can notify or set the layout information of
usage of the memories M0 to M7 to or in the ASIC 200. In the
description below, the portions different from Examples 1 and 2
described above will be mainly described, and the description of
the same portion will be omitted.
[0111] FIG. 18 is an example of the format of the parameter 300
generated by the processors 760 and 761 to issue a transfer
instruction to the DMAs 210 to 212 in Example 3. In the parameter
300 of Example 3, in addition to the fields 3001 to 3006 included
in the parameter 300 of the aforementioned embodiments, there is a
(I/O type) field 3007 where the I/O type of the transfer and the
priority of the processing are specified. The I/O type field 3007
is an eight-bit field, and the upper four-bit indicates the I/O
type and the lower four-bit indicates the priority. For example, as
the I/O types, a random I/O from a host is set to 1000b (binary
digit), a sequential I/O from a host is set to 0100b, and a
transfer for a synchronization function between volumes included in
a storage is set to 0110b through 0111b according to the types.
Meanwhile, as to the priorities, a transfer with the top priority
is set to 1000b, while a transfer with the minimum priority is set
to 0001b, and the like. Each of the processors 760 and 761
specifies the value by combining the aforementioned values when
generating the parameter 300.
[0112] FIG. 19 is a memory layout information setting table 800.
This table is provided in a setting register of each of the
internal data transfer path arbiters 220 and 222 in the ASIC 200.
Here, the information in the memory layout information setting
table 800 may be arranged at a different location as long as the
internal data transfer path arbiters 220 and 222 can utilize the
table, and the same effects can be obtained. For example, the
memory layout information setting table 800 can be arranged in the
local memories 770 and 771, the memories M0 to M7 or the like.
[0113] In the memory layout information setting table 800, memory
ranges used by the DMAs 210 to 212 for a memory transfer are
specified. In general, the entire regions of the memories M0 to M7
cannot be used as the cache memory regions of the storage, and a
part of the memories M0 to M7 is used as a control information
storage area of a protocol chip, a cache management information
storage area of the processors 760 and 761, and the like. In this
respect, a region other than the aforementioned regions is
specified in this table 800, and thereby, it is made possible to
notify the ASIC 200, particularly, the internal data transfer path
arbiters 220 and 222 that a large number of accesses occurs to
which one of the memory controllers 230 to 233, the internal data
transfer path arbiters 220 and 222 controlling passing of data
through the internal data transfer paths 101 and 102,
respectively.
[0114] A configuration of the memory layout information table 800
will be described. In a memory number field 8010, a memory plane
number is recorded. In Example 3, an assumption is made that the
memory numbers match the numbers that show the memory controllers
230 to 233 corresponding to the memories M0 to M7. A start address
field 8020 shows the start address of the effective range of access
by the DMAs 210 to 212. An end address field 8030 shows the end
address of the effective range of access by the DMAs 210 to 212. A
ratio field 8040 is a field where a ratio showing the number of
times the ASIC 200 accesses each of the memory controllers 230 to
233 is set. The value of the ratio is determined according to the
size of the address range specified by the processors 760 and 761.
In Example 3, while employing an area of a certain address space as
a unit, the number of the address space existing in the address
range is set as the ratio value. In the example shown in FIG. 19,
the address space 1000h (addresses 0000h-0FFFh) is employed as a
unit, and the ratio value 16 is set for the memory M2 and the
memory M3 providing a space of 10000h, and the ratio value 5 is set
for the memory M0 providing a space of 5000h. A counter field 8050
indicates a count value by an operation counter used by the arbiter
for managing a change in the control ratio, and the same value as
that of the ratio field 8040 is set in the counter field 8050 as
the default value.
[0115] FIG. 20 is a diagram schematically showing exchanges of data
between the DMA 210 and the internal data transfer path arbiter 220
in this embodiment. Since how data is exchanged between the DMAs
210 to 213 and the internal data transfer path arbiters 220 and 221
is the same, a description will be given using the DMA 210 and the
internal data transfer path arbiter 220, herein. Between the DMA
210 and the internal data transfer path arbiter 220, the following
signal lines exist: an REQ signal line 601 that requests the right
of usage of the path from the DMA 210 to the internal data transfer
path arbiter 220; an REQ_MEM_NO signal line 602 that transmits an
access destination memory controller number at the time of issuing
the request; an ACT signal line 603 that transmits permission of
the right of usage of the path from the internal data transfer path
arbiter 220 to the DMA 210; and various signal lines 60n for data
to be sent and received between both of them, a signal for
synchronization, communications of an error status and the like.
The internal data transfer path arbiter 220, using the
aforementioned group of signal lines, is capable of communicating
with all of the DMAs 210 to 212 and the other resources in the ASIC
200 in the same controller that use the path via the switch block
250.
[0116] FIG. 21 is a flowchart showing an operation of the internal
data transfer path arbiters 220 and 222 in Example 3. Hereinafter,
the internal data transfer path arbiter is simply referred to as an
arbiter. In addition, since the arbiters 220 and 222 perform the
same operation, a description will be given of only the arbiter
220, herein.
[0117] When the DMAs 210 to 212 are started by the same procedure
as the procedure shown in FIG. 5 described above, each of the DMAs
210 to 212 analyzes the parameter 300 and specifies one of the
memory controllers 230 to 233 of the access destination by the same
procedure as the procedure shown in FIG. 7. When accessing the one
of the memory controllers 230 to 233 of the access destination via
the internal data transfer path 101 or 102, each of the DMAs 210 to
212 sends a request signal for acquiring the right of usage of the
path together with a signal showing the access destination memory
controller number to the arbiter 220 via the REQ signal line 601
and the REQ_MEM_NO signal line 602.
[0118] Upon receipt of the request signals from the DMAs 210 to
212, the arbiter 220 starts an operation to control the right of
usage in sequence (S9000).
[0119] When receiving one request signal (S9010: Yes), the arbiter
220 grants the right of usage for the received request signal
(S9100) and returns an ACT signal via the ACT signal line 603. Upon
receipt of the ACT signal, one of the DMAs performs a data transfer
with the corresponding one of the memory controllers 230 to 233.
Then, when the data transfer is completed, the DMA withdraws the
REQ signal on the REQ signal line 601 (negates the signal) and then
releases the right of usage. The arbiter 220 waits until the right
of usage is released (S9110), and then waits for a request signal
again.
[0120] If the arbiter 220 receives multiple requests at the same
time, or there are multiple requests that are waiting to be
processed (S9010: No), the arbiter 220 checks the access
destination memory controller number signals as well as the request
signals received via the REQ_MEM_NO signal line 602 (S9020). Then,
the arbiter 220 compares the values of the counter fields 8050
corresponding to the respective memory controller numbers with
reference to the memory layout information setting table 800
(S9030). Then, the arbiter 220 selects the memory controller number
whose value in the counter field 8050 is the largest among the
values of the memory controllers 230 to 233 that are requested as
the access destinations, and then grants the right of usage of the
path to one of the DMAs 210 to 212 that specifies the number
(S9040). If multiple DMAs 210 to 212 are applicable, the arbiter
220 controls the right of usage by round robin. Then, the arbiter
220 performs processing to decrement the counter value
corresponding to the memory controller by one and to lower the
priority by one (S9050). When all of the counter values become 0
(S9060: Yes), the arbiter 220 resets the counter value of each of
the memories with reference to the ratio field 8040 in order that
the counter value of each of the memories can match the value of
the ratio field 8040 (S9070). Then, the arbiter 220 waits for the
granted right of usage to be released (S9080), and then waits for a
signal again.
(Effects of Example 3)
[0121] According to Example 3, the memory resource usage
information set by the processors 760 and 761, i.e., the control
software is utilized, and thereby, the usage of the internal data
transfer paths 101 and 102 can be more efficiently controlled than
a case where the usage thereof is controlled by hardware alone.
Thus, the data transfer via the internal data transfer paths 101
and 102 can be more efficient.
[0122] Although the three Examples and the modifications belonging
thereto have been described above, the aforementioned Examples are
provided to describe an example of the means to solve at least one
of the problems cited in the present invention. Thus, the
configurations thereof are not intended to impose any limitation on
the present invention. In addition, the aforementioned Examples can
be applied individually or in combination.
[0123] The present invention can be applied not only to one or
multiple storage devices described in Examples 1 through 3
described above, as a matter of course, but also broadly applied to
one or more data transfer devices coupled to each other via one or
more networks of an environment without issuing a response
indicating completion of writing is used in the device, without
limited to PCI Express.
[0124] In the above, the embodiments have been described; however,
the above-mentioned embodiments are intended to facilitate
understanding of the present invention and not to be taken in a
limiting sense. The present invention may be modified or improved
without departing from the spirit of the invention, and the present
invention also includes the equivalents thereof.
* * * * *