U.S. patent application number 10/217485 was filed with the patent office on 2003-09-18 for data communication method in shared memory multiprocessor system.
This patent application is currently assigned to Hitachi, Ltd.. Invention is credited to Nakamura, Tomohiro, Sukegawa, Naonobu.
Application Number | 20030177273 10/217485 |
Document ID | / |
Family ID | 28035040 |
Filed Date | 2003-09-18 |
United States Patent
Application |
20030177273 |
Kind Code |
A1 |
Nakamura, Tomohiro ; et
al. |
September 18, 2003 |
Data communication method in shared memory multiprocessor
system
Abstract
N shared data registers are provided for N+1 processors,
respectively. For allowing all the processors to read the same data
from the shared data registers, the processors are connected by
interprocessor communication channels. The processors are
classified into a master processor and subordinate processors. All
data writing into the shared data registers are executed from the
master processor. Further, data writing into the shared data
registers from the subordinate processor is executed from the
master processor after a write request is sent to the master
processor.
Inventors: |
Nakamura, Tomohiro;
(Hachioji, JP) ; Sukegawa, Naonobu; (Inagi,
JP) |
Correspondence
Address: |
Stanley P. Fisher
Reed Smith LLP
Suite 1400
3110 Fairview Park Drive
Falls Church
VA
22042-4503
US
|
Assignee: |
Hitachi, Ltd.
|
Family ID: |
28035040 |
Appl. No.: |
10/217485 |
Filed: |
August 14, 2002 |
Current U.S.
Class: |
709/251 ;
709/234 |
Current CPC
Class: |
G06F 15/167 20130101;
H04L 12/427 20130101; G06F 2213/0038 20130101 |
Class at
Publication: |
709/251 ;
709/234 |
International
Class: |
G06F 015/16 |
Foreign Application Data
Date |
Code |
Application Number |
Mar 14, 2002 |
JP |
2002-070099 |
Claims
What is claimed is:
1. A data communication method in a shared memory multiprocessor
system having a plurality of processors and a memory, comprising:
providing a register for storing shared data in each of said
processor; connecting said processors by a ring type network; and
setting one of said processors to being a main processor, and
setting other processors to be subordinate processors; wherein when
said subordinate processor performs data writing into said
registers of said processors, said subordinate processor transfers
data, which is to be written on said ring type network, to said
main processor, thereby said data is written into said registers of
all said processors from said main processor through said ring type
network.
2. The data communication method according to claim 1, wherein each
of said processors has a buffer for temporarily holding data
transferred through said ring type network; each of said processors
holds said data in said buffer until a signal for allowing writing
to all said processors is arrived from said main processor via said
ring type network; and each of said processors writes said data
into said register in response to arrival of said signal.
3. The data communication method according to claim 1, wherein each
of said processors has a unique identification number; each of said
processors performs data transfer through said ring type network by
adding said identification number; and when said main processor
performs data transfer for writing said data into said registers of
all said processors through said ring type network, each of said
processors performs said data transfer by adding a delay which
depends on said identification number added upon said data
transfer.
4. The data communication method according to claim 2, wherein each
of said processors has a unique identification number; each of said
processors performs data transfer through said ring type network by
adding said identification number; and when said data is stored
into said register from said buffer in response to arrival of said
signal for allowing said writing to all said processors from said
main processor via said ring type network, each of said processors
adds a delay which depends on said identification number.
Description
BACKGROUND OF THE INVENTION
[0001] The present invention relates to a data communication method
in a shared memory multiprocessor system and, in particular, to
realizing data communication with coherence being maintained and
speed-up of such data communication, and further to a coherence
control.
[0002] In a shared memory multiprocessor system where a plurality
of processors share a memory, it is necessary to communicate shared
data between the processors upon executing parallel programs.
[0003] Conventionally, when performing communication of the shared
data, a process called a lock process has been executed for
maintaining coherence.
[0004] This process is a process for executing an exclusive control
of data to allow one of the processors to exclusively read/write
data shared by the processors.
[0005] The exclusive control can avoid an occurrence of wrong
processing wherein, for example, the plurality of processors
read/write simultaneously so that a readout value of data differs
depending on order of access.
[0006] For realizing the exclusive control, a test and set
instruction or the like provided for the processors is normally
used.
[0007] The test and set instruction (1) reads out data, which is
stored at a certain address on a memory, into a register; (2)
writes a value "1" into the memory at the same address; and (3)
determines whether a value read out into the register is "0".
[0008] The test and set instruction ensures inseparable execution
of the foregoing operations (1) to (3) without being influenced or
interrupted from the other processors during the execution
thereof.
[0009] Only when the value read out into the register by the test
and set instruction is "0", the subject processor determines that
the exclusive control is realized, so that the subsequent exclusive
control performs necessary processing.
[0010] After this processing is finished, this processor writes a
value "0" into the memory at the same address, thereby to allow the
other processors to realize the exclusive control.
[0011] On the other hand, if the value read out into the register
by the test and set instruction is "1", the subject processor,
determining that the exclusive control is not realized, executes
again the test and set instruction, and then repeats it until a
value read out into the register becomes "0".
[0012] The exclusive control using the test and set instruction is
described in Kisaburo Nakazawa, "Computer Architecture and
Configuration Method", Asakura Bookstore, November 1995,
pp.388-389.
[0013] It is time-consuming to perform the data communication with
coherence being maintained between the processors by executing the
exclusive control based on the conventional lock process.
[0014] Particularly, when data transfer is frequently required, the
processing speed may be largely lowered due to influence of the
lock process.
[0015] This is caused by the fact that the lock process always
requires data read/write processing on a main memory so that every
execution of the test and set instruction requires much time.
[0016] Further, since the data read/write processing on the main
memory based on the lock process is executed with respect to one
same address, lowering of the memory access performance may be
induced.
SUMMARY OF THE INVENTION
[0017] It is an object of the present invention to perform
high-speed data communication with coherence being maintained
between processors by adding a simple hardware structure, without
using a time-consuming instruction such as a test and set
instruction.
[0018] For accomplishing the foregoing object, in the present
invention, each of processors in a shared memory multiprocessor
system has a register allocated thereto for shared data
communication, and shared data is written into these registers.
[0019] Each processor has a duplicate of the registers of the other
processors. The processors have ring type network communication
channels therebetween, and transfer the contents of the registers
via these communication channels.
[0020] In data communication between the registers connected via
the ring type network communication channels, one of a plurality of
processors is set to be a main processor, and the other processors
are set to be subordinate processors. When performing data writing
into the registers of the respective processors, the subordinate
processor transfers data to be written to the main processor
through the ring type network. The data to be written is written
into the registers of all the processors from the main processor
through the ring type network.
[0021] Further, in the data transfer, buffers are provided for
temporarily holding data transferred through the ring type network.
Until a signal for allowing writing to all the processors is
arrived from the main processor via the ring type network, the data
is held in the buffer, and then the data is written into the
register after arrival of the signal.
[0022] Other objects, features and advantages of the invention will
become apparent from the following description of the embodiments
of the invention taken in conjunction with the accompanying
drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0023] FIG. 1 is a diagram showing a configuration of a shared
memory multiprocessor system in the field of the present
invention;
[0024] FIG. 2 is an exemplary diagram showing a relationship
between parallel processing and data transfer in a multiprocessor
system to be dealt with by the present invention;
[0025] FIG. 3 is a diagram showing a configuration of a shared
memory multiprocessor system having shared data registers and
interprocessor communication channels, according to the present
invention;
[0026] FIG. 4 is a diagram showing an example of data transfer
processing in a ring type network, according to the present
invention;
[0027] FIG. 5 is an exemplary diagram showing a processing method
for maintaining coherence in data transfer processing in a ring
type network, according to the present invention;
[0028] FIG. 6 is an exemplary diagram showing an operation example
based on a processing method for maintaining coherence in data
transfer processing in a ring type network, according to the
present invention;
[0029] FIG. 7 is a diagram showing a configuration of a shared
memory multiprocessor system which is provided with buffers in
addition to shared data registers and interprocessor communication
channels, according to the present invention;
[0030] FIG. 8 is an exemplary diagram showing an operation example
based on a processing method for maintaining coherence using
buffers in data transfer processing in a ring type network,
according to the present invention;
[0031] FIG. 9 is a diagram showing a configuration of a shared
memory multiprocessor system which is provided with registers for
delay time in addition to shared data registers and interprocessor
communication channels, according to the present invention; and
[0032] FIG. 10 is a diagram showing a configuration of a shared
memory multiprocessor system which is provided with buffers and
buffers for delay time in addition to shared data registers and
interprocessor communication channels, according to the present
invention.
DESCRIPTION OF THE EMBODIMENT
[0033] A shared memory multiprocessor system in the field of the
present invention includes, as shown in FIG. 1, a plurality of
processors (1), a system bus (31), a memory controller (32) and a
main memory (4).
[0034] A cache memory (2) is normally provided for each processor
(1), and a shared cache (33) may also be provided for the
processors (1).
[0035] In the figure, the system bus (31), the memory controller
(32) and the shared cache (33) are collectively shown as memory
controller, bus, shared cache (3).
[0036] In the shared memory multiprocessor system, the processors
(1) logically share the single main memory (4), and access to the
main memory (4) from each processor (1) is carried out via the
system bus (31) and the memory controller (32). The system bus (31)
may be replaced with a switch arrangement.
[0037] Each cache memory (2) allows the associated processor (1) to
read/write a portion of data of the main memory (4) at high speed,
and thus stores data necessary for the associated processor
(1).
[0038] Accordingly, data stored in the cache memories (2) of the
respective processors (1) include data of different addresses.
[0039] On the other hand, the shared cache (33) is shared by the
plurality of processors (1), and thus stores data and so on
required by the respective processors (1).
[0040] The shared cache (33) is arranged at a position nearer to
the main memory (4) as compared with the cache memories (2) in the
processors (1). Therefore, although much time is required for
read/write processing, a cache memory with a larger capacity is
normally used for the shared cache (33), thus resulting in high
possibility of required data located in the shared cache (33).
[0041] Access to the shared cache (33) from each processor (1) is
performed via the system bus (31) and the memory controller (32).
Data transfer is carried out between the shared cache (33) and the
main memory (4).
[0042] FIG. 2 exemplarily shows an operation of parallel processing
in the multiprocessor system.
[0043] FIG. 2, in which a time axis is vertically shown,
illustrates an example wherein eight processors execute one program
(6), a processor #0 executes a sequential section (7), and eight
processors #0 to #7 execute parallel sections (8).
[0044] At a portion where shifting occurs from the sequential
portion (7) to the parallel portions (8) and vice versa or at a
portion where shifting occurs from a certain parallel processing to
another parallel processing (in FIG. 2, a portion where shifting
occurs from the parallel processing 2 to the parallel processing
3), a synchronization point (5) is provided for matching the timing
of, for example, performing data transfer (9) between the
processors.
[0045] For ensuring efficient processing in the multiprocessor
system, it is necessary to perform processing at high speed at each
synchronization point to efficiently achieve the data transfer
(9).
[0046] At the synchronization point (5) where shifting occurs from
the sequential processing 1 to the parallel processing 1, the data
transfer (9) is carried out for distributing necessary data from
processor #0 executing the sequential processing 1 to the
processors which will execute the parallel processing 1. At the
synchronization point (5) where shifting occurs from parallel
processing 1 to sequential processing 2, the data transfer (9) is
carried out for collecting data from the processors executing the
parallel processing 1 to processor #0 which will execute the
sequential processing 2. At the synchronization point (5) where
shifting occurs from the parallel processing 2 to parallel
processing 3, the data transfer (9) is carried out for exchanging
data between the processors.
[0047] In the multiprocessor processing, it has been a large
problem as to how to maintain coherence of shared data between the
processors. In particular, while data transfer relating to
synchronization among the processors requires strict coherence, the
speed of read/write processing for shared data is a factor which
largely influences the parallel processing performance.
[0048] However, another problem arises in view of cost if
complicated hardware is provided for solving such a problem.
[0049] Therefore, the present invention realizes a technique for
performing high-speed data communication with simple hardware while
maintaining coherence of shared data.
[0050] FIG. 3 shows shared data registers (10) and interprocessor
communication channels (11).
[0051] As interprocessor communication channels, there may be
proposed a crossbar type which forms a network closely in a
one-to-one fashion, a star type which forms a star type network
using a communication hub, and a ring type which forms a ring type
network.
[0052] The crossbar type is excellent in communication speed, but
complicated and high in cost, and further, flexibility is low
against increment in number of processors.
[0053] The star type has a problem in that communication is
concentrated to the hub.
[0054] The ring type is the simplest in structure and flexible
against increment in number of processors, but low in
performance.
[0055] The present invention adopts the ring type as the
interprocessor communication channels (11) and realizes a coherence
maintained communication system in the ring type network. FIG. 3
shows an embodiment wherein the ring type is used as the
interprocessor communication channels (11). It may also be arranged
that the shared data registers (10) are not provided and shared
data is directly written into memories.
[0056] FIG. 4 shows an operation example of writing into the shared
data registers (10) in the multiprocessor system having the
interprocessor communication channels (11) according to the ring
type network.
[0057] FIG. 4 shows in time sequence an operation wherein processor
#0 and processor #2 simultaneously write different data A and B at
the same address. A box of each shared data register (10)
exemplarily shows data buffering relative to an address X.
[0058] Through the interprocessor communication channels (11) of
the ring type network, a write A request for writing data A at
certain address X is transferred from processor #0 to processors
#1.fwdarw.#2.fwdarw.#3.
[0059] Similarly, a write B request for writing data B at the same
address X is transferred from processor #2 to processors
#3.fwdarw.#0.fwdarw.#1.
[0060] If the write request is transferred in the interprocessor
communication channels 11 in this manner, the final state will be
that data B is written at address X in processors #0 and #1, while
data A is written at the same address X in processors #2 and
#3.
[0061] This state means that if data is read out pursuant to a load
instruction for the same address X, the read-out data differs
depending on a reading processor.
[0062] Generally, simultaneous writing of different data at the
same address should be inhibited by software. However, even if such
a state occurs due to a program bug, coherence should be ensured
among final values of data by means of hardware.
[0063] FIG. 5 exemplarily shows a coherence maintained
communication system in the interprocessor communication channels
(11) of the ring type network according to the present
invention.
[0064] In an embodiment of the present invention, a processor whose
processor number is the smallest is determined to be a master
processor, while the other processors are determined to be
subordinate processors, and only the master processor is allowed to
start writing.
[0065] In FIG. 5, processor #0 becomes a master processor, and
processors #1, #2 and #3 become subordinate processors.
[0066] In case of a write request from processor #0 being the
master processor, writing into the shared data registers 10 of the
processors are executed in order of
(i).fwdarw.(ii).fwdarw.(iii).fwdarw.(iv) as shown in FIG. 5.
[0067] On the other hand, in case of a write request from processor
#1 being the subordinate processor, a write request is transferred
in order of (i).fwdarw.(ii).fwdarw.(iii) without actual writing and
reaches processor #0 being the master processor.
[0068] Then, from processor #0, actual writing into the shared data
registers (10) of the processors is executed in order of
(iv).fwdarw.(v).fwdarw.(vi).fwdarw.(vii).
[0069] Likewise, in case of the other subordinate processors, only
a write request is transferred with no execution of actual writing
until it reaches processor #0 being the master processor, and then
actual writing is executed in order from processor #0.
[0070] If arrival of a request from another processor and
generation of a write request from the subject processor occur
simultaneously, each processor preferentially processes the request
from the other processor and makes its own request when there is no
request from the other processors.
[0071] Determination of the master processor can be carried out by
transferring processor numbers through the interprocessor
communication channels (11) in the following manner.
[0072] (1) Each processor transmits its own processor number into
the interprocessor communication channel (11);
[0073] (2) Upon receipt of a processor number of another processor
from the interprocessor communication channel (11), the subject
processor compares it with its own processor number. If its own
processor number is smaller, the subject processor transmits its
own processor number into the interprocessor communication channel
(11). On the other hand, if its own processor number is greater,
the subject processor transmits the received processor number of
the other processor into the interprocessor communication channel
(11) and becomes a subordinate processor.
[0074] If the received processor number is equal to its own
processor number, the subject processor becomes a master
processor.
[0075] FIG. 6 is a diagram showing an operation, wherein writing
into the shared data registers (10) is executed according to the
method of the present invention when, like FIG. 4, a request from
processor #0 for writing data A at address X and a request from
processor #2 for writing data B at the same address X occur
simultaneously.
[0076] First, with respect to the write request for data A from
processor #0, since processor #0 is the master processor, writing
of data A into the shared data registers (10) is executed in order
from processor #0.
[0077] On the other hand, since processor #2 is the subordinate
processor, processor #2 transfers only a write request to processor
#3 where no actual writing of data B is executed.
[0078] When this request reaches processor #0 being the master
processor, writing of data B into the shared data registers (10) is
executed in order from processor #0.
[0079] Through the foregoing operation, data A is first written
into the shared data registers (10) of all the processors, then
data B is overwritten thereon. Accordingly, values of the shared
data registers (10) of all the processors finally become data B and
thus match with each other.
[0080] FIG. 7 shows another embodiment of the present invention,
wherein a buffer (12) is provided in each processor (1), in
addition to the shared data register (10).
[0081] Each of the interprocessor communication channels (11)
includes a data communication channel (15) and an address/command
communication channel (16). Data transferred in each of the data
communication channels (15) is first recorded in a buffer for data
(13), i.e. a data buffer, of the buffer (12).
[0082] On the other hand, address/data transfer ID transferred in
each of the address/command communication channels (16) is first
recorded in a buffer for address (14), i.e. an address buffer, of
the buffer (12).
[0083] In the address/command communication channel are
transferred, in addition to an address at which data should be
written, an ID for corresponding data transfer, a command for
commanding data writing, a command for commanding transfer of data
without writing it, a command for commanding writing of data
identified by the ID into the shared data register (10) from the
buffer (12), a command for notifying completion of data writing, a
command for notifying a processor number for determining the master
processor or the subordinate processor, a command for commanding
data reading, and a command for commanding initialization of data
and address.
[0084] If correspondence between data and address/command can be
identified, it is not necessary that the data and the
address/command arrive simultaneously with each other.
[0085] FIG. 8 is a diagram showing an operation, wherein writing
into the shared data registers (10) is executed according to the
method of the present invention as shown in FIG. 7 when, like FIG.
4, a request from processor #0 for writing data A at address X and
a request from processor #2 for writing data B at the same address
X occur simultaneously.
[0086] In FIG. 8, a dot-meshed arrow represents that data or
address/command is transmitted in a corresponding communication
channel or wiring.
[0087] Processor #0 is the master processor, and transmits data A
into the data communication channel (15) and a command for writing
at address X into the address/command communication channel
(16).
[0088] In response to receipt of this data transfer, each processor
first records data A and address X in the buffer (12), then writes
data A into the shared data register (10) at a portion where data
of address X should be recorded.
[0089] Processor #3 being the final subordinate processor, in
response to receipt of data transfer of data A, notifies only the
completion of data transfer processing of data A. For enabling it,
processor #3 needs to recognize itself as being the final
subordinate processor.
[0090] Processor #3 may transfer the received data and
address/command to processor #0 being the master processor. In this
event, the master processor may interpret a write command received
from the interprocessor communication channel as a notification of
the completion.
[0091] On the other hand, with respect to a request for writing
data B at address X transferred from processor #2, since processor
#2 is the subordinate processor, processor #2 transmits into the
address/command communication channel (16) a command for
transferring data without writing it at address X.
[0092] Through this data communication, each of the processors
records data B and address X into the buffer (12), but executes no
writing into the shared data register (10).
[0093] When this data transfer reaches the master processor, the
master processor changes the received address/command, which is to
be transmitted to the address/command communication channel (16),
into a command for writing at address X, and then transmits it into
the address/command communication channel (16).
[0094] Thereafter, data B is written into the shared data register
(10) in each processor in the same manner as data A.
[0095] When the command for writing data B at address X arrives
processor #2 being the original sender of data B, processor #2
changes the received command into a command for writing data
identified by the corresponding data transfer ID into the shared
data register (10) from the buffer (12), and transmits it, but does
not transmit data B. This is because data B has already been
recorded in the buffer (12) of processor #3.
[0096] FIG. 9 shows a configuration wherein registers for delay
time (18), i.e. delay time registers, are further provided in
addition to the shared data registers (10) and the interprocessor
communication channels (11) shown in FIG. 3.
[0097] Each processor carries out data transfer by adding a
processor ID unique to each processor, and records this processor
ID in the register for delay time (18).
[0098] The processor ID is determined in the following manner.
Assuming that the number of all the processors is Y, the master
processor is assigned Y-1, and subsequently, the other processors
are assigned Y-2, Y-3, . . . , 1, 0 in order of connection to the
ring type network, i.e. the processor ID is reduced one by one in
order of the connection.
[0099] Assuming that a time required for data transfer from one
processor to the next processor is "1", actual data recording into
the shared data register 10 is delayed by a time corresponding to a
value recorded in the register for delay time (18).
[0100] According to the foregoing processor ID determining method,
the processor having the processor ID of Z completes data transfer
time Z earlier than the processor having the processor ID of "0".
Thus, by delaying actual data recording into the shared data
register (10) by the time Z, a relationship of order of data
writing can be guaranteed in terms of an absolute time on which
data transfer is started.
[0101] In case of data of which data transfer is simultaneously
started, those data are simultaneously recorded into the shared
data registers (10). In this event, if an address is the same or
contention occurs with respect to a resource for writing, a
preferential control based on processor ID may be executed commonly
over all the processors, thereby to maintain coherence.
[0102] In FIG. 10, buffers for delay time similar to the registers
for delay time (18) are added to the configuration of FIG. 7 having
the buffers (12).
[0103] Like the configuration of FIG. 9, data recording into the
shared data registers (10) can be carried out by ensuring a
relationship of order of data writing in terms of an absolute time
on which data transfer is started. A difference from the
configuration of FIG. 9 resides in that data recording is delayed
by holding data in the buffers (12).
[0104] According to the present invention, the data transfer
processing with coherence being maintained can be performed at high
speed in the multiprocessor system.
[0105] Further, by providing the buffers, traffic in the
communication channels can be reduced to realize the data transfer
processing with increased using efficiency of the communication
channels. Moreover, by providing delays with respect to the timing
of data writing depending on a processor as an initial data sender,
a relationship of order of data writing can be guaranteed in terms
of an absolute time on which data transfer is started.
[0106] It should be further understood by those skilled in the art
that although the foregoing description has been made on
embodiments of the invention, the invention is not limited thereto
and various changes and modifications may be made without departing
from the spirit of the invention and the scope of the appended
claims.
* * * * *