U.S. patent number 8,832,324 [Application Number 13/835,205] was granted by the patent office on 2014-09-09 for first-in-first-out queue-based command spreading.
This patent grant is currently assigned to International Business Machines Corporation. The grantee listed for this patent is International Business Machines Corporation. Invention is credited to Mark R. Hodges, Patrick J. Meaney, Vesselina K. Papazova.
United States Patent |
8,832,324 |
Hodges , et al. |
September 9, 2014 |
First-in-first-out queue-based command spreading
Abstract
Embodiments relate to first-in-first-out (FIFO) queue based
command spreading. An aspect includes receiving a plurality of
commands by a first level priority stage of a memory control unit
(MCU), wherein each of the plurality of commands is associated with
one of a plurality of ports located on a buffer chip. Another
aspect includes storing each of the plurality of commands in a FIFO
queue of a plurality of FIFO queues in the MCU, wherein each of the
plurality of commands is assigned to a FIFO queue based on the
command's associated port, and each of the plurality of FIFO queues
is associated with a respective one of the plurality of ports
located on the buffer chip. Another aspect includes selecting a
FIFO queue of the plurality of FIFO queues and forwarding a command
from the selected FIFO queue to the buffer chip by the second level
priority stage. Another aspect includes a third level priority on
the buffer chip associated with each respective FIFO queue to help
optimize the bandwidth on the returning upstream fetch bus.
Inventors: |
Hodges; Mark R. (Endicott,
NY), Papazova; Vesselina K. (Highland, NY), Meaney;
Patrick J. (Poughkeepsie, NY) |
Applicant: |
Name |
City |
State |
Country |
Type |
International Business Machines Corporation |
Armonk |
NY |
US |
|
|
Assignee: |
International Business Machines
Corporation (Armonk, NY)
|
Appl.
No.: |
13/835,205 |
Filed: |
March 15, 2013 |
Field of
Search: |
;710/40,120,244 |
References Cited
[Referenced By]
U.S. Patent Documents
Foreign Patent Documents
|
|
|
|
|
|
|
WO2011160923 |
|
Dec 2011 |
|
WO |
|
WO2012089507 |
|
Jul 2012 |
|
WO |
|
Primary Examiner: Unelus; Ernest
Attorney, Agent or Firm: Cantor Colburn LLP McNamara;
Margaret
Claims
What is claimed is:
1. A computer implemented method for first-in-first-out (FIFO)
queue based command spreading in a memory control unit (MCU), the
method comprising: receiving a plurality of commands from one or
more command queues by a first level priority stage of the MCU,
wherein each of the plurality of commands is associated with one of
a plurality of ports located on a buffer chip that is in
communication with the MCU; storing each of the plurality of
commands in a FIFO queue of a plurality of FIFO queues in the MCU,
wherein each of the plurality of commands is assigned to a FIFO
queue based on the command's associated port, and each of the
plurality of FIFO queues is associated with a respective one of the
plurality of ports located on the buffer chip; selecting, by a
second level priority stage of the MCU, a FIFO queue of the
plurality of FIFO queues; and forwarding a command from the
selected FIFO queue to the buffer chip by the second level priority
stage.
2. The method of claim 1, wherein the plurality of FIFO queues
comprises a plurality of fetch FIFO queues, each of the plurality
of fetch FIFO queues being associated with a respective port on the
buffer chip; wherein the MCU further comprises a second plurality
of FIFO queues comprising a plurality of store FIFO queues, each of
the plurality of store FIFO queues being associated with a
respective port on the buffer chip; and further comprising storing
each of the plurality of commands in a FIFO queue of the plurality
of fetch FIFO queues and store FIFO queues based on whether the
command is a fetch or a store.
3. The method of claim 2, further comprising tracking by the second
level priority stage a number of fetch and store commands currently
being handled by the buffer chip, and selecting, by the second
level priority stage, a FIFO queue of the plurality of fetch FIFO
queues and store FIFO queues based on the tracking.
4. The method of claim 1, wherein selecting, by the second level
priority stage of the MCU, a FIFO queue of the plurality of FIFO
queues comprises toggling sequentially among the plurality of FIFO
queues.
5. The method of claim 1, further comprising tracking by the second
level priority stage a number of previously forwarded commands from
the plurality of FIFO queues, and selecting, by the second level
priority stage, a FIFO queue of the plurality of FIFO queues based
on the tracking.
6. The method of claim 1, further comprising storing an address
associated with each of the plurality of commands in an address
register associated with the command's assigned FIFO queue.
7. The method of claim 1, further comprising assigning a command
tag to the forwarded command by the second level priority stage,
and determining that a command done indication received from the
buffer chip is associated with the forwarded command based on the
command tag.
8. The method of claim 1, further comprising, based on storing a
command from a command queue of the one or more command queues by
the first level priority stage in a FIFO queue of the plurality of
FIFO queues, releasing the command queue.
9. The method of claim 1, further comprising selecting, by a third
level priority stage associated with a port queue in the buffer
chip, commands from the port queue for processing by the buffer
chip.
10. A computer system for first-in-first-out (FIFO) queue based
command spreading, the system comprising: a memory control unit
(MCU) comprising: a first level priority stage; a second level
priority stage; and a plurality of FIFO queues, wherein each of the
plurality of FIFO queues is associated with a respective one of a
plurality of ports located on a buffer chip that is in
communication with the MCU, the system configured to perform a
method comprising: receiving a plurality of commands from one or
more command queues by the first level priority stage, wherein each
of the plurality of commands is associated with one of the
plurality of ports located on the buffer chip; storing each of the
plurality of commands in a FIFO queue of the plurality of FIFO
queues, wherein each of the plurality of commands is assigned to a
FIFO queue based on the command's associated port; selecting, by
the second level priority stage, a FIFO queue of the plurality of
FIFO queues; and forwarding a command from the selected FIFO queue
to the buffer chip by the second level priority stage.
11. The system of claim 10, wherein the plurality of FIFO queues
comprises a plurality of fetch FIFO queues, each of the plurality
of fetch FIFO queues being associated with a respective port on the
buffer chip; wherein the MCU further comprises a second plurality
of FIFO queues comprising a plurality of store FIFO queues, each of
the plurality of store FIFO queues being associated with a
respective port on the buffer chip; and further comprising storing
each of the plurality of commands in a FIFO queue of the plurality
of fetch FIFO queues and store FIFO queues based on whether the
command is a fetch or a store.
12. The system of claim 11, further comprising tracking by the
second level priority stage a number of fetch and store commands
currently being handled by the buffer chip, and selecting, by the
second level priority stage, a FIFO queue of the plurality of fetch
FIFO queues and store FIFO queues based on the tracking.
13. The system of claim 10, wherein selecting, by the second level
priority stage of the MCU, a FIFO queue of the plurality of FIFO
queues comprises toggling sequentially among the plurality of FIFO
queues.
14. The system of claim 10, further comprising tracking by the
second level priority stage a number of previously forwarded
commands from the plurality of FIFO queues, and selecting, by the
second level priority stage, a FIFO queue of the plurality of FIFO
queues based on the tracking.
15. The system of claim 10, further comprising storing an address
associated with each of the plurality of commands in an address
register associated with the command's assigned FIFO queue.
16. The system of claim 10, further comprising assigning a command
tag to the forwarded command by the second level priority stage,
and determining that a command done indication received from the
buffer chip is associated with the forwarded command based on the
command tag.
17. The system of claim 10, further comprising, based on storing a
command from a command queue of the one or more command queues by
the first level priority stage in a FIFO queue of the plurality of
FIFO queues, releasing the command queue.
18. The system of claim 10, further comprising a third level
priority stage associated with a port queue in the buffer chip, the
third level priority stage configured to select commands from the
port queue for processing by the buffer chip.
19. A computer program product for implementing first-in-first-out
(FIFO) queue based command spreading in a memory control unit
(MCU), the computer program product comprising: a tangible,
non-transitory storage medium readable by a processing circuit and
storing instructions for execution by the processing circuit for
performing a method comprising: receiving a plurality of commands
from one or more command queues by a first level priority stage of
the MCU, wherein each of the plurality of commands is associated
with one of a plurality of ports located on a buffer chip that is
in communication with the MCU; storing each of the plurality of
commands in a FIFO queue of a plurality of FIFO queues in the MCU,
wherein each of the plurality of commands is assigned to a FIFO
queue based on the command's associated port, and each of the
plurality of FIFO queues is associated with a respective one of the
plurality of ports located on the buffer chip; selecting, by a
second level priority stage of the MCU, a FIFO queue of the
plurality of FIFO queues; and forwarding a command from the
selected FIFO queue to the buffer chip by the second level priority
stage.
20. The computer program product of claim 19, wherein the plurality
of FIFO queues comprises a plurality of fetch FIFO queues, each of
the plurality of fetch FIFO queues being associated with a
respective port on the buffer chip; wherein the MCU further
comprises a second plurality of FIFO queues comprising a plurality
of store FIFO queues, each of the plurality of store FIFO queues
being associated with a respective port on the buffer chip; and
further comprising storing each of the plurality of commands in a
FIFO queue of the plurality of fetch FIFO queues and store FIFO
queues based on whether the command is a fetch or a store.
Description
BACKGROUND
The present invention relates generally to computer memory, and
more specifically, to first-in-first-out (FIFO) queue-based command
spreading.
Contemporary high performance computing main memory systems are
generally composed of one or more memory devices, which are
connected to one or more memory controllers and/or processors via
one or more memory interface elements such as buffers, hubs,
bus-to-bus converters, etc. The memory devices are generally
located on a memory subsystem such as a memory card or memory
module and are often connected via a pluggable interconnection
system (e.g., one or more connectors) to a system board (e.g., a PC
motherboard).
Overall computer system performance is affected by each of the key
elements of the computer structure, including the
performance/structure of the processor(s), any memory cache(s), the
input/output (I/O) subsystem(s), the efficiency of the memory
control function(s), the performance of the main memory devices(s)
and any associated memory interface elements, and the type and
structure of the memory interconnect interface(s).
Extensive research and development efforts are invested by the
industry, on an ongoing basis, to create improved and/or innovative
solutions to maximizing overall system performance and density by
improving the memory system/subsystem design and/or structure.
High-availability systems present further challenges as related to
overall system reliability due to customer expectations that new
computer systems will markedly surpass existing systems in regard
to mean-time-between-failure (MTBF), in addition to offering
additional functions, increased performance, increased storage,
lower operating costs, etc. Other frequent customer requirements
further exacerbate the memory system design challenges, and include
such items as ease of upgrade and reduced system environmental
impact (such as space, power and cooling). In addition, customers
are requiring the ability to access an increasing number of higher
density memory devices (e.g., DDR3 and DDR4 SDRAMs) at faster and
faster access speeds.
In view of varying cost, capacity, and scalability requirements, a
wide number of memory system options may need to be considered.
Often a choice needs to be made between using an asynchronous
boundary between a processor and memory buffer chips or designing a
fully synchronous system. An asynchronous design allows the
flexibility of running the processor at a fixed frequency, while
memory buffer chips can be programmed to varying frequencies to
match the desired memory components. For example, if cost is most
important, a slower more widely available dual in-line memory
module (DIMM) can be used. In contrast, if performance is
paramount, then a leading edge technology DIMM can be used. This
type of memory system architecture may work well in systems where
each memory channel runs independently. However, this approach
typically falls short in high-availability systems.
Redundant array of independent memory (RAIM) systems have been
developed to improve performance and/or to increase the
availability of storage systems. RAIM distributes data across
several independent memory modules, where each memory module
contains one or more memory devices. There are many different RAIM
schemes that have been developed, each having different
characteristics, and different pros and cons associated with them.
Performance, availability, and utilization/efficiency (e.g., the
percentage of the memory devices that actually hold customer data)
are perhaps the most important. The tradeoffs associated with
various schemes have to be carefully considered because
improvements in one attribute can often result in reductions in
another. Examples of RAIM systems may be found, for instance, in
U.S. Patent Publication Number 2011/0320918 titled "RAIM System
Using Decoding of Virtual ECC", filed on Jun. 24, 2010, the
contents of which are hereby incorporated by reference in its
entirety, and in U.S. Patent Publication Number 2011/0320914 titled
"Error Correction and Detection in a Redundant Memory System",
filed on Jun. 24, 2010, the contents of which are hereby
incorporated by reference in its entirety.
High availability systems, such as RAIM systems, can include a
number of clock domains in various subsystems. Efficient
integration of subsystems including different clock domains
presents a number of challenges to establish synchronization
timing, detection of synchronization issues, and recovery of
synchronization.
Various memory functions, such as command scheduling to the DRAM,
power management, refresh and periodic calibration procedures may
be moved from the memory control unit (MCU) to a buffer chip.
Moving such functions to the buffer chip may impact utilization of
the interface, or bus, between the MCU and the buffer chip.
Utilization of the interface between the MCU and the buffer chip
needs to be maintained at a high level to achieve good performance
and memory throughput. A buffer chip may have multiple command
schedulers, or DRAM ports, for parallel memory access. In
multiple-port buffer chips with independent engines, it may be
difficult to keep all ports as busy as possible and increase
throughput of the buffer chip. Commands may be sent to the buffer
chip from the MCU sequentially. However, this may cause one or more
of the ports on the buffer chip to starve if, for example, a
relatively large number of earlier, sequentially-received commands
are sent to a first port on the buffer chip while a second port is
idle, and a later-received command is then sent to the second port
after the relatively large number of earlier-received commands.
SUMMARY
Embodiments include a method, system, and computer program product
for first-in-first-out (FIFO) queue based command spreading in a
MCU. An aspect includes receiving a plurality of commands from one
or more command queues by a first level priority stage of the MCU,
wherein each of the plurality of commands is associated with one of
a plurality of ports located on a buffer chip that is in
communication with the MCU. Another aspect includes storing each of
the plurality of commands in a FIFO queue of a plurality of FIFO
queues in the MCU, wherein each of the plurality of commands is
assigned to a FIFO queue based on the command's associated port,
and each of the plurality of FIFO queues is associated with a
respective one of the plurality of ports located on the buffer
chip. Another aspect includes selecting, by a second level priority
stage of the MCU, a FIFO queue of the plurality of FIFO queues. Yet
another aspect includes forwarding a command from the selected FIFO
queue to the buffer chip by the second level priority stage.
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
The subject matter which is regarded as embodiments is particularly
pointed out and distinctly claimed in the claims at the conclusion
of the specification. The forgoing and other features, and
advantages of the embodiments are apparent from the following
detailed description taken in conjunction with the accompanying
drawings in which:
FIG. 1 depicts a memory system in accordance with an
embodiment;
FIG. 2 depicts a memory subsystem in a planar configuration in
accordance with an embodiment;
FIG. 3 depicts a memory subsystem in a buffered DIMM configuration
in accordance with an embodiment;
FIG. 4 depicts a memory subsystem with dual asynchronous and
synchronous memory operation modes in accordance with an
embodiment;
FIG. 5 depicts a memory subsystem channel and interfaces in
accordance with an embodiment;
FIG. 6 depicts a process flow for providing synchronous operation
in a memory subsystem in accordance with an embodiment;
FIG. 7 depicts a process flow for establishing alignment between
nest and memory domains in a memory subsystem in accordance with an
embodiment;
FIG. 8 depicts a timing diagram of synchronizing a memory subsystem
in accordance with an embodiment;
FIG. 9 depicts a MCU that implements FIFO queue-based command
spreading in accordance with an embodiment;
FIG. 10 depicts a process flow for FIFO queue-based command
spreading in a MCU in accordance with an embodiment; and
FIG. 11 illustrates a computer program product in accordance with
an embodiment.
DETAILED DESCRIPTION
Exemplary embodiments provide FIFO queue-based command spreading.
The command spreading is implemented via multiple FIFO queues
located in a MCU. Each FIFO queue corresponds to a particular port
on a buffer chip that is located between the MCU and the DRAM. An
arbitration mechanism, including first and second level priority
stages, is used in conjunction with the FIFO queues. The various
ports on the buffer chip are kept busy by use of the FIFO queues,
allowing good utilization of the bus between the MCU and the buffer
chip, and increasing throughput of the buffer chip. The number of
command queues in the MCU, and power consumption of the computing
system, may also be reduced, as the command queues may be released
after commands are issued from the command queues to the FIFO
queues.
The first level priority stage receives command requests from the
command queues of the MCU, and directs the commands associated with
the command requests to a particular FIFO queue of the set of FIFO
queues based on the buffer chip port address associated with each
command. The FIFO queues may each correspond to a particular port
on a buffer chip that is in communication with the MCU. In some
embodiments, commands may be further assigned to a FIFO queue by
the first level priority stage based on command type, i.e., fetch
and store. In such an embodiment, each port may have a pair of
respective FIFO queues in the MCU: a first FIFO queue of the pair
of FIFO queues may correspond to fetches for a particular port, and
the second FIFO queue of the pair of FIFO queues may correspond to
stores for the particular port. In some embodiments, a single first
level priority stage may assign commands from the command queues to
the various FIFO queues for all of the buffer chips that are in
communication with the MCU. In other embodiments, each buffer chip
may have a separate first level priority stage. In further
embodiments, a first level priority stage may include separate
selection mechanisms for fetch and store commands.
The commands wait in the FIFO queues until they are forwarded to
the buffer chip by the second level priority stage. The second
level priority stage selects among the various FIFO queues to send
commands from the FIFO queues to the buffer chip. In some
embodiments, each buffer chip that is in communication with the MCU
may have a respective second level priority stage in the MCU. In
some embodiments, a second level priority stage may comprise a
toggle mechanism that sequentially steps through the FIFO queues
that are assigned to a particular buffer chip. This keeps the ports
in the buffer chip busy by not allowing a port to starve if a
command is available for that port. For example, if there is a
command for port 0 and a command for port 1 waiting in the FIFO
queues corresponding to port 0 and port 1 in the MCU, and a command
to port 0 was last selected and forwarded to the buffer chip by the
second level priority stage, the FIFO queue corresponding to port 1
will be selected next by the second level priority stage to send a
command to the buffer chip. In further embodiments, the toggle
mechanism may be used in conjunction with tracking, by the second
level priority stage, the commands that have been sent to
particular ports, and selecting the next FIFO queue from which to
send a command to the buffer chip based on the tracked commands.
The tracking may store the destination of a predetermined number of
previously forwarded commands. For example, in an embodiment in
which five previously forwarded commands are tracked by the second
level priority stage, and four of the previously forwarded commands
were made to port 0 and one of the previously forwarded commands
was made to port 1, if a single command is arrives in the FIFO
queue for port 1, the command for port 1 will be sent to the buffer
chip next by the second level priority stage regardless of the
state of the toggle mechanism. The tracking may further take into
account the number of fetch and store commands that are currently
being handled by the various ports on the buffer chip, allowing
good utilization of fetch and store command reorder queues that are
located on the buffer chip.
FIG. 1 depicts an example memory system 100 which may be part of a
larger computer system structure. A control processor (CP) system
102 is a processing subsystem that includes at least one processor
104 configured to interface with a memory control unit (MCU) 106.
The processor 104 can be a multi-core processor or module that
processes read, write, and configuration requests from a system
controller (not depicted). The MCU 106 includes a memory controller
synchronous (MCS) 108, also referred to as a memory controller,
that controls communication with a number of channels 110 for
accessing a plurality of memory devices in a memory subsystem 112.
The MCU 106 and the MCS 108 may include one or more processing
circuits, or processing may be performed by or in conjunction with
the processor 104. In the example of FIG. 1, there are five
channels 110 that can support parallel memory accesses as a virtual
channel 111. In an embodiment, the memory system 100 is a
five-channel redundant array of independent memory (RAIM) system,
where four of the channels 110 provide access to columns of data
and check-bit memory, and a fifth channel provides access to RAIM
parity bits in the memory subsystem 112.
Each of the channels 110 is a synchronous channel which includes a
downstream bus 114 and an upstream bus 116. Each downstream bus 114
of a given channel 110 may include a different number of lanes or
links than a corresponding upstream bus 116. In the example of FIG.
1, each downstream bus 114 includes n-unidirectional high-speed
serial lanes and each upstream bus 116 includes m-unidirectional
high-speed serial lanes. Frames of commands and/or data can be
transmitted and received on each of the channels 110 as packets
that are decomposed into individual lanes for serial communication.
In an embodiment, packets are transmitted at about 9.6 gigabits per
second (Gbps), and each transmitting lane transmits four-bit groups
serially per channel 110. The memory subsystem 112 receives,
de-skews, and de-serializes each four-bit group per lane of the
downstream bus 114 to reconstruct a frame per channel 110 from the
MCU 106. Likewise, the memory subsystem 112 can transmit to the MCU
106 a frame of packets as four-bit groups per lane of the upstream
bus 116 per channel 110. Each frame can include one or more
packets, also referred to as transmission packets.
The CP system 102 may also include a cache subsystem 118 that
interfaces with the processor 104. A cache subsystem interface 122
of the CP system 102 provides a communication interface to the
cache subsystem 118. The cache subsystem interface 122 may receive
data from the memory subsystem 112 via the MCU 106 to store in the
cache subsystem 118.
FIG. 2 depicts an example of a memory subsystem 112a as an instance
of the memory subsystem 112 of FIG. 1 in a planar configuration 200
in accordance with an embodiment. The example of FIG. 2 only
depicts one channel 110 of the memory subsystem 112a; however, it
will be understood that the memory subsystem 112a can include
multiple instances of the planar configuration 200 as depicted in
FIG. 2, e.g., five instances. As illustrated in FIG. 2, the planar
configuration 200 includes a memory buffer chip 202 connected to a
plurality of dynamic random access memory (DRAM) devices 204 via
connectors 206. The DRAM devices 204 may be organized as ranks of
one or more dual in-line memory modules (DIMMs) 208. The each of
the connectors 206 is coupled to a double data rate (DDR) port 210,
also referred to as a memory interface port 210 of the memory
buffer chip 202, where each DDR port 210 can be coupled to more
than one connector 206. In the example of FIG. 2, the memory buffer
chip 202 includes DDR ports 210a, 210b, 210c, and 210d. The DDR
ports 210a and 210b are each coupled to a pair of connectors 206
and a shared memory buffer adaptor (MBA) 212a. The DDR ports 210c
and 210d may each be coupled to a single connector 206 and a shared
memory buffer adaptor (MBA) 212b. The DDR ports 210a-210d are
JEDEC-compliant memory interfaces for issuing memory commands and
reading and writing memory data to the DRAM devices 204.
The MBAs 212a and 212b include memory control logic for managing
accesses to the DRAM devices 204, as well as controlling timing,
refresh, calibration, and the like. The MBAs 212a and 212b can be
operated in parallel, such that an operation on DDR port 210a or
210b can be performed in parallel with an operation on DDR port
210c or 210d.
The memory buffer chip 202 also includes an interface 214
configured to communicate with a corresponding interface 216 of the
MCU 106 via the channel 110. Synchronous communication is
established between the interfaces 214 and 216. As such, a portion
of the memory buffer chip 202 including a memory buffer unit (MBU)
218 operates in a nest domain 220 which is synchronous with the MCS
108 of the CP system 102. A boundary layer 222 divides the nest
domain 220 from a memory domain 224. The MBAs 212a and 212b and the
DDR ports 210a-210d, as well as the DRAM devices 204 are in the
memory domain 224. A timing relationship between the nest domain
220 and the memory domain 224 is configurable, such that the memory
domain 224 can operate asynchronously relative to the nest domain
220, or the memory domain 224 can operate synchronously relative to
the nest domain 220. The boundary layer 222 is configurable to
operate in a synchronous transfer mode and an asynchronous transfer
mode between the nest and memory domains 220, 224. The memory
buffer chip 202 may also include one or more multiple-input
shift-registers (MISRs) 226, as further described herein. For
example, the MBA 212a can include one or more MISR 226a, and the
MBA 212b can include one or more MISR 226b. Other instances of
MISRs 226 can be included elsewhere within the memory system 100.
As a further example, one or more MISRs 226 can be positioned
individually or in a hierarchy that spans the MBU 218 and MBAs 212a
and 212b and/or in the MCU 106.
The boundary layer 222 is an asynchronous interface that permits
different DIMMs 208 or DRAM devices 204 of varying frequencies to
be installed into the memory domain 224 without the need to alter
the frequency of the nest domain 220. This allows the CP system 102
to remain intact during memory installs or upgrades, thereby
permitting greater flexibility in custom configurations. In the
asynchronous transfer mode, a handshake protocol can be used to
pass commands and data across the boundary layer 222 between the
nest and memory domains 220, 224. In the synchronous transfer mode,
timing of the memory domain 224 is phase adjusted to align with the
nest domain 220 such that a periodic alignment of the nest and
memory domains 220, 224 occurs at an alignment cycle in which
commands and data can cross the boundary layer 222.
The nest domain 220 is mainly responsible for reconstructing and
decoding the source synchronous channel packets, applying any
necessary addressing translations, performing coherency actions,
such as directory look-ups and cache accesses, and dispatching
memory operations to the memory domain 224. The memory domain 224
may include queues, a scheduler, dynamic power management controls,
hardware engines for calibrating the DDR ports 210a-210d, and
maintenance, diagnostic, and test engines for discovery and
management of correctable and uncorrectable errors. There may be
other functions in the nest or memory domain. For instance, there
may be a cache of embedded DRAM (eDRAM) memory with a corresponding
directory. If the cache is created for some applications and other
instances do not use it, there may be power savings by connecting a
special array voltage (e.g., VCS) to ground. These functions may be
incorporated within the MBU 218 or located elsewhere within the
nest domain 220. The MBAs 212a and 212b within the memory domain
224 may also include logic to initiate autonomic memory operations
for the DRAM devices 204, such as refresh and periodic calibration
sequences in order to maintain proper data and signal
integrity.
FIG. 3 depicts a memory subsystem 112b as an instance of the memory
subsystem 112 of FIG. 1 in a buffered DIMM configuration 300 in
accordance with an embodiment. The buffered DIMM configuration 300
can include multiple buffered DIMMs 302 within the memory subsystem
112b, e.g., five or more instances of the buffered DIMM 302, where
a single buffered DIMM 302 is depicted in FIG. 3 for purposes of
explanation. The buffered DIMM 302 includes the memory buffer chip
202 of FIG. 2. As in the example of FIG. 2, the MCS 108 of the MCU
106 in the CP system 102 communicates synchronously on channel 110
via the interface 216. In the example of FIG. 3, the channel 110
interfaces to a connecter 304, e.g., a socket, that is coupled to a
connector 306 of the buffered DIMM 302. A signal path 308 between
the connector 306 and the interface 214 of the memory buffer chip
202 enables synchronous communication between the interfaces 214
and 216.
As in the example of FIG. 2, the memory buffer chip 202 as depicted
in FIG. 3 includes the nest domain 220 and the memory domain 224.
Similar to FIG. 2, the memory buffer chip 202 may include one or
more MISRs 226, such as one or more MISR 226a in MBA 212a and one
or more MISR 226b in MBA 212b. In the example of FIG. 3, the MBU
218 passes commands across the boundary layer 222 from the nest
domain 220 to the MBA 212a and/or to the MBA 212b in the memory
domain 224. The MBA 212a interfaces with DDR ports 210a and 210b,
and the MBA 212b interfaces with DDR ports 210c and 210d. Rather
than interfacing with DRAM devices 204 on one or more DIMMs 208 as
in the planar configuration 200 of FIG. 2, the DDR ports 210a-210d
can interface directly with the DRAM devices 204 on the buffered
DIMM 302.
The memory subsystem 112b may also include power management logic
310 that provides a voltage source for a voltage rail 312. The
voltage rail 312 is a local cache voltage rail to power a memory
buffer cache 314. The memory buffer cache 314 may be part of the
MBU 218. A power selector 316 can be used to determine whether the
voltage rail 312 is sourced by the power management logic 310 or
tied to ground 318. The voltage rail 312 may be tied to ground 318
when the memory buffer cache 314 is not used, thereby reducing
power consumption. When the memory buffer cache 314 is used, the
power selector 316 ties the voltage rail 312 to a voltage supply of
the power management logic 310. Fencing and clock gating can also
be used to better isolate voltage and clock domains.
As can be seen in reference to FIGS. 2 and 3, a number of memory
subsystem configurations can be supported in embodiments. Varying
sizes and configurations of the DRAM devices 204 can have different
address format requirements, as the number of ranks and the overall
details of slots, rows, columns, banks, bank groups, and/or ports
may vary across different DRAM devices 204 in embodiments. Various
stacking architectures (for example, 3 die stacking, or 3DS) may
also be implemented, which may include master ranks and slave ranks
in the packaging architecture. Each of these different
configurations of DRAM devices 204 may require a unique address
mapping table. Therefore, generic bits may be used by the MCU 106
to reference particular bits in a DRAM device 204 without having
full knowledge of the actual DRAM topology, thereby separating the
physical implementation of the DRAM devices 204 from the MCU 106.
The memory buffer chip 202 may map the generic bits to actual
locations in the particular type(s) of DRAM that is attached to the
memory buffer chip 202. The generic bits may be programmed to hold
any appropriate address field, including but not limited to memory
base address, rank (including master or slave), row, column, bank,
bank group, and/or port, depending on the particular computer
system.
FIG. 4 depicts a memory subsystem 112c as an instance of the memory
subsystem 112 of FIG. 1 with dual asynchronous and synchronous
memory operation modes in accordance with an embodiment. The memory
subsystem 112c can be implemented in the planar configuration 200
of FIG. 2 or in the buffered DIMM configuration 300 of FIG. 3. As
in the examples of FIGS. 2 and 3, the MCS 108 of the MCU 106 in the
CP system 102 communicates synchronously on channel 110 via the
interface 216. FIG. 4 depicts multiple instances of the interface
216 as interfaces 216a-216n which are configured to communicate
with multiple instances of the memory buffer chip 202a-202n. In an
embodiment, there are five memory buffer chips 202a-202n per CP
system 102.
As in the examples of FIGS. 2 and 3, the memory buffer chip 202a as
depicted in FIG. 4 includes the nest domain 220 and the memory
domain 224. Also similar to FIGS. 2 and 3, the memory buffer chip
202a may include one or more MISRs 226, such as one or more MISR
226a in MBA 212a and one or more MISR 226b in MBA 212b. In the
example of FIG. 4, the MBU 218 passes commands across the boundary
layer 222 from the nest domain 220 to the MBA 212a and/or to the
MBA 212b in the memory domain 224. The MBA 212a interfaces with DDR
ports 210a and 210b, and the MBA 212b interfaces with DDR ports
210c and 210d. The nest domain 220 and the memory domain 224 are
established and maintained using phase-locked loops (PLLs) 402,
404, and 406.
The PLL 402 is a memory controller PLL configured to provide a
master clock 408 to the MCS 108 and the interfaces 216a-216n in the
MCU 106 of the CP system 102. The PLL 404 is a nest domain PLL that
is coupled to the MBU 218 and the interface 214 of the memory
buffer chip 202a to provide a plurality of nest domain clocks 405.
The PLL 406 is a memory domain PLL coupled the MBAs 212a and 212b
and to the DDR ports 210a-210d to provide a plurality of memory
domain clocks 407. The PLL 402 is driven by a reference clock 410
to establish the master clock 408. The PLL 404 has a reference
clock 408 for synchronizing to the master clock 405 in the nest
domain 220. The PLL 406 can use a separate reference clock 414 or
an output 416 of the PLL 404 to provide a reference clock 418. The
separate reference clock 414 operates independent of the PLL
404.
A mode selector 420 determines the source of the reference clock
418 based on an operating mode 422 to enable the memory domain 224
to run either asynchronous or synchronous relative to the nest
domain 220. When the operating mode 422 is an asynchronous
operating mode, the reference clock 418 is based on the reference
clock 414 as a reference clock source such that the PLL 406 is
driven by separate reference clock and 414. When the operating mode
422 is a synchronous operating mode, the reference clock 418 is
based on the output 416 of an FSYNC block 492 which employs PLL 404
as a reference clock source for synchronous clock alignment. This
ensures that the PLLs 404 and 406 have related clock sources based
on the reference clock 408. Even though the PLLs 404 and 406 can be
synchronized in the synchronous operating mode, the PLLs 404 and
406 may be configured to operate at different frequencies relative
to each other. Additional frequency multiples and derivatives, such
as double rate, half rate, quarter rate, etc., can be generated
based on each of the multiplier and divider settings in each of the
PLLs 402, 404, and 406. For example, the nest domain clocks 405 can
include multiples of a first frequency of the PLL 404, while the
memory domain clocks 407 can include multiples of a second
frequency of the PLL 406.
In an asynchronous mode of operation each memory buffer chip
202a-202n is assigned to an independent channel 110. All data for
an individual cache line may be self-contained within the DRAM
devices 204 of FIGS. 2 and 3 attached to a common memory buffer
chip 202. This type of structure lends itself to lower-end cost
effective systems which can scale the number of channels 110 as
well as the DRAM speed and capacity as needs require. Additionally,
this structure may be suitable in higher-end systems that employ
features such as mirroring memory on dual channels 110 to provide
high availability in the event of a channel outage.
When implemented as a RAIM system, the memory buffer chips
202a-202n can be configured in the synchronous mode of operation.
In a RAIM configuration, memory data is striped across multiple
physical memory channels 110, e.g., five channels 110, which can
act as the single virtual channel 111 of FIG. 1 in order to provide
error-correcting code (ECC) protection for continuous operation,
even when an entire channel 110 fails. In a RAIM configuration, all
of the memory buffer chips 202a-202n of the same virtual channel
111 are operated synchronously since each memory buffer chip 202 is
responsible for a portion of a coherent line.
To support and maintain synchronous operation, the MCU 106 can
detect situations where one channel 110 becomes temporarily or
permanently incapacitated, thereby resulting in a situation wherein
the channel 110 is operating out of sync with respect to the other
channels 110. In many cases the underlying situation is
recoverable, such as intermittent transmission errors on one of the
interfaces 216a-216n and/or interface 214 of one of more of the
memory buffer chips 202a-202n. Communication on the channels 110
may utilize a robust cyclic redundancy code (CRC) on transmissions,
where a detected CRC error triggers a recovery retransmission
sequence. There are cases where the retransmission requires some
intervention or delay between the detection and retransmission. A
replay system including replay buffers for each of the channels can
be used to support a recovery retransmission sequence for a faulty
channel 110. Portions of the replay system may be suspended for a
programmable period of time to ensure that source data to be stored
in the replay buffer has been stored prior to initiating automated
recovery. The period of time while replay is suspended can also be
used to make adjustments to other subsystems, such as voltage
controls, clocks, tuning logic, power controls, and the like, which
may assist in preventing a recurrence of an error condition that
led to the fault. Suspending replay may also remove the need for
the MCU 106 to reissue a remaining portion of a store on the
failing channel 110 and may increase the potential of success upon
the replay.
Although the recovery retransmission sequence can eventually
restore a faulty channel 110 to fully operational status, the
overall memory subsystem 112 remains available during a recovery
period. Tolerating a temporary out of sync condition allows memory
operations to continue by using the remaining good (i.e.,
non-faulty) channels 110 until the recovery sequence is complete.
For instance, if data has already started to transfer back to the
cache subsystem 118 of FIG. 1, there may need to be a way to
process failing data after it has been transmitted. While returning
data with gaps is one option, another option is to delay the start
of data transmission until all error status is known. Delaying may
lead to reduced performance when there is a gapless requirement.
After recovering a faulty channel 110, the MCU 106 resynchronizes
the recovered channel 110 to the remaining good channels 110
thereby re-establishing a fully functional interface across all
channels 110 of the virtual channel 111 of FIG. 1.
To support timing alignment issues that may otherwise be handled
using deskewing logic, the MCU 106 and the memory buffer chip 202
may support the use of tags. Command completion and data
destination routing information can be stored in a tag directory
424 which is accessed using a received tag. Mechanisms for error
recovery, including retrying of read or write commands, may be
implemented in the memory buffer chips 202 for each individual
channel 110. Each command that is issued by the MCU 106 to the
memory buffer chips 202 may be assigned a command tag in the MCU
106, and the assigned command tag sent with the command to the
memory buffer chips 202 in the various channels 110. The various
channels 110 send back response tags that comprise data tags or
done tags. Data tags corresponding to the assigned command tag are
returned from the buffer chip in each channel to correlate read
data that is returned from the various channels 110 to an original
read command. Done tags corresponding to the assigned command tag
are also returned from the memory buffer chip 202 in each channel
110 to indicate read or write command completion.
The tag directory 424, also associated with tag tables which can
include a data tag table and a done tag table, may be maintained in
the MCU 106 to record and check the returned data and done tags. It
is determined based on the tag tables when all of the currently
functioning channels in communication with the MCU 106 return the
tags corresponding to a particular command. For data tags
corresponding to a read command, the read data is considered
available for delivery to the cache subsystem 118 of FIG. 1 when a
data tag corresponding to the read command is determined to have
been received from each of the currently functioning channels 110.
For done tags corresponding to a read or write command, the read or
write is indicated as complete from a memory control unit and
system perspective when a done tag corresponding to the read or
write command is determined to have been received from each of the
currently functioning channels 110. The tag checking mechanism in
the MCU 106 may account for a permanently failed channel 110 by
removing that channel 110 from a list of channels 110 to check in
the tag tables. No read or write commands need to be retained in
the MCU 106 for retrying commands, freeing up queuing resources
within the MCU 106.
Timing and signal adjustments to support high-speed synchronous
communications are also managed at the interface level for the
channels 110. FIG. 5 depicts an example of channel 110 and
interfaces 214 and 216 in greater detail in accordance with an
embodiment. As previously described in reference to FIG. 1, each
channel 110 includes a downstream bus 114 and an upstream bus 116.
The downstream bus 114 includes multiple downstream lanes 502,
where each lane 502 can be a differential serial signal path to
establish communication between a driver buffer 504 of interface
216 and a receiver buffer 506 of interface 214. Similarly, the
upstream bus 116 includes multiple upstream lanes 512, where each
lane 512 can be a differential serial signal path to establish
communication between a driver buffer 514 of interface 214 and a
receiver buffer 516 of interface 216. In an exemplary embodiment,
groups 508 of four bits are transmitted serially on each of the
active transmitting lanes 502 per frame, and groups 510 of four
bits are transmitted serially on each of the active transmitting
lanes 512 per frame; however, other group sizes can be supported.
The lanes 502 and 512 can be general data lanes, clock lanes, spare
lanes, or other lane types, where a general data lane may send
command, address, tag, frame control or data bits.
In interface 216, commands and/or data are stored in a transmit
first-in-first-out (FIFO) buffer 518 to transmit as frames 520. The
frames 520 are serialized by serializer 522 and transmitted by the
driver buffers 504 as groups 508 of serial data on the lanes 502 to
interface 214. In interface 214, serial data received at receiver
buffers 506 are deserialized by deserializer 524 and captured in a
receive FIFO buffer 526, where received frames 528 can be analyzed
and reconstructed. When sending data from interface 214 back to
interface 216, frames 530 to be transmitted are stored in a
transmit FIFO buffer 532 of the interface 214, serialized by
serializer 534, and transmitted by the driver buffers 514 as groups
510 of serial data on the lanes 512 to interface 216. In interface
216, serial data received at receiver buffers 516 are deserialized
by deserializer 536 and captured in a receive FIFO buffer 538,
where received frames 540 can be analyzed and reconstructed.
The interfaces 214 and 216 may each include respective instances of
training logic 544 and 546 to configure the interfaces 214 and 216.
The training logic 544 and 546 train both the downstream bus 114
and the upstream bus 116 to properly align a source synchronous
clock to transmissions on the lanes 502 and 512. The training logic
544 and 546 also establish a sufficient data eye to ensure
successful data capture. Further details are described in reference
to process 600 of FIG. 6.
FIG. 6 depicts a process 600 for providing synchronous operation in
a memory subsystem in accordance with an embodiment. In order to
accomplish high availability fully synchronous memory operation
across all multiple channels 110, an initialization and
synchronization process is employed across the channels 110. The
process 600 is described in reference to elements of FIGS. 1-5.
At block 602, the lanes 502 and 512 of each channel 110 are
initialized and calibrated. The training logic 544 and 546 can
perform impedance calibration on the driver buffers 504 and 514.
The training logic 544 and 546 may also perform static offset
calibration of the receiver buffers 506 and 516 and/or sampling
latches (not depicted) followed by a wire test to detect permanent
defects in the transmission media of channel 110. Wire testing may
be performed by sending a slow pattern that checks wire continuity
of both sides of the clock and data lane differential pairs for the
lanes 502 and 512. The wire testing may include driving a simple
repeating pattern to set a phase rotator sampling point,
synchronize the serializer 522 with the deserializer 524 and the
serializer 534 with the deserializer 536, and perform lane-based
deskewing. Data eye optimization may also be performed by sending a
more complex training pattern that also acts as a functional data
scrambling pattern.
Training logic 544 and 546 can use complex training patterns to
optimize various parameters such as a final receiver offset, a
final receiver gain, peaking amplitude, decision feedback
equalization, final phase rotator adjustment, final offset
calibration, scrambler and descrambler synchronization, and
load-to-unload delay adjustments for FIFOs 518, 526, 532, and
538.
Upon detecting any non-functional lanes in the lanes 502 and 512, a
dynamic sparing process is invoked to replace the
non-functional/broken lane with an available spare lane of the
corresponding downstream bus 114 or upstream bus 116. A final
adjustment may be made to read data FIFO unload pointers of the
receive FIFO buffers 526 and 538 to ensure sufficient timing
margin.
At block 604, a frame transmission protocol is established based on
a calculated frame round trip latency. Once a channel 110 is
capable of reliably transmitting frames in both directions, a
reference starting point is established for decoding frames. To
establish synchronization with a common reference between the nest
clock 405 and the master clock 408, a frame lock sequence is
performed by the training logic 546 and 544. The training logic 546
may initiate the frame lock sequence by sending a frame including a
fixed pattern, such as all ones, to the training logic 544 on the
downstream bus 114. The training logic 544 locks on to the fixed
pattern frame received on the downstream bus 114. The training
logic 544 then sends the fixed pattern frame to the training logic
546 on the upstream bus 116. The training logic 546 locks on to the
fixed pattern frame received on the upstream bus 116. The training
logic 546 and 544 continuously generate the frame beats. Upon
completion of the frame lock sequence, the detected frame start
reference point is used as an alignment marker for all subsequent
internal clock domains.
A positive acknowledgement frame protocol may be used where the
training logic 544 and 546 acknowledge receipt of every frame back
to the transmitting side. This can be accomplished through the use
of sequential transaction identifiers assigned to every transmitted
frame. In order for the sending side to accurately predict the
returning acknowledgment, another training sequence referred to as
frame round trip latency (FRTL) can be performed to account for the
propagation delay in the transmission medium of the channel
110.
In an exemplary embodiment, the training logic 546 issues a null
packet downstream and starts a downstream frame timer. The training
logic 544 responds with an upstream acknowledge frame and
simultaneously starts an upstream round-trip timer. The training
logic 546 sets a downstream round-trip latency value, when the
first upstream acknowledge frame is received from the training
logic 544. The training logic 546 sends a downstream acknowledge
frame on the downstream bus 114 in response to the upstream
acknowledge frame from the training logic 544. The training logic
544 sets an upstream round-trip delay value when the downstream
acknowledge frame is detected. The training logic 544 issues a
second upstream acknowledge frame to close the loop. At this time
the training logic 544 goes into a channel interlock state. The
training logic 544 starts to issue idle frames until a positive
acknowledgement is received for the first idle frame transmitted by
the training logic 544. The training logic 546 detects the second
upstream acknowledge frame and enters into a channel interlock
state. The training logic 546 starts to issue idle frames until a
positive acknowledgement is received for the first idle frame
transmitted by the training logic 546. Upon receipt of the positive
acknowledgement, the training logic 546 completes channel interlock
and normal traffic is allowed to flow through the channel 110.
At block 606, a common synchronization reference is established for
multiple memory buffer chips 202a-202n. In the case of a fully
synchronous multi-channel structure, a relative synchronization
point is established to ensure that operations initiated from the
CP system 102 are executed in the same manner on the memory buffer
chips 202a-202n, even when the memory buffer chips 202a-202n are
also generating their own autonomous refresh and calibration
operations. Synchronization can be accomplished by locking into a
fixed frequency ratio between the nest and memory domains 220 and
224 within each memory buffer chip 202. In exemplary embodiments,
the PLLs 404 and 406 from both the nest and memory domains 220 and
224 are interlocked such that they have a fixed repeating
relationship. This ensures both domains have a same-edge aligned
boundary (e.g., rising edge aligned) at repeated intervals, which
is also aligned to underlying clocks used for the high speed source
synchronous interface 214 as well as frame decode and execution
logic of the MBU 218. A common rising edge across all the
underlying clock domains is referred to as the alignment or
"golden" reference cycle.
Multi-channel operational synchronization is achieved by using the
alignment reference cycle to govern all execution and arbitration
decisions within the memory buffer chips 202a-202n. Since all of
the memory buffer chips 202a-202n in the same virtual channel 111
have the same relative alignment reference cycle, all of their
queues and arbiters (not depicted) remain logically in lock step.
This results in the same order of operations across all of the
channels 110. Even though the channels 110 can have inherent
physical skew, and each memory buffer chip 202 performs a given
operation at different absolute times with respect to the other
memory buffer chips 202, the common alignment reference cycle
provides an opportunity for channel operations to transit the
boundary layer 222 between the nest and memory domains 220 and 224
with guaranteed timing closure and equivalent arbitration among
internally generated refresh and calibration operations.
As previously described in reference to FIG. 4, each memory buffer
chip 202 includes two discrete PLLs, PLL 404 and PLL 406, for
driving the underlying clocks 405 and 407 of the nest and memory
domains 220 and 224. When operating in asynchronous mode, each PLL
404 and 406 has disparate reference clock inputs 408 and 414 with
no inherent phase relationship to one another. However, when
running in synchronous mode, the memory PLL 406 becomes a slave to
the nest PLL 404 with the mode selector 420 taking over the role of
providing a reference clock 418 to the memory PLL 406 such that
memory domain clocks 407 align to the common alignment reference
point. A common external reference clock, the master clock 408, may
be distributed to the nest PLLs 404 of all memory buffer chips
202a-202n in the same virtual channel 111. The PLL 404 can be
configured into an external feedback mode to ensure that all PLLs
404 align their output nest clocks 405 to a common memory
sub-system reference point. This common point is used by dedicated
sync logic to drive the appropriate reference clock 418 based on
PLL 404 output 416 into the memory domain PLL 406 and achieve a
lock onto the target alignment cycle (i.e., the "golden"
cycle).
FIG. 7 depicts a process 700 for establishing alignment between the
nest and memory domains 220 and 224 in a memory subsystem 112
accordance with an embodiment. The process 700 is described in
reference to elements of FIGS. 1-6. The process 700 establishes an
alignment or "golden" cycle first in the nest domain 220 followed
by the memory domain 224. All internal counters and timers of a
memory buffer chip 202 are aligned to the alignment cycle by
process 700.
At block 702, the nest domain clocks 405 are aligned with a frame
start signal from a previous frame lock of block 604. The nest
domain 220 can use multiple clock frequencies for the nest domain
clocks 405, for example, to save power. A frame start may be
defined using a higher speed clock, and as such, the possibility
exists that the frame start could fall in a later phase of a
slower-speed nest domain clock 405. This would create a situation
where frame decoding would not be performed on an alignment cycle.
In order to avoid this, the frame start signal may be delayed by
one or more cycles, if necessary, such that it always aligns with
the slower-speed nest domain clock 405, thereby edge aligning the
frame start with the nest domain clocks 405. Clock alignment for
the nest domain clocks 405 can be managed by the PLL 404 and/or
additional circuitry (not depicted). At block 704, the memory
domain clocks 407 are turned off and the memory domain PLL 406 is
placed into bypass mode.
At block 706, the MCS 108 issues a super synchronize ("SuperSync")
command using a normal frame protocol to all memory buffer chips
202a-202n. The MCS 108 may employ a modulo counter matching an
established frequency ratio such that it will only issue any type
of synchronization command at a fixed period. This establishes the
master reference point for the entire memory subsystem 112 from the
MCS 108 perspective. Even though the SuperSync command can arrive
at the memory buffer chips 202a-202n at different absolute times,
each memory buffer chip 202 can use a nest cycle upon which this
command is decoded as an internal alignment cycle. Since skew among
the memory buffer chips 202a-202n is fixed, the alignment cycle on
each of the memory buffer chips 202a-202n will have the same fixed
skew. This skew translates into a fixed operational skew under
error free conditions.
At block 708, sync logic of the memory buffer chip 202, which may
be part of the mode selector 420, uses the SuperSync decode as a
reference to trigger realignment of the reference clock 418 that
drives the memory domain PLL 406. The SuperSync decode is
translated into a one cycle pulse signal 494, synchronous with the
nest domain clock 405 that resets to zero a modulo counter 496 in
the FSYNC block 492. The period of this counter 496 within the
FSYNC block 492 is set to be the least common multiple of all
memory and nest clock frequencies with the rising edge marking the
sync-point corresponding to the reference point previously
established by the MCS 108. The rising edge of FSYNC clock 416
becomes the reference clock of PLL 406 to create the memory domain
clocks. By bringing the lower-frequency output of PLL 406 back into
the external feedback port, the nest clock 405 and memory clock 407
all have a common clock edge aligned to the master reference point.
Thus, the FSYNC block 492 provides synchronous clock alignment
logic.
At block 710, the memory domain PLL 406 is taken out of bypass mode
in order to lock into the new reference clock 418 based on the
output 416 of the PLL 404 rather than reference clock 414. At block
712, the memory domain clocks 407 are turned back on. The memory
domain clocks 407 are now edge aligned to the same alignment
reference cycle as the nest domain clocks 405.
At block 714, a regular subsequent sync command is sent by the MCS
108 on the alignment cycle. This sync command may be used to reset
the various counters, timers and MISRs 226 that govern internal
memory operation command generation, execution and arbitration. By
performing a reset on the alignment cycle, all of the memory buffer
chips 202a-202n start their respective internal timers and counters
with the same logical reference point. If an arbiter on one memory
buffer chip 202 identifies a request from both a processor
initiated memory operation and an internally initiated command on a
particular alignment cycle, the corresponding arbiter on the
remaining memory buffer chips 202 will also see the same requests
on the same relative alignment cycle. Thus, all memory buffer chips
202a-202n will make the same arbitration decisions and maintain the
same order of operations.
Embodiments may provide internally generated commands at memory
buffer chip 202 to include DRAM refresh commands, DDR calibration
operations, dynamic power management, error recovery, memory
diagnostics, and the like. Anytime one of these operations is
needed, it must cross into the nest domain 220 and go through the
same arbitration as synchronous operations initiated by the MCS
108. Arbitration is performed on the golden cycle to ensure all the
memory buffer chips 202 observe the same arbitration queues and
generate the same result. The result is dispatched across boundary
layer 222 on the golden cycle which ensures timing and process
variations in each memory buffer chip 202 is nullified.
Under normal error free conditions, the order of operations will be
maintained across all of the memory buffer chips 202a-202n.
However, there are situations where one channel 110 can get out of
sync with the other channels 110. One such occurrence is the
presence of intermittent transmission errors on one or more of the
interfaces 214 and 216. Exemplary embodiments include a hardware
based recovery mechanism where all frames transmitted on a channel
110 are kept in a replay buffer for a prescribed period of time.
This time covers a window long enough to guarantee that the frame
has arrived at the receiving side, has been checked for errors, and
a positive acknowledgement indicating error free transmission has
been returned to the sender. Once this is confirmed, the frame is
retired from the replay buffer. However, in the case of an
erroneous transmission, the frame is automatically retransmitted,
or replayed, along with a number of subsequent frames in case the
error was a one-time event. In many cases, the replay is sufficient
and normal operation can resume. In certain cases, the transmission
medium of the channel 110 has become corrupted to the point that a
dynamic repair is instituted to replace a defective lane with a
spare lane from lanes 502 or 512. Upon completion of the repair
procedure, the replay of the original frames is sent and again
normal operation can resume.
Another less common occurrence can be an on-chip disturbance
manifesting as a latch upset which results in an internal error
within the memory buffer chip 202. This can lead to a situation
where one memory buffer chip 202 executes its operations
differently from the remaining memory buffer chips 202. Although
the memory system 100 continues to operate correctly, there can be
significant performance degradation if the channels 110 do not
operate in step with each other. In exemplary embodiments, the
MISRs 226 monitor for and detect such a situation. The MISRs 226
receive inputs derived from key timers and counters that govern the
synchronous operation of the memory buffer chip 202, such as
refresh starts, DDR calibration timers, power throttling, and the
like. The inputs to the MISRs 226 are received as a combination of
bits that collectively form a signature. One or more of the bits of
the MISRs 226 are continually transmitted as part of an upstream
frame payload to the MCU 106, which monitors the bits received from
the MISRs 226 of the memory buffer chips 202a-202n. The presence of
physical skew between the channels 110 results in the bits from the
MISRs 226 arriving at different absolute times across the channels
110. Therefore, a learning process is incorporated to calibrate
checking of the MISRs 226 to the wire delays in the channels
110.
In exemplary embodiments, MISR detection in the MCU 106
incorporates two distinct aspects in order to monitor the
synchronicity of the channels 110. First, the MCU 106 monitors the
MISR bits received on the upstream bus 116 from each of the memory
buffer chips 202a-202n and any difference seen in the MISR bit
stream indicates an out-of-sync condition. Although this does not
pose any risk of a data integrity issue, it can negatively impact
performance, as the MCU 106 may incur additional latency waiting
for an entire cache line access to complete across the channels
110. Another aspect is monitoring transaction sequence identifiers
(i.e., tags) associated with each memory operation and comparing
associated "fetch" tags or "done" tags as the operations complete.
Once again, skew of the channels 110 is taken into account in order
to perform an accurate comparison. In one example, this skew can
manifest in as many as 30 cycles of difference between the fastest
and slowest channel 110. If the tags are 7-bits wide, with five
channels 110, and a maximum 30-cycle difference across channels
110, this would typically require 5.times.7.times.30=1050 latches
to perform a simplistic compare. There may be some cases that
equate to about 40 bit-times which is about 4 cycles of deskew
after aligning to a frame. To further reduce the number of latches,
a MISR can be incorporated within the MCU 106 to encode the tag
into a bit stream, which is then pipelined to eliminate the skew.
By comparing the output of the MISR of the MCU 106 across all of
the channels 110, a detected difference indicates an out-of-order
processing condition.
In either of these situations, the afflicted channel 110 can at
least temporarily operate out of sync or out of order with respect
to the other channels 110. Continuous availability of the memory
subsystem 112 may be provided through various recovery and
self-healing mechanisms. Data tags can be used such that in the
event of an out-of-order or out-of-sync condition, the MCU 106
continues to function. Each read command may include an associated
data tag that allows the MCS 108 to handle data transfers received
from different channels 110 at different times or even in different
order. This allows proper functioning even in situations when the
channels 110 drift out of sync.
For out-of-sync conditions, a group of hierarchical MISRs 226 can
be used accumulate a signature for any sync-related event. Examples
of sync-related events include a memory refresh start, a periodic
driver (ZQ) calibration start, periodic memory calibration start,
power management window start, and other events that run off a
synchronized counter. One or more bits from calibration timers,
refresh timers, and the like can serve as inputs to the MISRs 226
to provide a time varying signature which may assist in verifying
cross-channel synchronization at the MCU 106. Hierarchical MISRs
226 can be inserted wherever there is a need for speed matching of
data. For example, speed matching may be needed between MBA 212a
and the MBU 218, between the MBA 212b and the MBU 218, between the
MBU 218 and the upstream bus 116, and between the interfaces
216a-216n and the MCS 108.
For out-of-order conditions, staging each of the tags received in
frames from each channel 110 can be used to deskew the wire delays
and compare them. A MISR per channel 110 can be used to create a
signature bit stream from the tags received at the MCU 106 and
perform tag/signature-based deskewing rather than hardware
latch-based deskewing. Based on the previous example of 7-bit wide
tags, with five channels 110, and a maximum 30-cycle difference
across channels 110, the use of MISRs reduces the 1050 latches to
about 7.times.5+30.times.5=185 latches, plus the additional support
latches.
To minimize performance impacts, the MCS 108 tries to keep all
channels 110 in lockstep, which implies that all commands are
executed in the same order. When read commands are executed, an
associated data tag is used to determine which data correspond to
which command. This approach also allows the commands to be
reordered based on resource availability or timing dependencies and
to get better performance. Commands may be reordered while keeping
all channels 110 in lockstep such that the reordering is the same
across different channels 110. In this case, tags can be used to
match the data to the requester of the data from memory regardless
of the fact that the command order changed while the data request
was processed.
Marking a channel 110 in error may be performed when transfers have
already started and to wait for recovery for cases where transfers
have not yet occurred. Data blocks from the memory subsystem 112
can be delivered to the cache subsystem interface 122 of FIG. 1 as
soon as data is available without waiting for complete data error
detection. This design implementation is based on the assumption
that channel errors are rare. Data can be sent across clock domains
from the MCS 108 to the cache subsystem interface 122
asynchronously as soon as it is available from all channels 110 but
before data error detection is complete for all frames. If a data
error is detected after the data block transfer has begun, an
indication is sent from the MCS 108 to the cache subsystem
interface 122, for instance, on a separate asynchronous interface,
to intercept the data block transfer in progress and complete the
transfer using redundant channel information. Timing requirements
are enforced to ensure that the interception occurs in time to
prevent propagation of corrupt data to the cache subsystem 118 of
FIG. 1. A programmable count-down counter may be employed to
enforce the timing requirements.
If the data error is detected before the block data transfer has
begun to the cache subsystem 118, the transfer is stalled until all
frames have been checked for any data errors. Assuming errors are
infrequent, the performance impact is minimal. This reduces the use
of channel redundancy and may result in avoidance of possible
uncorrectable errors in the presence of previously existing errors
in the DRAM devices 204.
The CP system 102 and/or the memory subsystem 112 may also include
configurable delay functions on a per-command type or destination
basis to delay data block transfer to upstream elements, such as
caches, until data error detection is completed for the block.
Command or destination information is available for making such
selections as inputs to the tag directory. This can selectively
increase system reliability and simplify error handling, while
minimizing performance impacts.
To support other synchronization issues, the MCU 106 can
re-establish synchronization across multiple channels 110 in the
event of a channel failure without having control of an underlying
recovery mechanism used on the failed channel. A programmable
quiesce sequence incrementally attempts to restore channel
synchronization by stopping stores and other downstream commands
over a programmable time interval. The quiesce sequence may wait
for completion indications from the memory buffer chips 202a-202n
and inject synchronization commands across all channels 110 to
reset underlying counters, timers, MISRs 226, and other
time-sensitive circuitry to the alignment reference cycle. If a
failed channel 110 remains out of synchronization, the quiesce
sequence can be retried under programmatic control. In many
circumstances, the underlying root cause of the disturbance can be
self healed, thereby resulting in the previously failed channel 110
being reactivated and resynchronized with the remaining channels
110. Under extreme error conditions the quiesce and recovery
sequence fails to restore the failed channel 110, and the failed
channel 110 is permanently taken off line. In a RAIM architecture
that includes five channels 110, the failure of one channel 110
permits the remaining four channels 110 to operate with a reduced
level of protection.
FIG. 8 depicts an example timing diagram 800 of synchronizing a
memory subsystem in accordance with an embodiment. The timing
diagram 800 includes timing for a number of signals of the memory
buffer chip 202. In the example of FIG. 8, two of the nest domain
clocks 405 of FIG. 4 are depicted as a higher-speed nest domain
clock frequency 802 and a lower-speed nest domain clock frequency
804. Two of the memory domain clocks 407 of FIG. 4 are depicted in
FIG. 8 as a higher-speed memory domain clock frequency 806 and a
lower-speed memory domain clock frequency 808. The timing diagram
800 also depicts example timing for a nest domain pipeline 810, a
boundary layer 812, a reference counter 814, a memory queue 816,
and a DDR interface 818 of a DDR port 210. In an embodiment, the
higher-speed nest domain clock frequency 802 is about 2.4 GHz, the
lower-speed nest domain clock frequency 804 is about 1.2 GHz, the
higher-speed memory domain clock frequency 806 is about 1.6, GHz
and the lower-speed memory domain clock frequency 808 is about 0.8
GHz.
A repeating pattern of clock cycles is depicted in FIG. 8 as a
sequence of cycles "B", "C", "A" for the lower-speed nest domain
clock frequency 804. Cycle A represents an alignment cycle, where
other clocks and timers in the memory buffer chip 202 are reset to
align with a rising edge of the alignment cycle A. Upon receiving a
SuperSync command, the higher and lower-speed memory domain clock
frequencies 806 and 808 stop and restart based on a sync point that
results in alignment after a clock sync window 820. Once alignment
is achieved, the alignment cycle A, also referred to as a "golden"
cycle, serves as a common logical reference for all memory buffer
chips 202a-202n in the same virtual channel 111. Commands and data
only cross the boundary layer 222 on the alignment cycle. A regular
sync command can be used to reset counters and timers within each
of the memory buffer chips 202a-202n such that all counting is
referenced to the alignment cycle.
In FIG. 8 at clock edge 822, the higher and lower-speed nest domain
clock frequencies 802 and 804, the higher and lower-speed memory
domain clock frequencies 806 and 808, and the nest domain pipeline
810 are all aligned. A sync command in the nest domain pipeline 810
is passed to the boundary layer 812 at clock edge 824 of the
higher-speed memory domain clock frequency 806. At clock edge 826
of cycle B, a read command is received in the nest domain pipeline
810. At clock edge 828 of the higher-speed memory domain clock
frequency 806, the read command is passed to the boundary layer
812, the reference counter 814 starts counting a zero, and the sync
command is passed to the memory queue 816. At clock edge 830 of the
higher-speed memory domain clock frequency 806, the reference
counter 814 increments to one, the read command is passed to the
memory queue 816 and the DDR interface 818. At clock edge 832 of
the higher-speed memory domain clock frequency 806 which aligns
with an alignment cycle A, the reference counter 814 increments to
two, and a refresh command is queued in the memory queue 816.
Alignment is achieved between clocks and signals of the nest domain
220 and the memory domain 224 for sending commands and data across
the boundary layer 222 of FIG. 2.
FIG. 9 illustrates an embodiment of a MCU 900 that implements
FIFO-queue based command spreading. MCU 900 includes command queues
901, which receive command requests from a processor (such as
control processor 102 of FIG. 1) that is in communication with MCU
900 (which may comprise MCU 106 of FIG. 1). The requested commands
from the command queues 901 are each assigned by first level
priority stage 902, via command buses 910, for storage in a
particular FIFO queue of FIFO queues 903A-B and 904A-B based on the
port and command type (i.e., fetch or store) of each command.
Addresses associated with the commands are stored in address
registers 903C-D and 904C-D, each of which is associated with a
respective FIFO queue of FIFO queues 903A-B and 904A-B, from the
command queues 901 via address buses 911. The second level priority
stage 905 selects commands from the various FIFO queues to send to
buffer chip 906 via bandwidth (BW) limited command interface 908.
On buffer chip 906, the commands are then stored in queues 907A-B
and 908A-B, corresponding to fetch and store queues for port 1 and
port 0 on the buffer chip 906, until they are executed. Each of
queues 907A-B and 908A-B are associated with one or more DIMMs that
are in communication with the buffer chip 906. Data and command
done indications are returned from the buffer chip 906 to the MCU
900 via BW limited data interface 909. If the BW limited command
interface 908 goes into recovery, the buffer chip 906 may continue
issuing commands to the DIMMs from queues 907A-B and 908A-B.
The first level priority stage 902 may operate in the processor
clock domain (i.e., based on master clock 408 of FIG. 4). The first
level priority stage 902 may accept command requests from the
multiple command queues 901 based on, for example, a least recently
used (LRU) basis. The commands for the various buffer chips (e.g.,
for DRAM port 0 to DRAM port 1, which may be distributed across any
appropriate number of buffer chips) are loaded into FIFO queues
such as FIFO queues 903A-B and 904A-B based on command port, and,
in some embodiments, command type. Command valid bits for DRAM PORT
0 to DRAM PORT 1 are also stored with the commands in the FIFO
queues 903A-B and 904A-B, and are visible to the second level
priority stage 905. There may be separate selection mechanisms in
first level priority stage 902 for fetch commands and store
commands in some embodiments. The fetch and store commands may be
loaded in separate FIFO queues (e.g., FIFO queues 903A and 904A for
fetch commands, and FIFO queues 903B and 904B for store commands)
and presented to the second level priority stage 905 in parallel.
Once a command is stored in a FIFO queue of FIFO queues 903A-B and
904A-B, the command queue of command queues 901 from which the
command was issued is released, and may then be reloaded with a new
command request. The addresses associated with the various commands
are loaded in address registers 903C-D and 904C-D, which are
associated with the FIFO queues 903A-B and 904A-B, and are used to
match data (sent in response to fetch commands) and command done
indications that are sent from buffer chip 906 to
previously-forwarded commands via BW limited data interface 909.
This reduces the need for a relatively large number of command
queues 901 to hold addresses until the commands are completed in
the MCU 900.
The second level priority stage 905 may operate in the memory clock
domain (i.e., based on memory domain clocks 407 of FIG. 4). The
second level priority stage 905 selects among commands targeting
different buffer chip ports, and forwards the selected commands to
buffer chip 906 via BW limited command interface 908. The second
level priority stage 905 may also select among FIFO queues that are
designated for fetch commands or store commands, allowing good
utilization of the interfaces 908/909 between the MCU 900 and the
buffer chip 906. The second level priority stage 905 may assign
command tags to the commands that are forwarded to the buffer chip
906; these command tags may be used to keep track of commands that
are completed based on command done indications that are sent from
the buffer chip 906 to the MCU 900 via BW limited data interface
909. The second level priority stage 905 may also track how many
fetch and stores are currently being processed by the buffer chip
906 and select commands to send to the buffer chip based on this
tracking, allowing good utilization of fetch and store command
reorder queues that are located on the buffer chip 906.
In some embodiments, there may be a FIFO ordering of commands in
the queues 907A-B and 908A-B on the buffer chip 906. In other
embodiments, there is a third level priority stage within each
queue 907A-B and 908A-B on the buffer chip 906 that acts to
optimize the BW limited data interface 909. For example, if port 1
fetch FIFO queue 907A is holding multiple fetch entries and one
DRAM bank is occupied with a `bank busy` conflict, a fetch entry
for another DRAM bank may be selected from the queue 907A for
processing. In further embodiments in which there is only a single
queue per port, the third level of priority may select between
fetch and store operations within a queue for a particular port.
While inclusion of a third level priority stage within each queue
907A-B and 908A-B on the buffer chip 906 may optimize the usage of
BW limited data interface 909 in the upstream direction from the
buffer chip 906 to the MCU 900, without optimizing the spreading of
the queue entries across all the appropriate queues on the BW
limited command interface 908 by second level priority stage 905,
the usage of the BW limited command interface 908 may not be fully
optimized.
FIG. 9 is shown for illustrative purposes only. A memory controller
such as memory controller 900 may be in communication with any
appropriate number of buffer chips such as buffer chip 906, and the
buffer chips may have any appropriate number of ports. The number
of FIFO queues in the memory controller is based on the number of
ports on the various buffer chips that are in communication with
the memory controller, and, in some embodiments, there may be
separate FIFO queues for fetch commands and store commands. Each of
the buffer chips in communication with a memory controller such as
memory controller 900 may have a respective second level priority
stage in the memory controller that arbitrates between the buffer
chip and the FIFO queues in the memory controller that are assigned
to the ports on that buffer chip. Command queues 901 may include
any appropriate number of command queues.
FIG. 10 illustrates an embodiment of a method for FIFO-queue based
command spreading. FIG. 10 is discussed with respect to FIG. 9.
First, in block 1001, commands are received from the command queue
901 by first level priority stage 902. The first level priority
state 902 may select among command queues 901 on a LRU basis in
some embodiments. Then, in block 1002, the first level priority
stage assigns the received commands for storage in the various FIFO
queues 903A-B and 904A-B based on port and, in some embodiments,
command type (i.e., fetch or store). After a command from a
particular command queue has been stored in a FIFO queue of FIFO
queues 903A-B and 904A-B, that command queue is released, and may
then be reloaded with a new command request. Then, in block 1003,
the second level priority stage 905 selects among the FIFO queues
903A-B and 904A-B to forward commands from the FIFO queues 903A-B
and 904A-B to the buffer chip 906 via BW limited command interface
908. In some embodiments, the second level priority stage 905 may
comprise a toggle mechanism that sequentially steps through the
FIFO queues 903A-B and 904A-B that are assigned to the buffer chip
906. In further embodiments, the toggle mechanism may be used in
conjunction with tracking, by the second level priority stage 905,
the commands that have been sent to particular ports in the buffer
chip 906, and selecting the next FIFO queue of FIFO queues 903A-B
and 904A-B from which to send a command to the buffer chip based on
the tracked commands. The number of fetch commands and store
commands currently being handled by the buffer chip 906 may also be
tracked by second level priority stage 905 in order to select the
next FIFO queue of FIFO queues 903A-B and 904A-B. Then, in block
1004, the second level priority stage 905 assigns command tags to
each command that is forwarded to buffer chip 906, and in block
1005, these command tags are used to determine that commands have
been completed based on done indications sent from buffer chip 906
to MCU 900 via BW limited data interface 909. Done indications and
data may then be sent by MCU 900 to a cache (such as cache
subsystem 118 of FIG. 1) based on the addresses associated with the
commands that are stored in the address registers 903C-D and 904C-D
associated with FIFO queues 903A-B and 904A-B.
As will be appreciated by one skilled in the art, one or more
aspects of the present invention may be embodied as a system,
method or computer program product. Accordingly, one or more
aspects of the present invention may take the form of an entirely
hardware embodiment, an entirely software embodiment (including
firmware, resident software, micro-code, etc.) or an embodiment
combining software and hardware aspects that may all generally be
referred to herein as a "circuit," "module" or "system".
Furthermore, one or more aspects of the present invention may take
the form of a computer program product embodied in one or more
computer readable medium(s) having computer readable program code
embodied thereon.
Any combination of one or more computer readable medium(s) may be
utilized. The computer readable medium may be a computer readable
storage medium. A computer readable storage medium may be, for
example, but not limited to, an electronic, magnetic, optical,
electromagnetic, infrared or semiconductor system, apparatus, or
device, or any suitable combination of the foregoing. More specific
examples (a non-exhaustive list) of the computer readable storage
medium include the following: an electrical connection having one
or more wires, a portable computer diskette, a hard disk, a random
access memory (RAM), a read-only memory (ROM), an erasable
programmable read-only memory (EPROM or Flash memory), an optical
fiber, a portable compact disc read-only memory (CD-ROM), an
optical storage device, a magnetic storage device, or any suitable
combination of the foregoing. In the context of this document, a
computer readable storage medium may be any tangible medium that
can contain or store a program for use by or in connection with an
instruction execution system, apparatus, or device.
Referring now to FIG. 11, in one example, a computer program
product 1100 includes, for instance, one or more storage media
1102, wherein the media may be tangible and/or non-transitory, to
store computer readable program code means or logic 1104 thereon to
provide and facilitate one or more aspects of embodiments described
herein.
Program code, when created and stored on a tangible medium
(including but not limited to electronic memory modules (RAM),
flash memory, Compact Discs (CDs), DVDs, Magnetic Tape and the like
is often referred to as a "computer program product". The computer
program product medium is typically readable by a processing
circuit preferably in a computer system for execution by the
processing circuit. Such program code may be created using a
compiler or assembler for example, to assemble instructions, that,
when executed perform aspects of the invention.
Technical effects and benefits include not needing to store read or
write commands in the MCU for retrying commands, freeing up queuing
resources within the MCU.
The terminology used herein is for the purpose of describing
particular embodiments only and is not intended to be limiting of
embodiments. As used herein, the singular forms "a", "an" and "the"
are intended to include the plural forms as well, unless the
context clearly indicates otherwise. It will be further understood
that the terms "comprises" and/or "comprising," when used in this
specification, specify the presence of stated features, integers,
steps, operations, elements, and/or components, but do not preclude
the presence or addition of one or more other features, integers,
steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of
all means or step plus function elements in the claims below are
intended to include any structure, material, or act for performing
the function in combination with other claimed elements as
specifically claimed. The description of embodiments have been
presented for purposes of illustration and description, but is not
intended to be exhaustive or limited to the embodiments in the form
disclosed. Many modifications and variations will be apparent to
those of ordinary skill in the art without departing from the scope
and spirit of the embodiments. The embodiments were chosen and
described in order to best explain the principles and the practical
application, and to enable others of ordinary skill in the art to
understand the embodiments with various modifications as are suited
to the particular use contemplated.
Computer program code for carrying out operations for aspects of
the embodiments may be written in any combination of one or more
programming languages, including an object oriented programming
language such as Java, Smalltalk, C++ or the like and conventional
procedural programming languages, such as the "C" programming
language or similar programming languages. The program code may
execute entirely on the user's computer, partly on the user's
computer, as a stand-alone software package, partly on the user's
computer and partly on a remote computer or entirely on the remote
computer or server. In the latter scenario, the remote computer may
be connected to the user's computer through any type of network,
including a local area network (LAN) or a wide area network (WAN),
or the connection may be made to an external computer (for example,
through the Internet using an Internet Service Provider).
Aspects of embodiments are described above with reference to
flowchart illustrations and/or schematic diagrams of methods,
apparatus (systems) and computer program products according to
embodiments. It will be understood that each block of the flowchart
illustrations and/or block diagrams, and combinations of blocks in
the flowchart illustrations and/or block diagrams, can be
implemented by computer program instructions. These computer
program instructions may be provided to a processor of a general
purpose computer, special purpose computer, or other programmable
data processing apparatus to produce a machine, such that the
instructions, which execute via the processor of the computer or
other programmable data processing apparatus, create means for
implementing the functions/acts specified in the flowchart and/or
block diagram block or blocks.
These computer program instructions may also be stored in a
computer readable medium that can direct a computer, other
programmable data processing apparatus, or other devices to
function in a particular manner, such that the instructions stored
in the computer readable medium produce an article of manufacture
including instructions which implement the function/act specified
in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a
computer, other programmable data processing apparatus, or other
devices to cause a series of operational steps to be performed on
the computer, other programmable apparatus or other devices to
produce a computer implemented process such that the instructions
which execute on the computer or other programmable apparatus
provide processes for implementing the functions/acts specified in
the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the
architecture, functionality, and operation of possible
implementations of systems, methods, and computer program products
according to various embodiments. In this regard, each block in the
flowchart or block diagrams may represent a module, segment, or
portion of code, which comprises one or more executable
instructions for implementing the specified logical function(s). It
should also be noted that, in some alternative implementations, the
functions noted in the block may occur out of the order noted in
the figures. For example, two blocks shown in succession may, in
fact, be executed substantially concurrently, or the blocks may
sometimes be executed in the reverse order, depending upon the
functionality involved. It will also be noted that each block of
the block diagrams and/or flowchart illustration, and combinations
of blocks in the block diagrams and/or flowchart illustration, can
be implemented by special purpose hardware-based systems that
perform the specified functions or acts, or combinations of special
purpose hardware and computer instructions.
* * * * *