U.S. patent application number 16/683095 was filed with the patent office on 2021-05-13 for storage class memory access.
The applicant listed for this patent is Western Digital Technologies, Inc.. Invention is credited to Cyril Guyot, Adam Manzanares.
Application Number | 20210141736 16/683095 |
Document ID | / |
Family ID | 1000004480149 |
Filed Date | 2021-05-13 |
![](/patent/app/20210141736/US20210141736A1-20210513\US20210141736A1-2021051)
United States Patent
Application |
20210141736 |
Kind Code |
A1 |
Manzanares; Adam ; et
al. |
May 13, 2021 |
STORAGE CLASS MEMORY ACCESS
Abstract
A write request is received to write byte-addressable data
corresponding to a first entry of a plurality of entries in a page
table, and the byte-addressable data is written in a buffer of a
host memory. A read request is received to read byte-addressable
data corresponding to a second entry of the plurality of entries in
the page table, and a read command is sent to a device using a
memory device interface to read the byte-addressable data from a
Storage Class Memory (SCM) of the device. According to another
aspect, control circuitry of the device uses a block device
interface for receiving commands from a host to read and write data
in blocks in the SCM. The control circuitry also uses a memory
device interface for receiving read commands from the host to read
byte-addressable data from the SCM.
Inventors: |
Manzanares; Adam; (Union
City, CA) ; Guyot; Cyril; (San Jose, CA) |
|
Applicant: |
Name |
City |
State |
Country |
Type |
Western Digital Technologies, Inc. |
San Jose |
CA |
US |
|
|
Family ID: |
1000004480149 |
Appl. No.: |
16/683095 |
Filed: |
November 13, 2019 |
Current U.S.
Class: |
1/1 |
Current CPC
Class: |
G06F 3/0611 20130101;
G06F 3/0659 20130101; G06F 9/3013 20130101; G06F 3/0656 20130101;
G06F 12/1009 20130101; G06F 12/0882 20130101; G06F 9/30047
20130101; G06F 3/0673 20130101; G06F 3/064 20130101 |
International
Class: |
G06F 12/1009 20060101
G06F012/1009; G06F 12/0882 20060101 G06F012/0882; G06F 9/30
20060101 G06F009/30; G06F 3/06 20060101 G06F003/06 |
Claims
1. A method for interfacing with a device including a Storage Class
Memory (SCM), the method comprising: creating a page table
including a plurality of entries corresponding to memory locations
in the SCM of the device; setting the plurality of entries as
read-only in the page table; receiving a write request to write
byte-addressable data corresponding to a first entry of the
plurality of entries; writing the byte-addressable data for the
write request in a buffer of a memory of a host; receiving a read
request to read byte-addressable data corresponding to a second
entry of the plurality of entries; and sending a read command to
the device to read the byte-addressable data for the read request
using a memory device interface.
2. The method of claim 1, further comprising changing the first
entry to indicate read and write access and to point to a location
in the local memory of the host where the byte-addressable data for
the write request was written.
3. The method of claim 1, further comprising: reading a block of
data for the first entry from the SCM; storing the read block of
data in the buffer of the local memory; modifying the block of data
in the buffer to perform the write request; and sending a write
command to the device using a block device interface to flush the
modified block of data from the buffer of the memory of the host to
the SCM.
4. The method of claim 3, further comprising: indicating that data
for the first entry in the page table is unavailable for writing
data in preparation for the flushing of the modified block of data
to the SCM; setting the first entry as read-only in the page table
in response to the completion of the flushing of the modified block
of data to the SCM; and updating the first entry in the page table
to point to a location in the SCM where the modified block of data
was flushed.
5. The method of claim 1, further comprising redirecting the write
request from the memory device interface to a block device
interface for performance of the write request.
6. The method of claim 1, wherein the memory device interface
accesses a Base Address Register (BAR) of the SCM to read data from
the SCM.
7. The method of claim 1, wherein the write request is performed in
the buffer of the memory using an Operating System (OS) and the
read command is sent to the device using the memory device
interface without using the OS.
8. The method of claim 1, further comprising communicating with the
device using Peripheral Component Interconnect express (PCIe) via
the memory device interface and a block device interface.
9. A device, comprising: a Storage Class Memory (SCM) for storing
data; and control circuitry configured to: use a block device
interface for receiving write commands from a host to write data in
blocks to the SCM and for receiving read commands from the host to
read data in blocks from the SCM; and use a memory device interface
for receiving read commands from the host to read byte-addressable
data from the SCM.
10. The device of claim 9, wherein the control circuitry is further
configured to expose a Base Address Register (BAR) of the device to
the host via the memory device interface.
11. The device of claim 9, wherein the control circuitry is further
configured to: receive a write command from the host via the block
device interface to write a block of data in the SCM; and update
addresses used by the memory device interface for byte-addressable
data included in the block of data written in the SCM.
12. The device of claim 9, wherein the control circuitry is further
configured to block performance of a write command received via the
memory device interface.
13. The device of claim 9, wherein the control circuitry is further
configured to communicate with the host using Peripheral Component
Interconnect express (PCIe) via the memory device interface and the
block device interface.
14. A host, comprising: a memory for storing data; and processor
circuitry configured to: receive a write request to write data
corresponding to a first entry of a plurality of entries in a page
table; write data for the write request in a buffer of the local
memory; receive a read request to read data corresponding to a
second entry of the plurality of entries in the page table; and
send a read command to a device using a memory device interface to
read the data for the read request from a Storage Class Memory
(SCM) of the device.
15. The host of claim 14, wherein the processor circuitry is
further configured to change the first entry in the page table to
indicate read and write access and to point to a location in the
memory where data for the write request was written.
16. The host of claim 14, wherein the processor circuitry is
further configured to: read a block of data for the first entry
from the SCM; store the read block of data in the buffer of the
memory; modify the block of data in the buffer to perform the write
request; and send a write command to the device using a block
device interface to flush the modified block of data from the
buffer of the memory to the SCM.
17. The host of claim 14, wherein the processor circuitry is
further configured to: indicate that the first entry in the page
table is unavailable for writing data in response to preparing to
flush the modified block of data to the SCM; set the first entry as
read-only in the page table in response to the completion of the
flushing of the modified block of data to the SCM; and update the
first entry in the page table to point to a location in the SCM
where the modified block of data was flushed.
18. The host of claim 14, wherein the processor circuitry is
further configured to redirect the write request from the memory
device interface to a block device interface for performance of the
write request.
19. The host of claim 14, wherein the memory device interface
accesses a Base Address Register (BAR) of the SCM to read data from
the SCM.
20. The host of claim 14, wherein the processor circuitry is
further configured to: perform the write request in the buffer of
the local memory using an Operating System (OS) of the host; and
send the read command to the device using the memory device
interface without using the OS.
Description
BACKGROUND
[0001] Storage Class Memory (SCM) has recently been developed as a
non-volatile storage option that is capable of providing a fine
granularity of data access (i.e., byte-addressable or cache line
size). In addition, SCMs typically provide a shorter data access
latency, as compared to traditional non-volatile storage devices,
such as a Solid-State Drive (SSD) using flash memory or a Hard Disk
Drive (HDD) using a rotating magnetic disk. SCM can include, for
example, memories, such as a Magnetic Resistance Random Access
Memory (MRAM), a Phase Change Memory (PCM), and a Resistive RAM
(RRAM).
[0002] Although SCM can allow for byte-addressable access of data
(i.e., in units less than a page size or a block size), the time to
write data to SCM may be much longer than the time to read data
from SCM. This has slowed the adoption of SCM as a more affordable
and power efficient alternative to memories conventionally used for
host memory, such as Dynamic Random Access Memory (DRAM) or Static
Random Access Memory (SRAM).
BRIEF DESCRIPTION OF THE DRAWINGS
[0003] The features and advantages of the embodiments of the
present disclosure will become more apparent from the detailed
description set forth below when taken in conjunction with the
drawings. The drawings and the associated descriptions are provided
to illustrate embodiments of the disclosure and not to limit the
scope of what is claimed.
[0004] FIG. 1 is a block diagram of an example environment
including a host and a device including a Storage Class Memory
(SCM) according to one or more embodiments.
[0005] FIG. 2 is a block diagram showing the handling of read
requests and write requests according to one or more
embodiments.
[0006] FIG. 3 is an example of a page table according to one or
more embodiments.
[0007] FIG. 4 is a state diagram for a page table entry according
to one or more embodiments.
[0008] FIG. 5 is a flowchart for a page table creation process
according to one or more embodiments.
[0009] FIG. 6 is a flowchart for a write request process according
to one or more embodiments.
[0010] FIG. 7 is a flowchart for a read request process for
byte-addressable data according to one or more embodiments.
[0011] FIG. 8 is a flowchart for a flush process from a host memory
to an SCM according to one or more embodiments.
[0012] FIG. 9 is a flowchart for a multiple interface process for a
device including an SCM according to one or more embodiments.
[0013] FIG. 10 is a flowchart for a block write process for a
device including an SCM according to one or more embodiments.
DETAILED DESCRIPTION
[0014] In the following detailed description, numerous specific
details are set forth to provide a full understanding of the
present disclosure. It will be apparent, however, to one of
ordinary skill in the art that the various embodiments disclosed
may be practiced without some of these specific details. In other
instances, well-known structures and techniques have not been shown
in detail to avoid unnecessarily obscuring the various
embodiments.
Example System Environments
[0015] FIG. 1 is a block diagram of an example environment
including host 101 and device 111 according to one or more
embodiments. Host 101 communicates with device 111 to retrieve data
from and store data in device 111. As further described below,
device 111 can serve as a memory device and/or as a storage device
for host 101 via respective device interfaces. Host 101 and device
111 may be separate devices or may be housed together as part of a
single electronic device, such as, for example, a server, computing
device, embedded device, desktop, laptop or notebook computer or
another type of electronic device such as a tablet, smartphone,
network media player, portable media player, television, digital
camera, or Digital Video Recorder (DVR). In other implementations,
host 101 may be a client computer or a storage controller, and
device 111 may be a memory/storage server or memory/storage node in
a network, such as in a cloud storage network or data center. As
used herein, a host can refer to a device that is capable of
issuing commands to a device to store data or retrieve data. In
this regard, host 101 may include another storage device such as a
smart data storage device that is capable of executing applications
and communicating with other memory/storage devices.
[0016] As shown in FIG. 1, device 111 includes Storage Class Memory
(SCM) 120 that provides non-volatile storage of data that can be
accessed at a byte level (i.e., at a cache line size) that is less
than a page size or a block size. SCM 120 can include, for example,
Chalcogenide RAM (C-RAM), Phase Change Memory (PCM), Programmable
Metallization Cell RAM (PMC-RAM or PMCm), Ovonic Unified Memory
(OUM), Resistive RAM (RRAM), Ferroelectric Memory (FeRAM),
Magnetoresistive RAM (MRAM), Fast NAND, and/or 3D-XPoint memory.
Such SCMs provide for faster reading and writing of data than
conventional non-volatile storage, such as flash memory or a
rotating magnetic disk. In some implementations, device 111 may
also include other types of non-volatile storage in addition to SCM
120, such as a rotating magnetic disk or flash memory.
[0017] Although SCM can provide faster reading and writing of data
than conventional forms of non-volatile storage, SCM generally
takes longer for writing data than for reading data. This can be
especially apparent in cases where address indirection is used in
the SCM, such as for wear leveling. As noted above, the longer
write latency of SCMs can prevent the use of SCM as a replacement
for volatile local memory, such as more expensive and greater power
consuming Dynamic Random Access Memory (DRAM) or Static Random
Access Memory (SRAM). According to one aspect of the present
disclosure, a Base Address Register (BAR) is exposed by device 111
to host 101 so that read commands may be sent for byte-addressable
data (e.g., for cache lines or less than a page size or block size)
using a memory device interface, while write commands are sent from
host 101 using a block device interface for writing data in larger
blocks of data. As discussed in more detail below, data to be
written in SCM 120 can be aggregated or modified in buffer 107 of
memory 106 of host 101 before being flushed to SCM 120. Host 101
can then send a write command for writing the aggregated or
modified block of data in SCM 120. This arrangement reduces the
latency for reading and writing data in SCM 120 so that SCM 120 can
be used for storing byte-addressable data that would otherwise be
stored in memory 106.
[0018] In the example of FIG. 1, host 101 includes processor
circuitry 102 for executing computer-executable instructions, such
as an Operating System (OS) of host 101. Processor circuitry 102
can include circuitry such as one or more processors for executing
instructions and can include, for example, a Central Processing
Unit (CPU), a Graphics Processing Unit (GPU), a microcontroller, a
Digital Signal Processor (DSP), an Application Specific Integrated
Circuit (ASIC), a Field Programmable Gate Array (FPGA), hard-wired
logic, analog circuitry and/or a combination thereof. In some
implementations, processor circuitry 102 can include a System on a
Chip (SoC). In this regard, the example of FIG. 1 shows Memory
Management Unit (MMU) 104 as being part of processor circuitry 102
or included in the same casing as processor circuitry 102. Those of
ordinary skill in the art will appreciate that processor circuitry
102, memory 106, and/or device interface 108 may be combined into a
single component or casing in some implementations. Other
implementations may not include an MMU. As used herein, an MMU can
be considered processor circuitry.
[0019] In the example of FIG. 1, processor circuitry 102 can access
memory 106 via MMU 104 to execute instructions, such as
instructions for executing a block device interface or a memory
device interface used to read data from or write data to device
111. In this regard, and as discussed in more detail below, buffer
107 of memory 106 can store byte-addressable data for write
requests that are aggregated or buffered to reach a block size for
SCM 120.
[0020] Memory 106 serves as a main memory for host 101 and can
include, for example, a volatile RAM such as DRAM or SRAM, a
non-volatile RAM, or other solid-state memory. While the
description herein refers to solid-state memory generally, it is
understood that solid-state memory may comprise one or more of
various types of memory devices such as flash integrated circuits,
C-RAM, PC-RAM or PRAM, Programmable Metallization Cell RAM (PMC-RAM
or PMCm), OUM, RRAM, NAND memory (e.g., Single-Level Cell (SLC)
memory, Multi-Level Cell (MLC) memory (i.e., two or more levels),
or any combination thereof), NOR memory, EEPROM, FeRAM, MRAM, other
discrete Non-Volatile Memory (NVM) chips, or any combination
thereof. In some implementations, memory 106 may be located
external to host 101, but used as a main memory for host 101.
[0021] Processor circuitry 102 also uses MMU 104 to access SCM 120
of device 111 via device interface 108. In some implementations,
MMU 104 can access a page table that translates virtual addresses
used by processor circuitry 102 into physical addresses (e.g., byte
addresses) indicating a location of where data for the virtual
addresses are to be stored in or retrieved from memory 106 or SCM
120. In this regard, MMU 104 may keep track of the locations for
byte-addressable data. In addition, MMU 104 may execute a memory
device interface (e.g., memory device interface 10 in FIG. 2) for
accessing byte-addressable data.
[0022] Device interface 108 allows host 101 to communicate with
device 111 via bus or interconnect 110. In some implementations,
device interface 108 may communicate with host interface 118 of
device 111 via bus or interconnect 110 using a standard, such as
Peripheral Component Interconnect express (PCIe), Ethernet, or
Fibre Channel. As discussed in more detail below, bus or
interconnect 110 can include a bus or interconnect that can allow
commands for both byte-addressable data with a memory device
interface and block-addressable data with a block device interface.
In other embodiments, host 101 and device 111 may communicate via
two or more buses or interconnects, each providing a memory device
interface, a block device interface, or both.
[0023] In this regard, processor circuitry 102 uses a plurality of
logical interfaces for reading data from and writing data to SCM
120 of device 111. For writing data and reading block-addressable
data, host 101 interfaces with device 111 using a block device or
storage device interface such as, for example, Non-Volatile Memory
express (NVMe) that may be implemented, for example, by an OS
driver executed by processor circuitry 102. For reading
byte-addressable data, host 101 interfaces with device 111 using a
memory device interface, such as a PCIe Base Address Register (BAR)
interface, Gen-Z, Open Coherent Accelerator Processor Interface
(OpenCAPI), or Cache Coherent Interconnect for Accelerators (CCIX),
that may be executed by processor circuitry 102. In some
implementations, the memory device interface may be implemented by
MMU 104, or by other circuitry of processor circuitry 102, such as
a hardware accelerator.
[0024] As shown in FIG. 1, device 111 includes host interface 118,
control circuitry 112, memory 116, and SCM 120. Host interface 118
allows device 111 to communicate with device interface 108 of host
101 via bus or interconnect 110. Host interface 118, in some
implementations, may communicate with device interface 108 of host
101 using a standard, such as PCIe, Ethernet, or Fibre Channel.
[0025] In addition, control circuitry 112 uses a plurality of
logical interfaces for receiving and performing read and write
commands from host 101 to access data in SCM 120. For reading and
writing block-addressable data, control circuitry 112 interfaces
with host 101 using a block device interface, which may include,
for example, an NVMe interface. For reading byte-addressable data,
control circuitry 112 interfaces with host 101 using a memory
device interface. The memory device interface may include, for
example, a PCIe BAR interface, Gen-Z, OpenCAPI, or CCIX.
[0026] Control circuitry 112 can include circuitry such as one or
more processors for executing instructions and can include, for
example, a CPU, a GPU, a microcontroller, a DSP, an ASIC, an FPGA,
hard-wired logic, analog circuitry and/or a combination thereof. In
some implementations, control circuitry 112 can include an SoC such
that one or both of host interface 118 and memory 116 may be
combined in a single chip with control circuitry 112. As with
processor circuitry 102 of host 101 discussed above, control
circuitry 112 of device 111 in some implementations can include
separate components, such as separate hardware accelerators for
implementing a memory device interface and a block device
interface.
[0027] Memory 116 of device 111 can include, for example, a
volatile RAM such as DRAM, a non-volatile RAM, or other solid-state
memory. Control circuitry 112 can access memory 116 to execute
instructions, such as a firmware of device 111 that can include
instructions for implementing the memory device interface and the
block device interface. In addition, control circuitry 112 may
access memory 116 for data used while executing a firmware of
device 111, data to be written in SCM 120, and/or data that has
been read from SCM 120.
[0028] Those of ordinary skill in the art will appreciate that
other implementations can include more or less than the elements
shown in FIG. 1, and that the processes discussed herein can be
implemented in other environments. For example, other environments
may not include an MMU in host 101, may include a separate MMU or
hardware accelerator for implementing a memory device interface, or
can include a different number of SCMs or different types of
non-volatile storage in device 111 in addition to SCM 120.
[0029] FIG. 2 is an example block diagram showing the handling of
read requests and write requests by host 101 and device 111
according to one or more embodiments. As shown in FIG. 2, memory
device interface 10 receives read request A and write request B,
while block device interface 12 receives read request C and write
request D. The read and write requests can come from applications
executed by processor circuitry 102. In the example of FIG. 2,
memory device interface 10 is implemented by MMU 104 of processor
circuitry 102 and block device interface 12 is implemented by an OS
driver of host 101 executed by processor circuitry 102.
[0030] Write request B is initially received by memory device
interface 10, but is redirected by memory device interface 10 to
block device interface 12, since memory device interface 10 is only
used for handling read requests for byte-addressable data, as
opposed to write requests. In some implementations, MMU 104 hands
control of the write request to an OS of host 101 since the memory
mapping to SCM 120 is marked as read-only. As noted above, SCM 120
generally performs read commands faster than write commands. In the
present disclosure, SCM 120 can serve as a local memory or a
partial DRAM replacement for host 101 for read requests, while
write requests are performed in memory 106 of host 101. This
ordinarily allows for a smaller sized local memory at host 101,
which can reduce power consumption and the cost of the overall
system including host 101 and device 111.
[0031] As used herein, read and write requests refer to data
accesses made at a byte-level (i.e., byte-addressable data), such
as cache line requests made by applications executed by processor
circuitry 102 of host 101. On the other hand, read and write
commands refer to commands sent to device 111 from host 101 to
access data either at a byte-level in the case of read commands
from memory device interface 10, or at a block-level (i.e., page or
block-addressable data) from block device interface 12. A page size
or block size can correspond to a unit of data in a virtual memory
that is managed by an OS of host 101. Data accessed in device 111
by block device interfaces 12 and 22 in FIG. 2 is read and written
in units of this page or block size. Some examples of a block or
page size can include 512 bytes, 4 KB, 8 KB, 2 MB, or 4 MB. In
contrast, byte-addressable data accessed by memory device
interfaces 10 and 20 in FIG. 2 allow data to be read in units of a
byte, including the reading of a single byte of data from SCM 120
of device 111.
[0032] As shown in FIG. 2, read request A for byte-addressable data
A is repackaged by memory device interface 10 as read command A,
which is sent to memory device interface 20 of device 111 to
retrieve byte-addressable data A from SCM 120. Memory device
interface 20 at device 111 receives read command A and uses
optional logical-to-physical mapping module 24 to identify a
physical address in SCM 120 storing byte-addressable data A. In
other implementations, logical-to-physical mapping module 24 may be
omitted, such as in implementations where SCM 120 does not use
address indirection for memory techniques such as, for example,
wear-leveling to more evenly distribute writing throughout SCM 120.
In such implementations, memory device interface 20, which may be
executed by control circuitry 112 in FIG. 1, can perform read
operation A in SCM 120. In performing read operation A,
byte-addressable data A is returned to memory device interface 20
and may be temporarily stored in a buffer, such as a buffer in
memory 116 of device 111 in FIG. 1, before being returned to memory
device interface 10 of host 101 to complete read command A.
[0033] Memory device interface 20 executed by control circuitry 112
of device 111 is configured to only receive and perform read
commands for byte-addressable data. The performance of write
commands received by memory device interface 20 may be blocked or
trigger an error at device 111. Such errors may or may not be
reported back to host 101.
[0034] In the example of FIG. 2, write request B is received by
memory device interface 10 of host 101. In some cases, write
request B can include a request to store byte-addressable data,
such as data from a cache flush command to flush one or more cache
lines from a cache or caches of processor circuitry 102 of host 101
(e.g., from an L1/L2/L3 cache). In other cases, write request B can
include a request to store block or page-addressable data, such as
data from an application executed by processor circuitry 102.
Memory device interface 10 identifies write request B as a write
request, and in response, redirects write request B to block device
interface 12. In some implementations, write requests received by
memory device interface 10 can trigger a fault handler that allows
the OS of host 101 via block device interface 12 to handle the
write request.
[0035] In the case where write request B is to store
byte-addressable data, block device interface 12 uses buffer 107 to
aggregate or modify one or more portions of the block of data
including the byte-addressable data to be written to SCM 120. Block
device interface 12 sends a read command for the block of data
including the byte-addressable data to be written to device 111.
Block device interface 22 of device 111 receives the read command
for the block and performs a read operation on SCM 120 and returns
the read block including the byte-addressable data to block device
interface 12 of host 101. Block device interface 12 buffers the
read block of data in buffer 107 and modifies one or more
byte-addressable portions of the buffered block for write request
B. In some cases, additional write requests for byte-addressable
data included in the buffered block may also be performed while the
block is stored in buffer 107.
[0036] Block device 12 then sends a write command for the modified
block including the byte-addressable data to flush the data for
write request from buffer 107 to SCM 120. In some cases, the write
command may include additional blocks that have been modified or
written, such as data for write request D. Block device interface
22 of device 111 receives the write command, and uses optional
logical-to-physical mapping module 24 to identify one or more
physical addresses in SCM 120 storing one or more blocks including
data B and D for write requests B and D. As noted above,
logical-to-physical mapping module 24 may be omitted in other
implementations, such as where SCM 120 does not use address
indirection. In such implementations, block device interface 22,
which may be executed by control circuitry 112 in FIG. 1, can
identify one or more addresses in SCM 120 for performing write
operations B and D. In performing write operations B and D, block
device interface 22 stores one or more blocks of data including
data B and data D in SCM 120. In addition, block device interface
22 may update byte-addresses used by memory device interface 20 for
data included in the one or more blocks of data with the locations
where the data has been written in SCM 120.
[0037] One or more write completion indications are returned to
block device interface 22 after completing the write operations.
Block device interface 22 may forward or send a write completion
indication to block device interface 12 of host 101 to indicate
that the write command or write commands have been completed and
may also provide the new byte-addressable physical addresses for
data stored in the write operations in addition to
block-addressable locations for the data. In other implementations,
memory device interface 20 may instead provide the updated
byte-addressable physical addresses to memory device interface 10
of host 101.
[0038] Read request C is also received by block device interface 12
of host 101. The data to be retrieved for read request C is
addressed in terms of pages or blocks, as opposed to being a
request for byte-addressable data, such as with read request A
discussed above. Block device interface 12 repackages the request
as read command C and sends read command C to block device
interface 22 of device 111. For its part, block device interface 22
performs read command C by using optional logical-to-physical
mapping module 24, which provides a physical address for reading
block-addressable data C from SCM 120. Block-addressable data C is
read from SCM 120 and returned to block device interface 22, which
passes the data on to block device interface 12 of host 101 to
complete the command. In some cases, data C may be buffered in a
memory of device 111, such as memory 116, before sending the data
to host 101.
[0039] As will be appreciated by those of ordinary skill in the
art, other implementations can include different components or
modules than those shown in the example of FIG. 2. For example,
other implementations may not include logical-to-physical mapping
module 24, such that memory device interface 20 and block device
interface 22 access SCM 120 without using a logical-to-physical
mapping module.
Page Table Examples
[0040] FIG. 3 is an example of page table 16 according to one or
more embodiments. Page table 16 may be created, for example, by
processor circuitry 102 of host 101 executing memory device
interface 10 in FIG. 2. In some implementations, processor
circuitry 102 may maintain multiple page tables, such as those for
mapping virtual addresses for byte-addressable data and also for
mapping virtual addresses for block-addressable data. In addition,
those of ordinary skill in the art will appreciate that page table
16 may include other information than shown in the example of FIG.
3, such as, for example, statistics information. Page table 16 may
be stored, for example, in memory 106 of host 101 or in a memory of
MMU 104 or other memory of processor circuitry 102.
[0041] As shown in FIG. 3, virtual addresses are assigned to
different data accessed by processor circuitry 102 and page table
16 indicates a type of access for the data and a physical address
where the data is stored in either memory 106 or SCM 120 of device
111. For example, byte-addressable data for virtual addresses A and
C in page table 16 is stored in SCM 120 at physical addresses SCM 1
and SCM 2, respectively. The data for virtual addresses A and C is
also indicated in page table 16 as having read-only access. In this
regard, memory device interface 20 of device 111 in some
implementations may expose a read-only Base Address Register (BAR)
to memory device interface 10 of host 101. The access for the
byte-addressable data represented in page table 16 is initially
read-only, and remains read-only until the data is written to or
otherwise modified.
[0042] In some implementations, memory device interface 20 of
device 111 may expose a portion of the BAR as a readable and
writable address range that maps to a memory of device 111, such as
memory 116 in FIG. 1. In such implementations, the portion of the
BAR that maps to SCM 120 can remain read-only, while the portion of
the BAR exposed to host 101 that maps to memory 116 can allow
byte-addressable writes and reads via memory device interface 10 of
host 101. This read/write portion of the BAR mapped to memory 116
can include, for example, a battery-backed or power fail-safe
volatile DRAM, or portion thereof, to effectively provide
non-volatile storage or non-volatile memory. For example, the
read/write portion of the BAR mapped to memory 116 may be protected
from data loss due to power interruption with a capacitor or
battery that may allow control circuitry 112 to transfer data from
the BAR mapped portion of memory 116 to SCM 120 after a loss in
power at device 111.
[0043] In the example of FIG. 3, after a write request has been
received for data represented in page table 16, such as for virtual
addresses B and D, the data for the write request is stored in
buffer 107 of memory 106. This is indicated in the example of FIG.
3 where the allowed access for virtual addresses B and D is
read/write and the physical addresses storing this data is
indicated as Mem. 1 and Mem. 2, respectively, in memory 106. Memory
device interface 10 in FIG. 2 can update page table 16 after
byte-addressable data included in the pages or blocks corresponding
to virtual addresses B and D have been rewritten or modified in
buffer 107 of memory 106. If needed, byte-addressable data included
in the buffered pages or blocks can be modified in buffer 107 or
read from buffer 107 by the same application that issued an
original write request for the data or by a different application.
As discussed in more detail below with reference to FIG. 4, the
state of such data remains read/write until the data is flushed
from buffer 107 to SCM 120.
[0044] FIG. 4 is a state diagram for a page table entry in page
table 16 of FIG. 3, such as an entry for one of virtual addresses
A, B, C, or D according to one or more embodiments. As shown in
FIG. 4, the entry begins in an initial state with read-only access
for a physical address in SCM 120. The physical address may
correspond to an address of a BAR provided by device 111 to memory
device interface 10 of host 101. This allows memory device
interface 10 to directly access the address in SCM 120 to read data
without using an OS of host 101.
[0045] The entry moves to the second state after a write request is
received for the data represented by the entry. As discussed above,
the write request can be handled as a software event by memory
device interface 10 and/or block device interface 12. This
ordinarily allows for more flexibility in the design and
implementation of host-side buffering than hardware solutions that
may rely exclusively on MMU 104.
[0046] In the second state, a block or page including the data for
the write request has been retrieved by block device interface 12
of host 101 and stored in buffer 107 of memory 106 in host 101. The
prior or obsolete version of the block may remain in SCM 120, but
the modified block or page in buffer 107 is the current or valid
version of the data for the virtual address. Memory device
interface 10 or block device interface 12 also updates page table
16 to change the access to read/write and to map the virtual
address for the entry to the physical address where the data has
been written in buffer 107 of memory 106.
[0047] In some implementations, memory device interface 10 or block
device interface 12 may identify that there have been no previous
writes to the block or page or that the write request is the first
write to the block or page. In such implementations, the data to be
written for the block or page may be stored in buffer 107 without
first retrieving the block or page from device 111. The write
request is then performed on the buffered block or page.
[0048] While the entry is in the second state, the block or page
for the entry stored in memory 106 can be modified or overwritten
by the same application that issued the write request or by a
different application. Data corresponding to the entry, such as
byte-addressable data within the buffered block or page, can also
be read from the physical address in memory 106 while the entry is
in the second state. Memory device interface 10 may refer to the
entry in page table 16 in response to read and write requests to
modify or read the byte-addressable data corresponding to the
virtual address that is stored in memory 106. Temporarily storing
the data in memory 106 can ordinarily allow for a faster write
operation than would be possible by writing the data to SCM 120. In
this regard, the buffering of the modified byte-addressable data in
memory 106 can be advantageous when the buffered data is soon
reused, since it can also be quickly read from memory 106. Data
buffered in memory 106 may also be read quicker than data read from
SCM 120. This can be especially beneficial for cache lines, which
are often read or modified soon after an initial write.
[0049] In addition, the aggregation or modification of data in
memory 106 and using a separate block device interface to flush an
aggregated or modified block of data in one write operation is more
efficient than making numerous smaller write operations in SCM 120,
which has a greater write latency than its read latency. The
foregoing use of both a block device interface and a memory device
interface with page table 16, and the buffering of written data in
buffer 107, can also provide a more efficient arrangement than
switching access of a BAR of SCM 120 from read-only to read/write
or switching or temporarily modifying a single interface of SCM 120
to accommodate byte-addressed and block-addressed data. Deferring
writes to SCM 120 can improve performance of the system including
host 101 and device 111 by allowing the writes to occur more
quickly in memory 106 of host 101, and writing the aggregated or
modified blocks to SCM 120 at a later time when write latency to
SCM 120 is less critical for processes or threads being executed by
processor circuitry 102 that may need to wait until the data has
been written before continuing execution.
[0050] After the data for the entry has been modified or aggregated
into one or more blocks by block device interface 12, the data for
the entry is flushed or de-staged by block device interface 12 from
buffer 107 to SCM 120 via block device interface 22 of device 111.
Block device interface 12 of host 101 updates the entry so that
access to the virtual address is unavailable or blocked while the
data is being flushed to SCM 120. In some implementations,
indicating in the page table that the virtual address is
unavailable can include removing or deleting an entry for the
virtual address or marking the entry unavailable or obsolete. This
can ensure consistency of the data so that different applications
are not modifying data in memory 106 before access of the flushed
data in SCM 120 is returned to read-only, which could result in
reading an old or obsolete version of the data. The use of memory
106 to temporarily buffer write requests provides an asynchronous
storage of data where the writing of the data to SCM 120 is
deferred to improve system performance in terms of Input/Output
Operations Per Second (IOPS), while the foregoing use of access
permissions in page table 16 allows for the data to remain
consistent.
[0051] As shown in FIG. 4, the entry returns to the first state of
read-only access in SCM 120 after the flush or de-staging operation
has completed. In some implementations, block device interface 22
of device 111 returns a command completion indication to block
device interface 12 of host 101 to indicate the completion of the
flush command. As discussed above with reference to FIG. 2, block
device interface 22 of device 111 may update memory device
interface 20 of device 111 with the byte-addressable addresses for
data written in SCM 120 for a write operation. Memory device
interface 20 of device 111 may then update memory device interface
10 of host 101 with the new byte-addressable addresses for the
flushed data so that memory device interface 10 can update the
entries in the page table with the new addresses for the flushed
data in SCM 120. In other implementations, block device interface
12 of host 101 may receive updated byte-addressable addresses from
block device interface 22 of device 111, which block device
interface 12 may use to update the page table.
Example Processes
[0052] FIG. 5 is a flowchart for a page table creation process
according to one or more embodiments. The process of FIG. 5 may be
performed by, for example, processor circuitry 102 of host 101
executing memory device interface 10.
[0053] In block 502, memory device interface 10 accesses the BAR of
SCM 120. In some implementations, control circuitry 112 of device
111 executing memory device interface 20 may expose a read-only BAR
of SCM 120 to memory device interface 10 of host 101. This allows
memory device interface 10 to have size and data type information
for SCM 120 for mapping virtual addresses of host 101 to physical
addresses of SCM 120 and enable direct memory access of SCM 120 by
host 101 for read operations. In addition, device 111 in some
implementations may also expose a read/write portion of the BAR
that maps to memory 116.
[0054] In block 504, memory device interface 10 creates a page
table including a plurality of entries corresponding to memory
locations in SCM 120. In more detail, the entries in the page table
correspond to the exposed BAR of device 111. The page table can
include entries for different virtual addresses and the mapped
physical addresses in SCM 120. In this regard, the created page
table can include entries for virtual addresses or pages that allow
memory device 10 to determine a physical location in SCM 120 of
device 111 for byte-addressable data that is smaller than a page or
block size. The created page table can also include an indication
of the allowed access for the physical address, as in the case of
page table 16 discussed above with reference to FIG. 3.
[0055] In block 506, memory device interface 10 sets the plurality
of entries in the page table as read-only. As discussed above, data
can be read from SCM 120 much quicker than data of the same size
can be written to SCM 120. The byte-addressable access to SCM 120
is therefore limited to read-only access. As discussed in more
detail below with reference to FIG. 6, writes to SCM 120 are
handled by block device interface 12 of host 101 so that data is
more efficiently written to SCM 120 in units of a block size (e.g.,
512 bytes, 4 KB, 8 KB, 2 MB, or 4 MB) instead of more numerous
writes that are smaller than a block size, such as for a single
byte of data. In addition, deferring writes to SCM 120 can improve
performance of the system including host 101 and device 111 by
allowing the writes to occur more quickly in memory 106 of host
101, and writing the aggregated or modified data to SCM 120 at a
later time when the effect of the write latency of SCM 120 does not
delay execution of a process or thread.
[0056] FIG. 6 is a flowchart for a write request process according
to one or more embodiments. The process of FIG. 6 may be performed
by, for example, processor circuitry 102 of host 101 executing
memory device interface 10 and block device interface 12.
[0057] In block 602, memory device interface 10 or block device
interface 12 receives a write request to write data corresponding
to an entry in a page table (e.g., page table 16 in FIG. 3). The
write request may come from, for example, an application executed
by processor circuitry 102. The write request is for data smaller
than a page or block size and is therefore byte-addressable. In
such cases, the write request may result from flushing or
de-staging a cache of processor circuitry 102. For example, an L1,
L2, or L3 cache of processor circuitry 102 may have one or more
cache lines flushed or de-staged.
[0058] In block 604, data for the write request is written in
buffer 107 of memory 106 of device 111 using block device interface
12. As discussed above, the write request to store byte-addressable
data may be received by block device interface 12 of host 101 after
redirection from memory device interface 10 or from another module,
such as from a portion of an OS of host 101. For example, in cases
where the write request is initially received by memory device
interface 10, the write request may trigger a fault handler that
redirects the write request to block device interface 12. As
discussed above with reference to FIG. 2, block device interface 12
can also receive write requests for block-addressable data.
[0059] The byte-addressable data written in buffer 107 for the
write request received in block 602 may be aggregated into units of
a page or block size or a current version of the block or page
including the byte-addressable data may be read from device 111 and
stored in buffer 107 for performing the write request. As noted
above, write operations take much longer to perform in SCM 120 than
read operations for a given amount of data. Performing write
requests in buffer 107 can result in performing less overall writes
to SCM 120 and in completing the smaller intermediate writes faster
in memory 106 to improve the efficiency and performance of host 101
and device 111. Write requests for data that is already in units of
a block or page size may also be buffered in memory 106 in some
implementations to improve the performance of write operations by
reducing the latency of performing the write operations. As noted
above, the faster completion of write requests can allow for
processes and threads to continue execution rather than wait for
data to be written to SCM 120. In other embodiments,
block-addressable data may instead be written to SCM 120 without
deferring the writing of such data to SCM 120 with the use of
memory 106. Such an arrangement may be preferred in cases where the
size of memory 106 is limited.
[0060] In block 606, block device interface 12 changes the entry
for the virtual address in the page table from read-only access to
both read and write access. Block device interface 12 also changes
the entry for the first virtual address to point to a location or
physical address in memory 106 where data for the first virtual
address was written. As discussed above with reference to the page
table entry state diagram of FIG. 4, the data buffered in memory
106 can be written and read in memory 106 until it is flushed or
de-staged from memory 106 to SCM 120 in page-sized or block-sized
units. The modification and flushing of data from memory 106 is
discussed in more detail below with reference to the process of
FIG. 8.
[0061] FIG. 7 is a flowchart for a read request process for
byte-addressable data according to one or more embodiments. The
process of FIG. 7 may be performed by, for example, processor
circuitry 102 of host 101 executing memory device interface 10.
[0062] In block 702, a read request is received by memory device
interface 10 at host 101 to read byte-addressable data
corresponding to an entry in a page table (e.g., page table 16 in
FIG. 3). The read request may come from, for example, an
application executed by processor circuitry 102. The read request
can be for data smaller than a page or block size and is therefore
byte-addressable. In some cases, the read request may result from a
process or thread that is executed by processor circuitry 102 to
load data into a cache of processor circuitry 102.
[0063] In block 704, memory device interface 10 uses a page table
to determine whether the requested data is located in memory 106.
If so, memory device interface 10 reads the data from memory 106 in
block 706. On the other hand, if it is determined that the
requested byte-addressable data is located in SCM 120, memory
device interface 10 in block 708 sends a read request to memory
device interface 20 of device 111 to directly access the requested
data from SCM 120. Unlike data access performed by block device
interface 12 of host 101, the read requests performed by memory
device interface 10 of host 101 may not require use of the OS of
host 101.
[0064] As discussed above, by allowing read-only access to a BAR of
SCM 120, it is ordinarily possible to take advantage of the
relatively quick read access of SCM 120 for byte-addressable data,
without incurring the greater performance penalty of writing
byte-addressable data to SCM 120. This can allow for a smaller main
memory used by host 101 (e.g., memory 106) or a storage space
savings for the host's main memory, which may be internal or
external to host 101. As noted above, memory 106 can include a DRAM
or SRAM in some implementations that can provide faster read and
write access than SCM 120, but costs more and consumes more power
for a given amount of data storage.
[0065] FIG. 8 is a flowchart for a flush process from a local
memory to an SCM according to one or more embodiments. The process
of FIG. 8 may be performed by, for example, processor circuitry 102
executing block device interface 12.
[0066] In block 802, block device interface 12 receives a write
request to write byte-addressable data corresponding to an entry in
a page table. The byte-addressable data to be written can include
data within a page or block represented by the page table entry.
Such write data may come from, for example, processes or threads
executed by processor circuitry 102 that may flush or de-stage
dirty cache lines from a cache of processor circuitry 102.
[0067] In block 804, block device interface 12 reads a block of
data from SCM 120 for the block or page of data represented by the
page table entry. The block or page is stored in buffer 107 of
memory 106. In addition, block device interface 12 updates the page
table to indicate that an entry or virtual address for the buffered
block or page has read/write access and that the data for the entry
is located at a physical address in memory 106.
[0068] In block 806, block device interface 12 modifies the
byte-addressable data for the write request by writing the data to
the block or page buffered in buffer 107. As noted above,
additional write requests and read requests may also be performed
on the same byte-addressable data or on other byte-addressable
portions of the buffered block or page while the block or page is
stored in memory 106.
[0069] In block 808, block device interface 12 indicates in the
page table that the data for the buffered block or page is
unavailable for reading and writing data in preparation for
flushing of the modified block to SCM 120. As noted above with
respect to the state diagram of FIG. 4, indicating that data for
the entry or virtual address is unavailable in the page table can
help ensure consistency of the data by not allowing the data to be
read or written while it is flushed to SCM 120.
[0070] In block 810, block device interface 12 sends a write
command to device 111 to flush or de-stage the modified block of
data from buffer 107 to SCM 120. In some implementations, block
device interface 12 may wait until a threshold number of blocks
have been aggregated in buffer 107 or may wait a predetermined
amount of time with no accesses to the data in a block before
flushing the modified block or blocks to SCM 120 via block device
interface 22 of device 111. In other cases, block device interface
12 may flush an aggregated block of data in response to reaching a
block's worth of data in buffer 107, such as when new write data is
collected in buffer 107 for a page or block that has not been
previously stored in device 111. In yet other implementations, the
flushing of a block or blocks of data from buffer 107 may depend on
a remaining storage capacity of buffer 107. For example, block
device interface 12 may flush one or more blocks of data from
buffer 107 in response to reaching 80% of the storage capacity of
buffer 107.
[0071] In block 812, block device interface 12 sets the entry for
the modified block of data as read-only in the page table in
response to completion of the flushing operation. This corresponds
to returning to the first state from the third state in the example
state diagram of FIG. 4. The read-only access can correspond to a
read-only BAR of SCM 120 used by memory device interface 10. The
completion of the flushing operation may be determined by block
device interface 12 based on the receipt of a command completion
indication from block device interface 22 of device 111.
[0072] In block 814, block device interface 12 or memory device
interface 10 updates the entry for the flushed block of data in the
page table to point a location in SCM 120 where the block was
flushed. In this regard, the new physical address of the data in
SCM 120 may be received by block device interface 12 as part of the
flush command completion indication or alternatively may be
received by memory device interface 10 via an update process of
memory device interface 20 of device 111.
[0073] FIG. 9 is a flowchart for a multiple interface process of a
device including SCM according to one or more embodiments. The
process of FIG. 9 may be performed by, for example, control
circuitry 112 of device 111.
[0074] In block 902, control circuitry 112 uses block device
interface 22 for receiving write commands from host 101 to write
data in blocks to SCM 120. As discussed above, block device
interface 22 is also used to read block-addressable data from SCM
120.
[0075] In addition, control circuitry 112 uses memory device
interface 20 in block 904 for receiving read commands from host 101
to read byte-addressable data from SCM 120. The use of two
interfaces at device 111 allows SCM 120 to be used by host 101 as a
main memory for reading byte-addressable data and as a non-volatile
storage for blocks of data.
[0076] In block 906, memory device interface 20 exposes a read-only
BAR for SCM 120 to host 101. As noted above, the exposed BAR for
device 111 may also include a read/write portion located in memory
116 of device 111. The BAR may be exposed via, for example, a PCIe
bus or interconnect that allows commands for byte-addressable data
using a memory device interface, and also allows commands for
block-addressable data using a block device interface, such as
NVMe, for example. The use of a BAR can allow processor circuitry
102 at host 101 to create and update a page table that maps virtual
addresses used by applications executed by processor circuitry 102
to physical addresses in SCM 120.
[0077] FIG. 10 is a flowchart for a device block write process
according to one or more embodiments. The process of FIG. 10 may be
performed by, for example, control circuitry 112 of device 111.
[0078] In block 1002 of FIG. 10, control circuitry 112 of device
111 receives a write command from host 101 via block device
interface 22 to write one or more blocks of data in SCM 120. In
some implementations, control circuitry 112 may use a
logical-to-physical mapping module to determine a location for
writing the one or more blocks for the write command in a write
operation sent to SCM 120. In other implementations, control
circuitry 112 may not use such a logical-to-physical mapping
module.
[0079] After receiving a confirmation of the completion of the
write operation in SCM 120, control circuitry 112 updates memory
device interface 20 in block 1004 with the physical addresses for
the data written. In some implementations, the updated addresses
are shared with host 101 via memory device interface 20 of device
111 so that memory device interface 10 at host 101 can update a
page table.
[0080] As discussed above, the use of an SCM for reading
byte-addressable data and writing to the SCM in blocks can allow
the SCM replace at least some of a host's main memory, while
reducing the effects of the greater write latency of the SCM. In
addition, the use of the host's main memory for temporarily
buffering byte-addressable data that has been modified and updating
a page table for the buffered data can help ensure that an old or
obsolete version of the data is not read from the SCM.
Other Embodiments
[0081] Those of ordinary skill in the art will appreciate that the
various illustrative logical blocks, modules, and processes
described in connection with the examples disclosed herein may be
implemented as electronic hardware, computer software, or
combinations of both. Furthermore, the foregoing processes can be
embodied on a computer readable medium which causes processor or
control circuitry to perform or execute certain functions.
[0082] To clearly illustrate this interchangeability of hardware
and software, various illustrative components, blocks, and modules
have been described above generally in terms of their
functionality. Whether such functionality is implemented as
hardware or software depends upon the particular application and
design constraints imposed on the overall system. Those of ordinary
skill in the art may implement the described functionality in
varying ways for each particular application, but such
implementation decisions should not be interpreted as causing a
departure from the scope of the present disclosure.
[0083] The various illustrative logical blocks, units, modules,
processor circuitry, and control circuitry described in connection
with the examples disclosed herein may be implemented or performed
with a general-purpose processor, a DSP, an ASIC, an FPGA or other
programmable logic device, discrete gate or transistor logic,
discrete hardware components, or any combination thereof designed
to perform the functions described herein. A general-purpose
processor may be a microprocessor, but in the alternative, the
circuitry may be any conventional processor, controller,
microcontroller, or state machine. Processor or control circuitry
may also be implemented as a combination of computing devices,
e.g., a combination of a DSP and a microprocessor, a plurality of
microprocessors, an SoC, one or more microprocessors in conjunction
with a DSP core, or any other such configuration.
[0084] The activities of a method or process described in
connection with the examples disclosed herein may be embodied
directly in hardware, in a software module executed by processor or
control circuitry, or in a combination of the two. The steps of the
method or algorithm may also be performed in an alternate order
from those provided in the examples. A software module may reside
in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM
memory, registers, hard disk, a removable media, an optical media,
or any other form of storage medium known in the art. An exemplary
storage medium is coupled to the processor or control circuitry
such that the circuitry can read information from, and write
information to, the storage medium. In the alternative, the storage
medium may be integral to processor or control circuitry. The
circuitry and the storage medium may reside in an ASIC or an
SoC.
[0085] The foregoing description of the disclosed example
embodiments is provided to enable any person of ordinary skill in
the art to make or use the embodiments in the present disclosure.
Various modifications to these examples will be readily apparent to
those of ordinary skill in the art, and the principles disclosed
herein may be applied to other examples without departing from the
spirit or scope of the present disclosure. The described
embodiments are to be considered in all respects only as
illustrative and not restrictive. In addition, the use of language
in the form of "at least one of A and B" in the following claims
should be understood to mean "only A, only B, or both A and B."
* * * * *