Hyperscale artificial intelligence and machine learning infrastructure — Pure Storage, Inc. (US11494692B1)

ABSTRACT

Abstract

A hyperscale artificial intelligence and machine learning infrastructure includes a plurality of racks, where: at least one or more of the racks include one or more GPU servers; at least one or more of the racks include one or more storage systems; each of the racks include one or more switches coupled to at least one switch in another rack; and the one or more GPU servers are configured to execute one or more artificial intelligence or machine learning applications, wherein data stored within the one or more storage systems is used as input to the one or more artificial intelligence or machine learning applications.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation-in-part application of and claims priority from U.S. patent application Ser. No. 16/045,814, filed Jul. 26, 2018, and is a non-provisional application of and claims priority from: U.S. Provisional Patent Application Ser. No. 62/648,368, filed Mar. 26, 2018, U.S. Provisional Patent Application Ser. No. 62/650,736, filed Mar. 30, 2018, and U.S. Provisional Patent Application Ser. No. 62/818,326, filed Mar. 14, 2019. Each of the priority documents are hereby incorporated by reference in their entirety.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A illustrates a first example system for data storage in accordance with some implementations.

FIG. 1B illustrates a second example system for data storage in accordance with some implementations.

FIG. 1C illustrates a third example system for data storage in accordance with some implementations.

FIG. 1D illustrates a fourth example system for data storage in accordance with some implementations.

FIG. 2A is a perspective view of a storage cluster with multiple storage nodes and internal storage coupled to each storage node to provide network attached storage, in accordance with some embodiments.

FIG. 2B is a block diagram showing an interconnect switch coupling multiple storage nodes in accordance with some embodiments.

FIG. 2C is a multiple level block diagram, showing contents of a storage node and contents of one of the non-volatile solid state storage units in accordance with some embodiments.

FIG. 2D shows a storage server environment, which uses embodiments of the storage nodes and storage units of some previous figures in accordance with some embodiments.

FIG. 2E is a blade hardware block diagram, showing a control plane, compute and storage planes, and authorities interacting with underlying physical resources, in accordance with some embodiments.

FIG. 2F depicts elasticity software layers in blades of a storage cluster, in accordance with some embodiments.

FIG. 2G depicts authorities and storage resources in blades of a storage cluster, in accordance with some embodiments.

FIG. 3A sets forth a diagram of a storage system that is coupled for data communications with a cloud services provider in accordance with some embodiments of the present disclosure.

FIG. 3B sets forth a diagram of a storage system in accordance with some embodiments of the present disclosure.

FIG. 3C sets forth an example of a cloud-based storage system in accordance with some embodiments of the present disclosure.

FIG. 3D illustrates an exemplary computing device that may be specifically configured to perform one or more of the processes described herein.

FIG. 4 sets forth a diagram illustrating an example computer architecture for implementing an artificial intelligence and machine learning infrastructure that is configured to fit within a single chassis according to some embodiments of the present disclosure.

FIG. 5 sets forth an example of a hyperscale artificial intelligence and machine learning infrastructure that includes multiple artificial intelligence and machine learning infrastructures according to some embodiments of the present disclosure.

FIG. 6 sets forth an example of a hyperscale artificial intelligence and machine learning infrastructure that includes multiple artificial intelligence and machine learning infrastructures according to some embodiments of the present disclosure.

FIG. 7 sets forth an example of a single rack that includes an artificial intelligence and machine learning infrastructure according to some embodiments of the present disclosure.

FIG. 8 sets forth an example where components of the artificial intelligence and machine learning infrastructure are decomposed into a compute rack and a storage rack according to some embodiments of the present disclosure.

FIG. 9 sets forth a diagram illustrating an example artificial intelligence and machine learning infrastructure according to some embodiments of the present disclosure.

FIG. 10 sets forth a diagram illustrating an example artificial intelligence and machine learning infrastructure according to some embodiments of the present disclosure.

FIG. 11 sets forth a diagram illustrating an example implementation of an artificial intelligence and machine learning infrastructure software stack according to some embodiments of the present disclosure.

FIG. 12 sets forth a flow chart illustrating an example method of performing artificial intelligence and machine learning processes in a hyperscale artificial intelligence and machine learning infrastructure according to embodiments of the present disclosure.

FIG. 13 sets forth a flow chart illustrating another example method of performing artificial intelligence and machine learning processes in a hyperscale artificial intelligence and machine learning infrastructure according to embodiments of the present disclosure.

DESCRIPTION OF EMBODIMENTS

Example methods, apparatus, and products for hyperscale artificial intelligence and machine learning infrastructure in accordance with embodiments of the present disclosure are described with reference to the accompanying drawings, beginning with FIG. 1A . FIG. 1A illustrates an example system for data storage, in accordance with some implementations. System 100 (also referred to as “storage system” herein) includes numerous elements for purposes of illustration rather than limitation. It may be noted that system 100 may include the same, more, or fewer elements configured in the same or different manner in other implementations.

System 100 includes a number of computing devices 164 A-B. Computing devices (also referred to as “client devices” herein) may be embodied, for example, a server in a data center, a workstation, a personal computer, a notebook, or the like. Computing devices 164 A-B may be coupled for data communications to one or more storage arrays 102 A-B through a storage area network (‘SAN’) 158 or a local area network (‘LAN’) 160 .

The SAN 158 may be implemented with a variety of data communications fabrics, devices, and protocols. For example, the fabrics for SAN 158 may include Fibre Channel, Ethernet, Infiniband, Serial Attached Small Computer System Interface (‘SAS’), or the like. Data communications protocols for use with SAN 158 may include Advanced Technology Attachment (‘ATA’), Fibre Channel Protocol, Small Computer System Interface (‘SCSI’), Internet Small Computer System Interface (‘iSCSI’), HyperSCSI, Non-Volatile Memory Express (‘NVMe’) over Fabrics, or the like. It may be noted that SAN 158 is provided for illustration, rather than limitation. Other data communication couplings may be implemented between computing devices 164 A-B and storage arrays 102 A-B.

The LAN 160 may also be implemented with a variety of fabrics, devices, and protocols. For example, the fabrics for LAN 160 may include Ethernet ( 802 . 3 ), wireless ( 802 . 11 ), or the like. Data communication protocols for use in LAN 160 may include Transmission Control Protocol (‘TCP’), User Datagram Protocol (‘UDP’), Internet Protocol (‘IP’), HyperText Transfer Protocol (‘HTTP’), Wireless Access Protocol (‘WAP’), Handheld Device Transport Protocol (‘HDTP’), Session Initiation Protocol (‘SIP’), Real Time Protocol (‘RTP’), or the like.

Storage arrays 102 A-B may provide persistent data storage for the computing devices 164 A- B. Storage array 102 A may be contained in a chassis (not shown), and storage array 102 B may be contained in another chassis (not shown), in implementations. Storage array

102 A and 102 B may include one or more storage array controllers 110 A-D (also referred to as “controller” herein). A storage array controller 110 A-D may be embodied as a module of automated computing machinery comprising computer hardware, computer software, or a combination of computer hardware and software. In some implementations, the storage array controllers 110 A-D may be configured to carry out various storage tasks. Storage tasks may include writing data received from the computing devices 164 A-B to storage array 102 A-B, erasing data from storage array 102 A-B, retrieving data from storage array 102 A-B and providing data to computing devices 164 A-B, monitoring and reporting of disk utilization and performance, performing redundancy operations, such as Redundant Array of Independent Drives (‘RAID’) or RAID-like data redundancy operations, compressing data, encrypting data, and so forth.

Storage array controller 110 A-D may be implemented in a variety of ways, including as a Field Programmable Gate Array (‘FPGA’), a Programmable Logic Chip (‘PLC’), an Application Specific Integrated Circuit (‘ASIC’), System-on-Chip (‘SOC’), or any computing device that includes discrete components such as a processing device, central processing unit, computer memory, or various adapters. Storage array controller 110 A-D may include, for example, a data communications adapter configured to support communications via the SAN 158 or LAN 160 . In some implementations, storage array controller 110 A-D may be independently coupled to the LAN 160 . In implementations, storage array controller 110 A-D may include an I/O controller or the like that couples the storage array controller 110 A-D for data communications, through a midplane (not shown), to a persistent storage resource 170 A-B (also referred to as a “storage resource” herein). The persistent storage resource 170 A-B main include any number of storage drives 171 A-F (also referred to as “storage devices” herein) and any number of non-volatile Random Access Memory (‘NVRAM’) devices (not shown).

In some implementations, the NVRAM devices of a persistent storage resource 170 A-B may be configured to receive, from the storage array controller 110 A-D, data to be stored in the storage drives 171 A-F. In some examples, the data may originate from computing devices 164 A-B. In some examples, writing data to the NVRAM device may be carried out more quickly than directly writing data to the storage drive 171 A-F. In implementations, the storage array controller 110 A-D may be configured to utilize the NVRAM devices as a quickly accessible buffer for data destined to be written to the storage drives 171 A-F. Latency for write requests using NVRAM devices as a buffer may be improved relative to a system in which a <figure-callout id="

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation-in-part application of and claims priority from U.S. patent application Ser. No. 16/045,814, filed Jul. 26, 2018, and is a non-provisional application of and claims priority from: U.S. Provisional Patent Application Ser. No. 62/648,368, filed Mar. 26, 2018, U.S. Provisional Patent Application Ser. No. 62/650,736, filed Mar. 30, 2018, and U.S. Provisional Patent Application Ser. No. 62/818,326, filed Mar. 14, 2019. Each of the priority documents are hereby incorporated by reference in their entirety.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A illustrates a first example system for data storage in accordance with some implementations.

FIG. 1B illustrates a second example system for data storage in accordance with some implementations.

FIG. 1C illustrates a third example system for data storage in accordance with some implementations.

FIG. 1D illustrates a fourth example system for data storage in accordance with some implementations.

FIG. 2A is a perspective view of a storage cluster with multiple storage nodes and internal storage coupled to each storage node to provide network attached storage, in accordance with some embodiments.

FIG. 2B is a block diagram showing an interconnect switch coupling multiple storage nodes in accordance with some embodiments.

FIG. 2C is a multiple level block diagram, showing contents of a storage node and contents of one of the non-volatile solid state storage units in accordance with some embodiments.

FIG. 2D shows a storage server environment, which uses embodiments of the storage nodes and storage units of some previous figures in accordance with some embodiments.

FIG. 2E is a blade hardware block diagram, showing a control plane, compute and storage planes, and authorities interacting with underlying physical resources, in accordance with some embodiments.

FIG. 2F depicts elasticity software layers in blades of a storage cluster, in accordance with some embodiments.

FIG. 2G depicts authorities and storage resources in blades of a storage cluster, in accordance with some embodiments.

FIG. 3A sets forth a diagram of a storage system that is coupled for data communications with a cloud services provider in accordance with some embodiments of the present disclosure.

FIG. 3B sets forth a diagram of a storage system in accordance with some embodiments of the present disclosure.

FIG. 3C sets forth an example of a cloud-based storage system in accordance with some embodiments of the present disclosure.

FIG. 3D illustrates an exemplary computing device that may be specifically configured to perform one or more of the processes described herein.

FIG. 4 sets forth a diagram illustrating an example computer architecture for implementing an artificial intelligence and machine learning infrastructure that is configured to fit within a single chassis according to some embodiments of the present disclosure.

FIG. 5 sets forth an example of a hyperscale artificial intelligence and machine learning infrastructure that includes multiple artificial intelligence and machine learning infrastructures according to some embodiments of the present disclosure.

FIG. 6 sets forth an example of a hyperscale artificial intelligence and machine learning infrastructure that includes multiple artificial intelligence and machine learning infrastructures according to some embodiments of the present disclosure.

FIG. 7 sets forth an example of a single rack that includes an artificial intelligence and machine learning infrastructure according to some embodiments of the present disclosure.

FIG. 8 sets forth an example where components of the artificial intelligence and machine learning infrastructure are decomposed into a compute rack and a storage rack according to some embodiments of the present disclosure.

FIG. 9 sets forth a diagram illustrating an example artificial intelligence and machine learning infrastructure according to some embodiments of the present disclosure.

FIG. 10 sets forth a diagram illustrating an example artificial intelligence and machine learning infrastructure according to some embodiments of the present disclosure.

FIG. 11 sets forth a diagram illustrating an example implementation of an artificial intelligence and machine learning infrastructure software stack according to some embodiments of the present disclosure.

FIG. 12 sets forth a flow chart illustrating an example method of performing artificial intelligence and machine learning processes in a hyperscale artificial intelligence and machine learning infrastructure according to embodiments of the present disclosure.

FIG. 13 sets forth a flow chart illustrating another example method of performing artificial intelligence and machine learning processes in a hyperscale artificial intelligence and machine learning infrastructure according to embodiments of the present disclosure.

DESCRIPTION OF EMBODIMENTS

Example methods, apparatus, and products for hyperscale artificial intelligence and machine learning infrastructure in accordance with embodiments of the present disclosure are described with reference to the accompanying drawings, beginning with FIG. 1A . FIG. 1A illustrates an example system for data storage, in accordance with some implementations. System 100 (also referred to as “storage system” herein) includes numerous elements for purposes of illustration rather than limitation. It may be noted that system 100 may include the same, more, or fewer elements configured in the same or different manner in other implementations.

System 100 includes a number of computing devices 164 A-B. Computing devices (also referred to as “client devices” herein) may be embodied, for example, a server in a data center, a workstation, a personal computer, a notebook, or the like. Computing devices 164 A-B may be coupled for data communications to one or more storage arrays 102 A-B through a storage area network (‘SAN’) 158 or a local area network (‘LAN’) 160 .

The SAN 158 may be implemented with a variety of data communications fabrics, devices, and protocols. For example, the fabrics for SAN 158 may include Fibre Channel, Ethernet, Infiniband, Serial Attached Small Computer System Interface (‘SAS’), or the like. Data communications protocols for use with SAN 158 may include Advanced Technology Attachment (‘ATA’), Fibre Channel Protocol, Small Computer System Interface (‘SCSI’), Internet Small Computer System Interface (‘iSCSI’), HyperSCSI, Non-Volatile Memory Express (‘NVMe’) over Fabrics, or the like. It may be noted that SAN 158 is provided for illustration, rather than limitation. Other data communication couplings may be implemented between computing devices 164 A-B and storage arrays 102 A-B.

The LAN 160 may also be implemented with a variety of fabrics, devices, and protocols. For example, the fabrics for LAN 160 may include Ethernet ( 802 . 3 ), wireless ( 802 . 11 ), or the like. Data communication protocols for use in LAN 160 may include Transmission Control Protocol (‘TCP’), User Datagram Protocol (‘UDP’), Internet Protocol (‘IP’), HyperText Transfer Protocol (‘HTTP’), Wireless Access Protocol (‘WAP’), Handheld Device Transport Protocol (‘HDTP’), Session Initiation Protocol (‘SIP’), Real Time Protocol (‘RTP’), or the like.

Storage arrays 102 A-B may provide persistent data storage for the computing devices 164 A- B. Storage array 102 A may be contained in a chassis (not shown), and storage array 102 B may be contained in another chassis (not shown), in implementations. Storage array

102 A and 102 B may include one or more storage array controllers 110 A-D (also referred to as “controller” herein). A storage array controller 110 A-D may be embodied as a module of automated computing machinery comprising computer hardware, computer software, or a combination of computer hardware and software. In some implementations, the storage array controllers 110 A-D may be configured to carry out various storage tasks. Storage tasks may include writing data received from the computing devices 164 A-B to storage array 102 A-B, erasing data from storage array 102 A-B, retrieving data from storage array 102 A-B and providing data to computing devices 164 A-B, monitoring and reporting of disk utilization and performance, performing redundancy operations, such as Redundant Array of Independent Drives (‘RAID’) or RAID-like data redundancy operations, compressing data, encrypting data, and so forth.

Storage array controller 110 A-D may be implemented in a variety of ways, including as a Field Programmable Gate Array (‘FPGA’), a Programmable Logic Chip (‘PLC’), an Application Specific Integrated Circuit (‘ASIC’), System-on-Chip (‘SOC’), or any computing device that includes discrete components such as a processing device, central processing unit, computer memory, or various adapters. Storage array controller 110 A-D may include, for example, a data communications adapter configured to support communications via the SAN 158 or LAN 160 . In some implementations, storage array controller 110 A-D may be independently coupled to the LAN 160 . In implementations, storage array controller 110 A-D may include an I/O controller or the like that couples the storage array controller 110 A-D for data communications, through a midplane (not shown), to a persistent storage resource 170 A-B (also referred to as a “storage resource” herein). The persistent storage resource 170 A-B main include any number of storage drives 171 A-F (also referred to as “storage devices” herein) and any number of non-volatile Random Access Memory (‘NVRAM’) devices (not shown).

In some implementations, the NVRAM devices of a persistent storage resource 170 A-B may be configured to receive, from the storage array controller 110 A-D, data to be stored in the storage drives 171 A-F. In some examples, the data may originate from computing devices 164 A-B. In some examples, writing data to the NVRAM device may be carried out more quickly than directly writing data to the storage drive 171 A-F. In implementations, the storage array controller 110 A-D may be configured to utilize the NVRAM devices as a quickly accessible buffer for data destined to be written to the storage drives 171 A-F. Latency for write requests using NVRAM devices as a buffer may be improved relative to a system in which a storage array controller 110 A-D writes data directly to the storage drives 171 A-F. In some implementations, the NVRAM devices may be implemented with computer memory in the form of high bandwidth, low latency RAM. The NVRAM device is referred to as “non-volatile” because the NVRAM device may receive or include a unique power source that maintains the state of the RAM after main power loss to the NVRAM device. Such a power source may be a battery, one or more capacitors, or the like. In response to a power loss, the NVRAM device may be configured to write the contents of the RAM to a persistent storage, such as the storage drives 171 A-F.

In implementations, storage drive 171 A-F may refer to any device configured to record data persistently, where “persistently” or “persistent” refers as to a device&#39;s ability to maintain recorded data after loss of power. In some implementations, storage drive 171 A-F may correspond to non-disk storage media. For example, the storage drive 171 A-F may be one or more solid-state drives (‘SSDs’), flash memory based storage, any type of solid-state non-volatile memory, or any other type of non-mechanical storage device. In other implementations, storage drive 171 A-F may include mechanical or spinning hard disk, such as hard-disk drives (‘HDD’).

In some implementations, the storage array controllers 110 A-D may be configured for offloading device management responsibilities from storage drive 171 A-F in storage array 102 A-B. For example, storage array controllers 110 A-D may manage control information that may describe the state of one or more memory blocks in the storage drives 171 A-F. The control information may indicate, for example, that a particular memory block has failed and should no longer be written to, that a particular memory block contains boot code for a storage array controller 110 A-D, the number of program-erase (‘P/E’) cycles that have been performed on a particular memory block, the age of data stored in a particular memory block, the type of data that is stored in a particular memory block, and so forth. In some implementations, the control information may be stored with an associated memory block as metadata. In other implementations, the control information for the storage drives 171 A-F may be stored in one or more particular memory blocks of the storage drives 171 A-F that are selected by the storage array controller 110 A-D. The selected memory blocks may be tagged with an identifier indicating that the selected memory block contains control information. The identifier may be utilized by the storage array controllers 110 A-D in conjunction with storage drives 171 A-F to quickly identify the memory blocks that contain control information. For example, the storage controllers 110 A-D may issue a command to locate memory blocks that contain control information. It may be noted that control information may be so large that parts of the control information may be stored in multiple locations, that the control information may be stored in multiple locations for purposes of redundancy, for example, or that the control information may otherwise be distributed across multiple memory blocks in the storage drive 171 A-F.

In implementations, storage array controllers 110 A-D may offload device management responsibilities from storage drives 171 A-F of storage array 102 A-B by retrieving, from the storage drives 171 A-F, control information describing the state of one or more memory blocks in the storage drives 171 A-F. Retrieving the control information from the storage drives 171 A-F may be carried out, for example, by the storage array controller 110 A-D querying the storage drives 171 A-F for the location of control information for a particular storage drive 171 A-F. The storage drives 171 A-F may be configured to execute instructions that enable the storage drive 171 A-F to identify the location of the control information. The instructions may be executed by a controller (not shown) associated with or otherwise located on the storage drive 171 A-F and may cause the storage drive 171 A-F to scan a portion of each memory block to identify the memory blocks that store control information for the storage drives 171 A-F. The storage drives 171 A-F may respond by sending a response message to the storage array controller 110 A-D that includes the location of control information for the storage drive 171 A-F. Responsive to receiving the response message, storage array controllers 110 A-D may issue a request to read data stored at the address associated with the location of control information for the storage drives 171 A-F.

In other implementations, the storage array controllers 110 A-D may further offload device management responsibilities from storage drives 171 A-F by performing, in response to receiving the control information, a storage drive management operation. A storage drive management operation may include, for example, an operation that is typically performed by the storage drive 171 A-F (e.g., the controller (not shown) associated with a particular storage drive 171 A-F). A storage drive management operation may include, for example, ensuring that data is not written to failed memory blocks within the storage drive 171 A-F, ensuring that data is written to memory blocks within the storage drive 171 A-F in such a way that adequate wear leveling is achieved, and so forth.

In implementations, storage array 102 A-B may implement two or more storage array controllers 110 A-D. For example, storage array 102 A may include storage array controllers 110 A and storage array controllers 110 B. At a given instance, a single storage array controller 110 A-D (e.g., storage array controller 110 A) of a storage system 100 may be designated with primary status (also referred to as “primary controller” herein), and other storage array controllers 110 A-D (e.g., storage array controller 110 A) may be designated with secondary status (also referred to as “secondary controller” herein). The primary controller may have particular rights, such as permission to alter data in persistent storage resource 170 A-B (e.g., writing data to persistent storage resource 170 A-B). At least some of the rights of the primary controller may supersede the rights of the secondary controller. For instance, the secondary controller may not have permission to alter data in persistent storage resource 170 A-B when the primary controller has the right. The status of storage array controllers 110 A-D may change. For example, storage array controller 110 A may be designated with secondary status, and storage array controller 110 B may be designated with primary status.

In some implementations, a primary controller, such as storage array controller 110 A, may serve as the primary controller for one or more storage arrays 102 A-B, and a second controller, such as storage array controller 110 B, may serve as the secondary controller for the one or more storage arrays 102 A-B. For example, storage array controller 110 A may be the primary controller for storage array 102 A and storage array 102 B, and storage array controller 110 B may be the secondary controller for storage array

102 A and 102 B. In some implementations, storage array controllers

110 C and 110 D (also referred to as “storage processing modules”) may neither have primary or secondary status. Storage array controllers

110 C and 110 D, implemented as storage processing modules, may act as a communication interface between the primary and secondary controllers (e.g., storage array controllers

110 A and 110 B, respectively) and storage array 102 B. For example, storage array controller 110 A of storage array 102 A may send a write request, via SAN 158 , to storage array 102 B. The write request may be received by both storage array controllers

110 C and 110 D of storage array 102 B. Storage array controllers

110 C and 110 D facilitate the communication, e.g., send the write request to the appropriate storage drive 171 A-F. It may be noted that in some implementations storage processing modules may be used to increase the number of storage drives controlled by the primary and secondary controllers.

In implementations, storage array controllers 110 A-D are communicatively coupled, via a midplane (not shown), to one or more storage drives 171 A-F and to one or more NVRAM devices (not shown) that are included as part of a storage array 102 A-B. The storage array controllers 110 A-D may be coupled to the midplane via one or more data communication links and the midplane may be coupled to the storage drives 171 A-F and the NVRAM devices via one or more data communications links. The data communications links described herein are collectively illustrated by data communications links 108 A-D and may include a Peripheral Component Interconnect Express (‘PCIe’) bus, for example.

FIG. 1B illustrates an example system for data storage, in accordance with some implementations. Storage array controller 101 illustrated in FIG. 1B may similar to the storage array controllers 110 A-D described with respect to FIG. 1A . In one example, storage array controller 101 may be similar to storage array controller 110 A or storage array controller 110 B. Storage array controller 101 includes numerous elements for purposes of illustration rather than limitation. It may be noted that storage array controller 101 may include the same, more, or fewer elements configured in the same or different manner in other implementations. It may be noted that elements of FIG. 1A may be included below to help illustrate features of storage array controller 101 .

Storage array controller 101 may include one or more processing devices 104 and random access memory (‘RAM’) 111 . Processing device 104 (or controller 101 ) represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processing device 104 (or controller 101 ) may be a complex instruction set computing (‘CISC’) microprocessor, reduced instruction set computing (‘RISC’) microprocessor, very long instruction word (‘VLIW’) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. The processing device 104 (or controller 101 ) may also be one or more special-purpose processing devices such as an application specific integrated circuit (‘ASIC’), a field programmable gate array (‘FPGA’), a digital signal processor (‘DSP’), network processor, or the like.

The processing device 104 may be connected to the RAM 111 via a data communications link 106 , which may be embodied as a high speed memory bus such as a Double-Data Rate 4 (‘DDR4’) bus. Stored in RAM 111 is an operating system 112 . In some implementations, instructions 113 are stored in RAM 111 . Instructions 113 may include computer program instructions for performing operations in in a direct-mapped flash storage system. In one embodiment, a direct-mapped flash storage system is one that that addresses data blocks within flash drives directly and without an address translation performed by the storage controllers of the flash drives.

In implementations, storage array controller 101 includes one or more host bus adapters 103 A-C that are coupled to the processing device 104 via a data communications link 105 A-C. In implementations, host bus adapters 103 A-C may be computer hardware that connects a host system (e.g., the storage array controller) to other network and storage arrays. In some examples, host bus adapters 103 A-C may be a Fibre Channel adapter that enables the storage array controller 101 to connect to a SAN, an Ethernet adapter that enables the storage array controller 101 to connect to a LAN, or the like. Host bus adapters 103 A-C may be coupled to the processing device 104 via a data communications link 105 A-C such as, for example, a PCIe bus.

In implementations, storage array controller 101 may include a host bus adapter 114 that is coupled to an expander 115 . The expander 115 may be used to attach a host system to a larger number of storage drives. The expander 115 may, for example, be a SAS expander utilized to enable the host bus adapter 114 to attach to storage drives in an implementation where the host bus adapter 114 is embodied as a SAS controller.

In implementations, storage array controller 101 may include a switch 116 coupled to the processing device 104 via a data communications link 109 . The switch 116 may be a computer hardware device that can create multiple endpoints out of a single endpoint, thereby enabling multiple devices to share a single endpoint. The switch 116 may, for example, be a PCIe switch that is coupled to a PCIe bus (e.g., data communications link 109 ) and presents multiple PCIe connection points to the midplane.

In implementations, storage array controller 101 includes a data communications link 107 for coupling the storage array controller 101 to other storage array controllers. In some examples, data communications link 107 may be a QuickPath Interconnect (QPI) interconnect.

A traditional storage system that uses traditional flash drives may implement a process across the flash drives that are part of the traditional storage system. For example, a higher level process of the storage system may initiate and control a process across the flash drives. However, a flash drive of the traditional storage system may include its own storage controller that also performs the process. Thus, for the traditional storage system, a higher level process (e.g., initiated by the storage system) and a lower level process (e.g., initiated by a storage controller of the storage system) may both be performed.

To resolve various deficiencies of a traditional storage system, operations may be performed by higher level processes and not by the lower level processes. For example, the flash storage system may include flash drives that do not include storage controllers that provide the process. Thus, the operating system of the flash storage system itself may initiate and control the process. This may be accomplished by a direct-mapped flash storage system that addresses data blocks within the flash drives directly and without an address translation performed by the storage controllers of the flash drives.

The operating system of the flash storage system may identify and maintain a list of allocation units across multiple flash drives of the flash storage system. The allocation units may be entire erase blocks or multiple erase blocks. The operating system may maintain a map or address range that directly maps addresses to erase blocks of the flash drives of the flash storage system.

Direct mapping to the erase blocks of the flash drives may be used to rewrite data and erase data. For example, the operations may be performed on one or more allocation units that include a first data and a second data where the first data is to be retained and the second data is no longer being used by the flash storage system. The operating system may initiate the process to write the first data to new locations within other allocation units and erasing the second data and marking the allocation units as being available for use for subsequent data. Thus, the process may only be performed by the higher level operating system of the flash storage system without an additional lower level process being performed by controllers of the flash drives.

Advantages of the process being performed only by the operating system of the flash storage system include increased reliability of the flash drives of the flash storage system as unnecessary or redundant write operations are not being performed during the process. One possible point of novelty here is the concept of initiating and controlling the process at the operating system of the flash storage system. In addition, the process can be controlled by the operating system across multiple flash drives. This is contrast to the process being performed by a storage controller of a flash drive.

A storage system can consist of two storage array controllers that share a set of drives for failover purposes, or it could consist of a single storage array controller that provides a storage service that utilizes multiple drives, or it could consist of a distributed network of storage array controllers each with some number of drives or some amount of Flash storage where the storage array controllers in the network collaborate to provide a complete storage service and collaborate on various aspects of a storage service including storage allocation and garbage collection.

FIG. 1C illustrates a third example system 117 for data storage in accordance with some implementations. System 117 (also referred to as “storage system” herein) includes numerous elements for purposes of illustration rather than limitation. It may be noted that system 117 may include the same, more, or fewer elements configured in the same or different manner in other implementations.

In one embodiment, system 117 includes a dual Peripheral Component Interconnect (‘PCI’) flash storage device 118 with separately addressable fast write storage. System 117 may include a storage controller 119 . In one embodiment, storage controller 119 A-D may be a CPU, ASIC, FPGA, or any other circuitry that may implement control structures necessary according to the present disclosure. In one embodiment, system 117 includes flash memory devices (e.g., including flash memory devices 120 a - n ), operatively coupled to various channels of the storage device controller 119 . Flash memory devices 120 a - n , may be presented to the controller 119 A-D as an addressable collection of Flash pages, erase blocks, and/or control elements sufficient to allow the storage device controller 119 A-D to program and retrieve various aspects of the Flash. In one embodiment, storage device controller 119 A-D may perform operations on flash memory devices 120 a - n including storing and retrieving data content of pages, arranging and erasing any blocks, tracking statistics related to the use and reuse of Flash memory pages, erase blocks, and cells, tracking and predicting error codes and faults within the Flash memory, controlling voltage levels associated with programming and retrieving contents of Flash cells, etc.

In one embodiment, system 117 may include RAM 121 to store separately addressable fast-write data. In one embodiment, RAM 121 may be one or more separate discrete devices. In another embodiment, RAM 121 may be integrated into storage device controller 119 A-D or multiple storage device controllers. The RAM 121 may be utilized for other purposes as well, such as temporary program memory for a processing device (e.g., a CPU) in the storage device controller 119 .

In one embodiment, system 117 may include a stored energy device 122 , such as a rechargeable battery or a capacitor. Stored energy device 122 may store energy sufficient to power the storage device controller 119 , some amount of the RAM (e.g., RAM 121 ), and some amount of Flash memory (e.g., Flash memory 120 a - 120 n ) for sufficient time to write the contents of RAM to Flash memory. In one embodiment, storage device controller 119 A-D may write the contents of RAM to Flash Memory if the storage device controller detects loss of external power.

In one embodiment, system 117 includes two data communications links

123 a , 123 b . In one embodiment, data communications links

123 a , 123 b may be PCI interfaces. In another embodiment, data communications links

123 a , 123 b may be based on other communications standards (e.g., HyperTransport, InfiniBand, etc.). Data communications links

123 a , 123 b may be based on non-volatile memory express (‘NVMe’) or NVMe over fabrics (‘NVMf’) specifications that allow external connection to the storage device controller 119 A-D from other components in the storage system 117 . It should be noted that data communications links may be interchangeably referred to herein as PCI buses for convenience.

System 117 may also include an external power source (not shown), which may be provided over one or both data communications links

123 a , 123 b , or which may be provided separately. An alternative embodiment includes a separate Flash memory (not shown) dedicated for use in storing the content of RAM 121 . The storage device controller 119 A-D may present a logical device over a PCI bus which may include an addressable fast-write logical device, or a distinct part of the logical address space of the storage device 11

CLAIMS

Claims ( 18 )

What is claimed is:

1. A hyperscale artificial intelligence and machine learning infrastructure, the hyperscale artificial intelligence and machine learning infrastructure including:

a plurality of racks, wherein:

one or more of the racks include one or more GPU (‘Graphical Processor Unit’) servers;

one or more of the racks includes at least one storage system that separates network and data planes to expose one or more storage resources as a single namespace;

one or more of the racks include one or more switches coupled to at least one switch in another rack; and

the one or more GPU servers are configured to:

execute the one or more artificial intelligence or machine learning applications, using input data stored within the one or more storage systems that is accessed using the single namespace.

2. The infrastructure of claim 1 , wherein each of the storage systems comprises a fabric module configured to provision a software defined network for storage resources.

3. The infrastructure of claim 2 , wherein the fabric module of each of the storage systems presents a single namespace for the plurality of racks.

4. The infrastructure of claim 3 , wherein:

the plurality of racks is configured to maintain the single namespace upon coupling of an additional rack for scale out.

5. The infrastructure of claim 1 , wherein the switches and racks are configured in a leaf-spine network topology.

6. The infrastructure of claim 1 , wherein the switches and racks are configured in a torus network topology.

7. The infrastructure of claim 1 , wherein the switches and racks are configured in a hierarchical network topology.

8. The infrastructure of claim 1 , wherein at least one of the plurality of racks comprises:

one or more GPU servers, one or more storage systems, and one or more switches.

9. The infrastructure of claim 1 , wherein:

at least one of the plurality of racks comprises one or more GPU servers, one or more switches, and no storage systems; and

at least one other of the plurality of racks comprises one or more storage systems, one or more switches, and no GPU servers.

10. A method of performing artificial intelligence and machine learning processes in a hyperscale artificial intelligence and machine learning infrastructure, the infrastructure comprising:

a plurality of racks, wherein:

one or more of the racks include one or more GPU (‘Graphical Processor Unit’) servers;

one or more of the racks includes at least one storage system that separates network and data planes to expose one or more storage resources as a single namespace;

one or more of the racks include one or more switches coupled to at least one switch in another rack; and

wherein, the method further comprises executing, by the one or more GPU servers, one or more artificial intelligence or machine learning applications, including utilizing as input to the one or more artificial intelligence or machine learning applications, data stored within the one or more storage systems that is accessed using the single namespace.

11. The method of claim 10 , further comprising,

provisioning, by a fabric module of each of the storage systems, a software defined network for storage resources.

12. The method of claim 11 , further comprising presenting a single namespace for the plurality of racks.

13. The method of claim 12 , further comprising maintaining the single namespace upon coupling of an additional rack for scale out.

14. The method of claim 10 , wherein the switches and racks are configured in a leaf-spine network topology.

15. The method of claim 10 , wherein the switches and racks are configured in a torus network topology.

16. The method of claim 10 , wherein the switches and racks are configured in a hierarchical network topology.

17. The method of claim 10 , wherein at least one of the plurality of racks comprises:

one or more GPU servers, one or more storage systems, and one or more switches.

18. The method of claim 10 , wherein:

at least one of the plurality of racks comprises one or more GPU servers, one or more switches, and no storage systems; and

at least one other of the plurality of racks comprises one or more storage systems, one or more switches, and no GPU servers.

US16/365,648

2017-10-19

2019-03-26

Hyperscale artificial intelligence and machine learning infrastructure

Active

2040-05-09

US11494692B1

( en )

Priority Applications (3)

Application Number

Priority Date

Filing Date

Title

US16/365,648

US11494692B1

( en )

2018-03-26

2019-03-26

Hyperscale artificial intelligence and machine learning infrastructure

US17/979,467

US12067466B2

( en )

2017-10-19

2022-11-02

Artificial intelligence and machine learning hyperscale infrastructure

US18/809,094

US20240412113A1

( en )

2017-10-19

2024-08-19

Non-Disruptively Scaling Artificial Intelligence and Machine Learning Hyperscale Infrastructures

Applications Claiming Priority (5)

Application Number

Priority Date

Filing Date

Title

US201862648368P

2018-03-26

2018-03-26

US201862650736P

2018-03-30

2018-03-30

US16/045,814

US10360214B2

( en )

2017-10-19

2018-07-26

Ensuring reproducibility in an artificial intelligence infrastructure

US201962818326P

2019-03-14

2019-03-14

US16/365,648

US11494692B1

( en )

2018-03-26

2019-03-26

Hyperscale artificial intelligence and machine learning infrastructure

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/045,814

Continuation-In-Part

US10360214B2

( en )

2017-10-19

2018-07-26

Ensuring reproducibility in an artificial intelligence infrastructure

Related Child Applications (1)

Application Number

Title

Priority Date

Filing Date

US17/979,467

Continuation

US12067466B2

( en )

2017-10-19

2022-11-02

Artificial intelligence and machine learning hyperscale infrastructure

Publications (1)

Publication Number

Publication Date

US11494692B1

true

US11494692B1 ( en )

2022-11-08

Family

ID=83902496

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/365,648

Active

2040-05-09

US11494692B1

( en )

2017-10-19

2019-03-26

Hyperscale artificial intelligence and machine learning infrastructure

Country Status (1)

Country

Link

US

( 1 )

US11494692B1

( en )

Cited By (16)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20200257512A1

( en )

*

2019-02-07

2020-08-13

Microsoft Technology Licensing, Llc

Efficient scaling of a container-based application in a distributed computing system

US20210191762A1

( en )

*

2019-12-22

2021-06-24

Samsung Electronics Co., Ltd.

Method and apparatus for scaling resources of graphics processing unit in cloud computing system

US20220286498A1

( en )

*

2019-07-29

2022-09-08

Zte Corporation

Service request message sending method, distributed database architecture and computer readable storage medium

US20220318071A1

( en )

*

2019-12-19

2022-10-06

Huawei Technologies Co., Ltd.

Load balancing method and related device

US20220407770A1

( en )

*

2019-11-26

2022-12-22

Daikin Industries, Ltd.

Blockchain system, program and network connection apparatus

US20230093868A1

( en )

*

2021-09-22

2023-03-30

Ridgeline, Inc.

Mechanism for real-time identity resolution in a distributed system

CN116074179A

( en )

*

2023-03-06

2023-05-05

鹏城实验室

Highly scalable node system and training method based on CPU-NPU collaboration

US20230156083A1

( en )

*

2021-11-17

2023-05-18

Nutanix, Inc.

System and method for offloading preprocessing of machine learning data to remote storage

US11768636B2

( en )

2017-10-19

2023-09-26

Pure Storage, Inc.

Generating a transformed dataset for use by a machine learning model in an artificial intelligence infrastructure

US20230410090A1

( en )

*

2021-11-23

2023-12-21

Strong Force TX Portfolio 2018, LLC

Access layers with asset controls, data planes, and control planes

US11861423B1

( en )

2017-10-19

2024-01-02

Pure Storage, Inc.

Accelerating artificial intelligence (‘AI’) workflows

US20240113937A1

( en )

*

2019-06-04

2024-04-04

Telefonaktiebolaget Lm Ericsson (Publ)

Methods, Function Manager and Orchestration Node of Managing a Port Type

US12067466B2

( en )

2017-10-19

2024-08-20

Pure Storage, Inc.

Artificial intelligence and machine learning hyperscale infrastructure

US12373428B2

( en )

2017-10-19

2025-07-29

Pure Storage, Inc.

Machine learning models in an artificial intelligence infrastructure

US12493431B2

( en )

*

2022-11-29

2025-12-09

Samsung Electronics Co., Ltd.

Storage device and storage system for direct storage

US20260197234A1

( en )

*

2025-01-08

2026-07-09

MIPS Tech, LLC

Converged Scale-UP and Scale-Out Networks

Citations (224)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

EP0725324A2

( en )

1995-02-01

1996-08-07

Hewlett-Packard Company

Methods for avoiding overcommitment of virtual capacity in a redundant hierarchic data storage system

US5706210A

( en )

1995-03-01

1998-01-06

Fujitsu Limited

Network monitoring device

US5799200A

( en )

1995-09-28

1998-08-25

Emc Corporation

Power failure responsive apparatus and method having a shadow dram, a flash ROM, an auxiliary battery, and a controller

US5933598A

( en )

1996-07-17

1999-08-03

Digital Equipment Corporation

Method for sharing variable-grained memory of workstations by sending particular block including line and size of the block to exchange shared data structures

US6012032A

( en )

1995-11-30

2000-01-04

Electronic Data Systems Corporation

System and method for accounting of computer data storage utilization

US6085333A

( en )

1997-12-19

2000-07-04

Lsi Logic Corporation

Method and apparatus for synchronization of code in redundant controllers in a swappable environment

US20020013802A1

( en )

2000-07-26

2002-01-31

Toshiaki Mori

Resource allocation method and system for virtual computer system

US20020198921A1

( en )

2001-06-21

2002-12-26

Jameson Kevin Wade

Collection command applicator

US20030145172A1

( en )

2002-01-25

2003-07-31

International Business Machines Corporation

Method and system for updating data in a compressed read cache

US20030191783A1

( en )

2002-04-03

2003-10-09

Sun Microsystems, Inc., A Delaware Corporation

Fast lifetime analysis of objects in a garbage-collected system

US6643641B1

( en )

2000-04-27

2003-11-04

Russell Snyder

Web search engine with graphic snapshots

US6647514B1

( en )

2000-03-23

2003-11-11

Hewlett-Packard Development Company, L.P.

Host I/O performance and availability of a storage array during rebuild by prioritizing I/O request

US20030225961A1

( en )

2002-06-03

2003-12-04

James Chow

Flash memory management system and method

US20040080985A1

( en )

2002-10-28

2004-04-29

Sandisk Corporation, A Delaware Corporation

Maintaining erase counts in non-volatile storage systems

US20040111573A1

( en )

2002-12-06

2004-06-10

Garthwaite Alexander T.

Better placement of objects reachable from outside a generation managed by the train algorithm

US20040153844A1

( en )

2002-10-28

2004-08-05

Gautam Ghose

Failure analysis method and system for storage area networks

US6789162B1

( en )

2000-10-17

2004-09-07

Sun Microsystems, Inc.

Storage controller configured to select unused regions of a storage device for data storage according to head position

US20040187111A1

( en )

2003-02-21

2004-09-23

Eakin William Joseph

Content management portal and method for communicating media content

US20040193814A1

( en )

2003-03-27

2004-09-30

International Business Machines Corporation

Secure pointers

US20040260967A1

( en )

2003-06-05

2004-12-23

Copan Systems, Inc.

Method and apparatus for efficient fault-tolerant disk drive replacement in raid storage systems

US20050010565A1

( en )

2003-05-27

2005-01-13

David Cushing

System and method of transforming queries based upon E/R schema into multi-dimensional expression queries

US20050160416A1

( en )

2004-01-21

2005-07-21

International Business Machines Corporation

Method for determining a close approximate benefit of reducing memory footprint of a Java application

US20050188246A1

( en )

2004-02-25

2005-08-25

Emberty Robert G.

Persistent worldwide names assigned to removable media storage

US20050216800A1

( en )

2004-03-24

2005-09-29

Seagate Technology Llc

Deterministic preventive recovery from a predicted failure in a distributed storage system

US20060015771A1

( en )

2004-07-15

2006-01-19

International Business Machines Corporation

Management method for spare disk drives a RAID system

US7032089B1

( en )

2003-06-09

2006-04-18

Veritas Operating Corporation

Replica synchronization using copy-on-read technique

US20060129817A1

( en )

2004-12-15

2006-06-15

Borneman Christopher A

Systems and methods for enabling trust in a federated collaboration

US20060161726A1

( en )

2005-01-14

2006-07-20

M-Systems Flash Disk Pioneers Ltd.

Method for storing control information in flash memory devices

US7089272B1

( en )

2003-06-18

2006-08-08

Sun Microsystems, Inc.

Specializing write-barriers for objects in a garbage collected heap

US7107389B2

( en )

2002-08-29

2006-09-12

Matsushita Electric Industrial Co., Ltd.

Semiconductor memory device and method for writing data into flash memory

US20060230245A1

( en )

2005-04-08

2006-10-12

Microsoft Corporation

Data storage safety indicator and expander

US20060239075A1

( en )

2005-04-26

2006-10-26

Microsoft Corporation

NAND flash memory management

US7146521B1

( en )

2002-08-21

2006-12-05

3Pardata, Inc.

Preventing damage of storage devices and data loss in a data storage system

US20070022227A1

( en )

2005-07-22

2007-01-25

Nec Corporation

Path control device, system, cluster, cluster system, method and computer readable medium embodying program

US20070028068A1

( en )

2005-07-29

2007-02-01

International Business Machines Corporation

System and method for managing resources in a distributed storage system

US20070055702A1

( en )

2005-09-07

2007-03-08

Fridella Stephen A

Metadata offload for a file server cluster

US20070109856A1

( en )

2005-11-08

2007-05-17

Stmicroelectronics S.R.I

Method of managing fails in a non-volatile memory device and relative memory device

US20070150689A1

( en )

2005-12-22

2007-06-28

Pandit Anil K

Effective wear-leveling and concurrent reclamation method for embedded linear flash file systems

US20070168321A1

( en )

2004-05-07

2007-07-19

Hiroshi Saito

Data accessing method, device in which the data accessing method is executed and recording medium

US20070203937A1

( en )

2005-11-28

2007-08-30

Anand Prahlad

Systems and methods for classifying and transferring information in a storage network

US20070220227A1

( en )

2006-03-17

2007-09-20

Emc Corporation

Techniques for managing data within a data storage system utilizing a flash-based memory vault

US20070294564A1

( en )

2006-04-27

2007-12-20

Tim Reddin

High availability storage system

US20070294563A1

( en )

2006-05-03

2007-12-20

Patrick Glen Bose

Method and system to provide high availability of shared data

US20080005587A1

( en )

2006-06-30

2008-01-03

Ahlquist Brent M

Accelerating integrity checks of code and data stored in non-volatile memory

US7334124B2

( en )

2002-07-22

2008-02-19

Vormetric, Inc.

Logical access block processing protocol for transparent secure file storage

US20080077825A1

( en )

2006-09-26

2008-03-27

Keith Anthony Bello

Tracking error events relating to data storage drives and/or media of automated data storage library subsystems

US20080117220A1

( en )

2006-09-25

2008-05-22

Neurala Llc

Graphic Processor Based Accelerator System and Method

US20080133568A1

( en )

2006-11-30

2008-06-05

Cognos Incorporated

Generation of a multidimensional dataset from an associative database

US20080162674A1

( en )

2006-12-27

2008-07-03

Hewlett-Packard Development Company, L.P.

System and method for hot deployment/redeployment in grid computing environment

US20080195833A1

( en )

2007-02-13

2008-08-14

Samsung Electronics Co., Ltd.

Systems, methods and computer program products for operating a data processing system in which a file system&#39;s unit of memory allocation is coordinated with a storage system&#39;s read/write operation unit

US7437530B1

( en )

2003-04-24

2008-10-14

Network Appliance, Inc.

System and method for mapping file block numbers to logical block addresses

US20080270678A1

( en )

2007-04-25

2008-10-30

Cornwell Michael J

Command resequencing in memory operations

US20080282045A1

( en )

2007-05-09

2008-11-13

Sudeep Biswas

Garbage collection in storage devices based on flash memories

US7493424B1

( en )

2004-04-30

2009-02-17

Netapp, Inc.

Network storage system with shared software stack for LDMA and RDMA

US20090077340A1

( en )

2007-09-14

2009-03-19

Johnson Randy S

Storage area network (san) forecasting in a heterogeneous environment

US20090100115A1

( en )

2007-10-12

2009-04-16

Samsung Electronics Co., Ltd.

Methods and Apparatus for File Management Using Partitioned File Metadata

US20090198889A1

( en )

2008-02-05

2009-08-06

Sony Corporation

Recording apparatus, recording method, program for recording method, and storage medium that stores program for recording method

US7606813B1

( en )

*

2006-09-27

2009-10-20

Emc Corporation

Model consolidation in a database schema

US7669029B1

( en )

2006-11-15

2010-02-23

Network Appliance, Inc.

Load balancing a data storage system

US20100052625A1

( en )

2008-09-04

2010-03-04

International Business Machines Corporation

In Situ Verification of Capacitive Power Support

US7689609B2

( en )

2005-04-25

2010-03-30

Netapp, Inc.

Architecture for supporting sparse volumes

US7743003B1

( en )

2007-05-16

2010-06-22

Google Inc.

Scaling machine learning using approximate counting that uses feature hashing

US7743191B1

( en )

2007-12-20

2010-06-22

Pmc-Sierra, Inc.

On-chip shared memory based device architecture

US20100211723A1

( en )

2009-02-19

2010-08-19

Tdk Corporation

Memory controller, memory system with memory controller, and method of controlling flash memory

US20100246266A1

( en )

2009-03-25

2010-09-30

Samsung Electronics Co., Ltd.

Nonvolatile memory device and related programming method

US20100257142A1

( en )

2009-04-03

2010-10-07

Microsoft Corporation

Differential file and system restores from peers and the cloud

US20100262764A1

( en )

2009-04-14

2010-10-14

Chao-Yin Liu

Method for accessing storage apparatus and related control circuit

US20100325345A1

( en )

2009-06-22

2010-12-23

Hitachi, Ltd.

Method for managing storage system using flash memory, and computer

US20100332754A1

( en )

2003-04-09

2010-12-30

Corel Inc.

System and Method for Caching Multimedia Data

US7899780B1

( en )

2006-03-30

2011-03-01

Emc Corporation

Methods and apparatus for structured partitioning of management information

US20110072290A1

( en )

2009-09-24

2011-03-24

Xyratex Technology Limited

Auxiliary power supply, a method of providing power to a data storage system and a back-up power supply charging circuit

US20110125955A1

( en )

2009-11-25

2011-05-26

Silicon Motion, Inc.

Flash storage device, data storage system, and data writing method

US20110131231A1

( en )

2009-11-30

2011-06-02

International Business Machines Corporation

Method to efficiently locate meta-data structures on a flash-based storage device

US20110167221A1

( en )

2010-01-06

2011-07-07

Gururaj Pangal

System and method for efficiently creating off-site data volume back-ups

US8042163B1

( en )

2004-05-20

2011-10-18

Symatec Operating Corporation

Secure storage access using third party capability tokens

US8086585B1

( en )

2008-09-30

2011-12-27

Emc Corporation

Access control to block storage devices for a shared disk based file system

US20120023144A1

( en )

2010-07-21

2012-01-26

Seagate Technology Llc

Managing Wear in Flash Memory

US20120054264A1

( en )

2010-08-31

2012-03-01

International Business Machines Corporation

Techniques for Migrating Active I/O Connections with Migrating Servers and Clients

US20120079318A1

( en )

2010-09-28

2012-03-29

John Colgrove

Adaptive raid for an ssd environment

US20120079175A1

( en )

2010-09-28

2012-03-29

Fusion-Io, Inc.

Apparatus, system, and method for data transformations within a data storage device

US20120131253A1

( en )

2010-11-18

2012-05-24

Mcknight Thomas P

Pcie nvram card based on nvdimm

US8200887B2

( en )

2007-03-29

2012-06-12

Violin Memory, Inc.

Memory management system and method

WO2012087648A2

( en )

2010-12-22

2012-06-28

Intel Corporation

Method and apparatus to provide a high availability solid state drive

US20120172088A1

( en )

2010-12-31

2012-07-05

Motorola-Mobility, Inc.

Method and System for Adapting Mobile Device to Accommodate External Display

US8271700B1

( en )

2007-11-23

2012-09-18

Pmc-Sierra Us, Inc.

Logical address direct memory access with multiple concurrent physical ports and internal switching

US20120303919A1

( en )

2009-07-29

2012-11-29

International Business Machines Corporation

Write-erase endurance lifetime of memory storage devices

US20120311000A1

( en )

2011-06-03

2012-12-06

Apple Inc.

Pre-organization of Data

US20130007845A1

( en )

2011-06-30

2013-01-03

International Business Machines Corporation

Authentication and authorization methods for cloud computing security platform

US20130006976A1

( en )

2011-07-01

2013-01-03

International Business Machines

Search tool that utilizes scientific metadata matched against user-entered parameters

US20130031414A1

( en )

2011-07-27

2013-01-31

Cleversafe, Inc.

Identifying an error cause within a dispersed storage network

US20130036272A1

( en )

2011-08-02

2013-02-07

Microsoft Corporation

Storage engine node for cloud-based storage

US8387136B2

( en )

2010-01-05

2013-02-26

Red Hat, Inc.

Role-based access control utilizing token profiles

US20130054716A1

( en )

2011-07-25

2013-02-28

Stuart J. Levinson

System and method for abstract communication

US20130071087A1

( en )

2011-09-15

2013-03-21

Google Inc.

Video management system

US8437189B1

( en )

2006-12-20

2013-05-07

Marvell International Ltd.

Method and apparatus for storing data in a NAND flash memory

WO2013071087A1

( en )

2011-11-09

2013-05-16

Unisys Corporation

Single sign on for cloud

US20130145447A1

( en )

2011-12-01

2013-06-06

Dashlane SAS

Cloud-based data backup and sync with secure local storage of access keys

US20130145172A1

( en )

2011-12-06

2013-06-06

Wwpass Corporation

Token activation

US8465332B2

( en )

2011-01-13

2013-06-18

Tyco Electronics Corporation

Contact assembly for an electrical connector

US20130191555A1

( en )

2012-01-19

2013-07-25

Peter Chi-Hsiung Liu

Intelligent storage controller

US20130198459A1

( en )

2012-01-27

2013-08-01

Fusion-Io, Inc.

Systems and methods for a de-duplication cache

US20130205173A1

( en )

2012-02-03

2013-08-08

Fujitsu Limited

Storage device, and storage device control method

US20130219164A1

( en )

2011-12-29

2013-08-22

Imation Corp.

Cloud-based hardware security modules

US20130227201A1

( en )

2010-12-13

2013-08-29

Fusion-Io, Inc.

Apparatus, System, and Method for Accessing Auto-Commit Memory

US8527544B1

( en )

2011-08-11

2013-09-03

Pure Storage Inc.

Garbage collection in a storage system

US8566546B1

( en )

2010-09-27

2013-10-22

Emc Corporation

Techniques for enforcing capacity restrictions of an allocation policy

US20130290607A1

( en )

2012-04-30

2013-10-31

Jichuan Chang

Storing cache metadata separately from integrated circuit containing cache controller

Related documents

• Generating a transformed dataset for use by a machine learning model in an … — Pure Storage, Inc. (US11768636B2)#607043
• Testing nonstabilizerness only with stabilizer states#1142563
• MemExchange: Utility-Driven Distributed Memory Reallocation for Multi-Tenant Datacenters#366235
• 42 U.S.C. § 17085 — Tenant Star program#588778
• Protocol to Access White-Space (PAWS) Databases#227156