This article attempts to provide a basis for assessing the fitness of computer platforms for desktops and servers in terms of storage.  A desirable pre-requisite article to read is “Computing Subsystem Performance” for a fuller context.  This article focuses on internal storage connectivity.  Technologies discussed are SAS2, SAS3, RAID, and PCIe.  Storage devices discussed include SSD and HDD.

SAS (Serial Attached SCSI)

o SAS2 or 6G/s SAS has been around in the market since 2010 and SAS3 or 12G/s SAS will be introduced in late 2013 or early 2014 timeframe.  Do we need to care about these standards?  Yes in case storage performance is critical for the application.

o SAS2 came in at a time when SSD started receiving public attention.  Assuming SAS2 has a bandwidth of 600MB/s, it is not useful at all for most HDD which achieved a maximum read rate of around 100MB/s.  HDD did not hit the wall in a SAS1 pipe (half of SAS2), and therefore the widening of the pipe did not allow HDD to perform higher or allowed a marginal performance gain of HDD at most.  However SSD achieved a higher maximum read rate than HDD that is close to SAS1 pipe width.  The widening of SAS to 600MB/s saw SSD achieving 560MB/s of continuously read rate.  Will another doubling of SAS pipe facilitate another doubling of SSD read rate?  

PCIe (Peripheral Connect Interface Express)

o Some SSD vendors have been very ambitious in breaking the SAS2 limit by aggregating multiple SSD onto a PCIe card.  PCIe v2 has a bandwidth of 500MB/s per lane or 4GB/s for 8 lanes aggregate.  SSD vendors have borrowed this aggregation idea to aggregate multiple SSD units together.  A read rate of 3GB/s has been published for products with an interface of PCIe v2x8. 

o Bandwidth aggregation is not new for HDD in RAID arrangements.  RAID controllers are able to put 3, 4, 5 or more low cost SATA HDD units together and fill up a SAS channel.  In this respect, RAID is a tool for reducing the cost of achieving high read/write performance beyond the fundamental purposes of providing data security and integrity. 

o To understand how RAID vendors maximize their opportunities, see this example.  Compucon 4UD72 big data system (72 x 4TB = 288TB) can be configured with 3 RAID control cards each with 8 SAS2 ports (8 x 600MB/s) on PCIe v2 x8 (4GB/s) for a total of 20 SAS2 channels (20 x 600MB/s = 12GB/s).  This is an extreme case of one 4U PC providing 288TB of storage with 12GB/s of theoretical max read output.  

Summary

PCIe v3 has been around for a couple of years and we expect PCIe SSD and RAID vendors to be more aggressive in this respect as soon as SAS3 is released.

SAS3 = 1.2GB/s
PCIe v3 = 1GB/s per lane

END

This article attempts to provide the basis for assessing the fitness of computer platforms for desktops and servers.  The discussions refer to Von Neumann system architecture as all other architectures do not have a similar level of market dominance.

Von Neumann Stored Program Architecture

o The architecture was first known in 1945 and has slightly evolved to mean a stored program computer in which an instruction fetch and a data operation cannot occur at the same time because they share a common bus. It keeps its programmed instructions and data in RAM (random access memory).  Modern processors include separate instruction cache and data cache on the CPU die, but they are still considered Von Neumann in this article.  The original Harvard Architecture was meant to have separate bus and spaces for instructions and data, and the modified version applies separation at the cache level and not main memory.  As such, there is a universal concern on the single bus arrangement creating a performance bottleneck. 

Connectivity of Modern x86 Systems

o Modern desktop processors such as Intel Core and AMD APU have incorporated the GPU, memory controller, display controller, and PCIe controller inside the CPU die.  They are close in design to System-on-Chip (SOC) such as Intel Atom Centerton which includes the above lot and other I/O controllers on die.  Intel Core and AMD APU employ a south bridge to handle low speed I/O device control.  Note: The GPU mentioned above is normally for display purposes and not in the class that Nvidia touted for high performance general purpose computing.

o In terms of connectivity, the CPU interfaces with memory and display devices through dedicated controllers, and with other devices through the PCIe controller.  Display devices go through the PCIe controller as well for some modern CPU designs.  As such, the memory controller and PCIe controller have crucial roles in deciding the performance of a computer.

o Compucon system platform design philosophy carries the caution that a system is as strong as its weakest link.  In reality, not all applications have the same appetite for different resources of a computer system and this “as strong as weak link” concept shall be modified to adapt.

Bottleneck of Computing Sub-system

o Is the CPU continually forced to wait for needed data to be transferred to or from memory?  One answer comes from the Arithmetic Intensity (see separate article) required by the application relative to the CPU design.  AI is the ratio of CPU processing rate to memory data transfer rate. 

o The current memory type is DDR3.  It is known to have a high latency (lead time before data transfer takes place) and current efforts for DDR4 development are to reduce the latency.  Intel E5 Xeon and Core 2011p i7 processors have 4 memory controllers- Intel website has quoted a memory transfer rate of 51GB/s for DDR3.  We guess the same processors may hit 100GB/s if DDR4 is used.

END

This article attempts to provide the basis for assessing the fitness of computer platforms for desktops and servers.  The discussions refer to Von Neumann system architecture as all other architectures do not have a similar level of market dominance.

Von Neumann Stored Program Architecture

o The architecture was first known in 1945 and has slightly evolved to mean a stored program computer in which an instruction fetch and a data operation cannot occur at the same time because they share a common bus. It keeps its programmed instructions and data in RAM (random access memory).  Modern processors include separate instruction cache and data cache on the CPU die, but they are still considered Von Neumann in this article.  The original Harvard Architecture was meant to have separate bus and spaces for instructions and data, and the modified version applies separation at the cache level and not main memory.  As such, there is a universal concern on the single bus arrangement creating a performance bottleneck. 

Connectivity of Modern x86 Systems

o Modern desktop processors such as Intel Core and AMD APU have incorporated the GPU, memory controller, display controller, and PCIe controller inside the CPU die.  They are close in design to System-on-Chip (SOC) such as Intel Atom Centerton which includes the above lot and other I/O controllers on die.  Intel Core and AMD APU employ a south bridge to handle low speed I/O device control.  Note: The GPU mentioned above is normally for display purposes and not in the class that Nvidia touted for high performance general purpose computing.

o In terms of connectivity, the CPU interfaces with memory and display devices through dedicated controllers, and with other devices through the PCIe controller.  Display devices go through the PCIe controller as well for some modern CPU designs.  As such, the memory controller and PCIe controller have crucial roles in deciding the performance of a computer.

o Compucon system platform design philosophy carries the caution that a system is as strong as its weakest link.  In reality, not all applications have the same appetite for different resources of a computer system and this “as strong as weak link” concept shall be modified to adapt.

Bottleneck of Computing Sub-system

o Is the CPU continually forced to wait for needed data to be transferred to or from memory?  One answer comes from the Arithmetic Intensity (see separate article) required by the application relative to the CPU design.  AI is the ratio of CPU processing rate to memory data transfer rate. 

o The current memory type is DDR3.  It is known to have a high latency (lead time before data transfer takes place) and current efforts for DDR4 development are to reduce the latency.  Intel E5 Xeon and Core 2011p i7 processors have 4 memory controllers- Intel website has quoted a memory transfer rate of 51GB/s for DDR3.  We guess the same processors may hit 100GB/s if DDR4 is used.

END