High-End LSI controllers and the effect of large caches on I/O

On all of my Poweredge machines, I have a 6Gbps 2Tb Samsung EVO 870 as boot drive. They are configured as a single LUN RAID-0. Here's an example with my T140 (sporting an H740P PERC):

# megaclisas-status

-- Controller information --
-- ID | H/W Model          | RAM    | Temp | BBU    | Firmware
c0    | PERC H740P Adapter | 8192MB | 54C  | Good   | 50.5.1-2818

-- Array information --
-- ID | Type   |    Size |  Strpsz |   Flags | DskCache |   Status |  OS Path | CacheCade |InProgress
c0u0  | RAID-0 |   1818G |   512KB | ADRA,WB |  Enabled |  Optimal | /dev/sda | None      |None
[....]

-- Disk information --
-- ID   | Type | Drive Model                                      | Size     | Status          | Speed    | Temp | Slot ID  | LSI ID
c0u0p0  | SSD  | S620NG0R303366N Samsung SSD 870 EVO 2TB SVT02B6Q | 1.818TB  | Online, Spun Up | 6.0Gb/s  | 38C  | [:0]     | 0
[....]

These drives typically max out unbuffered I/O at around 550MB/s, yet on a freshly booted idle machine, I was getting higher readings by running hdparm repetitively.

# hdparm -tT /dev/sda
/dev/sda:
Timing cached reads:   19358 MB in  2.00 seconds = 9696.52 MB/sec
Timing buffered disk reads: 2894 MB in  3.00 seconds = 963.91 MB/sec

# hdparm -tT /dev/sda
/dev/sda:
Timing cached reads:   18772 MB in  2.00 seconds = 9403.51 MB/sec
Timing buffered disk reads: 1588 MB in  3.00 seconds = 529.11 MB/sec

# hdparm -tT /dev/sda
/dev/sda:
Timing cached reads:   18702 MB in  2.00 seconds = 9365.57 MB/sec
Timing buffered disk reads: 2950 MB in  3.00 seconds = 983.32 MB/sec

# hdparm -tT /dev/sda
/dev/sda:
Timing cached reads:   19044 MB in  2.00 seconds = 9536.41 MB/sec
Timing buffered disk reads: 4130 MB in  3.00 seconds = 1376.28 MB/sec

# hdparm -tT /dev/sda
/dev/sda:
Timing cached reads:   18786 MB in  2.00 seconds = 9408.40 MB/sec
Timing buffered disk reads: 5140 MB in  3.00 seconds = 1712.92 MB/sec

# hdparm -tT /dev/sda
/dev/sda:
Timing cached reads:   19914 MB in  2.00 seconds = 9974.09 MB/sec
Timing buffered disk reads: 1586 MB in  3.00 seconds = 528.66 MB/sec

Where did the higher numbers come from? hdparm does unbuffered I/O and bypasses the VFS cache.

I believe that this is the PERC Controller Cache at work.

The machine used in this test run has a PERC H750P with an 8GB cache.

With unbuffered I/O I never saw anything above 550 MB/sec on a Samsung EVO 870 but here it's the larger cache doing its job.

Such things are why I tend to favor RAID-0 LUNs, even for ZFS or Ceph.

Sure, 8G of cache is minimal compared to the size of modern disks, but in the case of I/O hotspots, a beefy cache can help.