Introduction

General informations

HOWTO content

This HOWTO describes the (networked) storage performance measurements methodology and tips.
As the members of TERENA's TF-Storage we try to collect general informations HOW TO
prepare and perform the storage performance testing, basing on our own experience and
knowledge as well as the material accessible in the Internet.

This HOWTO re-uses the presentation prepared by IBBT/Ghent University,
Storage Benchmarking Cookbook.

HOWTO limitations

Obviously, the world is too complicated to be described in a single work.
However, we hope that both background and practical, hands-on informations included in this HOWTO
can be useful for the community.
HOWTO does not cover all possible aspects, problems and approaches to the
storage benchmarking process. We try to describe the most important issues,
most popular approaches and tools used to evaulate the storage systems.

HOWTO organisation

The organisation of the sections of this HOWTO follows the real-world, networked storage systems layers layout (see table below).

We provide general and layer-by-layer comments on how to perform storage benchmarking
along with some background information, practical benchmarking tips and examples
as well as links to the external sources.

System layers

Typical storage environment is composed of layers similar to presented above.
It is important to understand, that each of these layers can have important influence of the data access performance.

In following subsections we shortly discuss the layers and mention the main factors,
that have the most important impact on the efficiency of the whole storage system.

Clients and applications layer

Client applications run in the client hosts. These hosts are the NAS clients. NAS services include file system-level
services, basic user authentication and access control.

Following application details may influence the data access performance:

• data access pattern,
• application-level block size,
• data caching,
• another storage and data virtualisation technologies implemented by the application.

Also the hardware and software components of the client systems such as:

• NIC's (Network Interface Cards) or HBA (Host Bus Adapters) their device drivers and settings, including transport protocol parameters,
• CPUs, RAM, PCI buses etc. along with their features and configuration settings,
• Operating system and their devices drivers, configuration and settings
  impact the observed storage performance.

Therefore, the networked storage benchmarking procedure should include examining the features
of the hardware and software components of the client system.

Note, that client hosts can use block-level services provided by the storage system: DAS (Directly Attached Storage) or SAN (Storage Area Network) instead of NAS services. This case is discussed in details in one of
the following sections.

Client network

Client network interconnects the clients and the filesystem cluster, called also 'NAS cluster'.

The efficiency of data access performed by the client network may depend on:

• network topology (e.g. loop, switch, multiple fabric),
• network technology (e.g. Gigabit Ethernet, 10 Gigabit Ethernet or Infiniband in HPC clusters),
• transmission protocol (e.g. IP) and protocol parameters (such as MTU, buffer sizes),
• network delay.

Most popular client network is the IP connectivity implemented over Gigabit Ethernet.
In the HPC clusters IP over Inifinband is often used. Concerning IB networks, some
solutions to encapsulate high-level NAS protocol traffic directly in the IB frames exist, e.g.
NFS over RDMA.

As the client-NAS server network can introduce data transmission delays and
significant performance limits, its features must be carefull examined
in the process of benchmarking the networked storage solution.

Filesystem and cluster nodes... and block-level storage clients

Filesystem and cluster nodes implement the NAS (Network Attached Storage) functionality,
including filesystem services, users authentication and data access control.

NAS services implementation

Physically, there are two most typical ways of implementing the NAS services:

• the NAS gateway that implement the file-level data services over the block-level data services provided by the external block storage box; the gateway itself can be build as:
• dedicated, specialised NAS gateway appliance,
• software NAS gateway run on the typical operating system,
• the complete NAS appliance that includes NAS engine and the block-level storage modules.
  Nevertheless the physical implementation, the logic remains similar.

The physical and logical characteristics of the implementation of NAS services
have very important impact on the performance of the networked storage.
Therefore, their features must be carefully tested in the confines of the networked storage bemchmarking evaluation.

The NAS elements may consitute a bottle-neck, among others due to:

• a lack of scalability of data processing performance,
• overhead of the file-level services compared to block-level storagem
• specifics of the NAS gateways hardware and software including:
• file system settings: caching, journaling, prefetching, redundancy,
• operating systems limits on NAS gateways,
• transport protocol parameters in both client and storage front-end network.

Block-level storage clients

Note, that while being a NAS servers, NAS gateways act as the block-level services clients in the same time. Block-level services are provided by storage controllers accesible through storage front-end network, e.g.
a FC-based SAN.

The performance of file system-level services strongly depends on the performance
of the block-level volumes. Therefore, the evaluation of the performance of the
networked storage environment should include examining the block storage components.
This is possible in open systems, where we can have access to the block-level storage directly.

In typical networked storage setups also the application/client systems can be connected to the storage network directly, exploiting the block-level storage. In such case, storage benchmarking can be focused
on the performance evaluation of SAN compoments, SAN network interfaces (so called HBAs)
(Host Bus Adapters) and SAN driver characteristics.

Storage front-end connectivity (SAN or DAS)

The block-level storage can be served directly, locally (DAS - Direct Attached Storage) or by the Storage Area Network. In both cases, the performance evaluation of the block-level storage can be performed using similar methods and tools, as the virtualisation mechanismes implemented in modern operating systems hide the fact that the SAN storage is not local.

SAN network may constitute a performance bootleneck, depending on:

• network topology and technology,
• links speed,
• latency,
• multipathing support (active-active vs active-passive).

Beside the FC technology, iSCSI and FCIP, iFCP and SRP (over IB) protocols are used to implement the data transport in the front-end storage networks.

Storage controllers and RAID structures

Storage controllers implement the basic virtualisation functions such as RAID structures, volumes (LUNs) and volume
mapping to hosts and volumes access control. They are crucial elements of the storage system, so their features may determine its performance characteristics.

Advanced functions of modern controllers can have the significant impact on the data access efficiency.
They should be taken into account while planning the storage controllers testing. Examples are:

• hardware-aid RAID implementation,
• front-end ports load balancing,
• advanced data caching:
• reads: read-ahead caching, sequential access pattern detection,
• writes: write-back and write-through caching modes and RAID stripe writes optimisation,
• LUN-in-cache functionality.

Solid benchmarking of the storage controllers should include (but should not be limited to):

• sustained mode operation testing,
• cached mode operation (client-cache efficiency)
• evaluation of the influence of the processes that take place in the RAID system (e.g sparing the failed disks, snapshots management etc.) on the performance observed by the clients.

RAID structures are the basic virtualisation technique used in storage controllers. The techniques and tricks used to optimise particular RAID implementations can be a differentiating factor of various storage controllers available on the market.
Therefore, controller settings that relate to RAID structures should be carefully examined in the confines
of storage system testing. Possible configuration settings include:

• RAID level,
• RAID chunk and stripe size,
• read-ahead and write cache operation modes.

Also the logical volumes can be configured in different ways. The settings that should be taken into account while performing the storage testing are:

• LUN distribution over RAID structures,
• LUN-specific cache setting,
• support for the access to the volume through multiple paths.

Particular storage controllers may include advanced configuration parameters, that can be used
for performance tunnig. Examples of such parameters are:

• cache mirroring and cache coherency settings,
• active-active vs active-passive controllers operation,
• and many others.

Significant amount of time should be spent in order to get a realistic picture
of a given controller performance characteristics.

Storage back-end connectivity

Back-end storage network interconnects the storage controllers with he disks drives.
Various type of the technologies are used for that purpose. Traditionally, Fibre Channel
loops span disk drives and storage controller back-end ports. In enterprise solutions,
switched back-end topologies are used. Currently, SAS technology is often used
for the back-end connectivity as it provides high bandwidth at optimal cost.

The storage back-end links can be a bootle-neck, especially in a sustained transfer
operations. Therefore its characteristics such as latency and bandwidth should be taken into account while
running the performance tests.

Disk drives

As the disk drives hold the actual data, their characteristics are very important
component of storage system performance.

The efficiency of accessing data held on disk drives may depend on:

• drives technology (FC, SATA/FATA, SSD),
• disk rotational speed (typically 7,2krpm, 10krpm. 145krpm),
• internal cache size and efficiency,
• additional techniques that can be implemented in the drives (e.g. NCQ - Native Command Queueing, Tagged Command Queueing),
• disk segment size,
• physical layout/placement of the data on the disk cylinders, e.g. outer sectors - higher throughput, middle sectors - lower seek latency.

As the data processed in storage systems ultimately goes to disk drives, it might be useful to examine the performance of the individual disk drives (if possible), in order to determine the realistic performance that can be achieved after combining multiple disks into RAID arrays.

Storage system layers summary

In above sections we showed that networked storage systems are typically composed of multiple layers.
Each of this layers has its own features, complexity and performance characteristics.
Solid storage benchmarking should take into account these facts. Appropriate testing metodology
should be used in order to reveal the real characteristics of particular components
or to find a source of the actual or potential bottle-neck observed in the system.

Overview of the benchmarking methodologies is provided in the next section of the HOTWO.

Storage Benchmarking methodologies overview

The complexity of the networked storage system makes solid storage benchmarking a difficult task.
One of the basic decisions that must be taken while preparing the benchmarking procedure
is the PURPOSE of running the benchmark and the EVALUATION CRITERIA for the test results.

Benchmarking purposes and evaluation criteria

There are many possible benchmarking purposes. Here are some examples of them:

• evaluation of the actual storage system efficiency (without any expectations formulated a priori):
• in a default setup,
• in a tunned configuration,
• determining the source of the observed bottle-neck (makes sense especially if the performance is lower than expected),
• tunning the storage system for a given application.

In order to evaluate the benchmark results, the EVALUATION CRITERIA must be set up.
The criteria can differ significantly, depending on the TARGET APPLICATION of the storage system.

For instance, when considering home directories storage, both sequential and random access pattern performance
are important. For high-performance computing, rich-media and backup/archive applications,
the sequental data access performance matters the most. Databases storage requires good
IOPS characteristics of the storage system.

Benchmarking procedure and tools should be able to reveal the characteristics of the examined
storage system that matches the defined criteria.

Benchmarking tactics and organisation

Depending on the purpose, the benchmarking process should be organised in different ways.

For instance, the evaluation of the storage system efficiency in a tunned configuration
require running multiple rounds of tests, while changing configuration parameters on single
or multiple storage system layers beetween rounds and doing the analysis phase after each test.

While planning the benchmarking process we shoul try to:

• plan the correct sequence of the benchmarking actions, e.g.:
• work bottom-up to know the efficiency of each system layer and the overhead it introduces comparing to lower layers;
• work top-down to determine see the limit on the highest level and try to find its source on the lower layers.
• choose the appropriate set of the examined configuration parameters and determine the set of the values used for them,
• eliminate the influence of unwanted optimisation mechanisms (in case we want it), e.g.:
• storage controller-side caching,
• NAS gateway filesystem-side caching,
• client-level, filesystem-side and application-side caching
• make sure, that the benchmarking load reflects the real load of the target applications and the target environment,

Benchmarking tactics: bottom up or top-down?

Both approaches has some advantages and disadvantages. The accepted tactics should fit the purpose
of the test.
If we want to quickly determine the overall performance of the storage system, we can
start from the top, e.g. by running a file-system level benchmark (more on benchmarks in the BENCHMARKING
TOOLS section), and go down in case the performance is lower than expected. This is most probably less
time consuming approach, however we may get an incomplete picture of the actual system efficiency.
Going bottom-up is more costly, but gives as the knowledge about the efficiency of each system layer and the overhead it introduces comparing to lower layers. This approach is suitable if we want to find a bottle-neck in the
system or we want to determine the realistic maximum performance in a tunned setup.

Benchmarking parameters selection

Having in mind the complexity of multi-layer storage system, we may be forced to:

• use heuristics,
• trust our experience and intuition
  in order to determine (and limit!!) the list of the configuration parameters modified
  during the benchmarking process and their examined values. Choosing the correct parameters
  and their values is a difficult task and can be performed basing on the deep knowledge
  of the storage systems, networks and computers systems architectures and characteristics.

Measuring the correct bottle-neck

Benchmarking the multi-layered system we have to make sure that we are measuring the correct
component of the system.

Data caching influence

Typical issue faced while benchmarking on the file-system level is the influence of the data
caching which can take place on the client side (file system-level, operating system-level), NAS gateway side,
or on the storage controller.

In order to measure the selected storage system element, e.g. the storage backend network bandwidth,
we may want to disable caching or plan the benchmark in the way, where the influence of the caching
will be eliminated.

Some practical methods to achieve that are:

• using test files big enough to not fit into the cache memory,
• using different files between multiple test rounds,
• make sure you do not read the file that was recently written (if not big enough, it can still reside in the cache),
• force filesystem sync'ing between test rounds (flushing the data from the previous round that still residing in cache can disturb the current round of tests),
• allocating the buffer cache before running the benchmark (on the client size),
• restart NFS servers, filesystem deamons, or even client machines and storage controllers between the test rounds,
• re-mount the filesystems,
• use flags that force the direct I/O processing while mounting the filesystem.
• use the benchmark that support direct I/O.
• use block-level benchmarks instead of file-level.

Another side effect of the data caching is that, the speed of writing the the buffer
can be measured instead of the actual tranmission speed. This can be avoided by using the data set large enought to fill up the buffers
and transmit enough data over the real communication channels. We may also use external monitoring tools,
such as Gigabit Ethernet network or SAN switches performance analysers, host ports usage monitoring tools to
analise the actual data traffic taking place in the network.

Monitoring the system compoments

While doing the benchmark we have to make sure, that we are able to determine the reason of the
the performance bottle-neck visible in the results. On the other hand, we should try to avoid
incorrect intepretation of the results in case if some caching/buffering effect makes the results
better than actual system features. We can use monitoring tools in addition to the benchmarking tools for that purpose.

Example monitoring tools we could while benchmarking the storage systems are:

• network interfaces monitoring tools: ntop, ethreal, tcpdump
• CPU monitoring tools:: top, dstat, vmstat
• system statistics collectors: vmstat, sar, dstat
• /proc filesystem in linux
• virtual machines monitoring tools: xm top, virt-top, virt-manager for Xen;
• and many others.

Avoiding the artificial setup

Note however, that the decision to switch off or eleminate the effects of caching should be taken carefully.
Disabling the cache, can make the operation of some optimisation technologies implemented in storage system elements
impossible. For instance, if we disable the write cache in the RAID controller, full-stripe write technique cannot
be exploited.

Switching the caching off, we also make the testing environment a bit artificial – the results we get
in such setup, can be useless in the real-life configuration.

Benchmarking workloads

Getting the realistic picture of the storage system performance, requires that we apply
the benchmark that generates the workload that matches our needs. Againt, the kind
of the testing workload should fit the purpose of performing the benchmark.

Possible purposes of the storage testing are:

• evaluate the storage performance for a specific application,
• to compare storage systems without a specific applicationin mind.

If we have a specific application in mind, we should make sure to be able to prepare
the benchmark that reflects the application characteristics.

This can be achieved in some ways:

• by finding the benchmark that generates the workload that is similar to the application data access pattern (micorbenchmarks or macrobenchmarks); the similarity should include:
• access pattern: sequential, random or mixed,
• read/write ratio,
• temporal and spatial locality of storage access requests,
• number of simultaneous access requests (concurrent application/benchmark threads).
• by choosing the test that focus on the storage characteristic that are crucial for application:
• throughput-intensive (performance measured e.g. in MB/s),
• I/O intensive (performance measured e.g. in IOPS or in request serving time).

Workload generators and benchmark types

There is a lot of benchmarks types available for free or under the license.
They differ in:

• the level on which they run: e.g. :
• application level,
• filesystem level,
• block-level and device level;
• the storage system component they examine, e.g.:
• complete storage system,
• client network,
• storage controller elements.

The correct benchmark should generate the workload that is similar to the target application needs.
Some application-level benchmarks simulate the behaviour of typical computing systems applications.

The table below provides some examples of benchmarks along with the links to Internet resources related to them.

Selected benchmarks discussion

In this section we discuss some benchmarks in details. We selected them basin on our experience
and interests. Therefore, the selection may be not optimal for every situation.

SPC benchmarks

Storage Performance Council (SPC) try to standardize storage system evaluation.
The organisation defines industry-standard storage workloads.
This "forces" vendors to publish a standardized performance of their storage systems.

SPC-1 and SPC-2 benchmarks evaluate complete storage systems, while SPC-1C and SPC-2C evaluate storage subsystems,
e.g. individual disk drives, HBAs, storage software (e.g. LVM, ...)

The table below summarizes SPC-1 and SPC-2 specifics.

Good practices and tips for benchmarking

Storage system components benchmarking

If we decide to perform layer-by-layer testing of the storage system or we want to find a source
of the bottleneck observed in the system we may want to examine a single element of the system and
avoid the influence of the other elements.

Network-only benchmarking

Example problem is to examine the network ability to carry the data traffic using a given protocol,
without the influence of the disk access latency. To do so, we may use some tricks, for instance:

• measure the network link features with a dedicated tool, that performs RAM to RAM transfers, e.g. iperf,
• configure a RAM disk and export it, e.g. using NFS (say we want to test NFS transmission efficiency).

Network benchmarking tips

As mentioned before, the client network can be a source of significant performance bootlenecks.
However, the degree the network parameters impact the performance observed by the application
is related to the fact, what is the application sensitive for.
In case, the application is bandwidth intensive, we should examine mainly the bandwidth of the network, while
for I/O intensive application the network delay should be carefully tested.

The parameters of the network link we should evaluate depend also on the kind of the protocol used
for data transmission. For instance, in case of NFS, both bandwidth and delay of the network link matters,
as typicall NFS performs a synchronous data transmission. Another example is the GridFTP protocol,
which can exploit multiple parallel tranmission streams. Therefore, the network benchmark should
be able to examine the network bandwidth using multiple parallel transmission streams (for instance, we
can use iperf benchmark with multiple TCP/IP streams).

Benchmarking the storage systems compoments

This section of the HOTWO contains the more detailed informations about benchmarking selected storage system
compoments. Each of the subsections contains some background informations as well as practical informations
on testing the storage system element.

Clients network

This part of the HOWTO describes how to perform clients network testing. Some background information is
provided along with the practical informations related to TCP network testing using iperf tool.
Iperf is a open-source benchmark working both under Linux and Windows operating system.
In this HOWTO we focus on testing the network under Linux, i.e. between two machines running Red Hat Linux Enterprise.

Background information

As mentioned in the introduction, the efficiency of data access performed by the client network may depend on
network topology, technology, transmission protocol and network delay which in turn can result from both the physical distance between communication parties and the feature of communication equipment that processes the network traffic.

As the most popular client network in storage systems is the IP connectivity implemented over Gigabit Ethernet
we focus on this technology in this part of the HOWTO. We will show, how to examine data transmission delays
present in the network and the bandwidth available in the link. As some data transmission protocols are able to exploit
multiple parallel data streams (TCP connections), we also show how to examine the influence of the number
of the TCP streams used in parallel on the actual bandwidth observed by the application.

Test preparation

Because Iperf is an client-server application, you have to install iperf on both machines involved in tests. Make sure that you use iperf 2.0.2 using p-threads or newer due to some multi-threading issues with older versions. You can check the version of this installed tools with a following command:

 [:userathostname ~]$ iperf -v
 iperf version 2.0.2 (03 May 2005) pthreads

Because in many cases the low network performance is caused by high CPU load, you should measure CPU usage at both link ends during every test round. In this HOWTO we use an open-source vmstat LINK-! tool which you probably have already installed on your machines.

Link properties

Before we start the tests, we should take a look on our network link setup. First, we should ensure that we can use MTU larger than the standard Ethernet MTU. We should try to use MTU 9000. Using jumbo frames is recommended especially in reliable and fast networks, as bigger frames boost network performance due to a better header-to-body-frame ratio. But we should remember that, it is possible to use MTU 9000 only if all network hardware between tested hosts (routers, switches, NICs etc.) support jumbo frames.

In order to enable MTU 9000 on the machine network interfaces you may use ifconfig command.

 [root@hostname ~]$ ifconfig eth1 mtu 9000

Alternatively, you can put this settings into interface configuration scripts, e.g. /etc/sysconfig/network-scripts/ifcfg-eth1 (on RHEL, Centos, Fedora etc.).

If jumbo frame are working properly, you should be able to ping one host from another using large MTU:

 [root@hostname ~]$ ping 10.0.0.1 -s 8960

In the example above, we use 8960 instead of 9000 because ping tool option -s needs frame size minus frame header which lenght is equal to 40 bytes. If you cannot use jumbo frames set the mtu to default value 1500.

To tune your link you should measure the average Round Trip Time (RTT) between machines. RTT can be obtained by multiplying the value returned by a ping command by 2. When you have RTT measured, you can set TCO read and write buffers sizes. There are three values you can set: minimum, initial and maximum buffer size. The theoretical value (in bytes) for initial buffer size is BPS / 8 * RTT, where BPS is the link bandwidth in bits/second. Example commands that set these values for the whole operating system are:

 [root@hostname ~]# sysctl -w net.ipv4.tcp_rmem="4096 500000 1000000"
 [root@hostname ~]# sysctl -w net.ipv4.tcp_wmem="4096 500000 1000000"

Probably, it is best if you start with values computed using the formula mentioned above and then tune these values according to the tests results.

You can also experiment with maximum socket buffer sizes:

 [root@hostname ~]# sysctl -w net.core.rmem_max=1000000
 [root@hostname ~]# sysctl -w net.core.wmem_max=1000000

Another options that should boost performance are:

 [root@hostname ~]# sysctl -w net.ipv4.tcp_no_metrics_save=1
 [root@hostname ~]# sysctl -w net.ipv4.tcp_moderate_rcvbuf=1
 [root@hostname ~]# sysctl -w net.ipv4.tcp_window_scaling=1
 [root@hostname ~]# sysctl -w net.ipv4.tcp_moderate_rcvbuf=1
 [root@hostname ~]# sysctl -w net.ipv4.tcp_sack=1
 [root@hostname ~]# sysctl -w net.ipv4.tcp_fack=1
 [root@hostname ~]# sysctl -w net.ipv4.tcp_dsack=1

COMMENT: the meaning of these parameters is explained in the Linux documentation (sysctl command) LINK-!.

Iperf tool description

After setting up the network link parameters, we are ready to run the test. Obviously, configuring
the network settings can be an iterative process, where we check different settings by running the
tests and evaluating their results.

To perform the test, we should run iperf in server mode in one host:

 [root@hostname ~]# iperf -s -M $mss

On the other host we should run command like this:

 [root@hostname ~]# iperf -c $serwer -M $mss -P $threads -w $\{window\} -i $interval -t $test_time

There is a description of names and symbols used in the command line:

• s - server
• c - client
• $server - address of machine on which iperf server is running
• M - MSS=MTU-40
• P number of threads which are sending data through tested link simultaneously
• w - TCP initial buffer size
• i - interval between two test rounds in seconds
• t - test time

Testing methodology

As we want to perform several actions before, during and after a single test. These actions include:

• starting vmstat on both machines - in order to collect system statistics,
• run iperf server and iperf client - to perform actual test,
• stop stop vmstat and iperf server - after the test round,
  we may want to automate the process. To achieve this, we can write some bash scripts to run tests.

To simplify the remote machine access (i.e. to avoid providing the password each time we want to execute a command on a remote host,
we may setup ssh keys properly.

To generate key-pair we use a following command:

 [root@hostname ~]# ssh-keygen -t dsa

Then we copy the public key to the remote server and add it to authorized keys file:

 [root@hostname ~]# cat identity.pub >> /home/sarevok/.ssh/authorized_keys

Now we can login from the remote serve to bug without password. So can also run programs remotely, e.g. from the bash script.

Below, we provide some simple scripts to run iperf and vmstat on both machines. The iperf and vmstat runs in rounds for various number of threads.
Note, that after each round both vmstat and iperf are killed and restarted to obtain identical testing environment.
The test is performed for different number of threads (up to 128), because the worse the link RTT, the greater performance boost can be achieved
by using multiple transmission threads. We may expect that, if RTT of the link is poor, we will achieve performance boost when
many threads are used.

There is a simple shell script to run iperf test:

 #!/bin/sh
 file_size=41
 dst_path=/home/stas/iperf_results
 script_path=/root
 curr_date=`date +%m-%d-%y-%H-%M-%S`
 serwer="10.0.1.1"
 user="root"
 test_time=60
 interval=1
 mss=1460
 window=1000000
 min_threads=1
 max_threads=128

 for threads in 1 2 4 8 16 32 64 80 96 112 128 ; do
 	ssh $user@$serwer $script_path/run_iperf.sh -s -w $\{window\}  -M $mss &
 	ssh $user@$serwer $script_path/run_vmstat 1 vmstat-$window-$threads-$mss-$curr_date &
 	vmstat 1 > $dst_path/vmstat-$window-$threads-$mss-$curr_date &
         
 	iperf -c $serwer -M $mss -P $threads -w $\{window\} -i $interval -t $test_time   >> $dst_path/iperf-$window-$threads-$mss-$curr_date 

 	ps ax | grep vmstat | awk '\{print $1\}' | xargs -i kill \{\} 2&>/dev/null
 	ssh  $user@$serwer $script_path/kill_iperf_vmstat.sh
 done

Script run_iperf.sh can look like this:

 #\!/bin/sh

 iperf $1 $2 $3 &

run_vmstat.sh script can contain:

 #\!/bin/sh
 vmstat $1 > $2 &
  

kill_iperf_vmstat.sh may look like this:

 #\!/bin/sh
 ps -elf | egrep "iperf" | egrep -v "egrep" |awk '\{print $4\}' | xargs -i kill -9 \{\}
 ps -elf | egrep "vmstat" | egrep -v "egrep" |awk '\{print $4\}' | xargs -i kill -9 \{\}

To start test script that can ignoring hangup signals, you can use nohup command.

 [:stasatworm]$ nohup script.sh &

This command keeps the test running when you close the session with server.

To obtain reliable and recurrent test results be sure that both machines are not performing any other tasks when the test is running. Run a number of tests and count an average the values acquired from all of them.

To present obtained results you may use the open-source gnuplot LINK-! program, as the graphical results presentation may help yuo to interpret them. Generating the plots can also be automated.

RAID structures

...

RAID background information

One of the most known and important storage virtualisation techniques is the RAID (Redundant Array of Independent Disks) technology. It allows to combine independent disk resources into structures that can provide advanced reliability and performance features, which are not possible to deliver using individual disk resources (e.g. individual drives).
In this section we describe standard RAID levels including RAID 0, 1, 5 and 6 as well as nested RAID structures such as 10, 50 and 60. At the end of the section we summarize fault tolerance, performance and storage efficiency characteristics of particular RAID structures.

Standard RAID structures

RAID0 (striping) does not provide any data redundancy. The main purpose of using RAID0 structures is to distribute the data traffic load over RAID components. Each file is split into blocks of a certain size and those are distributed to the various drives. This block is called chunk. A number of chunks for whom one parity chunk was counted is called stripe. Chunk size is a user-defined parameter characteristic to array. The chunks are send to all disk in the array in a way that one chunk is written to only one disk. Distributing the I/O operations among multiple drives allow to accumulate the performance of particular RAID components.

RAID1 is implemented as mirroring. The data that are written to both RAID components (disks) are ideal copies of each other. This means that every write operation performed on array must be done on both RAID components (drives). Variant of this technology is duplexing , where two independent RAID controllers are used to perform parallel writes on both mirrors.

Parity mechanism provides mechanism to write redundant data in more sophisticated way than in case of mirroring. This mechanism is used to im- plement RAID5 and RAID6 structures. If there are N drives in array, the disk controller splits data into N-1 stripe-sized chunks to write simultaneously on N-1 disks. Additionally the controller counts additional stripe-sized data called parity, and writes it to N-th disk. The parity is calculated using XOR operation. While recovering the broken data in the array (e.g. when one stripe is damaged), XOR operation is performed on the parity stripe and valid N-2 stripes. The result equals the broken stripe.

Parity information can be written to one special disk (RAID3 and RAID7structures) or can be spread across all the drives in the array (RAID5 and RAID6). The latter approach has some performance advantages, as the parity-writing load is distributed over all RAID components, opposite to the former case in which parity stripe writing can be a bottleneck. The known limitation of parity mechanism is that calculating the parity data can affect the write performance. RAID5 uses single parity stripe, while RAID6 uses double parity stripes, which provides extra fault tolerance - the array can deal with two broken drives. However, double-parity calculation can affect the write performance.

Nested RAID structures

Standard RAID structures have contradictory performance and redundancy features, e.g. mirroring provides high data redundancy, while limiting the writing performance. In order to provide both redundancy and performance, nested RAID structures are used. RAID10 combines some number of RAID1 structures by striping the data over them RAID0. In that way, superior fault tolerance can be achieved. The array can deal with 50% broken drives if for every broken drive there is his mirror drive working. RAID10 can achieve performance similar or or even better (in random read case) than RAID0. Another commonly used nested RAID structure is RAID50. It is RAID0 made of some number of RAID5 structures. RAID50 improve the performance of RAID5 thanks to the fact that the I/O traffic is distributed over particular RAID5 structures. This approach is effective especially for write operations. It also provides better fault tolerance than the single RAID level does. The drawback of nested RAID structures is that they require a relatively high number of drives to implement a given storage space.

Taking into account all theoretical information shown above, we summarize the storage efficiency information in Table 1

Table below contains a RAID levels comparison. The notes range is 0 - the worst to 5 - the best. The notes are based on http://www.pcguide.com/ref/hdd/perf/raid/levels/comp.htm and modified according to gathered experience and the fact that we are considering the same amount of drives in every RAID structure. To compare performance we assume that we make each RAID structure using the same number of drives and we use one thread to read or write data from or to the RAID structure.

RAID benchmarking assumptions

This part of the HOWTO explains the methodology and tips for measuring performance of client's Block-level storage configured on certain types of RAID storage.

Note, that the RAID storage used in this section is the software RAID implemented using the Linux MD mechanism LINK-!, over the disk drives directly attached
to the server (DAS storage).

The purpose of this part of the HOWTO is to:

• show how to examine block level storage features using both block-level and filesystem-level test (dd, iozone),
• provide examples of configuring RAIDs using Linux MD mechanism, including simple or nested RAID structures.

For the purpose of this howto we also suppose that:

• we have the same number of drives to build each RAID structure we examine,
• we create all tested RAID structures using the same identical pool of disks,
• we use the same type and size of the filesystem for filesystem-level testing.
  In that way we assure that we measure and compare only the RAID structures performance differences.

RAID structures preparation

Here, we present how to make software raid structure using Linux md tool. To create simple raid level from devices sda1 sda2 sda3 sda4 you should use following command:

 mdadm --create --verbose /dev/md1 --spare-devices=0 --level=0 --raid-devices=4 /dev/\{sda1, sda2, sda3, sda4\}

Where:

• /dev/md1 - the name of created raid group,
• spare devices - you can specify number on drives to be spare ones,
• level - simple raid level you want to create (Currently, Linux supports LINEAR (disks concatenation) md devices, RAID0 (striping), RAID1 (mirroring), RAID4, RAID5, RAID6, RAID10),
• raid-devices - number of devices you want to use to make a RAID structure.

When you create a RAID structure you should be able to see some RAID details similar to the informations shown below:

 [root@sarevok bin]# mdadm --detail /dev/md1
 /dev/md1:
         Version : 00.90.03
   Creation Time : Mon Apr  6 17:41:43 2009
      Raid Level : raid0
      Array Size : 6837337472 (6520.59 GiB 7001.43 GB)
    Raid Devices : 4
   Total Devices : 4
 Preferred Minor : 3
     Persistence : Superblock is persistent

     Update Time : Mon Apr  6 17:41:43 2009
           State : clean
  Active Devices : 4
 Working Devices : 4
  Failed Devices : 0
   Spare Devices : 0

      Chunk Size : 64K

  Rebuild Status : 10% complete

            UUID : 19450624:f6490625:aa77982e:0d41d013
          Events : 0.1

     Number   Major   Minor   RaidDevice State
        0      65       16        0      active sync   /dev/sda1
        1      65       32        1      active sync   /dev/sda2
        2      65       48        2      active sync   /dev/sda3
        3      65       64        3      active sync   /dev/sda4

When performance is considered the chunk size of the md device may be important parameter to tune. There is -c option of the mdadm command, that cen be usedto specify chunk size of kilobytes. The default is 64 kB, however it should be setup up according to some factors such as:

• physical segment size of the disk drives,
• planned block size of the filesystem.

Important!! When you want to run any benchmark on a new RAID structure you should wait until the RAID structure (re)building is finished. When you run a test when the RAID is being (re)building you achieve poor performance results, because system resources and disks are involved in (re)build RAID structure.

Because iozone benchmark is used in the next part of the HOWTH, we should firstly make a file system on the newly created md device.

The file system choice is not the topic of this part of the HOWTO. However, we assume that ext2 or ext3 file system is a good choice for our test.
When using ext2 and ext3 we may be interested in tunnning some basic file system parameters such as journaling type. We can do it using tune2fs Linux command.

To create the file system on md device and then mount it to some directory we use command like this:

 mkfs.ext3 /dev/md1
 mount /dev/md1 /mnt/md1

Again, there are some filesystem parameters that are interesting from the performance point of view. One of them is
the blocksize. It should be set taking into account the application features and the underlying storage components.
One of the rules of thumb is to use the block size equal to the size of the RAID stripe. You can set the the blocksize
using mkfs's -b parameter. There is also possibility to influence the filesystem behaviouds by using using mount command options.

Methods and tools for RAID levels benchmarking

To examine the performance of md device we normally use iozone tool. However, for making a quick test (for example to receive fast results) we may use dd tool.

dd tool

The idea of dd is to copy file from 'if' location to 'of' location. Using this tool to measure disk devices requires some trick. To measure write speed you read data from /dev/zero to file on the tested device. For measuring the read performance you should read the data from the file on tested device and write it to /dev/zero.
In that way we avoid measuring more that one storage system at a time. To measure time of reading or writing the file we use time tool. The example commands to test write and read 32 GB of data are:

for writing performance (please note using the sync command before and during the benchmark, so you are not measuring your operating system's cache performance) :

 [root@sarevok ~]# sync; time (dd if=/dev/zero of=/mnt/md1/test_file.txt bs=1024M count=32; sync)

and for reading performance:

 [root@sarevok ~]# time dd if=/mnt/md1/test_file.txt of=/dev/zero bs=1024M count=32

where:

• if - input file/device path
• of - output file/device path
• bs - size of a chunk of data to copy
• count - how many times a chunk defined by bs is copied

iozone tool

For more precise tests we use iozone tool. Iozone allows to run test in many modes, including:

• read, reread,
• random read,
• write, rewrite,
• random write .

In order to evaluate storage system features you should use a mode that fits best the application data access pattern. However, for all tests you have to run the write/rewrite mode. benchmark first (in order to generate the files for reading).

The other parameters which are interesting is the amount of Linux p-threads which are used for simultaneous read or write operations. Thanks to that feature we can realize how the performance changes while using more than 1 benchmark thread.

We can also use the block-size parameter to set the read/write chunk size (compare with md and file system chunk size).

Another important options allow to decide if the file close and flush operations are included in the measured time. Including flush time is especially significant, because this is the time when the cache buffers are written to disks.

To perform one round of the test we can use command:

 iozone -T -t $threads -r $\{blocksize\}k -s $\{file_size\}G -i 0 -i 1 -i 2 -c -e

where:

• -T - Use POSIX pthreads for throughput tests
• -t - how many threads use for test
• -r - chunk size used to test
• -s - test file size Important!! It is file size PER THREAD, because each thread writes or reads from it's own file.
• -i - test modes - we choose 0 - write/rewrite 1 - read/reread and 2 - random write/read
• -c - Include close() in the timing calculations
• -e - Include flush (fsync,fflush) in the timing calculations

It is also vital thing to realize that we measure the performance of software raid. This means that all tasks of the RAID controller (for example parity counting and checking) is performed by machine CPU. So it is necessary to measure CPU load when you perform a md device test, since CPU can be the performance bottleneck in such system. To measure CPU load we use vmstat tool.

To automate the testing we can write some simple SH script like this:

 #\!/bin/sh
 dst_path=/home/sarevok/wyniki_test_iozone
 curr_date=`date +%m-%d-%y-%H-%M-%S`

 file_size=128
 min_blocksize=1
 max_blocksize=32

 min_queuedepth=1
 max_queuedepth=16

 mkdir $dst_path
 cd /mnt/sdaw/

 blocksize=$min_blocksize
 while [: _blocksize -le $max_blocksize ]; do
         queuedepth=$min_queuedepth
 	while [: _queuedepth -le $max_queuedepth ]; do

 		vmstat 1 > $dst_path/vmstat-$blocksize-$queuedepth-$curr_date 
  
 		/root/iozone -T -t $queuedepth -r $\{blocksize\}k -s $\{file_size\}G -i 0 -i 1  -c -e > $dst_path/iozone-$blocksize-$queuedepth-$curr_date

 	    ps ax | grep vmstat | awk '\{print $1\}' | xargs -i kill \{\} 2&>/dev/null

         queuedepth=`expr $queuedepth \* 2`
         file_size=`expr $file_size \/ 2`
        done
  	blocksize=`expr $blocksize \* 2`	
 done

To start test script that can ignore hangup signals, you can use nohup command:

: root@sarevok$ nohup script.sh &

This command keeps the test running when we close the session with server. To present obtained results we use the open-source gnuplot program.

Remarks

When you perform any disk device or file system benchmark you should have in mind that there are many levels of cache in the system, e.g. file system cache, operating system cache, disk drive cache etc. The simplest way to avoid cache influence is to use such amount of data to fill all cache levels buffers. To do this we use the amount of data that is at least is equal to the machine RAM size doubled. Such data size should successfully eliminate the caching influence on measured md device performance.

After every round of tests, when we want to change some RAID (mD) or the file system parameters it is recommended to make the fresh file system in md device, in order to avoid the influence of the filesystem state on the test results.

Benchmarking tools discussion

This section of the HOTWO discusses the details of particular benchmarking tools and provides
practical information about theirs usage, automation, interpretation of the results and so on.

TO BE EXPANDED.

Links:

Practical information:

Real life benchmark requirements in RFPs:

One of the most common usages of storage benchmarking is making sure that the storage systems you buy meets your requirements. As always there are practical limits how complex the benchmark can be. This section lists benchmark procedures actually used in tenders.

CESNET - ~400TB disk array for HSM system using both FC and SATA disks (2011)

Brief disk array requirements: 

• Two types of disk in one disk array, no automatic tiering within the array required (there was an HSM system for doing this on a file level)
• Tier 1 - FC, SAS or SCSI drives, min. 15k RPM, totally min. 50TB consisting of 120x 600GB drives + 3 hot spares
• Tier 2 - SATA drives, min. 7.2.k RPM, totally min. 300TB, min. 375x1TB  + 10 hot spares OR 188x2TB + 5 hot spares

Performance requirements: 

• Sequential: there will be 10TB cluster filesystem  on the disk array using RAID5 or R6, this file system will be part of the HSM system. This filesystem will connected to one of front end servers (technical solutions of the connection is up to the candidates, e.g. MPIO, # FC channels, etc., but the solution must be identical to what is used in the proposal). The following benchmark will be run using iozone v3.347:
  
  iozone -Mce -t200 -s15g -r512k -i0 -i1 -F path_to_files
  
  The result of the test is an average value of three runs of the abovementioned command as „Children see throughput for 200 initial writers”, and , „Children see throughput for 200 readers”. 
  Minimum read speed 1600MB/s, minimum write speed 1200MB/s.

• Random: 

  Same setup of the volume as in the sequential test, but for this test, it will be connected without any filesystem (on a block level). The following test will be run on the connected LUN using fio v1.4.1 with this test definition:

  [global]
  description=CESNET_test
  [cesnet]
  # change it to name of the block device used
  filename=XXXX
  rw=randrw
  # 70% rand read, 30% rand write
  rwmixread=70
  size=10000g
  ioengine=libaio
  bs=2k
  runtime=8h
  time_based
  numjobs=32
  group_reporting
  # --- end ---
  

  The result of the test is sum of write and read IO operations divided by total elapsed time of the test in seconds. 

  Minimum required performance 9000 IOPs.

Results of the tests required as a part of proposal:  YES

Notes after evaluation: the tests themselves were OK, but the test architectures could be defined a bit better: The tests actually measured only performance of the FC disks (candidates obviously configured the volume in such a way that it was faster), performance of SATA volumes was not evaluated at all. Also, the winner used RAID5 as required but there was a big RAID0 volume above the 20 individual RAID5s (thus creating RAID50) which was allowed but not used in production afterwards.

File system benchmarking examples:

• Filesystem contest
• Benchmarking Filesystems

Independent storage benchmarking organisations:

Storage Performance Council (SPC)

• Storage Performance Council
• SPC-1 Benchmark Results
• SPC-2 Benchmark Results

Enterprise Strategy Group (ESG)

• Enterprise Strategy Group

Howto authors:

The text re-uses the material presented by Stijn Eeckhaut in Espoo, Finland,
during the 1st TF-Storage Meeting, 8 April 2008.
It also includes the material prepared by Maciej Brzezniak from Poznan Supercomputing
and Networking Centre, Poland and Stanislaw Jankowski, student at Poznan University of Technology.

If you want to contact the HOWTO authors directly, send the email to:

• maciekb -=at=- man.poznan.pl
• staszek -=at=- man.poznan.pl