Perf Tools
Content
CESNET technical report number 18/2003
Performance Testing Tools
Jan BartoĊ
30/10/2003
1
Abstract
The report describes properties and abilities of software tools for performance testing. It also shows the tools comparison according to requirements for testing tools described in RFC 2544 (Benchmarking Terminology for Network Interconnection Devices).
2
Introduction
This report is intended as a basic documentation for auxiliary utilities or programs that have been made for the purposes of evaluating transfer performance between one or more PCs in the role of traffic generators and another one on the receiving side. Section 3 3 describes requirements for software testing tools according to RFC 2544. Available tools for performance testing are catalogued by this document in section 4 4. This section also shows the testing tools compatibility with RFC 2544 and the evaluation of IPv6 support. The summary of supported requirements for testing tools and IPv6 support can be seen in section 5 5.
3
3.1
Requirements for software performance testing tools according to RFC-2544
User defined frame format
Testing tool should be able to use test frame formats defined in RFC 2544 Appendix C: Test Frame Formats. These exact frame formats should be used for specific protocol/media combination (ex. IP/Ethernet) and for testing other media combinations as a template. The frame size should be variable, so that we can determine a full characterization of the DUT (Device Under Test) performance. It might be useful to know the DUT performance under a number of conditions, so we need to place some special frames into a normal test
frames stream (ex. broadcast, management, routing update frames . . . ). These modifiers could have a significant impact on an ability of a router to forward data frames. The testing tool should be able to use a random destination address to simulate multiple streams of data. For more details about recommended frame formats see Appendix C included with RFC 2544.
3.2
Verifying received frames
The receiver should discard any frames received during a test run that are not actual forwarded test frames (ex. management frames, routing update frames . . . ). It should verify the length of received frames and report the number of dropped, duplicated frames, frames that were received out of order and the number of gaps in the received frame numbering sequence. The testing tool should verify that the all of the routing updates (see above in section User defined frame format) were processed by the DUT.
3.3
Bidirectional traffic
Real network traffic is not in a single direction. To test the bidirectional performance of a DUT, we need a testing tool, which can be run with the same data rate in each direction. The sum of the data rates should not exceed the theoretical limit for the media.
3.4
Setting inter-frame time gap
All the tests should be performed with both steady state traffic and with traffic consisting of repeated bursts of frames. Because of needs to determine the minimum interval between bursts, which the DUT can process with no frame loss, we need to set the inter-frame time gap between defined frames (bursts).
4
Tools
Each tool is defined as follows: Description A description of the tools construction, and the implementation methodology of the tests. Automation What steps are required to complete the test? What human intervention is required?
CESNET technical report number 18/2003
2
Settings possibilities Summary of program settings possibilities. Availability How do you retrieve this tool and get more information about it? Required Environment Compilers, OS version, etc. required to build and/or run the associated tool. RFC 2544 compatibility Summary of requirements to testing tools according to RFC 2544 as defined in section 3 3. References A list of publications relating to the tool, if any.
4.1
4.1.1
DBS 1.1.5
Description
DBS (Distributed Benchmark System) is aiming to give performance index with multi-point configuration and also in order to measure changes of throughput. It measures the performance of entire TCP functions in various operational environments. DBS has the capability of both measuring and analyzing TCP performance more in details. DBS is able to evaluate the three TCP control mechanisms - flow control, retransmission control and congestion avoidance control. The DBS can generate various situations where the three controls are working together. The DBS can also generate UDP traffic for more realistic benchmarking.
Figure 1: DBS architecture
CESNET technical report number 18/2003
3
The DBS is composed of three programs. Figure shows an overview of the DBS System Structure. The dbsc is a control program to handle monitoring program launched on observed hosts. dbsd is a program for sending and receiving data among observed hosts. These two programs are used for actual benchmarking. The dbs view is used for data analysis. The details of these programs are described below: <h4>dbsc: DBS controller (Controlling Host)</h4> The DBS controller is a program controlling the experiment of TCP/UDP data transfer. Controller reads commands from a command file, then it asks the DBS daemons to start data transfer experiments, and after receiving results from the daemons, the DBS controller saves them into the local files. <h4>dbsd: DBS daemon (Measuring Host)</h4> The DBS daemon is a daemon program that is launched on the experimental hosts. It sends and receives network traffic according to the commands instructed by the DBS controller. <h4>dbs view: DBS viewer</h4> The DBS viewer is a program for analysis data which is gathered by the DBS controller. It draws graphs to reveal the transitions of sequence numbers, changes of throughput, changes of delay times or other performance indexes. If measuring host has only one network interface card, traffic of command/result and measured traffic are transferred on the same network. This may influence the measurement results. To avoid this influence, DBS controls command/result and measured traffic are not transferred at the same time. DBS implementation assumes that clocks on all the hosts participating to the benchmark are synchronized. 4.1.2 Automation
Commands of execution are driven by a command file. Format of the command file is as follows. When multiple data streams are transferred in the same test, many configurations should be written in the same file.
# First Configuration { sender {’configurations of sender’; ’sending traffic pattern {s}’;} receiver {’configurations of receiver’;’receiving message pattern {s}’;} ’configuration of the connection’; }
CESNET technical report number 18/2003
4
# Second Configuration { sender {’configurations of sender’; ’sending traffic pattern {s}’;} receiver {’configurations of receiver’;’receiving message pattern {s}’;} ’configuration of the connection’; } . . .
Format of ’sending traffic pattern s’
pattern { data size, message size, interval, wait time; data size, message size, interval, wait time; . . . ; }
Figure 2: Patern parameters meaning The traffic is modeled as a sequence of data chunks called ”frames”. The size of each frame may vary. Each frame may consist of several messages. A single message is defined that it can be transferred in a single UDP datagram. If a frame is longer than the UDP maximum transfer unit (64KB), the frame is split into several messages. Between frames, there is a time gap called ”wait time” which implies the application overhead. The preparation time of each frame can be treated as this wait time. Moreover, this model controls the frame intervals. This frame interval can be used for modeling of an application level rate control. Command File Sample
CESNET technical report number 18/2003
5
# Sample file { sender { hostname = host1; port = 0; send_buff = 65535; recv_buff = 65535; mem_align = 2048; pattern {2048, 2048, 0.0, 0.0} } receiver { hostname = host2; port = 20001; recv_buff = 65535; send_buff = 65535; mem_align = 2048; pattern {2048, 2048, 0.0, 0.0} } file = test1; protocol = TCP; start_time = 0.0; end_time = 30; send_times = 2048; }
See http://www.kusa.ac.jp/ yukio-m/dbs/dbs man.html for more information about constructing command file. 4.1.3
Settings possibilities send/receive buffer size modification setting the TCP no delay option
CESNET technical report number 18/2003
specify the starting time of data transfer for each connection specify frame size and inter-frame time gap specify test duration sending complicated traffic patterns
6
Through ”tcp trace” function, several TCP internal information such as TCP/IP pseudo-headers, TCP congestion window size, round trip time, retransmission timeout and other values in the TCB structure can be obtained. 4.1.4 Availability
See http://www.ai3.net/products/dbs1 for details of precise OS versions supported, and for download of the source code. Current implementation supports BSDI BSD/OS, FreeBSD, NetBSD, Linux, mkLinux, SunOS, IRIX, Ultrix, Digital UNIX, NEWS OS, HP-UX. 4.1.5 Required Environment
C language compiler, UNIX-style socket API support. 4.1.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames Checks only sequence numbers of received frames. Bidirectional traffic It could be made of two single traffics between specified hosts running at the same time in the opposite direction. Setting inter-frame time gap Full Support 4.1.7 References
Performance and Control of Network Systems, Proceedings of SPIE, Volume 3231, November 1997(English) ”DBS: a powerful tool for TCP performance evaluations”2 Informating Processing Society of Japan, DPS, 95-DPS-71, July 1995 (Japanese) ”A proposal of DBS: performance evaluation for TCP over multipoint connection”3 Internet Conference’ 96, July 1996 (Japanese) ”Design and Implementation of DBS: a performance evaluation system for TCP” 4
1 2
http://www.ai3.net/products/dbs http://www.kusa.ac.jp/ yukio-m/papers/dbs paper.ps 3 http://www.kusa.ac.jp/ yukio-m/papers/dps9507.ps 4 http://www.kusa.ac.jp/ yukio-m/papers/conf96.ps
CESNET technical report number 18/2003
7
Master’s Thesis, Graduate School of Information Science, Nara Institute of Science and Technology, March 7, 1996 (Japanese) ”Design and Implementation of DBS: a performance evaluation system for multipoint TCP connections”5 ”Digest of Master’s Thesis”6
4.2
4.2.1
IPerf 1.7.0
Description
IPerf is a ttcp like tool with considerable advantages over it. Using a clientserver model to determine maximum bandwidth you can also measure delay jitter, packet loss, determine MTU, support TCP window size, run tests by amount transferred or for a specified period of time. Server can handle multiple simultaneous connections. Client can create UDP streams of specified bandwidth. Client-server model can use for testing bidirectional mode called ”dual testing mode”. IPerf uses representative streams to test out how link layer compression affects your achievable bandwidth and prints periodic intermediate bandwidth, jitter, and loss reports at specified intervals. As one of the few also supports IPv6. 4.2.2 Automation
It is command-line driven. Use the -D command line option to run the server as a daemon and redirect the output to a file. E.g. iperf -s -D > iperfLog. Command line samples:
node2> iperf -s -u -l 32k -w 128k -i 1
-s = run IPerf in server mode
-u = use UDP instead TCP
-l 32k = buffer length
-w 128k = largest receivable datagram size
-i 1 = interval time in seconds between periodic reports
node1> iperf -c node2 -b 10m -l 32k -w 128k
5 6
http://www.kusa.ac.jp/ yukio-m/papers/mthesis.ps http://www.kusa.ac.jp/ yukio-m/papers/mthesis digest.ps
CESNET technical report number 18/2003
8
-c = run IPerf in client mode
node2 = server address
-b 10m = UDP bandwidth to send at, in bits/sec - 10Mbit/sec
4.2.3
Settings possibilities send/receive buffer size modification specify TCP maximum segment size
4.2.4
Iperf is released as a distribution of the C++ source. Pre-compiled binaries for selected operating systems are also available (Linux, FreeBSD, IRIX, MacOS, MS Windows, OpenBSD, Solaris). See http://dast.nlanr.net/Projects/Iperf/7 for more information about program, for details of precise OS versions supported, and for download of the source code. 4.2.5 Required Environment
C++ non cross-compiler
7
http://dast.nlanr.net/Projects/Iperf/
CESNET technical report number 18/2003
setting the TCP No Delay option UDP server provides multicast mode bidirectional testing mode tradeoff testing mode (request/response test) setting the number of simultaneous connections to make to the server (requires thread support on both the client and server) specify the type-of-service (as defined in RFC 1349) for outgoing packets specify the time-to-live for outgoing multicast packets use a representative stream (from file or stdin) to measure bandwidth specify test duration Availability
9
4.2.6
RFC 2544 compatibility
User defined frame format Iperf can specify only data content of UDP packet in frame definition. Verifying received frames It determines packet loss only. Bidirectional traffic Full Support Setting inter-frame time gap No Support
4.3
4.3.1
NetPerf 2.2pl4
Description
NetPerf is a benchmark that can be used to measure the performance of many different types of networking. It provides tests for both unidirectional throughput and end-to-end latency. It also includes provisions for CPU utilization measurement. Its primary focus is on bulk data transfer and request/response performance using either TCP or UDP and the BSD Sockets interface. There are optional tests available to measure the performance of DLPI, Unix Domain Sockets, the Fore ATM API and the HP HiPPI LLA interface. NetPerf is designed around the basic client-server model. There are two programs - netperf and netserver. The first thing after running program is establishing a control connection. This connection is used to pass test configuration information and results to and from remote system. After this process is established new connection - measurement connection. The test is performed and the results are displayed. NetPerf places no traffic on the control connection while a test is in progress. NetPerf provides three types of transfers: Bulk Data Transfer - also referred to as ”stream” or ”unidirectional stream”. This test measures how fast one system can send data to another and/or how fast that other system can receive it. Request/Response Transfer - request/response performance is quoted as ”transactions/sec” for a given request and response size. A transaction is defined as the exchange of a single request and a single response. Connect/Request/Response - instead of simply measuring the performance of request/response in the same connection, it establishes a new connection for each request/response pair.
CESNET technical report number 18/2003
10
NetPerf includes test which use a socket interface to IPv6, but the control connection remains IPv4. 4.3.2 Automation
Execution as child of inetd requires editing of /etc/services and /etc/inetd.conf. To assist in measuring, script files are provided with the NetPerf distribution (script for measuring stream performance, script for measuring request/response performance ...). NetPerf is command-line driven. Command line samples:
node1> netserver -p 20000 -n 2
-p = listen on the specified port
-n = number of CPU’s in the system
node2> netperf -t TCP_STREAM -H node1 -- -s 16384 -S 16K
-t = test name to perform (script filename)
-H = name of the remote system
-s = local send and receive socket buffer size
-S = the same as -s, but for remote system
4.3.3
Settings possibilities send/receive buffer size modification of both systems (local/remote) setting TCP No Delay option
CESNET technical report number 18/2003
setting the size of a burst of packets (used to pace the send rate when is no flow control provided by the protocol being measured) specify test duration pre-fill buffers with data from file CPU rate calibration
11
4.3.4
Availability
See http://www.netperf.org/netperf/NetperfPage.html 8 for more details or email Rick Jones ([email protected]). Binaries are available here for HP/UX Irix, Solaris, and Win32. 4.3.5 Required Environment
C language compiler, sockets. 4.3.6 RFC 2544 compatibility
User defined frame format NetPerf can only specify data content of UDP packet in frame definition. Verifying received frames No Support Bidirectional traffic No Support Setting inter-frame time gap Full Support
4.4
4.4.1
NetPIPE 3.3
Description
NetPIPE (Network Protocol Independent Performance Evaluator) is a protocol independent performance tool for comparing different networks and protocols. NetPIPE performs simple ping-pong tests, bouncing messages of increasing size between two processes, whether across a network or within an SMP system. Message sizes are chosen at regular intervals, and with slight perturbations, to provide a complete test of the communication system. Each data point involves many ping-pong tests to provide an accurate timing. It also has an option to measure performance without cache effects. NetPIPE consists of two parts: a protocol independent driver, and a protocol specific communication section. The communication section contains the necessary functions to establish a connection, send and receive data, and close a connection. This part is different for each protocol. However, the interface between the driver and protocol module remains the same. Therefore, the driver does not have to be altered in order to change communication protocols. NetPIPE is a variable time benchmark, which increases the transfer block size from a single byte until transmission time exceeds 1 second. For each block size
8
http://www.netperf.org/netperf/NetperfPage.html
CESNET technical report number 18/2003
12
c, three measurements are taken: c - p bytes, c bytes, and c + p bytes, where p is a perturbation factor with a default value of 3. This perturbation allows analysis of block sizes that are possibly slightly smaller or larger than an internal network buffer. NetPIPE uses a ping-pong transfer. This forces the network to transmit just the data block without streaming other data blocks in with the message. The result is the transfer time of a single block, thus providing the information necessary to answer which block size is best, or what is the throughput given a block of size k. NetPIPE produces a file that contains the transfer time, throughput, block size, and transfer time variance for each data point and is easily plotted by any graphing package. Some typical uses: Measuring the overhead of message-passing protocols.
Help in tuning the optimization parameters of message-passing libraries.
Identify dropouts in networking hardware.
Optimizing driver and OS parameters (socket buffer sizes, etc.).
4.4.2
Automation
NetPIPE is a command-line driven program. Command line samples:
node1> NPtcp -r -b 32768 -l 1 -u 1048576
-r = receiver
-b = send and receive TCP buffer sizes -l = lower bound of block size -u = upper bound of block size
node2> NPtcp -t -h node1 [options]
-t = transmitter -h = remote host
CESNET technical report number 18/2003
13
4.4.3
Settings possibilities specify TCP send/receive buffer sizes setting lower/upper bound of variable block size
4.4.4
See http://www.scl.ameslab.gov/netpipe available.
You can download the latest version NetPIPE 3.5.tar.gz 10 . Infiniband module for the Mellanox VAPI has been added, an integrity check option (-I) has been incorporated, and problems with the streaming mode have been fixed. 4.4.5 Required Environment
C language compiler 4.4.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames No Support Bidirectional traffic No Support Setting inter-frame time gap No Support 4.4.7 References
Quinn O. Snell, Armin R. Mikler, and John L. Gustafson: NetPIPE: A Network Protocol Independent Performance Evaluator, IASTED Conference Paper
9 10
http://www.scl.ameslab.gov/netpipe http://www.scl.ameslab.gov/netpipe/code/NetPIPE 3.5.tar.gz
CESNET technical report number 18/2003
specify increment step size specify output filename specify buffers alignment specify buffer offset setting stream option (default mode is ”ping-pong”) Availability
9
for more details. Binaries are not
14
4.5
4.5.1
NetSpec 3.0
Description
NetSpec is a network level end-to-end performance evaluation tool for Network Experimentation and Measurement. It provides a fairly generic framework that allows a user to control multiple processes across multiple hosts from a central point of control. It uses a scripting language that allows the user to define multiple traffic flows from/to multiple computers. This allows an automatic and reproducible test to be performed. NetSpec exhibits many features like parallel and serial multiple connections, a range of emulated traffic types (FTP, HTTP, MPEG, etc.) on the higher levels, the most widely used transport protocols today, that is TCP and UDP, three different traffic modes, scalability, and the ability to collect system level information from the communicating systems as well as intermediate network nodes. The following figure shows the basic NetSpec architecture.
Figure 3: NetSpec architecture The controller is a process that supports the user interface, which is currently a file containing a description of an experiment using a simple block structured language in which the connection is the basic unit for an experiment and via the control daemon controls the daemons implementing the test. For every connection in the experiment, the corresponding test daemons are created. These test daemons send or receive data transferred across the connection. Each daemon is responsible for its own report generation after experiment
CESNET technical report number 18/2003
15
execution is complete. The output report is delivered to the controller via the control daemon for viewing by the user. NetSpec supports three basic traffic modes: <h4>Full Stream Mode</h4> Also known as ”full blast mode”, it instructs the test daemons to transmit data as fast as possible. <h4>Burst Mode</h4> The hosts under test transmit data every some specific intervals, specified by the burst period. The burst size and the burst period is passed as a parameter by the user (in blocks/burst and bytes/block). This mode is very useful in real world experiments where rate mismatches might reduce the throughput dramatically. <h4>Queued Burst Mode</h4> The hosts under test transmit data every some specific intervals, specified by the burst period. This mode is a variation of the basic burst algorithm and the burst size and burst period are passed as a parameter by the user. The advantage of this algorithm is that variations in available line rate will not cause it to miss blocks generated by interrupts arriving before previous write completes. The drawback is that characteristics of the traffic are influenced by the queuing delay. 4.5.2 Automation
NetSpec consists of several daemon types that are started and controlled by the main netspec daemon (netspecd). NetSpec uses a scripting language that allows the user to define multiple traffic flows from/to multiple computers. This allows an automatic and reproducible test to be performed. A test would be started with:
Netspec script.name
The script given below is of the most widely type used for a TCP point-topoint connection. The first block specifies the characteristics of the traffic source, while the second block specifies the characteristics of the receiver host. Script file sample
1 2 3 cluster { test host1 { type = full (blocksize=32768, duration=30);
CESNET technical report number 18/2003
16
4 5 6 7 8 9 10 11 12 13 14
protocol = tcp (window=32768); own = host1:20200; peer = host2:20200; } test host2 { type = sink (blocksize=32768, duration=30); protocol = tcp (window=32768); own = host2:20200; peer = host1:20200; } }
Figure 4: Test setup with the invoked daemons over TCP In this particular example, the host with name host1 is the sender system and the host with name host2 is the receiver system. The sender (line 2) sends data in full stream (line 3) mode; it transmits 32768 bytes as fast as possible for 30 seconds (duration of test). For this data transfer TCP (line 4) is used in the transport layer with a window size of 32.7KB. NetSpec can be installed in either of the two ways - inetd installation and standalone installation. 4.5.3
Settings possibilities specify size of frames and inter-frame time gap in burst modes setting traffic mode (full stream mode, burst mode or queued burst mode)
CESNET technical report number 18/2003
specify type of connection (point-to-point, multipoint-to-point connections)
point-to-multipoint or
17
setting connection mode (serial, parallel or cluster)
specify test duration
specify protocol type
4.5.4
Availability
See http://www.ittc.ku.edu/netspec 11 for more details. NetSpec can run on a variety of platforms. The following binaries are available: Digital Unix, Solaris, Linux, SunOs, FreeBSD, Irix. NetSpec manual can be found at http://www.ittc.ukans.edu/netspec/docs/NetSpecUser.pdf
12
4.5.5
Required Environment
C language compiler 4.5.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames No Support Bidirectional traffic It could be made of two single traffics between specified hosts running at the same time in the opposite direction. Setting inter-frame time gap Full Support
4.6
4.6.1
RUDE & CRUDE 0.70
Description
RUDE (Real-time UDP Data Emitter) is a small and flexible program that generates traffic to the network, which can be received and logged on the other side of the network with the CRUDE (Collector for RUDE). These programs can generate and measure only UDP traffic. To observe several variables describing the utilization of hardware resources (CPU load, number of interrupts etc.) can be used another free software package - atsar. It can record in regular intervals the
11 12
http://www.ittc.ku.edu/netspec http://www.ittc.ukans.edu/netspec/docs/NetSpecUser.pdf
CESNET technical report number 18/2003
18
values of various system counters and parameters together with timestamps, for example the CPU load and number of interrupts generated by the network interface cards. The rude/crude distribution contains several example Perl scripts for basic processing of the decoded crude output and computing jitter. Rude allows definitions of a number of concurrent UDP flows with varying packet size and rate. The TOS field in the IP header can also be set. Reordered and duplicated packets may arrive with arbitrary delays, hence a bitmask with 32 bits describing the fate of the last 32 packets of the flow is kept. Two types of generated flow are implemented - constant and trace. Constant flow means a constant-rate UDP flow, where the packet rate per second and packet size in bytes can be specified. The trace option gives a reference to a text file where the parameters of every single packet has to be given. The smallest time unit for flow definition is 1 ms. 4.6.2 Automation
RUDE is driven by a script file, which is used to specify the generated flows. Example of a simple script file follows:
START NOW ## FLOW 1: (flow ID = 25) ## ## Starts immediately at the specified START time with following parameters: ## 400 packets/second with 100 bytes/packet = 40kbytes/sec (1kbyte=1000bytes) ## ## Sets the TOS for this flow to LOW_DELAY (0x10) ## ## 9 seconds after that the flow is turned off... ## 0000 25 ON 3001 10.1.1.1:10001 CONSTANT 400 100 TOS 25 0x10 9000 25 OFF ## FLOW 2: (flow ID = 1) ## ## This flow acts as specified in the TRACE configuration file. ## 0000 1 ON 3002 10.1.1.1:10001 TRACE trace_file.txt 9999 1 OFF
Here, the flows 25 and 1 (second field in each row) are specified. The numeric values at the beginning of the rows are time offsets related to a predefined
CESNET technical report number 18/2003
19
global origin when the flow is to be started or its parameters modified. The destination IP address and port are defined as 10.1.1.1 and 10001, respectively, and the source port as 3001, respectively 3002. The source IP address will be determined at run-time as the address of egress interface. The second flow uses for packets definition the external file - trace file.txt. The following command line samples shows, how to run the specified script file. Command line samples:
sender> rude -s sample.cfg
-s <script file> = defines the script file to be used
receiver> crude
4.6.3
Settings possibilities control the length of the test specify the type-of-service (TOS) for outgoing packets
4.6.4
For more details about installation and using RUDE/CRUDE see http://rude.sourceforge.net.13 On this website can be downloaded also the old releases of this utilities. The newest release can be found at http://gd.tuwien.ac.at/opsys/linux/sf/r/rude/14 .
13 14
http://rude.sourceforge.net http://gd.tuwien.ac.at/opsys/linux/sf/r/rude/
CESNET technical report number 18/2003
plenty of flows definition TRACE flow - definition of packet size and time gap between packets (max. time resolution = 1 microsecond) CONSTANT flow = constant bit rate traffic (you may change packet size and packet rate) calculate some statistics on-the-fly setting the process real-time priority some visualization and statistical analysis - using grude script Availability
20
4.6.5
Required Environment
C language compiler 4.6.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames No Support Bidirectional traffic No Support Setting inter-frame time gap Full Support 4.6.7 References
Lhotka L.: Software tools for router performance testing, Technical Report 10/2001, CESNET, Prague, 2001.
4.7
4.7.1
TRENO (07/30/97)
Description
TRENO (Traceroute RENO) is a TCP throughput measurement tool, which is based on sending UDP packets with low TTL in patterns that are controlled at the user-level. Hosts and routers along the path to the final destination will send back ICMP TTL Exceeded messages which have similar characteristics to TCP ACK packets. TRENO also has an ICMP mode, which uses ICMP ECHO Requests instead of low TTL UDP packets. In this mode, you only get information about the final destination. The same sized packets are sent in both directions, giving you some information about the return path (request-response test). This allows to measure throughput independent of the TCP implementation of end hosts. TRENO has some limitations: Each hop should run for at least 10 seconds.
Some routers do not respond to the Treno probes as quickly as they forward packets. So the results of the Treno test will not accurately reflect the bandwidth at these hops. The Treno Server is single threaded, so if someone else is running tests you will need to wait until they complete their work.
CESNET technical report number 18/2003
21
For more details about these limitations, see TRENO’s homepage. 4.7.2 Automation
Command-line driven. No ”server” is required, and it only requires a single argument of the machine to run the test to. Command line samples:
sender> treno -p 10 hostname
-p < s > = set the test duration
4.7.3
Settings possibilities specify test duration specify the (initial) MTU
4.7.4
See http://www.psc.edu/networking/treno info.html15 or e-mail Matt Mathis ([email protected]) or Jamshid Mahdavi ([email protected]). 4.7.5 Required Environment
C compiler, raw sockets. 4.7.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames Checks only sequence numbers of received frames. Bidirectional traffic No Support Setting inter-frame time gap No Support
15
http://www.psc.edu/networking/treno info.html
CESNET technical report number 18/2003
specify used mode (ICMP or UDP packets) Availability
22
4.8
4.8.1
TTCP6 Revision: 3.8
Description
Originally written to move files around, TTCP became the classic throughput benchmark or load generator, with the addition of support for sourcing to/from memory. It can also be used as a traffic absorber. It has spawned many variants, recent ones include support for UDP, IPv6, data pattern generation, page alignment, and even alignment offset control. 4.8.2 Automation
It is the command-line driven tool. To use it, start the receiver on one side of the path, then start the transmitter on the other side. The transmitting side sends a specified number of TCP packets to the receiving side. At the end of the test, the two sides display the number of bytes transmitted and the time elapsed for the packets to pass from one end to the other. Command line samples:
receiver> ttcp6 -r -s -v -n100
-s (ttcp6 -r) : sink (discard) all data from network
sender> ttcp6 -t -s -v -n100 host
-s = (ttcp6 -t) : source a pattern to network
-v = verbose: print more statistics
-n = number of source buffers written to network
4.8.3
Settings possibilities send/receive buffer size modification setting the TCP no delay option
CESNET technical report number 18/2003
using UDP instead of TCP setting socket buffer size
23
4.8.4
Availability
See ftp://ftp.arl.mil/pub/ttcp/16 which includes the most common variants available or e-mail ARL ([email protected]). Download the latest version with IPv6 support from ftp://ftp.bieringer.de/pub/linux/IPv6/ttcp/ttcp+ipv6-3.tar.gz 17 . 4.8.5 Required Environment
C compiler, BSD sockets. 4.8.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames No Support Bidirectional traffic No Support Setting inter-frame time gap No Support
5
5.1 5.2
Summary
RFC 2544 compatibility IPv6 support
6
Conclusion
As we can see in the section 5 5 (Summary), none of the performance tools listed above, supports all of the requirements mentioned in RFC 2544 and described in section 3 3. Also IPv6 support is not an ordinary character. Only three of these tools support IPv6 protocol. The DBS 4.1 (Distributed Benchmark System) testing tool seems to be the best of these performance testing tools we have tested. This utility allows a user to control multiple processes across multiple hosts from a central point of control.
16 17
ftp://ftp.arl.mil/pub/ttcp/ ftp://ftp.bieringer.de/pub/linux/IPv6/ttcp/ttcp+ipv6-3.tar.gz
CESNET technical report number 18/2003
24
Figure 5: Summary of RFC 2544 compatibility
Figure 6: Summary of IPv6 support
CESNET technical report number 18/2003
25
It uses a scripting language that allows the user to define multiple traffic flows from/to multiple computers. This allows an automatic test to be performed. It also supports definition of inter-frame time gap. Of course we have to modify this auxiliary utility, so it can fulfill all of necessary requirements. Verifying received frames is not well done. It checks only sequence numbers of received frames, so the complete verification of received frames as defined in section 3 3 have to be made. A possibility of sending user defined frame format should be appended to the DBS utility too, because we need to generate special traffic. An alternative testing tool we can modify and use is NetSpec 4.5. The conclusive factor is simplicity of modifying the source codes of these tools.
References
[1] [2] Parker S., Schmechel C.: Some Testing Tools for TCP Implementors RFC 2398, August 1998 Bradner S., McQuaid J.: Benchmarking Methodology for Network Interconnect Devices RFC 2544, March 1999 Bradner S.: Benchmarking Terminology for Network Interconnection Devices RFC 1242, July 1991. DBS WWW pages http://www.ai3.net/products/dbs18 IPerf WWW pages http://dast.nlanr.net/Projects/Iperf19 NetPerf WWW pages and NetPerf manual included in archive http://www.netperf.org/netperf/NetperfPage.html20 NetPIPE WWW pages http://www.scl.ameslab.gov/netpipe21 NetSpec WWW pages http://www.ittc.ku.edu/netspec22
[3]
[4] [5] [6] [7] [8]
18
http://www.ai3.net/products/dbs http://dast.nlanr.net/Projects/Iperf 20 http://www.netperf.org/netperf/NetperfPage.html 21 http://www.scl.ameslab.gov/netpipe 22 http://www.ittc.ku.edu/netspec
19
CESNET technical report number 18/2003
26
[9] [10]
NetSpec User Manual http://www.ittc.ukans.edu/netspec/docs/NetSpecUser.pdf23 RUDE & CRUDE WWW pages and the documentation added to the source code http://rude.sourceforge.net24 Treno WWW pages http://www.psc.edu/networking/treno info.html25 TTCP documentation added to source code
[11] [12]
23 24
http://www.ittc.ukans.edu/netspec/docs/NetSpecUser.pdf http://rude.sourceforge.net 25 http://www.psc.edu/networking/treno info.html
CESNET technical report number 18/2003
27
Performance Testing Tools
Jan BartoĊ
30/10/2003
1
Abstract
The report describes properties and abilities of software tools for performance testing. It also shows the tools comparison according to requirements for testing tools described in RFC 2544 (Benchmarking Terminology for Network Interconnection Devices).
2
Introduction
This report is intended as a basic documentation for auxiliary utilities or programs that have been made for the purposes of evaluating transfer performance between one or more PCs in the role of traffic generators and another one on the receiving side. Section 3 3 describes requirements for software testing tools according to RFC 2544. Available tools for performance testing are catalogued by this document in section 4 4. This section also shows the testing tools compatibility with RFC 2544 and the evaluation of IPv6 support. The summary of supported requirements for testing tools and IPv6 support can be seen in section 5 5.
3
3.1
Requirements for software performance testing tools according to RFC-2544
User defined frame format
Testing tool should be able to use test frame formats defined in RFC 2544 Appendix C: Test Frame Formats. These exact frame formats should be used for specific protocol/media combination (ex. IP/Ethernet) and for testing other media combinations as a template. The frame size should be variable, so that we can determine a full characterization of the DUT (Device Under Test) performance. It might be useful to know the DUT performance under a number of conditions, so we need to place some special frames into a normal test
frames stream (ex. broadcast, management, routing update frames . . . ). These modifiers could have a significant impact on an ability of a router to forward data frames. The testing tool should be able to use a random destination address to simulate multiple streams of data. For more details about recommended frame formats see Appendix C included with RFC 2544.
3.2
Verifying received frames
The receiver should discard any frames received during a test run that are not actual forwarded test frames (ex. management frames, routing update frames . . . ). It should verify the length of received frames and report the number of dropped, duplicated frames, frames that were received out of order and the number of gaps in the received frame numbering sequence. The testing tool should verify that the all of the routing updates (see above in section User defined frame format) were processed by the DUT.
3.3
Bidirectional traffic
Real network traffic is not in a single direction. To test the bidirectional performance of a DUT, we need a testing tool, which can be run with the same data rate in each direction. The sum of the data rates should not exceed the theoretical limit for the media.
3.4
Setting inter-frame time gap
All the tests should be performed with both steady state traffic and with traffic consisting of repeated bursts of frames. Because of needs to determine the minimum interval between bursts, which the DUT can process with no frame loss, we need to set the inter-frame time gap between defined frames (bursts).
4
Tools
Each tool is defined as follows: Description A description of the tools construction, and the implementation methodology of the tests. Automation What steps are required to complete the test? What human intervention is required?
CESNET technical report number 18/2003
2
Settings possibilities Summary of program settings possibilities. Availability How do you retrieve this tool and get more information about it? Required Environment Compilers, OS version, etc. required to build and/or run the associated tool. RFC 2544 compatibility Summary of requirements to testing tools according to RFC 2544 as defined in section 3 3. References A list of publications relating to the tool, if any.
4.1
4.1.1
DBS 1.1.5
Description
DBS (Distributed Benchmark System) is aiming to give performance index with multi-point configuration and also in order to measure changes of throughput. It measures the performance of entire TCP functions in various operational environments. DBS has the capability of both measuring and analyzing TCP performance more in details. DBS is able to evaluate the three TCP control mechanisms - flow control, retransmission control and congestion avoidance control. The DBS can generate various situations where the three controls are working together. The DBS can also generate UDP traffic for more realistic benchmarking.
Figure 1: DBS architecture
CESNET technical report number 18/2003
3
The DBS is composed of three programs. Figure shows an overview of the DBS System Structure. The dbsc is a control program to handle monitoring program launched on observed hosts. dbsd is a program for sending and receiving data among observed hosts. These two programs are used for actual benchmarking. The dbs view is used for data analysis. The details of these programs are described below: <h4>dbsc: DBS controller (Controlling Host)</h4> The DBS controller is a program controlling the experiment of TCP/UDP data transfer. Controller reads commands from a command file, then it asks the DBS daemons to start data transfer experiments, and after receiving results from the daemons, the DBS controller saves them into the local files. <h4>dbsd: DBS daemon (Measuring Host)</h4> The DBS daemon is a daemon program that is launched on the experimental hosts. It sends and receives network traffic according to the commands instructed by the DBS controller. <h4>dbs view: DBS viewer</h4> The DBS viewer is a program for analysis data which is gathered by the DBS controller. It draws graphs to reveal the transitions of sequence numbers, changes of throughput, changes of delay times or other performance indexes. If measuring host has only one network interface card, traffic of command/result and measured traffic are transferred on the same network. This may influence the measurement results. To avoid this influence, DBS controls command/result and measured traffic are not transferred at the same time. DBS implementation assumes that clocks on all the hosts participating to the benchmark are synchronized. 4.1.2 Automation
Commands of execution are driven by a command file. Format of the command file is as follows. When multiple data streams are transferred in the same test, many configurations should be written in the same file.
# First Configuration { sender {’configurations of sender’; ’sending traffic pattern {s}’;} receiver {’configurations of receiver’;’receiving message pattern {s}’;} ’configuration of the connection’; }
CESNET technical report number 18/2003
4
# Second Configuration { sender {’configurations of sender’; ’sending traffic pattern {s}’;} receiver {’configurations of receiver’;’receiving message pattern {s}’;} ’configuration of the connection’; } . . .
Format of ’sending traffic pattern s’
pattern { data size, message size, interval, wait time; data size, message size, interval, wait time; . . . ; }
Figure 2: Patern parameters meaning The traffic is modeled as a sequence of data chunks called ”frames”. The size of each frame may vary. Each frame may consist of several messages. A single message is defined that it can be transferred in a single UDP datagram. If a frame is longer than the UDP maximum transfer unit (64KB), the frame is split into several messages. Between frames, there is a time gap called ”wait time” which implies the application overhead. The preparation time of each frame can be treated as this wait time. Moreover, this model controls the frame intervals. This frame interval can be used for modeling of an application level rate control. Command File Sample
CESNET technical report number 18/2003
5
# Sample file { sender { hostname = host1; port = 0; send_buff = 65535; recv_buff = 65535; mem_align = 2048; pattern {2048, 2048, 0.0, 0.0} } receiver { hostname = host2; port = 20001; recv_buff = 65535; send_buff = 65535; mem_align = 2048; pattern {2048, 2048, 0.0, 0.0} } file = test1; protocol = TCP; start_time = 0.0; end_time = 30; send_times = 2048; }
See http://www.kusa.ac.jp/ yukio-m/dbs/dbs man.html for more information about constructing command file. 4.1.3
Settings possibilities send/receive buffer size modification setting the TCP no delay option
CESNET technical report number 18/2003
specify the starting time of data transfer for each connection specify frame size and inter-frame time gap specify test duration sending complicated traffic patterns
6
Through ”tcp trace” function, several TCP internal information such as TCP/IP pseudo-headers, TCP congestion window size, round trip time, retransmission timeout and other values in the TCB structure can be obtained. 4.1.4 Availability
See http://www.ai3.net/products/dbs1 for details of precise OS versions supported, and for download of the source code. Current implementation supports BSDI BSD/OS, FreeBSD, NetBSD, Linux, mkLinux, SunOS, IRIX, Ultrix, Digital UNIX, NEWS OS, HP-UX. 4.1.5 Required Environment
C language compiler, UNIX-style socket API support. 4.1.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames Checks only sequence numbers of received frames. Bidirectional traffic It could be made of two single traffics between specified hosts running at the same time in the opposite direction. Setting inter-frame time gap Full Support 4.1.7 References
Performance and Control of Network Systems, Proceedings of SPIE, Volume 3231, November 1997(English) ”DBS: a powerful tool for TCP performance evaluations”2 Informating Processing Society of Japan, DPS, 95-DPS-71, July 1995 (Japanese) ”A proposal of DBS: performance evaluation for TCP over multipoint connection”3 Internet Conference’ 96, July 1996 (Japanese) ”Design and Implementation of DBS: a performance evaluation system for TCP” 4
1 2
http://www.ai3.net/products/dbs http://www.kusa.ac.jp/ yukio-m/papers/dbs paper.ps 3 http://www.kusa.ac.jp/ yukio-m/papers/dps9507.ps 4 http://www.kusa.ac.jp/ yukio-m/papers/conf96.ps
CESNET technical report number 18/2003
7
Master’s Thesis, Graduate School of Information Science, Nara Institute of Science and Technology, March 7, 1996 (Japanese) ”Design and Implementation of DBS: a performance evaluation system for multipoint TCP connections”5 ”Digest of Master’s Thesis”6
4.2
4.2.1
IPerf 1.7.0
Description
IPerf is a ttcp like tool with considerable advantages over it. Using a clientserver model to determine maximum bandwidth you can also measure delay jitter, packet loss, determine MTU, support TCP window size, run tests by amount transferred or for a specified period of time. Server can handle multiple simultaneous connections. Client can create UDP streams of specified bandwidth. Client-server model can use for testing bidirectional mode called ”dual testing mode”. IPerf uses representative streams to test out how link layer compression affects your achievable bandwidth and prints periodic intermediate bandwidth, jitter, and loss reports at specified intervals. As one of the few also supports IPv6. 4.2.2 Automation
It is command-line driven. Use the -D command line option to run the server as a daemon and redirect the output to a file. E.g. iperf -s -D > iperfLog. Command line samples:
node2> iperf -s -u -l 32k -w 128k -i 1
-s = run IPerf in server mode
-u = use UDP instead TCP
-l 32k = buffer length
-w 128k = largest receivable datagram size
-i 1 = interval time in seconds between periodic reports
node1> iperf -c node2 -b 10m -l 32k -w 128k
5 6
http://www.kusa.ac.jp/ yukio-m/papers/mthesis.ps http://www.kusa.ac.jp/ yukio-m/papers/mthesis digest.ps
CESNET technical report number 18/2003
8
-c = run IPerf in client mode
node2 = server address
-b 10m = UDP bandwidth to send at, in bits/sec - 10Mbit/sec
4.2.3
Settings possibilities send/receive buffer size modification specify TCP maximum segment size
4.2.4
Iperf is released as a distribution of the C++ source. Pre-compiled binaries for selected operating systems are also available (Linux, FreeBSD, IRIX, MacOS, MS Windows, OpenBSD, Solaris). See http://dast.nlanr.net/Projects/Iperf/7 for more information about program, for details of precise OS versions supported, and for download of the source code. 4.2.5 Required Environment
C++ non cross-compiler
7
http://dast.nlanr.net/Projects/Iperf/
CESNET technical report number 18/2003
setting the TCP No Delay option UDP server provides multicast mode bidirectional testing mode tradeoff testing mode (request/response test) setting the number of simultaneous connections to make to the server (requires thread support on both the client and server) specify the type-of-service (as defined in RFC 1349) for outgoing packets specify the time-to-live for outgoing multicast packets use a representative stream (from file or stdin) to measure bandwidth specify test duration Availability
9
4.2.6
RFC 2544 compatibility
User defined frame format Iperf can specify only data content of UDP packet in frame definition. Verifying received frames It determines packet loss only. Bidirectional traffic Full Support Setting inter-frame time gap No Support
4.3
4.3.1
NetPerf 2.2pl4
Description
NetPerf is a benchmark that can be used to measure the performance of many different types of networking. It provides tests for both unidirectional throughput and end-to-end latency. It also includes provisions for CPU utilization measurement. Its primary focus is on bulk data transfer and request/response performance using either TCP or UDP and the BSD Sockets interface. There are optional tests available to measure the performance of DLPI, Unix Domain Sockets, the Fore ATM API and the HP HiPPI LLA interface. NetPerf is designed around the basic client-server model. There are two programs - netperf and netserver. The first thing after running program is establishing a control connection. This connection is used to pass test configuration information and results to and from remote system. After this process is established new connection - measurement connection. The test is performed and the results are displayed. NetPerf places no traffic on the control connection while a test is in progress. NetPerf provides three types of transfers: Bulk Data Transfer - also referred to as ”stream” or ”unidirectional stream”. This test measures how fast one system can send data to another and/or how fast that other system can receive it. Request/Response Transfer - request/response performance is quoted as ”transactions/sec” for a given request and response size. A transaction is defined as the exchange of a single request and a single response. Connect/Request/Response - instead of simply measuring the performance of request/response in the same connection, it establishes a new connection for each request/response pair.
CESNET technical report number 18/2003
10
NetPerf includes test which use a socket interface to IPv6, but the control connection remains IPv4. 4.3.2 Automation
Execution as child of inetd requires editing of /etc/services and /etc/inetd.conf. To assist in measuring, script files are provided with the NetPerf distribution (script for measuring stream performance, script for measuring request/response performance ...). NetPerf is command-line driven. Command line samples:
node1> netserver -p 20000 -n 2
-p = listen on the specified port
-n = number of CPU’s in the system
node2> netperf -t TCP_STREAM -H node1 -- -s 16384 -S 16K
-t = test name to perform (script filename)
-H = name of the remote system
-s = local send and receive socket buffer size
-S = the same as -s, but for remote system
4.3.3
Settings possibilities send/receive buffer size modification of both systems (local/remote) setting TCP No Delay option
CESNET technical report number 18/2003
setting the size of a burst of packets (used to pace the send rate when is no flow control provided by the protocol being measured) specify test duration pre-fill buffers with data from file CPU rate calibration
11
4.3.4
Availability
See http://www.netperf.org/netperf/NetperfPage.html 8 for more details or email Rick Jones ([email protected]). Binaries are available here for HP/UX Irix, Solaris, and Win32. 4.3.5 Required Environment
C language compiler, sockets. 4.3.6 RFC 2544 compatibility
User defined frame format NetPerf can only specify data content of UDP packet in frame definition. Verifying received frames No Support Bidirectional traffic No Support Setting inter-frame time gap Full Support
4.4
4.4.1
NetPIPE 3.3
Description
NetPIPE (Network Protocol Independent Performance Evaluator) is a protocol independent performance tool for comparing different networks and protocols. NetPIPE performs simple ping-pong tests, bouncing messages of increasing size between two processes, whether across a network or within an SMP system. Message sizes are chosen at regular intervals, and with slight perturbations, to provide a complete test of the communication system. Each data point involves many ping-pong tests to provide an accurate timing. It also has an option to measure performance without cache effects. NetPIPE consists of two parts: a protocol independent driver, and a protocol specific communication section. The communication section contains the necessary functions to establish a connection, send and receive data, and close a connection. This part is different for each protocol. However, the interface between the driver and protocol module remains the same. Therefore, the driver does not have to be altered in order to change communication protocols. NetPIPE is a variable time benchmark, which increases the transfer block size from a single byte until transmission time exceeds 1 second. For each block size
8
http://www.netperf.org/netperf/NetperfPage.html
CESNET technical report number 18/2003
12
c, three measurements are taken: c - p bytes, c bytes, and c + p bytes, where p is a perturbation factor with a default value of 3. This perturbation allows analysis of block sizes that are possibly slightly smaller or larger than an internal network buffer. NetPIPE uses a ping-pong transfer. This forces the network to transmit just the data block without streaming other data blocks in with the message. The result is the transfer time of a single block, thus providing the information necessary to answer which block size is best, or what is the throughput given a block of size k. NetPIPE produces a file that contains the transfer time, throughput, block size, and transfer time variance for each data point and is easily plotted by any graphing package. Some typical uses: Measuring the overhead of message-passing protocols.
Help in tuning the optimization parameters of message-passing libraries.
Identify dropouts in networking hardware.
Optimizing driver and OS parameters (socket buffer sizes, etc.).
4.4.2
Automation
NetPIPE is a command-line driven program. Command line samples:
node1> NPtcp -r -b 32768 -l 1 -u 1048576
-r = receiver
-b = send and receive TCP buffer sizes -l = lower bound of block size -u = upper bound of block size
node2> NPtcp -t -h node1 [options]
-t = transmitter -h = remote host
CESNET technical report number 18/2003
13
4.4.3
Settings possibilities specify TCP send/receive buffer sizes setting lower/upper bound of variable block size
4.4.4
See http://www.scl.ameslab.gov/netpipe available.
You can download the latest version NetPIPE 3.5.tar.gz 10 . Infiniband module for the Mellanox VAPI has been added, an integrity check option (-I) has been incorporated, and problems with the streaming mode have been fixed. 4.4.5 Required Environment
C language compiler 4.4.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames No Support Bidirectional traffic No Support Setting inter-frame time gap No Support 4.4.7 References
Quinn O. Snell, Armin R. Mikler, and John L. Gustafson: NetPIPE: A Network Protocol Independent Performance Evaluator, IASTED Conference Paper
9 10
http://www.scl.ameslab.gov/netpipe http://www.scl.ameslab.gov/netpipe/code/NetPIPE 3.5.tar.gz
CESNET technical report number 18/2003
specify increment step size specify output filename specify buffers alignment specify buffer offset setting stream option (default mode is ”ping-pong”) Availability
9
for more details. Binaries are not
14
4.5
4.5.1
NetSpec 3.0
Description
NetSpec is a network level end-to-end performance evaluation tool for Network Experimentation and Measurement. It provides a fairly generic framework that allows a user to control multiple processes across multiple hosts from a central point of control. It uses a scripting language that allows the user to define multiple traffic flows from/to multiple computers. This allows an automatic and reproducible test to be performed. NetSpec exhibits many features like parallel and serial multiple connections, a range of emulated traffic types (FTP, HTTP, MPEG, etc.) on the higher levels, the most widely used transport protocols today, that is TCP and UDP, three different traffic modes, scalability, and the ability to collect system level information from the communicating systems as well as intermediate network nodes. The following figure shows the basic NetSpec architecture.
Figure 3: NetSpec architecture The controller is a process that supports the user interface, which is currently a file containing a description of an experiment using a simple block structured language in which the connection is the basic unit for an experiment and via the control daemon controls the daemons implementing the test. For every connection in the experiment, the corresponding test daemons are created. These test daemons send or receive data transferred across the connection. Each daemon is responsible for its own report generation after experiment
CESNET technical report number 18/2003
15
execution is complete. The output report is delivered to the controller via the control daemon for viewing by the user. NetSpec supports three basic traffic modes: <h4>Full Stream Mode</h4> Also known as ”full blast mode”, it instructs the test daemons to transmit data as fast as possible. <h4>Burst Mode</h4> The hosts under test transmit data every some specific intervals, specified by the burst period. The burst size and the burst period is passed as a parameter by the user (in blocks/burst and bytes/block). This mode is very useful in real world experiments where rate mismatches might reduce the throughput dramatically. <h4>Queued Burst Mode</h4> The hosts under test transmit data every some specific intervals, specified by the burst period. This mode is a variation of the basic burst algorithm and the burst size and burst period are passed as a parameter by the user. The advantage of this algorithm is that variations in available line rate will not cause it to miss blocks generated by interrupts arriving before previous write completes. The drawback is that characteristics of the traffic are influenced by the queuing delay. 4.5.2 Automation
NetSpec consists of several daemon types that are started and controlled by the main netspec daemon (netspecd). NetSpec uses a scripting language that allows the user to define multiple traffic flows from/to multiple computers. This allows an automatic and reproducible test to be performed. A test would be started with:
Netspec script.name
The script given below is of the most widely type used for a TCP point-topoint connection. The first block specifies the characteristics of the traffic source, while the second block specifies the characteristics of the receiver host. Script file sample
1 2 3 cluster { test host1 { type = full (blocksize=32768, duration=30);
CESNET technical report number 18/2003
16
4 5 6 7 8 9 10 11 12 13 14
protocol = tcp (window=32768); own = host1:20200; peer = host2:20200; } test host2 { type = sink (blocksize=32768, duration=30); protocol = tcp (window=32768); own = host2:20200; peer = host1:20200; } }
Figure 4: Test setup with the invoked daemons over TCP In this particular example, the host with name host1 is the sender system and the host with name host2 is the receiver system. The sender (line 2) sends data in full stream (line 3) mode; it transmits 32768 bytes as fast as possible for 30 seconds (duration of test). For this data transfer TCP (line 4) is used in the transport layer with a window size of 32.7KB. NetSpec can be installed in either of the two ways - inetd installation and standalone installation. 4.5.3
Settings possibilities specify size of frames and inter-frame time gap in burst modes setting traffic mode (full stream mode, burst mode or queued burst mode)
CESNET technical report number 18/2003
specify type of connection (point-to-point, multipoint-to-point connections)
point-to-multipoint or
17
setting connection mode (serial, parallel or cluster)
specify test duration
specify protocol type
4.5.4
Availability
See http://www.ittc.ku.edu/netspec 11 for more details. NetSpec can run on a variety of platforms. The following binaries are available: Digital Unix, Solaris, Linux, SunOs, FreeBSD, Irix. NetSpec manual can be found at http://www.ittc.ukans.edu/netspec/docs/NetSpecUser.pdf
12
4.5.5
Required Environment
C language compiler 4.5.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames No Support Bidirectional traffic It could be made of two single traffics between specified hosts running at the same time in the opposite direction. Setting inter-frame time gap Full Support
4.6
4.6.1
RUDE & CRUDE 0.70
Description
RUDE (Real-time UDP Data Emitter) is a small and flexible program that generates traffic to the network, which can be received and logged on the other side of the network with the CRUDE (Collector for RUDE). These programs can generate and measure only UDP traffic. To observe several variables describing the utilization of hardware resources (CPU load, number of interrupts etc.) can be used another free software package - atsar. It can record in regular intervals the
11 12
http://www.ittc.ku.edu/netspec http://www.ittc.ukans.edu/netspec/docs/NetSpecUser.pdf
CESNET technical report number 18/2003
18
values of various system counters and parameters together with timestamps, for example the CPU load and number of interrupts generated by the network interface cards. The rude/crude distribution contains several example Perl scripts for basic processing of the decoded crude output and computing jitter. Rude allows definitions of a number of concurrent UDP flows with varying packet size and rate. The TOS field in the IP header can also be set. Reordered and duplicated packets may arrive with arbitrary delays, hence a bitmask with 32 bits describing the fate of the last 32 packets of the flow is kept. Two types of generated flow are implemented - constant and trace. Constant flow means a constant-rate UDP flow, where the packet rate per second and packet size in bytes can be specified. The trace option gives a reference to a text file where the parameters of every single packet has to be given. The smallest time unit for flow definition is 1 ms. 4.6.2 Automation
RUDE is driven by a script file, which is used to specify the generated flows. Example of a simple script file follows:
START NOW ## FLOW 1: (flow ID = 25) ## ## Starts immediately at the specified START time with following parameters: ## 400 packets/second with 100 bytes/packet = 40kbytes/sec (1kbyte=1000bytes) ## ## Sets the TOS for this flow to LOW_DELAY (0x10) ## ## 9 seconds after that the flow is turned off... ## 0000 25 ON 3001 10.1.1.1:10001 CONSTANT 400 100 TOS 25 0x10 9000 25 OFF ## FLOW 2: (flow ID = 1) ## ## This flow acts as specified in the TRACE configuration file. ## 0000 1 ON 3002 10.1.1.1:10001 TRACE trace_file.txt 9999 1 OFF
Here, the flows 25 and 1 (second field in each row) are specified. The numeric values at the beginning of the rows are time offsets related to a predefined
CESNET technical report number 18/2003
19
global origin when the flow is to be started or its parameters modified. The destination IP address and port are defined as 10.1.1.1 and 10001, respectively, and the source port as 3001, respectively 3002. The source IP address will be determined at run-time as the address of egress interface. The second flow uses for packets definition the external file - trace file.txt. The following command line samples shows, how to run the specified script file. Command line samples:
sender> rude -s sample.cfg
-s <script file> = defines the script file to be used
receiver> crude
4.6.3
Settings possibilities control the length of the test specify the type-of-service (TOS) for outgoing packets
4.6.4
For more details about installation and using RUDE/CRUDE see http://rude.sourceforge.net.13 On this website can be downloaded also the old releases of this utilities. The newest release can be found at http://gd.tuwien.ac.at/opsys/linux/sf/r/rude/14 .
13 14
http://rude.sourceforge.net http://gd.tuwien.ac.at/opsys/linux/sf/r/rude/
CESNET technical report number 18/2003
plenty of flows definition TRACE flow - definition of packet size and time gap between packets (max. time resolution = 1 microsecond) CONSTANT flow = constant bit rate traffic (you may change packet size and packet rate) calculate some statistics on-the-fly setting the process real-time priority some visualization and statistical analysis - using grude script Availability
20
4.6.5
Required Environment
C language compiler 4.6.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames No Support Bidirectional traffic No Support Setting inter-frame time gap Full Support 4.6.7 References
Lhotka L.: Software tools for router performance testing, Technical Report 10/2001, CESNET, Prague, 2001.
4.7
4.7.1
TRENO (07/30/97)
Description
TRENO (Traceroute RENO) is a TCP throughput measurement tool, which is based on sending UDP packets with low TTL in patterns that are controlled at the user-level. Hosts and routers along the path to the final destination will send back ICMP TTL Exceeded messages which have similar characteristics to TCP ACK packets. TRENO also has an ICMP mode, which uses ICMP ECHO Requests instead of low TTL UDP packets. In this mode, you only get information about the final destination. The same sized packets are sent in both directions, giving you some information about the return path (request-response test). This allows to measure throughput independent of the TCP implementation of end hosts. TRENO has some limitations: Each hop should run for at least 10 seconds.
Some routers do not respond to the Treno probes as quickly as they forward packets. So the results of the Treno test will not accurately reflect the bandwidth at these hops. The Treno Server is single threaded, so if someone else is running tests you will need to wait until they complete their work.
CESNET technical report number 18/2003
21
For more details about these limitations, see TRENO’s homepage. 4.7.2 Automation
Command-line driven. No ”server” is required, and it only requires a single argument of the machine to run the test to. Command line samples:
sender> treno -p 10 hostname
-p < s > = set the test duration
4.7.3
Settings possibilities specify test duration specify the (initial) MTU
4.7.4
See http://www.psc.edu/networking/treno info.html15 or e-mail Matt Mathis ([email protected]) or Jamshid Mahdavi ([email protected]). 4.7.5 Required Environment
C compiler, raw sockets. 4.7.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames Checks only sequence numbers of received frames. Bidirectional traffic No Support Setting inter-frame time gap No Support
15
http://www.psc.edu/networking/treno info.html
CESNET technical report number 18/2003
specify used mode (ICMP or UDP packets) Availability
22
4.8
4.8.1
TTCP6 Revision: 3.8
Description
Originally written to move files around, TTCP became the classic throughput benchmark or load generator, with the addition of support for sourcing to/from memory. It can also be used as a traffic absorber. It has spawned many variants, recent ones include support for UDP, IPv6, data pattern generation, page alignment, and even alignment offset control. 4.8.2 Automation
It is the command-line driven tool. To use it, start the receiver on one side of the path, then start the transmitter on the other side. The transmitting side sends a specified number of TCP packets to the receiving side. At the end of the test, the two sides display the number of bytes transmitted and the time elapsed for the packets to pass from one end to the other. Command line samples:
receiver> ttcp6 -r -s -v -n100
-s (ttcp6 -r) : sink (discard) all data from network
sender> ttcp6 -t -s -v -n100 host
-s = (ttcp6 -t) : source a pattern to network
-v = verbose: print more statistics
-n = number of source buffers written to network
4.8.3
Settings possibilities send/receive buffer size modification setting the TCP no delay option
CESNET technical report number 18/2003
using UDP instead of TCP setting socket buffer size
23
4.8.4
Availability
See ftp://ftp.arl.mil/pub/ttcp/16 which includes the most common variants available or e-mail ARL ([email protected]). Download the latest version with IPv6 support from ftp://ftp.bieringer.de/pub/linux/IPv6/ttcp/ttcp+ipv6-3.tar.gz 17 . 4.8.5 Required Environment
C compiler, BSD sockets. 4.8.6 RFC 2544 compatibility
User defined frame format No Support Verifying received frames No Support Bidirectional traffic No Support Setting inter-frame time gap No Support
5
5.1 5.2
Summary
RFC 2544 compatibility IPv6 support
6
Conclusion
As we can see in the section 5 5 (Summary), none of the performance tools listed above, supports all of the requirements mentioned in RFC 2544 and described in section 3 3. Also IPv6 support is not an ordinary character. Only three of these tools support IPv6 protocol. The DBS 4.1 (Distributed Benchmark System) testing tool seems to be the best of these performance testing tools we have tested. This utility allows a user to control multiple processes across multiple hosts from a central point of control.
16 17
ftp://ftp.arl.mil/pub/ttcp/ ftp://ftp.bieringer.de/pub/linux/IPv6/ttcp/ttcp+ipv6-3.tar.gz
CESNET technical report number 18/2003
24
Figure 5: Summary of RFC 2544 compatibility
Figure 6: Summary of IPv6 support
CESNET technical report number 18/2003
25
It uses a scripting language that allows the user to define multiple traffic flows from/to multiple computers. This allows an automatic test to be performed. It also supports definition of inter-frame time gap. Of course we have to modify this auxiliary utility, so it can fulfill all of necessary requirements. Verifying received frames is not well done. It checks only sequence numbers of received frames, so the complete verification of received frames as defined in section 3 3 have to be made. A possibility of sending user defined frame format should be appended to the DBS utility too, because we need to generate special traffic. An alternative testing tool we can modify and use is NetSpec 4.5. The conclusive factor is simplicity of modifying the source codes of these tools.
References
[1] [2] Parker S., Schmechel C.: Some Testing Tools for TCP Implementors RFC 2398, August 1998 Bradner S., McQuaid J.: Benchmarking Methodology for Network Interconnect Devices RFC 2544, March 1999 Bradner S.: Benchmarking Terminology for Network Interconnection Devices RFC 1242, July 1991. DBS WWW pages http://www.ai3.net/products/dbs18 IPerf WWW pages http://dast.nlanr.net/Projects/Iperf19 NetPerf WWW pages and NetPerf manual included in archive http://www.netperf.org/netperf/NetperfPage.html20 NetPIPE WWW pages http://www.scl.ameslab.gov/netpipe21 NetSpec WWW pages http://www.ittc.ku.edu/netspec22
[3]
[4] [5] [6] [7] [8]
18
http://www.ai3.net/products/dbs http://dast.nlanr.net/Projects/Iperf 20 http://www.netperf.org/netperf/NetperfPage.html 21 http://www.scl.ameslab.gov/netpipe 22 http://www.ittc.ku.edu/netspec
19
CESNET technical report number 18/2003
26
[9] [10]
NetSpec User Manual http://www.ittc.ukans.edu/netspec/docs/NetSpecUser.pdf23 RUDE & CRUDE WWW pages and the documentation added to the source code http://rude.sourceforge.net24 Treno WWW pages http://www.psc.edu/networking/treno info.html25 TTCP documentation added to source code
[11] [12]
23 24
http://www.ittc.ukans.edu/netspec/docs/NetSpecUser.pdf http://rude.sourceforge.net 25 http://www.psc.edu/networking/treno info.html
CESNET technical report number 18/2003
27
