Okay, this is all very much preliminary, and I appreciate any useful commentary, especially wrt to tuning on different operating systems. I have tried my best to pull any tricks I know of, but I am far from knowing enough about the individual operating systems

Update: After tweaking the FreeBSD kernel and recompiling, numbers got a bit better

Overview

This page collects and represents the benchmark data I have collected while implementing and porting my streaming server (flss, FIXME: link to download the sources) to different platforms. Currently it is known to be working on the following platforms:

• Linux Kernels 2.4.x/2.6.x; i386 and ppc
• FreeBSD 4.7 and FreeBSD 5.2; i386
• Darwin 7.x (aka Mac OS X); ppc

Additionally it *should* run under the aforementioned operating systems on different hardware architectures (however at a significant performance penalty, see the source of the event dispatcher for an explanation why). It *should* be easily portable to all other Unix-like platforms. A native Windows port is being prepared.

Benchmark data for the above mentioned hardware/software combinations have been collected and will be publicised here, and it will hopefully be updated as the platforms evolve.

Test Systems

The following test hardware for the streaming server is available to me (and all benchmarks are run on the server):

• x86: Dual Intel Pentium III, 933MHz (home-grown system)
• ppc: Dual Motorola PPC 7455, 1GHz (Apple Powermac G4)
• Intel E1000 Gigabit NIC (64 Bit PCI Card)
• Apple GMAC Gigabit Ethernet NIC (built-in Apple Powermac G4)

The systems were used with the following hardware/software combinations:

It should be noted that the x86 system is far older than the ppc system, and the hardware in the ppc system is far more powerful than the x86 system in almost all respects.

System specific notes

• linux-i386/linux-ppc: The epoll system call is available only in the 2.6 series of the Linux kernel; a patch implementing this functionality for the 2.4 kernels was applied
• freebsd-i386: Two different thread implementations exist for the FreeBSD 5.2 system: traditional user-space threads which cannot make use of multiple CPUs; and KSE-based threads which can make use of multiple CPUs, but apparently incurs a noticeable performance overhead. However, the target application would not run correctly with user-space threads, so KSE is a requirement (strangely enough, it *did* run correctly on FreeBSD 4.x series...).

LMBench

Results. This benchmark measures basic OS parameters, such as system call overhead. Annotations:

• FreeBSD uses tcsh as default shell, whereas all other systems use bash; the "shell proc" timings should therefore be taken with care
• Porting lmbench to windows is very difficult because of the very different native operating system API; there is not much point in measuring the performance of a Posix emulation subsystem as it can reasonably be expected without any testing to be inferior to using native API calls.

Dispatcher bench

This benchmark tries to measure the efficiency of the operating system at delivering socket IO events to applications. For this purpose either one or two dispatcher threads are created, and a given number of UDP sockets for each thread. Both threads will then wait for events on each of "their" sockets, and upon receiving data will transmit the data to the next of "its" sockets. An intial message of 16 bytes is posted to the first socket of each thread, so that both threads are in effect busy passing a token of 16 bytes to itself via the UDP sockets.

This benchmark stresses both the network subsystem and the event notification mechanism of the operating system at the same time. It also exposes scalability problems on SMP systems - although two dispatcher loops could in theory be running completely independently, some interlocking between the two dispacher threads is absolutely necessary inside the kernel to guarantee the integrity of the operating system (e.g. file descriptor table, network stack, thread state, scheduler, ...).

The benchmark measures the number of dispatching operations (receive notification from kernel; receive data from kernel; send data to next socket) the system was able to perform per second. In case of two threads running in parallel the figures for both threads were added to demonstrate the throughput of the system as a whole.

Results. Annotations:

• Mac OS X consistently crashed while running this benchmark. Probability of a crash increased with the number of active sockets, and it also greatly increased when running two dispatcher threads in parallel. It was therefore impossible to get any meaningful numbers for large numbers of active UDP sockets.
• The behaviour of Mac OS X in the two-thread benchmark was very erratic, and the resultast numbers are hardly consistent. They were far more stable in the single-threaded case.
• During the single-threaded benchmark runs both Mac OS X and FreeBSD exhibited equal load on both CPUs. This indicates that the scheduler was unable to keep the running threads on one CPU to utilize the cache affinity of the threads. This probably explains the bad benchmark numbers both FreeBSD and Mac OS X are receiving in comparison to Linux.
• During the multi-threaded benchmark all operating systems show almost full load on both CPUs; this indicates that all systems have the basic infrastructure in place to run the threads in parallel on both CPUs. However the poor (or rather, non-existant) scalability of FreeBSD and Mac OS X indicate that the locking between the threads inside the kernel does not allow to utilize the second CPU in any useful fashion during this benchmark.

RTP streaming test

The efficiency of the server serving multiple clients is exercised during this test. Still preliminary due to some hardware problems.

The bottom line however is that flss has no trouble concurrently serving around 400 clients @1150 kbit/s sustained on a moderately-specced machine.

Results. Annotations:

Windows-specific notes

The most efficient way to do network IO and event notification on the Windows platform are Completion Ports. Unfortunately this IO model does not match the Unix-style separation of "readyness notification" and "IO request", which the streaming server is currently based on for dispatching. As a result it is impossible without major restructuring of the code to make use of Completion Ports, so only the "second-best" dispatching mechanism can be used on Windows; performance numbers for windows are therefore to be taken with great care.

Conclusions

This section could as well be titled "food for flamewar", but... I will postpone writing this until all source material is presented on this website

I appreciate comments