Tech: Setting up and Running a Siege-based Sustained Load Test...

So its been a while since I had to do any kind of load test, and as chance would have it, I needed to generate around 2,000 hits/s against a remote server sustained for a couple of days.  It turned out to be a bit more complicated than I expected so I decided to share my setup with others and aggregate some disparate information.

1. First, to get the siege source, download it from joedog (yes, its legit).

curl -C - -O http://www.joedog.org/pub/siege/siege-latest.tar.gz

2. Extract and uncompress the tarball

tar -xvf siege-latest.tar.gz

3. cd to the new directory (version 2.7 in this instance)

cd siege-2.70/

4. Make sure that the user you are logged in as can write to /usr/local where the program will be installed to, and then execute the following commands in order

./configure; make; make install;

OSX HINT: install the XCode Command Line Tools before building siege by launching XCode -> Preferences -> Downloads .  Then select Install next to Command Line Tools .

5. Create the siege config file in your home directory

/usr/local/siege.config

6. Edit the values in $HOME/.siegerc, specifically the value for failures should be quite high generating this kind of load from a single app server

failures = 1000000

7. Run a test to make sure everything is working properly

siege -c1 -r1 'http://www.google.com'

========

8. Next make sure you have enough File Descriptors to be able to handle the network socket connections:

sudo echo "username soft nofile        2048" >> /etc/security/network.conf

sudo echo "username hard nofile         5020" >> /etc/security/network.conf

9. Verify the changes by logging back in as "username" and verify using

ulimit -n

2048

========

10. Now create a few scripts (startsiege, stopsiege, trimlogs):

#!/bin/bash

#############
#startsiege
#  this script works from a loop that will restart siege if the current process terminates 
#  prematurely due to a high number of errors (such as File Descriptors being exhausted).
#  It also looks for the presence of a file called "bar" to help exit cleanly if the stop script was used
############


echo "Preparing for assault...";
#Create data pipe for nohup output
pipeHandle="./siegepipe";
if [ -e "$pipeHandle" ];
then
  echo -n "File handle exits...";
else
  echo -n "Creating new file handle for data pipe...";
  mknod ./siegepipe p;
fi
echo "Continuing...";

#Create a log dir if one does not exist
if [ -d ./log ]; then
  echo -n "Log dir exists...";
else
  echo -n "Creating new log dir...";
  mkdir ./log;
fi
echo "Continuing...";

#Redirect output from the pipe to a file
nohup cat < ./siegepipe >./log/siege.log > /dev/null 2>&1 &
catPID=$!;
echo "$catPID" > ./cat.pid;

#Start the siege process
echo -n "File Descriptors: ";
ulimit -n
echo "Initiating assault...";
grep foo bar > /dev/null 2>&1;
while (($? != 0)); do
  date;
  nohup siege -c2000 -t4H 'http://crackme.edgeplatform.com/?cacheH=on&test=monkey' >./log/siege.log &
  #nohup siege -c2000 -t4H 'http://crackme.edgeplatform.com/?cacheH=on&test=monkey' >./log/siege.log > /dev/null 2>&1 &
  siegePID=$!;
  echo "$siegePID" > ./siege.pid;
  wait $siegePID;
  grep foo bar > /dev/null 2>&1;
 if [[ $? = 0 ]]; then
   rm bar;
   exit 0;
 else
   echo "Aborted due to errors...Cleaning Up.";
   echo "Rotating log...";
   mv ./log/siege.log ./log/siege.log$$;
   echo "Re-establishing output redirect...";
   kill -9 $catPID;
   nohup cat < ./siegepipe >./log/siege.log > /dev/null 2>&1 &
   catPID=$!;
   echo "$catPID" > ./cat.pid;
   echo "Starting a new assault...";
   grep foo bar > /dev/null 2>&1;
 fi
done;


#!/bin/bash


#############
#stopsiege
#  This script creates the file bar used by the start script to exit cleanly, and kills
#  the process id of siege
############

echo "foo" > bar;
echo "Stopping Assault...";
kill -9 `cat ./siege.pid`;
echo "Stopped...";
echo -n "Cleaning up...";
# Stop Output Reidrect
kill -9 `cat ./cat.pid`;
# Remove pipe
rm ./siegepipe;
# Remove logs
rm -rf ./log/siege.log*;
rmdir ./log;
# Remove PID files
rm ./siege.pid ./cat.pid;
echo "Done...";



#! /bin/bash


#############
#trimlogs
  This script trims any logs that are over 15 minutes in age 
# in case there is a sudden error runup
############

while (true); do
  echo "Trimming archived error logs older than 15 minutes...";
  find ./log -name "siege.log[0-9]*" -cmin +15 -exec rm -v '{}' ';' ;
  sleep 900;
done;



========

To run simply use ./startsiege

 $ ./startsiege
Preparing for assault...
Creating new file handle for data pipe...Continuing...
File Descriptors: 2048
Initiating assault...
Mon Dec 10 12:18:10 PST 2012
nohup: redirecting stderr to stdout
Aborted due to errors...Cleaning Up.
Rotating log...
Re-establishing output redirect...
Starting a new assault...
./startsiege: line 49: 15049 Killed                  nohup cat < ./siegepipe > ./log/siege.log > /dev/null 2>&1
Mon Dec 10 13:04:30 PST 2012
nohup: redirecting stderr to stdout
Aborted due to errors...Cleaning Up.
Rotating log...
Re-establishing output redirect...
Starting a new assault...
./startsiege: line 49: 17156 Killed                  nohup cat < ./siegepipe > ./log/siege.log > /dev/null 2>&1
Mon Dec 10 13:51:46 PST 2012
nohup: redirecting stderr to stdout

To trim logs while siege is running, use ./trimlogs from a separate shell

 $ ./trimlogs

Trimming archived error logs older than 15 minutes...

removed `./log/siege.log15047'

To stop, use ./stopsiege

$ ./stopsiege

Stopping Assault...

Stopped...

Cleaning up...Done...

========

Thats all there is to it!  You have increased the file descriptors to reduce errors, created error handling that will restart the process if it dies unexpectedly, and trimmed the error logs that get generated when the process is nohup'd.