Automatically kill a process if it exceeds a given amount of RAM

killf.sh

I would strongly advise not to do it. As suggested by @chrisamiller , setting ulimit will limit the RAM available with process.

But still if you are insisting then follow this procedure.

Save the following script as killif.sh:
#!/bin/sh

if [ $# -ne 2 ];
then echo "Invalid number of arguments"
exit 0
fi

while [ 1=1 ];
do

SIZE=`pmap $1|grep total|cut -d" " -f13`
SIZE=${SIZE%%K*}
SIZEMB=$(($SIZE/1024))
echo "Process id ="$1" Size = "$SIZEMB" MB"
if [ $SIZEMB > $2 ];
then echo "SIZE has exceeded.\nKilling the process......"
`kill -9 $1`
echo "Killed the process"
exit 0
else
echo "SIZE has not yet exceeding"
fi

sleep 10
done
Now make it executable.

chmod +x killif.sh
Now run this script on terminal. Replace PROCID with actual process id and SIZE with size in Mb.

./killif.sh PROCID SIZE

For example: ./killif.sh 132451 100
If SIZE is 100 then process will be killed if it's ram usage goes up beyond 100 MB.

Caution: You know what are you trying to do. Killing process is not a good idea. If that process has any shutdown or stop command then edit the script and replace kill -9 command with that shutdown command

http://askubuntu.com/questions/54747/automatically-kill-a-process-if-it-exceeds-a-given-amount-of-ram