Fixing SSH timeout problems

This post is about resolving ssh idle timeout issues.

Foreword

Using SSH to connect to a remote unix host is invaluable, but most users are frustrated by the default behaviour where sessions left idle for a short time are automatically disconnected. Here I'll talk about some strategies for overcoming this.

Preventing auto SSH disconnection on the server.
This strategy involves adjusting the configuration of the server's ssh daemon, so you'll need root access to the server for this.
The ssh daemon is the server side service that allows ssh connections to be made, we can configure it by adjusting it's configuration file:

vi  /etc/ssh/sshd_config

locate the line:
#ClientAliveInterval 0
and modify it to:

ClientAliveInterval 60

Save the file.
Reload the new sshd config with:
service sshd reload

This adjustment causes the server to send a keep-alive message to the client every 60 seconds. As long as your client is still connected this is enough to prevent the connection from being idle and closing.
The default value of 0 meant that no keep-alive message was ever sent and your connection dies from being idle.

Read more at
man sshd_config

Preventing auto SSH disconnection on the client.

Linux/Unix Clients
Similar to the server side strategy presented above, here we adjust the ssh client configuration so it sends the keep-alive message to the server every 60 seconds.

vi  /etc/ssh/ssh_config

locate the line:
#ClientAliveInterval 0
and modify it to:

ClientAliveInterval 60

Save the file.

Windows Clients
Most windows users use PuTTY , probably because it's free ;-)

PuTTY Settings -> Connection

Set PuTTY to send keep-alive messages every 60 secs (as shown above).

Other Strategies

If you don't fancy any of the above, when you know you're going to leave your ssh connection idle for some time simply issue the top command:

top

This starts the interactive top program that auto refreshes your server state every few seconds and will be enough to stop a timeout. Hit q to exit top.


Conclusion
All strategies presented above achieve the same goal. The method you choose will largely be determined by whether you have root access to the server and whether you want changes to affect all clients connecting to the server.

Installing Tomcat7 with Java7 to Centos5 / RHEL5

This post describes how to install Tomcat7 with Java7 to a Centos5 / RHEL5 server.
The target here is run Tomcat on port 80 as a deamon service but not as root. I will however be using root privileges to install the software below.

Foreword
There are many ways to achieve the same goal of getting Tomcat to serve your dynamic content, I've been configuring the Tomcat servers on production websites since Tomcat3 and I've watched Tomcat mature as a product. I used to advocate putting Tomcat behind Apache httpd server but now I view Tomcat as mature and fast enough to listen on port 80 directly. There are still very legitimate reasons you may want to sit Apache in front of Tomcat, these include needing to use Apache mods or using Apache as a loadbalancer with multiple tomcats behind. I won't be covering those instances here.


Installing Java 7
We need to start by installing the JVM. Oracle provide a couple of different options: an RPM or a Compressed Binary. Here I use the RPM as it does slightly more for you.

My Centos box is 64bit, use uname -a , so I'll use 64bit java.

Download the rpm and optionally use md5sum to check the file, this is good practice as it ensures the file has transferred correct and hasn't been tampered with.

md5sum jdk-7-linux-x64.rpm

gives us 3c5c52922766ba365f83ee6a60dd2e60  jdk-7-linux-x64.rpm , as we expected

You may wish to uninstall you previous JVM, exercise caution if you have services using this JVM, transfer your web traffic and stop them.

Here I uninstall Java6/JDK1.6 . I use rpm -qa to locate the rpm package name and uninstall with:

rpm -e jdk-1.6.0_27-fcs

Install the new java:

rpm -ivh jdk-7-linux-x64.rpm

I now add JAVA_HOME to the environment, this is again good practice, but not strictly necessary if you are going to use the scripts we use below.

vi /etc/profile

and add to the bottom:

export JAVA_HOME=/usr/java/latest

relogin to shell and test with:

echo $JAVA_HOME

You will notice that the Java RPM has made the /usr/java/latest link to point to the latest JVM we can see this with the command ls -al /usr/java/

java -version

Should confirm Java is ready, we'll install Tomcat next:


Installing Tomcat 7

Download and md5sum the latest Tomcat7 to your home directory. this is apache-tomcat-7.0.21.tar.gz at time of writing.

Extract it:
tar xvzf apache-tomcat-7.0.21.tar.gz

Move the extracted folder:

mv apache-tomcat-7.0.21 /usr/share/

Set the CATALINA_HOME env variable, using the same technique we use to set JAVA_HOME above
export CATALINA_HOME=/usr/share/apache-tomcat-7.0.21

A primary objective here is to run Tomcat as the server on port 80 but not as the root user. Running a public facing process on a port as root is dangerous as if the service is compromised the attacker will be able to cause far more damage as root has unlimited permissions. Instead we'll make an arbitrary user tomcat to run the service under:

useradd tomcat

This will make a user and group called tomcat.

Set the tomcat installation to be owned by the tomcat user:
chown -Rf tomcat.tomcat /usr/share/apache-tomcat-7.0.21/

Change Tomcat to run on port 80

vi $CATALINA_HOME/conf/server.xml

Change all references of 8080 to 80, and comment out the Connector on port="8009" , this is used when we have Apache httpd server on port 80 in front of Tomcat.


Running Tomcat as a daemon

Tomcat ships with a nice tool that enables Tomcat to run as a unix daemon service as a user other than root, run the following commands to build jsvc:

cd $CATALINA_HOME/bin
tar xvfz commons-daemon-native.tar.gz
cd commons-daemon-1.0.x-native-src/unix
./configure
make
cp jsvc ../..
cd ../..

If you encounter any problems with configure or make, you may need to install additional components to Centos, use:

yum install gcc

and/or
yum install make

You will now have jsvc in your $CATALINA_HOME/bin folder.

Although it isn't in the docs Tomcat7 ships with a service script to operate jsvc:
$CATALINA_HOME/bin/daemon.sh

cd $CATALINA_HOME/bin/

Try starting Tomcat:
./daemon.sh start

You can check it's running by hitting port 80 with your web-browser or using

ps aux

to look for the tomcat process, you'll see the process running as non-root:
tomcat   26372  4.0 44.2 818540 464112 ?       Sl   14:25   6:31 jsvc.exec -java-home /usr/java/latest

or

netstat -nlp

to see the open ports, you'll see something like:

tcp        0      0 :::80      :::*     LISTEN      26372/jsvc.exec

If you have problems check the logs:

ls -al $CATALINA_HOME/logs

specifically:
catalina-daemon.out

Next step is to ensure Tomcat starts automatically on boot. We'll use the unix service facility to achieve this.

cd /etc/init.d

cp $CATALINA_HOME/bin/daemon.sh ./tomcat7

vi tomcat7

And replace the comment at the top of the file with:

    #!/bin/bash
    # description: Tomcat Start Stop Restart
    # processname: tomcat7
    # chkconfig: 234 20 80
    TOMCAT_USER=tomcat
    JAVA_HOME=/usr/java/latest
    export JAVA_HOME

    CATALINA_HOME=/usr/share/apache-tomcat-7.0.21

additionally you may add a line like:

CATALINA_OPTS="-Xms256m -Xmx512m"

to pass options such as memory limits to tomcat.

Add tomcat to levels 234 of startup:

chkconfig --add tomcat7

You can check it with:
chkconfig --list tomcat7

You can now control tomcat with the commands:

service tomcat7 start

service tomcat7 stop

Diagnostics:
service tomcat7 version

Try rebooting your Centos server, it should now auto restart tomcat running as the tomcat user.
shutdown -r now

Troubleshooting
Some problems I have seen on various boxes:


Jsvc reports Cannot find any VM in Java Home
"Cannot find any VM in Java Home" appears when starting jsvc, you may also see "Cannot locate JVM library file" in the logs.
I experienced this after building jsvc on a 64bit architecture server using a 32bit jvm, swapping the jvm for 64bit and rebuilding jsvc did the trick.


Firewall restricts access to port 80.
If it's iptables

iptables -I INPUT -p tcp --dport 80 -j ACCEPT

service iptables save

Conclusion
Hopefully this has helped you achieve the best of both worlds, you should now have Tomcat 7 listening on a privileged port but not running as a superuser. The service will also auto restart along with your server.
If for some reason you can't run jsvc you have a few options:
1) Use the 3rd party software authbind to allow a normal user access to a port below 1024.
2) Use iptables to reroute requests from port 80 to 8080.
3) Run apache httd along with a connector to proxy back to tomcat.

If you are successfully using Tomcat on port 80 you'll now most likely want to use virtual hosts to run multiple sites from a single server. I'll write about the pros and cons of this in the future...