Tomcat truncating cookie values problem

The Problem
When using Java running in the Tomcat servlet container you may observe cookie values fetched from javax.servlet.http.Cookie using getValue() seem incomplete.

Example
I like to parse various cookies in Java code, one particular high value cookie for me is Google Anaytics' Campaign Tracking Cookie the UTMZ.
UTMZ typically looks like:
__utmz=1.1322309384.1.1.utmcsr=anothersite.com|utmccn=(referral)|utmcmd=referral|utmctr=9371183|utmcct=/

However when reading and displaying this cookie in Java and Tomcat you'll see the value:
1.1322309384.1.1.utmcsr

This is because Tomcat 6.0.18 now adheres to the cookie spec more tightly than previous versions.
This may cause you multiple issues including:

• The value you expect to see is incomplete (a quick look with Google Chrome -> Inspect Element -> Network -> Headers -> Request Headers shows that the browser is sending the full cookie value to Tomcat).
• Code that did work on a previous Tomcat suddenly breaks.

Solution

If you are in control of the cookie, i.e. you write it then you need to ensure a version 0 cookie doesn't contain any of the following characters: equals =, parentheses (), colon : . 

If like the above example you are powerless to control the cookie format, you need to run Tomcat with some additional system properties set. The following added to catalina.properties will do the trick:

org.apache.tomcat.util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE=true

org.apache.tomcat.util.http.ServerCookie.ALLOW_HTTP_SEPARATORS_IN_V0=true

Further info can be found in the Tomcat Config Docs. 

Hosting multiple SSL sites on Tomcat 7 with Virtual Hosting

Foreword
Virtual Hosting is where you serve multiple websites from the same server. This post describes how to achieve virtual hosting with tomcat on linux where each site requires it's own certified SSL capability. As per my previous posts we're assuming Tomcat is your public facing server and you are using APR for performance thus negating the need to sit apache in front. (If you do have apache in front you'd configure SSL there instead).
I haven't found anywhere else on the web explaining how to do this so thought I'd share it with you.

Virtual Hosting Concepts
Virtual hosting comes in 2 guises:

1. Name-based Virtual Hosting: This is where all your sites can share a common IP and the server looks to the HTTP request header to decide what site to serve up. This is the most common type of virtual hosting.
2. IP-based Virtual Hosting: Here your sites each have their own distinct IP address. The server listens to multiple IP addresses to determine which site to serve. This type of virtual hosting is less common as it uses more IP addresses, which with IPv4 are scarce and have a cost implication.

The Problem
We are interested in setting up virtual hosting on tomcat with SSL, it's fine to use name-based hosting if all your sites want to share a single ssl certificate. However, this is rarely the case. With a shared cert users receive a scary browser warning if the ssl cert presented doesn't match the requested domain name exactly. For this reason for commercial sites you're going to need a separate, officially certified ssl certificate for each site.
The problem is that the SSL handshake between the client-server happens very early on, before the server looks at the HTTP request header: The server presents the ssl cert tied to the IP before it even looks at the website being asked for!

The Solution 
We must use multiple IP's in conjunction with name-based Virtual Hosting. I'm assuming your server has a single ip address and you only wish to host 2 ssl secured sites, this approach will scale but you'll need a new IP address assigned to your server for each additional SSL site you wish to host. Talk to you hosting provider about this as adding IPs is out of scope but basically once the IP is assigned to your server you'll need to cd /etc/sysconfig/network-scripts then add a file ifcfg-eth0:1 to bind the IP to your main Ethernet device, you don't need >1 NIC. Then service network restart

So now you have 2 public-facing IPs configured for your server, e.g.
10.0.0.1
10.0.0.2

Now you need to configure tomcat to listen on them:

vi $CATALINA_HOME/conf/server.xml

Set up your sites as Host elements as if you were using only Name-based Virtual Hosting, this is simple and well documented.
Now turn your attention to your connectors. It needs to look something like:

    <Connector port="80" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="443" />

    <Connector port="443" protocol="HTTP/1.1" address="10.0.0.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               SSLCertificateFile="/home/tomcat/ssl/www_domain1_com.crt"
               SSLCertificateKeyFile="/home/tomcat/sslkey/myserver.pem"
               SSLCertificateChainFile="/home/tomcat/ssl/www_domain1_com.ca-bundle"
               clientAuth="false" sslProtocol="TLS" />

    <Connector port="443" protocol="HTTP/1.1" address="10.0.0.2" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               SSLCertificateFile="/home/tomcat/ssl/www_domain2_com.crt"
               SSLCertificateKeyFile="/home/tomcat/sslkey/myserver.pem"
               SSLCertificateChainFile="/home/tomcat/ssl/www_domain2_com.ca-bundle"
               clientAuth="false" sslProtocol="TLS" />

Some explanation:

• The connector on port 80 will listen to all IP's by default.
• We launch 2 https connectors each listening to port 443 on a separate address. We specify the address attribute to dictate the ip they bind to, see The HTTP Connector.
• Each https/ssl connector is tied to it's own SSL Cert.
• We set our DNS A-records for the domains to ensure the requests go to the correct connector.

Conclusion
We solve our problem using name-based virtual hosting over multiple IP's. Strict IP-based hosting on tomcat involves setting the useIPVHosts attribute on the connector, but doesn't help in serving multiple SSL sites as you still need multiple https connectors because each can only have a single ssl cert specified (SSLCertificateFile).

Configuring Tomcat 7 to support SSL

Foreword

• This post talks about how to get SSL up and running with Tomcat 7, this means you'll be able to accept https/secure connections to your websites, this is especially relevant as Facebook now require all canvas apps to support https. 
• I'm assuming that you're using Tomcat as your client-facing server as described in my previous post, i.e. it isn't proxied behind Apache httpd.
• While I specifically target Tomcat 7, much of this post is relevant to all prior Tomcat versions.
• Many of the required processes are well documented on the web, rather than reproduce content instead I'll point you to the best resources I know.

Acquiring an SSL Certificate
When establishing an https connection the server will present the client/browser with an ssl certificate to aid the client in knowing whether to accept the ssl connection. In setting up SSL for Tomcat you'll need such a certificate. There are 2 ways:

1) Make your own certificate.
In this scenario you generate your private key and sign your own certificate. There are several tools to help you do this including keytool for Java , openssl on linux and Key-Manager on windows.
It's useful to sign your own cert if you are only worried about establishing data encryption on your private sites but it isn't too much use for hosting customer facing sites as your customers will get a scary browser warning that will probably scare them off for good. For this reason I won't dwell on this any further.

2) Buy a signed certificate
There's a huge variation in pricing and offerings, if you just want the padlock symbol on your site you can buy the cheapest offering from a company such as Comodo InstantSSL, they also offer a 90 day trial which is longer than the most free trials.

You'll need to create a private key which will be used to generate a Certificate Signing Request (CSR).
You send this CSR to your chosen certificate vendor, they'll review it and send you back a certificate you can use. Once again there are several alternative tools to help:

• To use openssl follow the procedure described here. This article covers generating a new private key AND CSR. If you already have an exiting private key you can just generate a new CSR with the command:  openssl req -out www_mydomain.csr -key myserver.key -new
• To use java keytool follow the procedure described here.

You may also want to convert your private key to pem format for use later, you can do this with openssl like this: 

openssl rsa -in myserver.key -out myserver.pem

Once you have received the signed certificate back from your vendor, your ready to install if to Tomcat (or Apache)

Configuring Tomcat for SSL
It's time to install our new ssl cert into tomcat. Life is never easy with Tomcat and here is no exception, it supports SSL in one of 2 ways and will select the most appropriate based on your operating system. Briefly these are:
1) Using a keystore file capable of containing many certificates and keys. Certificates are referenced by an alias name. Java JSSE is used here and thus is platform independent, but slower than method 2.
2) Using APR (Apache Runtime Library) that can access certificates and keys directly as files thus bypassing the keystore mechanism of method 1.

I recommend method 2, APR brings benefits to your entire tomcat distribution including native SSL handling and speed improvements, also you won't have to use the keytool command to insert certs into the keystore!
The downside is the APR can be troublesome to install:
Here's the official APR howto.
Here's how I do it:

cd $CATALINA_HOME/bin

tar xvfz tomcat-native.tar.gz
cd tomcat-native-<version>-src/jni/native

yum install apr openssl apr-devel openssl-devel

./configure --with-apr=/usr/bin/apr-1-config
make
make install
Note that it's very important to have the ssl packages installed by yum (as above) before you make APR, without them apr will still build but without SSL support and you'll get browser errors later. If you do this correctly apr will install to /usr/local/apr/lib

If you have followed my previous post on running tomcat using jsvc then you can benefit from switching the secure port to 443, you'll also need to insert the line:

export LD_LIBRARY_PATH="/usr/local/apr/lib"

at the top of your /etc/init.d/tomcat7 file, this ensures the APR library can be found by Tomcat.



Now we need to modify the $CATALINA_HOME/conf/server.xml file.
In server.xml locate the connector with port=8443 and uncomment it. You will need to add additional attributes to make it work. Unfortunately the attributes you use in configuring this vary depending on whether you're using method 1 or 2 I outline above.
With method 1 you reference the keystore, with method 2 you reference the certificates and key file directly. Here's a link to the supported attributes for each method.
As I'm advocating method 2, here's what it will look like:

    <Connector port="443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               SSLCertificateFile="/home/tomcat/ssl/www_yourdomain_com.crt"
               SSLCertificateKeyFile="/home/tomcat/ssl/myserver.pem"
               SSLCertificateChainFile="/home/tomcat/ssl/www_yourdomain_com.ca-bundle"
               clientAuth="false" sslProtocol="TLS" />

Where:
SSLCertificateFile and SSLCertificateChainFile were provided by your signing authority.
and SSLCertificateKeyFile is your private key (don't share this with anyone)


Save the server.xml file and start tomcat.

When starting tomcat you can determine whether it has loaded APR successfully by looking at the start of the $CATALINA_HOME/logs/catalina.<date>.log file.

It will either contain something like:

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/latest/jre..etc

or
INFO: Loaded APR based Apache Tomcat Native library 1.1.22.
or
a bunch of errors saying why it didn't work out.



You should now be able to hit your site securely https://www.yoursite.com :-)


Conclusion
If you've followed my recommendations you'll have a best of breed tomcat install running SLL natively fast for not much financial outlay. It's overly complex though, but I'm afraid there's no easy way when talking SSL.

Fixing SSH timeout problems

This post is about resolving ssh idle timeout issues.

Foreword

Using SSH to connect to a remote unix host is invaluable, but most users are frustrated by the default behaviour where sessions left idle for a short time are automatically disconnected. Here I'll talk about some strategies for overcoming this.

Preventing auto SSH disconnection on the server.
This strategy involves adjusting the configuration of the server's ssh daemon, so you'll need root access to the server for this.
The ssh daemon is the server side service that allows ssh connections to be made, we can configure it by adjusting it's configuration file:

vi  /etc/ssh/sshd_config

locate the line:
#ClientAliveInterval 0
and modify it to:

ClientAliveInterval 60

Save the file.
Reload the new sshd config with:
service sshd reload

This adjustment causes the server to send a keep-alive message to the client every 60 seconds. As long as your client is still connected this is enough to prevent the connection from being idle and closing.
The default value of 0 meant that no keep-alive message was ever sent and your connection dies from being idle.

Read more at
man sshd_config

Preventing auto SSH disconnection on the client.

Linux/Unix Clients
Similar to the server side strategy presented above, here we adjust the ssh client configuration so it sends the keep-alive message to the server every 60 seconds.

vi  /etc/ssh/ssh_config

locate the line:
#ClientAliveInterval 0
and modify it to:

ClientAliveInterval 60

Save the file.

Windows Clients
Most windows users use PuTTY , probably because it's free ;-)

PuTTY Settings -> Connection

Set PuTTY to send keep-alive messages every 60 secs (as shown above).

Other Strategies

If you don't fancy any of the above, when you know you're going to leave your ssh connection idle for some time simply issue the top command:

top

This starts the interactive top program that auto refreshes your server state every few seconds and will be enough to stop a timeout. Hit q to exit top.


Conclusion
All strategies presented above achieve the same goal. The method you choose will largely be determined by whether you have root access to the server and whether you want changes to affect all clients connecting to the server.

Installing Tomcat7 with Java7 to Centos5 / RHEL5

This post describes how to install Tomcat7 with Java7 to a Centos5 / RHEL5 server.
The target here is run Tomcat on port 80 as a deamon service but not as root. I will however be using root privileges to install the software below.

Foreword
There are many ways to achieve the same goal of getting Tomcat to serve your dynamic content, I've been configuring the Tomcat servers on production websites since Tomcat3 and I've watched Tomcat mature as a product. I used to advocate putting Tomcat behind Apache httpd server but now I view Tomcat as mature and fast enough to listen on port 80 directly. There are still very legitimate reasons you may want to sit Apache in front of Tomcat, these include needing to use Apache mods or using Apache as a loadbalancer with multiple tomcats behind. I won't be covering those instances here.


Installing Java 7
We need to start by installing the JVM. Oracle provide a couple of different options: an RPM or a Compressed Binary. Here I use the RPM as it does slightly more for you.

My Centos box is 64bit, use uname -a , so I'll use 64bit java.

Download the rpm and optionally use md5sum to check the file, this is good practice as it ensures the file has transferred correct and hasn't been tampered with.

md5sum jdk-7-linux-x64.rpm

gives us 3c5c52922766ba365f83ee6a60dd2e60  jdk-7-linux-x64.rpm , as we expected

You may wish to uninstall you previous JVM, exercise caution if you have services using this JVM, transfer your web traffic and stop them.

Here I uninstall Java6/JDK1.6 . I use rpm -qa to locate the rpm package name and uninstall with:

rpm -e jdk-1.6.0_27-fcs

Install the new java:

rpm -ivh jdk-7-linux-x64.rpm

I now add JAVA_HOME to the environment, this is again good practice, but not strictly necessary if you are going to use the scripts we use below.

vi /etc/profile

and add to the bottom:

export JAVA_HOME=/usr/java/latest

relogin to shell and test with:

echo $JAVA_HOME

You will notice that the Java RPM has made the /usr/java/latest link to point to the latest JVM we can see this with the command ls -al /usr/java/

java -version

Should confirm Java is ready, we'll install Tomcat next:


Installing Tomcat 7

Download and md5sum the latest Tomcat7 to your home directory. this is apache-tomcat-7.0.21.tar.gz at time of writing.

Extract it:
tar xvzf apache-tomcat-7.0.21.tar.gz

Move the extracted folder:

mv apache-tomcat-7.0.21 /usr/share/

Set the CATALINA_HOME env variable, using the same technique we use to set JAVA_HOME above
export CATALINA_HOME=/usr/share/apache-tomcat-7.0.21

A primary objective here is to run Tomcat as the server on port 80 but not as the root user. Running a public facing process on a port as root is dangerous as if the service is compromised the attacker will be able to cause far more damage as root has unlimited permissions. Instead we'll make an arbitrary user tomcat to run the service under:

useradd tomcat

This will make a user and group called tomcat.

Set the tomcat installation to be owned by the tomcat user:
chown -Rf tomcat.tomcat /usr/share/apache-tomcat-7.0.21/

Change Tomcat to run on port 80

vi $CATALINA_HOME/conf/server.xml

Change all references of 8080 to 80, and comment out the Connector on port="8009" , this is used when we have Apache httpd server on port 80 in front of Tomcat.


Running Tomcat as a daemon

Tomcat ships with a nice tool that enables Tomcat to run as a unix daemon service as a user other than root, run the following commands to build jsvc:

cd $CATALINA_HOME/bin
tar xvfz commons-daemon-native.tar.gz
cd commons-daemon-1.0.x-native-src/unix
./configure
make
cp jsvc ../..
cd ../..

If you encounter any problems with configure or make, you may need to install additional components to Centos, use:

yum install gcc

and/or
yum install make

You will now have jsvc in your $CATALINA_HOME/bin folder.

Although it isn't in the docs Tomcat7 ships with a service script to operate jsvc:
$CATALINA_HOME/bin/daemon.sh

cd $CATALINA_HOME/bin/

Try starting Tomcat:
./daemon.sh start

You can check it's running by hitting port 80 with your web-browser or using

ps aux

to look for the tomcat process, you'll see the process running as non-root:
tomcat   26372  4.0 44.2 818540 464112 ?       Sl   14:25   6:31 jsvc.exec -java-home /usr/java/latest

or

netstat -nlp

to see the open ports, you'll see something like:

tcp        0      0 :::80      :::*     LISTEN      26372/jsvc.exec

If you have problems check the logs:

ls -al $CATALINA_HOME/logs

specifically:
catalina-daemon.out

Next step is to ensure Tomcat starts automatically on boot. We'll use the unix service facility to achieve this.

cd /etc/init.d

cp $CATALINA_HOME/bin/daemon.sh ./tomcat7

vi tomcat7

And replace the comment at the top of the file with:

    #!/bin/bash
    # description: Tomcat Start Stop Restart
    # processname: tomcat7
    # chkconfig: 234 20 80
    TOMCAT_USER=tomcat
    JAVA_HOME=/usr/java/latest
    export JAVA_HOME

    CATALINA_HOME=/usr/share/apache-tomcat-7.0.21

additionally you may add a line like:

CATALINA_OPTS="-Xms256m -Xmx512m"

to pass options such as memory limits to tomcat.

Add tomcat to levels 234 of startup:

chkconfig --add tomcat7

You can check it with:
chkconfig --list tomcat7

You can now control tomcat with the commands:

service tomcat7 start

service tomcat7 stop

Diagnostics:
service tomcat7 version

Try rebooting your Centos server, it should now auto restart tomcat running as the tomcat user.
shutdown -r now

Troubleshooting
Some problems I have seen on various boxes:


Jsvc reports Cannot find any VM in Java Home
"Cannot find any VM in Java Home" appears when starting jsvc, you may also see "Cannot locate JVM library file" in the logs.
I experienced this after building jsvc on a 64bit architecture server using a 32bit jvm, swapping the jvm for 64bit and rebuilding jsvc did the trick.


Firewall restricts access to port 80.
If it's iptables

iptables -I INPUT -p tcp --dport 80 -j ACCEPT

service iptables save

Conclusion
Hopefully this has helped you achieve the best of both worlds, you should now have Tomcat 7 listening on a privileged port but not running as a superuser. The service will also auto restart along with your server.
If for some reason you can't run jsvc you have a few options:
1) Use the 3rd party software authbind to allow a normal user access to a port below 1024.
2) Use iptables to reroute requests from port 80 to 8080.
3) Run apache httd along with a connector to proxy back to tomcat.

If you are successfully using Tomcat on port 80 you'll now most likely want to use virtual hosts to run multiple sites from a single server. I'll write about the pros and cons of this in the future...