Performance and Scalability with Redis NoSQL Database

Introduction

Hibernate is a great technology, it frees developers from the pain of database schema creation and the maintenance of object-relational mapping code. They are free to create rich domain models quickly and easily! But with this simplicity comes a hidden danger, these highly normalised data models may mirror the problem domain accurately giving you a warm fuzzy feeling, but under load they are unlikely to perform or scale well!

In this article I present a scenario where the Redis NoSQL database was used to complement a rich domain model in MySQL to achieve dramatic performance improvements and near linear scalability.

The Problem

We have a social networking site for meeting new people, the homepage is a News-Feed, the News-Feed consists of News-Items of different types. News Item types include but are not limited to:

• Someone Likes you.
• A Match, (you both like each other).
• Someone Viewed you.
• A New Member matching your criteria.
• An Upload by someone you like.
• etc etc

The News-Feed looks like this:

The News-Feed is the aggregate of all the different types of News-Items sorted by their timestamp.

Additionally we want to be able to filter on just one News Item type, e.g. show a list of just one News Item type such as Who Likes Me.

And above all, this has got to be fast, really fast, it’s a homepage after all and our users expect a near instantaneous response.

The Relational database trap

MySql (Percona) is our authoritative datastore, as a relational database it brings a lot of advantages with it. We used Hibernate to build this system, and on top of that we used Spring Roo to generate our domain entities (Domain Driven Design). This gave us huge advantages in speed of development, we pretty much drove the creation of the domain layer by writing Roo script and Hibernate took care of the data access layer. Roo even helped us by generating some simple starting point finders in JPQL.

After this, we pretty much had a one-to-one mapping between a News Item and a relational database table sitting underneath, it looked something like this:

It seemed pretty straight forward, especially when we had no site traffic, to create these News-Items by pulling them out of the database with a straightforward SQL Select statement.

We modified the JPQL a little to include some JOINs to filter certain records out etc, and functionally it worked well. 

For the aggregate News-Feed, the easiest way forward was to issue several bounded database queries, combine the results and sort by timestamp in the application. We knew it wasn’t too pretty but this was a minimum viable product, we had no traffic and were keen to get this product out into the real world!

To our delight the site was well received and we watched Google Analytics intently as traffic numbers rapidly grew. Pretty soon we needed to scale…

Vertical Scaling will only get you so far.

A bit of poking around with New Relic confirmed what we initially suspected, the performance bottle necks were centred around the data tier. Optimisation wouldn’t be immediate, so to buy time we threw more hardware at it, aka vertical scaling. This is an acceptable short term approach, hardware scaling including provisioning greater memory and faster IO usually makes stuff quicker, but eventually you will hit a ceiling and in the meantime you’ll be spending far more cash on hardware than you’d like. It was time to address the data tier…

Optimising the Data Tier

Relational query optimisation shouldn’t really be left until you have a problem before being addressed. There’s a lot you can do with the sql explain statement and you’ll get tremendous insight into the operation of your queries if you do. Engineering queries and database indexes to mirror each other will in most cases yield orders of magnitude performance improvements with large datasets. With Hibernate you should also analyse the fetch strategies between objects to minimise n+1 problems. Furthermore, compared to re-architecture or database denormalisation, these are comparatively cheap exercises in terms of developer time and buy you a lot of scalability. I’ve written more about this here.

Other ways to scale a relational database.

If you’ve exhausted the avenues suggested above there are other ways of scaling a relational datastore. Firstly consider database read-replicas. These work fantastically well where there’s a strong read bias on your database load, which is the pattern you see on most web apps. Most sites see 90%+ of their DB queries just being fetches, and for sites like StackOverflow or Wikipedia I’d say it’s probably 99%+. Database sharding is also another valuable technique to consider for very large datasets, I won’t discuss it here.

Recap

Optimising our queries and the clever use of SQL Joins bought us well over 10X performance gains with our current hardware and dataset. However we could foresee that this approach wasn’t going to scale well for the News-Feed going forward. We were also aware that we were not the first to travel this path. Our site has a lot in common with giants such as Facebook, Twitter and Tumblr. All have a News-Feed type user interface backed by an authoritative relational datastore, and all needed to find an alternative approach to solve their scalability problem once and for all. We took the time to research the architectural approaches used by these sites and the information made public by them was of great inspiration to us.

Redis to the Rescue

Redis is so much more than just another NoSQL database. It’s capabilities go far beyond a simple key/value datastore, sure it can operate like this, but it’s true strength lies in it’s ability to store and manipulate data structures at mind-blowing speed. Redis natively supports Lists, Sets and SortedSets. These data structures allow you to logically bundle data together in a data structure within a Redis database. A data structure can then referenced within Redis by a unique key that you define.

Choosing a Redis Data Structure

At first glance it appeared that the Redis List structure would be ideal for our holding our News-Items. Lists maintain insertion order and could work well as a fast backing store for us. However the SortedSet structure seemed an even better fit for our solution. Sets maintain exclusivity of their elements, but the SortedSet also orders it’s elements according to an associated numerical score. It quickly occurred to us that using our News-Item timestamp as the score would result in Redis handling our time-series ordering for us.

To save space in Redis, and to avoid unnecessary duplication of data we decided that the values we’d store in Redis would be ID references to data stored authoritatively elsewhere. These id references could later be resolved by consulting a separate Lookup Service in our service oriented architecture (SOA).

As I’ve mentioned above, our News-Feed consists of News-Items of many different types. We need to be able to distinguish between these types as they are retrieved from Redis, so we devised a simple protocol for storing byte streams in Redis:

Note: We actually support over 10 NewsItem types, but we’ll stick with these 3 to demonstrate our concepts.

The Inbox Model

Using a NoSQL database for anything other than key/value based cache (the origins of Memcached) requires a developer to think about data in a fundamentally different way. With a relational database you model real world data into relational tables, with a NoSQL store the developer needs to consider instead the data structure that lends itself most closely to how that data will be required by the application. 

In our case we are building an Inbox of News-Items specific to a user. In effect we are pre-calculating the News-Items that will appear in a user’s inbox before they even request that data. It is this Inbox approach that enables Internet giants such a Twitter and Facebook to serve up your timeline with blistering speed at immense scale.

Our News-Feed Inbox

With reference to the above protocol, creating a Redis SortedSet for each User’s News-Feed would look something like this:

This works great for the News-Feed view, but remember above I said we also had the requirement to filter down to a single News-Item type, e.g. so we could just show News-Item types for who Likes You etc. Retrieving just the values starting with R (likes you) is not possible in Redis, there is no Structured Query Language to fall back upon! 

This highlights one of the shortcomings of NoSQL databases that one has to keep in mind, if you want to cut and slice data according to criteria you may not have even thought of yet then RDBMS wins every time. With NoSQL you need to anticipate your data views or queries in advance and have a structure to match.

Given the above structure, in order to extract just the R values we’d need to fetch all values and filter the values in the application. Whilst this is certainly possible, it isn’t particularly efficient for the database or the application and makes features like pagination especially difficult.

Redis Set Union

Fortunately there’s a graceful solution to this. Redis offers powerful Set manipulation features including intersection and union operations. After writing some proof of concept code we decided that the best approach would be for each user to have an Inbox per News-Item type; we could then perform union operations within Redis to construct the News-Feed on demand. It looks like this:

To create the News-Feed we then instruct Redis to perform a union operation combining the Inboxes we specify. This is a very flexible approach, for some application views we may want to union almost all Inboxes and for others we may just union together a couple of Inboxes. 

The resulting Union operation on the above Inboxes would produce a SortedSet looking like this:

Union Performance

Initially we were concerned that these union operations would be computationally expensive for Redis and therefore slow, but in our proof of concept code we observed that Redis was able to union 20 Inboxes each containing hundreds of entries within single digit millisecond times. 

The resulting SortedSets were of course automatically sorted by Redis according to our timestamp based scores. This allowed us to retrieve subsections of the resulting structure based on the score thus making implementation of user interface features like pagination and lazy-load a breeze.

System Architecture

Putting the above into context, here’s how the News-Feed system architecture looks:

• Every thing you see here with the exception of Redis itself and the Message Bus is written in Java Spring.
• It’s a Service Oriented Architecture (SOA) with services defined as a set of interfaces.
• The logical flow starts at the top of the diagram, with various events, usually as a result of user actions resulting in the creation or modification of domain entities.
• The Business Logic Services communicate these actions to the Event Publisher which maps them to concrete classes within the Event hierarchy, these event objects are suitable for transport over the Message Bus.
• The Message Bus is asynchronous and provides guaranteed delivery of messages passed to it. It’s a fire-and-forget operation for the Event Publisher and decouples event publication from event consumption and the processing that follows.
• The Message Bus provides opportunity for multiple subscribers to consume the events it carries. The Redis Fanout Service subscriber is responsible for populating our Redis database, but we may add other subscribers to fulfil other purposes such as providing push style notifications to apps etc.
• The Redis Fanout Service consumes Message Bus events and fans them out to Redis. The ‘fanout’ terminology arises because a single event may result in multiple redis entries, e.g. a Match (mutual like) event will result in a Match entry being added to the Redis inboxes of both the users involved in the match.
• Redis Fanout Service creates an object of the RedisItem class hierarchy for each required redis entry.
• RedisItem subclasses encapsulate our Redis storage protocol introduced earlier. RedisItems have the capability to produce a byte stream representing themselves suitable for insertion into Redis or indeed to construct themselves once retrieved from Redis.
• Redis Service completely abstracts the Redis database from the rest of the system. It uses Spring Data Redis to simplify Redis interactions and enable us to code with familiar Java constructs and collections instead of native Redis commands.
• Redis Service is able to accept batches of RedisItems for storage and uses pipelining to submit multiple storage requests in a single I/O cycle thus achieving very significant performance gains.
• Redis Service provides a concise set of finder methods for querying Inboxes, results are returned as RedisItems. There are also automatic mechanisms here for fault detection, robustness and inbox reconstruction should a state inconsistency be detected.
• News Service is an example of a service that wishes to retrieve the contents of the Redis database, it is invoked by an MVC controller as a direct consequence of a user request. It may retrieve further information by consulting the authoritative Lookup Service, and produces ViewDTOs.
• The Lookup Service generally returns domain entities from their IDs. This provides opportunity to resolve these from a Level2 cache, there are many swappable options here ranging from EHCache, Memcached, another Redis database or some other datagrid. Only a cache miss results in load on the relational database.
• ViewDTOs are suitable for passing to the View of the MVC or can instead create a JSON representation for passing to a client such as an app.

Conclusion

This architecture allows us to satisfy client News-Feed requests within single-digit millisecond server execution times. In effect we are using the Redis as a sophisticated cache to serve precomputed result sets to individual users. However, our approach is only made possible by Redis’ native support for data structures combined with it’s breakneck speed with large datasets. User requests are resolved from Redis and are never synchronously blocked waiting for complex SQL queries to complete. This also opens the possibility of serving the results of more computationally intense queries such a recommendations, which would not be possible within an acceptable timeframe with a synchronous query. Furthermore, this approach scales in a linear fashion, Redis retrieval times remain near constant regardless of the dataset size. 

Optimizing your Database Queries using MySql, Explain and Indexes

Problem:

Imagine you run a successful website or app, month after month your traffic grows until you notice page views are getting slower and slower and the user experience is suffering. A little investigation reveals the database is the bottleneck.

 

Solutions:

This is a common scenario and there are several lines of attack you can follow to alleviate this problem:

1)      Caching. All enterprise level, high traffic websites use caching in one form or another to operate at in-memory speeds and provide a lightning quick user experience. Memcached or Terracotta BigMemory would be a good solution here.

2)      NoSql databases. Adding a NoSql database tier such as Cassandra or Solr/Lucene can provide huge advantages in terms of scalability and specialized search, such a geospatial search.  This doesn’t mean ditching your relational database, many enterprise web-architectures have both Sql and NoSql database tiers running in harmony together, with the relational database as the authoritative data repository while something like Solr satisfies the majority of the user search queries.

3)      Optimize your database queries.

In reality you probably want to employ a combination of all 3 approaches, but for this article I’m going to focus on approach 3.

Optimize your database queries

All databases (even NoSql ones) have the notion of indexes. Proper index design is absolutely essential to maintain performance as a table grows in size. Indexes need to be closely aligned to the actual queries being executed by the database to be effective. I’m also going to cover several important nuances of designing effective indexes.

Align your Indexes to your Queries

Start by capturing the exact queries being sent to the database. There are several alternative ways:

1)      Enable the MySql General Query Log.  This causes MySql to write all queries it receives to a log.

To activate the log use MySql Workbench or MySql Monitor to issue the command:

                                SET GLOBAL general_log = 1;

Then locate the log on the computer running MySql , e.g.

On Windows: C:\ProgramData\MySQL\MySQL Server 5.5\data\<hostname>.log

On Linux: /var/lib/mysql/ <hostname>.log

2)      Enable the Slow Query Log.

This causes MySql to log all queries taking longer than a default of 10 secs to execute.

# activate for all queries

SET GLOBAL long_query_time = 0;                       

# activate log

SET GLOBAL slow_query_log = 1;

Find the log:

Windows:

C:\ProgramData\MySQL\MySQL Server 5.5\data\<hostname>-slow.log

Linux:

 /var/lib/mysql/ <hostname>-slow.log

3)      Application Logging

If you are building your queries in your application, you’ll be able to print them to your application logs before they execute.

If you’re using an ORM such as Hibernate you can enable SQL debugging, setting these properties is often a good start:

        <property name="hibernate.show_sql" value="true"/>

         <property name="hibernate.format_sql" value="true"/>

        <property name="hibernate.use_sql_comments" value="true"/>

Which method is best?

I recommend method (2) to capture all queries to the database over a period of time, e.g. you could enable this on a test server during a load simulation for 15 mins to capture a reasonable sample of database queries. You should ensure your load simulation provides an accurate coverage of queries that are typically executed by your users, if you don’t have this confidence then you cannot beat capturing a log from your live database server as the logging is pretty lightweight and should not have a negative effect. Note: you can even capture slow query logs from Amazon RDS cloud databases.

Processing a Slow Query Log.

MySql now provides a tool to do this but I recommend using Percona’s excellent toolkit, specifically the tool pt-query-digest.

After you have installed this, process your captured slow log with a command like:

~/ptools/bin/pt-query-digest /var/lib/mysql/mysql-slow.log > ~/pqd.txt

Here pt-query-digest processes your mysql-slow.log to produce the file: pqd.txt , this is an aggregation of all the queries in the slow log , it shows you the top 20 slow running queries thus enabling you to target the lowest hanging fruit first.

Replacing SubSelects with Joins

If you notice your slowest queries use subselects  (a dependent select statement within the where clause of the main select) then I’d generally advise you to rewrite the query using Joins. Subselects often scale horribly on tables that contain lots of data, whereas the database optimizer can use Joins to its benefit to narrow down a result set very quickly. There is often an orders of magnitude difference between the 2 techniques with identical outputs.

Using Explain

Prefixing a query with the EXPLAIN keyword causes MySql to explain how it resolves a query. This is critical in devising the correct indexes to speed up the query.

For example, suppose we have a social networking site with a table containing messages between users.  When a user checks his message inbox, this may cause the following query to run:

SELECT * FROM message AS msg WHERE msg.dest_user=489764 AND msg.del_by_user=0 ORDER BY msg.priority DESC, msg.created DESC LIMIT 10

The query is running slow, but you have no idea why, EXPLAIN can help, issue the command:

EXPLAIN SELECT * FROM message AS msg WHERE msg.dest_user=489764 AND msg.del_by_user=0 ORDER BY msg.priority DESC, msg.created DESC LIMIT 10

MySql responds with:

id,select_type,table,type,possible_keys,key,key_len,ref,rows,Extra

1,SIMPLE,msg,ALL,NULL,NULL,NULL,NULL,1618964,"Using where; Using filesort"

 

This shows us 2 main things;

1)      No Key (Index) was used (or available) to satisfy this query.

2)      1618964 rows were scanned to deliver this result of up to 10 messages, this is not efficient!

Running SELECT COUNT(*) FROM message;  also returns 1618964 , meaning we did a full-table scan. This is the worst case scenario, it means the database had to read every record in the table to determine if it should form part of the result set. This really won’t scale as the message table gets bigger.

Fortunately, Indexes can really help us here…

Index Selectivity

Indexes are most effective when they have high selectivity, this means that the column we are indexing has a high number of possible values.

It’s defined as:

Selectivity = of distinct values in a column / number of records ➔ S = d/n

Thus a good candidate for an index in the above example would be dest_user. In our example app every new user receives a welcome message and additionally we don’t actually remove a message from the database when a user deletes it (hence the del_by_user flag), so it’s possible there’s a message in the table for each of the 1210778 users in existence. Thus our selectivity is:

S = 1210778 / 1618964

S is 0.75

As ‘distinct values in a column’ can never exceed ‘number of records’ the highest selectivity possible is 1, so 0.75 is a pretty good candidate for an index on this column.

Contrast this with putting an index on the del_by_user column. It’s Boolean so there are only 2 possible values. Thus our selectivity is:

S = 2 / 1618964

S is 0.0000012

Very low, not a particularly effective index, however it would narrow down the number of records to be scanned to half of the total if there’s an even distribution of read messages.

The Overhead of Indexes

Indexes do have an overhead, every time there’s a query that changes the data being indexed then both the table data and its associated indexes must be modified.  This is a price worth paying for an effective index that speeds up your most frequent queries but it’s also a compelling argument against blindly applying indexes against all columns of every table!

When executing a query the database optimizer decides whether it’s better to use an existing index or not.

If you notice by using explain that an index exists for a column but is never actually used, it’s usually advisable to remove that index to clear the overhead.

Creating Indexes

Given the selectivity calculations above, dest_user is looking like a great candidate for an index, lets create one by issuing this command to MySql:

CREATE INDEX msgidx ON message (dest_user);

Now we run explain again, and get the following result:

id,select_type,table,type,possible_keys,key,key_len,ref,rows,Extra

1,SIMPLE,message0_,ref, msgidx, msgidx,8,const,1282,"Using where"

 

This is a huge improvement, it means MySql only had to scan 1282 records to produce the result set, this is going to be infinitely faster than scanning the whole table, in fact it tells us there are 1282 messages in the database to this user and MySql will then apply the next criteria in the WHERE clause to narrow the result set down further. The next criteria  is del_by_user. It could be worthwhile to index this column too.

Compound Indexes

A compound index is one index containing multiple columns. So knowing the query above has a WHERE clause of:

WHERE msg.dest_user=489764 AND msg.del_by_user=0

A sensible compound index to match this query would be:

CREATE INDEX msgidxDestusrDel ON message (dest_user, del_by_user);

Running EXPLAIN on the query again now gives:

id,select_type,table,type,possible_keys,key,key_len,ref,rows,Extra

1,SIMPLE,message0_,ref," msgidx,msgidxDestusrDel",msgidxDestusrDel,9,"const,const",141,"Using where"

 

This was really worth doing, we’ve got the rows scanned down to 141, which is exactly the number of rows returned by the WHERE clause of this query (excluding the LIMIT). This is always our goal when using EXPLAIN and this should be the point at which we move on to optimizing the next query.

You may be wondering why I chose a compound index here rather than just adding a new standalone index for del_by_user, let’s try that, I remove the compound index msgidxDestusrDel and add:

CREATE INDEX msgidxDel ON message (del_by_dest);

Running EXPLAIN gives:

id,select_type,table,type,possible_keys,key,key_len,ref,rows,Extra

1,SIMPLE,message0_,ref, “msgidx, msgidxDel”, msgidx,8,const,1282,"Using where"

 

Although 2 indexes exist the MySql optimizer in this case decides not make use of our new index msgidxDel, this is due to the tree implementation MySql uses to represent indexes. Index implementation differs between database vendors, so this outcome is MySql specific. Due to the overhead I described earlier we’d be better off without this new index.

 

Index Ordering Matters

Given a compound index of:

CREATE INDEX msgidxDestusrDel ON message (dest_user, del_by_user);

This would be useful for a WHERE clause of:

WHERE msg.dest_user=489764 AND msg.del_by_user=0

And

WHERE msg.del_by_user=0 AND msg.dest_user=489764

i.e. MySql is smart enough to select the best index regardless of the ordering within the WHERE clause.

However, if the message table also contains a Boolean msg_read column and the only compound index we create is:

CREATE INDEX msgidxDestusrDel ON message (dest_user, msg_read, del_by_user);

This index would not be useful for the WHERE clause:

WHERE msg.dest_user=489764 AND msg.del_by_user=0

MySql would in fact revert back to using the single index we have defined on the dest_user column, this is because the order of our new compound index is not compatible with this WHERE clause, msg_read is between the two column we are actually searching on.

Had the order of the compound index been:

CREATE INDEX msgidxDestusrDel ON message (dest_user, del_by_user, msg_read);

This would be used, and the last msg_read component would be ignored.

Indexes are also used for Sorting results

If you include your ORDER BY columns after your search columns within a compound index your query will be even faster, e.g.

In the context of our social networking site, imagine we have a table of ratings made by users. It has the following compound index:

CREATE INDEX srcRegRatingDate ON rating (src_user, rating_value, updated);

For the query:

SELECT * FROM rating AS rating0_ WHERE rating0_.src_user=8019 AND rating0_. rating_value=100 ORDER BY rating0_.updated DESC LIMIT 15

MySql can use the index order for the results rather than conducting a separate sorting operation.

The damaging effect of Range operators

Using a range (LIKE or ‘>’ etc) operator on a column in a compound index means that any additional criteria we have for subsequent columns cannot benefit from the index.

e.g. With reference to the above ‘rating’ example. The subtle difference of using a >= range operator on the rating_value rather than just ‘=’ would mean MySql could not use the ‘updated’ column of the index for the sort.

Covering Indexes

Covering indexes are indexes that contain all the columns necessary to completely satisfy the return requirements of the query solely using the index. In this case MySql can provide results without reference to the data table itself. This provides a huge speed boost.

If you name the columns to be returned by your select statement you can selectively ensure these columns appear within your compound index.

The position of the returned columns within the compound index is not important, the return columns merely have to be present.

If you are using an ORM such as hibernate and returning complex objects into your application your chances of using a covering index are more limited. This is because hibernate fetches all columns in every Select statement in order to fully populate the return object. If speed is crucial you may choose to take this responsibility away from hibernate and write your own database fetch routines to maximise the opportunity of covering indexes.

Conclusion

As you can see, designing the most efficient indexes for an application requires great insight to the queries actually being executed by the database. You need to carefully design your indexes around your most important queries, there is no effective catch-all situation.

The performance gains to be made by implementing effective indexing can be enormous. A little engineering effort here is often much more cost-effective than simply scaling-up or scaling-out database hardware.

Compact Policy usage on Facebook Apps with Internet Explorer

The Problem
Most useful Facebook apps you write will be stateful, the most common way of achieving this is to allow your chosen web container to run user sessions. As HTTP is stateless, cookies are used to reunite the user with their session on each request thus maintaining state.
The problem arises when for whatever reason the browser doesn't accept the session cookie and your facebook app can't maintain state for that user. This most commonly happens with Internet Explorer, every developer's favourite browser!
In this article I dig a little deeper and look at ways to mitigate this problem.
This post relates to privacy issues and you should seek advice from your legal advisor before using the techniques I describe here in a public facing deployment.

Example
A user hits your fb app running Internet Explorer (IE) with elevated privacy settings. As modern facebook applications run within a facebook iframe any cookies issued by your webapp are not regarded as 1st party and thus subject to even stricter controls. Such cookies are usually blocked by the browser and cause your user to lose state and your app to malfunction, often sending your user to your login page.
The exact same thing can happen if your user is running 3rd party security software from some well known vendors.

Solutions
You need to either:
1) Increase the chances of your session identifier cookie being accepted or
2) Make your webapp function without cookies at all.

Solution 1.
Internet Explorer now stands alone in that it's the only mainstream browser implementing W3C's Privacy Policy framework P3P. This allows a site to specify it's privacy policy in a machine readable format and a browser to make intelligent decisions based on a user's privacy preferences.... that's the idea anyway. The fact most browsers don't implement the standard shows the industry has lost interest in this standard.
However it matters to you as the lack of a Compact Policy (derived from a Full P3P Privacy Policy) has a serious impact of whether IE is going to accept your cookies. Take a look at Microsoft's own article at deploying P3P. You will find a range of tools for generating policies on the web.

Many sites, Facebook included, contain a message in place of the Compact Policy explaining why they don't comply. JUST the presence of this is enough to persuade IE to accept your cookie.

The Compact Policy is specified in the response header. Here's what facebook's looks like:

P3P:

CP="Facebook does not have a P3P policy. Learn why here: http://fb.me/p3p"

In Java you can do something similiar like this:
response.addHeader("P3P","CP=\"Please see our privacy policy at xxxx page.\"");

You could do this easily in a servlet filter so that it effects every response passed through the filter.

I'd of course recommend issuing a real policy containing a series of the accepted shortcodes.

Solution 2:
If you really can't get your cookies accepted by the browser there is one fallback position. URL-Rewriting is a technique where the Session ID is appended to the end of each browser request. This allows the session to be determined by the container and the client reunited with their session.
This technique obviously requires all links to be rewriten to bounce along the session with every request. This approach is supported by most containers and if your using the great Spring Security it all works automatically when needed.
Note: Exposing the SessionID does have security implications beyond the scope of this article, be aware of these before you go live using this, check out OWASP. UrlRewriting can easily be disabled too.

Conclusion
Implementing one or both of these techniques is as close as you can get to ensure your facebook app works as expected.
Note: Many sites Facebook and Google included can detect if cookies are completely disabled and will simply display an appropriate error message before refusing to work, you should take this approach if solution 2 is not appropriate for your application.

Tomcat truncating cookie values problem

The Problem
When using Java running in the Tomcat servlet container you may observe cookie values fetched from javax.servlet.http.Cookie using getValue() seem incomplete.

Example
I like to parse various cookies in Java code, one particular high value cookie for me is Google Anaytics' Campaign Tracking Cookie the UTMZ.
UTMZ typically looks like:
__utmz=1.1322309384.1.1.utmcsr=anothersite.com|utmccn=(referral)|utmcmd=referral|utmctr=9371183|utmcct=/

However when reading and displaying this cookie in Java and Tomcat you'll see the value:
1.1322309384.1.1.utmcsr

This is because Tomcat 6.0.18 now adheres to the cookie spec more tightly than previous versions.
This may cause you multiple issues including:

• The value you expect to see is incomplete (a quick look with Google Chrome -> Inspect Element -> Network -> Headers -> Request Headers shows that the browser is sending the full cookie value to Tomcat).
• Code that did work on a previous Tomcat suddenly breaks.

Solution

If you are in control of the cookie, i.e. you write it then you need to ensure a version 0 cookie doesn't contain any of the following characters: equals =, parentheses (), colon : . 

If like the above example you are powerless to control the cookie format, you need to run Tomcat with some additional system properties set. The following added to catalina.properties will do the trick:

org.apache.tomcat.util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE=true

org.apache.tomcat.util.http.ServerCookie.ALLOW_HTTP_SEPARATORS_IN_V0=true

Further info can be found in the Tomcat Config Docs. 

Hosting multiple SSL sites on Tomcat 7 with Virtual Hosting

Foreword
Virtual Hosting is where you serve multiple websites from the same server. This post describes how to achieve virtual hosting with tomcat on linux where each site requires it's own certified SSL capability. As per my previous posts we're assuming Tomcat is your public facing server and you are using APR for performance thus negating the need to sit apache in front. (If you do have apache in front you'd configure SSL there instead).
I haven't found anywhere else on the web explaining how to do this so thought I'd share it with you.

Virtual Hosting Concepts
Virtual hosting comes in 2 guises:

1. Name-based Virtual Hosting: This is where all your sites can share a common IP and the server looks to the HTTP request header to decide what site to serve up. This is the most common type of virtual hosting.
2. IP-based Virtual Hosting: Here your sites each have their own distinct IP address. The server listens to multiple IP addresses to determine which site to serve. This type of virtual hosting is less common as it uses more IP addresses, which with IPv4 are scarce and have a cost implication.

The Problem
We are interested in setting up virtual hosting on tomcat with SSL, it's fine to use name-based hosting if all your sites want to share a single ssl certificate. However, this is rarely the case. With a shared cert users receive a scary browser warning if the ssl cert presented doesn't match the requested domain name exactly. For this reason for commercial sites you're going to need a separate, officially certified ssl certificate for each site.
The problem is that the SSL handshake between the client-server happens very early on, before the server looks at the HTTP request header: The server presents the ssl cert tied to the IP before it even looks at the website being asked for!

The Solution 
We must use multiple IP's in conjunction with name-based Virtual Hosting. I'm assuming your server has a single ip address and you only wish to host 2 ssl secured sites, this approach will scale but you'll need a new IP address assigned to your server for each additional SSL site you wish to host. Talk to you hosting provider about this as adding IPs is out of scope but basically once the IP is assigned to your server you'll need to cd /etc/sysconfig/network-scripts then add a file ifcfg-eth0:1 to bind the IP to your main Ethernet device, you don't need >1 NIC. Then service network restart

So now you have 2 public-facing IPs configured for your server, e.g.
10.0.0.1
10.0.0.2

Now you need to configure tomcat to listen on them:

vi $CATALINA_HOME/conf/server.xml

Set up your sites as Host elements as if you were using only Name-based Virtual Hosting, this is simple and well documented.
Now turn your attention to your connectors. It needs to look something like:

    <Connector port="80" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="443" />

    <Connector port="443" protocol="HTTP/1.1" address="10.0.0.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               SSLCertificateFile="/home/tomcat/ssl/www_domain1_com.crt"
               SSLCertificateKeyFile="/home/tomcat/sslkey/myserver.pem"
               SSLCertificateChainFile="/home/tomcat/ssl/www_domain1_com.ca-bundle"
               clientAuth="false" sslProtocol="TLS" />

    <Connector port="443" protocol="HTTP/1.1" address="10.0.0.2" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               SSLCertificateFile="/home/tomcat/ssl/www_domain2_com.crt"
               SSLCertificateKeyFile="/home/tomcat/sslkey/myserver.pem"
               SSLCertificateChainFile="/home/tomcat/ssl/www_domain2_com.ca-bundle"
               clientAuth="false" sslProtocol="TLS" />

Some explanation:

• The connector on port 80 will listen to all IP's by default.
• We launch 2 https connectors each listening to port 443 on a separate address. We specify the address attribute to dictate the ip they bind to, see The HTTP Connector.
• Each https/ssl connector is tied to it's own SSL Cert.
• We set our DNS A-records for the domains to ensure the requests go to the correct connector.

Conclusion
We solve our problem using name-based virtual hosting over multiple IP's. Strict IP-based hosting on tomcat involves setting the useIPVHosts attribute on the connector, but doesn't help in serving multiple SSL sites as you still need multiple https connectors because each can only have a single ssl cert specified (SSLCertificateFile).

Configuring Tomcat 7 to support SSL

Foreword

• This post talks about how to get SSL up and running with Tomcat 7, this means you'll be able to accept https/secure connections to your websites, this is especially relevant as Facebook now require all canvas apps to support https. 
• I'm assuming that you're using Tomcat as your client-facing server as described in my previous post, i.e. it isn't proxied behind Apache httpd.
• While I specifically target Tomcat 7, much of this post is relevant to all prior Tomcat versions.
• Many of the required processes are well documented on the web, rather than reproduce content instead I'll point you to the best resources I know.

Acquiring an SSL Certificate
When establishing an https connection the server will present the client/browser with an ssl certificate to aid the client in knowing whether to accept the ssl connection. In setting up SSL for Tomcat you'll need such a certificate. There are 2 ways:

1) Make your own certificate.
In this scenario you generate your private key and sign your own certificate. There are several tools to help you do this including keytool for Java , openssl on linux and Key-Manager on windows.
It's useful to sign your own cert if you are only worried about establishing data encryption on your private sites but it isn't too much use for hosting customer facing sites as your customers will get a scary browser warning that will probably scare them off for good. For this reason I won't dwell on this any further.

2) Buy a signed certificate
There's a huge variation in pricing and offerings, if you just want the padlock symbol on your site you can buy the cheapest offering from a company such as Comodo InstantSSL, they also offer a 90 day trial which is longer than the most free trials.

You'll need to create a private key which will be used to generate a Certificate Signing Request (CSR).
You send this CSR to your chosen certificate vendor, they'll review it and send you back a certificate you can use. Once again there are several alternative tools to help:

• To use openssl follow the procedure described here. This article covers generating a new private key AND CSR. If you already have an exiting private key you can just generate a new CSR with the command:  openssl req -out www_mydomain.csr -key myserver.key -new
• To use java keytool follow the procedure described here.

You may also want to convert your private key to pem format for use later, you can do this with openssl like this: 

openssl rsa -in myserver.key -out myserver.pem

Once you have received the signed certificate back from your vendor, your ready to install if to Tomcat (or Apache)

Configuring Tomcat for SSL
It's time to install our new ssl cert into tomcat. Life is never easy with Tomcat and here is no exception, it supports SSL in one of 2 ways and will select the most appropriate based on your operating system. Briefly these are:
1) Using a keystore file capable of containing many certificates and keys. Certificates are referenced by an alias name. Java JSSE is used here and thus is platform independent, but slower than method 2.
2) Using APR (Apache Runtime Library) that can access certificates and keys directly as files thus bypassing the keystore mechanism of method 1.

I recommend method 2, APR brings benefits to your entire tomcat distribution including native SSL handling and speed improvements, also you won't have to use the keytool command to insert certs into the keystore!
The downside is the APR can be troublesome to install:
Here's the official APR howto.
Here's how I do it:

cd $CATALINA_HOME/bin

tar xvfz tomcat-native.tar.gz
cd tomcat-native-<version>-src/jni/native

yum install apr openssl apr-devel openssl-devel

./configure --with-apr=/usr/bin/apr-1-config
make
make install
Note that it's very important to have the ssl packages installed by yum (as above) before you make APR, without them apr will still build but without SSL support and you'll get browser errors later. If you do this correctly apr will install to /usr/local/apr/lib

If you have followed my previous post on running tomcat using jsvc then you can benefit from switching the secure port to 443, you'll also need to insert the line:

export LD_LIBRARY_PATH="/usr/local/apr/lib"

at the top of your /etc/init.d/tomcat7 file, this ensures the APR library can be found by Tomcat.



Now we need to modify the $CATALINA_HOME/conf/server.xml file.
In server.xml locate the connector with port=8443 and uncomment it. You will need to add additional attributes to make it work. Unfortunately the attributes you use in configuring this vary depending on whether you're using method 1 or 2 I outline above.
With method 1 you reference the keystore, with method 2 you reference the certificates and key file directly. Here's a link to the supported attributes for each method.
As I'm advocating method 2, here's what it will look like:

    <Connector port="443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               SSLCertificateFile="/home/tomcat/ssl/www_yourdomain_com.crt"
               SSLCertificateKeyFile="/home/tomcat/ssl/myserver.pem"
               SSLCertificateChainFile="/home/tomcat/ssl/www_yourdomain_com.ca-bundle"
               clientAuth="false" sslProtocol="TLS" />

Where:
SSLCertificateFile and SSLCertificateChainFile were provided by your signing authority.
and SSLCertificateKeyFile is your private key (don't share this with anyone)


Save the server.xml file and start tomcat.

When starting tomcat you can determine whether it has loaded APR successfully by looking at the start of the $CATALINA_HOME/logs/catalina.<date>.log file.

It will either contain something like:

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/latest/jre..etc

or
INFO: Loaded APR based Apache Tomcat Native library 1.1.22.
or
a bunch of errors saying why it didn't work out.



You should now be able to hit your site securely https://www.yoursite.com :-)


Conclusion
If you've followed my recommendations you'll have a best of breed tomcat install running SLL natively fast for not much financial outlay. It's overly complex though, but I'm afraid there's no easy way when talking SSL.