We’ve recently moved Coggle’s login sessions from a database-storage model to signed cookies, where session data is stored the session cookie itself.

There aren’t many real-world examples of how to handle this migration, so we’re sharing what we’ve learned doing this with node and express, and hopefully it’ll be a useful and interesting read!

Part 1: How Old Sessions Worked

Previously we handled sessions with the express-session module and connect-mongo data store, and then we used passport to load our actual user data based on the session. Our middleware setup looked like this:

const session = require('express-session');
const MongoStore = require("connect-mongo")(session);
const sessionStore = new MongoStore({ ...  });

// loads req.session from the database store, if the request included a valid session cookie
app.use(session({store: sessionStore, ...}));

// passport middleware loads req.user from our users collection based on the user ID stored in the session
app.use(passport.initialize());
app.use(passport.session());
// csrfMiddleware saves a CSRF token in the session
app.use(csrfMiddleware);

For each request that included a session cookie, the process was basically:

1. Check the connect-mongo sessions collection in the database to see if the cookie is valid
2. If it’s valid, load the session data (the user ID and anti-CSRF token) from the sessions collection
3. Passport middleware loads req.user based on the user ID
4. Our actual app logic runs
5. Finally, if the session is updated (for example the cookie expiry is extended), re-save the session to the database. (express-session does this when the response is sent by hooking the response object)

The corresponding data for every single session cookie that hadn’t expired had to be saved in the database. This added up to a lot of session records!

Before the migration sessions were the biggest cause of writes to our database, a significant source of reads, and the majority of data we actually stored in our main database (The actual content of Coggle diagrams is stored separately). Our goal with moving to signed sessions is to significantly reduce the resources needed to host this.

Part of the reason for the volume of session data is that we have very long-lived session cookies, as we prioritise people being able to easily return to their Coggle diagrams. People forgetting which email address they used to log in and ‘losing’ their diagrams as a result is our biggest source of support requests.

Part 2: Choosing a Signed Cookie Implementation

An alternative to storing sessions in the database is to instead store the session data in the cookie itself, so when each page is loaded the session data needed is immediately available in the cookies of the request. This is possible as long as there’s a cryptographic signature on the cookie to stop it from being tampered with. Someone can’t change their cookie to log in to someone elses account, as they have no way to forge the cryptographic signature.

There isn’t a formal standard for signing cookies, but the most common approach is to store a second cookie alongside each cookie to be signed, with a .sig extension to the name. This is the approach used by the cookies npm module, and the cookie-session middleware wraps this module into a convenient middleware which initialises req.session if the session cookie’s signature is valid.

We already use JSON Web Tokens in Coggle for authentication between our back-end services, so we also considered using JWTs as session cookie values. There would be a number of advantages/disadvantages to this:

• Public-keys could be used for signing, enabling our back-end services to verify signatures without access to the private signing key
• Cookie values could be easily encrypted, as well as signed, by using the related JWE standard.

• The additional information that makes JWTs portable (key ID, issuer, and using public-key signatures) also makes them bigger
• Public-key signatures are significantly more expensive to sign and verify.
• There are no readily available open source node modules for JWT-based session cookies.

Since we don’t need encryption and would prefer to use symmetric keys, we chose the cookie-session middleware. If you’re considering the same route, then think carefully about whether all of the data stored in your session should be unencrypted.

Part 3: Implementation

Secure Configuration:

The default for cookie-session (inherited from the cookies module), is to use the SHA1-HMAC signing algorithm. SHA1 has some weaknesses, so to be cautious we use SHA256-HMAC instead by passing our own Keygrip instance when creating the session middleware:

const signingKeys = new Keygrip([superSecretKey, ...], 'sha256');

const cookieSessionMiddleware = cookieSession({
  name: 'session-cookie',
  keys: signingKeys,
  maxAge: Session_Duration,
  httpOnly: true,
  sameSite: 'lax',
  signed: true,
  secure: true,
});

Handling CSRF:

We set SameSite=Lax on our session cookies, so it would not normally be possible for code on other sites to send potentially state-changing POST requests with the session cookie. However, in case people are using old browsers which do not support SameSite, or there is a bug in browser’s implementation, we still also use an anti-CSRF token for state changing requests.

Previously the CSRF token for each session was stored in the database, and the value sent with each request from the client compared against this - with signed cookies it’s instead stored in the cookie itself.

As the session cookie is stored as a HTTPOnly cookie, it is not possible for a CSRF script to read the value, even though it exists on the client.

It might be possible for malicious javascript to overwrite the HTTPOnly cookie, but in that case the cookie signature would be invalid.

This CSRF protection set-up is definitely a compromise, but as Coggle isn’t handling payments, we think it’s reasonable.

Migrating Old Sessions

It's important to migrate existing sessions so we don't log out users - running both the express-session and cookie-session middleware simultaneously isn't possible, as they both hook req.session and the response object.

As a result, we had to extract the logic from express-session which actually reads and verifies cookies (the getcookie function), and manually check the connect-mongo store, which is relatively straightforward:

const session = require('express-session'); // just for passing to connect-mongo, not used as middleware!
const MongoStore = require("connect-mongo")(session);
const legacySessionStore = new MongoStore({ ...  });

const loadLegacySession = function(req, callback){
  const session_id = getcookie(req, legacyCookieName, [legacyCookieSecret])
  if(session_id){
    legacySessionStore.get(session_id, function(err, session){
      return callback(err, session_id, session);
    });
  }else{
    return callback(null, null, null);
  }
};

With this in place, the final middleware for migrating sessions is straightforward. The migration is only temporary - once all old sessions have expired, we’ll be able to just use the new cookieSessionMiddleware directly instead.

const sessionMiddleware = function(req, res, next){
  // first delegate to the new session middleware:
  cookieSessionMiddleware(req, res, function(err){
    if(err) return next(err);
    // then, ONLY IF there's no user ID in the new style 
    // session, try to load one from the legacy session 
    // so we can migrate it:
    if(!(req.session && req.session.passport && req.session.passport.user)){
      loadLegacySession(req, function(err, legacySessionID, legacySession){
        if(err) return next(err);
        // if there was a passport user ID in the old 
        // session, migrate it:
        if(legacySession && legacySession.passport && legacySession.passport.user){
          req.session.passport = {user:legacySession.passport.user};
          // also migrate any existing CSRF token, so 
          // CSRF tokens used in pages which are already  
          // open remain valid:
          if(legacySession._csrf){
            req.session.csrf = legacySession._csrf;
          }
        }
        // delete the old session:
        if(legacySessionID){
          deleteLegacySession(legacySessionID, req, res);
        }
        return next();
      });
    }else{
      // if we have an authed session from the new cookie 
      // already then we're done:
      return next();
    }
  });
});

After this, the passport middleware works exactly the same as before, loading req.user from session.passport.user

Part 4: The Results!

We deployed the new sessions around January 27th. Based on one week either side of that, we saw some dramatic differences:

Database update operations, and corresponding db journal data, were reduced by approximately 80%, from 0.4MB/s to 0.08MB/s

Database volume busy time (which previously limited our peak scaling), reduced from approximately 15% to approximately 3%. In theory we can now handle peaks of over 30x our normal traffic volume, instead of peaks of only 7x!

(The reduction in the read ops of two of the volumes is primarily because they were being used as syncing sources for our off-site replicas - less journal data means less to be read for syncing)

And finally, 311 GB of session data and corresponding indexes can eventually be dropped from our database (multiplied across replicas, that’s over 1.5TB of disk space, or about $160/month)

---

Hopefully this has been an interesting read. If you thought we were crazy to store our sessions in MongoDB in the first place, well, we also used to store the entire contents of Coggle documents in a MongoDB database too… maybe we’ll write about that next!