Wednesday, January 22, 2014

Content-Security-Policy, Browser Extensions and JavaScript Templates

A couple of weeks ago I started seeing weird advertisements on some pages. I didn't pay it too much attention, but it was obviously wrong since some ads where displaying even in Youtube (as in, interrupting the Youtube video). Then one day I received this e-mail in my inbox:

This is our automatic e-mail notification of a Content-Security-Policy violation. Apparently I was injecting JavaScript pointing to into my own web page. WTF?

Ads and Trojans in Chrome extensions

Adware vendors have recently been acquiring Chrome Extensions to inject ads while browsing. Widely successful extensions are being sold to shady companies that turn them into trojans, in my case Awesome New Tab Page. These extensions have so many positive reviews that it will take a while for all the 1-star ratings to pull it down, or for Google to notice and drop it from the Chrome Store.

So Content-Security-Policy was saving my page from ads, even when it shouldn't. Chrome extensions are not constrained by Content-Security-Policy violations by design. What would be the point of notifying XSS injections, if everyone with a Chrome or Firefox extension will trigger a false positive? That would be an obvious way to disable AdBlock.

What is Content-Security-Policy?

Content-Security-Policy is the simplest way to restrict the list of allowed JavaScript locations for your site. Just by adding the Content-Security-Policy header you are 90% into a working solution for JavaScript injection attacks.

Why only 90%? There are different ways of including JavaScript code in your web page:

  • External, Embedded or Inlined: this is what CSP is all about. It will constrain the list of valid locations for your JavaScript files, and there are workarounds to pass arguments to your JavaScript code with CSP.
  • eval(). It was evil, right? Like, "friends don't let friends use eval()" evil?

Some libraries like Google Maps are still using eval, but that's considered a bug and should eventually get fixed. This is the real problem:

var s = _.template(
   'Complete ({{numresults}} processed). Your report is <a href="{{url}}">here</a>',
   { numresults: 5, url: '' }

All template implementations use eval under the hoods. Underscore, Lo-dash, Handlebars. Precompiled templates are still safe, but they require a separate file for each template. This is killer for simple cases like i18n messages, so we ended up using our own dumbed-down mustache implementation for this.

JavaScript libraries should not use eval() in order to be used by applications that specify Content-Security-Policy headers.

Content-Security Policy violations

One of the best things of CSP is that you can be proactive about attacks: you don't have to wait until being hacked to do something about it. By enabling CSP reports your admin account will be spammed about suspicious activities, in time to block the user or send him a horse head.

The only problem is that all browser extensions should be left out of the game. Right now support for this is partial: some extensions are still triggering a false positive (like the Awesome Tab Extension above), which renders the feature useless for the time being.

We are really happy with the status of CSP today. We trust more in the security of the system, even if not all browsers support it. It would be great if the notification system didn't send a daily reminder of browser extensions faved by our users, but that will eventually get fixed.

With CSP and mandatory SSL being included in http 2.0, things are definitely improving.

Wednesday, January 08, 2014

Running long tasks in App Engine for Java

Processing an unknown number of results in App Engine is a problem. "Unknown" means "potentially long-lived", and "potentially long-lived" is a ticket to pain.

The problem

App Engine is designed to scale by enforcing some requirements like short-lived requests: any query will timeout after 60 seconds for front-end requests and 10 minutes for Task Queues. Any Datastore queries should be processed before this limit expires, and any pending work postponed using cursors and Task Queues. These limits do not apply to backends, but neither do automatic scaling; with backends, you must manage scaling yourself.

And then, there is the expiration timeout for queries: as a protection from getting stale data, queries have a 30-second timeout. Forget about 60-sec front-end, 10-min queues and unlimited backends, your queries will timeout in all these cases after 30 seconds.

Since we are going to be forced to use cursors and postpone work, we can go with task queues and get automatic retries and exponential backoff for free.

Postponing work

Let's say that we are changing a attribute in the Datastore, and we need to propagate the change to other denormalized entities bound to it.

public class User {

   private Key key;
   private String name;


public class Ticket {

   private Key key;

   /** the owner of this ticket */
   private Key userKey;

   /** DENORMALIZED: The user name of the owner of this ticket */
   private String userName;


This "cascade update" is usually done in seconds, but you never know. As a general rule you don't want to design a system that fails on your best customers.

Through trial-and-error, this is the workflow that we devised:

  • Execute a query and start looping through results. For each result:
    • If still below the 30-second limit, process result.
    • If the 30-second limit has been exceeded, grab a cursor and execute the query again immediately, starting on that cursor.
    • If the 10-minute limit has been exceeded, grab a cursor and post the task again, starting on that cursor.
  • When there are no more results to process, end.

Wrap it up in a library, label it "open source", and we get queue4gae:

public class UpdateTicketsTask extends CursorTask {

    private Key userKey;

    private String newUserName;

    private UpdateTicketsTask() {
        // for Jackson deserialization

    public UpdateTicketsTask(Key userKey, String newUserName) {
        super("user-changes"); // the name of the Task Queue to use
        this.userKey = userKey;
        this.newUserName = newUserName;

    protected Cursor runQuery(Cursor startCursor) {
        EntityManager entityManager = EntityManagerFactory.getEntityManager();
        CursorIterator it = entityManager.createQuery(Ticket.class)
            .equal("userKey", userKey)
        while (it.hasNext() && !queryTimeOut()) {
            Ticket t =;
        return it.hasNext()? it.getCursor() : null;

} UpdateTicketsTask(userKey, "Foobar from Hell"));

As long as runQuery() returns a not-null Cursor the method will be invoked again, passing the cursor as an argument (null on its first invocation). The Task class should keep an eye on queryTimeout() to check if the 30-second time limit is close to expire. Anything else (re-executing immediately, reposting to execute later, JSON serialization) is being taken care of automatically.

Queue4gae is framework-agnostic: it works with any web and persistence framework for App Engine (as long as the later supports Cursors) and uses Jackson for serialization to be able to inspect tasks in the App Engine Console.

Some closing thoughts

This is a simple example: specifically, transactions and batch updates have been left out. You may want to add those for real-world code.

Is this implementation fool-proof? No. If a user starts changing names like crazy there is the possibility that two tasks will collide, leaving a ticket instance in an inconsistent state. If this is important, consider adding some controls to delay a task until the previous one has finished.

Is this implementation super-fast? No. This implementation will not run over a Map-Reduce but use linear execution, and it will require some time to process a large result set. If a new release of your application requires updating everything you will be better with appengine-mapreduce. On the other side, queue4gae is optimized for cases that are "usually done in seconds, but maybe not", consuming less quota.

Tasks should be idempotent: if anything fails, the same task could be executed more than once. You should plan for that.

First post as GDE!

This is my first post as Google Developer Expert for the Cloud Platform. Let me know if you like or hate it!

Tuesday, January 07, 2014

Google Developer Expert

It's official. I am starting 2014 as a Google Developer Expert for the Cloud Platform. This is the first time a GDE has been nominated in Spain.

This is extremely exciting news. I am keeping my independence as CTO at Extrema and Koliseo, but as a member of the GDE program I will get early access to features to be released by Google, more visibility of the Cloud Platform roadmap and a closer relationship with developer advocates at Google. This is awesome.

The process has been quite a trip. In order to get into the program, several Googlers had to inspect my history on the Internet and I had to pass a couple of interviews via Hangout. Some comments on that:

  • When I get nervous, I tend to talk a lot.
  • After some Red Bulls, I tend to talk a lot.
  • German supermarkets have some jumbo-sized 500ml cans of RedBull. That is a lot of fuel to speech.

The culprit

All things considered, it's a miracle how everything went fine. As a bonus, I enjoyed an entire night of sheer productivity (note to self: sleep seems to be optional. More research required).

Being a GDE means that I will be blogging and attending to more events (Google I/O, here I come), sometimes to talk. So, if you want to see me somewhere just drop me a line on Google+ or Twitter.

This is going to be an amazing year. See you in 2014!

Wednesday, October 09, 2013

Our love-hate relationship with OAuth

A couple of years ago we integrated our login page at Koliseo using all the social networks that made sense, up to the point were we had to stop before hitting the paradox of choice. We ended up with this interface:

This covers the most common OAuth2 providers that we could think about, using email (GMail, HotMail and Yahoo) or social networks (Google+, Facebook, Linkedin or Twitter). Half of them are using OAuth2, the rest are still stuck with OAuth1.

These are our conclusions after debugging this feature to death.

We love OAuth2

OAuth2 is a bit too good to be true. If you were developing your own user authentication system, this is a non-exhaustive list of features you should consider fitting in:

  • An interface to modify your password, maybe with a password strength meter in JavaScript.
  • Database encryption for passwords, in which case you must choose your encryption algorithm. Be prepared for a couple of nights reading about the demise of MD5 or flavors of salt for your passwords.
  • Password recovery via e-mail.
  • E-mail warnings whenever the user changes their password.
  • Detect when the same user has failed to log-in multiple times to display a captcha for a better entertaining experience.
  • Detect when a significant set of users are trying to log-in using the same (wrong) password in a short period of time, probably from a concrete set of IPs.

I'm no security expert, so probably I am forgetting something. But hey, you have an alternative:

  • Integrate OAuth.

Yup. Man, do we like shorter TODO lists.

Extras included with OAuth

If you are opting for OAuth, your friendly authentication provider may be including the following extras for free:

  • Two-factor authentication.
  • Password recovery by phone.
  • Backup e-mail account.
  • Suspicious activity reports, like someone trying to access your account using Tor.

Also, you are contributing to a more secure Internet: less user accounts means less security holes, because - breaking news - people reuse passwords for the hundreds of websites they visit.

Little does it matter that you are encrypting and salting your passwords if someone else is storing the same passwords in a text file in a shared Windows folder. OAuth does not remove the need to secure your system (SSL, XSS, Content Security Policy all apply the same) but anything password-related suddenly becomes someone else's problem.

Not everyone is happy with this system, and we got a 1% of users that didn't have or didn't want to login using any of these providers. We were OK with that; others may consider Mozilla Persona to be open to everybody.

We also hate OAuth

You know the joke, right? When you are integrating six different providers, you have seven different problems:

  • Google has the best implementation, which made it our first choice for testing (it helps that the majority of our users are using Google accounts). Overall, the experience was fluid and reliable.
  • Twitter has never abandoned OAuth1 officially, and their adoption of the OAuth2 standard has been partial and slow. They also insist in not including a scope to retrieve the e-mail of the user, which is a major inconvenience and made us almost drop it from our login page. Instead, we had to include an e-mail confirmation step just for Twitter. Thanks, guys.
  • Facebook will add a #_=_ hash to your callback URL. That's something to consider in JavaScript if you are using the hash for anything, like pointing to the selected tab. They also forget to return JSON sometimes.
  • Linkedin may return two different OAuth ids for the same user depending on the application that is asking. TL;DR if you are forced to change your Linkedin API key at any time in the future, your entire database of users will be lost.

OAuth is complicated

For all we tried, nothing saved us from was^H^H^H investing significant time reading the standard. We needed to at least get the concepts, the different authorization workflows and figure out the right integration for what we intended to do. For anybody looking to get introduced into OAuth, we can recommend the excellent OAuth bible by the Mashape guys.

The Bible didn't exist when we implemented this, so we did it the hard way: eat RFC, bitch.

OAuth adds new security holes

The following is the list of apps that I had authorized at the time of this writing. All providers like Google or Twitter have a page like this.

How often do I check this page? Believe it or not, once a year I empty this list thoroughly. With a less paranoid user it would be quite easy to introduce an entry that keeps their personal data hacked for the foreseeable future.

Steal the user password and use it to authorize an app, for example TweetDeck accessing a hacked Twitter account. Now the account will remain hacked even if the user changes their password, as long as they don't revoke the authorization.

But we are still happy

If you are looking for conclusions:

  • We still love OAuth, sort of, in a cyanide and happiness way. It's not perfect, but it lets us focus on real problems.
  • If you can choose, go with OAuth2.
  • Nothing will remove LastPass from our laptops, ever.
  • Remember to have a working e-mail address
If you want to give it a spin, you can test our implementation. Comments and feedback are welcome!

Monday, September 24, 2012

Sublime Text Editor tricks for web developers

Last weekend we were at Apache Barcamp Spain with a talk about cool tools for web developers.

Aside: if you are coming to a single event in Spain on a given year, make it this one. Barcamp Spain has developed its own soul, where the beer+networking experience is so much fun, and the food by itself is totally worth the trip.

Some people were asking for a transcript of the third part, our own set of tips and tricks to boost Sublime Text Editor for web development. So here it comes:

Feel comfortable

Chances are Sublime Text Editor is not your first editor. You probably had a previous life, in which you were already proficient with some editor (Eclipse, Bean, TextMate, whatever). This is about getting things done, so I would recommend to configure Sublime with the typical shortcuts that are hard-coded into your brain. Me, I am used to Eclipse shortcuts.

The idea is to feel comfortable. Get whatever you need to make Sublime as good as your previous editor of choice. Now, awesomeness can start.

Maximize space

I am not quite a fan of the distraction-free mode in Sublime, but I do want to maximize the used space on screen (the data-ink ratio, for Tufte fans). So here it goes:
  • Toggle sidebar on/off: Ctrl + K, Ctrl + B. This is useful if you rarely use it. In my case, it's Shift+Ctrl+R to open any file, which makes the sidebar mostly ignored.
  • See several files at once: Shift + Alt + [1234589] (you have to choose one). You can compare them side by side, top vs bottom, etc. Once you are done, you can switch back with Shift+Alt+1.

Surround with tag: Shift+Alt+W

This is a useful shortcut for inline tags (bold, italics, links), but not so much for block tags (div, p, etc). Try it:
  • Write any text.
  • Select a word, press Shift+Alt+W, then type b.

Block select: Shift + Right-click drag

This is great when you need to insert tags at the start of each line (p, li).
  • Prepare your list of data
  • Select the code and hit Tab to indent it a bit.
  • While holding the Shift key, right click and drag to select a rectangular selection of code.
  • Now type the start tag to get it inserted at the beginning of each line.

Multiple cursors

The previous are just concrete examples of using multiple cursors. For the purpose of understanding, let's say you can have multiple instances of a SublimeCursor class (I do not have any insight of the source code, but I found this approach easier to understand). This is a compiled list of things you can do, extracted by trial and error:
  • Ctrl + click: create new cursor.
  • Ctrl + double click: create a new cursor selecting the word that was double-clicked.
  • Shift + click: select using the latest cursor (so: ctrl + click, then shift + click to add a selection to the set of cursors)
  • Any keyboard action: Operate on all cursors at once. Keyboard operations (like Ctrl+V) will work, and so will normal typing (like hitting <end> to write the ending tag at the end of each line).
This is best understood with a set of examples:
  • Write any text.
  • Ctrl+click at the beginning of each line.
  • Write <p> or <li>

Forgot to enclose tag attributes with quotes?
  • Ctrl + double click on every attribute
  • Type "

  • Ctrl + double click on words that you want to make bold. 
  • If you want to select more than one, Ctrl+Click at the beginning, then Shift+Click at the end.
  • Hit Shift + Alt + W, then type b

There are tons of possible applications, and we found ourselves doing some really funky stuff. What is your personal trick? Use the comments section or tweet me @nachocoloma.

Monday, June 11, 2012

Using LiveReload on Linux

This is a guest entry from José Roberto Vidal, our newest acquisition at Extrema Sistemas. It's a perfect combination between me, not having enough time to write something coherent, and Roberto having something interesting to say and nowhere to stick it on. You can also find a Spanish version here.

LiveReload is an application that allows you to refresh automatically your browser whenever any file gets modified, even compiling any Sass / Less / CoffeeScript files. In a nutshell, it avoids you the tedious-to-death Alt+Tab F5.

Its installation in Mac OS or Windows is quite straightforward, but things get a bit trickier with Linux (surprise, surprise). This is what this post is about.

Installation and configuration

Leave the usual sacrifice of blood and caffeine by the side table. Thank you

Where Ruby gems get involved:

sudo apt-get install ruby1.9.1 ruby1.9.1-dev

Now you can use RubyGems to get the application. Inside your project folder, do:

sudo gem install bundle
sudo gem install guard
sudo gem install guard-livereload
bundle init

Edit the recently created Gemfile and add these two lines:

gem 'guard'
gem 'guard-livereload'

And execute

bundle exec guard init livereload

Monitored files and browser extension

Your generated Guardfile includes some predefined monitoring folders that you may have to modify if yours is not a Rails project. After that, just start the Guard server:

$ bundle exec guard
Guard uses NotifySend to send notifications.
Guard is now watching at '/home/me/workspace/project'
LiveReload 1.6 is waiting for a browser to connect.

At this point you probably can't wait to crack your knuckles and give it a try... Download the LiveReload extension for your browser (Chrome, Firefox) and configure it (right click -> options) to specify the hostname and port used by the Guard server (by default localhost:35729).

Once enabled, the guard-livereload server will notice it:

> Browser connected.
Browser URL: http://mymachine:myport/


If you get here, any modification to monitored files (CSS stylesheets, images, even JavaScript files) is noticed and applied immediately at the browser without refreshing.

Guard comes whith a lot of different flavors to automatize tasks in different environments. Get your own, or follow their nice screencast.


This is one of the labs that you can find in our HTML5+CSS3 course, together with Bootstrap, Sass and another 25 labs. It's just three days that you may choose to come to one of our public scheduled dates or make one specific for your own company.

Friday, October 28, 2011

Eclipse shortcuts for Sublime Text 2

I am a Linux guy. I used to pimp gedit with all plugins under the sun in an effort to transform it into TextMate, but no matter what I try it keeps being an awkward coding experience. Still on the search for alternatives, last month I discovered Sublime Text 2 and its entire new way of understanding text editing.

Sublime Text is just perfect for lightweight coding. I am not going to get into the details, since there are better places documenting this (see for example this list of tips and tricks). It is the perfect tool for times where a full-blown IDE is overkill: after all, it makes little sense to open Eclipse if you are going to spend the next eight hours debugging JavaScript or HTML (Agile and all that).
Now, this is the thing: I work with multiple major projects in the same week, performing different tasks and roles. I switch environments A LOT. Having to think "where am I editing?" before duplicating lines or saving files was messing into my flow.
So, without further ado, behold the

Ultimate eclipse shortcuts for Sublime Text 2

Not exactly rocket science, but still: go to Preferences -> Key bindings - User and paste this:

 { "keys": ["shift+enter"], "command": "run_macro_file", "args": {"file": "Packages/Default/Add Line.sublime-macro"} },
 { "keys": ["alt+up"], "command": "swap_line_up" },
 { "keys": ["alt+down"], "command": "swap_line_down" },
 { "keys": ["ctrl+alt+j"], "command": "join_lines" },
 { "keys": ["ctrl+alt+down"], "command": "duplicate_line" },
 { "keys": ["shift+ctrl+r"], "command": "show_overlay", "args": {"overlay": "goto", "show_files": true} },
 { "keys": ["ctrl+shift+s"], "command": "save_all" },
 { "keys": ["ctrl+l"], "command": "show_overlay", "args": {"overlay": "goto", "text": ":"} },
 { "keys": ["shift+ctrl+f4"], "command": "close_all" },
 { "keys": ["shift+ctrl+y"], "command": "lower_case" },
 { "keys": ["shift+ctrl+x"], "command": "upper_case" },
 { "keys": ["ctrl+d"], "command": "run_macro_file", "args": {"file": "Packages/Default/Delete Line.sublime-macro"} }

Save, and you will have a selection of finest eclipse shortcuts configured right into your editor. It is as far as I could get without getting into macros. You can help yourself and find the complete list of commands under Preferences -> Key bindings - Global, and complement this with our own discoveries.
If you like this, do not miss our new post about Sublime Text Editor tips for Web Developers.

Thursday, October 13, 2011

Apache Barcamp Spain: a wrap-up

After ten years of IT events, one would think that I have already seen everything under the sun: Spain, Germany, Switzerland, Flanders, Java-related or not, we all end up doing similar things. Come, learn something, attend networking, maybe get some book signed by some mega-crack author with a funny name. Exchange business cards or LinkedIn ids or Twitter names or QR codes. That's it.

Then something like Apache Barcamp Spain gets together and it's like - WOW.

If I had to coin some term, it would be "beer-oriented event". This being my first barcamp, I must say that the format is a fresh and new perspective much more interesting that the typical "get together and talk" mumble-jumble.

The talks

First off, you don't know what you are getting into. Literally. You make the trip to Seville, which in our case means getting three people into the A-Team van to co-star a 500+ km of roadtrip movie. You do all that, yet you don't know the talks that will happen the day after. Everyone proposes a talk and a voting process happens, and for all you know you could end-up learning new, surprising flavors of freaky.

Fortunately this was not the case: from Play! framework to GIS, Agile, Maven, Groovy, Web or Mobile, the landscape was full of well-seasoned experts. I loved the talks, but also the pace: instead of the typical 45-minute sessions, these were shortened up to 30 which leaves just the time to get to the point. No fluff, no fillers, just beans.

With three parallel tracks, you know you will miss 66% of the talks. I know I did, and some great talks are not included here. Sorry for that.

The following is my contribution about CSS. Bear in mind that these are introductory talks, but I tried to make them fun and throw something in for the most experienced.

The people

CELEBRITIES. All of them. Hey, we got a T-shirt to prove it.

Seriously, the organization made a big effort to make it clear that this show was about the people. I mean, of course all events do, but here you could really feel the love. Beer-time was the required great excuse to meet great people, and I found this to be one of the best formulas to get in touch. Relaxed and natural, like friends at a party.

Pity that we were just 100 people. I know quite a crowd that was left out because of the limited capacity.

The party

This event was co-sponsored by several companies, Extrema between them, but one company really stood out: Atlassian.

Atlassian sponsored the closing party, and a huge one at that. Flamenco, beers, jamón serrano, mojitos when the time was right. It was exactly the kind of party I would ask of Seville.

Funny fact of the night: After one hour of animated chat, I ended up recognizing Carlos Sanchez because of the avatar in his business card. Avatares reales ya :)

Tuesday, May 24, 2011

Firefox 4 and 3.6 at the same time

With the arrival of HTML5 web development has turned into the sunny side of the hill again. These days the world would be a perfect place if I could launch two browser instances at the same time, one with support for the latest standards (almost) and another without.

This is something that has been documented online, but from fragmented sources: it is possible to have Firefox 3.X and 4 installed and running at the same time. Actually, it's quite simple as the whole installation process takes less than ten minutes.

Step one: Installing the thing

This one is a no-brainer: you should install both Firefox 3.6 and 4 in separate folders. For Windows, this means using different folders and creating separate shortcut icons.

For Linux, it means installing Firefox 4 from your favorite package manager (.rpm or .deb) and downloading Firefox 3.6 as .tar.gz and uncompressing it somewhere like /usr/local/firefox3.6 (depending on your Linux version, you may prefer to do it the other way around).

In both cases, you need the link to download an older version of Firefox, which is somehow hidden in their website (search for "All systems and languages" and "Other Firefox Downloads", or just click here).

Step two: add -no-remote

Firefox will check at launch time if there is a running instance and attach to it instead of starting a new one, which is definitely not what we want since it will not check for versions in the process. Launching 3.6 will result in your existing Firefox 4.0 window getting the focus, but nothing else. No new window. No "ta-daaaa!"

This behavior can be overriden by adding -no-remote to your program shortcuts (Windows or Linux). That will skip the check, and your new window will open.

Step three: use different profiles

You may be able at this point to launch separate windows for Firefox 3.6 and 4 at the same time, but this is no good if they start stepping on the plugins of each other. The plugin release that may be valid for 3.6 is not for 4, and viceversa (fact is, anything other than Firebug has big chances of not having been ported to Firefox 4 yet). You need separate plugin folders.

The good news is that your plugins are stored in a folder relative to your Firefox profile. Most users (99.9% or so) have a single "default" profile that is selected automatically and are not aware about profiles (now you are; lucky you!). Just launch:

firefox -ProfileManager

Create two separate profiles, which will allow for two separate sets of plugins. I like to name them "Firefox 3.6" and "Firefox 4", in an attempt to be original and unexpected. They will be stored inside your user home folder. Accept and close the window.

To avoid the profile selection window over and over again, the profile can be set at your Firefox launcher (or shortcut icon, if you are using Windows). I will assume some locations and profile names here:

/usr/bin/firefox -no-remote -P "Firefox 4"
/usr/local/firefox3.6/firefox -no-remote -P "Firefox 3.6"

"C:\Program Files\Firefox 4.0\firefox" -no-remote -P "Firefox 4"
"C:\Program Files\Firefox 3.6\firefox" -no-remote -P "Firefox 3.6"

That's it. Now you can develop using both versions with separate Firebug installations.

More. More. More.

This is an excerpt of the first lab in our HTML5 and CSS3 course that is starting tomorrow. This 4-day course includes 20+ labs filled with polyfills, mobile and print CSS, Internet Explorer workarounds, transitions, form controls and mostly everything under the CSS3 / HTML5 sun.

Thursday, February 10, 2011

Applications dealing with multiple timezones

This is a requirement that can hardly be considered common. Most applications are not forced to deal with more than one timezone: intranet applications, nationwide deployments (most European Countries enjoy this great invention that is Central European Time), even air navigation systems place everything in UTC.

We recently deployed Koliseo on AppEngine. This is still a work in progress, where some features have raised interesting questions: what happens when a friend tells you to meet on Friday at 17:15? How should you store this in the database?

Multiple painzones

I love the way computers measure time: milliseconds since 1970. Holy cow. No leap years, no arbitrary seconds-in-a-minute conversion, no Daylight Saving Time and it doesn't matter where you are. Give me a number that I can sort and compare. Computers do not have problems with dates. Humans do.

Then, the only problem is parsing and formatting these dates.

Parse date

Ye olde way of providing time information says that you should put a test field, label it "when" and let the user do his thing. Suppose that he introduces "25 Oct 2010 at 00:00", sitting comfortably in his nice office in Madrid (GMT + 1, where DST applies).

  • What the user means: "25 Oct 2010 at 00:00" in NiceAndWarmOffice@Madrid
  • What the server understood: "25 Oct 2010 at 00:00" in local server time. For AppEngine this is UTC, which means one or two hours less, depending on the season.

The user wanted to express Oct 25 00:00 (Madrid time), but the server understood Oct 24 23:00 (Madrid time). This is the value that will get stored in the database.

You need to ask for the timezone of the user. It can be automatically deduced using javascript (it comes in small and jumbo sizes), but it is nice practice to let the user modify the timezone later, just in case.

This timezone must be stored associated to something, which in our case is a venue. From there on, any date introduced in Koliseo must be associated to this entity. Following this example, any Performance must specify where before telling when if we want to be able to parse the introduced timestamp.

The code to parse is quite straightforward thanks to Saint JodaTime:

DateTimeZone dtz = DateTimeZone.forID(timezoneID);
DateTimeFormatter formatter = DateTimeFormat
 .forPattern("yyyy-MM-dd HH:mm") // replace with your favorite format string
Date date = formatter.parseDateTime(dateAsString).toDate();

Format date

How do you want to display dates to the user?

  • Relative units: This is the best way to display unambiguous dates ("ten minutes ago", "one week ago", etc). This is done in Loom by using l:formatDate, but the first case I can remember of this practice was found in Ruby
  • User time: This is usually not interesting, since a date is associated to something happening (in Koliseo, a performance) and is usually meaningful inside their own context and timezone.
  • Original timezone: This is the most common case. Example: tell me that the show is scheduled at 17:30 (local time of the show, in this case Madrid), not browser time (Helsinki or whatever).

To format dates in the original timezone you can use the same snippet of code just replacing formatter.parse() with format().

Javascript code

Whenever you send or receive a Date from the server, the typical JSON serialization uses "milliseconds from 1970", which can be a problem. This value cannot be used as is because the timezone information is missing. 17:35 in Madrid will be deserialized as 16:35 if I am browsing from London, which is wrong.

Twitter solves this by serializing dates as text, then you can parse and separate the parts to display (day, month, year, weekday). To test your javascript code, just add "-Duser.timezone=UTC" to your server launch script and play a little with your application. If you do not see any date mismatch, everything is good.