Google App Engine and remote data caching

Aral Balkan is trying to run the website for his conference on Google App Engine, the same platform that snaptrip uses. In October, he posted twice on Twitter:

Great, you have no control over how Google App Engine caches data requests. Pulling in RSS feeds? Forget about it! (It uses Google Proxy and you can't tell it not to cache a feed or set the cache duration.)

I'd also noticed this, because the snaptrip login page (which does double-duty as an FAQ and news page - maybe I should rename it?) pulls in entries tagged 'snaptrip' from the Atom feed of this very weblog, and after my third post it failed to update for a good half a day. I wasn't that bothered, and didn't bother to double-check the documentation, which does clearly state that

App Engine uses a HTTP/1.1 compliant proxy to fetch the result.

Evidently this is the same problem as Aral has, and as usual, Tom Insam had an answer. It's from a slightly different direction (working with Google's Open Social containers), and as he said, it's what "everyone has done for years to bust caches you don't control": append an incrementing (or random) parameter to each request, which should mean that you're not hitting the cache. Having finally written a new blog post about snaptrip, I can confirm that this approach works. I'm not sure I'll leave it in - it seems a bit rude - but if timeliness is important, you might want to do the same.

It also occurs to me that, if every call to urlfetch is cached for some time, then you may find that repeated calls via API libraries might give somewhat unexpected results (although they're more likely to have changing arguments, anyway). Be careful out there.