3 posts from November 2008

Last week Safari 3.2 was released, with the usual minimal release notes: "This update includes stability improvements and is recommended for all Safari users." The security notes were somewhat more forthcoming, but even there, not everything is covered, for as well as bug fixes, 3.2 quietly added support for two big security features: EV SSL, and Google Safe Browsing.

Neither of these changes, obviously, is covered in the release information, but since the (very good) MacJournals writeup of details of the anti-phishing features was reposted at Macworld, there's been a small whirl of further commentary, especially as the latter includes data collection for Google. Most of the (sensible*) concern has been raised because Apple's terms and conditions, unlike those of Firefox (who also use the Google Safe Browsing API), allow Google to make use of the data sent as a result of surfing using this plugin for any purpose, not merely enhancing that particular service. This might not be so bad if it wasn't also for the fact that the Safe Browsing checks fetch and send data by default.

Personally, though, I can't say I'm bothered by either of these. I'm sure Google get far more useful information from searches and opt-in service usage than they get from partial hashes returned when browsing to potentially hacked sites. As for defaulting to using the service, well, both Chrome and Mozilla also do that, and as with Firefox, Safari offers a preference to disable phishing detection.

What is more surprising to me is that so few people have connected the release of 3.2, and its emphasis on security over features, to the removal of Safari as a "safe" browser from Paypal's list in February:

"Apple, unfortunately, is lagging behind what they need to do, to protect their customers," [PayPal security chief] Barrett said in an interview.

I have little doubt that there's been behind-the-scenes back and forth between PayPal, and similar organisations pushing these changes, led Apple to release this sooner rather than later, in the 3.0 branch (rather than waiting for Mac OS X 10.6 and Safari 4.) Perhaps a more sensible place for people to raise questions is whether EV-SSL and Safe Browsing are actually useful, or if they're merely security theatre? Now there's a well-researched comment piece I'd like to see.

* There's also a lot of kneejerk "OMG Google haz my datorz!" nonsense, but reading the article makes it clear that only hashes of URLs are checked, and even that's only when a partial hash is matched against a hash of your current URL.

Last week Kellan from Flickr published my interview on code.flickr. I'm still somewhat amazed that they chose me to ask, but then I'm also pleased at how much people are liking snaptrip, and I'm happy to see my words in print, as it were.

I actually compiled my answers a couple of weeks before it was posted, hence the reference to groupr as a "lost project". Now, of course, it's back, but I've already posted a couple of times about that. What I would like to do is - finally, and belatedly - document (and update the released version of) my EXIF machine tagger.

Why bother with such a thing? Flickr will extract EXIF metadata, but it won't allow you to do any aggregate queries on it. (Well, that's not quite true; at dConstruct 2007 Tom Coates leaked some URLs which I picked over, but they don't cover all the useful things I'd like. Plus, it's not documented.) By extracting all the data from my photos into machine tags (and a local SQLite database), it becomes possible to point people at all the photos taken at the wide end of my widest lens, or those taken with a particular make of camera (and to do more complex queries locally).

With that out of the way, how do you go about such a thing? Well, as usual, it's actually a fairly simple joining operation. Get a list of photos, and for each of them, get the EXIF data (using flickr.photos.getExif), then store the data locally, and add tags back to Flickr. There's not much munging invovled - I convert spaces in the EXIF field names to underscores, and some things get put in the "file:" or "camera:" namespace, rather than "exif:" - so it's all pretty straightforward. (I do preserve spaces in the EXIF values, though, by quoting my arguments to the addTags method.)

I also add an meta:exif field with either "none" or the epoch seconds of the time of tagging, so that it's easy to exclude previously-tagged images from being examined again. Another minor niggle is that, to add tags, a script has to be authorised. I copied the code chunk from the flickr_upload script in a Perl module, and it seems to work for me.

However, the fact that users need to get an API key, secret, and then a token, is naturally going to limit the audience for such a script. A few other users have metadata in the "exif:" namespace, but it's not exactly common. It's hard to turn the script into a web app, too, since it needs about a second per image to run, and the first run has to examine your entire library, which these days is typically thousands of images. I may still do it, but I haven't bothered for months, so I wouldn't count on it.

Another drawback is that machine tags are normalised at Flickr. This means that when I query on exposure bias, both -1/3EV and +1/3EV show as just "exif:exposure_bias=13ev". I've been thinking about ways around this - by querying raw tags - but it's not straightforward. (Ways around this normalising, and ways of getting all predicates for a namespace, and values for a namespace (at least within a given user's photos), would have made my list for "things you'd like to see in Flickr" if I'd felt able to get away with being so demanding.)

One final observation is that the script's in Perl, and uses XML (which is, apparently, sometimes compressed at Flickr's end; at least, I had to add Compress::Zlib at one point for some reason). If I was to redo it, either in Python or Ruby, the data would all be fetched as JSON, and it'd probably get a few more users. Ah well. Installing the prereqs shouldn't be too hard.

That said, of course the script, as is, proved useful. I run it manually after an upload, while Tom, who is (as ever) a bit more sensible, has his fork running as a cron job. Either way, please download it, play, and feel free to let me know what you think.

I've finally launched a new version of groupr that includes a view that I've wanted for ages: recent photos in your groups. Like this, in fact:

The obvious inspiration for this is Flickr's own recent photos from your contacts page, which, as the name suggests, shows photos that your friends, family and others have posted to the site, in reverse chronological order. It's a great leaping off point if you want to follow other people's work, and I'm sure that without it I wouldn't use Flickr nearly as much.

It's pretty self-evident that a similar page for your groups would be a good idea, then, and it has been much requested over the years. The problem is that building it isn't easy. You have to deal with the API join: to get information on the photos in each group you have to make an API call, so for 140 groups, that's 140 calls, each taking up to a second.

For a while, I thought this was an insurmountable problem for groupr: nobody's going to wait for over two minutes for a page to load. However, this problem also exists on the groups page, and so I solved it the same way: by using client-side calls (AJAX, if you like, although technically I use JSON by preference), cached by the server for later use in generated pages.

Last week I started putting the theory into practice, and the final conceptual leap was in the order I made the group calls. Initially, I queried the groups in alphabetical order, but it occured to me that the more photos were in a group, the more likely it was to have recent updates. Of course, this isn't perfectly reliable, but having made the change, it seems to work; typically, the "recent photos" page fills up with the most recent photos fairly quickly.

(I use a similar technique on the page that lists your groups, and lets you sort them in various ways. The list page needs a little more work, but I expect it's still handy.)

Having finally implemented this, I'm pleased at how easy it was now I've laid the groundwork, and also a bit surprised that nobody's ever done this before. Hopefully the recent view page will prove useful to some of you. Feel free to leave comments here if you're seeing errors or have suggestions on how to improve it.