11 posts tagged “api”

(With apologies to Samuel.)

Hi, I must be going. I'm running into more and more issues that stop me being happy posting here.

• Data ownership

  My other hosted blog is on Tumblr. Unlike Vox, they let you use a custom domain so that you control the URLs. I realise Six Apart make a nice living off doing this with TypePad, and Vox isn't really aimed at the sort of people who care, but I do, so maybe this isn't the right place for me. Related to that:

• Badly documented, badly supported API

  To get my content out of Tumblr, I need use only one API call: /api/read/json, with well-documented paging parameters. In contrast, I've spent several hours grappling with the Atom API that Vox supports, finding it inconsistent, barely-documented, broken, and otherwise infuriating. I dare say eventually I'll manage to liberate all my data, but damnit, it should be easier than this. (If you're lucky, there'll be a follow-up on how I managed to get on, with more technical details, later.)

• Lack of one-click export support

  ... in fact, it should be that easy. I believe Blogger may recently have added this; certainly Pownce had to when they were acquired and shut down. I don't want to wait until a crisis point, though; I want backups of my content as and when. The recent loss of JournalSpace and AOL Homepages show you can never be too paranoid.

• The HTML editor still doesn't work in Safari

  Well, it's better than it was; instead of locking up your browser, it does now allow you to post. Unfortunately, it also inserts loads of random tags that mess up formatting when I come to copy the entry to anywhere else

• Even in Camino, the WYSIWYG editor can mangle things
• There's no raw view, which makes fixing the editor's bad HTML even harder
• The best bits are now in Movable Type's UI

  When I started using Vox, I still had Movable Type 2.6 on my personal site. I still do, but I have a version of 4.1 or something on my laptop, and at some point (probably sooner rather than later, now) I'll deploy it to husk.org proper. A lot of the niceties of the interface in Vox are replicated over there.

  (Meanwhile, Six Apart still shuffle backwards and forwards on whether MT is free or not. I think for my uses, it's definitely free as in beer, but I can choose between whether I have a copy that's free as in speech or not. Sigh.)

• Lack of control over page design

  Editing a header image and choosing from some (admittedly pretty) backgrounds is a bit poor when you compare it to Tumblr. Sorry.

• No stats/analytics

  Even Flickr has stats now, and Tumblr lets you add in Google Analytics to your HTML.

• The UI feels too "heavy"

  When I started using Vox, it felt nice and simple compared to that MT 2.6 editing screen. Since then, however, we've seen Tumblr, Twitter and ffffound, where the posting interface is a simple text box, bookmarklet, or similarly stripped-down. Editing on Vox feels like it's a battle far too often.

• There's no feeling of community / not a one-stop shop

  Vox feels like it was intended to fix some of the issues with LiveJournal and the isolated blogging of MT and TypePad, but sadly it never hit critical mass. Similarly, the idea of allowing users to upload all their stuff was a nice idea, but it doesn't seem to have worked out, for me anyway. (Once again, Tumblr does both of these right, for me anyway.)

• It's not going anywhere

  I don't know what Six Apart's focus is, but Vox definitely doesn't feel like it's part of it. While I'll continue to watch them with interest, it feels like a lot of the work that's been going on hasn't really had any useful impact over the last couple of years.

I'm not going to give up posting; as I said above, I do have a Movable Type installation I'll be reverting to, and I'll continue reading what my friends and neighbours have to post. However, I don't feel comfortable posting here any more. Sorry.

When I posted about lib-flickr-minimal, I noted that the newly-launched flickr.places.placesForUser method made a more interesting demo of data you could fetch when authenticated than, say, showing a user's most recent private photos. Evidently the developers at Flickr agreed it was an interesting concept, because over the last couple of months that area of the API has been extended considerably, As a result, I've expanded the demo into an AppJet application of its own.

Where? What? When? is the result. It shows you, on a map, the locations with the most photos according to a given criterion: by default, that's a tag, but it can also show your photos, or those from your friends and family, or your contacts. You can then inspect a place and see the most recent relevant photos, or the most popular tags, for that location.

How did that evolve from the initial demo app? Instead of simply printing a table based on Flickr's response into the document, I directly plotted the results on the map. I added a small form to enable the choice of criteria, and when Flickr added the placesForTags method, I added that as a choice. Belatedly, I realised that would also work for users without authentication, so I removed the requirement to authenticate, and made tags the logged-out default. (The image above shows a slight change to the initial results: it's the same tag, London, but at the neighbourhood, not locality, level. All of the locations are within the greater city's area, which probably won't be a surprise, but that's not true for Paris. Evidently, what happens in Vegas doesn't always stay there.)

The design of the application isn't quite settled, but I knew I wanted to replace the standard Google Maps pushpins with partially-transparent circles. Initially, I went with red, but when I showed it to colleagues, they said it reminded them of maps of bomb blast radii, so I spent a while looking around for the right colour, before settling on a yellow. The circles themselves are scaled according to the natural log of the number of photos for that location; I played with square roots as well, but I feel that logarithms give the right sense of scale.

The last piece of work I did was adding tag display for locations, using the tagsForPlace method. These tags can be surfed: clicking on one will load a new search for the given tag. It's noticable that the first few tags for most places are almost always place names, while common tags seem to share a familiar pattern of scattered, similarly-sized circles across the US, Europe, south-east Asia and coastal Australia.

There's still a few things I could add; tag persistence in URLs (to make it easier to share pages), better loading indicators (especially initially), options on which photos are shown, and links to view the search on Flickr itself, for example. There's also a missing question: while the API methods support maximum and minimum times, I haven't yet added options to allow you to show When? However, for now I think I've done enough (and I'll note that the site has a link to view the source of the application, if you fancy hacking on it yourself.) Enjoy.

Last week Kellan from Flickr published my interview on code.flickr. I'm still somewhat amazed that they chose me to ask, but then I'm also pleased at how much people are liking snaptrip, and I'm happy to see my words in print, as it were.

I actually compiled my answers a couple of weeks before it was posted, hence the reference to groupr as a "lost project". Now, of course, it's back, but I've already posted a couple of times about that. What I would like to do is - finally, and belatedly - document (and update the released version of) my EXIF machine tagger.

Why bother with such a thing? Flickr will extract EXIF metadata, but it won't allow you to do any aggregate queries on it. (Well, that's not quite true; at dConstruct 2007 Tom Coates leaked some URLs which I picked over, but they don't cover all the useful things I'd like. Plus, it's not documented.) By extracting all the data from my photos into machine tags (and a local SQLite database), it becomes possible to point people at all the photos taken at the wide end of my widest lens, or those taken with a particular make of camera (and to do more complex queries locally).

With that out of the way, how do you go about such a thing? Well, as usual, it's actually a fairly simple joining operation. Get a list of photos, and for each of them, get the EXIF data (using flickr.photos.getExif), then store the data locally, and add tags back to Flickr. There's not much munging invovled - I convert spaces in the EXIF field names to underscores, and some things get put in the "file:" or "camera:" namespace, rather than "exif:" - so it's all pretty straightforward. (I do preserve spaces in the EXIF values, though, by quoting my arguments to the addTags method.)

I also add an meta:exif field with either "none" or the epoch seconds of the time of tagging, so that it's easy to exclude previously-tagged images from being examined again. Another minor niggle is that, to add tags, a script has to be authorised. I copied the code chunk from the flickr_upload script in a Perl module, and it seems to work for me.

However, the fact that users need to get an API key, secret, and then a token, is naturally going to limit the audience for such a script. A few other users have metadata in the "exif:" namespace, but it's not exactly common. It's hard to turn the script into a web app, too, since it needs about a second per image to run, and the first run has to examine your entire library, which these days is typically thousands of images. I may still do it, but I haven't bothered for months, so I wouldn't count on it.

Another drawback is that machine tags are normalised at Flickr. This means that when I query on exposure bias, both -1/3EV and +1/3EV show as just "exif:exposure_bias=13ev". I've been thinking about ways around this - by querying raw tags - but it's not straightforward. (Ways around this normalising, and ways of getting all predicates for a namespace, and values for a namespace (at least within a given user's photos), would have made my list for "things you'd like to see in Flickr" if I'd felt able to get away with being so demanding.)

One final observation is that the script's in Perl, and uses XML (which is, apparently, sometimes compressed at Flickr's end; at least, I had to add Compress::Zlib at one point for some reason). If I was to redo it, either in Python or Ruby, the data would all be fetched as JSON, and it'd probably get a few more users. Ah well. Installing the prereqs shouldn't be too hard.

That said, of course the script, as is, proved useful. I run it manually after an upload, while Tom, who is (as ever) a bit more sensible, has his fork running as a cron job. Either way, please download it, play, and feel free to let me know what you think.

I've finally launched a new version of groupr that includes a view that I've wanted for ages: recent photos in your groups. Like this, in fact:

The obvious inspiration for this is Flickr's own recent photos from your contacts page, which, as the name suggests, shows photos that your friends, family and others have posted to the site, in reverse chronological order. It's a great leaping off point if you want to follow other people's work, and I'm sure that without it I wouldn't use Flickr nearly as much.

It's pretty self-evident that a similar page for your groups would be a good idea, then, and it has been much requested over the years. The problem is that building it isn't easy. You have to deal with the API join: to get information on the photos in each group you have to make an API call, so for 140 groups, that's 140 calls, each taking up to a second.

For a while, I thought this was an insurmountable problem for groupr: nobody's going to wait for over two minutes for a page to load. However, this problem also exists on the groups page, and so I solved it the same way: by using client-side calls (AJAX, if you like, although technically I use JSON by preference), cached by the server for later use in generated pages.

Last week I started putting the theory into practice, and the final conceptual leap was in the order I made the group calls. Initially, I queried the groups in alphabetical order, but it occured to me that the more photos were in a group, the more likely it was to have recent updates. Of course, this isn't perfectly reliable, but having made the change, it seems to work; typically, the "recent photos" page fills up with the most recent photos fairly quickly.

(I use a similar technique on the page that lists your groups, and lets you sort them in various ways. The list page needs a little more work, but I expect it's still handy.)

Having finally implemented this, I'm pleased at how easy it was now I've laid the groundwork, and also a bit surprised that nobody's ever done this before. Hopefully the recent view page will prove useful to some of you. Feel free to leave comments here if you're seeing errors or have suggestions on how to improve it.

Overnight Flickr launched their video support, which I'm sure you can read about everywhere else. I'd like to note (as much for myself as anything) how to handle videos in the API.

As with the main site, videos are treated just as funny photos. As kellan explained on the Flickr API group, the API does just the same thing. There's a new "extras" parameter, "media", which returns either "photo" or "video". What isn't fully explained on the thread is how to display a video as a video, rather than as a static thumbnail (the way Flickr themselves do on, for example, pool pages).

The answer is to use the embedding code shown with each video. (You'll need to be logged in to see that page.) Seasoned API users will note that the photo ID and secret are in the code that Flickr gives you, so it looks like it'll be easy to check the media attribute for a photo, and either show just the image, or use the Flash code (with the appropriate substitiutions) to show a playable version. Unfortunately, if you can't figure out the aspect ratio of the image (typically by using the o_dims extra argument to methods like flickr.photos.search - and yes, it works for videos), you'll need to do an extra call (to flickr.photos.getSizes) to get those arguments for the <object> tag. (This is a problem for images, too, but I'd be happier leaving height and width tags out of an <img> than an <object> tag.)

In summary, then, supporting videos on Flickr through the API looks as if it should be straightforward, and in fact even if you do nothing then it'll work (albeit with videos treated as images). Nicely done.

Launched today at Etech, Fire Eagle is a new Yahoo service that lets apps easily store and retrieve your location. Its big selling point is that it can take almost any input (postcodes, addresses, cities, lat/long pairs) and then give a heirarchal output to apps, with levels of precision that the user chooses.

Unfortunately, while almost all of these levels ("country", "city", "exact") make sense whichever country you're in, there's one that doesn't. In the UK, "postal" will return a full postcode, which locates you to a building (at best) or a street (at worst). In the US, by contrast, it reutrns a five-digit zip code, which covers about 5000 people, and is obviously far less revealing. (I've no idea what it does outside these two countries, but I'd love to know.)

Now, obviously I can work around this in a couple of ways. Firstly, I could be fuzzier about the location I give to Fire Eagle, but this is silly; then nobody gets the better data. Much better, then, to use one of the other choices that are offered, but this begs the question; why have the postal option at all?

Still, it's early days, and as well as looking forward to seeing how the service (and its uses) evolve, I'm hoping there'll be a few tweaks for international consistency within the service itself.

A year or so ago, whilst writing groupr (RIP), I came up with what I thought was a useful name for something I found myself doing a lot: the API join. I'm fairly sure this is common to a lot of Web (2.0) APIs, but it's especially common with Flickr. For example, take groupr. First, it would do a call to get the groups you're a member of. For each of these, it then fetched the photos in the group. Obviously, this has a problem: as the number of groups you're a member of goes up, so does the number of calls to the API - and each call takes about a tenth of a second. The only way to mitigate this, and the solution groupr used, was to page the groups - and even that leaves you making as many calls as you have groups on the page.

The problem reared its head again when I was looking at doing a ffffound-inspired Flickr favourites app. I wanted to display the usual Flickr size, rather than square thumbnails (as Flickr's own favourites page does). Unfortunately, the standard call to get favourites didn't list the size of the photos, and I really didn't want to spend two seconds fetching them all. Other people have raised similar questions on the Flickr API group discussions; for example, here's one about getInfo and getExif, and here's another about getting photo sizes.

Imagine my surprise, then, when I looked at the documentation for flickr.photos.search and noticed a new argument to the "extras" parameter: o_dims. It turns out this returns the original height and width, and is also available in the favorites methods, so now it's possible to avoid doing those calls, and to embed derived height and width for web-scale images from a single call, even for the 36 or so images on Flickr's version of the page.

Of course, this is simply because the API has now moved the join deeper; instead of being at the API level it's being done inside Flickr (presumably at the database level). In fact, I suspect that last weekend's database downtime may not be unrelated (perhaps it was needed for the launch of Apple TV's Flickr slideshows?). It also doesn't help with the other methods, such as getExif (there's a reason I've moved some of my EXIF data to machine tags, which are fetchable with another extras parameter to many calls).

Facebook, interestingly, allows a SQL-like query language as part of their API access, but I wonder how they deal with queries that could bring the database to its knees. I do notice the line

In order to make your query indexable, the WHERE should contain an = or IN clause for one of the columns marked with a *.

Is that an enforced criteria, or is it merely a recommendation, and do they return long-running queries without results to keep up database performance? It's the sort of thing I'd love to see Flickr add to their API, but I can imagine the problems are far from trivial, and in the meantime, I'm very happy to see one API join bite the dust.

Things you should be able to do with Flickr favourites 1, but can't:

The idea of Hackday London 2007 was, unsurprisingly, to hack. Beforehand I'd had little idea of what to do, but candace managed to come up with a few ideas. Notably, one evening last week she noticed some photos on SpaceWeather.com of the International Space Station, as taken from the Netherlands, and thought that perhaps we'd have a chance of seeing it. We checked Heavens-Above, a venerable satellite tracking/prediction site, and we caught a flyby which included a moment of brightness as the newly-deployed solar panels caught the sun.

Wouldn't it be great, she mused, if it was possible to get messages to your phone when such things were going to happen? As well as ISS flybys, there are also Iridium flares, where the redundant communications satellites reflect sunlight down to the ground, and it'd be nice to be told about those, too. It looked like we had an idea.

Implementing the idea wasn't terribly tricky, either. There are two parts to it. Firstly, there's a scraper for Heavens-Above. We set up a special London account, and wrote a script that authenticates against the site,  and downloads and parses the data tables for the ISS and iridium flares. This goes into a plain text file, with the date and time as one field and the text message to send as the next. Since the data tables list events for seven days in advance, this script doesn't have to run frequently- at the moment it's doing so once a day.

Secondly, there's the sender script, which runs every five minutes. It reads in the data file, parses the date (slightly hackily- I'll need to fix that eventually), and, if the event is within twenty minutes, sends it to Twitter (which we use as it's a simple way of sending SMSes to multiple users). Also - and this is where where the required use of a BBC or Yahoo! API comes in - the script checks the Yahoo feed's "current weather conditions" value, and if it's likely you'll be able to see the event, continues onwards to send it. Otherwise, it doesn't bother (but I do get an email from cron telling me what the weather actually was).

I was able to get all the coding done and put it on my colo before one of the flares that evening, but sadly the weather wasn't quite clear enough and we didn't have visibility in the right direction. Still, we had text messages and a Twitter page that we could point to as proof of a working hack, so we went home. (There's an aside here about the difference between the SF culture and London's more lackadaisical one, perhaps, but it'll have to wait for another day.)

I spent another 30 minutes on Sunday morning tidying up the verbosity of the script (it now only prints, and hence sends email, at the same time it has output to send), and then came the slightly nervewracking presentation, which thankfully seemed to go down well, despite us having nothing really in the way of UI to demo. (One of the best things about Hackday- you don't have to write up yourself...)

That evening I added a feed for SF (and took the chance to comment alongside all the bits of code that needed changing). If you're interested in getting messages for passes Above London or Above SF then get a Twitter account and follow the appropriate user. (Longer-term Twitter users might want to adjust their phone notification settings if they want to get SMSes late at night.)

What's next, then? Well, I've since looked at a Perl module (Astro-satpass) that would have let us cut out Heavens-Above, and possibly opened the door to more customised notifications. In particular, we've made some arbitrary decisions (we don't send flares that don't climb above 20°, for example) and it's really hard to add a new location. It'd be nice to remove those limitations, but doing so introduces a new problem; namely, Twitter is a very easy platform for notifications, although I'm concerned about its reliability and timeliness. Customised messages mean either abandoning it or (ab)using the direct messaging syntax.

It was notable that Twitter was used in a fair amount of the hacks (from the live blog post written during the presentations, at least 10%). I think it'd be a perfect fit for Yahoo, alongside Flickr and del.icio.us, as a developer-friendly site that, perhaps, needs a bit of resourcing behind it to make it truly reliable (and, perhaps, more international; the UK number isn't always cheap). How about it?

Anyway, the two applications/accounts/bots should now run without human intervention (at least until that date hack I mentioned rears its head around Christmas), and hopefully I'll remain inspired to tackle the more complex project of personalised feeds and notifications later in the summer. For now, enjoy spotting satellites.

For a very long time I've been intending to learn a better way of writing web applications. I had a very small potter with the the obvious candidate, but that didn't seem to work out. Screencasts don't make good tutorials, perhaps. There were a raft of Perl MVC web frameworks to look at, but that never quite worked out either.

When Fotango announced a competition - with prizes! - for the best Zimki application, though, I seemed to find enough motivation to do something new. Zimki is a Javascript-based hosted web development service. You write the server-side logic with Javascript, and there's a provided persistency framework, advertise RESTian web services and a choice of templating solutions (Trimpath's JavaScript templates and TAL). There's also a client-side library which uses MochiKit to allow you to make web service calls.

So, what to code? Well, I'd been intending to develop a small web app to keep an eye on the photos in Flickr pools for a while, and this seemed to be the ideal opportunity. Furthermore, Sven-S. Porst wondered if you couldn't get your group memberships in an alphabetical list. Seemed like the idea was sorted out then.

There's an entire other post to be written about the pitfalls I found in developing the app, but I'll have to trawl IRC logs to see what I was complaining about. Skipping to the end, the app got to a state where I was happy to submit it on the closing date of the competition, and here it is: groupr.

It's pretty straightforward really, but there is a bit of hidden complexity. Firstly, any request to the root of the domain checks to see if there's a Flickr authentication token in your session. If not, you're prompted to sign in (and shown a lot of explanatory text about it). If you are, you're directed to the groups page. Then your group list is fetched (and cached), and the main template is displayed.

As this is the first time you're looking at the page, none of the detailed information about the groups (most importantly, the photos in them) are available. I could fetch this all first, then display the page having done so, but that could take a while. Instead, once loaded, the page calls a web service (on groupr itself) that proxies through to Flickr, caches the result, then returns the photo data. (The caching means that, the second time you load the page, it's included straight into the template.) Hopefully this all works slickly enough that nobody really notices the difference between the first (AJAX) load and the second (cached) one.

Unfortunately, if I did this for all the groups I'm in (65 or so), it'd still take far too long, hammer Flickr's picture servers, and possibly swamp the client under the weight of HTML, so only ten groups are shown, with paging through to the other chunks of ten groups.

groupr's still under development; I'm looking at ways of showing more than just the most recent photos in the group, and also at persisting data so that you can get a list of photos you've not seen (somewhat like Flickr's "photos from your contacts" page). It's certainly a fun little app to hack on and a nice enough environment in which to do so. Plus, I'm still waiting to see if I win a prize. Which would be nice.