No More Secrets

Using secret keys to identify applications communicating across the internet has become popular as people have copied the very successful Flickr authentication API. Unfortunately people trust that they can keep these keys secret from attackers, even as they distribute applications that contain the secret keys to other people. I decided to see how hard it would be to extract the secret key and API key from Flickr’s new XULRunner based Uploadr.

While the Uploadr is nominally open source, they don’t include the key in the source code. In the README they instruct anyone building it to acquire their own key and modify the source code to include that key:

You'll need your own API key and secret from Flickr to build Uploadr.
These can be obtained at http://flickr.com/services/api/.  The key
and secret must be placed in flKey.cpp as indicated.  This file is at:
  UPLOADR/MacUploadr.app/Contents/Resources/components/flKey.cpp

The API key is stored as a string.  The secret is stored as individual
characters so it is not easily readable from the binary.

Such noble goals.

Getting the API key was as simple as running tcpdump:

sudo tcpdump -i en1 -vvv -n -s 0 -w /tmp/DumpFile.dmp

running the Uploadr, and then looking through the dump file to find where the API key is passed. Look:

api_key=a5c4c07991a15c6b56b88005a76e7774

In theory getting the secret key is harder, but it turned out to be even easier. It’s stored in the source code for the flKey::Sign method:

	// The shared secret
#ifdef XP_MACOSX
	bytes[0] = '-';
	bytes[1] = '-';
	bytes[2] = '-';
	bytes[3] = '-';
	bytes[4] = '-';
	bytes[5] = '-';
	bytes[6] = '-';
	bytes[7] = '-';
	bytes[8] = '-';
	bytes[9] = '-';
	bytes[10] = '-';
	bytes[11] = '-';
	bytes[12] = '-';
	bytes[13] = '-';
	bytes[14] = '-';
	bytes[15] = '-';
#endif

All I need to do is find that code, disassemble it and reassemble the string. I ran the Uploadr under gdb:

gdb '/Applications/Flickr Uploadr.app/Contents/MacOS/xulrunner'

and once it was up and running tried to find something to break on that would expose the secret key. There weren’t symbols for the flKey::Sign method that’s defined in the source file but I found I could break on MD5Init that’s called from that function. So:

break MD5Init
kill
run

And now I stop in the middle of the signature function – and from the stack trace I can see where flKey::Sign begins:

Breakpoint 1, 0x149a086b in MD5Init ()
(gdb) where
#0  0x149a086b in MD5Init ()
#1  0x149a1588 in flKey::Sign ()
#2  0x017bb45d in NS_InvokeByIndex_P ()
...

All I have to do is disassemble flKey::Sign:

(gdb) disassemble 0x149a1588
Dump of assembler code for function _ZN5flKey4SignERK9nsAStringRS0_:
0x149a1498 <_ZN5flKey4SignERK9nsAStringRS0_+0>:	push   %ebp
0x149a1499 <_ZN5flKey4SignERK9nsAStringRS0_+1>:	mov    %esp,%ebp
0x149a149b <_ZN5flKey4SignERK9nsAStringRS0_+3>:	push   %edi
0x149a149c <_ZN5flKey4SignERK9nsAStringRS0_+4>:	push   %esi
...
0x149a153e <_ZN5flKey4SignERK9nsAStringRS0_+166>:	movb   $0xXX,(%eax)
0x149a1541 <_ZN5flKey4SignERK9nsAStringRS0_+169>:	movb   $0xXX,0x1(%eax)
0x149a1545 <_ZN5flKey4SignERK9nsAStringRS0_+173>:	movb   $0xXX,0x2(%eax)
0x149a1549 <_ZN5flKey4SignERK9nsAStringRS0_+177>:	movb   $0xXX,0x3(%eax)
0x149a154d <_ZN5flKey4SignERK9nsAStringRS0_+181>:	movb   $0xXX,0x4(%eax)
0x149a1551 <_ZN5flKey4SignERK9nsAStringRS0_+185>:	movb   $0xXX,0x5(%eax)
0x149a1555 <_ZN5flKey4SignERK9nsAStringRS0_+189>:	movb   $0xXX,0x6(%eax)
0x149a1559 <_ZN5flKey4SignERK9nsAStringRS0_+193>:	movb   $0xXX,0x7(%eax)
0x149a155d <_ZN5flKey4SignERK9nsAStringRS0_+197>:	movb   $0xXX,0x8(%eax)
0x149a1561 <_ZN5flKey4SignERK9nsAStringRS0_+201>:	movb   $0xXX,0x9(%eax)
0x149a1565 <_ZN5flKey4SignERK9nsAStringRS0_+205>:	movb   $0xXX,0xa(%eax)
0x149a1569 <_ZN5flKey4SignERK9nsAStringRS0_+209>:	movb   $0xXX,0xb(%eax)
0x149a156d <_ZN5flKey4SignERK9nsAStringRS0_+213>:	movb   $0xXX,0xc(%eax)
0x149a1571 <_ZN5flKey4SignERK9nsAStringRS0_+217>:	movb   $0xXX,0xd(%eax)
0x149a1575 <_ZN5flKey4SignERK9nsAStringRS0_+221>:	movb   $0xXX,0xe(%eax)
0x149a1579 <_ZN5flKey4SignERK9nsAStringRS0_+225>:	movb   $0xXX,0xf(%eax)
...
0x149a168e <_ZN5flKey4SignERK9nsAStringRS0_+502>:	pop    %esi
0x149a168f <_ZN5flKey4SignERK9nsAStringRS0_+503>:	pop    %edi
0x149a1690 <_ZN5flKey4SignERK9nsAStringRS0_+504>:	pop    %ebp
0x149a1691 <_ZN5flKey4SignERK9nsAStringRS0_+505>:	ret
End of assembler dump.

And sure enough if I reconstruct those bytes, the secret key is: REDACTEDREDACTED

That wasn’t too hard was it? The fact that the code around the key was open source made it easier to find but even if it was closed source if I knew what kind of algorithm I was looking for it wouldn’t be hard to find and examine. It’s pretty easy to trace forward from user input or backward from network IO to find the signature algorithm, and the secret’s always the input to that.

Unfortunately OAuth follows the same model as Flickr, placing some value in a consumer’s “private” key. They do mention in Appendix B.7 that it’s often possible for attackers to access a secret it’s still a vital part of the protocol.

Since it’s not possible to securely identify clients we should move to a model where users approve actions against their accounts rather than granting carte-blanche actions to clients. For example Flickr could put photos that are uploaded into a staging area and require approval on the web site before including them in a user’s photo stream. This would allow users to make a informed decisions about third-party access to their accounts.

Updated: at Flickr’s request I’ve removed the “secret” key.

Updated again: I made another post about this topic, expanding on the final paragraph of this post.

Posted in Default | Tagged , , | 8 Comments

Closed Source

So now that I’m working on something that’s proprietary, closed source and in stealth mode, I’m finally doing stuff and learning how to do things that are really cool! Typical. Perhaps I should just start queuing up blog posts about the stuff I’ve discovered to push out live once we launch something other people could look at.

Posted in Default | Tagged | 3 Comments

Flash Development with Flex Builder

Dear Lazyweb, I’ve started doing Flash and Flex development. For me the Flex Builder IDE is significantly better than the Flash CS4 IDE, but when you build a SWF in Flex Builder it includes all of the MX widgetry. That’s too heavyweight for building simple Flash applets. Is it possible to get around that so I can use Flex Builder for my non-visual Flash development without bringing in MX?

Update: David Zuckerman a Flex Builder Developer dropped in to provide this invaluable advice:

Hey Ian, have you tried an ActionScript project in Flex Builder? It sheds most of the Flex weight, but you’ll still need to modify your project settings. In your setttings, go to the “ActionScript Build Path” tab, and open up the Flex SDK that’s listed there. Remove everything but playerglobal.swc (you can keep utilities.swc if you want). That should give you just the language definition, and none of Flex

I haven’t moved the project over to this fully but it looks like it works perfectly. Thanks David!

Posted in Default | Tagged , , | 5 Comments

New challenges

So I’ve been at a new job for a couple of weeks now. I left Songbird around when we shipped 1.0 to seek some new challenges. I’ve been doing Mozilla browser development for seven and a half years and I’m sick of it. I still think building browsers is one of the most interesting fields to work in – since we all spend much of our time working and interacting through web sites small incremental improvements to web browsers can have a huge impact on a wide variety of activities. But building web browsers is hard.

Now I’m doing some web stuff. It’s a challenge but in different ways. I’m working a lot with Flash and AIR right now, I’ve just successfully evangelized jQuery to the other developers and about to head back into cross-browser html + css development. The project I’m working on is still under wraps but it’s really exciting and I’m working with some pretty amazing people. I can’t wait to be able to brag about it.

In the mean time I might start talking about more web-y things here.

Posted in Default | Tagged , , , , | Leave a comment

Free Technical Books, Online

(Inspired by James Tauber, I’m going to try to write a blog post every day for November. Some of them will be here but others will be over on my personal blog.)

When Oreilly originally launched their Safari Books Online service in 2001 I was really excited. I love technical books but they’re expensive to buy and heavy to carry around. It turned out that they were expensive to read online too. Full access to Safari costs almost $500/year and a limited ten-books-a-month plan costs over $250/year. I don’t spend nearly that much money on technical books as it is, and while I’d save some time over just Googling for information that’s available freely online I’m basically a cheapskate.

Today I noticed that San Francisco Public Library offers access to SBO. A bit of digging showed me that the Oakland Public Library (where my lovely wife works) also have a Safari subscription. With my library card number I can about a third of the books on the main site – but that seems to be a broad and deep collection. There’s no downloading and the lynda.com videos aren’t available. But the price is great.

Posted in Default | Tagged , | Leave a comment

Meanwhile, in the day job

A couple of months ago my role at Songbird shifted a little. Up till then I was working on the core product, fixing bugs and adding features across the whole product as part of the bird engineering team. Since we started working on 0.7 (aka Fugazi) I moved into a group initially called strategic development which then split and merged with the design and product group.

We’ve been looking through feedback from our users, primarily through surveys to determine what features we can add that will address most users’ feature requests. Doing this outside engineering has been great since they can focus on improving the core product, keeping a clear vision of what the product we want should be, while we’re working directly from end-user feedback.

My first project was a new Last.fm addon using our new playback history API. It was installed by default for all Songbird 0.7 users so it’s had quite a bit of use, some good feedback, some bug fixes and now some translations:

My next project was taking GeorgesSeeqpod addon, updating it and getting it ready for inclusion in Songbird. It wasn’t ready for Fugazi, but hopefully it’ll be ready for Genesis (our next release). Seeqpod is an MP3 search engine and our addon attempts to integrate it nicely into Songbird. I streamed a lot of random music while developing it. I spent a whole day listening to 80s Metallica.

Now I’m working on better music store integration using Songbird’s Web Page API.

Posted in Default | Tagged , , | Leave a comment

Tracking WordPress using Git

I publish this blog through WordPress, for reasons I’ve outlined before. I run it with a custom theme and a bunch of plugins though, and I wanted a convenient way to keep my WordPress install up to date without having to reinstall everything all the time. I wanted source control for my blog install.

My first attempt involved mirroring WordPress SVN into a Git repository on github so that I had a Git version of the SVN tree (including branches, tags and every checkin separate) and seperate repository holding the changes I’d made for my web site. This eventually failed for two reasons, first the script I was using to mirror the SVN into Git had a habit of failing in bizarre ways and secondly having two repositories confused me.

Yesterday I decided to update my fairly outdated WordPress install, it had been missing security fixes for some time and was one minor version behind. Since tracking SVN hadn’t worked I tried a simpler approach, a single Git repository containing a master branch that tracks releases and an ianloic.com branch to track the state of my site.

I set up master with a fresh download of WordPress 2.5 from wordpress.com, created my ianloic.com branch and applied the differences between my site and the 2.5 SVN tag (for all it’s failures my old approach at least let me do this easily). I switched back to the master branch, deleted all the files (leaving my .git directory intact) and unpacked the new WordPress 2.6.1 tarball. I checked that in (to the master branch), tagged it 2.6.1 and then merged that into my ianloic.com branch. I pushed all that to github and then checked it out on my web server (at Dreamhost).

Normally with Git you’re tracking just the master branch, but I want both master and ianloic.com branches to be tracked so my .git/config contains:

[remote "origin"]
	url = git@github.com:ianloic/wordpress.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master
[branch "ianloic.com"]
	remote = origin
	merge = refs/heads/ianloic.com

Now it’s easy to track changes that I’m making to my site and update to the latest WordPress without risking losing anything. The process for updating to a new WordPress release is:

  • on my laptop check out the master branch
  • rm all the files except for .git from the directory
  • unpack the new release into the directory
  • git-add . — now git-status will indicate what has changed, been added or removed
  • git-commit to check in the new version of wordpress
  • git-tag versionnum to tag which version is currently in master
  • git-checkout ianloic.com
  • git-merge versionnum to merge the latest version into the site’s branch
  • git-push –all –tags to push all the branches and tags to github
  • on my web server, git-pull to update to the latest release

I end up with a tree that looks like this:

ianloic.com WordPress in Git
Posted in Default | Tagged , , , | 3 Comments

OpenID Usability Non-solutions

At work we’re building our new centralized authentication solution. Allowing OpenID logins is not part of our first release, but it’ll follow at some point in the future, at least if Rob has any say in it. Even though I’ve had an OpenID identity for as long as anyone, use mine extensively and have even implemented my own provider, I’m not convinced it’s a good idea to only support OpenID logins.

The approach taken by Magnolia (who only support OpenID logins these days) and IDSelector (which is supposed to make OpenID usable) is allow users to log in with any of their existing accounts that offer OpenID (Yahoo, Livejournal, AOL, etc). The thinking behind this is that users don’t have to remember a new username and password this way. This thinking is backwards. Users already remember their usernames and passwords. Web browsers remember passwords and people use consistent usernames and password patterns across sites. Both software and humans have adapted to this problem. People haven’t adapted to remembering which account they used to sign into a site.

If I sign up for Magnolia using one of the accounts I have (of the 7 external account types they offer, I have 5) what happens in 2 weeks when my cookie expires and I need to log in again? Even though I might use the same password across all of those accounts there isn’t an easy way for me to remember which account I chose to use to log in. Fundamentally, this approach to OpenID doesn’t give users less things to remember, but more.

I think a better approach is for site to allow either local logins or OpenID identities. When offering OpenID logins it’s important that sites help educate users about the value of OpenID rather than hiding it.

Posted in Default | Tagged , | 3 Comments

Source Control for your Operating System

I think that I’ve realised why I feel so uncomfortable using MacOS X and Windows for “Real Work”. They’re fine for playing MP3s or browsing the web, but when it comes to developing software I get scared.

For me, developing my own software usually involves installing other software and upgrading existing software on my computer and when you don’t have a package manager that’s really hard to do reliably and reproducably. I’ve tried doing this by hand, keeping track of what versions of what I installed where with what build flags, trying to write scripts that can reproduce my build environment and trying to keep track of which dependencies are build-time and which are run-time. I’m bad at that, but that’s the kind of thing that computers are really good at.

Now, I know that Cygwin and MacPorts each have their own pseudo-package system, but that only covers the UNIX libraries and utilities that are ported, not native tools, and they don’t integrate with the tools you use when producing packages to ship. When you’re developing a Mac or Windows application how do you keep track of which libraries need to be included in your installer or disk image? There must be a trick, but I’ve never come across a tool for the job. The Debian packaging tools do an amazing job of analyzing the dependencies in software that has been built and mapping them to package dependencies. There are even tools for creating VMs that you can use to test that the autodetected dependencies are correct. All automatic, reproducable, testable.

So today, out of the blue, I found a metaphor that I’m satisfied with to explain to my Windows and Mac developer friends the way I feel about developing on their platforms. It’s like programming without source control. Developing on Linux and with source control I feel safe – if anything I do has unintended (or worse, unnoticed) side-effects I know that I can get things back to where they were before, or at least work out what happened. On Mac and Windows I’m back to when I was writing Pascal programs in DOS where I’d type a few lines, save and never be able to make my program work again.

Aside 1: At least with Windows I can do everything in VMWare which gives me snapshots and roll-back, Apple’s DRM prevents me doing the same for MacOS. Fail.

Aside 2: I feel almost the same about Ruby gems. It’s a package system but a terrible one. It’s like being forced to use RCS when I’ve got git sitting right here. I do understand why Mac Ruby developers seem to think it’s the best thing since sliced bread – I remember in 1995 when I was first introduced to RCS.

Posted in Default | Tagged , , , | 11 Comments

Twitter Translation

My friend Britt mentioned today that he was about to launch twitter.jp. How exciting! But I don’t understand Japanese. If only I could easily translate all those tweets in languages I don’t understand.

I played around with Google’s new AJAX Translation API before and I wondered how hard it would be to use that from a GreaseMonkey script. The answer: hard. I’m not sure what the exact problem was but every way I tried to include Google’s APIs into the pages I was manipulating, including creating my own iframe and using document.write failed. In the end I used a static proxy html file (hosted in one of my Amazon S3 buckets for cheap efficiency) with some sneaky cross-site communication (the request goes over in location.hash, the response comes back in window.name).

My script is now up on userscripts.org: twitlator.user.js

To use it simply go to a page listing a bunch of tweets, like the public timeline and find a tweet in a language you don’t under stand. For example:

Click the translate! link and several moments later you’ve got:

Yep. People aren’t posting anything interesting in Japanese either.

I’m pretty sure my script will only work on Firefox 3. I’m using getElementsByClassName which I think wasn’t introduced until Firefox 3. Why aren’t you running it already?

Posted in Default | Tagged , , , , , , | 2 Comments