Archive for the Category Tech

 
 

Programmatic Uploading With Vimeo – A Tragic Comedy

One of the apps we’re writing for our customer requires the ability to upload videos to Vimeo, so I got the chance to work with their web services APIs this weekend. There aren’t any publicly available Objective C classes for this, so I ended up rolling my own.

Unfortunately, this ended up being an exercise in the value of good documentation. Here’s what I found.

OAuth

Vimeo uses OAuth, and I find that Google’s OAuth classes work fine for this.

The Upload Process

The real problem was in step 3: “Post the Video Files.” This step didn’t work, and I spent about 20 hours altogether, with considerable help from my friend and business partner George, trying to figure out what was going on.

Here’s what the Vimeo Upload API site currently has to say:

Step 3 of the upload process

So, right off the bat, we have to do some special things around authentication. Okay, we’ll keep that in mind if anything goes wrong. This is a POST transaction, so the parameters probably need to go in the POST body.

So we try it, and get a response of “415 – media format not supported.”

Okay…

The problem here is that it’s a guessing game: we have been given the rules but haven’t actually been given any clear guidelines about where the parameters are supposed to be and what format the video is supposed to be in (I’m talking about upload format, not video codec format).

Eventually we started using a multipart MIME format for the document, which got us past the 415 response code. The entirety of the new error message was:

Upload failed. try again!

No error code, no explanation. Nothing.

Troubleshooting

The biggest red flag, to my mind, was the signature. The API documentation had gone out of its way to emphasize the importance of getting the signature parameters right, so this is where we looked first.

After several hours we learned:
* A hell of a lot about OAuth signature generation
* A hell of a lot about Google’s OAuth code
* That GData OAuth only uses URL parameters for generating a signature. It does not look at the body at all.

Enter Wireshark

Eventually I realized that there was too much ambiguity about where the service was expecting to find the parameters, so I downloaded Vimeo’s desktop uploader (an Adobe AIR app), and started looking at call traces with Wireshark. This was a lifesaver:

  1. The Vimeo app doesn’t sign its own uploads
  2. There was a missing parameter in the API documentation (“filename”).

Vimeo Wireshark Output

Once I added the filename attribute the upload worked on the first try. I was able to verify that I could upload chunks with or without authenticating the upload chunk request before sending it.

The winning combination

So, what worked for us was to do a POST, as the document indicates, with the ticket_id and chunk_id as URL parameters, and with the FileName and file_data parameters as multipart form data, followed by the actual chunk data, with a content type of video/quicktime.

Rant

At first I thought of the Vimeo documentation as a few tantalizing clues to help you solve the uploading puzzle, but now I consider it to be openly malicious documentation. You would actually be better off without it because they withhold information critical to your success while simultaneously throwing a gigantic red herring at you.

I don’t know why Vimeo states that they require this step to be authenticated, nor why they have stated so explicitly how the signature has to be constructed for this particular call. Perhaps they had to relax the requirement internally and didn’t update their documentation. What I do know is that it steered me in a completely different direction than I needed to go in.

Likewise, the response messages from the upload server are almost criminally opaque. If a parameter is missing, it would be helpful to indicate that in the error message. Most of the other error and status messages Vimeo provides are detailed and articulate.

Finally, the omission of the Filename parameter itself from the API documentation, which is mandatory for the upload to succeed, sucked. In retrospect, this is a standard part of the Content-Disposition field and probably would have been included anyway if we’d started out on this with directions appropriate for doing a low-level implementation of the protocol.

Hindsight

It is likely that the directions, as opposed, are spot on for someone looking to integrate with Vimeo from a website. The php and javascript uploaders that I have seen implement their code pretty much exactly as stated. I’m sure that there is a lot happening under the hood that we have to deal with explicitly ourselves.

In any case, if you are looking to do Vimeo uploading in native code, the bottom line is: use multipart-mime with a content disposition block for the actual binary data. Authentication for this particular call appears to be disabled, but could come back at any time. For now, if you’re having problems with uploading, it’s probably due to your message format and not your signature.

Details

HR Awesomeness!

The company I work for, Black Pixel, has been looking for a few new iOS developers, and the responses I’ve been getting to our ads have pretty much been uniformly subpar.

Finally, after reading yet another resume of someone claiming to have deep iOS development experience, I tweeted the following:

Protip: Comments like

”* Very familiar with X-Code”

in your resume automatically result in deletion.

The problem is, that the product in question is called “Xcode.”

I got a lot of comments from people, some of whom wanted to know if “XCode” was an acceptable mistake or not. Unfortunately, I couldn’t fit a reasonable, informative response in 140 characters this time.

No it is not. Hell, no, it is not.

Xcode is the main tool you use to make your apps. It is the fundamental place in which you do your work. If you’ve truly worked with it enough to claim solid familiarity with the tool, you will know how Apple spells its name.

Messing up the camel casing is definitely more forgivable, but not if you want to work for us. We push ourselves mercilessly to make the best software possible. If you are representing yourself as having solid iOS industry experience and being a good fit for our team, you wouldn’t mess this up.

Reality Check

None of this is a reliable indicator of good programming expertise. Knowing how to spell Xcode, iPhone, or knowing that iPod Touches are not referred to as ‘iTouches” doesn’t mean you are a great programmer.

But NOT knowing these things telegraphs a profound lack of attention to detail about things that are fundamental to how we work. Nothing screams “I will be a massive liability for your company” like an inability get these things right on something as important to their career as a resume.

Obviously the overall strength of the person’s resume and their previous successes will temper this response: if I saw “XCode” from an Apple employee that would clearly know better, I’d assume it was simply a shift key accident. On the other hand, if it was on their resume, well, I’d be shocked.

Efficient transfer of huge files between machines

I need to transfer 250gb of data between two machines over the lan.

Ladies and gentlemen, the tar trick. I only need this every two to five years, but when I need it, I really need it.

This tars up the contents of a directory and streams it as compressed data to a remote host. This is super efficient as:

  1. The data is compressed before it is transferred, so overall throughput is greatly reduced.

  2. Compression and decompression are distributed and run in parallel. Instead of zipping everything up, transferring it, and then unzipping it, both processes run simultaneously, the the compress/decompress time is cut in half.

How to tar something to a remote host.

tar cvf – source_dir | gzip | ( ssh target_host “cd target_dir; gunzip -d | tar xvf – ” )

How to tar something from a remote host.

ssh target_host “tar cvf – source_dir | gzip” | gunzip -d | tar xvf -

UPDATE:

Several people have mentioned more updated refinements, I haven’t tried all of these but I agree that they sound like smart approaches:

Jerry Chen: you can use ssh’s -C compression flag which accomplishes the same thing as gzip, i believe

Nate True: I like to use Netcat instead of SSH to transfer the data – cuts out the overhead of encryption but takes more effort to set up.

Zachery Bir: can’t you just use cvfz/xvfz and bypass piping to gzip/gunzip altogether?

RIM BlackPad: Checkmate II

RIM has announced the BlackPad: their acknowledging nod to the iPad’s success. Welcome to Checkmate 2.0: the only thing I need to do here is replace ‘iPhone’ with ‘iPad.’

End game

Because of that effort, since the iPhone was released, everyone else has been struggling to play catch up, and no one has really come close. Apple raised the bar higher than anyone else had before, and by the time the competition realized how much of an effort would be required to seriously compete, the public had already turned to them to see how they would meet Apple’s threat.

Being in the public eye, and forced to react, no competitor is really going to be able to put in the years of time required to offer Apple any serious competition. Everyone is reacting and trying to get something out the door as quickly as possible. I think that this is very well illustrated by the BlackBerry Storm, probably one of the most notorious handsets ever released by RIM. Perhaps it was the need for a fast response, or perhaps it was arrogance on the part of RIM, but there was no way that the Storm could ever be considered serious competition for the iPhone.

Can’t wait to hear the reviews.

Checkmate

Checkmate.jpg

When I read articles like last year’s Apple iPhone Doomed To Failure — Windows Mobile 7 Plans For 2009 Leaked, which predicted that competitors would rush in to dominate the iPhone’s market share, I find myself reflecting on the history of the iPhone with the awe that an acolyte has in the presence of a grand master. Because the competition was beaten before they even understood what was happening to them.

When Apple introduced the iPhone, they won the game in a single move – the perfect checkmate. Although the iPhone was arguably not perfect, by the time it was announced, Apple had already maneuvered its pieces in such a way that, by the time the other players saw the move, it was already too late for them to do anything about it.

I’m not referring to iPhone’s dominion over Windows Mobile 7 – Ballmer and his cronies will shoot themselves in the foot enough to ensure that it’s never any real threat, as history has already shown – I’m talking about its dominion over the entire mobile phone market.

Plans within plans, wheels within wheels

Apple spent years getting the design for the iPhone nailed, and they considered everything that was wrong with conventional mobile phones, and worked the problem until they had a solution that addressed those issues. They clearly understood how large of an effort the iPhone would require, and made a commitment to that effort years in advance of its release. I am certain that the amount of money, thought, and effort that Apple invested in the platform significantly outstripped anything their competitors had done previously, and perhaps since.

The extent of Apple’s efforts, and the remarkable patience and secrecy they practiced while getting things into place, paid off incredibly well for them.

End game

Because of that effort, since the iPhone was released, everyone else has been struggling to play catch up, and no one has really come close. Apple raised the bar higher than anyone else had before, and by the time the competition realized how much of an effort would be required to seriously compete, the public had already turned to them to see how they would meet Apple’s threat.

Being in the public eye, and forced to react, no competitor is really going to be able to put in the years of time required to offer Apple any serious competition. Everyone is reacting and trying to get something out the door as quickly as possible. I think that this is very well illustrated by the BlackBerry Storm, probably one of the most notorious handsets ever released by RIM. Perhaps it was the need for a fast response, or perhaps it was arrogance on the part of RIM, but there was no way that the Storm could ever be considered serious competition for the iPhone.

The Palm Pre: One notable, noble effort

One camp that has earned my admiration is Palm, the only company that I appears to have been desperate enough to try and put in the time required to produce any real competition. While I don’t think that the Pre is serious competition for the iPhone, it is definitely a commendable and respectable effort.

What next?

Despite Mitchell Ashley’s claims to the contrary, no one has seriously found a way to compete with Apple’s skyrocketing market share, except perhaps through attempting to offer more liberal application sales policies in an effort to woo more third party developers.

I have long maintained the RIM and Microsoft have always shot to deliver a product *just* good enough that people will pay for it and not demand their money back. I’m wondering at this point if they’re ever going to be willing to duplicate the effort Apple put into the original iPhone and try and beat them where Apple is strongest, or if they’re just going to keep phoning it in (pardon the pun).

At present it looks like they’re sticking with phoning it in – just pumping out crap and subsisting on the money of people too cheap or indiscriminate to know the difference, or enterprises that have already committed to infrastructure changes that lock customers into their brand.

If the meteoric rise of Apple’s market share is any indicator, that market ain’t working, boys.