Wil Shipley writes about the compromised perfection we must strive for in order to provide users an experience that meets their human expectations:

“Classic computer programming has largely failed, because it failed to copy nature. Nothing in nature works 100% of the time, but it sure works well MOST of the time – and when it fails, well, you die and get replaced. A human being, for instance, is an absolutely amazing machine, and is provably NOT provably correct.”

I particularly like the example in the second half, having to do with smartly interpreting a typed ISBN numbers for product search. How do you strip the meaningless dashes from a search term, except when they’re utterly meaningful? This kind of thinking is important to fine-tuning an application. Nobody will appreciate the hours you spent laboring over the question, but for some reason they’ll just think your product is particularly awesome.

It’s interesting that the pursuit of perfection in an application has to involve the pursuit of compromise. By solving a problem in a way that degrades gracefully to unsolvable, you offer a happy, possibly even surprise solution to many people who would not otherwise expect one.

The latest ridiculous App Store power-play to make it into the public limelight is Apple’s alleged censoring of Ninjawords, an iPhone interface to a community-edited dictionary called Wiktionary. Before being approved, even as a 17+ rated title, the app’s developers were asked to remove specific words from the dictionary’s index.

(Edit 8/6/2009: Since I wrote this article, John Gruber received a response from Apple’s Phil Schiller. He paints a slightly different picture of the alleged censoring, and defends Apple’s intentions as being noble. I still maintain my theory below helps explains the capriciousness of AppStore review policies.)

John Gruber excoriates Apple for censoring a reference book. Gruber also discovered through an interview with a Ninjawords developer that Apple must have gone out of its way to locate words they could find fault with. Apparently the developers had been careful to prevent casual users from stumbling upon an offensive word, by preventing auto-completion for common vulgarities:

“In other words, the App Store reviewer(s) explicitly searched for curse words they already knew, and found them.”

I’ve been thinking about the capriciousness of the App Store review process. It’s ridiculous the kinds of rejections and hoop-jumping we’ve observed in the past year, and one has to assume that the issues making their way into the public eye are only the tip of the iceberg.

Then I remembered something from my own experience that might shed light on the situation. I started as a Quality Assurance tester back in 1995, in a small engineering group. Our group was diligent in the pursuit of finding issues that would embarrass the company or hurt customers. But we worked with larger groups whose motives seemed more oriented to the systematic evaluation they were receiving from their bosses.

These testers didn’t care how good their bug reports were. It didn’t matter if the software gaffe they discovered would save the company a million dollars, or a metric shit-ton of public grief. All that mattered was that the bug was “valid” and that the reporter was “first.”

I learned about the subtleties of this system through the ways that those testers interacted with me. Sometimes a bug that I submitted was determined to be a duplicate of an earlier report one of these testers submitted. If mine had more detailed information, it might be marked as the “original” bug, while the less informative bug was designated a duplicate. This worked great for those of us trying to ship a great product, but not so good for people who were fighting for their reputations in the metric-oriented testing groups.

Because our group was committed to shipping a great product, we were always convinced that bug reports with more information were superior. But the testers who were under the gun to produce new, unique issues, wanted credit for having uncovered these issues first.

As you can imagine, the “thirst for first” led to a significant number of ridiculous bug reports. If a tester could reasonably defend a bug report as valid, then it counted in their statistics, and made them look like a useful member of their team. My impression was that promotions and raises were directly linked to these statistics.

Many of the mercenary testers I encountered were motivated to scrape the system for bugs, as ridiculous as they may be. They logged them into the bug system and then defended them at all costs, as if their lives depended on it. And it turned out, they did. At least, their paychecks did.

I would not be surprised to learn that App Store reviewers are working under a similar structure. A system that rewards “unique, valid rejections” would certainly explain the behavior we have seen coming to light in the past year.

Why would somebody waste time typing profane words into a dictionary, gathering screen captures, and sending them to developers, except to defend their prize “catch”? If perfecting the product was the goal, we’d see a lot more nuance and thoughtfulness. But excellence is one goal, and collecting proof of “doing one’s job” is quite another. I think I know what many App Store reviewers aspire to.

Afterhought: It occurred to me shortly after publishing the above that App Store reviewers can’t be working purely under a “catch all violations” directive, because if they were, there would be numerous rejections based on UI guideline violations, and we’re not seeing as many of those (or are we?). I’m sticking to my thesis, but I suspect that the number of rejections we’re seeing on contrived issues like “you can find ‘cock’ in the dictionary” is because these are the easiest for reviewers to defend with Apple’s published guidelines. Whether a text field is aligned properly is a lot harder to challenge than whether “cock” can be interpreted as profanity.

Brent Simmons writes on the Anatomy Of A Feature, using his recent work in NetNewsWire to add support for the popular Instapaper service:

It’s tempting to think that adding a feature like this is just about adding the functionality — but there’s a bunch more to it than that.

Here you see the gory, deliberate details that a responsible developer must consider when adding even what seems like an incredibly “easy feature.”

Without a doubt, the part of my job that slows me down the most is exactly this kind of consideration. I want to add a new feature, but not until I have evaluated how the feature best fits in the structure of the application as it exists today.

Sometimes even an “easy feature” requires massive overhaul of the application, so that it doesn’t feel tacked-on or cheaply done. If the feature is important enough, it’s worth practically rewriting your application, just to add something that looks like it must have taken all of 10 minutes to do.

Crashes suck. When an application experiences a crashing bug, it’s likely to stop running and take with it any unsaved work which you may have had open. Generally speaking, developers who take pride in their code also take pride in ensuring that it is resistant to crashing.

Unfortunately, it can be difficult for developers to know just how crash-prone an application is, because many users are not proactive about sending in bug reports when an application crashes. Who can blame them? They just had to suffer the indignity of a product gone completely wild, and now you want to be generous with their time and help you out?

Apple recognizes this, which is why they make it incredibly easy for users to report crashes on the Mac. When an application goes down in flames, a dialog appears offering to send the details of the crash directly to the company. This is great for Apple, but relatively useless for developers. You see, although Apple is happy to collect information about every crashing application on your Mac, they don’t share the information with the developers of the product in question.

Yeah, it sucks.

Rolling Our Own

It’s obvious that Apple should be helping developers by sharing this information. It’s good for us and it’s good for the platform. It’s good for Apple. A stable environment is so important to perceived quality, that it took less than a year of supporting 3rd party software on the iPhone before they decided to share crash information with iPhone application developers.

Who knows, we may get lucky. Apple’s generosity with this information may take a turn for the better at any moment. But for the time being if you want to have any sense of the stability of your application in the wild, you’re going to have to arrange an easy way for users to send the crash information directly to you.

I recently decided to bite this bullet, and I discovered that while there is some really useful open source code out there to help achieve this goal, the pluses and minuses of each package are not well documented. I hope this roundup of several options helps direct your search for the most appropriate solution in your product.

UKCrashReporter is a popular option that is proudly low-tech in its operation. It relies on finding crash logs in the user’s home directory. If they’re there, UKCrashReporter asks the user if it may transmit the crash information to your web server via an HTTP POST request.

SFBCrashReporter and CMCrashReporter are each quite similar to UKCrashReporter, detecting crash logs at application launch time, and offering to submit them via HTTP POST to a URL of your choosing.

FeedbackReporter also relies on launch detection of crash logs, but offers to report uncaught Cocoa exceptions in your application right as they occur. While other solutions listed here are focused tightly on crashes, FeedbackReporter also provides a general-purpose feedback interface, which users could invoke from a menu item in your application for general, non-crashing feedback.

HDCrashReporter adopts the same approach as UKCrashReporter, but opts to deliver the crash information by email instead of HTTP request. It uses Apple’s NSMailDelivery class, or if that fails, attempts to open a mailto: URL, engaging the user with their favorite email client.

ILCrashReporter and its branch ILCrashReporter-NG tackle a shortcoming of all the choices listed so far: that a user will not be presented the option of submitting the crash log until and unless they launch the application again. To be sure that users immediately have the opportunity, ILCrashReporter starts a second watchdog process, that sits around waiting for your application to terminate unexpectedly. If it does, the crash information is gathered up and sent, like HDCrashReporter, by email. Unlike HDCrashReporter, the email is delivered by connecting directly to an SMTP server either deduced from the user’s settings, or else explicitly provided by you.

plcrashreporter is one of the newest additions to the bunch, and takes a decidely different approach in that it ignores Apple’s own crash logs, opting instead to scrutinize the state of memory and write its own, custom-formatted crash log to disk. It installs custom signal and exception handlers, and includes code that has an in-depth understanding of how to examine memory on different platforms including iPhone, Mac i386, and Mac PPC. Although the crashreporter is active at the moment of the crash, it requires an approach such as that of UKCrashReporter or HDCrashReporter, of detecting and then submitting the crash informaton on the next launch of the application.

Breakpad is the crash reporting facility used by Mozilla, and is unique among solutions here in that it is cross-platform, supporting Mac OS X, Windows, and Linux. Crashes and exceptions are detected as they occur, spawning a separate crash-reporter application that offers to submit crash information via an HTTP POST. Also unlike other solutions, the project includes server-side software for collecting crash information and for browsing it via a web interface.

Unsanity’s Smart Crash Reports takes a more vigilante approach, installing software that gets loaded into every application on the system. By hijacking the standard crash report dialog, SCR is able to repurpose it as a developer-friendly dialog that offers to send the report to both Apple and the application’s developer. Developers can choose to have crash information sent by email or by an HTTP POST to your server. Of all the solutions listed here it is probably the most fragile because of the way it hijacks Apple’s UI. It also requires installing software that has an impact outside of your application, which may be more than many developers are willing to ask of their users.

Great! So Which One Is Best?

If you want crash reporting in your app you’ll need to decide from among the above solutions, or starting your own solution from scratch. The problem of course, is none of the solutions listed is the obvious best choice. It depends on your own particular priorities and how you answer many of the questions at hand, including:

• How invasive/fragile should the solution be?
• Report crashes immediately, or after the next application launch?
• What platforms and architectures must be supported?
• What kind of user interface should be presented to users?
• Should reports be delivered by email, HTTP, or some other means?

Hopefully the capsule reviews above give you some direction in evaluating the options that are available, and making an appropriate choice. Of course, after evaluating and discarding the options here, I’m sure some of you will decide to hunker down and add another project to the list. Please let me know if you do.

The Best Solution Of All

The list of solutions above was gathered in one afternoon (with the help of my Twitter followers!), and I’m sure it’s not exhaustive. What the insane variety of choices tells me first and foremost is that Apple could end a lot of suffering if they just shared their dang crash reports. Seriously, look that this list. The number of choices and variety of implementation details is out of control. And these are only some open source solutions. I’ve talked to a number of colleagues who have implemented similar plans from scratch for their own purposes.

Apple: stop the madness! Free the crash reports and spare us agony of reinventing this wheel again, and again, and again. Good for us, good for the platform, good for you. Thanks.