I was thrilled to see that Macworld today posted a review of MarsEdit 2.0. The application earned 4 mice, which for those of you who follow the magazine know, is a pretty great accomplishment. I’m betting that one day I’ll earn all five, but for the time being I’m quite pleased with this achievement.

Lisa Schmeiser, who reviewed the application, did a good job of identifying many of the improvements I made for the 2.0 release, such as built-in Flickr support, and enhanced markup macros. But she also tuned into some of the “soft features” that are not as suited to bullet lists, but nonetheless make a huge impact on users over time. One of these is the “peace of mind” that MarsEdit can provide:

MarsEdit 2.0.4’s big appeal is peace of mind: You can compose posts locally and archive your work. Anyone who’s ever lost a Weblog post owing to a browser freezing or site going down will appreciate this tool.

Ask any Mac software developer you can find, and they’ll attest that getting reviewed by Macworld is one of the things we hope and strive for. It’s a great honor just to get mentioned, but to receive such a positive and thoughtful review really makes my day.

When Leopard was released over a month ago, I proudly announced that Red Sweater’s applications were Leopard ready. Well, it was true, but as I suspect is the case for most software developers, little wrinkles in behavior have slowly shown up as more and more users upgrade to the latest cat.

Today I released two updates to applications that get a bit less attention, but which nonetheless have passionate fans: FastScripts and Clarion. It was thanks to these fans that I discovered and was able to fix a few Leopard-specific issues that they noticed.

FastScripts had a problem with running automator routines. Like, at all. I guess that shows how often I run automator routines myself! From a developer perspective, running Automator Workflows in Leopard is a lot easier than it was in Tiger, and is officially supported by a new AMWorkflow class for Cocoa programmers. Unfortunately, while providing a new and sanctioned method for running these workflows, they simultaneously broke the old technique that I was using for Tiger. Fixed in FastScripts 2.3.6!

Another issue with FastScripts had to do with the order of presentation for the user’s “personal scripts” compared with Apple’s standard scripts (from /Library/Scripts). It’s only natural that users would prefer to see their own scripts appear higher in the menu, since those are probably the ones they’re accessing more frequently. Well, it turns out the placement of these items has just sort of magically fallen the “desired” way for the past 3 major operating system releases. In Leopard, the order of the randomness flip-flopped, and I needed to explicitly ensure that the user’s scripts take priority. Also fixed in FastScripts 2.3.6!

Clarion is an application for studying and drilling yourself on musical interval recognition. It’s a pretty niche market, but it has a pretty nifty live piano keyboard view that I’m pretty proud of. While you’re contemplating the sounds of the pitches you are being quizzed on, you can tinker around on the piano keyboard to give yourself visual and audio clues as to what the interval might be. Unfortunately, some changes in CoreAudio on Leopard revealed a bug in my code for playing these notes. The end result? On Leopard any key you pressed would keep playing the sound even after you release the key. Nightmare! Fixed in Clarion 2.0.1.

If you’re running Leopard and noticing any glitches in my products, please let me know so I can get to the bottom of them. Right now I have only one outstanding issue, which is a text-rendering problem in MarsEdit, when opening existing posts for editing. Unfortunately I can’t find anything wrong with my behavior, so I’m also sort of quietly hoping for a fix from Apple in 10.5.2.

Michael Mistretta over at MacApper has posted a review of MarsEdit 2.

Before I heard about MarsEdit, I would have all my blogs in Safari’s Bookmarks Bar, and I would have to click each one and log in, in order to see if there were any new posts written, or if I wanted to edit something. Now, I am just a click away from MarsEdit, where I can check and edit all of my blogs at once.

Thanks for taking the time to share your extensive thoughts, Michael! I think that paragraph sums up the “life changing” nature of MarsEdit for bloggers.

Developers who write code inevitably write code that crashes. Luckily for our users, 99% of those crashes happen on our own development machines, and we get the opportunity to fix them before shipping them off to our unsuspecting computers.

But every once in a while even the most polished and refined software exhibits that nastiest of malfunctions: the crashing bug. In the middle of a possibly important user task, the application comes screeching to a halt and disappears, leaving the user with nothing more than a cryptic message from the operating system, along with an offer to send the crash log off to Apple.

Needless to say, crashes suck. A lot. For users, especially. If there’s one behavior of your application that you should focus on eliminating, it’s the behavior of crashing. Above all other “nuisances,” this is the one that is absolutely unacceptable.

Which is why crash logs are so important. Occasionally, when a user is sophisticated enough to understand the value of the crash log and, instead of choosing to send it to Apple, copies and pastes it into an email to you, you should be overjoyed! Finally! A crash has been captured in the wild and delivered to you for easy inspection. This artifact not only proves that a crashing bug exists, but usually includes highly detailed information about where and how the crash has happened.

Which is why it boggles the mind to think that some developers see these crash logs as relatively useless. In my own experience of reporting bugs to other developers, I have found frequently that my crash log is considered more as a statistic than as a specimen. While I expect a response like “great, I see what the bug is, and I’ll get this fixed,” I often get responses more like “sigh, I’ve seen this a few times but have no idea what’s going on!”

Sometimes I find that the developer in question simply needs to be educated about the tools at his or her disposal. I imagine the hours wasted speculating about what the crash log means, and can’t help but think if they knew how to analyze it, they could have the bug solved (in many cases) in moments. Oh – and I’m sounding high and mighty right now, so before I get too carried away, let me just say that I too need to be educated on a great many things, but making sense of crash logs is not particularly high on that list. I leave it to other bloggers to fill in the gaps of my education.

Crash Logs: The Crappy Way

So what is this big thing that developers are missing? Well, a crash log essentially shows the “path” that the little hamsters in your computer were running on when they fell off the cliff. It paints a picture of where the program got misguided and went off doing something it never should have done. Problem is, the picture is a little blurry. Below is an actual crash log submitted by an actual MarsEdit user (abbreviated and condensed to fit the blog):

Thread 0 Crashed:
Foundation 0x93cc2a2e _NSAPAddPage + 94
Foundation 0x93cc3b27 __NSAutoreleaseObject + 263
Foundation 0x93cc3962 -[NSObject(NSObject) autorelease] + 18
Foundation 0x93cceaca +[NSString stringWithFormat:] + 106
marsedit   0x000637fb 0x1000 + 403451
marsedit   0x00053883 0x1000 + 338051
marsedit   0x000539ea 0x1000 + 338410
marsedit   0x00053d7b 0x1000 + 339323
marsedit   0x00053e45 0x1000 + 339525
marsedit   0x0005238b 0x1000 + 332683

For any non-programmers who are still reading (bravi!), you may have a little trouble understanding what the heck this means. This is an ordered description of the path the computer followed while trying to do something. Problem is it ended up crashing. The path is shown in reverse-order, so the crash is at the top of the list, inside Foundation (an Apple system library). Trying to make human sense of this goes something like this:

1. Something in MarsEdit at address 0x0005238b…
2. … called something in MarsEdit at address 0x00053e45
3. … called something in MarsEdit at address 0x00053d7b
4. … called something in MarsEdit at address 0x000539ea
5. … called something in MarsEdit at address 0x00053883
6. … called something in MarsEdit at address 0x000637fb
7. … which called a standard string library method “stringWithFormat”
8. which eventually crashed in the system.

You can sort of see how some developers are forced to throw up their arms in despair, right? But what if you could turn this muddled picture into something a bit sharper? The fact is, you can. The reason the picture is so blurry is because MarsEdit shipped without debugging symbols. The reasons for this are mostly to keep the shipping application as small as possible, but there are also some good reasons to hide the symbols if they reveal too much about your business strategy.

Crash Logs: The Happy Way

Many developers seem to think that gathering useful crash logs requires shipping a symbol-laden application, but aren’t willing to do so. The fact is, you can have the best of both worlds by shipping a symbol-stripped version of your application, but keeping a symbol-laden version on-hand for the developer’s convenience.

What am I getting at? Apple offers a tool called “atos”, which makes it relatively easy to map an address from a symbol-stripped application to a name in a symbol-laden version of that same application. As long as the two applications were built with the same sources and using the same build options, the addresses are bound to match up. It’s just the symbols that are “stripped” from the shipping application.

Let’s take a look at what happens when I apply my atos powers to the crash log I pasted above. From the Terminal, I invoke atos with the literal memory addresses of all the mysterious points in the “path” from the crash log. You can list the addresses in whatever order you want, but I’m listing them so that the result will be in order from top to bottom, the same way I reasoned out the crash in plain “crappy” English above:

iBook> cd MarsEdit.app/Contents/MacOS
iBook> atos -arch i386 -o MarsEdit \ 
	0x0005238b 0x00053e45 0x00053d7b 0x000539ea 0x00053883 0x000637fb

Note also that I specified the architecture explicitly. This is because I am debugging this on a PowerPC Mac, but the crash log I received showed that the customer is running on an Intel machine. This yields a magical response with, behold, real meaningful symbols!

-[RSXMLRPCCall invoke:] (in MarsEdit) (RSXMLRPCCall.m:397)
-[RSXMLRPCRequest requestText] (in MarsEdit) (RSXMLRPCRequest.m:239)
-[RSXMLRPCRequest serializeParams:] (in MarsEdit) (RSXMLRPCRequest.m:216)
-[RSXMLRPCRequest serializeDictionary:toString:] (in MarsEdit) (RSXMLRPCRequest.m:153)
-[RSXMLRPCRequest serializeData:toString:] (in MarsEdit) (RSXMLRPCRequest.m:123)
-[NSData(extras) base64StringWithLineLength:] (in MarsEdit) (NSData+extras.m:215)

Now I can apply my human sense again, but with the benefit of symbolic names instead of raw memory addresses:

1. MarsEdit invoked an RPC network request
2. … which while building the actual request cargo text
3. … was trying to serialize the parameters
4. … when it tried to serialize a dictionary object
5. … whose data
6. … upon being converted to a Base64 representation
7. … caused a “stringWithFormat:” call that went blammo!

Not only does the plot to this story make a lot more sense now, but I’ve even got handy pointers to the line numbers in the source code. Let’s see, NSData+extras.m, around line 215:

NSString *charString = [NSString stringWithFormat:@"%c", base64EncodingTable [outbuf [i]]];

Survey says? That’s the line I’m going to zero in on now as I try to figure out why this crash is occurring. Sure, I’ll have to now speculate as to why, and under what circumstances the observed crash might occur. So the magic of “atos” doesn’t take all the work out of debugging a crash log, but it brings most crashes out of the “frustratingly impossible” into the realm of “all in a day’s work.” I can use my knowledge of the situation to ask intelligent questions of the user, and hopefully get to the bottom of it, so that no other user ever has to see the nasty crash again.

I can do more than just tell the user “thanks for the report, I have no clue what’s happening here. Sorry.”

Doing It For Yourself

I hope I’ve intrigued you by the power of what I was able to achieve with “atos”, but of course to do the same with your own project, you’ll need to make sure you can come up with a “symbol-laden version” of your release app, so you can apply the same magic. The best way I know to do this is to simply add another build configuration in xcode, that is just like your “Release” configuration, but doesn’t strip out debugging symbols. You can read more about this approach and others in Apple’s fine documentation.

If you find yourself dealing with crash logs on a regular basis, it might be a good idea to make building a “symbol-laden” version of your app part of the standard build and distribute process. For myself, I’ve found it happens rarely enough that I’m happy with a process whereby I re-checkout a tagged version of the sources from Subversion, and then rebuild a copy using the “Release With Symbols” build configuration.

Nota Bene: If you’re on Leopard and trying to replicate a symbol-laden version of an app you shipped with Tiger, you’ll probably have to tweak Xcode to use the old linker, in order to get a build whose memory addresses match up with your old Tiger-built application. Might be easier to just go back to Tiger and rebuild it there.

I know this was probably common knowledge for some of you, but I also know from experience of chatting with my colleagues that this is completely new, forehead-slapping “why didn’t I know about that?!” type material for many others. I hope it saves you hours of work. Possibly even enough hours to continue reading long articles such as this one!