From time to time, I get dragged into the Mac OS X Console application. Either it’s because I’m printing debugging output via NSLog and Xcode has suddenly stopped showing it in the “Run” window, or I’ve just started seeing some wiggy stuff happen with my computer and I need to check under the hood.

What starts out as an innocent enough expedition often ends up with me scratching my head about something else I see in the Console log. This is the festering trash heap where every programmer’s worst “it couldn’t really happen” nightmare output goes. Well, there’s also a lot of annoying “forgot to remove the NSLog” type innocuous output there, too. But at any rate, I usually feel a little ashamed that I wasn’t aware earlier that such and such app has been unable to open a window for the past 3 days. I wish I could keep up with this stuff, but I don’t want to be one of those nerds who leaves the Console window open all the time just waiting for junk to happen. I want to live!

So I had an idea. I’ll track this junk the way I track all the other junk. With NetNewsWire! What is syndication for? It’s for packaging data in a format conducive to my computer tracking changes in junk over time. I want to “subscribe” to my Console log.

What I’ve come up with is sort of an embarrassing hack, but it does kind of work:

But wait a minute. If it’s an embarrassing hack, then shouldn’t I keep my mouth shut? Why would I want to go spreading my half-assed solutions across the Internet? Isn’t that bad marketing? Yes, but I have ulterior motives:

1. I sort of want to show it off, anyway.
2. I need your help.

Yes, you! I need your geeky, unixy, sysloggy help. I’ve got this fun little hack, which I’m now prepared to let you download, because you’ve read along so patiently.

What are you downloading? It’s a small shell tool whose only purpose is to translate your Console log into RSS format. You get no options. You just run it, and hope that you like what you see. If you’re using NetNewsWire, you can “subscribe” directly to the executable. So just put the file somewhere on your disk, and then select “New Special Subscription…” from the File menu. You have to tell NetNewsWire that it’s a “Script,” and then you have to tell it that it’s a “Shell Script,” even though it’s a binary executable.

Now update and see your last 30 console “chunks” directly from NetNewsWire.

What’s a chunk? Aha! Yes, this is where you and the “help” thing come into play. One of the problems with syndicating something large and unwieldy is figuring out how to glom together related items. It would be overwhelming to the point of uselessness if every single line in the Console ended up as a separate feed item. So I have to try to be clever. What I’m doing right now roughly follows this logic: starting at the end of the file, move up line by line, collecting lines into a “chunk” until a line “looks like” it belongs to a particular application. Once a particular application is identified, keep moving up the file, until a line that doesn’t identify an application, or that identifies a different application, appears.

Sounds complicated, huh? Yeah, well that was the easy part. That works great when output looks like this:

2006-01-03 12:23:16.462 Xcode[4453] NOTE: Referenced project...[edited]
2006-01-03 12:23:16.562 Xcode[4453] NOTE: Referenced project...[edited]
2006-01-03 12:23:16.643 Xcode[4453] NOTE: Referenced project...[edited]

See – that’s the “dream chunk.” It is so easy to parse. I could even turn the timestamp into a feed item timestamp. It’s just to beautiful. Then you run into examples like these (grouped together for convenience of presentation):

(event handler):Undefined value
Authentication Service: Started
Jan  3 22:30:50 DanielG5 authexec: executing ...[edited]

Bad boys like this pop up all the time. Isn’t somebody in charge here? Argh – different application services are allowing logging to happen that follows different or no conventions. So my happy little “chunk things together” strategy starts to fall apart.

So I drop this funny little hack in your lap in the hopes that you’ll be inspired to try it out, and if you’re just the kind of thing who enjoys this kind of analysis, you’ll help me come up with a systematic approach for “chunking” the Console log. The real problems are when there’s some really well-defined line like the Xcode examples given above, followed by a dozen lines of output that were spewed by that program’s last “well-defined” timestamped log-point.

I suppose I could compromise and just say “all unidentifiable stuff” goes into a chunk of its own. Then you’d end up with random spewage orphaned from its owner, but it would still be temporally near and therefore show up contiguously in the feed.

A new logic for the tool might be:

1. Look at this line, does it “look like it starts with a date” and “have a decimal number in square brackets”? If so, try to find lines above it with the same number in the square brackets. Put all of these lines into a feed item with a date of the bottom-most line.
2. Look at this line, does it look unidentifiable? If so, group it with unidentifiable lines above it until I regain sanity. For consistency, give this feed item the same date as either the chunk below or the chunk above.

I’d be particularly curious to hear the opinions of anybody who’s done this kind of log file parsing before. Heck, one of you is probably going to post a link to something that already does exactly this. That would be beautiful! I’m looking forward to hearing your thoughts.

Anybody who has dabbled with CoreAudio programming on the Mac is probably familiar with the indispensable MTCoreAudio framework, by Michael Thornburgh. The framework makes it easy for Cocoa programmers to harness the power of CoreAudio’s HAL (Hardware Abstraction Layer) by wrapping some of the major components in Objective-C wrapper objects.

What has Thornburgh been up to lately? MTCoreAudio releases haven’t come as fast and furious as they did in the early days, so he must be keeping busy. It turns out he co-founded a company called amicima (pronounced like Italian – think “da Vinci”), whose mission it is to develop robust p2p infrastructure protocols for a variety of possible uses. Blah blah blah, right? Hasn’t everybody developed a p2p protocol suite? What’s left to develop? I thought so, too, but amicima’s technology offers some compelling features.

The company has just released a Mac OS X version of their flagship technology demo: amiciPhone. This program doesn’t make a great visual splash – it’s not going to win any beauty awards. It’s a “technology demo” in the best and worst senses of phrase, highlighting a robust infrastructure with minimal attention to the application’s user interface. Still, Thornburgh’s solid Cocoa-based interface provides access to the voice, file transfer, and text chat features of the program. And while he’s no visual designer, he couldn’t resist the temptation to add some Mac-only niceties. He sent me a PDF file through the program and, before agreeing to accept it, I was treated to a graphical preview of the document’s first page. That’s cool! amiciPhone is “a lot like Skype” when judged purely by its feature set. But the application isn’t out to dethrone the existing VOIP giants – it’s merely a demo of what the company’s underlying technologies might be used for. (Not that somebody else couldn’t use the technology to write a Skype-dethroner!)

Some things I noticed during my long-distance chat/demo with Thornburgh:

• Voice connection survives change of IP address. This is pretty amazing. Thornburgh opened up a voice chat with me on his PowerBook. He then proceeded to pull the plug on the ethernet, causing a few seconds timeout while his computer came to terms with the backup Airport connection. When his connection came back online, he was chatting away again. He plugged his computer back into ethernet and, because of the Mac’s much quicker switch to the hardwired connection, his IP address (confirmed with tcpdump) switched seamlessly without any interruption in voice quality.
• Slick congestion control. I’ll have to take Thornburgh’s word for it, but apparently most VOIP programs suck when it comes to managing voice and file transfer at the same time. I don’t have anything to compare with (unless you count the numerous times I’ve failed altogether when trying to transfer a file with iChat), but I was impressed when Thornburgh’s voice continued uninterrupted while a giant QuickTime movie started transferring from his house in California to mine in Massachusetts.
• High audio quality. This is a testament to CoreAudio, MTCoreAudio, and speex, the open source audio codec they’re using for the demo. I launched amiciPhone on my Mac with a MOTU 828 audio interface attached, and within seconds I was hearing Thornburgh’s voice through the speakers on either side of my office. The demo allows essentially any CoreAudio output or input device to be selected, so you can use a specific device for amiciPhone without changing your default sound settings.

The fact that this was all happening over a secure AES-128 connection with open source components got me wondering if it wouldn’t be too difficult for an application like Adium to incorporate parts of it, maybe facilitating a “voice chat” add-on separate from whatever protocol is being used for the text-chat functionality. I don’t know if the Adium team already has a plan to support voice, but since Thornburgh’s company is licensing the technology in both GPL and commercial forms, it could be an appealing option for both open source and well-funded software teams.

Of possible interest to Cocoa developers is something called MObj, which betrays Thornburgh’s passion for Objective-C style dynamic programming. This component of the amicima suite is licensed under a generous BSD license, and promises a number of Foundation’s core features in a cross-platform, ANSI-C format. I’ll have to keep that in mind next time I’m forced to write code away from the comforts of Cocoa.

I’m looking forward to seeing what technologies amicima comes up with in the future, and more importantly what creative developers out there might end up doing with them. In the mean time, maybe some kind soul will donate a UI overhaul to amiciPhone – or at least an icon!

It’s the beginning of the month, which means it’s invoice time for me. I very rarely send out paper invoices. These days most companies are hip enough to have addresses like AccountsPayable@<company>.com. So my low-volume invoicing mechanism involves a Pages templates document, and a little manual copy and paste from my web-based hour-tracking system.

One company I work for makes things “easier” by requiring that I submit invoices through a nasty web invoicing system. Just about everything about this site is wrong. To add insult to injury, they charge me, the “supplier” for the privilege of stumbling my way through their ridiculous invoicing system. To be fair I haven’t run out of “free credits” that I got when I first signed up, but I’m really not looking forward to the day when I have to pay for this abuse.

Among the ridiculous things I must endure every month as I fill out the invoice data, is the selection of “units” for my “hourly work” line item. It’s always “hours,” just like every other professional consulting service is almost always “hours.” But hours aren’t special to this company – they’re just another item smack-dab in the middle of the world’s most comprehensive list of invoicing units, and I have to scroll down to the middle of it every time:

Please Select From our Ridiculously Stupid List of Possible Units1,000 10 10 Metres 10 x 0.5ml 10 x 0.6ml 10 x 0.75ml 10 x 1ml 10,000 10-Pack 100 100 Metres 100-Pack 1000L 100g 100g 100L 100ml 10L 10ml 10ml 125Kg 1Kg 1L 2.5L 200L 20ml 250g 250ml 25g 25Kg 25L 25ml 25ml 4 Week 400L IBC 4L 4x48L 5 x 10ml 5 x 5ml 50 Pound Bag 50-Pack 500g 500ml 50Kg 5g 5L 5ml 6 Pack 9 x 17L 9 x 20L 9x15L 9x20L Acre Assembly Bag Bale Ball Barrel (US) (petroleum etc.) Barrel – Imperial Base Basket Batch Board Feet Bottle Box Box of 12 Box of 6 Bucket Bundle Bushel Can Canister Carload Carton Case Catchweight Centigram Centilitre Centimetre Coil Container Copies Crate Cubic centimetre Cubic decimetre Cubic Feet Cubic foot Cubic inch Cubic Inches Cubic metre Cubic metre (net) Cubic millimetre Cubic yard Cycle Cylinder Day Decilitre Decimetre Degree Dollars (US) Double Centre Dozen Drum Each Fifteen kg drum Fiftyfive gallon (US) drum Fluid ounce (UK) Fluid ounce (US) Foot Gallon (UK) Gallon (US) Gram Gross Hour Hucom Hundred Hundred cubic feet Hundred cubic metre Hundred feet Hundred kilogram Hundred Litres Hundred Pounds IBC Inch Inches Inner Item Items Jumbo Kilogram Kilometer Kilometre Kilos per Drum (wet) Kilowatt Kilowatt Hour Kilowatt Hour Kit Kit of 3 Large Length License Link Litre Load Load Lot Lump Sum Mega Watt Metre Metre cubed Mile (statute mile) Milligram milliliter Millimetre Million Minute Minutes Month Ounce Ounce Outer Overtime Hours Pack Packet Pages Pair Pallet (lift) Pallet/unit load Percent Perf. unit Piece Pint (UK) Pint (US) Pound Pounds Quart Quarter Quarter Hour Quire Rack Ream Reel Ring Role Roll Second Service Set Sheet SITA Skip Spool Square centimetre Square foot Square inch Square kilometre Square metre Square yard Synthetic Latex Dry Kilos Synthetic Latex Dry Tonnes Synthetic Latex Wet Kilos Synthetic Latex Wet Tonnes Tank Therms Thousand Thousand Dosages Thousand Feet Thousand Sq. Metre Tin Ton Ton (UK) or longton (US) Tonne (metric ton) Truckload Truckload Tube Twenty-Four Unit Vial Wallet Week Wholesaler Wraps Yard Year

I know – it’s funny! Like – Brazil funny. The first time, and then you start to loathe invoices. This should be a happy time, when you think “I’m going to get paid.” Instead I think, “I wonder if I’ll survive the invoicing system this month.” Maybe one month I’ll just charge them for “27 50-Pound Bags of Mac Consulting” and see what happens.

Other fun aspects of this system include the classic “works only if you click the Submit button” bug. Hitting return submits, but submits with bogus data. This allows me to regularly observe the classless manner in which the system conveys SQL database errors to me! “Oh! ’16:Can not init Service’, that explains it!”

This is a perfect follow-up to my Verizon post. It’s another example where a company must be making hundreds of thousands – no millions – of dollars peddling crap. On the bright side, when you swim in a sea of useless and buggy software, it’s easy to feel good about putting an honest day’s work into quality programming, and working with people who share these values.

Since moving to the Northeast I’ve been a customer of Verizon for my telephone and internet service. Being a customer of Verizon is basically the same as being a customer of SBC, as I used to be in San Francisco. Things basically work most of the time, but any time you need to change anything, install anything, pay anything, get anything fixed, etc., you better hold your breath and pray for the best.

Among the sad offerings at Verizon is its infuriating web page. Back in San Francisco I remember being upgraded to “SBC Yahoo!,” and that when I switched to the Yahoo-branded service, at least the web interface for things like billing and services was a bit better.

So when I got a letter in the mail from Verizon offering a “free upgrade” to Verizon Yahoo!, I thought I had nothing to lose.

Grr … it makes me download and install custom branded software. Apparently this is part of the “upgrade.” If you want to play in the Yahoo home page, you have to let it run a scary installer-type program that connects back to the Internet and displays clunky HTML status while it has its way with your Mail and Safari settings.

I should have backed up everything, but I was feeling lazy I guess. Knowing the Telco companies, I should never be lazy. I let it do its thing all over my computer, and when it finally relaunched Safari: surprise! no more bookmarks. And when I say “no” bookmarks, I mean it. It didn’t even stuff a bunch of Verizon marketing bookmarks in there to pretty up the browser. Just a big gaping hole where my bookmarks (in the bookmark bar and menu) used to be.

Of course, it also replaced my home page setting in Safari. To this! Yes, the garbage you see on your screen is what I see on mine, too. Lucky us – living the high life with Yahoo and Verizon.

I was relieved to learn that whoever programmed this little monster was at least concerned enough about their work to make a backup of the existing Bookmarks file. Not that this is acceptable for the 99% of users who won’t freaking know to look in ~/Library/Safari for the lost Bookmarks. But for anybody unfortunate enough to go through this upgrade and suffer my fate, maybe you’ll get “lucky” like I did and find your bookmarks thus preserved.

You can imagine that by this point I was pretty scared to relaunch Mail.app. Another “favor” the Verizon installer did was to reconfigure my Verizon email account in Mail so that it would go through a yahoo branded SMTP/POP server. I could easily imagine this programmer clobbering all of my accounts, assuming that the user “surely only uses Verizon email.”

I’m happy to say that my pessimism in this regard was ill-founded. Whether by skill or dumb-luck, the Verizon installer managed not to obliterate my Mail settings.

Now I’m just hoping it didn’t do anything else that I haven’t noticed yet. Keeping my fingers crossed.

Update By looking carefully through the “installer” program, I was able to find quite a bit of information. I know the name of the person who wrote this code, which I will omit to spare him the public humiliation. I know the name of the company, which I will omit to spare myself from legal harassment (though we should be suing them!), and I know the exact cause of the bug.

The installer is based on a collection of sloppy AppleScripts, and the author was kind enough to leave the offending AppleScript’s source code intact in the shipping image. I know the answer to my question above: was it skill or dumb-luck that caused my Bookmarks to be backed up? It was definitely dumb-luck. In fact, it was a sloppy piece of programming that led to that remnant being left around.

The Verizon installer attempts to inject its bookmarks into the user’s Bookmarks.plist file by:

1. Making a copy of the existing Bookmarks.plist file.
2. Erasing the original Bookmarks.plist.
3. Iterating through the original bookmarks one line at a time, echoing the line back out to the original Bookmarks.plist.
4. Sneaking its own items in when it feels the time is right.

So what happens when one little thing goes wrong in this nasty mess? You lose your Bookmarks.

This whole thing involves about a hundred bazillion dangerous, presumptuous “do shell script” calls. Instead of using AppleScript’s built-in file access routines, the author has chosen to use shell scripts and the “echo” command to perform all of his writes. For all of his reads, he uses a combination of the UNIX “cat” command with the “sed” and “tail” command. For each line of the original Bookmarks line to be read, he cats the *entire* file and pipes the result through sed and then tail. Seeing this kind of stuff makes me wonder whether half of my hard drive was deleted and I just haven’t found out yet.

So what caused the script to fail, and my bookmarks to disappear? The following line of AppleScript code:

do shell script "cd;/bin/echo '" & theLine & "' >> Library/Safari/Bookmarks.plist"

By assuming that “theLine” will never contain any characters that cause unexpected, perhaps dangerous side-effects to occur, the author has made his disastrous error. Among my Bookmarks is a javascript “bookmarklet” which contains, as part of its “URL” a lot of interesting characters. It’s a JavaScript! The author of the script probably felt pretty comfortable about putting single quotes around “theLine.” By doing so he essentially protected himself from lots of the hairy problems that could arise from special characters like ampersand and tilde popping into the command line. But what about the single quotes in my Bookmark? As soon as the first single quote from the Javascript bookmarklet is hit, the party’s over. As the rest of the string is processed, it’s a crap shoot whether we’re in “safe” mode or not. When he hits the next single quote in my Bookmarklet, *phew* we’re safe again. But soon there’s another one and we’re in danger. It so happens that during one of these unsafe stretches, the script runs into an ampersand. What does the shell do when it encounters an ampersand? It forks the process out and continues accepting commands. For instance, go the Terminal and type:

ls & ls

See how ls gets called twice – sort of concurrently? So every time this piece of work from Verizon gets to an unprotected ampersand in my Bookmark file, it ends the current command and starts up another one. This is what the AppleScript failure looks like when I narrow it down to the offending line:

sh: line 1: amp: command not found
sh: line 1: amp: command not found
sh: line 1: amp: command not found
sh: line 1: amp: command not found
sh: line 1: +(new%20Date).getTime();var%20request=
new%20XMLHttpRequest();request.open(POST,path,true);
request.setRequestHeader(Method,POST%20+path+%20HTTP/1.1);
request.setRequestHeader(Content-Type,application/x-www-
form-urlencoded);request.send(post);};: No such file or directory

It’s damn lucky that “amp” isn’t an alias for “format my hard drive.” Damn lucky for me, and damn lucky for this careless Verizon consultant. Hopefully every other person in the world this happens to will be able to share my relatively lucky escape.

I would try to inform Verizon about this, but to be honest I don’t think I could get through to them if I tried. If anybody in a position to be embarrassed by this happens to be reading, maybe you can help get this brought to the right person’s attention.

Update 2 It’s a lucky coincidence that the data being parsed in this case is fairly well-limited to what Safari will allow you to put in a bookmark. Since Safari converts all spaces in the bookmark (even if it’s inside a javascript block) into an escaped space character, there are no actual spaces in the bookmark text by the time Verizon’s installer gets to it. This is very lucky, because it means a bookmark like “javascript:alert(‘&quot;rm -rf ~; echo ‘)” won’t actually delete your entire home directory. Instead, the shell says something like “command not found: rm%20-rf%20~”.

This lucky break for us doesn’t make the installer any less frightening – it still kills your bookmarks, or at best, just corrupts all your bookmarklets. And I would be surprised if this is the only bug in the installer – I certainly advise any Mac user to avoid running this at all costs. If you do need to run it, I would do a full backup of your computer before doing so.