I have been plagued by a bug I see often in Terminal. I know it’s been at least months, and it may have even been years. But it’s gotten worse lately. I call it the “phantom [Process Completed] bug.” The symptoms of this issue are that every so often an innocent attempt to open a new Terminal window instead yields an immediate “[Process Completed]”. What this has meant for me is that I’ve basically gotten used to “trying again” whenever I see this. It sucks though, because it limits my ability to automate with confidence when it comes to opening new shell windows. The thing about this bug is seems to “happen only every other time per pseudo-tty.” So if it fails once, it will probably not fail the very next time you try it! A dangerous recipe for “acceptance,” but this is a terrible concession to have to make to a computer!

Things finally came to a head today and I decided to track down the root cause. I posted to the MacOSX-Dev mailing list for help, and Chad Leigh pointed me in the direction of some very helpful advice that, while it alleviates a separate problem I was experiencing (running out of processes, and getting fork failures), it doesn’t solve the “Process Completed” problem.

I needed to collect some hard data, so I started out by rebooting my machine. On a freshly booted machine, I am able to create a dozen new Terminal windows without running into the problem a single time. Close those dozen windows and try again, and about half of them fail! So this has something to do with reusing pseudo-TTYs? I decided to switch over to “iTerm” for a little comparison. Sure enough, opening a dozen windows there yields about the same failure rate.

I searched the web (again, I had tried this months ago), and came up with some interesting leads. Some people have noticed that the problem is worse with zsh. I run zsh, lucky me! Other people have noticed that the problem only happens when a Terminal window is closed without manually closing down the shell inside first. Sure enough, I almost always close my shells with Cmd-W, allowing Terminal to “take out the trash.” I run some more tests, and what do you know? I can “cleanse” ttys by making sure that I run “exit” on the shell before closing the window.

Since the rumor is that it happens more in zsh than other shells, I’m inclined to believe it’s a shell bug. I decide to test against a very simple shell: “bc” the built in UNIX calculator. Setting my default shell to “/usr/bin/bc”, I go on a rampage of opening a dozen or more new shells, closing them all, and trying again. Sure enough, “bc” is completely clean. It can take a beating and happily reopen on the same old tty that got slammed last time-around. I switch back to zsh, and open a bunch of terminals again. They’re all clean! So bc left them clean enough for zsh’s liking. It must be something about the way zsh “takes out the trash” (or doesn’t).

I switch the default shell to “/bin/sh”. In twelve opens, I witness the failures due to booby-traps set by zsh. Close twelve. Open twelve again: all clean. Sh (bash, actually) cleans up well! Try with tcsh: all clean, too!

Damn you zsh! I love ya, but you’re a messy sucker!

I’m not about to give up zsh, and I’m not quite ready to look into locating the bug in zsh, but I need to work around this. I know that if I type “exit” every time I need to close a shell window, I will never see the bug again, but I’m too lazy to remember to do that! FastScripts/AppleScript tag team to the rescue. I wrote this simple script:

With FastScripts I’m able to override just about any menu shortcut in just about any app, so I set it up as an Terminal-specific script, and assign it the shortcut Cmd-W. Now, whenever I type Cmd-W in Terminal, the same shortcut I’ve been using for years to close shell sessions, the Terminal first performs an explicit “exit” in the window, and then closes it. I get to keep my behavior, but the buggy zsh conditions are no longer produced.

I’m so excited by this that I’ve spent a few minutes just holding Cmd-N down for several seconds, closing all the windows, and repeating. Ah, it’s going to be a nice autumn.

Update: I thought I had observed some funkiness with the state of the pseudo tty files after zsh was closed on them, but I’ve come to find out that the behavior is not unique to zsh. I think it’s a red herring. Other shells sometimes leave the tty in this funky state and it doesn’t seem to affect their ability to reopen the tty as a healthy session.

Update 2: It occurred to me that one of the downsides to replacing Cmd-W with a scripted keyboard shortcut is that I can’t take advantage of the “hold down the keys for repeated invocation” functionality that you normally get for free with menu items.

I decided the only reason I was taking advantage of this in Terminal is because it lacks a proper “Close All” function. So, I wrote another script, this one mapped to Cmd-Opt-W in Terminal, which will first go through and call “exit” in all the open windows, and then close them all:

Usually when I get an email message with HTML content, I reach for the delete key faster than you can say “spam!” But there are some messages I receive which contain important HTML content. For instance, the invoicing system one of my large corporate clients uses (why, I don’t know), is completely unusable in most regards. The only way I have found to get a personal copy for my records of invoices I issue through it is to have it email me a copy. When the copy arrives, of course it is in a table-heavy HTML format, which is unreadable even if I attempt to save it as plain text. It looks terrible as HTML too, but at least it’s readable!

So, I get this HTML email regularly, and I need to save it to my “Unpaid Invoices” folder. Unfortunately, Apple Mail has no obviously easy way to do this. I have before today resorted to various steps including showing the raw source and manually copy/pasting the HTML portion out and into another file. Today I decided the B.S. stops and I get an easy solution.

I didn’t even bother looking around for an existing script. I knew what I wanted so I wrote it. The result is a script that should do a reasonably good job of grabbing the HTML out of a selected message and saving it to disk for you. I don’t think of this as something that most people want to do often, but if you find you do need it, now you don’t have to write it yourself:

Click here to download: Save HTML Content

Since I’m a little obsessed with AppleScript, I end up pasting AppleScript code into my blog. I recently decided it was a real shame that I couldn’t easily replicate the way AppleScript code looks in Script Editor, in my blog. I thought about this situation for a while, and then did some experimenting.

If I copy the contents of a Script Editor window, and paste it into TextEdit, it looks great! Just like Script Editor! Why can’t I get that in HTML? I decided to try saving as HTML from TextEdit. Sure enough, that yields an HTML look-alike, but it depends on a rather elaborate set of CSS definitions in the <head> block of the document. I’m not ambitious enough to figure out whether I can include non-inlined CSS styles in random locations like the content of my blog post. Besides, I’d like to be able to “grab some styled text” and run with it.

It occurred to me that maybe AppleScript itself could help me get the formatting I needed. After a little tinkering, Sure enough, with TextEdit and the Text Suite, I can parse a chunk of styled text and spit out reasonably similar-looking HTML!

In this blog entry, I submit to you a script that will turn the text from the front-most document in TextEdit into a self-contained, inline styled chunk of HTML. It’s not pretty, but it works! As a proof-of-concept, the following code was copied from Script Editor, pasted into TextEdit, and then run from Script Editor. Because of some WordPress funkiness I had to change a few minor details in the source for this page, but I think you’ll find that the script works as expected in most contexts.:

Wow, that looks quite a bit more Script-Editor-like than your average web posting! It’s shamefully inefficent (take a look at the HTML it generates), but it gets the job done! I am happy to have this new tool at my disposal, but aside from producing rotten HTML code, the script is also dog slow. Also, when I copy the script above and paste it into Script Editor, it comes out double-spaced. Anybody out there have any tips for achieving this goal in a slightly more gallant manner?

Update: The script above has been updated since the original post. I discovered a few prefetching tricks that dramatically improve performance of the script. How dramatically? Thanks to Shark I can tell you! The test case for both scripts is converting the styled script text above from TextEdit to inline HTML. The old script? 58 seconds! 40% of which was spent in TextEdit itself. The optimized version above, which prefetches the pertinent styled text data from TextEdit? 4.5 seconds, with only 5.4% of the processor taken up by TextEdit. More than 10-times faster! Just imagine how fast it will be when a good scripter rewrites it!

Most Mac developers have, by now, given Apple’s CHUD Performance Tools Suite at least a cursory run-through. These tools, and in particular Shark, are incredibly effective for pinpointing performance problems that may be plaguing your application (or other code).

The presence of these tools has helped many developers inside and outside of Apple produc more efficient code. By effectively painting the low hanging fruit neon green, Shark allows us to easily locate our most embarrassingly slow code – hopefully before anybody else does.

One area of “coding” on the Mac where optimization has been sorely overlooked is that of AppleScript authoring. AppleScript performance can range from instantaneous to mind-bogglingly slow, and usually clocks in at some random level between the two, having been left complete up to chance by the script’s author. While laziness in the past may have been excusable, poor script performance is inexcusable in a post-Shark world.

Chumming Along

When professionals want to lure a shark nearby for viewing or killing pleasure, they may litter the water with visceral mix of blood and flesh: irresistible to the carnivorous beasts. Luckily, attracting Shark to your AppleScript is much easier to stomach, if you know a few tricks.

In AppleScript, as in other code that is executed in linear fashion by a computer, it is frequently most interesting to examine the performance metrics over a quite specific period of time. If you’re experienced with Shark, you may know that a clumsy method of starting and stopping the sampler is provided by way of a keyboard shortcut. This is useful, but often not precise enough for gauging the minute differences between scripting choices. By taking advantage of the the often-overlooked chudRemoteCtrl tool, which can be used to (among other things) remotely start and stop sampling in the main Shark application, we can get fairly precise performance details which are furthermore broken down by process, a detail which can be especially useful in assigning blame.

Preparing Shark for Duty

We can put control of Shark literally at our fingertips, but as the CHUD tools do have some eccentric behaviors, there are a few steps which must be followed to prepare Shark for reliable, scripted obedience:

1. Launch Shark
2. Ensure that Shark is set up for “Remote” control. This is off by default but can be turned on by selecting the “Programmatic (Remote)” menu item from the Sampling menu.
3. Ensure that all sampling settings are to your liking. You probably want to target “Everything” as opposed to a specific process, because a given AppleScript is liable to enlist the help of many processes to achieve its goals. If your Shark window looks more or less like this, then you’re probably in good shape:

   

4. Optional: Since sifting through the large quantity of data Shark produces can be daunting, it may be helpful to quit as many running Applications as you can before running your tests.

The power of Shark is now “cocked and ready to fire.” Next we’ll look at what we need to do from an AppleScript to take advantage of this potential.

Unleashing the Beast

Programmers familiar with using Shark from languages like Objective-C or C++ may be familiar with the CHUD library calls that allow procedural starting and stopping of Shark’s profiler. While AppleScript doesn’t include this functonality, we can take advantage of AppleScript’s shell scripting interface to invoke a powerful tool: chudRemoteCtrl. By passing the appropriate arguments to this tool, we can easily start and stop the profiler during the execution of a script, while giving meaningful descriptions to the associated sample sets that are generated as a consequence.

To demonstrate the power of this technique, I will use Shark to profile the performance of an extremely trivial example script. This scripts purpose is to compute the 64th power of 2, through methods ranging from completely idiotic to downright obvious. I chose this example simply to give you dramatically different performance numbers to compare. In real life, the numbers may not always be as “no-brainer,” but the same principles apply.

on run
	-- Run some tests
	set powersOfTwo to 64
	
	my SharkStartSampling("Shell Scripted Dumb Calculator")
	set worstResult to 2
	repeat (powersOfTwo - 1) times
		set badScript to "echo '" & worstResult &"* 2' | bc"
		set worstResult to do shell script badScript
	end repeat
	my SharkStopSampling()
	
	my SharkStartSampling("AppleScript Built-In Dumb Calculator")
	set betterResult to 2
	repeat (powersOfTwo - 1) times
		set betterResult to betterResult * 2
	end repeat
	my SharkStopSampling()
	
	my SharkStartSampling("AppleScript built-in expression")
	set bestResult to 2 ^ powersOfTwo
	my SharkStopSampling()
	
	my SharkStartSampling("Null Operation")
	my SharkStopSampling()
end run

on SharkStartSampling(sampleLabel)
	do shell script "chudRemoteCtrl -s " & quoted form of sampleLabel
end SharkStartSampling

on SharkStopSampling()
	do shell script "chudRemoteCtrl -e"
end SharkStopSampling

Paste the above code into your script editor and, after double-checking that Shark is “ready”, run the script. It’s important to keep Shark happy because it seems to misbehave after unexpected events like attempting to start sampling twice without ending the previous sample. If you get into trouble, you have to quit and relaunch Shark, putting it back into the prepared state described above.

As the script runs, you’ll see Shark processing samples and popping up analysis windows – one for each of the named tests I included in the script. When the script is complete, take a look at the resulting data sets. The first thing that strikes me as I look at the results on my PowerMac G5, is that the dumbest of all the code takes at least 10 times longer to execute than the smartest. As you look at the results in Shark, you can focus on particular processes and apply the powerful data mining utilities to eliminate uninteresting stack calls. For instance, in a real world situation, you will often only be interested in the samples obtained from the program your script was executed in, and the program your script is “telling” to do something (if applicable).

As you glance through the resulting sample sets, you may notice that even the smartest technique comes up with a significant number of samples. There is some overhead to the profiling mechanism, and being a multi-tasking system, the other processes on your computer may be stealing time away from your test. For this reason, I included as a final example a “null operation” which simply starts and stops sampling as quickly as possible. On my computer, the total difference between the control case and the “built-in expression” case is only 15.7ms. By comparison, the dumbest case clocks in at a whopping 8 seconds!

In Summary…

Clearly this example was an extreme case of low hanging fruit, but the principles can be applied in any script to obtain information about its effect on any process in the system. You may also find yourself using this technique to verify or dismiss suspicions about the software you use. What exactly happens when an application asks the system to beep? Is perl faster than ruby? Does iTunes hesitate just a bit longer than usual when asked to play a Britney Spears song? Only one way to find out…

An Update…

On the AppleScript-Users mailing list, Has makes some interesting points about this approach. He points out that, because Shark doesn’t gather data about the particular AppleScript subroutines that time is spent in, it can’t really be described as profiling the script. While I think this is a fair point, I maintain that it can be very useful to figure out where the system is spending time while executing your script. The nice thing about Shark is that it will represent samples from code in any process that was invoked to fulfill the demands of your script. True, if your script is 100% AppleScript and doesn’t invoke any scripting additions that reach the system APIs, then the profile might be kind of boring, but that is in itself informative as well.