I recently had the mixed pleasure of spending a few nights in the Argent Hotel (recently bought by the Westin), a rather nice edifice located in downtown San Francisco. In the month of December, the hotel had really attractive rates – I guess in the big scheme of things, it makes sense that not many people travel to San Francisco for the holidays. It’s sort of a “while you’re young” kind of town, and I imagine most people rush off back to their family homelands when the holiday season arrives.

But being from the bay area, I found myself back in San Francisco for a few days and needing accommodation. Right next to the BART, clean rooms, courteous staff, what more could I ask for?

Not being reamed left and right with nickel-and-dime bullshit.

Honor-bar fridge: No personal use.
Each local telephone call: $1.
One two minute domestic long-distance call: $9. (!!!)
Wall-tethered internet access: $15/day.
Reminding oneself to finally remember to call ahead and check on this bullshit: slight comfort, at best.

I will not rent rooms from nickel-and-diming bastard hotels.
I will not rent rooms from nickel-and-diming bastard hotels.
I will not rent rooms from nickel-and-diming bastard hotels.

Oh yeah, The Westin. They ripped me off in Chicago, too. Hopefully I’ll be more careful next time.

“I drank from a faucet, and I kept my receipt
For when they weigh me on the way out, here nothing is free.”

Death Cab For Cutie – Why You’d Want To Live Here

The best (i.e. worst) part about the Argent was the sign on the honor-bar fridge. They meant to say “We can’t let you use this refrigerator for personal items, because we don’t make any money off of that.” Instead, they insulted my intelligence (and all 666 other guests – no joke), with the statement: Please do not place your personal items in the Refreshment Center, as they may spoil. They might as well shut off access to the bath and bed, as my skin might melt off in a freak chemical accident, too.

No way to to treat a customer. We pay the big sum of money up front, so we don’t have to pay the bullshit money as we go.

Today Garrett Murray let loose a rather provocative post, wherein he details the unfortunate series of events that unfolded after selling his company’s product and web site to Brian Ball of MacZOT.

Long story short: Brian agreed to pay roughly $5000 for the product, in installments. But a contractual loophole of Garrett’s own devising inadvertently allowed Brian to simply stop paying, and allow ownership to revert back to Garrett. Brian essentially claims that Garrett got more money out of the deal than he could have otherwise, and Garrett essentially claims that Brian was kind of a prick about the whole deal.

You be the judge. Also be sure to check out the extensive debate on the (now buried, it seems) digg entry. Here both Brian and Garrett have a chance to let their feelings be known in the public debate. My summary? Liberals will agree with Garrett and conservatives will agree with Brian. (“Have a heart” vs. “Tough shit, kid.”)

But what really interests me…

I find the drama of this story a bit engaging. It’s always at least a little bit interesting to watch a car crash, even if you can sort of tell that people are getting hurt. But what I can’t believe is that a relatively mature product like this sells for only $5000. And Garrett was including 50 hours of consulting with the deal. By my reckoning that means he was essentially selling 50 hours of work and throwing in the business for free.

But this got me thinking. Will anybody sell me their product for $5000? I am a good buyer. Make me an offer, and if I like what you’ve got, I’ll pay cold hard cash for it. No installment crap. If you won’t sell for $5000, how much will it take? Send me a private email detailing the terms of your offer (not legally binding), and I will let you know if I’m interested. I promise not to divulge any specific details to the public about your offer. I just want to know what my options are. If you’re a desperate shareware developers looking to get away from a particular product, maybe we can help each other out.

After a week or so, I’ll post a followup consisting only of statistical information such as average offering price, number of total offers, etc. And I’ll let you know if I ended up with any new products in the pipe.

Let’s make a deal!

Update January 5: I’m sure you’re probably curious whether this is garnering any results. So here’s a little update: as of this writing I’ve received at least 15 legitimate, reasonable offers comprising mostly Mac software titles, but also a web site, and a Palm-platform product. There have also been, needless to say, a few good jokes.

The viability of the serious offers, in my estimation, sort of run the gamut. But there’s definitely some interesting stuff in there. The range so far is from $2,000 to $20,000, with some quite interesting stuff in the lower end. I have to confess that if I make an acquisition I will probably be bargain-hunting. Some of the pricier offers would still be a great deal to a more monied investor, but are probably too rich for my blood right now.

Update: The product I acquired is MacXword from Advenio, LLC. The product was rebranded and released on March 21 as Black Ink! For more information about the process I went through and the offers I received, see my Acquisition Roundup post.

If you’re anything like me, the most challenging part of your development is when the process itself doesn’t seem to be working. OK, I designed the architecture of the project. I wrote the code. I expect everything to work. It’s not due until tomorrow, but what’s this? The linker’s failing?

Usually we get to take compiling and linking for granted. These tools are used by just about every shipping binary executable inside or outside of Apple, so chances are they’ve gotten all of the really critical bugs ironed out. Still, you have to wonder when you stare down a perplexing “undefined symbols” error, which you can’t for the life of you imagine being accurate:

ld: Undefined symbols:
_acl_copy_ext_native referenced from CarbonCore
expected to be defined in libSystem
_acl_copy_int_native referenced from CarbonCore
expected to be defined in libSystem
_task_name_for_pid referenced from CarbonCore
expected to be defined in libSystem

The disagreement between two of Apple’s low-level frameworks is a great tip that this isn’t really my problem. Still, I need to make sure I’m not leading the linker astray somehow. I am using SDKs (-isysroot from the command line), so maybe I’ve given the linker mixed signals about where to search for libraries. After scouring the makefiles (in this case it’s the Subversion project), looking for unwanted -L or -F flags that would point the build process back into an unwanted directory, I’m left empty-handed. This should be working!

Confirm LDs Complaints

Just to get my head on straight I decided to confirm that that SDK’s version of libSystem had the symbols that are allegedly “undefined”:

% cd /Developer/SDKs/MacOSX10.4u.sdk                                                                           
% nm usr/lib/libSystem.dylib | grep _acl_copy_ext_native
         U _acl_copy_ext_native
900fe38c T _acl_copy_ext_native

See the capital T? That means “implemented” to you. It actually means “Text Section Symbol,” but what’s important is it’s a symbol for code, not a symbol for an undefined import, like the U right above it. It’s OK for a single binary to have both U and T symbols, because the file will resolve against itself. If a file only has Us for a given symbol, it’s going to look elsewhere for resolution. That’ the situation that CarbonCore is in:

% cd /Developer/SDKs/MacOSX10.4u.sdk                                                                           
% cd System/Library/Frameworks/CoreServices.framework   
% cd Frameworks/CarbonCore.framework              
% nm CarbonCore | grep _acl_copy_ext_native 
         U _acl_copy_ext_native

Getting bored yet? Let me cut to the chase. My SDK’s CarbonCore and libSystem are in perfect agreement. CarbonCore wants some _acl symbols, and libSystem has them. Looking at my installed system, however, I discover that these symbols are completely absent. Apple has changed the inter-framework dependencies at some point, such that my OS (10.4.8) is not compatible with my 10.4 Universal SDK.

Which implies that the linker is looking in my OS instead of the SDK, when trying to resolve these symbols. But why!? And do I really have to start debugging the linker?

Spy On Dynamic Linking

If you find yourself in a similar boat, there are some debugging aids you should know about. Set the environment variables LD_TRACE_ARCHIVES and LD_TRACE_DYLIBS to encourage ld to spit out some potentially useful information. When I did so, I was surprised to see it confess openly just how many libraries it was pulling in from my root system, instead of from the SDK. In particular:

[Logging for XBS] Used dynamic library: /usr/lib/libSystem.B.dylib

Aha! So at least I know why it’s got its head screwed on wrong. But what causes this? I’m specifying the SDK path with the “-isysroot” parameter to the gcc, which in turn ends up getting passed as a “-syslibroot” parameter to ld in the link phase. What’s supposed to happen here is that the provided path gets prepended to all full path library references, and that path is given priority in the search .

So I don’t think it’s too far out of line for me to believe that the log line shown above is a sign of a bug in ld. In no case should a library be used from “/” when a perfectly good counterpart exists in the SDK tree.

Hack The Linker

Thanks to Darwin, I find myself downloading and building my own copy of ld. At least I can take a look at what is supposed to happen, if not necessarily fixing the problem. I peppered the sources with extra printf statements just so I could figure out what was happening when. It turns out that the linker’s failure to stay in the SDK tree is caused by the resolution of 2nd-level dependent libraries. That is, I link to one dylib and it has some other libraries listed in its dependencies. When ld gets to those libraries, it adds them to its list (if they’re not already there), but somehow the SDK path doesn’t get prepended, so it just uses the verbatim path. I tracked down a spot in the source code where the right thing is attempted:

if(next_root != NULL && *dylib_name == '/'){
	p->file_name = allocate(strlen(next_root) +
				  strlen(dylib_name) + 1);
	strcpy(p->file_name, next_root);
	strcat(p->file_name, dylib_name);
	p->dylib_file = p->dylib_name;
}

Emphasis mine to show that we’ve got a bug here: the code is trying to replace the existing dylib path with one that has “next_root” prepended to it. But after going to all the work of building that string, it simply ignores it, assigning the base name as the file name.

Unfortunately, even after fixing this bug and building a custom version of ld, the problem was not completely fixed. I am resisting the urge to become too much more familiar with this code, but so far I suspect that the “patched up dylib” continues to have the table of contents for the old dylib. So even though it’s now pointing into my SDK, it’s still not finding the desired symbol names in the list.

All in all I think this just reveals that “ld” is a lot less ready for prime-time in the SDK department than I’d hoped. I’ll admit that it takes some fairly unusual circumstances to detour the dylib search out of the SDK: I don’t even have the logic completely worked out yet. But I’m standing by my belief that “whenever a qualified library in the SDK path exists, it should be used in favor of a library outside the SDK.” Radar #4904317

By the way, I’ve found at least one instance via google of somebody else having exactly this same linking error (with the _acl* symbols). This was in the WebKit project, which I’m sure also has complex dynamic linking dependencies. I notice in that email thread that the problem just mysteriously disappeared one day. What I bet happened is the person updated their Xcode or system so that the SDK and system were once again in sync. It doesn’t mean the bug isn’t still happening, it just means its quiet because the wrong version of the dylib happens to look a lot like the right one. I suspect this bug is happening a lot more than people realize, but in mostly innocuous ways.

Update: Mea Culpa

Well, when you write your thoughts out to the public you’ve gotta take the victories with the defeats. I’m a little embarrassed to admit that I’ve been set straight by Apple’s Eric Albert, who informed me that everything would “just work” if I removed a bunch of full-path references to /usr/lib that were in my link line. It was dumb of me to assume a bug in ld … just like my second paragraph above says, “chances are they’ve gotten all the really critical bugs ironed out.”

So if it’s not a bug, what is it? A caveat! I jumped to conclusions when I got the source code and started mucking about. My interpretation of the source code supported my idea that ld should aggressively look for libraries in SDK, which might have been the goal at one point in the project’s history. But what’s in the source code doesn’t matter. The man page for ld clearly describes the -syslibroot option:

-syslibroot rootdir (32-bit only)
	Prepend rootdir to the standard directories
	when searching for libraries or frameworks.

Now that I’ve been set straight I know where to put the stress: on the word searching. The SDK rootdir doesn’t apply to libraries referenced by full path, some of which were erroneously present in my test case. Since these erroneous references showed up on the command line before the corresponding -l for some critical system libraries, they caused the insertion of non-SDK dylibs into ld’s search list.

In summary: if you’re using an SDK you must rely on ld’s search mechanism to find all the Apple system libraries you depend on. If you make the mistake of specifying any of them by full path, you run the risk of leading ld astray from the SDK directory. You can’t expect ld to prefix full-path specifications with the SDK path.

Developers, and some power users who are reading this have probably heard of launchd. It’s Apple’s “mama process,” responsible for launching other processes at startup, login, at regular intervals, or on demand. If you open the Activity Monitor application, and view “All Processes, Hierarchically,” you’ll see that there are only two top-level processes: kernel_task, and launchd. The kernel sits there and does its thing, while launchd spawns all the other processes that make up your Macintosh computing experience.

One big benefit of launchd to end-users is that most of what gets launched can be easily examined by perusing the configuration files. Launchd looks in five well-defined locations for launchable items:

/System/Library/LaunchDaemons
/System/Library/LaunchAgents
/Library/LaunchDaemons
/Library/LaunchAgents
~/Library/LaunchAgents

So all you have to do to get a feel for (most of) what launchd is up to is start poking around in those directories. For each independent “launch” that launchd controls, you’ll find a corresponding configuration property list file. For instance, you’re guaranteed to find a bunch of examples in /System/Library/LaunchDaemons, and depending on which third-party packages you’ve installed, you might find things in other places, as well.

Launchd In Context: FTP

Take a quick look at ftp.plist, one of the system launch daemons. Depending on how you’ve configured your system, you’ll either see that a “Disabled” key is present and set to “true,” or it is absent. This corresponds directly to the “FTP Access” item in the System Preference’s “Sharing” panel. Go ahead and toggle the setting in your preferences, and observe that the ftp.plist file is immediately updated to reflect the new setting. This tells launchd that it should no longer consider launching the ftp daemon when an incoming ftp connection is attempted.

What constitutes an incoming ftp connection? Look further down the ftp.plist and you’ll find an entry with key name “SockServiceName” and a value of “ftp”. Launchd uses this name to lookup the socket port/protocol information for the named service from /etc/services. In this case, when an incoming connection is attempted on port 21, launchd will, if this item is not disabled, launch the daemon at /usr/libexec/ftpd, in order to handle the incoming connection. Groovy, huh? Read more about network handlers and other on-demand daemons.

Boot-Time King

I said above that these configuration files reveal “most of” what launchd is up to with regard to running things on your system. Because launchd can’t handle all types of configurations, and because its introduction has been sort of gently staged, it’s responsible for running a lot of the older mechanisms like /etc/rc, StartupItems, etc. This means that launchd is basically responsible for everything from when the kernel gets loaded on. That includes booting your Mac!

Things have been in such flux over the past few years that it’s hard to keep a straight answer when it comes to exactly what takes place at boot time. Excellent overviews such as Mac OS X System Startup by Amit Singh, or The Boot Process by Apple, are liable to become outdated as launchd continues to evolve.

Fortunately, the entire process is transparently observable, because launchd is open source. If you’re ever in doubt about any nuance of the startup process, just look at the source code. The repository even contains the boot-time configuration file “/etc/rc”, and the source for antiques like SystemStarter. You can even check out how the boot process is going to look in Leopard. (Note: free registration on macosforge.com required). I couldn’t figure out how to check out those repository URLs from the command-line, but I discovered an alternate URL that works, and doesn’t seem to require registration.

Editing Launchd Configuration Files

It won’t take you long to discover that the launchd configuration files, while elegant and modular, are a bit of a pain to read, let alone edit. Fortunately, a couple of third-party solutions exist to ease this task.

Lingon, by Peter Borg, is a very powerful editor which also serves as a sort of browser for all the existing configuration files on your Mac. You can selectively start or stop any existing configuration, and it takes care of asking you to authenticate for editing those system-owned files (after warning you profusely not to do so!). It even sports a “wizard” interface that asks in relatively plain English what it is you’re trying to accomplish, and guides you through the creation of new, custom launchd configurations.

While Lingon is essentially easy to use, it has some rough edges which scream for refinement. I’d really like to see the application receive a UI overhaul. Lots of little awkward UI elements and poorly phrased dialog make it slightly less than a 100% success. But it sure beats editing in vi, or even Property List Editor.

One of Lingon’s nicest features is its “Expert” pane, which allows you to examine and edit the raw XML of a configuration at any time. This is so useful that I’d like to see if featured as a full-time view, perhaps in a hideable split view pane. It would be fun to be able to watch the changes live in the XML source while tweaking details in the UI. But Lingon is, in the spirit of launchd itself, open source. So any shortcomings that ultimately get me down can be remedied to my own (or your own) liking.

Apparently I wasn’t the only person to object slightly to the UI of Lingon. Launchd Editor is a $5, closed-source alternative that bites off a much smaller feature set, while trying to improve on the UI of Lingon. It’s clear by observing some of the configuration panes that Launchd Editor was inspired by Lingon (or vice-versa, I’m not positive which came first, but I’d guess Lingon did). Some of the control layout is almost identical to Lingon, but touched up a bit in alignment and wording, mostly for the better.

Launchd Editor is limited by its adoption of a document-centric model, such that the user has to specifically locate and open a configuration file. It also lacks the start/stop/load features of Lingon. Overall I’d say it’s worth using Lingon in spite of its rough UI, but minimalists might prefer Launchd Editor.

Launchd For Developers

So far I’ve talked about how the system uses launchd for its services, and sort of alluded to the fact that you might come up with your own configurations to suit your personal needs. But developers should also take heed of launchd as a mechanism for automating tasks on behalf of users. One example of this is Noodlesoft’s Hazel, which uses launchd configuration rules to periodically check the status of “watched folders” and perform housekeeping tasks on them. By saving configurations to the user’s LaunchAgents folder, and then loading them with the launchctl command-line tool, it pushes off timing responsibility to launchd. Nifty!

Summary

Launchd is a super addition to Mac OS X, and a stellar example of Apple’s commitment to improving Unix for both end-user and architectural benefit. Hopefully if you hadn’t gotten a chance to take a close look at it before, you’ll be encouraged to do so after reading this article.