Update. Well I am a complete idiot. Of course I should have assumed that if I wanted to build something out of Darwin, other people in the past have wanted the same thing. I’m a fool for not noticing this page at opendarwin.org. The “darwinbuild” tool basically “follows the dark dependencies” for you. I haven’t confirmed that it can build CFNetwork yet, but I’ve started it on its way, and I have little reason to believe that it will fail. It sure is downloading a bunch of the same junk I (painfully) did.

Perhaps my only criticism is that it appears to be downloading way more than necessary. Naturally, it must be assuming a raw system with no developer tools installed. It’s downloading things like gcc and gnumake. Maybe there’s some way to coerce it into only downloading the “not installed by Xcode” bits.

Update 2: It didn’t succeed. Probably just fine-tuning, but I suppose for the time being there may be some value in my crazy efforts shown below.

Update 3:After some tweaking, I got CFNetwork to build with darwinbuild. Basically, the dependency tree they have for CFNetwork is good but not perfect. I will get in touch with them to see if I can get the correct dependencies added to the tree.

Before discovering darwinbuild, I managed to get CFNetwork built on my own. But it was a painful experience. The long and miserable story below is enclosed for historical snickering and contempt:

I mentioned recently how Apple’s open source resources can be quite helpful in “cutting to the chase” in some debugging scenarios. A major obstacle for most of us, however, is the fact that most of these open source projects are set up with peculiar Apple-internal build variables, conventions, etc. I have a hard time building these things, and I used to work in that environment! (At Apple, you just get used to having a system with “internal headers” installed across the board. Sigh. Because most Apple employees work in this type of environment, it’s no wonder the open source projects are largely unbuildable by us normal folks.)

But I don’t like being limited in this way, so I’m resolving to take matters into my own hands. I want to have these projects at my easy disposal, so I’m going to figure them out. For every project I need to build that “takes signficant work,” I’ll add a blog entry here detailing what was required. Hopefully this will serve as a convenient “google destination” for others who could benefit from getting something built, but don’t have the time to follow the endless tunnels needed to reach the destination.

Because I’m debugging some funny behavior in CFNetwork, a project I worked on briefly while I was at Apple, I’ll start there.

CFNetwork is quite a small project, but it’s much trickier to build than you might guess. It includes a number of dependencies which are not part of your standard “Xcode install” development environment, as well as some build peculiarities that we’ll have to work around. We’ll coax the project into thinking our dingy little bachelor pad is every bit as luxurious (?!) as the environment it’s gotten used to at Apple. Many of the dependencies are just on private header files that need to be collected from various open source projects, but CFNetwork also links against some funny security frameworks that need to be installed into /usr/local.

CFNetwork – Step by Step:

1. CoreFoundation framework with PrivateHeaders. The open-source flavor of CoreFoundation (368.25) includes much, but not all, of the CoreFoundation binary that shipped in 10.4.4. It is therefore unsuitable as a “drop-in” replacement for the copy of CoreFoundation that came with your copy of Mac OS X. Nonetheless, it can be useful to build if, for example, you want to use parts of CoreFoundation in an open source application on another platform. Or perhaps you want to play around with tweaking the sources for some common collection classes, to see if the performance can be improved. In our case, building it gives us a convenient copy of the framework with private headers in tact, and it’s actually easier to do this than to figure out how to get the private headers out in any other way.

   CoreFoundation does not, thankfully, have dependencies on other Apple-internal sources or headers. There is just one minor issue to contend with if you’ve got gcc 4.0 set up as your default compiler (you probably do unless you changed it with gcc_select). CoreFoundation’s makefile passes the -Wno-precomp option to the compiler, which is no longer recognized by gcc 4.0. To work around this, you could probably remove the flag from the build options, but I just “went with the flow” and changed the compiler to 3.3. I did this by finding the following text in framework.make:

   ifeq "$(PLATFORM)" "Darwin"
   CC = /usr/bin/cc
   else
   

   And changing “cc” to “gcc-3.3”. After making this change, you should be able to type “make”, cringe at a few dozen build warnings, and eventually see the reassuring “Done!” feedback. We don’t really care if the binary is perfect – so I’m not going to fret about the warnings or whether I should have used gcc 4.0 for better code. The built-framework resides, by default, in the following directory:

   /tmp/CoreFoundation.sym/CoreFoundation.framework
   

   You’ll see that inside this framework exists the usual Headers directory as well as a PrivateHeaders directory containing “top secret” stuff!

   The easiest way to accommodate this dependency is to point CFNetwork at the same “SYMROOT” (where built objects go) as CoreFoundation. Since the CFNetwork build includes its own SYMROOT as part of the framework search path, building to a common directory as CoreFoundation will cause it to find the custom CoreFoundation you built above. The SYMROOT can be specified on the command line, so for starters, instead of simply typing make in the CFNetwork source directory, type the following:

   SYMROOT="/tmp/CoreFoundation.sym" make
   

   If this “define a variable on the commandline” trick doesn’t work with your shell you’ll have to find another way of getting the definition into the Make context. If it does work, you’ll find that your build is still failing, but at least it’s finding the required CFPriv.h and associated headers.

2. Asynchronous NetDB. CFNetwork takes advantage of the asynchronous getaddrinfo calls that are linked into the libSystem library on OS X. Unfortunately, the header files needed to compile against these, netdb_async.h, is not included. I discovered, with the help of Google and Ian Lister, that these header can be obtained by downloading LibInfo (222.1) and doing a “make installhdrs”. I didn’t really want to soil my pristine system headers, so I just copied the pertinent header file over to the CFNetwork source directory:

   cp lookup.subproj/netdb_async.h ../CFNetwork-129.9
   

   Don’t feel like downloading the whole project just to grab the header? Here’s a direct link.

   In the longer term, I could see the value of putting all this junk into a centralized “Darwin SDK” (does it already exist? if so, I’m wasting a lot of time here!). It would be cool to have an SDK among the others for 10.3.9, 10.4, etc., that specifically included all the extra junk you use to build common Darwin projects.

3. CDSA Security Pieces. Yet more crummy downloads you’ll have to make from the Darwin source repositories. CFNetwork relies on headers from libsecurity_utilities (25) and libsecurity_cdsa_utilities (16). The latter requires the former to be built and installed first. It also requires some private headers from the Security framework (25966), which in turn requires that you essentially download, build, and install all the various libsecurity_* projects. Ugh. Are you noticing a trend here? I just want to get the minimum I need to build this thing. So I’m going to take shortcuts left and right.

   To the Security folks’ credit, they include Xcode projects for their sources. Unfortunately, they are no more “self contained” than the other projects I’ve attempted to compile. Another rat’s hole! Let’s start with libsecurity_utilities:

   To build this project, I am going to tweak a few things in the project, and then build from the command-line with the xcodebuild command. I’m using Xcode 2.2, so I had to upgrade the project to the “.xcodeproj” format before proceeding.

   This project shared similarities with CFNetwork. It relies on CoreFoundation’s private headers, and it also seems pre-determined to use gcc 3.3. I’m going to point it at my existing “CoreFoundation.sym” magic directory, since it happens to be there and conveniently serving CFNetwork at the same time. I’ll also specify a gcc version on the command line using the Xcode build option GCC_VERSION:

   xcodebuild -target libsecurity_utilities GCC_VERSION=3.3 FRAMEWORK_SEARCH_PATHS=/tmp/CoreFoundation.sym/
   

   This gets me “almost there”. For some reason when CFPriv is brought in here, it objects to the “non-umbrella” form of including headers directly from CarbonCore. I can’t remember how to fix this correctly, so I just take the easy way out. I add a symbolic link to my magic directory:

   cd /tmp/CoreFoundation.sym
   ln -s /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework
   

   The only problem now is the post-install script phase from Apple that pops the build products into the correct system path. I’m using “per-deployment build folders” in my Xcode preferences, but this script isn’t expecting that. So I open up both Script phases and replace all instances of “SYMROOT” with “BIULT_PRODUCTS_DIR”. Now it will always point at the correct directory, regardless of how the preference is set. With that, the build works, and I’m ready to install the resulting libraries:

   sudo xcodebuild -target libsecurity_utilities GCC_VERSION=3.3 FRAMEWORK_SEARCH_PATHS=/tmp/CoreFoundation.sym/ BUILD_VARIANTS=normal DSTROOT=/ install
   

   Yes, I’m taking a whiz on my pristine installed system, but in this case I don’t feel so concerned by it since the stuff is all neatly confined to /usr/local/SecurityPieces.

   Are you still with me? We’re almost there! If you made it this far you deserve a medal. Now we’ve got to compile and install the libsecurity_cdsa_utilties project. Remember the aforementioned Security framework private headers? This is where those come into play. Specifically we need “SecTrustPriv.h” and “SecKeychainItemPriv.h”. These both come from the libsecurity_keychain (25886) project. I’m getting pretty tired of playing this game, so I’m just going to make a copy of the entire Security framework as it shipped on my machine, put it in my “magic” CoreFoundation.sym folder, and drop everything that looks like a private header in:

   cd /tmp/CoreFoundation.sym
   cp -r /System/Library/Frameworks/Security.framework ./
   mkdir PrivateHeaders
   cp ~/Sources/ExternalSources/darwin/libsecurity_keychain-25886/lib/*Priv.h PrivateHeaders
   

   I’m sorry to report you’ll also need checkpw.h, from a little rinky dink project called libsecurity_checkpw. Just grab it directly and drop it into your magic Security framework.

   Aside from the header dependencies, most things about this project look similar to the previous, so I try a slightly modified xcodebuild line, being careful to point it at the magic directory, where my “private headers” security framework now lives:

   sudo xcodebuild -target libsecurity_cdsa_utilities GCC_VERSION=3.3 FRAMEWORK_SEARCH_PATHS=/tmp/CoreFoundation.sym/ BUILD_VARIANTS=normal DSTROOT=/ install
   

   Success! Well, in a really pathetic, sort of almost dead from frustration kind of way. It turns out that the particular headers required by CFNetwork are not included in the libsecurity_cdsa_utilities project. They’re included in libsecurity_cdsa_utils (no “ities”), instead! (13). Oops, what a misread. But the “utils” project depends on the “utilities” framework being built and installed, so our time is not wasted. Just go through the same rigamarole as above one more time:

   sudo xcodebuild -target libsecurity_cdsa_utils GCC_VERSION=3.3 FRAMEWORK_SEARCH_PATHS=/tmp/CoreFoundation.sym/ BUILD_VARIANTS=normal DSTROOT=/ install
   

   It ended with no obvious error but a “Build Failed” message. Whatever. Ask me if I care? It got far enough to put the required header where CFNetwork wants it. At this point I’m starting to think I should have just downloaded all the “libsecurity” items and figured out how to build them kosher. But the important thing is, it doesn’t matter anymore. I’ve satisfied CFNetwork, and it’s time to move on to the next hurdle.

4. JavaScriptGlue (417). CFNetwork’s ProxySupport.c depends on something called JavaScriptGlue. I’m assuming this is so CFNetwork can support the javascript-based custom proxy specification files. Cool! I’m not really sure I need it for my testing, but heck I’ve come this far I might as well cover all the bases. Besides, it might be easier to placate it than to try to cut out functionality.

   It turns out that JavaScriptGlue relies in turn on JavaScriptCore and who knows what else. Fortunately I’m getting a little more desperate and a little less curious. Maybe it means I’m also getting smarter. I don’t have to BUILD the damn thing, I just have to get its headers installed. I pull a similar move to the Security framework trick I did earlier, copying JavaScriptGlue.framework from /System/Library/PrivateFrameworks to my “magic” CoreFoundation.sym directory. Then I just pop the a Headers folder in there containing every file from the JavaScriptGlue project that ends in “.h”:

   cp -r /System/Library/PrivateFrameworks/JavaScriptGlue.framework /tmp/CoreFoundation.sym/
   cd /tmp/CoreFoundation.sym/JavaScriptGlue.framework
   mkdir Headers
   cp ~/Sources/ExternalSources/darwin/JavaScriptGlue-417/*.h ./Headers/
   

5. Security Framework PrivateHeaders. Oh! Security again! Didn’t we already deal with that about 15 pages ago? We only dealt with the parts needed to get the funky cdsa frameworks building. The problem with the Security framework is it’s a weird amalgamation of code from all those libsecurity* projects. The headers come from all across those projects, too. We grabbed some of the headers but not all of them. So our magic Security.framework isn’t quite magic enough. CFNetwork wants SecureTransportPriv.h, which I found in the libsecurity_ssl project. Grab it directly and drop it in your magic Security.framework. You’ll also need ASN1 headers from libsecurity_asn1 (9). It’s brute-force time – just grab all the *.h you can find and drop them into Security.framwork

Summary

Well, there you have it. How to build CFNetwork in just a few easy steps. In retrospect, I could probably do some things a bit faster and more logically. But the problem is, when you’re looking at a source project from outside the company, you don’t know what you’re going to need ahead of time. You start building and just keep hoping that every tweak or accommodation will be the last one you need to make.

My experience getting CFNetwork to build informs an opinion that people interested in building it regularly should bite the bullet, download and figure out how to cleanly build their own copy of all the Security stuff. Just do it once and start building your own private “Darwin SDK.” I don’t have the time or energy to do that now, but you can bet I made a copy of my “CoreFoundation.sym” folder. Next time I need to build something like this, I’ll be starting with magic in hand. If somebody does start putting something together, maybe it will be shareable under the same license as all of the projects from which it was composed (probably APSL).

Since many of the “private headers” are available via the variety of open source projects, it’s a shame Apple doesn’t just make such an SDK available. They have an internal installer that puts Apple engineers in good shape for this stuff, so they would have something to start with in building such a thing.

(If they do, by some chance already do this, please don’t tell me about it. Give me a week or so to feel like a badass for compiling a CFNetwork of my very own.)

Now, about that bug…

This entry was posted to Apple, Debugging, Programming. Comments are closed. Thanks for reading!