For the upcoming Gatekeeper feature in Mac OS X 10.8, Apple will make it easy for customers to prevent software from running that has not been digitally “signed” by developers with a certificate from Apple called the Developer ID certificate.

Many developers already choose to sign software using self-generated signing certificates. I wrote many years ago about the value of doing this for applications that access information from the user’s Keychain. With an unsigned application, every new version of your app needs the user’s approval to access the same keychain item (e.g. a blog password). But with a signed application, Apple allows a new version of the app from the same signature authority to continue accessing the keychain without nagging the user.

You might think that switching to Developer ID would be as simple as switching from the self-generated certificate to a certificate from Apple. I certainly assumed it would be, and so apparently did The Omni Group when they shipped a version of OmniFocus signed with their Developer ID certificate.

But there’s a catch. As The Omni Group’s own Ken Case pointed out on Twitter, simply substituting the Apple certificate will lead to code signature validation failure on 10.6 and earlier systems:

OmniFocus for Mac v1.10.1 is now available for those who were running into the keychain bug in our Developer ID builds on pre-Lion systems.

— Ken Case (@kcase) March 16, 2012

What is the cause for this failure? I asked Ken and he kindly went into more detail over Twitter. I decided to elaborate on his response and to do some more research of my own. I’m sharing a summary of my findings here for the reference of other developers.

Identity Confirmation

Every item in the keychain has individual settings for how it may be accessed in the future. The desirable Keychain behavior of allowing an app continued access to keychain items painstakingly documented in an Apple tech note, but the gist of it is that after gaining approval to access a keychain item from the user, an app will be allowed to continue accessing that item if it is in fact the same app it claims to be. The thinking here is, if the user indicated to the system that she trusts MarsEdit, then any future application that can prove it is also MarsEdit should inherit that same trust.

An app is determined to be itself by meeting its designated requirement (DR). The DR is the list of policies, as specified within the code signature, for determining the authenticity of the app. The code signing process generates a default DR that is suitable for most applications. You can use the codesign command-line tool to examine the DR of any signed application on your Mac. For brevity, I’m omitting all output here except the DR:

% codesign -d -r- /Applications/MarsEdit.app

designated => identifier "com.red-sweater.marsedit" and
certificate root = H"3a3292a0f7b9b5f5492d956d8f561621fe446e51"

This makes sense. The app should be considered to be “MarsEdit” if it has the right bundle ID, and is properly signed by a certificate whose trust chain leads to a certificate with the listed SHA1 fingerprint. In this case, the certificate listed is the self-generated certificate I made a few years ago for Red Sweater’s apps. Since the certificate is embedded into the app by the code-signing process, the resuilting app is imminently verifiable and will “satisfy its own DR” on any Mac:

% codesign -dv /Applications/MarsEdit.app

/Applications/MarsEdit3.4.4.app: valid on disk
/Applications/MarsEdit3.4.4.app: satisfies its Designated Requirement

I never had a role in choosing the DR for MarsEdit. It was the default policy installed into my app by the codesign utility. I didn’t care about the DR, I just cared that users weren’t nagged about accessing the keychain on successive updates to my app. Thanks to Apple’s sensible defaults, it did in fact “just work.”

A Betrayal Of Trust

When I switched over to Apple’s Developer ID certificate, I continued to assume that Apple’s default signing behaviors would lead to reasonable behavior with regard the Keychain. And, for the most part, it does. Here is the DR for a version of MarsEdit that was signed with Apple’s certificate:

designated => identifier "com.red-sweater.marsedit" and 
anchor apple generic and
certificate 1[field.1.2.840.113635.100.6.2.6] /* exists */ and 
certificate leaf[field.1.2.840.113635.100.6.1.13] /* exists */ and
certificate leaf[subject.OU] = "493CVA9A35"

Here the requirements take a little more decoding to make sense of. I’m learning as I go here but I’ll try to translate. The first line is pretty obvious, specifying the bundle ID. The other lines proceed to state requirements of the certificates in the signing trust chain. It identifies an anchor, certificate 1, and a leaf, so it expects three certificates in the chain in all. And specifically for these certificates it expects:

1. The root certificate must be an “apple generic” certificate. This is a special certificate designation specifically documented by Apple as being any certificate “for code signed by Apple, including code signed using a signing certificate issued by Apple to other developers.” I take this to mean that the root must either be an Apple certificate or a Developer Developer ID Certification Authority certificate that Apple provided to me for signing my app.
2. Certificate 1 (the certificate in the middle of the trust chain) must have a particular field, which I was able to verify on my Mac exists in the “Developer ID Certification Authority” certificate.
3. The leaf (my private Developer ID certificate) must have a particular field, which I again verified exists in my certificate.
4. The leaf must also have a specific subject value of “493CVA9A35”, which I take to be the unique ID Apple assigned to me for my certificate.

In short, the application must be signed with my specific Developer ID certificate, that certificate must be trusted by a certificate that has a special field indicating it’s a Developer ID authority, and that certificate must be trusted by Apple.

You can examine the certificate trust chain for a signed app with codesign:

% codesign -dvvv MarsEdit.app

Executable=/Volumes/daniel/Desktop/newmars/MarsEdit.app/Contents/MacOS/MarsEdit
Identifier=com.red-sweater.marsedit
Format=bundle with Mach-O universal (i386 x86_64)
CodeDirectory v=20100 size=4953 flags=0x0(none) hashes=241+3 location=embedded
Hash type=sha1 size=20
CDHash=f20bbe82230831579e301989faf2956a0142e838
Signature size=4234
Authority=Developer ID Application: Red Sweater Software
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Signed Time=Mar 17, 2012 9:01:47 PM
Info.plist entries=31
Sealed Resources rules=4 files=149
Internal requirements count=1 size=260

I’ve highlighted the certificate chain in green. In this case, the trust chain has my Developer ID certificate as the leaf, the Apple Developer ID authority a step removed, and Apple’s Root CA a step above that. This all meshes nicely with the designated requirement rules we outlined above. So what’s the catch?

The catch is that the intermediate certificate is not likely to actually exist on any customer’s Mac. Because that’s a special certificate provided by Apple to developers, it’s only likely to be installed in the keychains of other developers. So when the Keychain goes to validate the app, it does not meet the designated requirement, so it is not trusted to inherit the access to the keychain.

A little bit that I still don’t completely understand is that apps that are signed in this way are allowed access on 10.7, even if the intermediate certificate is missing. I did just a small amount of research into this, but I think this has something to do with the new “SystemPolicy” file located at /var/db/SystemPolicy. It’s an sqlite3 file that you can freely examine. It’s where the system stores information about all the Gatekeeper signed apps on your Mac, but also where it maintains a list of what I think are high-level security overrides. Examining the SystemPolicy file on my 10.7 machine, I find a couple interesting “authority” elements:

INSERT INTO "authority" VALUES(4,1,'anchor apple generic and
certificate 1[field.1.2.840.113635.100.6.2.6] exists and
certificate leaf[field.1.2.840.113635.100.6.1.13] exists',
1,NULL,0.0,'Developer Seed',NULL,0,NULL);

INSERT INTO "authority" VALUES(5,2,'anchor apple generic and
certificate 1[field.1.2.840.113635.100.6.2.6] exists and
certificate leaf[field.1.2.840.113635.100.6.1.14] exists',
1,NULL,0.0,'Developer Seed',NULL,0,NULL);

Since Apple’s security code is open sourced, you can actually poke around at libsecurity_codesigning and try to make some sense of it. I don’t want to get too sucked into this curiosity, but I was able to confirm that the items above specify “authority” policies for running applications, and for installing applications. I was also able to confirm these are included in the source code to libSecurity, so they weren’t added on my system as a result of anything I did.

The naming of those items, “Developer Seed”, is pretty interesting. I speculate that Apple knows that the certificate chain for Developer ID-signed applications will not validate, and so they added these high level overrides in 10.7 so that we can test our signed applications, and so that customers can run them without incident. The only problem is there is no such workaround on 10.6 and earlier systems.

Getting Back To Basics

In order for our Developer ID-signed applications to work as expected on 10.6 or earlier, the designated requirement has to be met. Since the odds of the default DR being met are low, we need to override Apple’s default DR and install a simpler one that self-verifies with rules more along the lines of the way our self-generated certificate did it.

As Ken Case pointed out in his tweets, you can do this by specifying the requirements information explictily on the command line:

% codesign -f --sign "Developer ID" 
  -r='designated => certificate leaf H"xxx" and
  identifier "com.red-sweater.marsedit"' MarsEdit.app

(Split into three lines for readability on my blog. It should all be on one line.)

This will sign the app using your Keychain-installed Developer ID certificate from Apple, but it will specify the given explicit DR. The xxx string is replaced by the SHA1 fingerprint for your Developer ID signing certificate, which you can find by examining the certificate in the Keychain Access app and looking at the very bottom.

Now, when the Keychain validates the designated requirement on either Mac OS X 10.7 or 10.6, it passes the test.

A Proper Solution

I have some concerns about working around the problem using this custom, dumbed down designated requirement. Sure, it’s no “dumber” than the self-generated signature’s DR, but what concerns me is whether Apple’s support for Gatekeeper in 10.8 will make assumptions about an app’s, ahem, Gateworthiness, based on some of those semi-magical certificate field checks in the problematic DR. If I’m correct in my guesses about how those SystemPolicy entries work, then there is already “magical” behavior happening in 10.7 for Developer ID signed apps. Who’s to say there won’t be continued magic going forward?

I’m not sure what the right fix is. Maybe it’s just this, dumbing down the requirements string. But it would be nice if Apple’s default code-signing requirements “just worked”…

Hey, wait. Why does this version of my app built with Xcode 4.3 “just work”? All of my shipping apps are currently built using Xcode 3 on a 10.6 machine. All of the examples above are based on analysis of an app that has been code-signed in that environment. But when I look at a version of my app built Mac OS X 10.7, and with Xcode 4.3, the requirements are a little different:

designated => anchor apple generic and
identifier "com.red-sweater.marsedit" and
(certificate leaf[field.1.2.840.113635.100.6.1.9] /* exists */ or
certificate 1[field.1.2.840.113635.100.6.2.6] /* exists */ and 
certificate leaf[field.1.2.840.113635.100.6.1.13] /* exists */ and
certificate leaf[subject.OU] = "493CVA9A35")

I’ve highlighted in green the part that is different from my old DR. Where it used to require that intermediate Developer ID authority certificate that is missing from most Macs, it now requires either that certificate OR that a field in my Developer ID signing certificate exists. This effectively restores the passing status of the DR on any Mac, whether it has an intermediate installed, or whether or not there are any alleged funky overrides going on in SystemPolicy.

So, my recommended fix either to use Xcode 4.3 to build your app, or mimic Xcode 4.3’s code signing behavior by stealing its codesign arguments (look in the build log after building on 4.3) and applying them verbatim in your build process.

I’ve learned a whole lot about code signing and certificates today. If you got to the bottom of this post, I hope that you have too. In any case, this should give you a good idea of a situation you need to be aware of, and test against: if you are signing your app with Developer ID, and need to run on 10.6 or earlier, make sure the Keychain is still kind to your app.

Update: Now With More Facts

My conclusions above were based on some false assumptions on my part about how designated requirements are synthesized and how the certificate validation process actually works. Based on some valuable comments from Chris Suter and Evan Schoenberg I think it’s fair to say the root problem here is that “default” DR synthesis is actually just a placeholder that allows the host system to insert whatever default it thinks is suitable for the app. This explains a weird thing that I thought may have just been a bug: that the designated requirement showed up differently on 10.6 than on 10.7. It turns out this reflects the fact that if you don’t specify a specific DR, the system will do it’s best to make sense of your app (maybe based on its signatures to some extent?).

So, I’m not 100% fact based yet, but I think the new moral of the story is: if you need to run earlier than 10.7, specify your DR literally because the default DR generated for a Developer ID certificiate will not work perfectly on 10.6 and earlier.

Update: Now With Even More Facts

My friends at Fetch Softworks went even deeper on this issue, discovering that Apple’s own code-signing defaults for Mac App Store submissions offer a useful clue for a better, more generalized designated-requirement for Gatekeeper-signed apps. In a nutshell: you want to key the designated requirement off of your developer ID being listed in the certificate, rather than a precise fingerprint of a certificate.

I just posted MarsEdit 3.4.4 to the MarsEdit home page for direct-purchase customers, and am submitting to the Mac App Store for release as soon as it’s approved by Apple.

This release is important for MarsEdit customers who use the integrated Flickr browser to insert images into blog posts. In particular, if you tend to use the “Medium” size embed, this update is critical to restore the app’s ability to insert photos of that size.

What happened is Flickr added some new image sizes: “Large Square” and “Small 320”. MarsEdit didn’t know what to make of either of these, and ended up calling each of them “Medium” too. So customers were seeing three medium items in the menu, and it was only ever possible to select the first one.

This update fixes the problem at hand and further fixes the way MarsEdit handled Flickr embed sizes so that should they add additional items or rename the existing ones, MarsEdit should adapt more elegantly.

Dave Winer writes about the intermingling of of tech and advertising (via Brent Simmons):

The tech industry has been absorbed by the ad industry, and vice versa.

However, there is, imho, still room for a tech industry that is not merged with the ad industry.

I’ll take this a step further: advertising is on the way out. Technology loathes a middle-man, and advertising as an industry is the king of all middle-men. The purpose of advertising is to connect customers with companies, so as to facilitate a transfer of money in exchange for goods or services. As time goes by, customers and companies will be more and more capable of achieving this on their own.

In the history of the world so far, there has been considerable opportunity for advertisers to misguide customers, and to lure their money toward products or services that can be framed as perfect for them, even when they are not. That’s the art and the holy grail of advertising. But going forward, technology will offer customers and companies the tools to connect effortlessly, optimizing for compatibility without the help of the bogus, outdated advertising system.

Most of us base purchasing decisions on vague hunches derived from a mix of advertising influences, word-of-mouth, and the relative trendiness of a product. But more and more as customers we are cutting out the advertising middle-man, in favor of systems based on education and trust. Amazon is a good example of this. With the notable exception of their Kindle line of products, they have little concern about which products their customers buy. It only matters that they buy things, and that they buy things often. They provide detailed product information, and allow honest, often scathing reviews. The goal is for customers to make self-serving decisions. In this case, defying the advertisers’ best interests is in Amazon’s best interest as well.

Extrapolate the technology-assisted consumption process out over the next 10, 50, 100 years, and I have a hard time imagining a meaningful role for conventional advertising. If I search Google for “lawnmower,” it’s not interesting that some tractor company has paid Google for the privilege of putting their brand’s information at the top of the list. At some point in the future, customers will assume that companies who choose to advertise conventionally are afraid of the outcome when consulting various self-empowering resources. Where am I more likely to search for “lawnmower?” If I want to know what a lawnmower is, Google. If I want to know which lawnmower to buy? Amazon, or another site that strives to empower customers, not advertisers.

I do worry about what happens to some of our beloved, advertising-driven services. We’ve all grown accustomed to the subsidization of news reporting and analysis. In recent decades, advertising has crept further into our lives, even subsidizing municipal infrastructures such as public transit. What impact will the end of advertising have on these important services?

In the old world, technology for connecting customers directly to companies did not exist, so companies were satisfied in buying advertising. It is tool that serves to expose customers to the concept of a product, and to crudely attempt to educate them about the suitability of the product for their purposes.

In the new world, mass-exposure will be replaced by social networking, and education will be not only replaced by, but massively bolstered by trusted systems such as Amazon’s review database, Consumer Reports, and other much better stuff that is presumably coming in the future. Presumably? It has to be coming, and it has to be better, because everything’s riding on it.

Everything’s riding on it because this is the salvation for current advertising-subsidized industries. They will shift from being exposure-focused, to education-focused. Amazon, Apple, and many others already offer affiliate systems that reward anybody who can produce a sale. The old way to produce a sale is by blasting customers with unwanted information until you happen upon something that sticks. The new way is to provide customers with a trustworthy, opt-in system for determining what’s best for the customer. To stay alive in the changing world, these subsidized industries will change their business plans, or go out of business.

Earlier today, before I even followed Brent’s link to Dave’s piece, I read this short, thought-provoking essay, allegedly by the graffiti artist Banksy. Here is an excerpt that I think is pertinent to my predictions here:

They have access to the most sophisticated technology the world has ever seen and they bully you with it. They are The Advertisers and they are laughing at you.

…

You owe the companies nothing. Less than nothing, you especially don’t owe them any courtesy. They owe you. They have re-arranged the world to put themselves in front of you. They never asked for your permission, don’t even start asking for theirs.

As a businessman who is dedicated to my own commercial success, I embrace the challenge of getting the word out to potential customers. I will shout my message from the rooftops to anybody who will listen. But only to those who will listen. I don’t want to annoy, interrupt, cajole, or appeal to a customer’s feelings of inferiority. I don’t want a customer to choose my product over a competitor’s unless it’s better for them. In short: I want a future without advertising, where my products sell themselves through word-of-mouth and through trusted systems that educate customers about making the right choice for them. Not what’s right for companies, and certainly, so long as they’re still around, not what’s right for advertisers.

MarsEdit 3.4.3 is now available for download from the MarsEdit home page and as an update in the Mac App Store.

I highly recommend this update for anybody who has installed the Safari 5.2 beta release on Lion, or who is running a developer preview of a future OS X release.

MarsEdit 3.4.3

• Fixes to improve with the Safari 5.2 beta
• Fixes to improve with a future OS X update
• Prevent a possible hang while parsing Lightroom libraries