Lion is the first operating system to require, and to fully take advantage of, 64-bit addressing modes in the Intel chips that power Apple’s Macintosh computers. One of the side-effects of this is that every object identifier in Mac OS X’s Cocoa programming framework (typically an address in memory), is now twice as long as it was in a 32-bit environment.

Apple has apparently taken advantage of the 64-bit runtime in Lion by optimizing the Objective C runtime itself to use some of these extra bits for, shall we say, clever purposes. Bavarious describes an optimization through which Apple is able to replace previously full-fledged opaque objects such as NSNumber with an object-placeholder that exists entirely as the 64-bit “object address” itself. This means that, for a wide range of “simple” objects, no additional memory allocation is required, and no retain/release memory management is required for the “object.”

The trick relies on a implementation detail of the system, that allocated blocks of memory will always be aligned at 16-byte offsets into the address space. This leaves a bunch of numbers that can be represented in 64-bits, that cannot reasonably be assigned to any other object. To understand this practically, imagine that your neighborhood’s postal addresses are all assigned at offsets of 10: 30, 40, 50, etc. A clever postal service could institute an addressing system that uses an “invalid” address such as “31,” to perhaps mean “deliver to 30 with expedited afternoon delivery.”

Cleverness like this with encoding extra information in memory addresses is a time-honored tradition. I recall the days of 24-bit addressing on classic Mac OS, where Apple, and many 3rd party developers, observed that the high 8 bits of a typical memory address could be tweaked and used to store additional information, because the system would never reference those bits when resolving a particular address.

In those days, using those extra bits turned out to be a pretty significant headache when 32-bit addressing ultimately came along, and lots of code had this “crufty” treatment of addresses to clean up. Perhaps it is a memory of situations like this that caused Jon “Wolf” Rentzsch to comment in his bookmarking of the above-referenced blog post:

“Every tagged pointer has its lowest bit set, hence tagged pointers are odd integers” Strikes me as a really bad idea. [Emphasis Mine]

But the difference now, in this scenario, is the “cute hacking” is all being done by a central power, with and in terms of opaque objects that only Apple has the authority to change. I think this is a really clever hack that will undoubtedly lead to some serious performance gains in Lion and beyond. It’s hard to imagine specific outcomes that will make Apple regret adopting this strategy. In the worst case scenario, an addressing system of future Macs will not leave any “spare” bits to be exploited, so the runtime will simply revert to its previous behavior.

One of my favorite new features in Lion is a completely revamped “FileVault”, Apple’s brand-name for encryption techonologies that protect the data on your disk from eavesdroppers, should the disk be lost or stolen.

In Mac OS X 10.6 and earlier, FileVault was a feature that only affected your home directory. In OS X Lion, it applies encryption at a very low-level, encrypting an entire volume of your disk at a time, and keeping it encrypted as you use it.

I was able to enable FileVault for my boot volume with relative ease, using the Security & Privacy preference pane in System Preferences. However, the UI for this is pretty limited, and notably, it only allows you to protect the computer’s startup disk.

The way I have my Mac configured, most of my sensitive data is not on the startup volume, but is instead on a second partition called “Data” where I keep my home directory, media files, etc.  Apple’s Disk Utility allows you to erase and reformat a volume as encrypted from scratch, but what if you want to migrate a volume in-place, the way the system does the boot volume? You’re not completely out of luck.

OS X Lion ships with a low-level technology called “core storage,” which is used to facilitate a wide variety of disk-maintenance functionality, including whole-disk encryption. To get a quick look at what core storage supports, type “diskutil cs” at the Terminal command line. For a more in-depth look, type “man diskutil” and search for the core storage command documentation.

Important: This is the part of the blog article where I warn you to be very careful before proceeding. The diskutil command is capable of doing incredibly destructive things to your disk and to your data, so you should feel confident before doing anything that you have a 100% reliable backup of your data.

To convert an arbitrary volume to Lion’s whole-disk encryption, you use diskutil’s core storage “convert” command, and provide a passphrase. For example, if you have a volume called “Data” attached to your Mac, you would run something like this from the command line:

% diskutil cs convert /Volumes/Data -passphrase '[yourPasswordHere]'

Warning: at least one person has run into an issue where the passphrase was set to something unexpected because of characters such as ! and $ being interpreted by the shell before being passed to the tool. One more good reason to make sure you have a backup before messing with any of this stuff.

The “diskutil cs convert” command kicks off a conversion process similar to what the System Preferences panel does when allowing you to convert your main startup volume to core storage with encryption. At any time during the conversion, you can use the diskutil command again to see status of your volumes, whether they are encrypted, not encrypted, or in-progress while converting.

% diskutil cs list

You’ll see a bunch of information, but search carefully for the named volume (e.g. “Data”) that you just started the conversion process on. You’ll find a line starting with something like:

Size (Converted):

This shows you what the progress in the conversion is. From time to time, check this manually, to see how far along things have progressed.

Caveats

In addition to the major admonition above to backup your data carefully, you should also know that after you have converted a volume, it seems to be in a sort of provisionally encrypted state where it’s still being treated by the running OS as a “native volume” although it’s been converted and is ready to be treated as a “core storage” volume. I have to confess I don’t really understand it 100%, but it seemed like a really good idea to me to restart as soon as possible after the conversion is complete.

But before you restart, bear in mind that there appears to be a bug in the login process that will prevent a user whose home directory is on an encrypted (“locked”) secondary volume from being able to log in. It seems that whatever logic Apple applies to unlock volumes at login time is not applied early enough to allow the actual login to occur. This means that if you converted your secondary volume like I did, and it contains your home directory, you won’t be able to login.

For this reason, make sure that you have a valid account to log in to whose home directory is located on the main startup volume. In my experience, the process of logging in to this main-volume account will prompt the system to ask for the secondary volume’s password in order to unlock it. Once the secondary volume is unlocked, you can log out and log back in to your regular account, with the home directory on the secondary volume.

This bug is pretty annoying. Hopefully this is something that Apple will get fixed soon, and it may be for bugs like this that they haven’t enabled full-disk encryption as a full-fledged user-facing feature of the operating system. In the mean time, if encrypting your data is important to you, I hope these instructions and caveats will serve you well.

Update: Not surprisingly, this topic is covered in some detail in John Siracusa’s Lion review.

Today Apple released OS X Lion, aka Mac OS X 10.7, the latest in the decade-long run of incredible updates to Mac OS X.

I encourage everybody to upgrade to Lion. I’ve been running it for months in pre-release form, and even while the bugs were being ironed out, I found the experience of using it to be (mostly) superior to running 10.6.

If you want a more detailed analysis of Lion’s features, sit down with a tall glass of your favorite beverage, and read John Siracusa’s famously detailed review. If you’re looking for a quicker overview, check out Jason Snell’s Macworld review, or my developer buddy Matt Gemmell’s guest appearance at the Guardian UK.

Supported Systems

We have known for some time that Mac OS X Lion would drop support for a number of Macs. In particular, all Macs that do not support Intel’s 64-bit memory addressing are not qualified to install the OS. But I was curious to know a bit more about how Apple makes this determination when, say, a user is browsing the Lion “product” in the Mac App Store.

As I described in an an earlier post, you can learn a lot about a product in the App Store by inspecting the HTML that makes up the product page, and by manually loading the references resources. In the case of Lion, the “Buy Now” button contains a number of parameters intended to inform the Mac App Store client about whether a sale should be allowed, and what some of the conditions of sale are. In the case of Lion, here is the raw HTML:

<button is-rental="0" dk-id="30" is-pre-order="0" preflight="http://a5.mzstatic.com/us/r1000/
065/Purple/cd/82/29/mzm.tvvarwmu.pfpkg" item-name="OS X Lion" bundle-id="com.apple.InstallAssistant.Lion" version-string="10.7" buyparams="productType=C&amp;price=0&amp;salableAdamId=444303913&amp; pricingParameters=STDRDL" large-icon="http://a4.mzstatic.com/us/r1000/083/Purple/00/00/00/ lion.170x170-75.png" is-install-button="0" is-update="0" check-is-osx-server="http://r.mzstatic.com/static/isOSXServer.pfpkg" is-free-download="0" adam-id="444303913" metrics-leaf="1" metrics-loc="Buy" class="button-area" aria-label="Install, OS X Lion, Free">    <span class="price">Install</span>    <span class="left-cap"></span>    <div class="inner"><span>Install</span></div>    <span class="right-cap"></span>  </button>

If you load the “preflight” URL content from the command line, and unarchive it with the “xar” tool, you get a few files including a file called “Distribution”:

% mkdir TestFolder; cd TestFolder
% curl -O http://a5.mzstatic.com/us/r1000/065/Purple/cd/82/29/mzm.tvvarwmu.pfpkg
% xar -x -f mzm.tvvarwmu.pfpkg
% more Distribution

Inside you will see a number of constant constraints such as hostArchitectures=”x86_64,i386″, but also more nuanced tests that, for example, test the current computer’s motherboard model ID with a fixed list of allowed values. Presumably this will prevent the App Store from allowing (without some hacking) downloading Lion and installing it, for example, on a Dell PC that happens to meet the ostensive hardware requirement.

Perhaps the most heartwarming bit of code in this relatively complex document is a small function call from the start of the test for whether the install should be allowed on this computer:

function isSupportedPlatform(){

        if( isVirtualMachine() ){
                return true;
        }

It has already been announced that Lion 10.7 will finally support virtualization, e.g. with the use of products such as VMWare or Parallels. This install-time test seems to drive home how true that is.

I, like most, have been perplexed by the persistent rumors that Apple is gearing up to release a third iPad this fall. For those of you who don’t follow the product lineup closely, the iPad 2 was released just earlier this year, so Apple releasing a successor to that product so soon after would be a deviation from their more usual approach of shipping major updates to a product every 1 or 2 years.

I like Manton Reece’s take on recent speculation that it won’t be an iPad 3 so much as an iPad Pro:

This isn’t a replacement for the current iPad; it’s another layer to the product lineup. And like the awkwardly-named iPod Photo from 2004, I bet the iPad Pro is meant to be temporary.

But I disagree about the temporary nature of such a high-end counterpart to the iPad. Apple likes to segment the market and then keep it that way. There are four completely different classes of iPod for sale in the Apple Store, and iPhones that range in price from $49 (with a contract) to $649.00 (unlocked).  Mac Minis currently start at $699 while the beefiest Mac Pro starts at $4,999.

Apple’s segmentation seems to achieve the classic goal of taking a little or a lot of the customer’s money, depending on how much they have to spend. But many folks are inspired to buy in at several price-points for a single product line. I own a Mac Mini and a MacBook Pro. An iPod Nano and an iPod touch. An iPhone 4, and, well, my wife takes good care of my iPhone 3G. I also own an iPad. Will I buy an iPad Pro if it comes out this fall? If it puts my iPad 1 to shame (which the iPad 2, frankly, did not), then yes, I’ll probably buy one of those too.

I’ll be surprised if Apple doesn’t take the same approach with iPad that they usually do: the low-end unit is always surprising powerful for the price, but outdone by some whiz-bang innovations at the high end. No, they don’t apply this strategy to all of their products, but this isn’t some hobby for them. As Jony Ive says in a recent commercial, the iPad “defines an entire category.” Let’s see how Apple intends to fill that category out.