Magical Code
January 7th, 2006Back in my first days at Apple, I was privileged to work with brilliant engineers (and a few idiots, lest I get too nostalgic). One of these brilliant types, who I might go so far as to describe as a mentor, had a mantra he’d throw out when anybody started whining about the difficulty of programming something. Without listening to the details, he’d sigh and give a contemptuous reply: “It’s only code!”
When I arrived at Apple I was one of the whiners. I thought there were at least three types of code in the world: easy, difficult, and magical. I thought of myself at that time as incredibly good at easy code, and perhaps squeaking by with some panache at the difficult stuff. The magical stuff – device drivers, etc. – I simply hoped nobody would ever ask me to do. By the time I left Apple, I had learned that my mentor was essentially correct.
In fact, his mantra was just a bit simplistic. There is easy code, and there is hard code. But there is no magical code. Furthermore, the ways in which people typically classify code into easy and hard categories is completely backwards. Most average programmers tend to think of things like device drivers and operating system scheduling as “hard,” while something like writing a word processing application is “easy.” In fact, the low-level code that most people are so afraid of is often the easiest code imaginable. It’s easy because it does so little. Device drivers often boil down to nothing more than a glorified switch statement – a slack-jawed servant carrying out with precision the demands of a device and its client. These are so easy even PhDs can write them! Of course, many low-level device drivers and services require elegance and brilliance to do well, but in many instances there is simply less need for clever programming at this level.
People confuse familiarity with ease. They don’t know what goes on beneath the surface of their OS’s GUI, so they assume whatever it is must be magic. If anything, these low-level services can be difficult to debug, but they’re often not difficult to code. I’m angering a lot of overpaid, sloppy device-driver engineers right now.
What’s really difficult is writing a word processor, or any other GUI application with a rich feature set and elegant UI mechanics. Writing a GUI application essentially requires defining a protocol for each “object” in the application, and writing “device drivers” that manage the interactions between those objects. So a program like Word is a web of thousands of device-driver like pieces of code, all interoperating without (we hope) knocking unexpectedly into each other. When this is done correctly, the user perceives it as magic. User magic is good!
But bad programmers assume that if it’s easy to use a program, it must also be easy to write the program. This is why we end up with a bazillion terrible “To Do List” type applications. Everybody thought it was easy, and yet even something with such relatively modest requirements far outweighs the difficulty factor of a relatively complex device driver.
So when good programmers come to me and foolishly describe themselves as unqualified for low-level work like OS design or hardware interfaces, I set them straight. You’re not unqualified – you might be overqualified! Now write me a kick-ass To Do List program.
January 7th, 2006 at 12:07 pm
You had me nodding until you mentioned a good (implied GUI from todo list) GUI programmer is perhaps overqualified to work in the kernel. They are just different interests with different bodies of knowledge — having only one set of it doesn’t make you qualified for anything in the other camp. Some people like being at the 5000 foot level writing Cocoa applications, others like to be in the weeds working on device drivers. The reality is if you took almost anyone from either camp and tried to drag them to the other, they’d be kicking and screaming the whole way. There are a lot of Cocoa developers that probably wouldn’t be Cocoa developers if they had to micromanage all those freebies they get for using the API. The trick for programmers exceeding your expectations is to put them in an area that they are interested in… then you won’t have whiners, you will have people that although may not be good at it initially working on the problem all hours of the day.
I don’t believe that writing a great ToDo list application qualifies you for working on device drivers any more then the other way around. They are completely different skillsets requiring a non-overlapping body of knowledge.
Magical nine times out of ten translates to, I have a lot to read to gain enough of an understanding of what I’m working on before I can even set down to the first line of stone. I wouldn’t call them unqualified – they still may have the building blocks to write code, but I would call them underqualified until they took the time to research the specifics of what is needed to play in the other camp… and that’s dependent on their interest / motivation.
January 7th, 2006 at 12:25 pm
Hi Mark – thanks for your feedback. I intended my “devaluation” of device-driver type coding in a somewhat tongue-in-cheek manner. You’re right that not every great GUI developer will be able to cut it, but my point is that most great GUI developers would have no problem whatsoever adapting to the type of coding required in low level systems. I suppose I abuse the meaning of the word “overqualified” here as a sort of ironic response to the ridiculousness that somebody really brilliant would deem themselves unqualified for the work.
I disagree that the two programming jobs requiring “non-overlapping body of knowledge.” The vast majority of programming is, in my opinion, the same wherever you happen to be applying it. In this regard I agree most heartily with the sentiment of my old coworker’s “It’s just code” mantra. You’re right that there are tiny bits to be picked up in order to become competent in specific areas, but in this case I think the tiny bit that needs to be picked up is “permission to try” more than anything.
January 7th, 2006 at 12:54 pm
You’re right non-overlapping is a bit harsh — in my job I’ve worked in both camps, and I have tremendous appreciation for both sides. Without these Cocoa freebies and RAD in languages such as python and ruby we would not have the complexity in software we have today without significant man years. What we can write today in a couple weeks of Cocoa might take the better part of a year in ANSI/POSIX C and a pointer chasing widget toolkit. Stand on the shoulders of giants, etc etc.
I just recently took an active role in J2EE development at the office, I approached it with, software development is in my blood but don’t rely on me as a lead until I can get my bearings. In 43folders terms… a qualified Yes answer to agreeing to a transition in the work I’m doing. It’s relatively easy to pick up new languages, but coming from client/server to service-oriented architecture (SOA) is a learning curve (or leap of faith depending on how you look at it :)) Userland and kernel space are very similar… it’s hard to play exclusively in ring 0, attention to detail is far more pressing as it is mandatory not optional… just because an if..then..else is of the same construct doesn’t mean you understand spinlocks and dma.
I will certainly compromise on my statement with your addition, and you’re right it is just code, but the difference between these camps IMHO is far more then “tiny bits”. These “tiny bits” are often volumes of reference material and often require putting on a different hat and reevaluating how you look at a problem. This is most certainly true in other conceptual differences in for example functional versus object-oriented programming, but also applies to what you manage at different functional boundaries in a system.
Your point is well stated though about not referring to kernel coders as gods and userland as mere mortals… I know plenty of kernel coders that would appreciate a solid end-user application to get stuff done, but they probably have more fun from a software engineering standpoint, flipping bits and the challenge of supporting new gear with companies that are apprehensive about supplying docs.
January 7th, 2006 at 1:05 pm
Linkers. Now those are magic…
(Yeah, I know the basic concept. But this has got to be about the rarest application ever written — there are probably a handful in existence. So they’re mysterious.)
January 7th, 2006 at 1:38 pm
So far, this is turning out to be true with Mugshot – the backend Flickr “kit” that I built is pretty simple and what I originally thought would turn out to be 20 or more classes is one. And I’m the only person that has to deal with any problems with it. The UI, on the other hand, is consuming almost all of my time – how to make so much functionality simple, elegant, and obvious. Flickr provides a *lot* of functionality, and the goal of Mugshot is to make it easier and faster than using the website – but that’s not (so far) an easy goal to achieve.
January 7th, 2006 at 2:24 pm
Eh, linkers aren’t all that magic. They’re just code. :) In fact, I’ve seen both a linker and a dynamic (runtime) linker written in modern C++, and they were both easy to understand and easy to extend.
I think a lot of why people tend to regard low-level code as magic is that it often involves assembly code, and there aren’t too many programmers out there who know assembly. Thing is, assembly isn’t that hard in most cases. It’s just different. I’m planning to explain some of it in hopefully simple terms on my weblog sometime.
It’s weird — as someone who regularly writes and debugs assembly code and low-level things as well as various high-level things, the stuff that’s magic to me isn’t that part of the system or even high-level code. It’s the IBCocoa interaction. It’s the fact that when you drag something from one thing to another in IB, stuff just happens when the nib is opened and there isn’t any code to go along with it. Things that happen without code and which you can’t debug are magic.
January 7th, 2006 at 3:50 pm
[…] This is a good read No responses to ‘Easy, hard and magical code’. RSS feed for comments and Trackback URI for ‘Easy, hard and magical code’. […]
January 7th, 2006 at 5:17 pm
Eric Albert said: “Things that happen without code and which you can”™t debug are magic.”
Or maybe they are just poorly documented (e.g. file format generated by IB)?
January 8th, 2006 at 2:04 pm
No, the IB file format is pretty straightforward. It’s an NSArchive of the assorted objects in the nib. That’s how nibtool reads it — it just unarchives it. And I guess thinking about it that way makes the whole interaction less magic, since really it’s just an instantiation of the objects defined in code with certain variables getting initialized to point to other instantiated objects. Huh. I’d never thought about it that way before. :)
January 8th, 2006 at 9:16 pm
I think what’s most difficult about IB isn’t the data format itself, but the subtle, undocumented things that happen behind the scenes. Sure, it’s a “standard archive.” Until you find out that SuchAndSuchView isn’t actually SuchAndSuchView but IBHackedSuchAndSuchView or something, that gets recognized by AppKit on unarchiving and magically transformed into something functional. Or you discover after beating your head bloody that the an NSButton instantiated and archived in IB for some reason ends up with different default settings than an NSButton alloc’d and init’ed in your code.
That’s magic all right – black magic!
January 8th, 2006 at 9:28 pm
I don’t find any code magical anymore. I know that if I put enough time into it, eventually it will make sense. The question is whether that amount of time is worth it.
What I find magical is the ability to conjure up ideas for innovations like the GUI, the web, or the first person shooter. Even in smaller doses, like bar code scanning via iSight in Delicious Library, this ability amazes me. You could spend the rest of your life brainstorming and there is still a chance that you wouldn’t come up with anything even comparable.
January 8th, 2006 at 10:42 pm
“Magic”, to me, is in the programmer and his or her vision. I’ve done high-level and low-level work, and once you get your head wrapped around the situation the code itself is all pretty straightforward. However, there are some people for whom creating an elegant and precise *design* seems so easy, and yet for most mere mortals it’s an uphill battle.
January 9th, 2006 at 1:41 pm
I dunno. There are “magic” linkers and regular linkers.
Take the vintage MPW tool PPCLink for example. It did dead stripping and nobody considered that a special or advanced feature. In the PEF format is a bytecode system for expanding a relocating data (the “pidata” section type). PPCLink even had a peephole optimizer for those bytecodes.
By comparison to the linkers we use today, PPCLink _was_ magic.
January 14th, 2006 at 7:19 pm
Having recently made the jump from GUI all the way down to drivers, a lot of this talk sounds familiar, and I guess I don’t have anything revolutionary to add, except perhaps “permission to try” has a lot to do with the mountain of documentation you know must exist somewhere but [a] you’re not sure where to start, and [b] the documentation you expect simply does not exist. Mix these two facts together and you can spend a lot of time blaming yourself because other people seem to be getting along just fine with the paltry, obtuse, disorganized reference docs which do exist. Those other guys must be wizards! And I suck. I’m not criticizing technical publications departments; I know they’d love to write reams and reams more if they had the money and could find the people. I don’t think the execs who budget these things realize just how daunting it can be to try something without a few clues.