Things are getting ever-nearer with FlexTime, but as is the case with every project of relative complexity, the number of niggling details expands in proportion to the number of days of development ostensibly remaining.

Today I spent a lot longer than I wish I had working on a confounding bug report with my NSToolbar implementation: the contextual menu doesn’t update to reflect the changed state. If you control or right-click in any standard Cocoa application’s toolbar, you’ll see a contextual menu containing common configuration options. The indicate the enabled state of these options, a check mark is supposed to appear next to them. This is what the menu looks like in FlexTime 1.0b5:

No check marks! Obviously the icons are small, and I’m seeing “Icon and Text.” Something is definitely wrong here. But I’m not responsible for managing this menu. Who is?

If you guessed NSToolbar, don’t feel stupid. I guessed that too, but we’re wrong. My first reaction in debugging this problem was to search the documentation for NSToolbar to see if there are any funny delegate methods I’m supposed to implement. NSToolbar has nary a mention at all of even the existence of this contextual menu! Before getting too deep, I tried a few web searches, to see if anybody else had already run into the problem. I found this from 2001 describing an identical problem. And this from last year following up on it. But where is the fix?

The next phase in my debugging process was to start hacking out code until things work. I figured there must be something funny in my delegate methods, so I just unhooked the delegate and let ‘er rip. Result? An empty toolbar with exactly the same problem.

The Xcode developer tools ship with a source code example, Simple Toolbar (local link). I was running out of patience so I decided to attack the problem from the complete opposite direction. Instead of asking “why doesn’t my menu work right?” I would ask “why do all the other ones work right?” Who is responsible for setting those damn check marks, anyway?

From the SimpleToolbar project, I first launched and confirmed that the check marks show up as expected. Then, I “Paused” execution to get access to the gdb console, and set a breakpoint:

Somebody is setting those checkmarks, and unless they’re doing it in an extremely unsanctioned way, I’ll catch them at NSMenuItem’s setState: method. The only problem is everytime I “continue,” it reactivates the application and causes my breakpoint to be hit at a spot where I don’t care about it. Once the breakpoint hits I’m stuck back in the debugger and never get a chance to right-click in the toolbar! This is overcome with a simple conditional on the breakpoint. It turns out that it’s just one menu item that is always getting set when I activate the app, so I configure gdb to skip it in that case:

Now I’m able to right-click and provoke the anticipated action. I’m interrupted again by the debugger, this time pointing at one of my items of interest:

Note: you don’t have to use commands like “bt” at this point, since Xcode’s debugger is mimicking the same thing in the UI, but once I get started in the console I tend to stay there for a while.

So what have we learned by now? NSToolbar is not responsible for setting those menu item states, NSWindow is! It makes a sort of sick sense, if you consider the overlapping responsibilities of the toolbar and the window. After all, the window is what you “set” a toolbar on. The window has to know how to collapse and expand it. It’s a sick little behind-the-scenes game that fortunately we don’t have to see much of.

So why isn’t validateUserInterfaceItem: called in my app? I set a breakpoint on the method of NSWindow and it never triggers. Hmm, suspicious indeed. But when I, on a whim, search for the method in my sources, my memory is thankfully jostled. Of ourse! FlexTime uses a subclass of NSWindow, and within it, overrides validateUserInterfaceItem! It’s been a while since I did this, so it’s not fresh in my mind. This is what the method looks like:

// MENU Validation
- (BOOL)validateUserInterfaceItem:(id <NSValidatedUserInterfaceItem>)anItem
{
	BOOL validItem = YES;

	// We use validation to keep the state of NSMenu targets up to date
	SEL thisSelection = [anItem action];
	if ((thisSelection == @selector(undo:)) ||
		(thisSelection == @selector(redo:)))
	{
		validItem = (mUndoDisabled == NO);
	}
	return validItem;
}

At some point I figured out that if I override NSWindow’s menu validation, I get a chance to disable undo and redo. I wanted to do this because it turns out to be very inconvenient timing for a user to suddenly try to “undo” during the middle of a FlexTime routine playback. So what’s wrong with the above method? It cuts NSWindow out of the loop.

Simply adding a “[super validateUserInterfaceItem:anItem]” call at the top of the method does the trick functionally, but it leaves me with a bad taste in my mouth. Specifically, a bad compiler warning. The method is defined as the sole member of the formal NSUserInterfaceValidations protocol, but NSWindow isn’t declared as conforming to that protocol. So I’m stuck in a tough situation. I know my superclass implements this method – I saw it in gdb! I also know that if it doesn’t get its implementation called, my functionality suffers. I decide to code defensively and plan for the best while expecting the worst. If the superclass responds to the method in question, we’ll cast it as being protocol compliant, to satisfy the compiler:

// Let the standard NSWindow validation take place
if ([super respondsToSelector:@selector(validateUserInterfaceItem:)] == YES)
{
	validItem = [(id )super validateUserInterfaceItem:anItem];
}

Now, even in the unlikely event that Apple completely changes the way they handle this, we’ll probably still be safe. Added bonus? No compiler warnings.

But more importantly, the change works! My check marks now appear as expected. One more bug, laboriously down the drain.

How did this bug happen? Several factors played into the mistake on my part. Ultimately it came to an underestimation on my part of the method’s significance to NSWindow. Since NSWindow is not documented as using the method, it was a “lucky break” for me that implementing it in my subclass got me the short-term results I was seeking. But alarm bells should have been going off. I’m overriding a method in a standard system class and seeing interesting results. That means the system must have good reason to call it. I will be much more careful in the future when subclassing and overriding system classes.

The other problems here were with naming convention and habitual expectations. It’s important to remember that, although validateUserInterfaceItem’s main purpose is to enable or disable a part of the UI, its secondary purpose is to allow clients arbitrary access to the UI item. If it was called “validateAndMuckWithUserInterfaceItem,” I might have paused a second or two and thought more carefully about the implications of overriding it. Worse, the typical scenarios where this method is implemented are those where the system has set aside some number of UI elements as being “your responsibility.” I’m not in the habit of needing to defer to the system about whether my UI elements are enabled or not, but in this case by becoming the system, I acquired a whole new set of responsibilities.

Update: Chris Liscio pointed out that the behavior of validateUserInterfaceItem: is to bounce up the responder chain until somebody says “YES” to an item. So another failure here is based no my misunderstanding of the method. It’s well documented, but I never paid close attention:

To validate a control, the application calls validateUserInterfaceItem: for each item in the responder chain, starting with the first responder. If no responder returns YES, the item is disabled.

This means my typical approach to implementing the method needs to change. I have assumed that everything would be enabled unless I said otherwise, when in fact the opposite is true. I should insist that everything I don’t recognize gets a “NO” result from the validation method, as a general rule.

This entry was posted to Cocoa, Debugging, Hacking, Programming, Xcode. Comments are closed. Thanks for reading!