App Store Mercenaries
August 5th, 2009The latest ridiculous App Store power-play to make it into the public limelight is Apple’s alleged censoring of Ninjawords, an iPhone interface to a community-edited dictionary called Wiktionary. Before being approved, even as a 17+ rated title, the app’s developers were asked to remove specific words from the dictionary’s index.
(Edit 8/6/2009: Since I wrote this article, John Gruber received a response from Apple’s Phil Schiller. He paints a slightly different picture of the alleged censoring, and defends Apple’s intentions as being noble. I still maintain my theory below helps explains the capriciousness of AppStore review policies.)
John Gruber excoriates Apple for censoring a reference book. Gruber also discovered through an interview with a Ninjawords developer that Apple must have gone out of its way to locate words they could find fault with. Apparently the developers had been careful to prevent casual users from stumbling upon an offensive word, by preventing auto-completion for common vulgarities:
“In other words, the App Store reviewer(s) explicitly searched for curse words they already knew, and found them.”
I’ve been thinking about the capriciousness of the App Store review process. It’s ridiculous the kinds of rejections and hoop-jumping we’ve observed in the past year, and one has to assume that the issues making their way into the public eye are only the tip of the iceberg.
Then I remembered something from my own experience that might shed light on the situation. I started as a Quality Assurance tester back in 1995, in a small engineering group. Our group was diligent in the pursuit of finding issues that would embarrass the company or hurt customers. But we worked with larger groups whose motives seemed more oriented to the systematic evaluation they were receiving from their bosses.
These testers didn’t care how good their bug reports were. It didn’t matter if the software gaffe they discovered would save the company a million dollars, or a metric shit-ton of public grief. All that mattered was that the bug was “valid” and that the reporter was “first.”
I learned about the subtleties of this system through the ways that those testers interacted with me. Sometimes a bug that I submitted was determined to be a duplicate of an earlier report one of these testers submitted. If mine had more detailed information, it might be marked as the “original” bug, while the less informative bug was designated a duplicate. This worked great for those of us trying to ship a great product, but not so good for people who were fighting for their reputations in the metric-oriented testing groups.
Because our group was committed to shipping a great product, we were always convinced that bug reports with more information were superior. But the testers who were under the gun to produce new, unique issues, wanted credit for having uncovered these issues first.
As you can imagine, the “thirst for first” led to a significant number of ridiculous bug reports. If a tester could reasonably defend a bug report as valid, then it counted in their statistics, and made them look like a useful member of their team. My impression was that promotions and raises were directly linked to these statistics.
Many of the mercenary testers I encountered were motivated to scrape the system for bugs, as ridiculous as they may be. They logged them into the bug system and then defended them at all costs, as if their lives depended on it. And it turned out, they did. At least, their paychecks did.
I would not be surprised to learn that App Store reviewers are working under a similar structure. A system that rewards “unique, valid rejections” would certainly explain the behavior we have seen coming to light in the past year.
Why would somebody waste time typing profane words into a dictionary, gathering screen captures, and sending them to developers, except to defend their prize “catch”? If perfecting the product was the goal, we’d see a lot more nuance and thoughtfulness. But excellence is one goal, and collecting proof of “doing one’s job” is quite another. I think I know what many App Store reviewers aspire to.
Afterhought: It occurred to me shortly after publishing the above that App Store reviewers can’t be working purely under a “catch all violations” directive, because if they were, there would be numerous rejections based on UI guideline violations, and we’re not seeing as many of those (or are we?). I’m sticking to my thesis, but I suspect that the number of rejections we’re seeing on contrived issues like “you can find ‘cock’ in the dictionary” is because these are the easiest for reviewers to defend with Apple’s published guidelines. Whether a text field is aligned properly is a lot harder to challenge than whether “cock” can be interpreted as profanity.
August 5th, 2009 at 12:17 am
This is an interesting theory, yet it would seem counter-productive to what is in Apple’s (and everyone else’s) best interests. Personally I wish they would just stop approving/disapproving apps based on any sort of content or functionality criteria now that apps have ratings and just start approving stuff as long as it works, doesn’t do anything malicious, and has an appropriate rating. I don’t have an iPhone or iPod Touch but watching all this go down from the sidelines makes me question both my desire to own one and my desire to develop for them.
August 5th, 2009 at 12:38 am
Actually, they are rejecting apps based on UI guideline violations. One of my apps was recently rejected because it did not display a message when it could not connect to the Internet and could not, for that reason, display content. I believe that’s fair, keeps the base quality of apps high and should be done. I immediately fixed the issue and resubmitted (still pending approval).
On the other hand, another of my apps is a (Chinese) dictionary app very much like the one that got rejected. In April, I gave in and submitted a version without the “objectionable content”. They approved it. Well and good. Now, since we have the ratings system, I wanted to do the right thing and add all the filtered words back in at the risk of a higher rating. So, I rated my app as having “infrequent profanity” and submitted it with all the words. To my annoyance, after a week, I get an email telling me it’s been rejected because it contains “suggestive themes”. OK, added that to the ratings as well as and requested them via email to let me know if I need to change anything else. Got a canned “your app is encountering unexpected delays in processing”. Just today, got another email saying it’s been rejected again, now because it contains “sexual themes”.
I don’t know why they love to waste everyone’s time like this. Why couldn’t they have been clear about what they wanted from the get-go? I don’t know. Will my app be approved even now that I’ve rated it with the highest possible rating the App Store accepts? I don’t know, but I definitely am not finding out any time soon.
I wrote a rant about this back in April but the situation is just getting worse.
August 5th, 2009 at 12:39 am
This makes a whole lot of sense, especially in light of your afterthought.
Of course there’s not going to be a cadre of reviewers leaping after HIG violations. A misaligned text field or misused disclosure button isn’t going to cause Apple, as a company, public embarrassment.
On the other hand, something obscene slipping into the App Store and shaming Apple could be hugely damaging to the brand. If you want to have Scott Forstall’s job one day, you do everything you can to prove to your bosses that, hey, you’re looking out for the company.
This kind of laser focus would also explain how Ninjawords, which has obliquely (and overtly) naughty words as part of its dataset gets harassed and Babyshaker Fun Time gets approved. When you’re focused on denying access to specific, provable, boolean obscenity, common sense but gray area stuff is going to get missed.
The bigger questions we all want answered are whether or not Apple’s leadership is clear on how much they’re pissing off their developer community, whether or not that matters to them, and what’s in the works to make it better. But you know that.
In the meantime, this is a juicy bit to chew on. Good stuff.
August 5th, 2009 at 12:41 am
Precisely. Exactly. Yes. Your hypothesis might be right or it might be wrong, but I think you’re on the right track when you examine the reviewer’s possible incentives.
Who are these reviewers? Do we have any data on how many apps are reviewed per day, that might hint at how many reviewers there are? Is it a large, minimally-trained contract workforce, as I like to imagine? Or is it a dozen grumpy software engineers?
August 5th, 2009 at 1:03 am
@Karan Misra:
“I don”™t know why they love to waste everyone”™s time like this. Why couldn”™t they have been clear about what they wanted from the get-go?”
This is how they see the world: their time is expensive, your time is cheap.
August 5th, 2009 at 1:12 am
@bendin To some extent, they are wasting their own time as well because reviewing an app in its entirety, I would assume, would require more resources. This is especially applicable to apps like mine which are getting rejected purely on the basis of their rating. It would take them only half a minute to ensure that I have modified the ratings appropriately, whereas currently they ask me to submit a new binary every time (whether I need to or not) and thus reprocess the app entirely, wasting both their time and mine.
August 5th, 2009 at 1:51 am
It’s also possible that these “bugs” are filed by the same type of people who make it their mission in life to constrain the personal choices of others to match their own notions of what is permissible in the universe they believe they inhabit.
August 5th, 2009 at 3:30 am
Isn’t this whole App Store rejextion/censoring debacle an all american problem? I have traveled the states (I am from Europe) and I always thought that it was rather funny that (at least in California) most people are normal and intelligent humans that did use as much “offensive language” as every European does and didn’t tell any god-nonsense all the time. Most Californians I talked to even told me that Bush is the biggest idiot they have ever seen in their live and despite all patriotism the states are built on a ton of hot air and are shortly before a total financial collapse.
BUT: When switching on a TV or reading any newspaper in the states every nipple is covered, god is praised and loves america, bla bla bla. Even if you want to see your original cinema movies you have to travel to Europe, because they are all censored. Remember the Janet Jackson nipplegate? Europe didn’t care at all – wohoo a nipple – we have seen them before – yawn.
Don’t get me wrong – I am not bashing Americans here (as said, most of them who I meet were smart people), but there is a huge problem in America, some kind of double-moral/double-standard thing.
You can’t say fuck or show nipples in TV, but children are allowed to fire tank-breaking fully automatic weapons as a hobby. So my guess is – If you fix this cultural double standard, the AppStore will fix itself.
August 5th, 2009 at 4:08 am
I think you’re absolutely right with your theory. I suspect, the App Store mercenaries are judged by two major factors:
#1 sheer volume (review as many apps as possible)
#2 number of defendable rejections
To reach #2 with the smallest possible effort, they focus on easy to find rejection reasons: entering specific buzz words to find offensive content, checking whether an app requiring Internet access shows a proper alert in Airplane mode and stuff like that.
http://appreview.tumblr.com/ has a comprehensive list of popular (and as it seems partly automatically checked) rejection reasons.
To reach #2 it also makes sense to not work with the developer to propose suggested fixes (like specific ratings) but to wait until the developer submits again just to reject again. It all makes sense if the incentive system works like that.
August 5th, 2009 at 6:27 am
So what we need is a way to appeal the rejection, run by a different team at Apple. If a rejection gives a bonus to the reviewer, then a successful appeal (the rejection is deemed invalid) would give a bonus to the member of the appeal team and a penalty to the reviewer who rejected the app in the first place.
August 5th, 2009 at 7:37 am
I’ve mentioned this elsewhere, but I think the ultimate solution to this problem is for Apple to subcontract the content evaluation to a group like the ESRB. I think Apple can still retain control of ensuring that apps meet HIG guidelines and other technical requirements, but that the evaluation of profane/violent/sexual/etc. content should be handled by a 3rd party that specializes in those types of decisions.
Censorship is a nasty business, one that Apple does not need nor should it want to be involved in. It’s loose-loose as you’re constantly in the press for being too strict/not strict enough. When dealing with the sheer volume of apps that Apple is, there is sure to be a constant stream of edge cases that will be a perpetual PR nightmare. Apple should pay a third party to take this heat from them and can defer PR issues to that organization as they inevitably arise.
I found your article particularly insightful into the possible structural motivations behind the reviewers’ process. I suspect that it likely has to do with evaluating an overwhelming volume of apps and the reviewers’ picking the lowest-hanging fruit (so-to-speak) rather than having the luxury of thorough evaluations.
August 5th, 2009 at 12:53 pm
It’s time for some Euroweenie flambe`!
—-
Isn”™t this whole App Store rejextion/censoring debacle an all american problem? I have traveled the states (I am from Europe) and I always thought that it was rather funny that (at least in California) most people are normal and intelligent humans that did use as much “offensive language” as every European does and didn”™t tell any god-nonsense all the time.
—-
So you judge America by California? Even worse, you judge all of California by people you met? That is extremely illogical and statistically invalid. I expected more intellectual firepower from your average Eurosocialist atheist.
Censorship wasn’t born in America. America was founded upon freedom and one of those freedoms is freedom of the press. Guess who was busy censoring the press in the 1600’s-1700’s? That would be Europe.
—-
BUT: When switching on a TV or reading any newspaper in the states every nipple is covered, god is praised and loves america, bla bla bla.
—-
Any newspaper? Find me a single one of any circulation size (besides the Washington Times) that praises God and is patriotic. There isn’t a one. You are a liar.
—-
Even if you want to see your original cinema movies you have to travel to Europe, because they are all censored.
—-
Uhm, no, they’re not. You lie yet again. First of all, the government is the only entity which can censor. Second of all, the only movie censorship boards were state-run, and they disappeared in 1968, thanks to the US Supreme Court.
—
Remember the Janet Jackson nipplegate? Europe didn”™t care at all — wohoo a nipple — we have seen them before — yawn.
—-
Really? All of Europe didn’t care? Or just the people that you know?
—-
You can”™t say fuck or show nipples in TV, but children are allowed to fire tank-breaking fully automatic weapons as a hobby. So my guess is — If you fix this cultural double standard, the AppStore will fix itself.
—–
No, children are NOT allowed to fire tank-breaking full-auto weapons as a hobby. In fact, full autos have been restricted weapons since the middle ’60’s, which means very few people have them.
I have caught you in so many obvious untruths that it makes me wonder if you are intentionally satirizing the ignorance of many Europeans about America.
August 5th, 2009 at 12:59 pm
OK – we’ve had two opinions on the question of whether US culture norms are in part to blame for this problem.
I don’t think the value of pursuing that thread will outweigh the prejudice that will evidently come with it, so I’m going to delete any further comments on that particular tangent.
Thanks for chiming in, but it’s flame-kindling if I ever saw it :)
August 5th, 2009 at 1:35 pm
My theory( http://twitter.com/lakshmivyas/status/1904981233 | App store inital review is done by an automated test suite. ) is founded on the same basis. The key difference is not a rejection but an approval: Any human responsible for approving babyshaker wouldn’t have let it go through.
August 5th, 2009 at 1:55 pm
As Ortwin Gentz says, I think the reviewer incentive is based on two criteria but I think they are slightly different than he proposed:
1) Volume of apps process
2) Must reject any app that might embarrass Apple
#1 explains the lack of helpful rejection letters (e.g. fix X, Y and Z and then we’ll approve it) and #2 explains the almost pathological hunt for ways in which an app might be found offensive. It is far better to err on the side of caution than to let Apple be embarrassed by another Babyshaker.
Another possibility is that there are various review groups in various countries and some are more “strict” than others; Ninjawords and Eucalyptus got sent to one group and Babyshaker and the other dictionary apps got sent to another.
August 5th, 2009 at 2:49 pm
They definitely reject for HIG violations (perceived and real), but you don’t hear about those because they usually make more sense, or at least are more defendable.
August 5th, 2009 at 3:07 pm
So far, my apps have been rejected only for good reasons. And I’m grateful Richard and his team found the issues they reported or helped me find one edge case in which the feeling might have been : ‘this app doesn’t work’ because of network connections or such.
I even talked to Richard once: he had temporarily rejected my app because I had mentioned Crash Log Reports in the update textfield. Yes, the Richard who rejected all those Google Voice App last week (was it last week). After I changed that text, the update was accepted in hours.
Now, it seems that the rating thingie is going bezerk. I don’t have any content in my apps, but I’m gonna make sure I never add any, and never let users talk to each other because my apps would be prohibited because I used french words: pine, putain, connards.
OK, Now I can’t read that blog on my iPhone anymore…
Other ways to approve apps would be much better, such as crowdsourcing as Manton suggested. But, we’ll never see that. The siuation will just get worse. Has anything improved among the issues developers have with the AppStore review Process. I don’t think so.
August 5th, 2009 at 8:11 pm
One of my apps was rejected for a HIG violation: a user could navigate to the next screen by touching a table cell but NOT by touching the blue disclosure button in the cell. To Apple’s credit they provided a detailed explanation and it was fixable. The only problem I had with the rejection was that the app had been out for months already and the problem was never picked up.
Like everyone else who is trying to do this as a business I worried about all of these app rejections because it highlights just how vulnerable we are to Apple’s whims. We really are vulnerable… what happens if this is your only revenue stream and things like app rejections or changes to how apps are presented (like the issues that people are having with the keywords) can really make or break you?
August 6th, 2009 at 5:31 pm
As one who’s been rejected on UI button issues, I find the always humorous line “Applications must adhere to the iPhone Human Interface Guidelines”
I guess “iPhone Human Interface Immutable Laws” doesn’t have a peppy enough ring to it.
The other funny (strange, not ha-ha) thing is that the rejections are completely up to the whim of the reviewer-which lends more credence to the mercenary theory.. The last UI rejection was on a code-only update- the UI change had been approved for the last 3 revisions.
August 7th, 2009 at 1:59 am
Well I just got a rejection for a UI violation. I used a Mac OS X
kUserIcon(a silhouette of a head) as a button to show a “user” screen. But it turns out that the same icon is used on the iPhone OS as a “Show Contacts” button. It was my fault for not catching a difference between Mac and iPhone OS. And it’s a glaringly obvious mistake for anyone who’s familiar with all the standard icons, so this isn’t a case of rejection for a button being a few pixels off. But I’m still miffed that a difference of opinion on ratings meant I had to resubmit the application.And, when I got the rejected-for-unfiltered-wikipedia-access message, the reviewer didn’t point out my rejection-worthy UI mistake, even though it’s on the initial screen. Now maybe that particular reviewer didn’t notice. Or maybe they rejected my rating based on the description, w/o even running the app. Or maybe they were punished for taking the time to point out other mistakes, and rewarded for letting people re-submit something that was sure to be rejected…
August 7th, 2009 at 3:56 pm
I think your theory is spot-on, but misses one reason for The Way Things Are. As fast as the App Store has grown it is likely impossible to find, hire and train enough _GOOD_ testers as is required. So independent of the (likely flawed) reward system for the testers, they’re prolly scraping the bottom of their local barrel.
August 7th, 2009 at 4:44 pm
Just a couple things which you may take as you will.
* Schiller’s ex $MACR. Consider the source. If you want to trace a lot of strife follow the ex $MACR employees and see whose skirts they ran under. I’m sure many organizational lessons were learned at $MACR, all the wrong ones. And you can see this with the old mis-handling of OS X Security [then] and the AppStore [now]. All talk and no real policy.
* I’ve also seen and worked in divisions like the [cr]AppStore where they have ‘scoring’ for trouble tickets. Of course the staff starts to game the system and ‘stuff the box’ for perks, bonuses or incentives. It’s weighted to ‘number of closed issues per day and not resolution.
* Couple that with a NON-technical staff that barely knows the API issues but screens MOSTLY on policy rules.
August 7th, 2009 at 5:15 pm
According to reports from WWDC attendees who spoke with Apple employees, the situation is this: apps are reviewed by Apple employees, most of who are doing reviews *in addition to* their regular jobs, in many cases taking the review work home at night. As far as can be learned from talking with Apple staffers, there is no department or group whose responsibility it is to review iPhone apps. That alone probably accounts for many of the wildly inconsistent approvals and denials.
Then there’s the workload. According to http://148apps.biz/app-store-metrics/?mpage=submission, Apple has received an average of about 8,000 apps per month in the past five months. That’s 400 apps to be evaluated every working day by Apple employees who are doing reviews on top of their full-time jobs.
In short, it’s pretty obvious that Apple was unprepared for the flood of app submissions, and has dealt with it very poorly. Hiring a hundred or so full-time reviewers who followed consistent (and *published*) criteria would go a long way toward alleviating the App Store problems, but that’s an expense Apple would obviously rather avoid. But just as obviously – at least to us developers, if not to Phil Schiller – the current ad-hoc system is badly broken.
The question is when Apple will admit that it has a disaster on its hands, and take the kind of serious action needed to reform the App Store approval process. Unfortunately, as long as Apple execs can smugly count their billions of downloads and tens of thousands of apps, it’ll be easy for them to go on overlooking the festering sores at the heart of the enterprise.
August 7th, 2009 at 5:21 pm
I don’t have experience with iPhone development, but I do have a lot of experience with outsourced business functions (including being part of the outside team). I just can’t get over the impression that the App store approval process is outsourced, with a strict set of level-of-service agreements. It would certainly explain the obsession with the enforcement of petty rules.
August 7th, 2009 at 10:44 pm
> Isn”™t this whole App Store rejextion/censoring debacle an
> all american problem?
I’m from the UK but live in US for the past 10 years. What you are noticing is the influence of all the Puritans here in the US. They are maybe 10% but they terrorize the hell out of everyone else. You have to take proactive steps to avoid them attacking you. Anything you say or do that cannot be said or done in church is like sticking your arm out of a moving train. People and companies here just naturally avoid taking these risks without noticing it in most cases. This kind of proactive defense from Puritans is part of shareholder responsibility for a public corporation.
A big problem is that so many Puritans are politicians, police, judges. So it isn’t just that you can be hounded by them, you can be arrested or fined. It’s like the religious police in Saudi Arabia. So radio and TV stations are regularly fined millions of dollars for saying “penis” (I kid you not) and the most common arrest here is possession of cannabis, which gets you 5 years. We have the largest and most non-violent prison population in the world, the majority of which were taxpayers before they went in, because the cops are out hunting for sinners as well as criminals. You can lose your kids here, easy. If you have kids, you really have to watch what you say and do.
The funny thing is, most of the US Americans who are famous around the world for their accomplishments are actually persecuted here in the States. A couple of years after I got here I realized all of my American heroes had criminal records for things they would not be arrested for in Europe or Canada or most other places. Outside the US, American culture is much more respected than inside.
So when you submit an app to the App Store, first ask yourself: “could a preacher demo this app for a congregation on Sunday morning?” and if the answer is no then you still have work to do.
August 7th, 2009 at 11:30 pm
I suspect the same is the process at proxy server companies that block workers form using company computers to surf the web. It seems to me that many of the websites I was being blocked from seeing were perfectly legitimate and in fact critical to me to do my job. For example, I’m a photo editor. I find photos from all over the world.
One of the worse examples of an illegitimate blocking of a website was Corbis. Sure, we all hate Bill Gates and don’t want him to make more money than he already has (joking), but Corbis is a great place to find photos. Yet the proxy server our company employed blocked it because it had a few photos of naked women. Um, yeah, a stock photo agency of Corbis’ size is going to have a few of those no doubt. The IT person who was consulted about opening up the proxy server to let us in was giggling on the phone line as he was looking through the site. That’s when I developed my theory about the people who scour the web for sites to block, are going to find sites to block whether those sites deserve it or not, simply because they feel they have to keep finding more to keep their jobs.
August 8th, 2009 at 5:07 am
One of the problems of app reviewing being treated as a part time job is that reviewers will stop looking after they find the first reason for rejecting an app. If they have the time to look in more depth, they may find other issues/ problems and the more of these communicated in 1 rejection message, the less developer frustration there will be as there will be fewer submission/ rejection cycles.
One of the advantages of involving many Apple employees in the process is that problems experienced by many are likely to get fixed. Dealing with the number of submissions is clearly a problem but it is not a problem that any other company has had to deal with on this scale. So 12 months on Apple is still feeling its way.
The reason for rejection of Google Voice was covered in my article of last October “The unwritten Rule behind Apple’s App Store rejections”
http://lowendmac.com/nash/08tn/unwritten-app-store-rule.html
Maybe Google thought its partnership with Apple through Eric Schmidt, Al Gore, Arthur Levinson etc. was more important to Apple than Apple’s partnership with AT&T. Before developing, think about that “unwritten Rule” and ask yourself if your app will add more business to Apple (and the carrier) than it takes away.
August 8th, 2009 at 12:55 pm
You say, “It”™s ridiculous the kinds of rejections and hoop-jumping we”™ve observed in the past year, and one has to assume that the issues making their way into the public eye are only the tip of the iceberg.”
Really? You really believe that for every story getting press there are nine more, just as ridiculous, getting no attention at all? Or does it seem way more likely that, given the number of apps on the store, that MOST reviews are reasonable, and the stories you hear about are the exception?
Also, I think you got it backwards: I don’t think reviewers are rewarded for rejecting apps (even from a manager’s point of view, rejections just generate more work and no benefit for the organization), but they are probably severely punished (fired) for approving one that causes a problem.
September 5th, 2009 at 8:20 pm
There have been enough stories of this kind now that it’s clear that Apple is engaging in extremely wide-ranging censorship on the App store.
I bought the Chambers dictionary – one of the most respected dictionaries in the world, and it had searching for ‘offensive’ words disabled by default. I had to turn it on. I WANT my dictionary to be complete, as does every person who buys one.
It’s a DICTIONARY, for crying out loud. I thought we got over this ‘foul language has no place in the dictionary’ rubbish in the sixties. Frankly, anyone who believes in censoring dictionaries at all, for any audience, is a moron who deserves to be cleaning latrines for a living, as their stupidity warrants nothing better.
I’ve patronised Apple for years, but this has left a very bad taste in my mouth. With their recent success, as a company they seem to have adopted the mindset that everything they do is right by definition.
I currently have an iPhone; when it breaks, there’s no way I’ll be buying another one unless Apple pulls a 180 degree u-turn on this. I’m simply not interested in closed censored software, with material that any reasonable person would accept or desire removed.
Apple’s success has gone to their head. There have been a number of other distasteful ways that they’ve screwed their customers that they wouldn’t have dreamed of 5 years ago (Google ‘iDVD chapters’ to see examples of removed features). Censoring is not out of character for Apple now; I suggest that it’s their new, unpleaseant character.