Wednesday, April 17, 2013

The Challenge Index and Gilb's Law

Once again, Jay Mathews of The Washington Post has published his ranking of various schools according to his "challenge index", which is the ratio of the number of college-level tests given at the school to the number of students in the graduating class, a college-level test generally being Advanced Placement (AP), sometimes International Baccalaureate (IBB). A couple of Washington, DC, schools head his nationwide list of private schools: St. Anselm's Abbey School with an index of 8.533; and Washington International School with an index of 7.288.

St. Anselm's usual graduating class is in the upper 20s: if the number used is 30, that gives us 256 AP tests. These are generally taken by juniors and seniors, so there are probably about 60 to 65 boys taking them. Washington International School has about 900 students from kindergarten through 12th grade. Figure that as being a bit heavier in the lower grades, and reckon between 60 and 70 students in a graduating class: a graduating class of 67 works out to 481. In both cases, the Post is giving us four digits of index for three digits of tests.

I imagine that the index can be useful as an indicator of aspirations. The school that has most of its students taking a couple of AP exams is likely to be asking more of them than the school that has almost no students taking such exams. After a certain level, though, I don't know how much it matters. I can name half a dozen or more high schools in the District of Columbia where it would be difficult to graduate without getting an excellent preparation for college. I know the challenge index only for the two above, and will likely forget it within the week.

In Peopleware, the software consultant Tom DeMarco summarizes a discussion he had with Tom Gilb, the author of Software Metrics. Gilb is or was a believer in measuring everything, DeMarco something of a skeptic. Neither man convinced the other, but DeMarco took away from the discussion what he calls "Gilb's Law":
Anything you need to quantify can be measured in some way that is superior to not measuring it at all.


  1. For selecting students for university, I think it's important not to go just on marks, as they mostly do here in Australia, but to also require a statement written by the prospective student about why they want to study a particular thing, plus a longish, wide-ranging chat.

    1. All or nearly all US colleges require an essay or two. Many of them have an alumni interview as part of the process. An article in The Washington Post's Sunday magazine mentioned a school where every admissions officer reads about 30 applications a day, roughly one every 15 minutes. It sounded to me like very tough duty.