To the TNS Psychometrics Committee:
Pat Thomas has been kind enough to send me copies of the Psychometrics Committee correspondence to date. I was a member of the original TNS Psychometrics Committee, chaired by Ed Van Vleck, and I remember well the difficulties we faced and some of the approaches we used. I hope that this background will prove useful to the deliberations of this committee.
As you all know, I am the author of several high-range I.Q. tests. Most recently, I have published a new test, the LIGHT (Langdon Intellectual Gradient High-range Test), a successor test to the LAIT. The number series (too dependent on mathematical training) and the vocabulary (too easy to cheat, too culture-dependent) sections have been eliminated, along with ten other problems that were flawed in individual ways. Ten new problems have been added. The test consists of ten figure series, ten miscellaneous symbolic problems, and twenty miscellaneous spatial problems. I have sent Pat enough copies of the test to include one with each copy of the next PsyCom memo.
I have a few items in my files that bear on the current examination of the meaning of SAT scores. I spoke with Pat about this and have included a number of papers on the subject which he will share with you.
My own opinion, after reviewing all the material in my possession, is that TNS' SAT cutoff should be about 1460. We really goofed when we set it at 1550, but we had access to much less information on this test in the early 1980's and some of it was of dubious quality. At this point, I find myself coming to conclusions about the SAT that are very similar to what I've heard from others recently, an encouraging sign.
Now I'd like to respond to some of the ideas expressed in the PsyCom correspondence.
Pat is right, in his Dec. 15 memo, that the greatest need of the Psychometrics Committee is for data. Somebody needs to take the trouble to actually do some research into what's known about the various tests to be evaluated, and this is a lot like work. On the old committee, only Ed and I were willing to do the research. The other four members of the committee didn't even comment on the stuff we presented, but they were happy enough to vote on the proposals we submitted.
Quite frankly, if any of you don't intend to be actively involved in research, I believe that you should resign from this committee.
I agree with Pat, in his Jan. 12 memo, that test cutoffs can change over time, but I think that our first concern should be to determine whether some of our qualifying scores were set accurately in the first place, and even if it makes sense to accept some of the tests that are currently used for admission to TNS.
For example, we currently accept a number of tests that, in my opinion, we probably shouldn't, including:
The Wonderlic. This test, which is given in employment agencies all over the U.S. (and probably other places), is not at all secure. Hundreds of thousands of people have access to the answer key. It also lacks sufficient ceiling to be accurate at the 99.9th percentile.
The armed services tests. There have been a number of changes in the scoring systems, particularly for the Navy test. These tests are designed to discriminate well around the general population mean and don't have enough ceiling for our purposes.
The Stanford-Binet. This test doesn't have enough ceiling for adults; childhood scores (at least those made by children under ten) are very poorly correlated with adult intelligence. Also, the Binet yields large numbers of highly inflated scores.
The California Test of Mental Maturity. This is one of the poorest I.Q. tests ever devised. There are a number of bad items. There is almost no correlation between the CTMM and either the LAIT or the FSQT (the test did correlate moderately well with the PIAS). I don't trust the scoring system. Examination of previous scores listed by LAIT testees reveals that the CTMM yields too few high scores.
The WAIS. This test uses an idiotic age correction for adults. We shouldn't accept any WAIS score unless we are sure that it is not age-corrected.
There may be other tests we should drop. If we still accept either the BAT or any of Chris Harding's tests, they should be removed from our list of qualifying scores, because neither Bloom nor Harding has supplied us with the statistical data that would permit usto evaluate their tests. Both were asked by the Van Vleck committee; neither came forward with data.
Also, we need to be very wary of any test (Scott Jackson mentioned the LSAT) that changes its scoring system. My preference is to avoid any possible confusion by not accepting such tests.
In Dan La Coste's letter of Jan. 21, he wrote, ``The fact of achieving a lower IQ at certain tests simply shows that these ones are bad or inappropriate.'' Well, possibly. But if someone makes one score on one test and a different score on another it may well be that the lower score is closer to his or her ``true'' intellectual ability.
Dan, on what grounds do you conclude that ``. . . it would be necessary, for judging the value of a test, to use as reference the WAIS-R, which is the most reliable of all, in order to determine the adults' intellectual level''?
I disagree with Dan about what he calls I.Q. and E.Q. (the former referring to ability relative to those of one's own age and the latter to ability relative to the whole population). I don't give a damn whether somebody is ``smart for his age.'' I want to belong to a group with people who can think now--not who once were or might someday be intelligent.
In response to Dan's question about the Cattell, I think that this is a reasonably good test. The stupidity of Mensa members which Dan refers to is not a matter of inferior equipment, but of neurotic patterns which are very prevalent in contemporary society and not limited to Mensa members. The problem with the Cattell from our point of view is that it doesn't have a lot of ceiling.
I would be interested in seeing the LUTE, but claims to measure up to six sigma above the mean are absurd, whether put forward by Chris Harding, Ron Hoeflin, Grady Towers, or Dan La Coste. There just aren't enough people in the world to do the statistics that would make it possible to establish where the fifth sigma is, let alone the sixth. LAIT score reports for those who score at or above four sigma (on the total score or any subscore) bear a warning that such scores are extrapolated and are subject to large uncertainties.
I like Oscar Wehmanen's suggestion that the TNS membership be surveyed to gather information on the SAT (and other tests). I hope that this will actually be done. Also, there is some data available from TNS Membership Officers (past and present) on the distribution of qualifying and non-qualifying scores submitted on various tests.
Croft Warn wrote, in his letter of March 15, ``I'm probably not going to have much more to offer than what I indicated when I first volunteered: to make my computer useful in sifting data and doing statistical computations . . .'' We've got lots of computers, Croft; what we need is your brain power.
In Kjeld's postcard of April 13, he advances an explanation for the fact that 1600 is a more common score than 1590 on the SAT (different forms yielding 1590 and 1580 as the raw score for one missed item). There are two other ideas that should be included here: 1. The bunching at 1600 is probably due to the fact that there are people whose ``true'' ability level, according to the SAT scale, is 1610, 1620, 1630, etc.; these people tend to make 1600's; 2. On the LAIT, there have been no perfect scores of 176--but there also have been no 174's, though there have been a number of 175's; the reason has to do with the penalty for guessing: to get 174, you'd have to get all the items right but two, get one wrong, and omit one; two wrong yields a 173.
Finally, I would like to bring to the attention of this committee a project that Cyd Bergdorf and I are working on. We intend to publish a high-range psychometrics journal. This will be a scholarly publication which will provide a forum for the discussion of statistical methods for evaluating scores far out along the right tail of the normal distribution and tests designed to measure very high I.Q.'s.
It is absurd that each of the nine or so currently active high-I.Q. societies has its own psychometrics committee, reinventing the wheel. There is a need for a venue for free and open debate about this subject. Each society will be able to make use of this resource for data on which to base its determinations as to appropriate cutoffs on various tests.
All of you are on the distribution list for the new journal. We also intend to include the membership officers and psychometrics committees of all the high-I.Q. societies and a number of academic psychometricians. With regard to the latter, there is clearly a need for dialog between the academicians and the people working on high-range testing in the high-I.Q. societies. I think that we need to take steps toward putting our house in order here; otherwise, someone will eventually write a scathing denunciation of amateurs playing at psychometrics without effective peer review.
[Note: Unfortunately, the idea of a high-range psychometrics journal was not pursued at the time of this newsletter. I intend to begin publication of a Web-based journal soon.]