To the TNS Psychometrics Committee:
Congratulations to the members of the committee for ratifying a new, and much more reasonable, qualifying score on the SAT. However . . .
1. Patrick Thomas wrote in his memo to the committee of July 12: ``ON BEHALF OF THE PSYCHOMETRIC COMMITTEE OF THE TRIPLE NINE SOCIETY, I do hereby formally recommend to the Membership Officer that our SAT minimum be changed to 1470.'' This is technically incorrect; the following paragraph is quoted from my letter to the TNS ExCom of Sept. 29, 1992:
It's good that the Psychometrics Committee has been reactivated and has been active enough to review our admission score on the SAT, which has long been considered unrealistically high by many with an interest in high-range psychometrics. But the change of qualifying score should not have been phrased as a recommendation to the Membership Officer, who, according to the TNS constitution, ``shall . . . determine whether applicants meet the admission criteria established by the Society's psychometrician.'' The closest thing we have to a ``psychometrician'' is our Psychometrics Committee. Since the constitution is silent on this committee, addressing only the Executive Committee and committees in general, I believe that it is the Executive Committee which must formally ratify this change. Therefore, I move that TNS' qualifying score on the SAT be changed to 1470, as recommended by the Psychometrics Committee.
2. In the July 12 memo, Pat went on to say:
My thanks to all who participated in the vote; we had an 89% turnout, good even for an election year. Two members appear to have given explicit permission to broadcast their votes by including them in memos (enclosed) addressed ``to the PsyCom''; a third might also have done so, but I wasn't quite sure, so blacked out his vote. Votes ranged from a 1450 up to a pair of 1480's, and averaged 1466.375. The fourth lowest vote was 1467 and the third lowest was 1460, so we came fairly close to lowering the cutoff further.
This is contrary to the practice of the Executive Committee currently and of the Psychometrics Committee in the early days of the society. All committee votes are, or should be, public. This protects against fraud on the part of a committee chairman and places each committee member on record (as the members have every right to expect in exchange for putting him or her in position to act in their stead).
I agree that there can be a problem with admitting members of the other 99.9th-percentile societies to TNS on the basis of their membership in those groups. Some nominally 99.9th-percentile societies (at least four of the five others: MM, ISPE, Minerva, and OATH [One-in-A-THousand]; Cincinnatus never got around to setting separate qualifying scores from TNS) have accepted scores on tests not accepted by TNS or accepted by TNS with higher qualifying scores. But we are also trying to rebuild the society's membership after the losses caused by the high-I.Q. wars of a few years back and we, who are close to the admission-criteria-setting process, know that it's inexact at best. I don't think that the Executive Committee motion quoted in Pat's letter will lower the average intelligence of TNS members significantly; most of the qualifying scores accepted by these groups are in the right ballpark, at least.
We need to scrutinize our own list of qualifying scores and throw out some of the more questionable tests, as I mentioned in my letter to the committee accom-panying Pat's memo of July 12; I made a case against certain tests on the list in this letter. The case against some of the tests is open and shut, in my opinion, and if nobody has spoken up to defend them, I think that it would be a good idea to call for a vote on dropping:
The Bloom Analogies Test (Bloom never published data on the norming and didn't respond to requests for data from the PsyCom)
The Harding Skyscraper (Harding did not provide data, either)
The Stanford-Binet (not enough ceiling for adults; low correlation of childhood scores with adult scores; too many high scores)
The Wechsler Adult Intelligence Scale (age corrected; insufficient ceiling)
The Wonderlic (key not secure; insufficient ceiling)
The W-87 (another of Harding's tests without proper data)
The armed services tests and the California Test of Mental Maturity are also candidates for removal from our list, but need more scrutiny before a decision is made about dropping them.
Additionally, there are a number of tests on our list about which I know very little except that membership officers of high-I.Q. societies don't see a lot of them and I haven't seen any data on their norming. This list includes:
The ACE
The Eysenck Test
The Otis (several forms)
The Pintner General Abilities Test
Wechsler tests other than the WAIS
Perhaps some of those reading this memo could volunteer to look into one or more of these tests and report back to the committee.
Turning to ``new business,'' we need to determine how to evaluate the three-part GRE scores which have been used by ETS for the past several years. One approach, on an interim basis, would be to use the Verbal and Quantitative scores from the three-part test and discard the Analytical scores. But the Verbal and Quantitative parts of the old test were longer, so we'd be losing some reliability here. We should find out what ETS has to say about comparing scores on the old and new GRE.
We must also evaluate a number of recent tests, including:
The LUTE
The PIAS
The Quest Test
The Titan Test
I want to do another norming of the PIAS; then I will send data around. The PIAS data set includes over 1400 people who took the test before the answers were published in Games. We are looking at a very similar situation with regard to the Quest Test; as it will soon be published in Omni, we can expect that several thousand people will take this test before the answers are released.
These people's scores are every bit as valid as scores made on tests for which the answers have not been released. If you have any doubt about this, take note of the fact that the answers to any particular SAT exam are released every year, and we accept the SAT. As to the question of the acceptance of the Quest Test itself for TNS admission puposes, I will have more to say below.
Chris Harding and the ISPE (separately) have also produced new tests, but they have not been normed and may still be in the process of revision.
When evaluating a new test, we must ask for information which will enable us to reach informed opinions, including, at a minimum:
A sample size of at least several hundred
A characterization of the shape of the distribution of scores
Distribution of scores by age, sex, membership in high-I.Q. societies, etc.
A scatter diagram of scores on the test in question vs. previous test scores
Correlation of the test in question with other tests (some of the better tests, at least, should correlate at or above the .5 level)
A complete description of the algorithm used to compute scores on the test in question, with a statistical analysis justifying this score computation method
Additionally, the test author should be willing to provide data for independent analysis by other investigators (with personal identifying information removed), or at least raw score/previous score pairs.
It is too early for this committee to concern itself with determining appropriate qualifying scores on the LUTE or the Quest Test, because we have none of the required data for either. Neither Chris Harding's score-computation algorithm for the Quest Test nor Dan LaCoste's tables for the LUTE is accompanied by any indication of how these numerical relationships were derived, nor has any explanation of a norming method been advanced for either test. In his letter of May 12, LaCoste indicated that he had a sample of only twenty, far too small for a statistically meaningful norming.
Patrick, I hope that you will put the vote on a qualifying score on the Quest Test on hold, in light of the above; this committee cannot make decisions without scientific examination of the evidence.
I was sorry to see remarks in some of the committee memos speculating on the answers to certain items on the tests under examination. The whole higher-I.Q. societies community depends on the availability of secure psychometric instruments. Please, fire away at the tests in general, say that you think a particular item is flawed if you like, but don't go into specifics that could compromise anyone's ability to come to the test in question innocent of hints or cheat sheets.
In his memo of June 29, Kjeld Hvatum said he had data on the PSAT/NMSQT available. Please send the data for distribution to this committee, Kjeld. I believe that this test is among those which have changed their scoring systems, so we'll need to be clear about what years are involved for a particular cutoff score if we accept this test.