RE: BJ IQ
SB,
I recently took two different IQ tests online. On the first test, I got two questions incorrect. I was pissed off about these questions, because I thought the author did not understand what he asked (it involved true-false-neither logic for subset relationships). After that test, I took another online IQ test. When the same type of questions came up on the second exam as the first, I used the incorrect logic from the first test author, and once again (since the second author got it right), those were the only questions I missed in the second IQ test. I wrote an angry letter to the first author asking him to consider the effects of poorly written questions or unreasoned answers.
I have my own experience writing an AP IQ test. A couple of years ago I was trying to learn to write PHP scripts. I threw together an AP IQ test to help me learn PHP. It had a lot of questions that were of questionable value, and vague or incorrect choices for the answers. Even though the design was poor, and the questions were not well thought out, it was clear that experts were getting very high scores and those who were not experts got scores that were much lower. So, it worked. However, the amount of vitriol the exam created made the effort a very unpleasant experience for me. Just like I was with the IQ tests I took, those experts who didn't get perfect scores on my exams were (rightfully) pissed at my own lack of care in designing the AP test.
In case you want to have a look, here is the link -- this may be a model for how NOT to write such an exam:
http://www.cardcounter.com/BJ_IQ/BJIQ.html
I would advise against designing such an exam. You will not please the experts with your exam, they will let you know everything that's wrong with your test. You will never get it right, because there are always subtleties that are deeper than you can imagine. The only value it will have is the value you get from its design and implementation.
With best wishes,
Eliot