Examiner (Part V): Questioning the Questions

Last night, I had an interesting (though brief) discussion with the Hierophant. He asked, if I recall correctly, which of the questions on the list, in my opinion, would I leave out.

In some ways, the gurus of epistemology (or the theory of knowledge, as some prefer) would say that it depends on your world-view, your assumptions, your tolerance for relativity and your ideas of the underlying truth of the world. (If you think there is such a thing at all, that is.)

To me, however, all that is too much. I am a simple man in many ways. Faced with such an enormous question, I tackled the enormity head-on by falling back (haha) on the comforting trinity of validity, reliability and utility.

I therefore applied the simple idea that a question in any form of assessment must display all three traits. A question must test what it's supposed to test, it must return the same score (or grade, or mark, or examiner's response) for the same answer each time, and it must be do-able within the expected limit of the examinee's education while eliciting a sufficient response.

In my opinion (which I confess is that of a young hothead who has only a fair amount of experience at this), I would take out questions 2, 5, 6, 8 and 9. I'd keep the rest, which seem to me to be pitched at the right level, target, competency and scope. Remember, this is only my opinion, and perhaps shows nothing more than the inner workings of my own brain.

What's wrong (if anything) with these questions in my view?

Let's have a look at question 2: "Examine the ways empirical evidence should be used to make progress in different areas of knowledge."

The words that grab me here are 'examine', 'should', and 'progress'. It's a very difficult question because it places too much responsibility in the hands of the examinee. For a start, any attempt to 'examine the ways' would be infinite in scope; the examinee therefore has to do a solid enough survey within 1600 words or so, being careful to cover all the major kinds of ways. Secondly, 'should' requires the examinee to prescribe rather than describe; even 'can' or 'might' or 'could' would be easier than this, because now the candidate must establish sufficient grounds for his prescription. Thirdly, 'progress' — always a difficult word to define, and certain to be contentious in many disciplines. When you combine all three, difficulties, it is not a question I'd want to tackle in 1600 words.

Now take a look at question 6: "'All knowledge claims should be open to rational criticism.' On what grounds and to what extent would you agree with this assertion?"

I'm immediately a little suspicious of the 'rational criticism' part. The problem of course is that there are as many kinds of valid reasoning as there are disciplines that define reason. Should a claim in one area of knowledge be open to rational criticism based on another area of knowledge? Simple example: should scientists argue theological claims? This doesn't quite kill the question, but it does require the candidate to worry about what 'rational' means (and no, it doesn't necessarily mean 'reasonable', 'reasoning' or 'logical'), and to a lesser extent, 'open'. The examinee must also lay out the grounds for his position, which is not an easy thing to do, since such a laying-out is also a knowledge claim...

Question 8 is this: "'People need to believe that order can be glimpsed in the chaos of events' (adapted from John Gray, Heresies, 2004). In what ways and to what extent would you say this claim is relevant in at least two areas of knowledge."

You could write a book about this. Gleick and Talib (and to a lesser and rather skewed extent, Dawkins) are among the many people who have already done it. The crux is 'need'. Is it really a necessity? Is it biologically hard-wired into us? Is something that is hard-wired a need or a drive? And what is meant by 'chaos' — do we use a mathematical, social, or aesthetic definition? Ho ho, the student attempting this is like the curators of large museums such as the Smithsonian, who from the largest collections of specimens in the world must then decide what to display in order to best meet the unknown predilections and tastes of myriads of visitors. Not easy, and possibly unfair.

I've already addressed question 5 and question 9 elsewhere, so we'll stop there. Again, I must say that the five questions I've highlighted are questions I'd have qualms about either setting or doing, as examiner or candidate. This is not at all to say that I have official standing or that my opinions are valid; but this is what I think, and I'd be grateful for any input.