It All Comes Down To This

Every year, it all comes down to this, that one's life and its accomplishments and decorations are reified into a single number. That number, no longer satisfied to be half a right angle or the answer to life, the universe, and everything plus three, is the refined prism-exudation of enlightenment. Or so it seems.

It is only the beginning, young Jedi Wyverns. If you truly believed, as your forerunners did, that the best is yet to be, you would not be so focused on just this one point of the present, soon past and forgotten.

But that is life, nor are we out of it. For where it is, there we must ever be. Sorry, Kit.

Terms of Irritation

I'm not so young now, and not so prone to being lured into energetic disputation. However, I am sometimes irritated enough to fire off broadsides. Occasionally, I have been known to set satchel demolition charges around disputants' positions and blow them up while laughing like a maniac.

Of late, I've been most irritated by the terms 'alternative assessment' (alternative to what?), 'holistic assessment' (see elsewhere in this blog), and 'authentic assessment' (should we be giving inauthentic assessments?). These are silly terms.

Proponents of the first term say it is 'alternative to traditional'. Yeah, and when it becomes established, it will -be- traditional and probably the currently-traditional will have become alternative, no? I am reminded of the so-called 'alternative to practical assessment' which we were offered, and which later became the main mode of 'practical' assessment worldwide.

Proponents of the second term are mostly unable to say whether it is 'assessment of the holistic' or 'assessment that is holistic' that they mean. The former is not possible (the best map is 1:1, after all) and the latter is only slightly more plausible. Unless it means (as it often really does) assessment that offers a quick-and-dirty thumbnail sketch of whatever is supposed to be assessed.

Proponents of the last term are equally unable to say what 'authentic' means. Surely very few people use 'false assessment' or 'fake assessment'? Some change this to 'performance-based assessment', until I point out that 'performance-based' simply means 'based on carrying out whatever was supposed to be carried out'. Which of course says nothing about what is being assessed and how.

Times like this make one wish that education was not so much about hand-waving and vague platitudes, and more about hard thinking and ruthless disputation. Or at least, values. Ho ho ho.

Kinds of Assessment (Part II)

Of course, you can always consider other attributes of assessment when differentiating between the different kinds. You can consider the kind of assessor, the perspective of assessment, and the intention behind assessment, for example. This last is quite different from mere purpose.

1. Differentiated by assessor

1.1 Self: if the assessor is the person or institution being assessed, there is of course an opening for charges of bias. What people don't realise is that there is almost always bias; the real problem is groupthink or inability/refusal to resist bias.

1.2 Other, human: if the assessor is another human, then assessment is subject to indeterminate bias. No two humans assess exactly the same way; even the same human will assess differently under different conditions.

1.3 Other, machine: at least, machines tend to be consistently biased.

2. Differentiated by perspective

2.1 Internal: if the assessment is carried out within the same institution or group, the advantage is insider knowledge leading possibly to more useful, detailed, and relevant assessment — the disadvantage is insider knowledge leading to the exact opposite.

2.2 External: if the assessment is carried out from outside the institution or group, the advantage is that it looks more objective — the disadvantages are that lack of objectivity (if present) is harder to discern, and that feedback (if required) tends to be vague and less useful.

2.3 Subjective: if the assessment requires a personal standpoint (i.e. the assessor is the subject and the assessed is the object) then obviously all assessors will differ slightly in standpoint.

2.4 Objective: if the assessment has an enforced, pre-determined, external benchmark with no subjective choice required of the assessor, and the assessment is mechanical in a deterministic sense, then it is very reliable and accurate. Only the validity of the assessment is likely to be problematic. Note that the more realistic or complex the assessment is, the more unlikely a valid objective test can be made.

3. Differentiated by intention

3.1 Quantitative justification: some assessments are designed to convert a complex concept into simply measurable numbers. This is a highly debatable, occasionally politically-motivated practice.

3.2 Data consolidation: some assessments are designed to collect potentially useful data, and the candidate is not told that this data may be used for purposes beyond the explicit ones. This is an ethically dubious practice.

3.3 Normative ranking: some assessments are made so that a convenient ranked list can be made of all assessed candidates. Since assessments are of limited dimensions and are not supposed to contain interpretation (that's for evaluation), this practice is philosophically dubious.

3.4 Other intentions: there are as many intentions behind assessments as there are forms of assessment, but most of them lead to gratification, money, power or sex (just like crimes do). It's quite possible that assessment might lead to none of these, but if you think about it, society is generally complicit in using assessment in such ways. Sad.

=====

Well, I hope you who are reading this enjoyed my somewhat offbeat take on assessment. Some of it, I'm aware, was more true than we pretend; some of it less true than we believe. Most people wouldn't know how to assess assessment, though. Heh.

Kinds of Assessment (Part I)

There's way too much jargon around, and it confuses the stakeholders in education. If you confuse the stakeholders too much, they might misstake the vampires and die horribly. Heh.

And so, here is my convenient guide to kinds of assessment.

1. Differentiated by benchmark type

1.1 Criterion-based assessment: this type involves an assessor scoring the candidate assessed on the basis of whether specific behaviours or accomplishments match expectations of the assessor's master document or reference, model or exemplar, or expert judgement.

1.2 Norm-based assessment: given a sufficiently-large sample of population and a clear ranking system that would justify relative rankings within that sample, the candidate is assessed in terms of performance relative to that sample and ranked in comparison to it.

1.3 Self-based assessment: the candidate is assessed relative to earlier performances, with the assessor deciding if the candidate has performed better or worse compared to herself, himself or itself.

2. Differentiated by instrument type

This kind has more kinds on heaven and earth than are dreamt of in our philosophy, Horatio. Essentially, we have many ways of assigning quantitative values to candidate performances. All have flaws, but some are more appropriate to specific situations than others. The most comprehensive is probably apprenticeship, where the candidate's work over a period of years is evaluated by a recognized master at the task-set/skill-set which the candidate is learning.

Some common instrumental approaches are 'performance-based' in which the candidate's behaviour leading to an outcome is examined, and 'outcomes-based' in which the candidate's final product is examined.

3. Differentiated by purpose

3.1 Normative assessment: the candidate is assessed in order to develop a larger population of assessed candidates so that the assessor can figure out what the typical or desirable performance should be. This is normally used when the desired outcome is not clearly anticipated or known.

3.2 Formative assessment: the candidate is assessed in order to develop performance by response to assessment-based feedback.

3.3 Summative assessment: the candidate is assessed in order to compile a complete evaluation of one or more dimensions of performance; the word 'evaluation' is used because the assignment of value relative to a context (e.g. an overall grade) is a necessary outcome of this type of assessment.

3.4 Assessment for learning: this superficially resembles formative assessment. The problem here is the term 'learning'. The performance is to be developed in the assumption that improved performance implies learning. It can be difficult to prove this.

3.5 Assessment of learning: this superficially resembles summative assessment. The problem here is again the term 'learning'. The overall assessment is evaluated, with the candidate's relatively good performance in a given context assumed to imply superior learning to those with relatively worse performance. It can be likewise difficult to prove this.

3.6 Diagnostic assessment: assessment carried out at a fixed point in a process to determine the present state of the candidate, and especially what course or regime is required for further development or progress. It can be used for formative assessment.

3.7 Predictive assessment: assessment carried out by visualising the likely trajectory of the candidate given a hypothetical future state and the present state. Used to determine future value of candidate. It can also be used for formative assessment. (Actually, a sufficiently creative teacher can use anything formatively. Or reformatively, if necessary.)

=====

The next post will deal with the deeper (and more sinister, perhaps?) ideas behind assessment. See last line in 3.7 above.

Assessment and Evaluation, Knowledge and Mind

People have different philosophies of assessment and evaluation. The kind of philosophy you have depends a lot on your theory of knowledge, and perhaps your theory of mind.

Here are some crude approximations. A theory of knowledge is your concept of what it means to know, and what it means for something to be (usefully or otherwise acceptably) called 'knowledge'. [Good starting link here] A theory of mind is your concept of what mind is, how we conceptualise mental states, and how we decide what these mental states are. [Good starting link here.]

It follows that if your idea of knowledge is ontological — derived axiomatically through reason but without 'real' evidence — then you would test for abstract knowledge. It is true, so all you have to assess is whether the candidate knows it, and perhaps whether the candidate knows why it is logically necessary to believe in it (e.g. by mathematical or theological proof from basic principles or axioms).

It also follows if your idea of knowledge is empirical — derived from observation and collation of 'real-world' data — then you would test for concrete knowledge. It is true only if it has reliable existence which can be detected by an observer. So you would need to assess whether the candidate can produce what is required (e.g. by making a pot, by swimming 50m, by making up a 1.0M solution of sulphuric acid).

Very often, we have conflated our idea of knowledge so much that we have no idea what to test, and the act of a candidate writing down an ontological proof is taken as empirical proof of reality. We thus evaluate the candidate as intelligent despite knowing full well that we can't tell. Which brings us to the theory of mind.

If you believe that mental states are not only apprehensible but comprehensible, it is quite likely you believe so by congruence. In other words, you believe that if a person does X, that person would be thinking much as you would be if you were doing X. Some people get this right more often than others, and they are said to be high-empathy or to have high EQ. Some people get this wrong more often than others, and we think of them as psychotic or low EQ (or otherwise deficient).

This is why when students do things that our tests don't rate highly, we think of them as not having appropriate intelligence or preparation or skill (or whatever). That's if we're being cautious and/or charitable. Many of us would just say, "This candidate is low-performing." (Or 'dumb', or 'stupid', except that nowadays we only think it but try not to say it.) Our concepts of high performance depend on how difficult it would be for us, the educators, to carry out. What's easy for us should be easy for those we have taught, right?

Wrong, of course. In my ramblings on this blog, I've pointed out the circularity of our ideas about intelligence and its vulnerability to subjective evaluation even when considered by experienced educators and scientists.

In effect, we need to stop thinking of intelligence as a scored function — high-scoring, low-scoring or whatever-scoring. We need to think of it as a) what jobs need to be done, b) what are the ways they can be done, c) does this person figure out such ways quickly? We can omit b) if we just want to see whether a person can get the job done, regardless of whether we are able to figure out how it might be done. We need also to think of it as d) can our candidate figure out what jobs need to be done?

You see, we have no ability to actually get into someone's head, short of telepathy. We cannot tell how cognitively brilliant someone is except by whether that person gets the job done fast. We can plausibly rate candidates on how convincing their products are to us, but that would be subjective — the more their thinking corresponds with what we think is good, the more points they get. All these are old criticisms.

Moving forward, I suspect what we should do is empower candidates by teaching them every tool we know, how to evaluate the usefulness of tools, and how to create their own tools. A lot of that teaching will comprise meaningless exercises. We must be honest and tell them that we don't know how valuable the education we're giving them is. When we are long gone, they will have to prove those tools in the crucible of reality, and our hope should be that they will survive it.

Assessment and Evaluation

In education circles, these two flap around like much-reduced versions of Odin's ravens. Yet they are not a symmetric pair, or even orthogonal. They are linked by differences that were once obvious and are now subtle, much as 'right' and 'wrong' are, but not quite.

Assessment (from Latin assidere = 'sit down beside', 'be continually present with', 'constantly attend to'), implies that you sit down beside somebody, look over their shoulder, and decide exactly how much they score. The purpose here used to be the determination (from Latin terminus = 'boundary') of income for tax calculations (from Latin tangere = 'to touch' and calculi = 'stones'), a practice known to this day as 'assessment'. I've deliberately stressed the Latin etymology here to point out that the Romans were very heavy on tax assessment and related practices, because that is how you run a great empire administratively.

Assessment is meant to be a purely quantitative exercise. You touch up someone, estimate (from Latin aestimare = 'appraise') his wealth, and assess his tax. When we assess students, all we're looking for is their quantitative achievement so we can compare them to a chart or table and place them appropriately.

Evaluation, on the other hand, comes from a venerable linguistic stem from which we also get 'valour', 'valency' and 'value'. Also from the Latin, evaluation is about determining the potency or worth of something in its social context. When we evaluate something, we aren't looking only at its obvious quantitative attributes, but at its potential and actual usefulness or quality when compared to everything around it.

When we evaluate students, therefore, we are looking at them in terms of how much skill and potential are revealed by what they have done and are doing. We are thinking of them qualitatively, in as many dimensions as necessary to figure out how valuable they are in their present and future context.

Thus, evaluation is to assessment as sculpture is to sketching — the former needs the latter more than the latter needs the former, but it is much more well-rounded. Evaluation gives assessment a greater purpose, although assessment itself can have its own purposes.

Here's an example. Think of a physical fitness test battery. You might want to include chin-ups (to evaluate upper body strength), standing broad jump (to evaluate one form of lower body strength), juggling (to evaluate coordination), a 2.4 km run (to evaluate basic aerobic fitness), and so on. The numbers you get here form an assessment, and the methods used to obtain the numbers are called assessment rubrics.

Now consider what the entire test battery tells us about the subject's overall fitness levels and what they mean in terms of employment prospects and potential job scope. (If you had a trained observer watching the candidate's approach to taking these tests, you might also have one kind of psychological data.) Put it all together to form a narrative about this person's capabilities, and you have an evaluation.

Evaluations are too complicated to have fixed rubrics. Instead, you tell the evaluator what you are looking for, and the evaluator decides how to put the assessment data together to tell you what you will find. It requires a much higher level of cognition, and the burden of responsibility (moral or otherwise) is also much greater.

In examinations, anybody can use a marking scheme and turn in a reasonable assessment of an examinee's performance. In fact, for the so-called objective tests, a machine can do the assessment.

However, not all who get 27/40 for an MCQ test got there the same way — some may well be unusual candidates of great promise, while others may be drones who studied hard and got only the obvious questions right. It takes a trained interpreter to tell you what exactly the pattern of right and wrong answers means about the candidate, and even then, without asking a student why they chose their answers, the evaluation is very thin and may not very helpful.

In short, assessment is all about 'what do we have' and evaluation is all about 'what does it mean to us'. Assessment is simple, evaluation complex. Assessment can be a science; if it is, then evaluation is an art. And I'm quite sure I will have more to say about them.

The Big Change

Some time ago, I noted an amusing change which I believe to be of great benefit to everyone concerned...

In recognition of the concerns raised by colleagues regarding [redacted] assessment, the prescribed titles will be split from November 2012 and a new set of 6 titles will be issued for each examination session. The appropriate set of titles for each session will be made available on [redacted] as follows:

• 1 March for the year of the November session
• 1 September for the following May session

Marking guidance for examiners has been severely restricted because of the use of one set of titles to service two examination sessions. This improvement will enable the senior examining team to compile detailed marking notes for each title thus providing greater support to the examining team.

What it does mean is a lot less work all round for those of us who work on such things. Wonderful!

TOKenism

Over the last few months, I've been helping students who are doing the International Baccalaureate Organisation's 'Theory of Knowledge' core subject. Along the way, I've heard from them the various things that their teachers and peers have had to say about the subject, some of which are wise, some of which are less so.

I suppose that in the manner of such exact musings, I should get to the point without drifting too much. Theory of Knowledge (TOK) as an IB core subject is not just about epistemology, although TOK is certainly that. Rather, the point for the student is both pragmatic as well as conceptual. It is practically important because the entire IB syllabus hinges around the question of how a practitioner knows that what he or she asserts is true (or at least, justifiable) and to what degree this is so, and the grades awarded often directly relate to this; it is conceptually important because education is all about knowledge, and if you don't know why you claim to know something, then you probably can't claim knowledge of it.

That leads to the practical use of knowing this. In the TOK core subject, a student is assessed in two ways: a written essay of about 1200-1600 words, answering the student's choice of one of ten offered questions (there's a sample list elsewhere in this blog); and a presentation that is offered to a panel of teachers who use rubrics to produce an internally-moderated score. Essentially, both of these assessment modes are designed to see how much a student knows about the pragmatic and conceptual importance of TOK, and whether that student can present this in a form that is sound and easily understood.

The problem I've faced is that a lot of the output I've seen is neither sound nor easily understood; or if one, not the other. I have no idea why some people can't help their students to do both. The ability to guide students in this is the difference between what I call TOKism and TOKenism. The former is the belief and capacity a teacher has in transmitting a love for knowledge that includes respect for truth and clarity in its presentation; the latter is the lip-service a person pays in acknowledging its importance while actually requiring things like aesthetic or moral prettiness in what the student offers.

I say this because I've seen for myself (having been a senior moderator), and heard from many witnesses and victims, the odd things that TOKenists look at when they claim to be assessing students' work in TOK. Not a few seem to be looking at literary merit and the ability to mount an argument, regardless of how clear or fair that argument is. Worse, some are looking at the quality of style of presentation rather than the content and structure of the presentation itself. I remember one panelist actually remarking, "Well it may be a good argument, very clear, but I just don't like the way he said it."

I guess that in a sense, the ideals of TOK teaching are somewhat injured by that kind of approach. Good clear arguments, covering the process of how somebody knows something and how that person can justify the claim of knowledge, are all that is needed. The introductory definitions must frame the arguments that follow, and the concluding statements must summarise those arguments while being closely derived from them. If this answers the question posed, then that is all that is required, in the main.

The rubrics do indeed award points for other more specific and technical qualities in the essay and presentation. But style never comes into it, and neither does the moral or social approval of the moderator or supervisor.

I look forward to seeing the grades awarded for the dozens of essays I've already read and commented on. It will be one way to sort the TOKists from the TOKenists. It's fortunate that the IBDP examiners themselves don't seem to be biased in this way. I suspect that some school cultures tend to build up groupthink, and that teachers from such schools tend to start believing that there's only one way to think (or that students ought to think).

Assessing Your Teachers

I was musing about such heavy matters when I came across this post courtesy of my IP Hosting agent.

Let me summarise the matter for you.

There are two kinds of assessment available, criterion-based assessment and normative assessment. In the former, you must meet concrete objectives (e.g. has demonstrated ability to stand on the head without other bodily support, sip hot black coffee and mark 25 ELA1 answer scripts without breaks while singing the Hallelujah Chorus from Handel's Der Messias backwards). In the latter, you are ranked against others who are put through equivalent trials and you score points (normally with little attention paid to where you score them as long as you score them).

Organisations such as the International Baccalaureate Organisation prefer the former. Other organisations (including some without websites) prefer the latter. (For those of you who have recently watched a certain Godfarthur video clip, you know what I mean.)

Some institutions concentrate on the production of grades, irrespective of how those grades are produced and what raw material these grades originate from. Some concentrate on other things either to obscure the grade data or to show that they can do plenty of other tricks as well. In such institutions, officers will be appraised on anything and everything; this is called 'open-ended appraisal'.

In some institutions where there is too much to appraise, the person who can present the most convincing case will net the booty. It doesn't always mean that the best teacher will get the largest bonus; if bonus funds are limited, even good teachers may be paid less. Furthermore, as teaching ability is a non-linear convex curve which plateaus off, it might be rapidly outstripped by the non-linear non-continuous functions available in other areas. How does one compare a gold in some violent territorial spheroid-transfer sport with a fair crop of distinctions from a once-idle class?

It strikes me that this 'profession' is one in which the direct clients have little structured feedback into the remuneration of their assigned officers. Rather, feedback from competing officers is sometimes used to assign bonuses. I use inverted commas around the word 'profession' because in many cases there is no local body which licenses teachers in a criterion-referenced approach. Lawyers, doctors and engineers have professional organisations which provide discipline and professional enhancement; teachers do not, and frankly would not welcome criterion-based assessment.

In my previous life, I once compared the assessment criteria of a large national healthcare institution with that of a large national educational institution. Here is a very brief summary.

Healthcare:

• A = international or regional reputation/standards met (lists follow) in clinical competence, clinical workload, teaching, research, and administrative ability
• B = regional to national, high impact output...
• C = good quality, regular professional output...
• D = meets departmental standards, some professional output...
• E = below departmental standards, poor output for time spent...

Having seen the full document, I am impressed at the amount of work done in the attempt to set criteria for grading.

Education:

• A = performs at two grades above current level in academic work, para-academic work and non-academic work
• B = performs at one grade above...
• C = performs at...
• D = meets requirements...
• E = does not meet requirements...

Having seen a huge sample of these documents, I am not impressed by the accuracy or effectiveness of many such documents. I am not alone, and so national education agencies have changed the format. I am still somewhat unmoved.

So, do you think we will be a better healthcare or a better education hub? And can you propose better ways of ranking teachers?