Helen Myers suggested I might have a look at the different mark schemes for the four awarding bodies for England and Wales. If you missed it, I previously blogged about AQA's spec and the specifications in general, but did not comment on mark schemes. Mark schemes are significant, particularly for the the Speaking and Writing papers. If you are bored by the nitty-gritty of mark schemes, look away now.
Context first: there are essentially two types of mark scheme, a points-based one which tends to be most objective (i.e. one correct response gets one point, with little or no room for ambiguity) and a level of performance mark scheme, used typically in MFL to assess conversation or written composition. The latter is bound to be somewhat subjective, even when exam boards do their utmost to make each level as explicit as possible. There are some quite technical issues here, in fact. Let's say you have a maximum mark of 15/15 for a task. Research shows that marking is affected not just by level descriptors, but by how you split up the marks e.g 13-15, 10-12, 6-9 etc. The placement of these gaps may have an influence on how examiners choose a mark - they might be reluctant to give a full mark of 15/15, for example or choose the middle mark of three options.
Awarding bodies are getting smarter about this type of thing so we would hope that they have taken into account such issues when they design their schemes.
OK, with that said, I'm going to attempt to compare the level of performance mark schemes for Foundation Speaking. In another blog I shall look at Writing. I shall make the assumption that the more objective point-based schemes, used mainly in Listening and Reading and not worth going into. Are there significant differences between awarding bodies?
SPEAKING (Foundation)
AQA
30 marks (50% of Speaking marks)
Comm 10 Range/Acc 10 Pron/Int 5 Spont/Flu 5
Pearson Edexcel
30 marks (50% of Speaking marks)
Comm/content 10 Interaction/Spont 10 Linguistic Knowledge/Acc 10
OCR
20 marks (still 50% of Speaking marks)
Comm 10 Language 10
Eduqas
30 marks (50% of Speaking marks)
Comm/Interaction 10 Range 10 Knowledg/Acc 5 Pron/Int 5
Notes
Pearson Edexcel marks pronunciation and intonation as part of Communication. I understand this, but have to say I like to give a separate mark for quality of pronunciation (AQA and Eduqas do so). I have heard many candidates from centres who communicate a lot in a poor accent. On balance, I still like to reward a good accent if I hear it, but I do see the opposing argument. Communication should be the key factor.
Let's compare specific level descriptors towards the middle of the range for Conversation (5-6 marks out of 10). I have marked in bold certain phrases for comparison
OCR's Language mark for 5-6/10
Simple language is generally accurate. Errors occasionally impede communication. Some variety of vocabulary and structures, appropriately used in places. Repetition evident. Limited attempt at rephrasing and repair strategies
Pronunciation and intonation impede communication in places
AQA's Range and Accuracy descriptor for 5-6/10 is:
Basic conversational language which uses simple structures and vocabulary and may often be repetitive. There is little or no success in making reference to past or future events. There are likely to be frequent errors, which sometimes impede communication.
Is OCR's descriptor tougher? Does the reference to past and future in AQA's descriptor make it easier to assess a candidate and for teachers to prepare them?
In the case of Eduqas Range and Accuracy/Knowledge are marked separately. For 5-6/10 mark for range:
Is able to use a limited range of vocabulary
Is able to substitute words or phrases and occasionally uses a past or future tense
For Accuracy/Knowledge 3/5:
Shows some knowledge of grammar
Manages to convey some messages despite error
As a teacher marking conversations I find those descriptors too short and vague. Some knowledge? "...despite error" - how much? In general I would have some issues with the Eduqas descriptors. They make a lot of use of the word "spontaneity". How easy is this to identify when most answers are pre-learned anyway, especially by weaker candidates? Have a look for your self here (p.116). Compare with AQA'a descriptors here (p.94). I would find AQA'a easier to apply. Fortunately, examiners will be doing the job from 2018!
OCR's mark scheme is here (p.33). Pearson Edexcel's is here (p.126).
The Pearson Edexcel Linguistic Knowledge/Accuracy (same as Range/Acc) for 5-7 marks out of 10.:
Uses a narrow range of high frequency vocabulary and grammatical structures Some accurate language but with many basic errors Attempts to use tenses and time frames to refer to past, present, and future events with a little success
I have marked in bold points where you may compare with the descriptors above. Pearson Edexcel refer specifically to past and future tense (not or, as with AQA and Eduqas). This may imply AQA's and Eduqas's are marginally easier. OCR require past, present and future for 7-8 marks - this is inconsistent with the others and slightly harder.
Now, many teachers may find the differences I am pointing out as quite minor ones and that, in any case, once exams have been marked, grades are later fixed at meetings, with Ofqual overseeing similarity of standard setting between boards. So any conclusions about relative difficulty may be somewhat moot.
Ofqual will be looking at these issues, going through this material with a fine tooth comb to ensure comparability and clarity, so there may be changes made to these drafts. Ofqual have been known to reject draft specs, but the exam boards will have done their best to avoid this happening.
Ultimately, when departments make a choice of exam board they should be pretty confident that there is broad similarity in standards and content, but enough subtle difference to provide some genuine options.
Next blog: comparing Writing mark schemes.
Wake up at the back.
Context first: there are essentially two types of mark scheme, a points-based one which tends to be most objective (i.e. one correct response gets one point, with little or no room for ambiguity) and a level of performance mark scheme, used typically in MFL to assess conversation or written composition. The latter is bound to be somewhat subjective, even when exam boards do their utmost to make each level as explicit as possible. There are some quite technical issues here, in fact. Let's say you have a maximum mark of 15/15 for a task. Research shows that marking is affected not just by level descriptors, but by how you split up the marks e.g 13-15, 10-12, 6-9 etc. The placement of these gaps may have an influence on how examiners choose a mark - they might be reluctant to give a full mark of 15/15, for example or choose the middle mark of three options.
Awarding bodies are getting smarter about this type of thing so we would hope that they have taken into account such issues when they design their schemes.
OK, with that said, I'm going to attempt to compare the level of performance mark schemes for Foundation Speaking. In another blog I shall look at Writing. I shall make the assumption that the more objective point-based schemes, used mainly in Listening and Reading and not worth going into. Are there significant differences between awarding bodies?
SPEAKING (Foundation)
AQA
30 marks (50% of Speaking marks)
Comm 10 Range/Acc 10 Pron/Int 5 Spont/Flu 5
Pearson Edexcel
30 marks (50% of Speaking marks)
Comm/content 10 Interaction/Spont 10 Linguistic Knowledge/Acc 10
OCR
20 marks (still 50% of Speaking marks)
Comm 10 Language 10
Eduqas
30 marks (50% of Speaking marks)
Comm/Interaction 10 Range 10 Knowledg/Acc 5 Pron/Int 5
Notes
Pearson Edexcel marks pronunciation and intonation as part of Communication. I understand this, but have to say I like to give a separate mark for quality of pronunciation (AQA and Eduqas do so). I have heard many candidates from centres who communicate a lot in a poor accent. On balance, I still like to reward a good accent if I hear it, but I do see the opposing argument. Communication should be the key factor.
Let's compare specific level descriptors towards the middle of the range for Conversation (5-6 marks out of 10). I have marked in bold certain phrases for comparison
OCR's Language mark for 5-6/10
Simple language is generally accurate. Errors occasionally impede communication. Some variety of vocabulary and structures, appropriately used in places. Repetition evident. Limited attempt at rephrasing and repair strategies
Pronunciation and intonation impede communication in places
AQA's Range and Accuracy descriptor for 5-6/10 is:
Basic conversational language which uses simple structures and vocabulary and may often be repetitive. There is little or no success in making reference to past or future events. There are likely to be frequent errors, which sometimes impede communication.
Is OCR's descriptor tougher? Does the reference to past and future in AQA's descriptor make it easier to assess a candidate and for teachers to prepare them?
In the case of Eduqas Range and Accuracy/Knowledge are marked separately. For 5-6/10 mark for range:
Is able to use a limited range of vocabulary
Is able to substitute words or phrases and occasionally uses a past or future tense
For Accuracy/Knowledge 3/5:
Shows some knowledge of grammar
Manages to convey some messages despite error
As a teacher marking conversations I find those descriptors too short and vague. Some knowledge? "...despite error" - how much? In general I would have some issues with the Eduqas descriptors. They make a lot of use of the word "spontaneity". How easy is this to identify when most answers are pre-learned anyway, especially by weaker candidates? Have a look for your self here (p.116). Compare with AQA'a descriptors here (p.94). I would find AQA'a easier to apply. Fortunately, examiners will be doing the job from 2018!
OCR's mark scheme is here (p.33). Pearson Edexcel's is here (p.126).
The Pearson Edexcel Linguistic Knowledge/Accuracy (same as Range/Acc) for 5-7 marks out of 10.:
Uses a narrow range of high frequency vocabulary and grammatical structures Some accurate language but with many basic errors Attempts to use tenses and time frames to refer to past, present, and future events with a little success
I have marked in bold points where you may compare with the descriptors above. Pearson Edexcel refer specifically to past and future tense (not or, as with AQA and Eduqas). This may imply AQA's and Eduqas's are marginally easier. OCR require past, present and future for 7-8 marks - this is inconsistent with the others and slightly harder.
Now, many teachers may find the differences I am pointing out as quite minor ones and that, in any case, once exams have been marked, grades are later fixed at meetings, with Ofqual overseeing similarity of standard setting between boards. So any conclusions about relative difficulty may be somewhat moot.
Ofqual will be looking at these issues, going through this material with a fine tooth comb to ensure comparability and clarity, so there may be changes made to these drafts. Ofqual have been known to reject draft specs, but the exam boards will have done their best to avoid this happening.
Ultimately, when departments make a choice of exam board they should be pretty confident that there is broad similarity in standards and content, but enough subtle difference to provide some genuine options.
Next blog: comparing Writing mark schemes.
Wake up at the back.
Comments
Post a Comment