Having fun yet? While you are digging into the Standard 6 material, here are statistics that provide some information about standard 6.
In 2008-2010, QM studied inter-rater reliability or the level of agreement among peer reviewers conducting course reviews. The following standards had the highest levels of inter-rater reliability among the 40 standards:
• 6.5 The course components are compatible with current standards for delivery modes. (94%)
• 6.1 The tools and media support the learning objectives, and are appropriately chosen to deliver the content of the course. (91%)
• 6.4 Students have ready access to the technologies required in the course. (91%)
On the other end of the spectrum, the standard with one of the lowest levels of inter-rater reliability is:
• 6.7 The course design takes full advantage of available tools and media. (70%) The low level of inter-rater reliability indicates that peer reviewers have a broad range of interpretation for Standard 6.7.
Apparently, Standard 6.7 has a high level of interpretation. Your thoughts?
This comment has been removed by the author.
ReplyDeleteThese are impressive statistics!
ReplyDeleteI don't think well in numbers - but I was just reading about early findings in composition studies (1970s) that attempted to promote use of rubrics for grading and extremely poor inter-rater reliability was one of the issues brought up in support of scales and rubrics. And extremely is key here - the scores were literally all over the place, as scorers focused on different elements of successful writing and there were few common baselines/denominators.
So these numbers speak volumes even to a non-numbers person like myself!
I think the relative span of percentages is not at all startling. Indeed, 90s are stellar, 70s are still quite solid. A core reason for a larger response range for Standard 6 could be that concepts such as "full range" are a bit difficult to apply to a criterion focused on media. Available media change and expand constantly and, perhaps more importantly, some practitioners will opt to use some familiar or favorite tools/media and go into depth, while others will opt to use many different tools/media and either maintain this approach over time or "slim it down" based on polled learner preferences.
What do others think?
Thanks for posting!
-Sabine