While double coding is a standard process to ensure high quality data, it can (and should) also be used as a learning opportunity. In addition to comparing codes to see if the two coders are within one, there should be a rich debrief session. This is a time for observers to talk about the coding process, the evidence they saw, and how they used that evidence to arrive at the code they assigned. Even when coders have high levels of inter-rater reliability, I recommend that they have that conversation. Below is a list of just some of the questions observers can use to guide this conversation:
- Tell me about the process you use to assign codes
- What was challenging about this observation? What made it challenging?
- How did you arrive at your code? Tell me about your evidence
- What evidence did you have for that dimension?
- How did you sort the behaviors?
- What evidence did you find in the manual to support that code?
- How did you balance your code across activities? Across adults?
There are a couple of approaches to double coding. In one approach, one of the observers is considered to be the anchor coder and he or she takes the lead on the observation - including greeting the teacher, signaling the start and stop times, and signaling when to move around the room. In this model of double coding, the anchor leads the debrief session and his or her scores are the official codes for that session. In another model, both observers are equal and if their codes aren't an exact match, they talk until they reach consensus and the consensus codes are the official codes. One of the coders still needs to take the lead in terms of starting and stopping the observation.
These debrief sessions can really help build observers' knowledge of coding with CLASS. Hope this helps! Let me know if you have additional questions.
------------------------------
Sarah Hadden
Senior Advisor, Research and Professional Services
Teachstone
------------------------------