NSF Awards: 2016849
The Automated Collaboration Assessment Using Behavioral Analytics project is developing an instructional teaching tool to assist teachers’ instructional assessments of students’ collaboration quality in science. To achieve the goal of developing a reliable technological innovation that can be applied in various instructional settings with diverse populations, we seek to obtain a fundamental understanding of collaboration. We present evidence on collaboration and address how such findings contribute to current and future research on student collaborative behaviors and interactions and teaching practices that promote and support collaboration. We developed a multi-tiered Collaboration Conceptual Model based on contemporary learning sciences research and field research to identify and characterize nuanced collaboration behaviors and interactions among diverse participant groups as they participate in our designed collaboration task series. This multi-tiered conceptual model starts with individual collaboration behaviors (e.g., physical behaviors and utterances) which are gradually aggregated into the group collaboration dynamics and qualities (e.g., interdependence, division of intellectual labor, etc). We applied the conceptual model on collected data to determine how well our model aligned with authentic collaboration. We conducted mixed-method analyses to examine specific collaboration patterns that exist in our current sample and how individual collaboration behaviors relate to the collaboration outcomes (e.g., individual attitudes towards collaboration and task performance by group). Our research will inform the work of education practitioners, formal and informal educators, and researchers on gaining a comprehensive understanding of collaboration and enhance our collective understanding of how to promote meaningful and productive collaboration experiences for all learners in science.
Barry Fishman
Professor
This is such interesting work! I really admire the way you have combined different sets of theories with a variety of observation and analytic techniques to understand collaboration.
One thing I'm curious about is what you see as the relative "value added" by the behavioral analytics, since so much of the variation in outcomes (collaboration quality) seems related to student confidence in science.
I'm also curious about the bodily movements. I know this wasn't the focus of this video, but the gesture and body alignment tracking figure prominently in the images shown. Given the small variations in how different groups of students are aligned/seated, have you observed any differences that are meaningful for collaboration quality?
Nonye Alozie
Leiny Garcia
Nonye Alozie
Senior Education Researcher
Hi Barry! Nice to hear from you again! It was nice to see you at AERA, btw.
Thank you for your recognition of the theoretical foundation of this project. This is a great question, Barry. It is something that we are really digging into this year.
One of the things we know about collaboration research is that speech and discourse is a main component for how people understand the way people work together. While there are some studies that look at prominent gestures and behaviors that are associated with collaboration, there aren't a lot of studies that help us understand the way people interact with each other as part of group work and how they are associated with knowledge building, shared goals, shared plans, etc. we also know that a big part of what we say isn't just in the words but it's also in the mannerisms and the gestures that our company the words. So, we want to look deeper into that. Steinbock’s paper talks about the relational space of collaboration- which can be looked at from a behavioral lens. Something that we're looking into right now is how gesture and speech work together as participants communicate their ideas and negotiate knowledge and groups. Understanding behavior is part of what Steinbock calls being mindful of process which is where you attend to and develop the social relational space during- in addition to the subject matter development and negotiation. On a very practical note- we use vision to help us make sense of what is being said, and we are trying to understand how to potentially support teachers in using their resource of sight (in a loud and busy classroom, for example) when managing a classroom with multiple groups working simultaneously.
In terms of the small variation, yes. There are small variations that are present. We performed an extensive literature review to help us create an architecture of behaviors and a coding rubric to identify and capture those behaviors. We are using machine learning to help us organize them and determine how they relate to one another and whether there are indeed differences.
I feel like I am missing something in this questions- let me know.
Nonye Alozie
Ateng' Ogwel
Julie Harrison
Andresse St Rose
Director of Educational Research and Evaluation
Collaboration is such a key skill and practice in school, at work, and at play that I am glad to see this study! What age group(s) or grade level(s) are participating in the study, and is your conceptual model specific to this age group or do you think it could apply to collaboration across a broad age range- with adult learners for example?
Nonye Alozie
Kemi Ladeji-Osias
Nonye Alozie
Senior Education Researcher
Hi Andresse. Thank you for your question. For the study we presented, we worked with both adults and students. Our conceptual model for collaboration was initially made for middle school age students and was refined as we looked at more and more data. When we collected data from adults in online spaces, we revised our model and corresponding rubric to match the adults and the online work platform that we designed. We found that while there were some changes, there were not as many differences that we expected to see. There is some literature (I can't remember at the moment) that says that after 14 or so, behaviors start to even out and become more consistent. It is also worth mentioning that the study was done in a lab setting, so the students and adults were self selected and were more likely to behave similarly. We have yet to try this in authentic work spaces.
Nonye Alozie
Kemi Ladeji-Osias
Andresse St Rose
Julie Harrison
Very interesting work to measure CPS! We are working to do similar multimodal measurements of student collaboration within iSAT. My question is related to Barry's above. So much of the CPS measurement literature focuses on assessing communication data. Could you speak a little bit more on the raw data used to generate the insights presented in this video? e.g., How were students' perceptions of their collaboration measured?
Nonye Alozie
Nonye Alozie
Senior Education Researcher
Hi Julie, Thank you for your question. I wrote a novel in response to Barry- so please take a look at what I wrote and see if you have an additional questions. I will be happy to expound on anything. Arif Rachmatullah was the lead analyst for the student perception work and developed the concept behind that. I will invite him to respond to your question.
Nonye Alozie
Julie Harrison
Arif Rachmatullah
STEM Education Researcher
Hi Julie, thank you so much for your question. We used a survey after collaboration for students to reflect on their participation. We selected, modified, and re-validated the items developed by Dewiyanti et al. Please let us know if you are interested in these survey items; we are happy to share them with you.
Nonye Alozie
Julie Harrison
Julie Harrison
Hi Arif, thanks for your reply. I am definitely interested in these survey items. While we continue to build out more behavioral metrics for collaborative problem solving at iSAT, we are looking to self-report items in the mean time. If you wouldn't mind sending along the survey, my email is julieharrison@gatech.edu. Thank you!
Nonye Alozie
Ekundayo Shittu
The effort to measure/assess the impact of collaboration is commendable. I am curious to know whether there were some unexpected outcomes especially instances where collaboration was negatively correlated with group performance. I voted! Thank you.
Nonye Alozie
Nonye Alozie
Senior Education Researcher
Hi Ekundayo. Thank you for your vote! This is such a great question. Measuring collaboration is so difficult- as I am sure you know, it is the hardest type of group work out there. We really had to break things down into smaller parts and look at things separately. We are in the process of bringing those parts together to see how they interact. Some of the surprising things that we saw last year were that higher completion rates of tasks among groups did not necessarily translate into better collaboration quality. We often see collaboration turn into more competitive sports in classrooms, where students are racing against the clock. With this, we saw the quality of their group artifacts were not affected by the quality of their collaboration- this was surprising. This was based on a small sample size. We decided to dig deeper into this by looking at the features of the tasks/activities to determine whether there are certain types of tasks that correlate with certain types of collaboration qualities. This is still being investigated. I will invite Hui Yang, the other lead analyst on the behavioral roles to say more about what she is finding there.
Nonye Alozie
Arif Rachmatullah
STEM Education Researcher
Hi Ekundayo, adding to what Nonye has said, we actually have some findings related to the relationship between groups' mixed-gender status and the artifact quality the groups generated. We hypothesized that mixed-gender groups would be a positive predictor of artifact quality; however, the results showed that it was not a significant predictor. The only group demographic factor that significantly predicted artifact quality was group ethnically diverse status (whether the group consisted of members from different ethnicities). Again, these findings were still based on small sample size, and we are still collecting more data to obtain a more robust statistical model. And thank you so much for the vote!
Nonye Alozie
Hui Yang
Education Researcher (STEM & CS)
Thank you for watching our video. We would love to hear your feedback and questions about our projects. In the meantime, to our formal and informal educators, we are also interested in getting your thoughts on the questions below:
Nonye Alozie
Ilana Horn
this is super interesting. we are thinking about collaboration on our project as well, but we are interested in how teachers make sense of fostering and supporting it. i see a lot of point of contact between our project.
one question: I was not clear how the visualizations of kids' bodies was being used in this project? can you say more about that?
Nonye Alozie
Hui Yang
Education Researcher (STEM & CS)
Hi IIana, thank you for your comment. The visualization of students' collaboration behaviors that you saw in our video is a part of our behavioral analytic technology for analyzing students’ collaboration quality. Our team developed a process to stratify our collaboration conceptual model for developing the machine learning models in our automated collaboration assessment tool which could analyze students' collaboration behaviors (including the body movements) and provides feedback on their collaboration quality. My colleague Anirudh Som, the task lead for the tool development, could provide more details on the development of the visualizations of students’ bodies.
Nonye Alozie
Nonye Alozie
Senior Education Researcher
Hi there- thanks for your question. I just want to add to Hui's response. To use of the visualization that you saw give a multidimensional view of the all of the students so that the machine learning models can identify movements from different angles. The human annotators were also able to see different angles of the students when coding the videos. Those visualizations provide a more detailed account of what students are doing so that the behavioral analytics can have more to work with.
Nonye Alozie
Janet Coffey
Program Director, Science Learning
I agree with others that this is really interesting work. Thank you so much for sharing - and for your thoughtful responses. I appreciate the attention to behavioral analysis. Were you able to derive any insights from analysis of movement and gesture or other forms of participation to degrees of science confidence? And, at this point from this work, are you able to hypothesize possible ways to structure collaborative group work - aside from group composition - in order to help contribute to and shore up science confidence? Or does that measure really align with confidence in understanding of the specific content (as opposed to one's ability to figure something out, for example)?
Nonye Alozie
Hui Yang
Education Researcher (STEM & CS)
Hi Janet, thank you for your question. The current work which Arif has been leading suggests two types of group configuration that can be utilized to promote the quality of collaboration. The first configuration is when all students in a group have relatively high levels of science confidence with relatively equal chances to contribute to the collaboration process. The second configuration is when group members have different levels of science confidence and different chances to participate during collaboration. In this situation, group members who have higher science confidence levels have more opportunities to clarify concepts to members with lower science confidence. Arif can provide more details on these findings if you would like to learn more.
Nonye Alozie
Arif Rachmatullah
STEM Education Researcher
Hi Janet, thank you so much for your question. I want to add to Hui’s response, especially related to your question about the analysis of movement and gesture. We haven’t yet analyzed the relations between science confidence and types of movements/gestures. One of our in-progress work is looking into the relationship among types of gestures (pointing, representational, working), types of talk (explanation, questioning, suggesting, etc.), and the groups’ artifact quality. We’re finding fascinating insights based on these relationships, which actually bring in another factor (i.e., task features). Our preliminary findings, as well as literature, seem to show that students’ behavior and movements tend to be dependent on the tasks, such as whether the task is abstract enough so that talk only does not afford students to express all their ideas, so they need to offload the thoughts into gestures. We’re still digging into this topic and cannot wait to share our findings with others.
Nonye Alozie