1. Nonye Alozie
  2. Senior Education Researcher
  3. Automated Collaboration Assessment Using Behavioral Analytics
  4. SRI International
  1. Arif Rachmatullah
  2. https://www.sri.com/bios/arif-rachmatullah/
  3. STEM Education Researcher
  4. Automated Collaboration Assessment Using Behavioral Analytics
  5. SRI International
  1. Anirudh Som
  2. https://scholar.google.com/citations?user=dY2-WmIAAAAJ&hl=en
  3. Advanced Computer Scientist
  4. Automated Collaboration Assessment Using Behavioral Analytics
  5. SRI International
  1. Hui Yang
  2. https://scholar.google.com/citations?user=Nxz6FhIAAAAJ&hl=en
  3. Education Researcher (STEM & CS)
  4. Automated Collaboration Assessment Using Behavioral Analytics
  5. SRI International
Public Discussion

Continue the discussion of this presentation on the Multiplex. Go to Multiplex

  • Icon for: Barry Fishman

    Barry Fishman

    Facilitator
    Professor
    May 10, 2022 | 10:29 a.m.

    This is such interesting work! I really admire the way you have combined different sets of theories with a variety of observation and analytic techniques to understand collaboration.

    One thing I'm curious about is what you see as the relative "value added" by the behavioral analytics, since so much of the variation in outcomes (collaboration quality) seems related to student confidence in science.

    I'm also curious about the bodily movements. I know this wasn't the focus of this video, but the gesture and body alignment tracking figure prominently in the images shown. Given the small variations in how different groups of students are aligned/seated, have you observed any differences that are meaningful for collaboration quality?

     
    2
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
    Leiny Garcia
  • Icon for: Nonye Alozie

    Nonye Alozie

    Lead Presenter
    Senior Education Researcher
    May 10, 2022 | 10:48 p.m.

    Hi Barry! Nice to hear from you again! It was nice to see you at AERA, btw.

    Thank you for your recognition of the theoretical foundation of this project. This is a great question, Barry. It is something that we are really digging into this year.

    One of the things we know about collaboration research is that speech and discourse is a main component for how people understand the way people work together. While there are some studies that look at prominent gestures and behaviors that are associated with collaboration, there aren't a lot of studies that help us understand the way people interact with each other as part of group work and how they are associated with knowledge building, shared goals, shared plans, etc. we also know that a big part of what we say isn't just in the words but it's also in the mannerisms and the gestures that our company the words. So, we want to look deeper into that. Steinbock’s paper talks about the relational space of collaboration- which can be looked at from a behavioral lens. Something that we're looking into right now is how gesture and speech work together as participants communicate their ideas and negotiate knowledge and groups. Understanding behavior is part of what Steinbock calls being mindful of process which is where you attend to and develop the social relational space during- in addition to the subject matter development and negotiation. On a very practical note- we use vision to help us make sense of what is being said, and we are trying to understand how to potentially support teachers in using their resource of sight (in a loud and busy classroom, for example) when managing a classroom with multiple groups working simultaneously.

    In terms of the small variation, yes. There are small variations that are present. We performed an extensive literature review to help us create an architecture of behaviors and a coding rubric to identify and capture those behaviors. We are using machine learning to help us organize them and determine how they relate to one another and whether there are indeed differences.

     

    I feel like I am missing something in this questions- let me know.

     
    3
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
    Ateng' Ogwel
    Julie Harrison
  • Icon for: Andresse St Rose

    Andresse St Rose

    Facilitator
    Director of Educational Research and Evaluation
    May 10, 2022 | 12:10 p.m.

    Collaboration is such a key skill and practice in school, at work, and at play that I am glad to see this study! What age group(s) or grade level(s) are participating in the study, and is your conceptual model specific to this age group or do you think it could apply to collaboration across a broad age range- with adult learners for example?

     
    2
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
    Kemi Ladeji-Osias
  • Icon for: Nonye Alozie

    Nonye Alozie

    Lead Presenter
    Senior Education Researcher
    May 10, 2022 | 10:57 p.m.

    Hi Andresse. Thank you for your question. For the study we presented, we worked with both adults and students. Our conceptual model for collaboration was initially made for middle school age students and was refined as we looked at more and more data. When we collected data from adults in online spaces, we revised our model and corresponding rubric to match the adults and the online work platform that we designed. We found that while there were some changes, there were not as many differences that we expected to see. There is some literature (I can't remember at the moment) that says that after 14 or so, behaviors start to even out and become more consistent. It is also worth mentioning that the study was done in a lab setting, so the students and adults were self selected and were more likely to behave similarly. We have yet to try this in authentic work spaces.

     
    3
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
    Kemi Ladeji-Osias
    Andresse St Rose
  • Icon for: Julie Harrison

    Julie Harrison

    Graduate Student
    May 10, 2022 | 02:50 p.m.

    Very interesting work to measure CPS! We are working to do similar multimodal measurements of student collaboration within iSAT. My question is related to Barry's above. So much of the CPS measurement literature focuses on assessing communication data. Could you speak a little bit more on the raw data used to generate the insights presented in this video? e.g., How were students' perceptions of their collaboration measured?

     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
  • Icon for: Nonye Alozie

    Nonye Alozie

    Lead Presenter
    Senior Education Researcher
    May 10, 2022 | 11:00 p.m.

    Hi Julie, Thank you for your question. I wrote a novel in response to Barry- so please take a look at what I wrote and see if you have an additional questions. I will be happy to expound on anything. Arif Rachmatullah was the lead analyst for the student perception work and developed the concept behind that. I will invite him to respond to your question.

     
    2
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
    Julie Harrison
  • Icon for: Arif Rachmatullah

    Arif Rachmatullah

    Co-Presenter
    STEM Education Researcher
    May 10, 2022 | 11:36 p.m.

    Hi Julie, thank you so much for your question. We used a survey after collaboration for students to reflect on their participation. We selected, modified, and re-validated the items developed by Dewiyanti et al. Please let us know if you are interested in these survey items; we are happy to share them with you. 

     
    2
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
    Julie Harrison
  • Icon for: Julie Harrison

    Julie Harrison

    Graduate Student
    May 11, 2022 | 11:01 a.m.

    Hi Arif, thanks for your reply. I am definitely interested in these survey items. While we continue to build out more behavioral metrics for collaborative problem solving at iSAT, we are looking to self-report items in the mean time. If you wouldn't mind sending along the survey, my email is julieharrison@gatech.edu. Thank you!

     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
  • Icon for: Ekundayo Shittu

    Ekundayo Shittu

    Higher Ed Faculty
    May 10, 2022 | 04:06 p.m.

    The effort to measure/assess the impact of collaboration is commendable. I am curious to know whether there were some unexpected outcomes especially instances where collaboration was negatively correlated with group performance. I voted! Thank you.

     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
  • Icon for: Nonye Alozie

    Nonye Alozie

    Lead Presenter
    Senior Education Researcher
    May 10, 2022 | 11:14 p.m.

    Hi Ekundayo. Thank you for your vote! This is such a great question. Measuring collaboration is so difficult- as I am sure you know, it is the hardest type of group work out there. We really had to break things down into smaller parts and look at things separately. We are in the process of bringing those parts together to see how they interact. Some of the surprising things that we saw last year were that higher completion rates of tasks among groups did not necessarily translate into better collaboration quality. We often see collaboration turn into more competitive sports in classrooms, where students are racing against the clock. With this, we saw the quality of their group artifacts were not affected by the quality of their collaboration- this was surprising. This was based on a small sample size. We decided to dig deeper into this by looking at the features of the tasks/activities to determine whether there are certain types of tasks that correlate with certain types of collaboration qualities. This is still being investigated. I will invite Hui Yang, the other lead analyst on the behavioral roles to say more about what she is finding there.

     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
  • Icon for: Arif Rachmatullah

    Arif Rachmatullah

    Co-Presenter
    STEM Education Researcher
    May 10, 2022 | 11:46 p.m.

    Hi Ekundayo, adding to what Nonye has said, we actually have some findings related to the relationship between groups' mixed-gender status and the artifact quality the groups generated. We hypothesized that mixed-gender groups would be a positive predictor of artifact quality; however, the results showed that it was not a significant predictor. The only group demographic factor that significantly predicted artifact quality was group ethnically diverse status (whether the group consisted of members from different ethnicities). Again, these findings were still based on small sample size, and we are still collecting more data to obtain a more robust statistical model. And thank you so much for the vote!

     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
  • Icon for: Hui Yang

    Hui Yang

    Co-Presenter
    Education Researcher (STEM & CS)
    May 11, 2022 | 10:32 p.m.

    Thank you for watching our video. We would love to hear your feedback and questions about our projects. In the meantime, to our formal and informal educators, we are also interested in getting your thoughts on the questions below:

    • What are the benefits and challenges have you noticed about having collaboration activities in your classroom/programs? 
    • What kind of feedback do you usually provide to your students/learners when they work collaboratively? 
     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
  • Icon for: Ilana Horn

    Ilana Horn

    Higher Ed Faculty
    May 12, 2022 | 03:17 p.m.

    this is super interesting. we are thinking about collaboration on our project as well, but we are interested in how teachers make sense of fostering and supporting it. i see a lot of point of contact between our project.

     

    one question: I was not clear how the visualizations of kids' bodies was being used in this project? can you say more about that?

     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
  • Icon for: Hui Yang

    Hui Yang

    Co-Presenter
    Education Researcher (STEM & CS)
    May 12, 2022 | 05:23 p.m.

    Hi IIana, thank you for your comment. The visualization of students' collaboration behaviors that you saw in our video is a part of our behavioral analytic technology for analyzing students’ collaboration quality. Our team developed a process to stratify our collaboration conceptual model for developing the machine learning models in our automated collaboration assessment tool which could analyze students' collaboration behaviors (including the body movements) and provides feedback on their collaboration quality. My colleague Anirudh Som, the task lead for the tool development, could provide more details on the development of the visualizations of students’ bodies.

     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
  • Icon for: Nonye Alozie

    Nonye Alozie

    Lead Presenter
    Senior Education Researcher
    May 13, 2022 | 08:31 p.m.

    Hi there- thanks for your question. I just want to add to Hui's response. To use of the visualization that you saw give a multidimensional view of the all of the students so that the machine learning models can identify movements from different angles. The human annotators were also able to see different angles of the students when coding the videos. Those visualizations provide a more detailed account of what students are doing so that the behavioral analytics can have more to work with.

     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
  • Icon for: Janet Coffey

    Janet Coffey

    Facilitator
    Program Director, Science Learning
    May 13, 2022 | 02:34 a.m.

    I agree with others that this is really interesting work. Thank you so much for sharing - and for your thoughtful responses. I appreciate the attention to behavioral analysis. Were you able to derive any insights from analysis of movement and gesture or other forms of participation to degrees of science confidence?   And, at this point from this work, are you able to hypothesize possible ways to structure collaborative group work - aside from group composition - in order to help contribute to and shore up science confidence? Or does that measure really align with confidence in understanding of the specific content (as opposed to one's ability to figure something out, for example)? 

     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
  • Icon for: Hui Yang

    Hui Yang

    Co-Presenter
    Education Researcher (STEM & CS)
    May 15, 2022 | 08:51 p.m.

    Hi Janet, thank you for your question. The current work which Arif has been leading suggests two types of group configuration that can be utilized to promote the quality of collaboration. The first configuration is when all students in a group have relatively high levels of science confidence with relatively equal chances to contribute to the collaboration process. The second configuration is when group members have different levels of science confidence and different chances to participate during collaboration. In this situation, group members who have higher science confidence levels have more opportunities to clarify concepts to members with lower science confidence. Arif can provide more details on these findings if you would like to learn more. 

     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie
  • Icon for: Arif Rachmatullah

    Arif Rachmatullah

    Co-Presenter
    STEM Education Researcher
    May 16, 2022 | 12:00 a.m.

    Hi Janet, thank you so much for your question. I want to add to Hui’s response, especially related to your question about the analysis of movement and gesture. We haven’t yet analyzed the relations between science confidence and types of movements/gestures. One of our in-progress work is looking into the relationship among types of gestures (pointing, representational, working), types of talk (explanation, questioning, suggesting, etc.), and the groups’ artifact quality. We’re finding fascinating insights based on these relationships, which actually bring in another factor (i.e., task features). Our preliminary findings, as well as literature, seem to show that students’ behavior and movements tend to be dependent on the tasks, such as whether the task is abstract enough so that talk only does not afford students to express all their ideas, so they need to offload the thoughts into gestures. We’re still digging into this topic and cannot wait to share our findings with others.

     
    1
    Discussion is closed. Upvoting is no longer available

    Nonye Alozie