NSF Awards: 2019805
We present the results of a design sprint conducted at the NSF AI Institute for Student-AI Teaming (iSAT) where stakeholders, undergraduate students, graduate students, and researchers collaborated to innovate on the design of an artificially intelligent (AI) agent. We discuss the steps of the design process as well as ideas for the role of the AI that emerged from the collaborative effort.
iSAT is an interdisciplinary research community dedicated to transforming classrooms into more effective, engaging and equitable learning environments. We team with diverse groups of students and teachers to develop the next-generation of collaborative learning environments. Together we tackle one of the biggest challenges facing education today: How can one teacher create an equitable and collaborative classroom for all students? https://www.colorado.edu/research/ai-institute/
Dorothy Bennett
Hello, Was wondering how students related and reacted to these robots intervening in their group work? What roles did teachers find themselves playing? Thanks!
Rachel Dickler
Research Associate
Hello Dorothy! Thank you so much for your questions! We are just starting to take our next steps towards working with students to get their feedback on the implementation of the AI agent in their group. The exciting part about this work is that we are still deciding the form the agent will take (e.g., a robot embodiment, a disembodied voice, an avatar on a computer screen, a computer display, or other option). It will definitely be fun to see students' reactions to the robot embodiments and how the robots impact the students' experiences!
The AI agent will augment teacher instruction, but teachers will continue doing the amazing work that they are already doing in classrooms! Depending on the role of the agent, it may be able to help students with collaboration and understanding of content as well as display data to teachers. We aim for teachers to continue in their roles while the agent best supports teachers' particular needs. Our upcoming participatory design work with teachers will be critical for helping us to further define the exact role of the agent.
Pati Ruiz
Rachel Dickler
Research Associate
Welcome to the NSF National AI Institute for Student-AI Teaming (iSAT) video page! iSAT is imagining how AI-agents can help teachers and students during collaborative group work in classrooms. We are excited to read and respond to your questions and comments. We are particularly interested in your thoughts on the design sprint process and resulting AI-agent roles. We also welcome general discussions on the potential role AI-agents may play to support collaborative learning in classrooms.
Dorothy Bennett
Sounds interesting Rachel. I am curious to hear what you learn from the participatory design work. The role of agents in the classroom, I think, really needs to consider how teachers think of their role in instruction and what learning it can enhance.
Rachel Dickler
Research Associate
Thank you so much Dorothy, yes we completely agree! We look forward to sharing our results!
Chris Dede
Rachel, I'm glad to see your National AI Institute is participating in the STEM Videohall. Please take a look at our video: https://stemforall2022.videohall.com/presentations/2364 It would be great to find ways to collaborate
Julie Harrison
PhD Student
Chris, I am one of the co-presenters of this video--one of the Georgia Tech collaborators. I attended a talk you gave at GT a few months back. Your concept of teaching 'judgment' skills vs. 'reckoning' abilities has stuck with me and I think it is very applicable to the work we are doing at iSAT to teach and support collaborative problem solving. Glad to see another NSF AI institute represented--looking forward to seeing how your institute evolves. Thanks for stopping by our page.
Pati Ruiz
Chris Dede
Happy to stay in touch about this issue. Perhaps we can have a cross-institute discussion about it.
Dalila Dragnic-Cindric
Congratulations on this exciting work! In this video, Rachel Lieber mentioned exploring metaphors used to describe AI's role in classrooms. Can you share a bit more about what specific metaphors were being used, and did you see a difference in who was using which metaphor? For example, did you find that teachers and students were using different metaphors?
Rachel Dickler
Research Associate
Thank you so much Dalila! That's an excellent question! We started off the design sprint with several different ideas for how an AI agent could support teachers and students, and then narrowed our focus to three specific metaphors:
We had groups of undergraduate students, graduate students, post-doctoral researchers, and early career scholars who participated in the design sprint event to create these metaphors. The designs of the metaphors were heavily influenced by feedback we had received from teachers and students including through the iSAT Learning Futures Workshops. We are looking forward to bringing these ideas back to teachers and students for feedback in the upcoming months! It will definitely be interesting to see which components are most helpful to different teachers and students depending on their specific needs.
Judi Fusco
This is great. I look forward to talking more and we are happy to share anything you want on Educator CIRCLS. (We'll tweet about your video, too!) We think a lot about how AI systems and teachers might partner. And of course, relevant for the AI Engage Institute, too.
Rachel Dickler
Chris Dede
I hope we can build collaboration among the three Institutes focused on education. Some of the AI-agents we are building relate to community building.
Rachel Dickler
Julie Harrison
Judi Fusco
Rachel Dickler
Research Associate
Thank you so much Judi and Chris! Yes that would be great!
Judi Fusco
Nathan Holbert
Associate Professor
Love this idea of using AI to support collaboration between not just machines and humans, but also between humans and humans! As a research community our initial efforts around AI in education is aimed at building an answer machine. But of course we know the best teaching isn't just having the right answers, it's supporting learners in a complex process!
About the design sprint. Were there any surprises that emerged or any insights from a group or team that suggested some unexpected but exciting directions for AI?
Rachel Dickler
Research Associate
Hi Nathan! Thank you so much, we are enjoying deeply exploring the ways in which AI can support collaboration between humans (particularly between middle and high school students)!
One critical insight that emerged from the design sprint was the idea of using AI to support “non-normative” forms of collaboration (as part of the Community Builder metaphor for an AI Agent). Non-normative forms of collaboration include behaviors that are commonly discouraged in classrooms or are perceived as unproductive (e.g., “off-task” discussions). Often, however, these non-normative behaviors make for more equitable collaborative dynamics as well as present an opportunity for building relationships between students. This brings to the forefront exciting directions for AI in terms of being able to capture and support the variety of normative and non-normative ways in which students collaborate. This will likely require AI models that triangulate speech with gesture, posture, eye-movements, and other forms of multi-modal data to understand the different types of collaboration that are occurring across classroom settings (which can be particularly challenging to capture given the amount of movement and noise during group work). We are looking forward to pushing the boundaries in this area to see how we could best support our teachers and students!
Nathan Holbert
Joshua Danish
Professor and Program Chair
Ooh, this is a really nice angle. I wonder if you can say more about how you'll identify when normative or non-normative behaviors are "good"? Is that something you imagine the teacher might help with?
Julie Harrison
PhD Student
From a measurement perspective, we are are working to model CPS dynamics in real time. One example we discuss is with off-task/topic conversation. Sometimes this off-topic conversation is a tool students use to become integrated with group members. It first appears unproductive to CPS, but it is actually contributing to group member inclusion and group cohesion. We are working to model this communication behavior over time in order to assess what types of non-normative CPS communication tend to benefit the group during different phases of collaboration.
Rachel or Indrani can speak to ways the teacher might help with this with regard to the teacher dashboards.
Indrani Dey
PhD Student
Yes, teachers will be critical to this component! We have conducted participatory design sessions with teachers to better understand their dashboard needs in the classroom, which we will use to inform our designs. We are also in the process of considering how we can integrate teacher feedback within a dashboard to inform our AI models, particularly for complex areas that vary or may look different across classrooms.
Nonye Alozie
This is great. Thanks for your video. I met Arthur Graesser a couple of weeks ago- he and I sit on the advisory board for the AI ALOE Institute. He mentioned that your group was also studying collaboration. I look forward to seeing more about the role that an AI collaborator plays in student collaboration. What are you expecting to see, in terms of the role of the AI partner?
Judi Fusco
Julie Harrison
PhD Student
As the institute grows, we will be and currently are developing the role of the agent alongside young people/students and teachers. Currently, the students in the Learning Futures Workshops are generating their own creative ideas for the parter (e.g., a simple light up display that monitors collaboration norms--somewhat of an artifact of how collaboration is going to ground the whole group). The role of community builder has also resonated with them. When interacting with teachers, we are currently focused on what they would like to see on both after-action review dashboards and in-real-time dashboards. We are working with teachers to ensure the dashboards are customizable and allow teachers to engage fully with the class during class time (i.e., design in ways that prevent them from having to look only at their iPads during class time).
Arthur Graesser's work with CPS measurement has been influential to the skills we are looking to measure with the iSAT CPS measurement framework. The work presented in your video is also relevant to the iSAT framework. One of our goals is to build out more affective skills/behaviors that contribute to successful CPS -- I will be adding confidence to that body of skills after seeing your video!
Joshua Danish
Professor and Program Chair
This is really wonderful work - thanks for sharing! I look forward to hearing more about it! One question I had was: how are you thinking about diversity amongst your learners? Different learners naturally have different needs in collaboration. Are you working with diverse groups for your ongoing participatory design work? Or is that something you'll aim to adapt to later? I really appreciated the note above about non-normative forms of participation. Thanks!
Julie Harrison
PhD Student
Hi Joshua, thanks for your comment. We are certainly considering diversity -- along social and economic dimensions, as well as learning and interaction preferences for students. There are a couple hurdles we are working to address. 1) Much of the collaboration measurement literature is focused on drawing insights from students' communications (e.g., transcripts). We recognize communication is not every student's contribution to a group, so we are working on multimodal measurement, and will now also look into ideas we've seen in other presenters' STEM for All videos for this, as well. 2) The data we are collecting from schools tends to have a bias regarding those students (parents) who opt into our data collection efforts. Generally, those who skew higher on levels of socioeconomic status are more likely to be willing to participate in data collection. In forthcoming data collection efforts, this is a priority that we will address.
Josh Sheldon
Project Lead
This is really interesting work - thanks so much for sharing.
There's been quite a bit of good discussion already. I'll try to follow up on some points and perhaps unearth a few others:
Looking forward to hearing more!
Indrani Dey
PhD Student
Hi Josh, thanks for your great questions!
Josh Sheldon
Project Lead
Hi Indrani.
Thanks for your response.
I understand that the agent is not to replace a human. So, assuming the lead teacher in the classroom stays the same, my curiosity is around the value of an AI agent vs. a second trained human, either a less-experienced co-teacher or a paraprofessional. And of course, I was mainly curious about the costs & benefits around educational and social outcomes, recognizing that there are huge practical implications, financial and otherwise, but choosing to ignore those for now.