OTTRS Speaker Series: Multimodal Human-AI Interaction
Event Start Date:
Friday, October 27, 2023
- 12:00 pm
Event End Date:
Friday, October 27, 2023 - 1:00 pm
Location: Virtual - Registration Required
Add to Calendar
Friday, October 27, 2023 12:00 pm
Friday, October 27, 2023 1:00 pm
America/New York
OTTRS Speaker Series: Multimodal Human-AI Interaction
UMD students, faculty, staff, alumni, and friends—join us for the OTTRS Speaker Series!
Description:
People communicate through verbal and non-verbal cues. AI and ML (machine learning) have made tremendous progress in language understanding. Audio tone, gestures, gaze, and touch, along with speech, offer new challenges and opportunities. Snehesh Shrestha’s work dissects multimodal human expression, focusing on human-AI interaction in robotics and music.
In the first part, he will discuss creating a robot capable of understanding natural commands, emphasizing multimodal repair mechanisms. He will briefly share data collection challenges, which greatly impact data quality and validity. His team used a Wizard-of-Oz setup, deceiving participants into believing they had a human-level AI robot, to capture ‘natural’ interactions. Verbal and non-verbal strategies were studied to train machine learning algorithms for multi-modal commands, highlighting the importance of combining gestures with speech.
In the second part, he will explore AI-mediated Student-Teacher Interaction systems towards violin education. He will discuss challenges in remote music lessons, which became particularly pronounced during the COVID-19 pandemic. He will discuss data collection challenges for precise motion capture, especially with young students. He will share insights into using audio to enhance pose estimation algorithms for 3D player visualization.
Lastly, he will introduce a novel haptic band designed for remote feedback, prompts, and metronome functions, enhancing online music education experiences.
Bio:
Snehesh Shrestha is a Ph.D. candidate at the University of Maryland College Park. He works in the Perception and Robotics Group (PRG) lab in the Department of Computer Science under the guidance of Prof. Yiannis Aloimonos (CS), Dr. Cornelia Fermüller (UMAICS), Dr. Ge Gao (INFO), and Dr. Irina Muresanu (School of Music). He has also worked with Dr. Michelle Gelfand (Department of Psychology) in the Culture Lab. Additionally, he works at NIST, developing new standards towards recommended practices for the design of human subject studies in human-robot interaction. His research is at the intersection of robotics, artificial intelligence, human factors, arts, and culture. He is interested in multidisciplinary research aimed at building rich and intuitive experiences that ‘amplify human abilities, empowering people and ensuring human control’ inspired from Dr. Ben Shneiderman’s Human-Centered AI book. His recent work has focused on human-robot interaction and AI for music education.
About OTTRS:
The Organizational Teams and Technology Research Society (OTTRS) aims to advance research and collaboration on the study of teams as relevant to technology and information. Topics include, but are not limited to, teams and artificial intelligence, technology to support different types of teams, collaborative learning, ethics in teams and technology, leveraging technology to improve the study of teams, and more. While we have a human focus, we are inspired by computational potential and technological advances. We explicitly welcome multiple and interdisciplinary approaches.
Please contact ischoolevents@umd.edu at least one week prior to the event to request disability accommodations. In all situations, a good faith effort (up until the time of the event) will be made to provide accommodations.
Virtual - Registration Required
UMD students, faculty, staff, alumni, and friends—join us for the OTTRS Speaker Series!
Description:
People communicate through verbal and non-verbal cues. AI and ML (machine learning) have made tremendous progress in language understanding. Audio tone, gestures, gaze, and touch, along with speech, offer new challenges and opportunities. Snehesh Shrestha’s work dissects multimodal human expression, focusing on human-AI interaction in robotics and music.
In the first part, he will discuss creating a robot capable of understanding natural commands, emphasizing multimodal repair mechanisms. He will briefly share data collection challenges, which greatly impact data quality and validity. His team used a Wizard-of-Oz setup, deceiving participants into believing they had a human-level AI robot, to capture ‘natural’ interactions. Verbal and non-verbal strategies were studied to train machine learning algorithms for multi-modal commands, highlighting the importance of combining gestures with speech.
In the second part, he will explore AI-mediated Student-Teacher Interaction systems towards violin education. He will discuss challenges in remote music lessons, which became particularly pronounced during the COVID-19 pandemic. He will discuss data collection challenges for precise motion capture, especially with young students. He will share insights into using audio to enhance pose estimation algorithms for 3D player visualization.
Lastly, he will introduce a novel haptic band designed for remote feedback, prompts, and metronome functions, enhancing online music education experiences.
Bio:
Snehesh Shrestha is a Ph.D. candidate at the University of Maryland College Park. He works in the Perception and Robotics Group (PRG) lab in the Department of Computer Science under the guidance of Prof. Yiannis Aloimonos (CS), Dr. Cornelia Fermüller (UMAICS), Dr. Ge Gao (INFO), and Dr. Irina Muresanu (School of Music). He has also worked with Dr. Michelle Gelfand (Department of Psychology) in the Culture Lab. Additionally, he works at NIST, developing new standards towards recommended practices for the design of human subject studies in human-robot interaction. His research is at the intersection of robotics, artificial intelligence, human factors, arts, and culture. He is interested in multidisciplinary research aimed at building rich and intuitive experiences that ‘amplify human abilities, empowering people and ensuring human control’ inspired from Dr. Ben Shneiderman’s Human-Centered AI book. His recent work has focused on human-robot interaction and AI for music education.
About OTTRS:
The Organizational Teams and Technology Research Society (OTTRS) aims to advance research and collaboration on the study of teams as relevant to technology and information. Topics include, but are not limited to, teams and artificial intelligence, technology to support different types of teams, collaborative learning, ethics in teams and technology, leveraging technology to improve the study of teams, and more. While we have a human focus, we are inspired by computational potential and technological advances. We explicitly welcome multiple and interdisciplinary approaches.
Please contact ischoolevents@umd.edu at least one week prior to the event to request disability accommodations. In all situations, a good faith effort (up until the time of the event) will be made to provide accommodations.
Register
Research Talks/Events