Overview
Students will train a computer program to recognize hand gestures from images captured through their webcam. By collecting photos of different hand signs, labeling categories, and coding conditional responses, students engage with core data science skills like pattern recognition, image classification, and training machine learning algorithms. In an accessible way, the activity demystifies artificial intelligence technology that permeates fields from self-driving vehicles to medical diagnosis programs. Students interested in computer science careers could someday be designing and improving these types of interactive machine learning applications as software engineers or data analysts. And for any career path, understanding how algorithms interpret data will soon be as fundamental as basic computer literacy. This activity offers an entrypoint for students to not just use AI, but glimpse how it functions behind the scenes through firsthand experimentation.
NB Curricular Connections
Technology (6-8)
- Strand: Design Thinking Skills – Big Idea: Problem Solving and Computational Practice
- Strand: Infromation Technology Skills – Big Idea: Networking, Digital Citizenship
What You’ll Need
- Computer with webcam
- Internet connection
- Instructions PDF (included)
Instructions
- Download the instructions PDF (included).
- Have students open mBlock.cc in their internet browser and start a new project.
- Add the Teachable Machine extension in mBlock using the extensions menu.
- Open the Teachable Machine extension and create 3 image categories with at least 15 sample images in each – suggestions are “peace sign”, “thumbs up”, and “heart sign”.
- Label each image category as it’s added, like “peace” or “thumbs up”. Take images from slightly different angles and distances.
- Once enough sample images are collected in all 3 categories, click “Use the Model” to return to mBlock.
- In mBlock, code IF/ELSE conditional statements to check the category prediction result and display a text response for each one accordingly.
- Add a final ELSE statement to catch unrecognized signs and display “I don’t know”.
- Put all the code inside a Forever loop and click run.
- Show different hand signs to the computer camera and watch it predict and display the correct text response when it recognizes one of the trained gestures.
Reflection Activity
Please see the attached PDF for several choices on how you and your learners can reflect upon today’s activity.
Global Competencies
Acknowledgements
We would like to thank our Partners from Brilliant Labs, an Atlantic Canadian-based charity that supports students, teachers, and communities with cross-curricular, maker-centered learning. They integrate creativity, innovation, coding, and entrepreneurship into K-12 classrooms across Atlantic Canada and beyond. You can find more information about their amazing work on their website or follow them on Twitter and Facebook
