Project 71: Large-model-driven Perception and Planning for Humanoid Robots

Contact Information

Asso. Prof. Yue Gao

Project Description and Objectives

This project focuses on leveraging large-scale pre-trained models to enhance humanoid robot perception and planning capabilities. The objective is to design an advanced robot perception system that integrates pre-trained models, such as vision-based transformers or language models, to process and interpret complex sensory data. By utilizing this system, the project aims to explore intelligent planning and decision-making strategies for humanoid robots operating in dynamic and complex tasks.

Participants will engage in a hands-on approach, constructing perception and planning modules, implementing task-based simulations, and testing on real-world humanoid and legged robotic platforms. The key objectives include optimizing the integration of pre-trained models for sensory understanding, improving task coordination under dynamic conditions, and achieving robust performance in experimental environments. The project provides an end-to-end research opportunity, encompassing theoretical model development, system integration, and practical robot testing.

Eligibility Requirements

Basic understanding of reinforcement learning, computer vision, or motion planning. Knowledge of large-scale pre-trained models and their applications.

Main Tasks

Development of a Robot Perception System.

Exploration of Intelligent Planning and Decision-Making.

Experimental Integration and Validation.

Website

Lab: https://gaoyue.sjtu.edu.cn

School: http://www.seiee.sjtu.edu.cn/