Written by Matthew Busekroos | Produced by Nate Caldwell
Originally from Shenzhen, China, also known as the forefront of robotic innovation and technology, Lirui (Leroy) Wang grew up witnessing the growth of tech giants and startups that created products now used in our daily lives. As a child, Wang became obsessed with solving math puzzles and later joined the robotics club in middle school where his interest in science flourished.
Prior to his studies at MIT’s Computer Science and Artificial Intelligence Laboratory, Wang double majored in computer science and electrical engineering at the University of Washington. He focused on the problem of robotic grasping through model-based optimization such as advanced motion planning and model-free learning such as end-to-end policy approaches.
“I was amazed by how much advanced research in dexterous robotic manipulation can contribute to applications in households and industries,” Wang said. “I wanted to understand how machine learning, which has revolutionized other fields such as computer vision and natural language processing, can push the frontier of robotic applications.”
Subsequently Wang decided to apply for a PhD to continue his research studies in robotics and machine learning. He now works alongside Professor Russ Tedrake in the lab’s Robot Locomotion Group.
“Russ is extremely disciplined, rigorous, and passionate, and is multifaceted as a good researcher, teacher, and engineer,” Wang said. “Russ masters at both low-level and high-level research: he codes the most in our lab and knows so much about low-level details in simulation, real-robots, software engineering, while being a professor and overseeing and keeping up with the high-level research directions.”
Wang said the Robot Locomotion Group is extremely diverse in terms of the research topics and expertise, with deep insights from control theory and optimization to robotics and machine learning.
Wang’s research focuses on fleet learning and generative simulation for robotic manipulation. They are treated as two paths toward tackling the data problem in robotics industry and research. The first approach is bottom-up: we collect high-quality data and bootstrap from it to build applications which can autonomously collect more data, and thus build a data flywheel for better generalization. Due to the lack of existing datasets, we have to play carefully in this specialist-generalist problem (such as scaling with heterogeneous data). The second approach top-down: we leverage foundation models and advances from other fields (such as LLM/VLM) in robotics and greatly improve the capability of current robots (through the medium of robotic simulation for instance).
“Similar to humans, robots and embodied agents inherently have to deal with heterogeneous inputs and outputs,” Wang said. “The data format and distributions are varied in different modalities such as color, depth, tactile, and proprioceptive information, and collected in different domains such as simulation, real robots, and human videos. Moreover, fleets of robots and machines ingest massive amounts of streaming data generated by interacting with their environments in a distributed fashion.”
He said the core idea behind his research, fleet learning, is to embrace the heterogeneous nature of robot learning to develop efficient and general algorithms at different levels of abstraction, from data and representation to policy and action.
“Fleet learning is a combination of multitask learning, meta-learning, and distributed learning in that it requires learning from multiple environments, tasks, and modalities; fast adaptations and continual learning for new robots; and communication efficiency when deployed on large scales with a focus on robotic applications,” he said.
Wang added that he is passionate about advancing robotic simulation with generative AI.
“Building, training, and validating single robot in different environments are still very hard and costly even after decades of development, due to the bottlenecks of physical robot tests” he said. “To enable robot fleets to deploy in the real world, robotic simulation can play a critical role as a digital copy of the real world that scales software such as model-free learning and model-based verifications. I have studied sim to real, real to sim, generative simulation throughout different projects.”
There are a few robotic applications that Wang has investigated, including diverse robotic warehouse data in Amazon, tactile insertion tasks in manufacturing, robotic grasping in unstructured environments; human-robot interactions such as object hand-over, table-top manipulations (e.g. building structures); and dexterous tool uses such as using a hammer, wrench, and others tools.
Wang said these applications can be readily extended to home robots to help the elderly live higher-quality lives, manufacturing sectors, including car and consumer electronics, and warehouse logistics, such as sorting and palletizing.
“Fleet managements are very useful when the robots begin to deploy at scale. These systems should have a focus on efficient computation on edge AI and embracing the heterogeneity and the continual adaptations at each deployment scenarios,” Wang said. “Simulation and digital copies for robotics and agents are useful across any robotic industry, which need the ML training and model-based verifications at scale. Building and interacting with simulation data have already been the starting steps for building real robotic applications in the past few years. Combining generative AI with simulation and synthetic data can potentially revolutionize the process of creating realistic and diverse robotic simulation environments for real world deployments.”
Similar to other roboticists and computer scientists, Wang said he is passionate about making a positive impact to the world.
“To achieve this, I build robots and autonomous agents to do challenging and meaningful tasks,” he said. “Current robots that interact with the physical world are usually deployed in structured environments such as factories, and yet, we have not seen many robots in unstructured environments doing more useful tasks. Sometimes this is referred to as the “chicken and egg” problem in robot learning where we need useful robots to deploy to generate the data and we need to collect data to train useful robots.”
Wang said this is both challenging and exciting, as it requires some levels of “common sense reasoning” and “dexterity.” “My grand vision is to build systems and platforms that can enable robots to perform intelligent physical tasks in the real world just like humans,” Wang said.
Upon graduating, Wang said his dream job is to build a team to contribute to advancing robotics and embodied agents in industry by committing to building products, systems, and research projects toward practical robots and software. He said these include real-world robotic applications in households and industries, as well as simulation and AI software.
“In the long run, I would like to build a team from which I can learn a lot and that can execute my grand goal of bringing robots to assist people’s daily lives and improve people’s productivity with AI and machine learning,” he said.
For more information on Lirui Wang, check out his personal website: https://liruiw.github.io/ and his projects on Policy Composition, GenSim and HPT.