Earlier this week, the Toyota Research Institute opened the doors of its Bay Area offices to members of the media for the first time. It was a day full of demos ranging from driving simulators and drift instructors to talks about machine learning and sustainability.
Robotics, a longtime focus of Toyota’s research division, was also on display. SVP Max Bajracharya presented a few projects. The first was a little more along the lines of what you’d expect from Toyota: an industrial arm with a custom gripper designed for the surprisingly complex task of moving boxes from the back of a truck to nearby conveyor belts—something most factories hope to automate. the future.
The other is a bit more surprising – at least for those who haven’t followed the division’s work all that closely. A shopping robot pulls different products from the shelf based on barcodes and general location. The system can extend to the top shelf to find items before determining the best method for grabbing the wide range of different objects and dropping them into the basket.
The system is a direct result of the 50-person robotics team’s focus on elderly care, aimed at addressing Japan’s aging population. However, it represents a linchpin of their original work of building robots designed to perform household tasks such as washing dishes and food preparation.
You can read a longer description of that pivot in an article published on businessupdates.org earlier this week. That is taken from a conversation with Bajracharya, which we reproduce below in a more complete state. Note that the text has been edited for clarity and length.
businessupdates.org: I was hoping to get a demo of the home robot.
Max Bajracharya: We’re still doing some things with robots for the home[…] What we’ve done has shifted. Home was one of our original challenge tasks.
Elderly care was the first pillar.
Absolute. One of the things we learned in that process is that we couldn’t properly measure our progress. The house is so hard. We choose challenging tasks because they are difficult. The problem with the house isn’t that it was too hard. It was that it was too difficult to measure the progress we were making. We have tried many things. We tried to mess it up procedurally. We would put flour and rice on the tables and we would try to clean them up. We would put things all over the house to make the robot tidy. We were betting on Airbnbs to see how well we did, but the problem is we couldn’t get the same house every time. But if we did, we’d fit too much into that house.
Isn’t it ideal that you don’t get the same house every time?
Exactly, but the problem is we couldn’t measure how well we did. Let’s say we were a little better at cleaning this one house, we don’t know if it’s because our abilities got better or if that house was a bit easier. We did the standard, ‘show a demo, show a cool video. We’re not good enough yet, here’s a cool video.” We didn’t know if we were making good progress or not. The grocery challenge task where we said, we need an environment where it’s as hard as a house or has the same representational issues as a house, but where we can measure how much progress we’re making.
You’re not talking about specific goals for the home or the supermarket, but solving problems that can span both places.
Or even measuring whether we are pushing the latest in robotics technology. Are we able to do the sensing, the movement planning, the behavior that is actually general purpose. To be completely honest, the challenge issue doesn’t really matter. The DARPA Robotics Challenges, they were just made up tasks that were hard. The same goes for our challenge tasks. We love the house because it is representative of where we ultimately want to help people in the home. But that doesn’t have to be the house. The grocery market is a very good representation because it has that huge diversity.
However, there is a frustration. We know how hard these challenges are and how far out things are, but a random person sees your video and suddenly it’s something just over the horizon, even though you can’t live up to it.
Absolute. That’s why Gill [Pratt] says each time, “re-emphasize why this is a challenging task.”
How do you translate that to normal people? Normal people don’t like challenging tasks.
Exactly, but that’s why in the demonstration you saw today, we tried to show the challenge tasks, but also an example of how to apply the capabilities that arise from that challenge to a real-life application, such as unloading a container. That’s a real problem. We went to factories and they said, ‘Yeah, this is a problem. Can you help us?’ And we said, yes, we have technologies that apply to that. So now we’re trying to show these challenges face these few breakthroughs that we think are important, and then apply those to real-world applications. And I think that’s helped people understand that because they see that second step.
How big is the robotics team?
The division consists of about 50 people, evenly split between here and Cambridge, Massachusetts.
You have examples like Tesla and Figure, who are trying to make all-purpose humanoid robots. You seem to be going in a different direction.
A little. Something we’ve observed is that the world is built for people. If you just got a clean slate, say I want to build a robot to work in human spaces. You tend to end up in human proportions and human-level capabilities. You end up with human legs and arms, not because that’s necessarily the optimal solution. It’s because the world is designed around people.
How do you measure milestones? What does success look like for your team?
The move from the house to the supermarket is a good example of this. We made progress around the house, but not as quickly and not as clearly as when we went to the supermarket. When we go to the grocery store, it really becomes very clear how well you’re doing and what the real problems are in your system. And then you can really focus on solving those problems. As we toured both Toyota’s logistics and manufacturing facilities, we saw all of these opportunities where they’re basically the challenge of grocery shopping, except a little bit differently. Now, the part instead of the parts being groceries, the parts are all the parts in a distribution center.
You hear from 1000 people that you know home robots are really hard, but then you feel like you have to try it yourself and then you want, really, you all make the same mistakes they did.
I think I’m probably just as guilty as everyone else. It’s like our GPUs are better now. Oh, we have machine learning and now you know we can do this. Oh, okay, maybe that was harder than we thought.
Something has to tip it over at some point.
Maybe. I think it’s going to take a long time. Like automated driving, I don’t think there is a silver bullet. There’s no such thing as this magic thing, which becomes ‘okay, now we’ve solved it’. It’ll cut away step by step, cut away. That’s why it’s important to have that kind of roadmap with shorter timelines, you know, shorter or shorter milestones that get you the small wins so you can keep working towards really reaching that long-term vision.
What is the process for actually producing one of these technologies?
That’s a very good question that we’re trying to answer ourselves. I think we understand the landscape a little bit now. Maybe I was naive at first thinking that, okay, we need to find this person to whom we’re going to transfer the technology to a third party or someone inside Toyota. But I think we’ve learned that whatever it is — whether it’s a corporate entity, or a company, or like a startup or a unit within Toyota — they don’t seem to exist. So we’re trying to find a way to create and I think that’s kind of the story of TRI-AD as well. It was created to translate the research into automated driving that we were doing into something more real. We have the same problem in robotics and in many of the advanced technologies that we work on.
You’re thinking about possibly moving to a place where you can have spin-offs.
Possible. But it’s not the main mechanism by which we would commercialize the technology.
What is the main mechanism?
We do not know. The answer is that the diversity of things we do will most likely be different for different groups.
How has TRI changed since its inception?
When I first started, I feel like we were very clearly just doing research in robotics. Part of that is that we were so far from applying the technology to almost any challenging real-world application in a human environment. In the last five years, I feel like we’ve made enough progress on that very challenging problem that we’re now starting to see it changing in these real-world applications. We have consciously shifted. We’re still 80% on the state of the art with research, but we’ve now allocated maybe 20% of our resources to figuring out if that research might be as good as we think it is and if it can be applied to real-world applications. We could fail. We may realize we thought we’d made some interesting breakthroughs, but it’s nowhere near reliable or fast enough. But we put 20% of our effort into trying.
How does elderly care fit into this?
I’d say it’s still our North Star in some ways. The projects are still looking at how we ultimately strengthen people at home. But over time, as we pick out these challenge tasks and drip out things that apply to these other areas, we use these short-term milestones to show the progress in the research that we’re making.
How realistic is the possibility of a full lights-out factor?
I think if you could maybe start from scratch in the future, that could be a possibility. When I look at production today, specifically for Toyota, it seems very unlikely that you’ll even come close to that. We [told factory workers], we’re building robot technology, where do you think it could apply? They showed us many, many processes where it was like, you take this harness, you run it through here, then you pull it out here, then you cut it here, and you cut it here, and you take it here, and you take it here, and then you run it like this. And this takes a person five days to learn the skill. We were like, ‘Yeah, that’s way too hard for robot technology.’
But the things that are most difficult for humans are the ones you would like to automate.
Yes, difficult or potentially injury prone. Of course, we’d like to take steps to eventually get there, but where I see robot technology today, we’re a long way from that.