Interviewer: Sabine Hauert (Sabine)
Guest: Raffaello D’Andrea (Raff)

Sabine: Hi Raff! Welcome to ROBOTS!

Raff: Hi. Nice to talk to you again.

Sabine: It has always been great to have you on our show, and your students as well. They always make for really great interviews because they having amazing projects. And I wanted to start by saying congratulations because you are one of the Co-founders of Kiva Systems that was just acquired by Amazon. I thought we’d start by getting an overview of how you make a success story from the lab to the industry. So, where does Kiva come from?

Raff: The idea for doing something like Kiva System came from Mick Mountz. Mick worked at Webvan. He was, basically, involved in the distribution space, and he saw that there was no real, really good solution for solving a lot of the problems that distribution facilities were having. So he had this great idea of why not automate a lot of these processes with mobile robots. Instead of having people walk to the product, why doesn’t the product go directly to the person? So he had this idea, and through a mutual friend we met when I was just starting my sabbatical at MIT, and he told me what his vision was and I thought, hey, this is a great idea. And then we met again this next day. And then we met again, basically, the whole weekend, and he basically convinced me that this was a phenomenal idea and whose time was right in terms of the technology. So, along with Pete Wurman, we started Kiva.

Sabine: The press release says that Amazon will be paying 775 million US Dollars in cash for all the outstanding shares of Kiva, and I’m wondering: Is that what you imagined when you started? I guess in those days it was still very out there. You didn’t know if it was going to work.

Raff: What really motivated me is just the vision of being able to see thousands of these mobile robots all moving around autonomously. That was the vision that I had, and for Mick, I can’t speak for him. And for Pete, I can’t speak for him either. But I think a lot of it was just the creation aspect, to be able to, from Mick’s perspective, to solve a real world problem in a really cool way. And for Pete, I’m sure he had similar ideas as me — just to create something as cool as this. What a great opportunity. So that is really motivated us. The fact that it is a business is great too. It is validation that this model for innovation can be successful.

Sabine: And the reason they came to you in the beginning, I think, is because of Robocup and all the work you were doing in that system. How did that translate?

Raff: When Mick had this idea, he was basically using videos on the web to see what the technology was like at the time. And he saw Robocop videos of robots playing soccer and he thought: If people can do this, the technology can also be modified and adapted to solve a real world problem such as distribution facilities. So that is how we actually met — it was through these videos.

Sabine: Is it satisfying, for you as a researcher, to see that, what you work on has such success. And this is really one of the major successes in robotics.

Raff: Yeah, this is exciting. The very fact that Kiva is bought by Amazon is a validation of the idea. That is great for Mick, who had this idea, and also for the technology.

The fact that what research people have been doing for quite some time is ready for prime time, basically. And, I think it is going to open up a lot of doors. A lot of people are going to say: “Hey, this is possible. You can make a business out of this.” And people are going to try. And you don’t know ahead of time what is going to be successful and what is not. But, it is going to make people try. Ten times more people, maybe, are going to try. You are going to have ten times as many successes, so I think that is what is really exciting about having done something like Kiva.

From a personal perspective, it is kind of neat to know that there is going to be thousands, hundreds of thousands, of these vehicles running around using the algorithms that you’ve developed and coded and bring to life. So, that is absolutely exciting.

Sabine: That is really interesting. Do you think this will be like bio-medical engineering, that there will be a race in robotics because people all of a sudden people put value on it, and it is getting heavily funded by VC’s and it is just become a sexy field for industry?

Raff: I hate making predictions because I know how hard they are to make, and it is easy to rationalize after the fact. And it is easy for a whole bunch of people to guess and, obviously, some of them are bound to be right, and then to look back upon it as having some sort of innate wisdom. All I am going to say is that basically there is a lot of flexibility in the technology and robotics in general. I think there are a lot of things that still need to be solved. The main one is situational awareness, better sensing. But I think a big one that people are not thinking about are better business models — to really work with entrepreneurs at an early stage so that they have an appreciation of what the real capabilities of the technology are, and to bring this technology to bear on these problems so that you can actually solve something with the technology that is available now.

I think that is really exciting to see. It will be really exciting to see more entrepreneurs working closely with researchers. I think that is going to be a very exciting thing to see if it happens more.

Sabine: Your lab has been working on a lot of cool systems. Maybe you can start by telling us what dynamic systems are.

Raff: Dynamic systems are systems that move, systems that change over time. Dynamics just simply means the study of motion. So I’m interested in understanding these systems, modeling them, and then figuring out how to control them, how to make them do what it is that you want them to do. Robots are a very good example of this. But my interests, in general, are with the overall system. How do you design it? How to you understand it? How do you control it?

Sabine: And why is this difficult?

Raff: I think it is difficult because it involves so many different aspects. There is the modeling side of it. Trying to understand how physical systems behave. That tends to be a lot of engineering science, engineering physics. Then there is the mathematics aspect of it. The language of models is mathematics.

So, if you want to be able to understand their behavior, if you want to figure out what to do with them, you have to understand mathematics. Then there is the dynamics and controls research aspect of it. The algorithms that you can bring to bear to make a system do what it is supposed to do from a mathematical perspective. And there is the technological part which is, how do you actually implement all of this? How do you bring something to life? Take it back from the mathematical equations, from the symbols, from the simulations, and make it something real. So, it is a really broad set of skills and experiences that you have to bring in order to do something like this.

Sabine: And that is really great because I’ve always found all these algorithms and formulas a bit boring and hard to grasp, and I think by putting robots in the loop and showing things that really move, it sort of helps students and people understand how these systems work. I’m wondering what other math or physics would you love to sort of show with robots? Like what can you help teach people?

Raff: I think that the number of things that you can teach with robots is enormous. I mean, it is huge. Write a simple as basic differential equations. A simple robot is governed by a relatively straightforward set of dynamical equations, so you can teach ‘em a lot of control, you can teach ‘em a lot of modeling, basic Newtonian mechanics, basic feedback laws like single input-single output control loops, basic programming. How is it that you take sensor information and manipulate it so that you can implement a computer algorithm — a control algorithm? There is the electronic side of it. There is just a tremendous amount of learning opportunities to do with robots and feedback systems in general.

Sabine: So let us start by looking at the flying machinery where you’ve done so much of your amazing work with quadrotor. So what is this arena?

Raff: The flying machine arena is a large space. It is about 10 meters x 10 meters x 10 meters where we fly a whole bunch of vehicles either by themselves, but also together to do things like juggle balls, balance poles, fly in formation, dance to music, catch balls as well. Basically, we are really trying to make these vehicles do things that have never been before. And a lot of it is geared towards adaptation and learning. We want these things, through practice, to get better and better at what they do.

Sabine: We always see the beautiful videos coming out of your lab. How long does it take to get one of these videos? You make it look so simple some times.

Raff: Actually, the videos don’t take us long at all because one of the the things that we really strive for in our lab is the concept of a zero downtime, demo. We kind of have a policy that, if somebody comes and visits our lab, within five minutes they should be able to see everything that we have in our lab. Everything always has to be working. So making videos is actually quite easy because we just film what it is that we do all the time. Of course, there is some production values, there is some editing that takes a bit of time. But the actual filming of the process doesn’t take that much time at all.

Sabine: If you are juggling a ping pong ball or if you are flying in formation, how different are these problems?

Raff: They share a lot of similarities. Of course, you have to estimate what the system is doing, but once you have that figured out for one vehicle, you have it figured out for multiple vehicles and for balls, etc. Some of the aspects of control are similar. Especially the inner loops, the lower-level loops don’t change much. What tends to change are more of the higher-level abilities, right from trajectory generation, but even higher up to what decisions to make. Once you have that lower stuff figured out – and that is an interesting problem on its own, that is where a lot of our contributions are – then you are free to move up a level and then explore higher level issues. Robocop was like that. Kiva Systems is like that, too.

Sabine: And what prevents you from taking these robots out of the flying machinery? Now what would it take to make this a reality, let us say, in your yard.

Raff: Right now we are focusing mainly on the control-side of the problem, less on the estimation side. We use a motion-capture system that gives us the state of the vehicles. These motion-capture systems are not cheap, and they are impractical in many ways. You don’t want to be lugging around a motion-capture system whenever you want to deploy a vehicle. But what it does allow us to do, is do research in an area. And it gives us a controlled environment where we can explore local estimation schemes, but have a validation method that really allows us to determine how well our algorithms are working, our estimation algorithms are working. So it is a great test bed for developing capabilities – the control capabilities, but even the estimation ones – once you want to be able to validate exactly what it is that you do.

Sabine: So with all that experience on what it takes to make systems like these, could you give a specification for the perfect outdoor sensor that might be able to help you, and that is not necessarily a Vicon?

Raff: No. I think the simple answer to that is, it depends.

Sabine: For a test.

Raff: Exactly. And that is really where, I think, business ideas come to life, and also where there is a lot of opportunity for people to collaborate from different disciplines, including people from business and people in robotics and engineering. Because a lot of times, what makes a problem difficult from a engineering perspective may be simple to change from a business perspective, and vice versa. So, when you talk to each other, you can figure out how to make the problem as simple as possible, technically, without sacrificing any of the business capabilities.

Sabine: All your work gives rise to a lot of collaborations, and I love your recent projects. There was, in collaboration with architecture, one to build these walls out of bricks. Can you tell us a bit more about that?

Raff: It was an installation we did in France with architects Gramazio and Kohler. They are world-renowned for using automation in architecture. So they pioneered the use of using robot arms in building walls. And, what is really neat about these walls, is they are not just straight, they undulate. They are built in such a way that it would be impossible for a person, or nearly impossible for a person to be able to do it because they lack the precision.

So, it was natural for us to team up to do something that was at the intersection of architecture, engineering with flying vehicles, and I would even like to say art. I think doing an installation in front of a live audience, to build a structure, to me, that is also an artistic endeavor. That is what we did in France. We built a 6-meter tall structure with four flying vehicles. The structure was composed of 1500 modules that weighed about a hundred grams each, so the thing looks like this undulating tower. It took about 13 to 15 hours of flying time to make the structure. We did it over a period of three days, and it was done again in front of a live audience. So that was very exciting for us to see people walk around while these vehicles are flying overhead carrying these little foam modules. It was a very special experience.

Sabine: So they were made of foam.

Raff: That is correct.

Sabine: And does that mean that they have a certain level of compression? Does that help you in the compliance of putting them at the right place?

Raff: Not really. As far as we were concerned, they were rigid structures. They just happened to be light. Think about it, with something with 1500 modules, even with a hundred grams per module, that is 150 kilos. That is a pretty heavy structure. It is now part of their permanent collection, so they have to be able to take it down. They cut it into three parts, so that, if it is on exhibit at a different museum, it can be actually carried by a truck and then assembled on site. So, there are some pragmatic aspects that have to be considered when you are doing an installation, and basically something that becomes part of a permanent collection.

Sabine: And what were you tracking? Did you know where the bricks were or how does it start from the base? You give it a brick or they pick up the bricks or what is the normal way of doing this?

Raff: The process is that there are humans in the loop. The humans are responsible for putting a brick in the brick dispenser. There is a system that interfaces with the operator to let them know when it is okay to put a brick in the dispenser and when it is not. As soon as a brick is sensed by the system, a vehicle will come by, pick up the brick and fly to the proper location where it can place that brick. We keep track of the system. We know what bricks have been laid. We record the height and exact location of where the previous bricks were laid so we know exactly where to fly in order to drop off the new brick. Again, we are using this motion-capture system to track the vehicles. The bottom line, though, is that we actually didn’t have to track the bricks. We just had to track the vehicles.

Sabine: Because it is that precise.

Raff: Exactly.

Sabine: You mentioned the artistic aspect of this project, and you’ve really been working at the interface of art with, for example, your robotic chair, your robotic table. Why is art and design important for you?

Raff: First let me say that, to me, they are all pretty much the same thing. Whether it is mobile robots in a warehouse or a robotic chair that falls apart and puts itself back together again, or this recent installation. For me, the interesting part, the part that I really care about, is the active creation, the part of making something new, something that moves, something that reacts to its environment, something that can learn and adapt. There are so many different ways. Then you start to express yourself when you do that — in business, in academia, and doing art. So that is really what I think all these things have in common. That is why I do it.

Sabine: What has been the most interesting reaction from the public to all of these projects? They are all very visible.

Raff: I mean, it varies. Some people think, what the heck are these guys doing? Why are they doing these things that have no apparent utility? Of course, Kiva System is an example of why you’d want to do these things. But, certainly, a lot of times we get that reaction. Sometimes you get indifference, which is kind of neat. When we unveiled the table at the Venice Biennale - this was a collaboration with artist Max Dean, it is a table that interfaces with you, has a conversation with you based on movement, depending on how you move, the table moves - there would be some people that would just be mesmerized by this table. They would spend hours with it. And then, we’ve seen some people that would come into the room where the installation was held and then the table would come up to them and try to have a conversation and they wouldn’t even slow down. They would just look at it, look the other way, and keep on walking. Llike, this is something that I see every day. Don’t bother me with this. The spectrum of reaction is just tremendous. By the way, that also makes it a lot of fun, too.

Sabine: And what do you think will be your next exhibit?

Raff: I never speculate about what I’m going to do next because I always change my mind, so I don’t know. We’ll see.

Sabine: Have you already displayed the blind juggling machine?

Raff: Yep. We were recently in San Francisco and Phillip Reist showed the Blind Juggler at Swissnex, the Swiss Council in San Francisco, and also some other events at the Exploratorium, and it was a lot of fun. Phillip has actually showed the Blind Juggler in Italy and in Denmark as well. It has also been on tour. It is also part of the permanent collection of the Computer Museum in Paderborn.

Sabine: For those who don’t know what it is, can you briefly describe it?

Raff: The Blind Juggler is a robot that can juggle balls without sensing them, without seeing them. It does this via a careful design of the natural dynamics of the system — so the shape of the paddle that hits the ball and the specific motion of the ball. What is interesting about this device is that it exhibits rich dynamics, complex dynamics. We have another version of this device, called the Cloverleaf, where you can juggle up to four balls simultaneously and you can exhibit really interesting behavior like chaos — different attractors — and we are exploring different ways in which you can control these type of dynamical systems.

Sabine: And not only is it visually beautiful, but it also makes really nice music and sounds.

Raff: Yeah, if you have an array of them. We have this concept where, we have 32 of them, where you can basically play drumbeats with these jugglers. It is actually quite fun to hear the complex patterns that you can generate. This is actually an installation that we would like to build some day.

Sabine: You mention that some people say: what are they doing? This has no direct purpose. What do you answer to them?

Raff: I think the university environment is a very special one, and there are some great things that we can inherit from industry in terms of robotics research. And I think those things are not necessarily the utility aspects of it, because business people are great at coming up with reasons why to make something. But, rather, things like robustness, things like reliability, things like simplicity. So, those are the kind of things that we like to carry over in our research from industry. But not the utility, right? So, we focus on elegant algorithms. We focus on things that have never been before, but as simply as possible, in a robust way, in a repeatable way. That allows us to do great research. But it also leads to really direct ways in which we can interface with industry and business folks that do have great ideas on new markets and new ways to deploy the technology.

Sabine: People come to you with a question about how to make something roust or how to make a quadrotor?

Raff: The main product of our group is the process by which we make systems, so the system architecture and the algorithms that we use to make the system do what it does. The papers that we publish are mainly algorithmic in nature, so nobody would come to us and say, can you make a better quad-copter? even though we do modify things when we have to in terms of hardware. We build things when we have to. If we don’t have to build them, we buy them. We are perfectly happy in doing that. Kiva Systems is an example where we had to build everything just because things didn’t exist. So, we hired phenomenal mechanical engineers, electrical engineers, computer scientists, and we built the best robots that we could. But, in my lab, we mainly try to work on algorithms.

Sabine: I think, for a lot of people in industry, it might not be obvious for them, either, how to apply these things. So just how to push both the lab and the industry to understand each other better on these aspects?

Raff: The main product of universities are the people that we train. And that is at all levels. We publish great papers. Some of them are phenomenal, change the field, have huge industrial impact. But if you just look at the bulk of the contributions that research labs make, it is the people that we train. So, the point is that we are training huge number of people in how to make complex systems, and these are the people that go out and change the world. Or these are the people that go out and team up with entrepreneurs and make new things happen. That is how that transfer happens. It is mainly through people.

Sabine: And I think a lot of your students have actually gone into robotics industries, right?

Raff: That is correct. Some of them have gone to academia, some of them are professors, some of them are at startups, some of them are at well-established, large companies. They are just everywhere, and that is one of the fun things about being a professor.

Sabine: You mentioned before that universities should be able to do basic research and should be able to explore problems without worrying too much about what is going to come out of it. I wonder if anything needs to change in terms of funding or in terms of how research is motivated so that this actually happens.

Raff: It is a really good question. I think at the end of the day, what you need is a portfolio. You need some folks that are doing very basic research, and you need some folks that are doing very applied research. It is just that in engineering, we have a little bit of a different mentality in the concepts. In science and in mathematics, you have concepts of pure physics and pure mathematics. But, in engineering, there is no such thing as pure engineering.

All engineering is thought to be applied, solving a specific problem. But that misses the point that there is a lot of intangible things with engineering research that are not just about solving a specific problem. It is just the methodology. It is the exploratory part of how you design systems. That used to be unconstrained. I think that funding agencies, universities, need to recognize that and explore and fund more pure engineering-type of research. By that I mean - let is be specific to robotics – the design aspects, the architecture aspects, the intangible things that, perhaps, are not seen as an immediate industrial impact. Now that doesn’t mean that you can go off and do crazy stuff that has no interesting research problems associated with it or any potential industrial impact. There has to be a nugget there somewhere — interesting from a research perspective, from a mathematical perspective, from an art perspective, from an industrial perspective. It has to be interesting in some way. But having the metric only being direct utility, I think, is a big mistake.

Sabine: Excellent. Thank you, Raff, for being here with us on ROBOTS.

Raff: You are welcome.