forked from RussTedrake/manipulation
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
419 lines (384 loc) · 19.1 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
<!DOCTYPE html>
<html>
<head>
<title>Robotic Manipulation</title>
<meta name="Robotic Manipulation" content="text/html; charset=utf-8;" />
<link rel="canonical" href="http://manipulation.csail.mit.edu/index.html" />
<script src="https://hypothes.is/embed.js" async></script>
<script type="text/javascript" src="htmlbook/book.js"></script>
<script src="htmlbook/mathjax-config.js" defer></script>
<script type="text/javascript" id="MathJax-script" defer
src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml.js">
</script>
<script>window.MathJax || document.write('<script type="text/javascript" src="htmlbook/MathJax/es5/tex-chtml.js" defer><\/script>')</script>
<link rel="stylesheet" href="htmlbook/highlight/styles/default.css">
<script src="htmlbook/highlight/highlight.pack.js"></script> <!-- http://highlightjs.readthedocs.io/en/latest/css-classes-reference.html#language-names-and-aliases -->
<script>hljs.initHighlightingOnLoad();</script>
<link rel="stylesheet" type="text/css" href="htmlbook/book.css" />
</head>
<body onload="forwardOldChapterLink(); customTags(); MathJax.typeset();">
<div data-type="titlepage">
<header>
<h1><a href="index.html" style="text-decoration:none;">Robotic Manipulation</a></h1>
<p data-type="subtitle">Perception, Planning, and Control</p>
<p style="font-size: 18px;"><a href="http://people.csail.mit.edu/russt/">Russ Tedrake</a></p>
<p style="font-size: 14px; text-align: right;">
© Russ Tedrake, 2020-2022<br/>
Last modified <span id="last_modified"></span>.</br>
<script>
var d = new Date(document.lastModified);
document.getElementById("last_modified").innerHTML = d.getFullYear() + "-" + (d.getMonth()+1) + "-" + d.getDate();</script>
<a href="misc.html">How to cite these notes, use annotations, and give feedback.</a><br/>
</p>
</header>
</div>
<p><b>Note:</b> These are working notes used for <a
href="http://manipulation.csail.mit.edu/Fall2022/">a course being taught
at MIT</a>. They will be updated throughout the Fall 2022 semester. <!-- <a
href="https://www.youtube.com/channel/UChfUOAhz7ynELF-s_1LPpWg">Lecture videos are available on YouTube</a>.--></p>
<section id="search" pdf="no"><h1>Search these notes</h1>
<form id="ddg-search-form" action="https://duckduckgo.com/" method="get">
<input type="text" name="q" placeholder="Search these notes..." />
<input type="hidden" name="sites" value="manipulation.csail.mit.edu" />
</form>
</section>
<section id="pdf">
<h1>PDF version of the notes</h1>
<p pdf="no">You can also download a PDF version of these notes (updated much less frequently) from <a href="https://github.com/RussTedrake/manipulation/releases">here</a>.</p>
<p>The PDF version of these notes are autogenerated from the HTML version.
There are a few conversion/formatting artifacts that are easy to fix (please
feel free to point them out). But there are also interactive elements in the
HTML version are not easy to put into the PDF. When possible, I try to
provide a link. But I consider the <a
href="http://manipulation.csail.mit.edu">online HTML version</a> to be the
main version.
</p>
</section>
<section id="table_of_contents">
<h1>Table of Contents</h1>
<ul>
<li><a href="#preface">Preface</a></li>
<li><a href="intro.html">Chapter 1: Introduction</a></li>
<ul>
<li><a href=intro.html#section1>Manipulation is more than pick-and-place</a></li>
<li><a href=intro.html#section2>Open-world manipulation</a></li>
<li><a href=intro.html#section3>Simulation</a></li>
<li><a href=intro.html#section4>These notes are interactive</a></li>
<li><a href=intro.html#section5>Model-based design and analysis</a></li>
<li><a href=intro.html#section6>Organization of these notes</a></li>
<li><a href=intro.html#section7>Exercises</a></li>
</ul>
<li><a href="robot.html">Chapter 2: Let's get you a robot</a></li>
<ul>
<li><a href=robot.html#section1>Robot description files</a></li>
<li><a href=robot.html#section2>Arms</a></li>
<ul>
<li>Position-controlled robots</li>
<li>Torque-controlled robots</li>
<li>A proliferation of hardware</li>
<li>Simulating the Kuka iiwa</li>
</ul>
<li><a href=robot.html#section3>Hands</a></li>
<ul>
<li>Dexterous hands</li>
<li>Simple grippers</li>
<li>Soft/underactuated hands</li>
<li>Other end effectors</li>
<li>If you haven't seen it...</li>
</ul>
<li><a href=robot.html#section4>Sensors</a></li>
<li><a href=robot.html#section5>Putting it all together</a></li>
<li><a href=robot.html#section6>Exercises</a></li>
</ul>
<li><a href="pick.html">Chapter 3: Basic Pick and Place</a></li>
<ul>
<li><a href=pick.html#monogram>Monogram Notation</a></li>
<li><a href=pick.html#section2>Pick and place via spatial transforms</a></li>
<li><a href=pick.html#spatial_algebra>Spatial Algebra</a></li>
<ul>
<li>Representations for 3D rotation</li>
</ul>
<li><a href=pick.html#kinematics>Forward kinematics</a></li>
<ul>
<li>The kinematic tree</li>
<li>Forward kinematics for pick and place</li>
</ul>
<li><a href=pick.html#jacobian>Differential kinematics (Jacobians)</a></li>
<li><a href=pick.html#section6>Differential inverse kinematics</a></li>
<ul>
<li>The Jacobian pseudo-inverse</li>
<li>Invertibility of the Jacobian</li>
</ul>
<li><a href=pick.html#section7>Defining the grasp and pre-grasp poses</a></li>
<li><a href=pick.html#section8>A pick and place trajectory</a></li>
<li><a href=pick.html#section9>Putting it all together</a></li>
<li><a href=pick.html#diff_ik_w_constraints>Differential inverse kinematics with
constraints</a></li>
<ul>
<li>Pseudo-inverse as an optimization</li>
<li>Adding velocity constraints</li>
<li>Adding position and acceleration constraints</li>
<li>Joint centering</li>
<li>Alternative formulations</li>
</ul>
<li><a href=pick.html#section11>Exercises</a></li>
</ul>
<li><a href="pose.html">Chapter 4: Geometric Pose Estimation</a></li>
<ul>
<li><a href=pose.html#section1>Cameras and depth sensors</a></li>
<ul>
<li>Depth sensors</li>
<li>Simulation</li>
</ul>
<li><a href=pose.html#section2>Representations for geometry</a></li>
<li><a href=pose.html#registration>Point cloud registration with known
correspondences</a></li>
<li><a href=pose.html#icp>Iterative Closest Point (ICP)</a></li>
<li><a href=pose.html#section5>Dealing with partial views and outliers</a></li>
<ul>
<li>Detecting outliers</li>
<li>Point cloud segmentation</li>
<li>Generalizing correspondence</li>
<li>Soft correspondences</li>
<li>Nonlinear optimization</li>
<li>Global optimization</li>
</ul>
<li><a href=pose.html#section6>Non-penetration and "free-space" constraints</a></li>
<ul>
<li>Free space constraints as non-penetration constraints</li>
</ul>
<li><a href=pose.html#section7>Tracking</a></li>
<li><a href=pose.html#section8>Putting it all together</a></li>
<li><a href=pose.html#section9>Looking ahead</a></li>
<li><a href=pose.html#exercises>Exercises</a></li>
</ul>
<li><a href="clutter.html">Chapter 5: Bin Picking</a></li>
<ul>
<li><a href=clutter.html#section1>Generating random cluttered scenes</a></li>
<ul>
<li>Falling things</li>
<li>Static equilibrium with frictional contact</li>
</ul>
<li><a href=clutter.html#section2>A few of the nuances of simulating contact</a></li>
<li><a href=clutter.html#section3>Model-based grasp selection</a></li>
<ul>
<li>Spatial force</li>
<li>The contact wrench cone</li>
<li>Colinear antipodal grasps</li>
</ul>
<li><a href=clutter.html#section4>Grasp selection from point clouds</a></li>
<ul>
<li>Point cloud pre-processing</li>
<li>Estimating normals and local curvature</li>
<li>Evaluating a candidate grasp</li>
<li>Generating grasp candidates</li>
</ul>
<li><a href=clutter.html#section5>The corner cases</a></li>
<li><a href=clutter.html#section6>Programming the Task Level</a></li>
<li><a href=clutter.html#section7>Putting it all together</a></li>
<li><a href=clutter.html#exercises>Exercises</a></li>
</ul>
<li><a href="segmentation.html">Chapter 6: Object detection and
segmentation</a></li>
<ul>
<li><a href=segmentation.html#section1>Getting to big data</a></li>
<ul>
<li>Crowd-sourced annotation datasets</li>
<li>Segmenting new classes via fine tuning</li>
<li>Annotation tools for manipulation</li>
<li>Synthetic datasets</li>
<li>Self-supervised learning</li>
</ul>
<li><a href=segmentation.html#section2>Object detection and segmentation</a></li>
<li><a href=segmentation.html#section3>Putting it all together</a></li>
<li><a href=segmentation.html#section4>Exercises</a></li>
</ul>
<li><a href="force.html">Chapter 7: Force Control</a></li>
<ul>
<li><a href=force.html#section1>A simple model</a></li>
<ul>
<li>(Direct) force control</li>
<li>A force-based "flip-up" strategy</li>
<li>Indirect force control</li>
<li>A stiffness-control-based "flip-up" strategy</li>
<li>Hybrid position/force control</li>
</ul>
<li><a href=force.html#section2>Peg in hole</a></li>
<li><a href=force.html#section3>Manipulator control</a></li>
<ul>
<li>Joint stiffness control</li>
<li>Cartesian stiffness control</li>
<li>Some implementation details on the iiwa</li>
</ul>
<li><a href=force.html#section4>Putting it all together</a></li>
<li><a href=force.html#section5>Exercises</a></li>
</ul>
<li><a href="trajectories.html">Chapter 8: Motion Planning</a></li>
<ul>
<li><a href=trajectories.html#section1>Inverse Kinematics</a></li>
<ul>
<li>From end-effector pose to joint angles</li>
<li>IK as constrained optimization</li>
<li>Grasp planning as IK</li>
</ul>
<li><a href=trajectories.html#section2>Kinematic trajectory optimization</a></li>
<ul>
<li>Trajectory parameterizations</li>
<li>Optimization algorithms</li>
</ul>
<li><a href=trajectories.html#section3>Sampling-based motion planning</a></li>
<ul>
<li>Rapidly-exploring random trees (RRT)</li>
<li>The Probabilistic Roadmap (PRM)</li>
<li>Post-processing</li>
</ul>
<li><a href=trajectories.html#section4>Time-optimal path parameterizations</a></li>
<li><a href=trajectories.html#section5>Active areas of research</a></li>
<li><a href=trajectories.html#section6>Exercises</a></li>
</ul>
<li><a href="rl.html">Chapter 9: Reinforcement Learning</a></li>
<ul>
<li><a href=rl.html#section1>RL Software</a></li>
<li><a href=rl.html#section2>Policy-gradient methods</a></li>
<ul>
<li>Black-box optimization</li>
<li>Stochastic optimal control</li>
<li>Using gradients of the policy, but not the environment</li>
<li>REINFORCE, PPO, TRPO</li>
<li>Control for manipulation should be easy</li>
</ul>
<li><a href=rl.html#section3>Value-based methods</a></li>
<li><a href=rl.html#section4>Model-based RL</a></li>
<li><a href=rl.html#section5>Exercises</a></li>
</ul>
<p style="margin-bottom: 0; text-decoration: underline;font-variant: small-caps;"><b>Appendix</b></p>
<li><a href="drake.html">Appendix A: Drake</a></li>
<ul>
<li><a href=drake.html#section1>Pydrake</a></li>
<li><a href=drake.html#notebooks>Online Jupyter Notebooks</a></li>
<ul>
<li>Running on Deepnote</li>
<li>Running on Google Colab</li>
<li>Enabling licensed solvers</li>
</ul>
<li><a href=drake.html#section3>Running on your own machine</a></li>
<li><a href=drake.html#section4>Getting help</a></li>
</ul>
<li><a href="station.html">Appendix B: Setting up your own "Manipulation Station"</a></li>
<ul>
<li><a href=station.html#section1>Message Passing</a></li>
<li><a href=station.html#section2>Kuka LBR iiwa + Schunk WSG Gripper</a></li>
<li><a href=station.html#section3>Intel Realsense D415 Depth Cameras</a></li>
<li><a href=station.html#section4>Miscellaneous hardware.</a></li>
</ul>
<li><a href="misc.html">Appendix C: Miscellaneous</a></li>
<ul>
<li><a href=misc.html#cite>How to cite these notes</a></li>
<li><a href=misc.html#annotation>Annotation tool etiquette</a></li>
<li><a href=misc.html#projects>Some great final projects</a></li>
<li><a href=misc.html#feedback>Please give me feedback!</a></li>
</ul>
</ul>
</section>
<section id="preface"><h1>Preface</h1>
<p>I've always loved robots, but it's only relatively recently that I've
turned my attention to robotic manipulation. I particularly like the
challenge of building robots that can master physics to achieve
human/animal-like dexterity and agility. It was <a
href="http://underactuated.mit.edu/intro.html">passive
dynamic walkers</a> and the beautiful analysis that accompanies them that
first helped cement this centrality of dynamics in my view of the world and my
approach to robotics. From there I became fascinated with (experimental)
fluid dynamics, and the idea that birds with articulated wings actually
"manipulate" the air to achieve incredible efficiency and agility. Humanoid
robots and fast-flying aerial vehicles in clutter forced me to start thinking
more deeply about the role of perception in dynamics and control. Now I
believe that this interplay between perception and dynamics is truly
fundamental, and I am passionate about the observation that relatively
"simple" problems in manipulation (how do I button up my dress shirt?) expose
the problem beautifully.</p>
<p>My approach to programming robots has always been very
computational/algorithmic. I started out using tools primarily from machine
learning (especially reinforcement learning) to develop the control systems
for simple walking machines; but as the robots and tasks got more complex I
turned to more sophisticated tools from model-based planning and
optimization-based control. In my view, no other discipline has thought so
deeply about dynamics as has control theory, and the algorithmic efficiency
and guaranteed performance/robustness that can be obtained by the best
model-based control algorithms far surpasses what we can do today with
learning control. Unfortunately, the mathematical maturity of
controls-related research has also led the field to be relatively conservative
in their assumptions and problem formulations; the requirements for robotic
manipulation break these assumptions. For example, robust control typically
assumes dynamics that are (nearly) smooth and uncertainty that can be
represented by simple distributions or simple sets; but in robotic
manipulation, we must deal with the non-smooth mechanics of contact and
uncertainty that comes from varied lighting conditions, and different numbers
of objects with unknown geometry and dynamics. In practice, no
state-of-the-art robotic manipulation system to date (that I know of) uses
rigorous control theory to design even the low-level feedback that determines
when a robot makes and breaks contact with the objects it is manipulating. An
explicit goal of these notes is to try to change that.</p>
<p>In the past few years, deep learning has had an unquestionable impact on
robotic perception, unblocking some of the most daunting challenges in
performing manipulation outside of a laboratory or factory environment. We
will discuss relevant tools from deep learning for object recognition,
segmentation, pose/keypoint estimation, shape completion, etc. Now relatively
old approaches to learning control are also enjoying an incredible surge in
popularity, fueled in part by massive computing power and increasingly
available robot hardware and simulators. Unlike learning for perception,
learning control algorithms are still far from a technology, with some of the
most impressive looking results still being hard to understand and to
reproduce. But the recent work in this area has unquestionably highlighted
the pitfalls of the conservatism taken by the controls community. Learning
researchers are boldly formulating much more aggressive and exciting problems
for robotic manipulation than we have seen before -- in many cases we are
realizing that some manipulation tasks are actually quite easy, but in other
cases we are finding problems that are still fundamentally hard.</p>
<p>Finally, it feels that the time is ripe for robotic manipulation to have a
real and dramatic impact in the world, in fields from logistics to home
robots. Over the last few years, we've seen UAVs/drones transition from
academic curiosities into consumer products. Even more recently, autonomous
driving has transitioned from academic research to industry, at least in
terms of dollars invested. Manipulation feels like the next big thing that
will make the leap from robotic research to practice. It's still a bit risky
for a venture capitalist to invest in, but nobody doubts the size of the
market once we have the technology. How lucky are we to potentially be able
to play a role in that transition?</p>
<p>So this is where the notes begin... we are at an incredible crossroads
between learning and control and robotics with an opportunity to have
immediate impact in industrial and consumer applications and potentially even
to forge entirely new eras for systems theory and controls. I'm just trying
to hold on and to enjoy the ride.</p>
<section><h1>A manipulation toolbox</h1>
<p>Another explicit goal of these lecture notes is to provide high-quality
implementations of the most useful tools in a manipulation scientist's
toolbox. When I am forced to choose between mathematical clarity and
runtime performance, the clear formulation is always my first priority; I
will try to include a performant formulation, too, if possible or try to
give pointers to alternatives. Manipulation research is moving quickly, and
I aim to evolve these notes to keep pace. I hope that the software
components provided in
<drake></drake> and in these notes can be directly useful to you in your own
work.</p>
<p>If you would like to replicate any or all of the hardware that we use for
these notes, you can find information and instructions in the <a
href="station.html">appendix</a>.</p>
<p>As you use the code, please consider <a
href="https://drake.mit.edu/getting_help.html">contributing back</a>
(especially to the mature code in <drake></drake>). Even questions/bug
reports can be important contributions. If you have questions/find issues
with these notes, please submit them <a
href="https://github.com/RussTedrake/manipulation/issues">here</a>.</p>
</section>
<p style="text-align:right;"><a href="intro.html">First chapter</a></p>
</section> <!-- end preface -->
<div id="footer">
<hr/>
<table style="width:100%;">
<tr><td><a href="https://accessibility.mit.edu/">Accessibility</a></td><td style="text-align:right">© Russ
Tedrake, 2020-2022</td></tr>
</table>
</div>
</body>
</html>