Lecture 12 Exercise Solutions¶
Exercise 1¶
Consider the function $$f\left(w_{1}, w_{2}, w_{3}, w_{4}, w_{5}\right) = w_{1}w_{2}w_{3}w_{4}w_{5}.$$
- Calculate the gradient using the reverse mode.
- You may want to start by drawing the graph to help you visualize.
- Set up an evaluation table. Note that this table can have the same columns as the example in class.
- Write out the reverse mode based on the evaluation table.
- Calculate the gradient using forward mode.
- You can use the same graph as in the reverse mode.
- Set up a forward evaluation table. Note that this table will have more columns than the one you created for the reverse mode.
- For both forward and reverse mode, calculate the number of operations.
- Hint: You may count only the floating point operations (e.g. addition and multiplication). You are not required to count memory access steps, retrievals, etc.
Solution to # 1¶
| Node | Current Value | Numerical Value | $\partial_{1}$ | $\partial_{1}$ Value | $\partial_{2}$ | $\partial_{2}$ Value |
|---|---|---|---|---|---|---|
| $x_{1}$ | $x_{1}$ | $2$ | $1$ | $1$ | $-$ | $-$ |
| $x_{2}$ | $x_{2}$ | $1$ | $1$ | $1$ | $-$ | $-$ |
| $x_{3}$ | $x_{3}$ | $1$ | $1$ | $1$ | $-$ | $-$ |
| $x_{4}$ | $x_{4}$ | $1$ | $1$ | $1$ | $-$ | $-$ |
| $x_{5}$ | $x_{5}$ | $1$ | $1$ | $1$ | $-$ | $-$ |
| $x_{6}$ | $x_{4}x_{5}$ | $1$ | $x_{5}$ | $1$ | $x_{4}$ | $1$ |
| $x_{7}$ | $x_{3}x_{6}$ | $1$ | $x_{6}$ | $1$ | $x_{3}$ | $1$ |
| $x_{8}$ | $x_{2}x_{7}$ | $1$ | $x_{7}$ | $1$ | $x_{2}$ | $1$ |
| $x_{9}$ | $x_{1}x_{8}$ | $2$ | $x_{8}$ | $1$ | $x_{1}$ | $2$ |
\begin{align*}
\overline{x}_{9} &= \dfrac{\partial f}{\partial x_{9}} = 1 \\
\overline{x}_{8} &= \dfrac{\partial f}{\partial x_{9}}\dfrac{\partial x_{9}}{\partial x_{8}} = 1 \cdot 1 = 1 \\
\overline{x}_{7} &= \dfrac{\partial f}{\partial x_{8}}\dfrac{\partial x_{8}}{\partial x_{7}} = 2 \cdot 1 = 2 \\
\overline{x}_{6} &= \dfrac{\partial f}{\partial x_{7}}\dfrac{\partial x_{7}}{\partial x_{6}} = 2 \cdot 1 = 2 \\
\overline{x}_{5} &= \dfrac{\partial f}{\partial x_{6}}\dfrac{\partial x_{6}}{\partial x_{5}} = 2 \cdot 1 = 2 \\
\overline{x}_{4} &= \dfrac{\partial f}{\partial x_{6}}\dfrac{\partial x_{6}}{\partial x_{4}} = 2 \cdot 1 = 2 \\
\overline{x}_{3} &= \dfrac{\partial f}{\partial x_{7}}\dfrac{\partial x_{7}}{\partial x_{3}} = 2 \cdot 1 = 2 \\
\overline{x}_{2} &= \dfrac{\partial f}{\partial x_{8}}\dfrac{\partial x_{8}}{\partial x_{2}} = 2 \cdot 1 = 2 \\
\overline{x}_{1} &= \dfrac{\partial f}{\partial x_{9}}\dfrac{\partial x_{9}}{\partial x_{1}} = 1 \cdot 1 = 1 \\
\end{align*}
Solution to # 2¶
| Node | Elemental | Numerical Value | Derivative | $\partial_{1}$ Value | $\partial_{2}$ Value | $\partial_{3}$ Value | $\partial_{4}$ Value | $\partial_{5}$ Value |
|---|---|---|---|---|---|---|---|---|
| $x_{1}$ | $x_{1}$ | $2$ | $\dot{x}_{1}$ | $1$ | $0$ | $0$ | $0$ | $0$ |
| $x_{2}$ | $x_{2}$ | $1$ | $\dot{x}_{2}$ | $0$ | $1$ | $0$ | $0$ | $0$ |
| $x_{3}$ | $x_{3}$ | $1$ | $\dot{x}_{3}$ | $0$ | $0$ | $1$ | $0$ | $0$ |
| $x_{4}$ | $x_{4}$ | $1$ | $\dot{x}_{4}$ | $0$ | $0$ | $0$ | $1$ | $0$ |
| $x_{5}$ | $x_{5}$ | $1$ | $\dot{x}_{5}$ | $0$ | $0$ | $0$ | $0$ | $1$ |
| $x_{6}$ | $x_{4}x_{5}$ | $1$ | $x_{4}\dot{x}_{5} + x_{5}\dot{x}_{4}$ | $0$ | $0$ | $0$ | $1$ | $1$ |
| $x_{7}$ | $x_{3}x_{6}$ | $1$ | $x_{3}\dot{x}_{6} + x_{6}\dot{x}_{3}$ | $0$ | $0$ | $1$ | $1$ | $1$ |
| $x_{8}$ | $x_{2}x_{7}$ | $1$ | $x_{2}\dot{x}_{7} + x_{7}\dot{x}_{2}$ | $0$ | $1$ | $1$ | $1$ | $1$ |
| $x_{9}$ | $x_{1}x_{8}$ | $2$ | $x_{1}\dot{x}_{8} + x_{8}\dot{x}_{1}$ | $1$ | $2$ | $2$ | $2$ | $2$ |
Solution to # 3¶
Forward ops: $4 + 3 \cdot 5 \cdot 4 = 64$.
Reverse ops: $4 + 8 = 12$.