Lecture 12 Exercise Solutions

Exercise 1

Consider the function $$f\left(w_{1}, w_{2}, w_{3}, w_{4}, w_{5}\right) = w_{1}w_{2}w_{3}w_{4}w_{5}.$$

  1. Calculate the gradient using the reverse mode.
    • You may want to start by drawing the graph to help you visualize.
    • Set up an evaluation table. Note that this table can have the same columns as the example in class.
    • Write out the reverse mode based on the evaluation table.
  2. Calculate the gradient using forward mode.
    • You can use the same graph as in the reverse mode.
    • Set up a forward evaluation table. Note that this table will have more columns than the one you created for the reverse mode.
  3. For both forward and reverse mode, calculate the number of operations.
    • Hint: You may count only the floating point operations (e.g. addition and multiplication). You are not required to count memory access steps, retrievals, etc.

Solution to # 1

Node Current Value Numerical Value $\partial_{1}$ $\partial_{1}$ Value $\partial_{2}$ $\partial_{2}$ Value
$x_{1}$ $x_{1}$ $2$ $1$ $1$ $-$ $-$
$x_{2}$ $x_{2}$ $1$ $1$ $1$ $-$ $-$
$x_{3}$ $x_{3}$ $1$ $1$ $1$ $-$ $-$
$x_{4}$ $x_{4}$ $1$ $1$ $1$ $-$ $-$
$x_{5}$ $x_{5}$ $1$ $1$ $1$ $-$ $-$
$x_{6}$ $x_{4}x_{5}$ $1$ $x_{5}$ $1$ $x_{4}$ $1$
$x_{7}$ $x_{3}x_{6}$ $1$ $x_{6}$ $1$ $x_{3}$ $1$
$x_{8}$ $x_{2}x_{7}$ $1$ $x_{7}$ $1$ $x_{2}$ $1$
$x_{9}$ $x_{1}x_{8}$ $2$ $x_{8}$ $1$ $x_{1}$ $2$
\begin{align*} \overline{x}_{9} &= \dfrac{\partial f}{\partial x_{9}} = 1 \\ \overline{x}_{8} &= \dfrac{\partial f}{\partial x_{9}}\dfrac{\partial x_{9}}{\partial x_{8}} = 1 \cdot 1 = 1 \\ \overline{x}_{7} &= \dfrac{\partial f}{\partial x_{8}}\dfrac{\partial x_{8}}{\partial x_{7}} = 2 \cdot 1 = 2 \\ \overline{x}_{6} &= \dfrac{\partial f}{\partial x_{7}}\dfrac{\partial x_{7}}{\partial x_{6}} = 2 \cdot 1 = 2 \\ \overline{x}_{5} &= \dfrac{\partial f}{\partial x_{6}}\dfrac{\partial x_{6}}{\partial x_{5}} = 2 \cdot 1 = 2 \\ \overline{x}_{4} &= \dfrac{\partial f}{\partial x_{6}}\dfrac{\partial x_{6}}{\partial x_{4}} = 2 \cdot 1 = 2 \\ \overline{x}_{3} &= \dfrac{\partial f}{\partial x_{7}}\dfrac{\partial x_{7}}{\partial x_{3}} = 2 \cdot 1 = 2 \\ \overline{x}_{2} &= \dfrac{\partial f}{\partial x_{8}}\dfrac{\partial x_{8}}{\partial x_{2}} = 2 \cdot 1 = 2 \\ \overline{x}_{1} &= \dfrac{\partial f}{\partial x_{9}}\dfrac{\partial x_{9}}{\partial x_{1}} = 1 \cdot 1 = 1 \\ \end{align*}

Solution to # 2

Node Elemental Numerical Value Derivative $\partial_{1}$ Value $\partial_{2}$ Value $\partial_{3}$ Value $\partial_{4}$ Value $\partial_{5}$ Value
$x_{1}$ $x_{1}$ $2$ $\dot{x}_{1}$ $1$ $0$ $0$ $0$ $0$
$x_{2}$ $x_{2}$ $1$ $\dot{x}_{2}$ $0$ $1$ $0$ $0$ $0$
$x_{3}$ $x_{3}$ $1$ $\dot{x}_{3}$ $0$ $0$ $1$ $0$ $0$
$x_{4}$ $x_{4}$ $1$ $\dot{x}_{4}$ $0$ $0$ $0$ $1$ $0$
$x_{5}$ $x_{5}$ $1$ $\dot{x}_{5}$ $0$ $0$ $0$ $0$ $1$
$x_{6}$ $x_{4}x_{5}$ $1$ $x_{4}\dot{x}_{5} + x_{5}\dot{x}_{4}$ $0$ $0$ $0$ $1$ $1$
$x_{7}$ $x_{3}x_{6}$ $1$ $x_{3}\dot{x}_{6} + x_{6}\dot{x}_{3}$ $0$ $0$ $1$ $1$ $1$
$x_{8}$ $x_{2}x_{7}$ $1$ $x_{2}\dot{x}_{7} + x_{7}\dot{x}_{2}$ $0$ $1$ $1$ $1$ $1$
$x_{9}$ $x_{1}x_{8}$ $2$ $x_{1}\dot{x}_{8} + x_{8}\dot{x}_{1}$ $1$ $2$ $2$ $2$ $2$

Solution to # 3

Forward ops: $4 + 3 \cdot 5 \cdot 4 = 64$.

Reverse ops: $4 + 8 = 12$.