Key Word(s): Automatic differentiation, Forward mode
This week's exercises are all about getting familiar with automatic differentiation in the forward mode. There are no coding portions this week. Each exercise will provide information on how to submit.
Exercise 1¶
You will work with the following function for this exercise, \begin{align} f\left(x,y\right) = \exp\left(-\left(\sin\left(x\right) - \cos\left(y\right)\right)^{2}\right). \end{align}
- Draw the computational graph for the function.
- Note: This graph will have $2$ inputs.
- Create the evaluation trace for this function.
- Use the graph / trace to evaluate $f\left(\dfrac{\pi}{2}, \dfrac{\pi}{3}\right)$.
- Compute $\dfrac{\partial f}{\partial x}\left(\dfrac{\pi}{2}, \dfrac{\pi}{3}\right)$ and $\dfrac{\partial f}{\partial y}\left(\dfrac{\pi}{2}, \dfrac{\pi}{3}\right)$ using the forward mode of AD.
Hint¶
You can set up a table with the following headings:
| trace | elem op. | value | elem der. | $\nabla_{x}$ | $\nabla_{y}$ |
|---|---|---|---|---|---|
| $v_{2}$ | $x_{2}^{2}$ | $1$ | $2x_{2}\dot{x}_{2}$ | $1$ | $2$ |
| $v_{3}$ | $v_{2}^{2}$ | $1$ | $2v_{2}\dot{v}_{2}$ | $2$ | $4$ |
A few key observations:
- First of all, the values in the table is made up and for purely illustrative purposes.
- The first column represents where you are in tracing out the computation.
- The second column is the current elementary operation. In this example, we're just squaring one of the inputs.
- The third column is the value of the function at this point in the evaluation. Here we're just pretending it's $1$.
- The fourth column is the derivative of the elementary funtion. This is where the chain rule happens!
- The fifth and sixth columns are where things get interesting:
- Fifth column. The derivative value evaluated from the fourth column where the dot is interpreted as a derivative with respect to $x$.
- Sixth column. The derivative value evaluated from the fourth column where the dot is interpreted as a derivative with respect to $y$.
So, in this example, we need to calculate $2v_{2}\dot{v}_{2}$. From the previous row, we see that $\dot{v}_{2}$ could be $1$ or $2$ depending on which derivative the dot represents! In our $\nabla_{x}$ column, we know that the dot means derivatives with respect to $x$ and in our $\nabla_{y}$ column we know that the dot means derivatives with respect to $y$. Plugging the appropriate values in from the previous row gives us the right values.
Deliverables¶
exercise_1.md- A markdown table for the evaluation trace.
- Solutions for the derivative evaluations at the specified points.