Key Word(s): Automatic differentiation, Forward mode, Dual numbers
Automatic Differentiation and Dual Numbers¶
A dual number is an extension of the real numbers. Written out, the form looks similar to a complex number.
Review of Complex Numbers¶
Recall that a complex number has the form z=a+ib where we define the number i so that i2=−1.
No real number has this property but it is a useful property for a number to have. Hence the introduction of complex numbers.
Visually, you can think of a real number as a number lying on a straight line. Then, we "extend" the real line "up". The new axis is called the imaginary axis.
Complex numbers have several properties that we can use.
- Complex conjugate: z∗=a−ib.
- Magnitude of a complex number: |z|2=zz∗=(a+ib)(a−ib)=a2+b2.
- Polar form: z=|z|exp(iθ) where θ=tan−1(ba).
Towards Dual Numbers¶
A dual number has a real part and a dual part. We write z=a+ϵb and refer to b as the dual part.
We define the number ϵ so that ϵ2=0.
This does not mean that ϵ is zero! ϵ is not a real number.
Some properties of dual numbers:¶
- Conjugate: z∗=a−ϵb.
- Magnitude: |z|2=zz∗=(a+ϵb)(a−ϵb)=a2.
- Polar form: z=a(1+ba).
Example¶
Recall that the derivative of y=x2 is y′=2x.
Now if we extend x so that it has a real part and a dual part (x←a+ϵb) and evaluate y we have y=(a+ϵb)2=a2+2abϵ+b2ϵ2⏟=0=a2+2abϵ.
Notice that the dual part contains the derivative of our function evaluated at a!!¶
Expanding cos and sin in their Taylor series gives sin(ϵb)=∞∑n=0(−1)n(ϵb)2n+1(2n+1)!=ϵb+(ϵb)33!+⋯=ϵbcos(ϵb)=∞∑n=0(−1)n(ϵb)2n(2n)!=1+(ϵb)22+⋯=1. Note that the definition of ϵ was used which resulted in the collapsed sum.
So we see that y=sin(a)+cos(a)bϵ. And once again the real component is the function and the dual component is the derivative.
Automatic Differentiation and Dual Numbers¶
A dual number is an extension of the real numbers. Written out, the form looks similar to a complex number.
Review of Complex Numbers¶
Recall that a complex number has the form z=a+ib where we define the number i so that i2=−1. No real number has this property but it is a useful property for a number to have. Hence the introduction of complex numbers. Visually, you can think of a real number as a number lying on a straight line. Then, we "extend" the real line "up". The new axis is called the imaginary axis.
Complex numbers have several properties that we can use.
- Complex conjugate: z∗=a−ib.
- Magnitude of a complex number: |z|2=zz∗=(a+ib)(a−ib)=a2+b2.
- Polar form: z=|z|exp(iθ) where θ=tan−1(ba).
Towards Dual Numbers¶
A dual number has a real part and a dual part. We write z=a+ϵb and refer to b as the dual part. We define the number ϵ so that ϵ2=0. This does not mean that ϵ is zero! ϵ is not a real number.
Some properties of dual numbers:¶
- Conjugate: z∗=a−ϵb.
- Magnitude: |z|2=zz∗=(a+ϵb)(a−ϵb)=a2.
- Polar form: z=a(1+ba).
Example¶
Recall that the derivative of y=x2 is y′=2x.
Now if we extend x so that it has a real part and a dual part (x←a+ϵa) and evaluate y we have y=(a+ϵb)2=a2+2abϵ+b2ϵ2⏟=0=a2+2abϵ.
Notice that the dual part contains the derivative of our function!!¶
Example¶
Evaluate y=sin(x) when x←a+ϵb.
We have y=sin(a+ϵb)=sin(a)cos(ϵb)+cos(a)sin(ϵb). Expanding cos and sin in their Taylor series gives sin(ϵb)=∞∑n=0(−1)n(ϵb)2n+1(2n+1)!=ϵb+(ϵb)33!+⋯=ϵbcos(ϵb)=∞∑n=0(−1)n(ϵb)2n(2n)!=1+(ϵb)22+⋯=1. Note that the definition of ϵ was used which resulted in the collapsed sum.
So we see that y=sin(a)+cos(a)bϵ. And once again the real component is the function and the dual component is the derivative.
Exercise 1¶
Using dual numbers, find the derivative of y=ex2. Show your work!
You do not need to turn this exercise in.