## Why should I consider automatic differentiation?

The following are the reasons why you should consider automatic differentiation
in your computational project:

- your functions are extremely complicated;
- manually implementing analytical derivatives is a tedious and error-prone
  task; and
- computing derivatives using finite differences can be inaccurate and
  inefficient.

## What is the difference between reverse and forward modes?

Here is a brief, practical, and qualitative discussion on the differences
between these two automatic differentiation algorithms.

In a forward mode algorithm, each function evaluation produces not only its
output value but also its derivative. The evaluation of a vector function, for
example, computes both the output vector as well as the derivative of this
vector, in general, with respect to one of the input variables (e.g., *∂u/∂x*,
*∂u/∂y*). However, the forward mode algorithm can also be used to compute
directional derivatives. In this case, the derivative of the output vector with
respect to a given direction vector is performed. We could say that,
fundamentally, the forward mode always computes directional derivatives. The
simplest case of a derivative with respect to a single input variable
corresponds thus with a directional derivative whose direction vector is a unit
vector along the input variable of interest (e.g., the unit vector along *x*,
or along *y*, and so forth).

In a reverse mode algorithm, each function evaluation produces a complete
expression tree that contains the sequence of all mathematical operations
between the input variables to produce the output variable (a scalar). Once
this expression tree is constructed, it is then used to compute the derivatives
of the scalar output variable with respect to all input variables.

## Is one algorithm always faster than another?

Even though the reverse mode algorithm requires a single function evaluation
and all derivatives are subsequently computed in a single pass, as the
constructed expression tree is traversed, **the forward algorithm can still be
a much more efficient choice for your application**, despite its multiple,
repeated function evaluations, one for each input variable. This is because the
implementation of the forward mode algorithm in {{autodiff}} uses *template
meta-programming* techniques to avoid as much as possible temporary memory
allocations and to optimize the calculations in certain cases. The reverse mode
algorithm, on the other hand, requires the construction of an expression tree
at runtime, and, for this, several dynamic memory allocations are needed, which
can be costly. We plan to implement alternative versions of this algorithm, in
which memory allocation could be done in advance to decrease the number of
subsequent allocations. This, however, will require a slightly more complicated
usage than it is already provided by the reverse mode algorithm implemented in
{{autodiff}}.

## Which automatic differentiation algorithm should I use?

Ideally, you should try both algorithms for your specific needs, benchmark
them, and then make an informed decision about which one to use.

If you're in a hurry, consider:

**forward mode**: if you have a vector function, or a scalar function with not
many input variables.

**reverse mode**: if you have a scalar function with many (thousands or more)
input variables.

Have in mind this is a very simplistic rule, and you should definitely try both
algorithms whenever possible, since the forward mode could still be faster than
reverse mode even when many input variables are considered for a function of
interest.
