]> chapter08.xml

Chapter 8
Differentiation

8.1 Problem 1: Numerical Limits

A particle is moving through space, and you record its position as a function of time x ( t ) in a table. Your problem is to determine its velocity v ( t ) = d x / d t when all you have is this table of x versus t .

8.2 Method: Numeric

You probably did rather well in your first calculus course and feel competent at taking derivatives. However, you probably did not take derivatives of a table of numbers using the elementary definition:

d f ( x ) d x = def lim h → 0 f ( x + h ) - f ( x ) h .      (8.1) In fact, even a computer runs into errors with this kind of limit because it is wrought with subtractive cancellation; the computer's finite word length causes the numerator to fluctuate between 0 and the machine precision ε m as the denominator approaches zero.

8.2.1 Method: Forward Difference

The most direct method for numerical differentiation of a function starts by expanding it in a Taylor series. This series advances the function one small step forward:

f ( x + h ) = f ( x ) + h f ' ( x ) + h 2 2 f '' ( x ) + h 3 6 f ( 3 ) ( x ) + … ,      (8.2) where h is the step size . This is illustrated in Fig.  ?? . We obtain the forward-difference derivative algorithm by solving ( 8.2 ) for f ' ( x ) : f c ' ( x ) ≃ f ( x + h ) - f ( x ) h ,      (8.3) ≃ f ' ( x ) + h 2 f '' ( x ) + ⋯ ,      (8.4)

where the subscript c denotes a computed expression. You can think of this approximation as using two points to represent the function by a straight line in the interval from x to x + h .

The approximation ( 8.3 ) has an error proportional to h (unless the "ultrauser" looks down kindly upon you and makes f '' vanish). We can make the approximation error smaller and smaller by making h smaller and smaller. For too small an h , however, all precision will be lost through the subtractive cancellation on the LHS of ( 8.4 ), and the decreased approximation error becomes irrelevant.

PIC
Figure 8.1: Forward difference (solid line) and central-difference (dashed line) methods for numerical first derivative.
For example, consider a case where f ( x ) is a simple, analytic polynomial: f ( x ) = a + b x 2 .      (8.5) The exact derivative is f ' = 2 b x .      (8.6) The computed derivative is f c ' ( x ) ≈ f ( x + h ) - f ( x ) h = 2 b x + b h .      (8.7) This clearly becomes a good approximation only for small h ( h ≪ 2 x ).

8.2.2 Method: Central Difference

An improved approximation to the derivative starts with the basic definition ( 8.1 ). Rather than making a single step of h forward, we form a central difference by stepping forward by h / 2 and backward by h / 2 :

f c ' ( x ) ≈ f ( x + h / 2 ) - f ( x - h / 2 ) h = def D c f ( x , h ) .      (8.8) Here D c is the symbol for the central difference. The central-difference approximation is illustrated in Fig.  ?? . When the Taylor series for f ( x ± h / 2 ) are substituted into ( 8.8 ), we obtain f c ' ( x ) ≃ f ' ( x ) + 1 2 4 h 2 f ( 3 ) ( x ) + ⋯ .      (8.9) The important difference from ( 8.3 ) is that when f ( x - h / 2 ) is subtracted from f ( x + h / 2 ) , all terms containing an odd power of h in the Taylor series cancel. Therefore, the central-difference algorithm becomes accurate to one order higher in h ; that is, h 2 . If the function is well behaved; that is, if ( f ( 3 ) h 2 ) / 2 4 ≪ ( f ( 2 ) h ) / 2 , then you can expect the error with the central-difference method to be smaller than with the forward difference ( 8.4 ). If we now return to our polynomial example ( 8.5 ), we find that for a parabola, the central difference gives the exact answer regardless of the size of h : f c ' ( x ) ≈ f ( x + h / 2 ) - f ( x - h / 2 ) h = 2 b x .      (8.10)

8.2.3 Method: Extrapolated Difference

Because the Taylor series provides an analytic expression for the error, we can be even more clever. While the central difference ( 8.8 ) makes the error term proportional to h vanish, we can also make the term proportional to h 2 vanish by algebraically extrapolating from relatively large h (and, because of this, small roundoff error) to h → 0 :

f c ' ( x ) ≃ lim h → 0 D c f ( x , h ) .      (8.11) We introduce the additional information by forming the central difference with step size h / 2 : D c f ( x , h / 2 ) = def f ( x + h / 4 ) - f ( x - h / 4 ) h / 2      (8.12) ≈ f ' ( x ) + h 2 f ( 3 ) ( x ) 9 6 + ⋯ .      (8.13)

We now eliminate the quadratic error term as well as the linear error term in ( 8.9 ) with the combination

f c ' ( x ) ≈ 4 D c f ( x , h / 2 ) - D c f ( x , h ) 3      (8.14) ≈ f ' ( x ) - h 4 f ( 3 ) ( x ) 1 9 2 0 × 4 × 1 6 + ⋯ .      (8.15)

If h = 0 . 4 , there is only one place of roundoff error, and if f ( 3 ) = O ( 1 ) , the truncation error in ( 8.15 ) is now the same size as machine precision ε m , which is the best you can hope for.

A good way of computing ( 8.14 ) is to group the terms as

f c ' ( x ) = 1 3 h 8 f ( x + h 4 ) - f ( x - h 4 ) - f ( x + h 2 ) - f ( x - h 2 ) .      (8.16) The advantage to ( 8.16 ) is that it reduces the loss of precision that occurs when large and small numbers are added together, only to be subtracted from other large numbers; it is better to first subtract the large numbers from each other and then add the difference to the small numbers. When working with these and similar higher-order methods, it is important to remember that while they may work very well for well-behaved functions, they may fail badly for computed or measured functions containing noise. In these difficult cases it may be better to first fit the data with some analytic function using the techniques of Chapter  ?? , Data Fitting , and then differentiate the fit.

But regardless of the algorithm, you may remember that evaluating the derivative of f ( x ) at x requires you to know the values of f surrounding x . We shall use this same idea in Chapter  ?? , Differential Equations and Oscillations .

8.3 Assessment: Error Analysis

The approximation errors in numerical differentiation decrease with decreasing step size h while roundoff errors increase with a smaller step size (you have to take more steps and do more calculations). We know from our discussion in Chapter  ?? , Errors and Uncertainties in Computations , that the best approximation occurs for an h that makes the total error ε a p p r o x + ε r o a minimum. As a rough guide, this occurs when ε r o ≈ ε a p p r o x .

Because differentiation subtracts two numbers very close in value, the limit of roundoff error is essentially machine precision:

f ' ≈ f ( x + h ) - f ( x ) h ≈ ε m h ,      (8.17) ⇒ ε r o ≈ ε m h .      (8.18)

The approximation error with the forward-difference algorithm ( 8.4 ) is an O ( h ) term, while that with the central-difference algorithm ( 8.9 ) is an O ( h 2 ) term;

ε a p p r o x f d ≈ f ( 2 ) h 2 ,      (8.19) ε a p p r o x c d ≈ f ( 3 ) h 2 2 4 .      (8.20)

The h value for which roundoff and approximation errors are equal is therefore

ε m h ≈ ε a p p r o x f d = f ( 2 ) h 2 ,      (8.21) ε m h ≈ ε a p p r o x c d = f ( 3 ) h 2 2 4 ,      (8.22) ⇒ h f d 2 = 2 ε m f ( 2 ) , h c d 3 = 2 4 ε m f ( 3 ) .      (8.23)

We take f ' ≈ f ( 2 ) ≈ f ( 3 ) (which is crude, although not really so crude for e x or cos x ), and assume a single precision calculation ε m ≈ 1 0 - 7 . In this case we obtain

h f d ≈ 0 . 0 0 0 5 ,      (8.24) h c d ≈ 0 . 0 1 .      (8.25)

This may seem backward because the better algorithm leads to a larger h value. It is not. The ability to use a large h means that the error in the central-difference method is some 20 times smaller than the error in the forward-difference method. For the forward-difference algorithm, approximately half of the machine precision is lost.

8.4 Implementation: Differentiation, diff.f (.c)

8.5 Assessment: Error Analysis, Numerical

  1. Differentiate the functions cos x and e x at x = 0 . 1 , 1 . , and 1 0 0 using single-precision forward-, central-, and extrapolated-difference algorithms.
    1. Print out the derivative and its relative error E as a function of h . Reduce the step size h until it equals machine precision h ≈ ε m .
    2. Plot log 1 0 | E | versus log 1 0 h and check whether the number of decimal places obtained agrees with the estimates in the text.
    3. See if you can identify truncation error at large h and roundoff error at small h in your plot. Do the slopes agree with our predictions?

8.6 Problem 2: Second Derivatives

Let's say that you have measured the position versus time x ( t ) for a particle. Your problem is to determine the force on the particle.

8.7 Theory: Newton II

Newton's second law tells us that the force and acceleration are linearly related:

F = m a ,      (8.26) where F is the force, m is the particle's mass, and a is the acceleration. So if we can determine the acceleration a ( t ) = d 2 x / d t 2 from the x ( t ) table, we can determine the force.

8.8 Method: Numerical Second Derivatives

The concern about errors we expressed for first derivatives is even more of a concern for second derivatives because the additional subtractions lead to more cancellations. Let's go back to the central-difference method:

f ' ( x ) ≃ f ( x + h / 2 ) - f ( x - h / 2 ) h .      (8.27) This algorithm gives the derivative at x by moving forward and backward from x by h / 2 . The second derivative f ( 2 ) ( x ) is taken to be the central difference of the first derivative: f ( 2 ) ( x ) ≃ f ' ( x + h / 2 ) - f ' ( x - h / 2 ) h ,      (8.28) f ( 2 ) ( x ) ≃ [ f ( x + h ) - f ( x ) ] - [ f ( x ) - f ( x - h ) ] h 2 .      (8.29)

As was true for first derivatives, by evaluating a function in the region surrounding x , it is possible to determine the second derivative at x . But some care is required to preserve the grouping in ( 8.29 ) and not convert it to the neater form:

f ( 2 ) ( x ) ≃ f ( x + h ) + f ( x - h ) - 2 f ( x ) h 2 .      (8.30) This latter form increases subtractive cancellation because the computer first stores the "large" number f ( x + h ) + f ( x - h ) [well, if f does not change sign, then the sum is larger than either f ( x + h ) or f ( x - h ) ] only to subtract another large number 2 f ( x ) from it.

8.9 Assessment: Numerical Second Derivatives

Write a program to calculate the second derivative of cos x using the central-difference algorithm ( 8.29 ). Test it over four cycles. Start with h ≈ π / 1 0 and keep reducing h until you reach machine precision.