Units: A Unit Conversion Program and Scientific Calculator: Precision

Previous: Field Width, Up: Numeric Output Format [Contents][Index]

### 11.4 Precision

The meaning of “precision” depends on the format type. With ‘`g`’ or ‘`G`’, it specifies the number of significant digits (like the `--digits` option); with ‘`e`’, ‘`E`’, ‘`f`’, or ‘`F`’, it specifies the maximum number of digits to be shown after the decimal point.

With the ‘`g`’ and ‘`G`’ format types, trailing zeros are suppressed, so the results may sometimes have fewer digits than the specified precision (as indicated above, the ‘`#`’ flag causes trailing zeros to be displayed).

The default precision is 6, so ‘`%g`’ is equivalent to ‘`%.6g`’, and would show the output to six significant digits. Similarly, ‘`%e`’ or ‘`%f`’ would show the output with six digits after the decimal point.

The C `printf()` function allows a precision of arbitrary size, whether or not all of the digits are meaningful. With most compilers, the maximum internal precision with `units` is 15 decimal digits (or 13 hexadecimal digits). With the `--digits` option, you are limited to the maximum internal precision; with the `--output-format` option, you may specify a precision greater than this, but it may not be meaningful. In some cases, specifying excess precision can result in rounding artifacts. For example, a pound is exactly 7000 grains, but with the format ‘`%.18g`’, the output might be

    You have: pound
    You want: grain
            * 6999.9999999999991
            / 0.00014285714285714287

With the format ‘`%.25g`’ you might get the following:

    You have: 1/3
    You want:
            Definition: 0.333333333333333314829616256247

In this case the displayed value includes a series of digits that represent the underlying binary floating-point approximation to 1/3 but are not meaningful for the desired computation. In general, the result with excess precision is system dependent. The precision affects only the display of numbers; if a result relies on physical constants that are not known to the specified precision, the number of physically meaningful digits may be less than the number of digits shown.

See the documentation for `printf()` for more detailed descriptions of the format specification.

The `--output-format` option is incompatible with the `--exponential` or `--digits` options; if the former is given in combination with either of the latter, the format is controlled by the last option given.

Previous: Field Width, Up: Numeric Output Format [Contents][Index]