Chapter 2

Altitude Measurement

Although altitudes and zenith distances are equally suitable for navigational calculations, most formulas are traditionally
based upon altitudes which are easily accessible using the visible sea horizon as a natural reference line. Direct
measurement of the zenith distance, however, requires an instrument with an artificial horizon, e.g., a pendulum or spirit
level indicating the direction of the normal force (perpendicular to the local horizontal plane), since a reference point in
the sky does not exist.

Instruments

A marine sextant consists of a system of two mirrors and a telescope mounted on a metal frame. A schematic
illustration (side view) is given in Fig. 2-1. The rigid horizon glass is a semi-translucent mirror attached to the frame.
The fully reflecting index mirror is mounted on the so-called index arm rotatable on a pivot perpendicular to the frame.
When measuring an altitude, the instrument frame is held in a vertical position, and the visible sea horizon is viewed
through the scope and horizon glass. A light ray coming from the observed body is first reflected by the index mirror
and then by the back surface of the horizon glass before entering the telescope. By slowly rotating the index mirror on
the pivot the superimposed image of the body is aligned with the image of the horizon. The corresponding altitude,
which is twice the angle formed by the planes of horizon glass and index mirror, can be read from the graduated limb,
the lower, arc-shaped part of the sextant frame (not shown). Detailed information on design, usage, and maintenance of
sextants is given in [3] (see appendix).

On land, where the horizon is too irregular to be used as a reference line, altitudes have to be measured by means of
instruments with an artificial horizon:

A bubble attachment is a special sextant telescope containing an internal artificial horizon in the form of a small
spirit level whose image, replacing the visible horizon, is superimposed with the image of the body. Bubble attachments
are expensive (almost the price of a sextant) and not very accurate because they require the sextant to be held absolutely
still during an observation, which is difficult to manage. A sextant equipped with a bubble attachment is referred to as a
bubble sextant. Special bubble sextants were used for air navigation before electronic navigation systems became
standard equipment.

A pan filled with water, or preferably an oily liquid like glycerol, can be utilized as an external artificial horizon. Due
to the gravitational force, the surface of the liquid forms an exactly horizontal mirror unless distorted by vibrations or
wind. The vertical angular distance between a body and its mirror image, measured with a marine sextant, is twice the
altitude. This very accurate method is the perfect choice for exercising celestial navigation in a backyard.
A theodolite is basically a telescopic sight which can be rotated about a vertical and a horizontal axis. The angle of
elevation is read from the vertical circle, the horizontal direction from the horizontal circle. Built-in spirit levels are used
to align the instrument with the plane of the sensible horizon before starting the observations (artificial horizon).
Theodolites are primarily used for surveying, but they are excellent navigation instruments as well. Many models can
measure angles to 0.1' which cannot be achieved even with the best sextants. A theodolite is mounted on a tripod and has
to stand on solid ground. Therefore, it is restricted to land navigation. Traditionally, theodolites measure zenith
distances. Modern models can optionally measure altitudes.

Never view the sun through an optical instrument without inserting a proper shade glass, otherwise your eye
might suffer permanent damage !

Altitude corrections

Any altitude measured with a sextant or theodolite contains errors. Altitude corrections are necessary to
eliminate systematic altitude errors and to reduce the altitude measured relative to the visible or sensible horizon
to the altitude with respect to the celestial horizon and the center of the earth (chapter 1). Of course, altitude
corrections do not remove random errors.

Index error (IE)

A sextant or theodolite, unless recently calibrated, usually has a constant error (index error, IE) which has to be
subtracted from the readings before they can be processed further. The error is positive if the displayed value is greater
than the actual value and negative if the displayed value is smaller. Angle-dependent errors require alignment of the
instrument or the use of an individual correction table.

1st correction : H 1 = Hs − IE

The sextant altitude, Hs, is the altitude as indicated by the sextant before any corrections have been applied.

When using an external artificial horizon, H1 (not Hs!) has to be divided by two.

A theodolite measuring the zenith distance, z, requires the following formula to obtain H1:

H 1 = 90° − ( z − IE )

Dip of horizon

If the earth's surface were an infinite plane, visible and sensible horizon would be identical. In reality, the visible horizon
appears several arcminutes below the sensible horizon which is the result of two contrary effects, the curvature of the
earth's surface and atmospheric refraction. The geometrical horizon, a flat cone, is formed by an infinite number of
straight lines tangent to the earth and radiating from the observer's eye. Since atmospheric refraction bends light rays
passing along the earth's surface toward the earth, all points on the geometric horizon appear to be elevated, and thus
form the visible horizon. If the earth had no atmosphere, the visible horizon would coincide with the geometrical
horizon (Fig. 2-2).
The altitude of the sensible horizon relative to the visible horizon is called dip and is a function of the height of eye,
HE, the vertical distance of the observer's eye from the earth's surface:

Dip ['] ≈ 1.76 ⋅ HE[m] ≈ 0.97 ⋅ HE [ft ]

The above formula is empirical and includes the effects of the curvature of the earth's surface and atmospheric
refraction*.

*At sea, the dip of horizon can be obtained directly by measuring the vertical angle between the visible horizon in front of the observer and the
visible horizon behind the observer (through the zenith). Subtracting 180° from the angle thus measured and dividing the resulting angle by two
yields the dip of horizon. This very accurate method is rarely used because it requires a special instrument (similar to a sextant).

2nd correction : H 2 = H 1 − Dip

The correction for dip has to be omitted (dip = 0) if any kind of an artificial horizon is used since an artificial
horizon indicates the sensible horizon.

The altitude obtained after applying corrections for index error and dip is also referred to as apparent altitude, Ha.

Atmospheric refraction

A light ray coming from a celestial body is slightly deflected toward the earth when passing obliquely through the
atmosphere. This phenomenon is called refraction, and occurs always when light enters matter of different density at an
angle smaller than 90°. Since the eye can not detect the curvature of the light ray, the body appears to be at the end of a
straight line tangent to the light ray at the observer's eye and thus appears to be higher in the sky. R is the angular
distance between apparent and true position of the body at the observer's eye (Fig. 2-3).

Refraction is a function of Ha (= H2). Atmospheric standard refraction, R0, is 0' at 90° altitude and increases
progressively to approx. 34' as the apparent altitude approaches 0°:

Ha [°] 0 1 2 5 10 20 30 40 50 60 70 80 90

R0 ['] ~34 ~24 ~18 9.9 5.3 2.6 1.7 1.2 0.8 0.6 0.4 0.2 0.0

R0 can be calculated with a number of formulas like, e. g., Smart's formula which gives highly accurate results from 15°
through 90° altitude [2, 9]:

R 0 ['] = 0.97127 ⋅ tan (90° − H 2 [°] ) − 0.00137 ⋅ tan 3 (90° − H 2 [°] )

For navigation, Smart's formula is still accurate enough at 10° altitude. Below 5°, the error increases progressively.
For altitudes between 0° and 15°, the following formula is recommended [10]. H2 is measured in degrees:

34.133 + 4.197 ⋅ H 2 + 0.00428 ⋅ H 22

R0 ['] =
1 + 0.505 ⋅ H 2 + 0.0845 ⋅ H 22

A low-precision refraction formula including the whole range of altitudes from 0° through 90° was developed by
Bennett:

R 0 ['] =
1
 
tan  H 2 [°] +
7.31

 H 2 [°] + 4.4 

The accuracy is sufficient for navigational purposes. The maximum systematic error, occurring at 12° altitude, is approx.
0.07' [2]. If necessary, Bennett's formula can be improved (max. error: 0.015') by the following correction:

R 0, improved ['] = R 0 ['] − 0.06 ⋅ sin (14.7 ⋅ R 0 ['] + 13)

The argument of the sine is stated in degrees [2].

Refraction is influenced by atmospheric pressure and air temperature. The standard refraction, R0, has to be multiplied
with a correction factor, f, to obtain the refraction for a given combination of pressure and temperature if high precision
is required.

p [m bar ] 283 p [in. Hg ] 510

f = ⋅ = ⋅
1010 273 + T [°C ] 29.83 460 + T [° F ]

P is the atmospheric pressure and T the air temperature. Standard conditions (f = 1) are 1010 mbar (29.83 in) and
10°C (50°F). The effects of air humidity are comparatively small and can be ignored.

Refraction formulas refer to a fictitious standard atmosphere with the most probable density gradient. The actual
refraction may differ from the calculated one if abnormal atmospheric conditions are present (temperature inversion,
mirage effects, etc.). Particularly at low altitudes, anomalies of the atmosphere gain influence. Therefore, refraction at
altitudes below ca. 5° may become erratic, and calculated values are not always reliable. It should be mentioned that dip,
too, is influenced by atmospheric refraction and may become unpredictable under certain meteorological conditions.

3rd correction : H3 = H 2 − f ⋅ R0

H3 is the altitude of the body with respect to the sensible horizon.

Parallax

Calculations of celestial navigation refer to the altitude with respect to the earth's center and the celestial horizon. Fig.
2-4 illustrates that the altitude of a near object, e.g., the moon, with respect to the celestial horizon, H4, is noticeably
greater than the altitude with respect to the geoidal (sensible) horizon, H3. The difference H4-H3 is called parallax in
altitude, PA. It decreases with growing distance between object and earth and is too small to be measured when
observing stars (compare with chapter 1, Fig. 1-4). Theoretically, the observed parallax refers to the sensible, not to the
geoidal horizon. Since the height of eye is several magnitudes smaller than the radius of the earth, the resulting error in
parallax is not significant (< 0.0003' for the moon at 30 m height of eye).
The parallax (in altitude) of a body being on the geoidal horizon is called horizontal parallax, HP. The HP of the sun
is approx. 0.15'. Current HP's of the moon (ca. 1°!) and the navigational planets are given in the Nautical Almanac
[12] and similar publications, e.g., [13]. PA is a function of altitude and HP of a body:

PA = arcsin (sin HP ⋅ cos H 3 ) ≈ HP ⋅ cos H 3

When we observe the upper or lower limb of a body (see below), we assume that the parallax of the limb equals the
parallax of the center (when at the same altitude). For geometric reasons (curvature of the surface), this is not quite
correct. However, even with the moon, the body with by far the greatest parallax, the resulting error is so small that it
can be ignored (<< 1'').
The above formula is rigorous for a spherical earth. However, the earth is not exactly a sphere but resembles an oblate
spheroid, a sphere flattened at the poles (chapter 9). This may cause a small but measurable error in the parallax of the
moon (≤ 0.2'), depending on the observer's position [12]. Therefore, a small correction, OB, should be added to PA if
high precision is required:

OB =
HP
298
[
⋅ sin (2 ⋅ Lat ) ⋅ cos Az N ⋅ sin H 3 − sin 2 Lat ⋅ cos H 3 ]
PA improved = PA + OB

Lat is the observer's assumed latitude (chapter 4). AzN , the azimuth of the moon, is either measured with a compass
(compass bearing) or calculated using the formulas given in chapter 4.

4th correction : H 4 = H 3 + PA

Semidiameter

When observing sun or moon with a marine sextant or theodolite, it is not possible to locate the center of the body with
sufficient accuracy. It is therefore common practice to measure the altitude of the upper or lower limb of the body and
add or subtract the apparent semidiameter, SD, the angular distance of the respective limb from the center (Fig. 2-5).
We correct for the geocentric SD, the SD measured by a fictitious observer at the center the earth, since H4 refers to the
celestial horizon and the center of the earth (see Fig. 2-4). The geocentric semidiameters of sun and moon are given on
the daily pages of the Nautical Almanac [12]. We can also calculate the geocentric SD of the moon from the tabulated
horizontal parallax:

SDgeocentric = arcsin ( k ⋅ sin HP) ≈ k ⋅ HP k Moon = 0.2725

The factor k is the ratio of the radius of the moon (1738 km) to the equatorial radius of the earth (6378 km).

Although the semidiameters of the navigational planets are not quite negligible (the SD of Venus can increase to 0.5'),
the centers of these bodies are customarily observed, and no correction for SD is applied. Semidiameters of stars are
much too small to be measured (SD=0).

5th correction : H 5 = H 4 ± SDgeocentric

(lower limb: +SD, upper limb: –SD)

When using a bubble sextant which is less accurate anyway, we observe the center of the body and skip the correction
for semidiameter.

The altitude obtained after applying the above corrections is called observed altitude, Ho.

Ho = H 5

Ho is the geocentric altitude of the body, the altitude with respect to the celestial horizon and the center of the
earth (see chapter 1).

Alternative corrections for semidiameter and parallax

The order of altitude corrections described above is in accordance with the Nautical Almanac. Alternatively, we can
correct for semidiameter before correcting for parallax. In this case, however, we have to calculate with the topocentric
semidiameter, the semidiameter of the respective body as seen from the observer's position on the surface of the earth
(see Fig. 2-5), instead of the geocentric semidiameter.

With the exception of the moon, the body nearest to the earth, there is no significant difference between topocentric and
geocentric SD. The topocentric SD of the moon is only marginally greater than the geocentric SD when the moon is on
the sensible horizon but increases measurably as the altitude increases because of the decreasing distance between
observer and moon. The distance is smallest (decreased by about the radius of the earth) when the moon is in the zenith.
As a result, the topocentric SD of the moon being in the zenith is approximately 0.3' greater than the geocentric SD. This
phenomenon is called augmentation (Fig. 2-6).
The accurate formula for the topocentric (augmented) semidiameter of the moon is stated as:

k
SD topocentric = arctan
− ( cos H 3 ± k ) 2 − sin H 3
1
2
sin HP
(lower limb: +k, upper limb: –k)

The following, simpler formula is accurate enough for navigational purposes (error << 1''):

cos H 3
SDtopocentric ≈ k ⋅ HP ⋅
cos ( H 3 + HP ⋅ cos H 3 )

Thus, the fourth correction is:

4th correction (alt.) : H 4, alt = H 3 ± SD topocentric

(lower limb: +SD, upper limb: –SD)

H4,alt is the topocentric altitude of the center of the moon.

Using one of the parallax formulas explained earlier, we calculate PAalt from H4,alt, and the fifth correction is:

5th correction (alt.) : H 5, alt = H 4, alt + PA alt

Ho = H 5, alt

Since the geocentric SD is easier to calculate than the topocentric SD, it is generally recommendable to correct for the
semidiameter in the last place unless one has to know the augmented SD of the moon for special reasons.

Combined corrections for semidiameter and parallax of the moon

For observations of the moon, there is a surprisingly simple formula including the corrections for augmented
semidiameter as well as parallax in altitude:

Ho = H 3 + arcsin [sin HP ⋅ ( cos H 3 ± k )]

(lower limb: +k, upper limb: –k)

The formula is rigorous for a spherical earth but does not take into account the effects of the flattening. Therefore, the
small correction OB should be added to Ho.

To complete the picture, it should be mentioned that there is also a formula to calculate the topocentric (augmented)
semidiameter of the moon from the geocentric altitude of the center, Ho:

k
SD topocentric = arcsin
1 sin Ho
1+ 2
− 2⋅
sin HP sin HP
Phase correction (Venus and Mars)
Since Venus and Mars show phases similar to the moon, their apparent center may differ somewhat from the actual
center. Since the coordinates of both planets tabulated in the Nautical Almanac [12] refer to the apparent center, an
additional correction is not required. The phase correction for Jupiter and Saturn is too small to be significant.

In contrast, coordinates calculated with Interactive Computer Ephemeris refer to the actual center. In this case, the
upper or lower limb of the respective planet should be observed if the magnification of the telescope permits it.

The Nautical Almanac provides sextant altitude correction tables for sun, planets, stars (pages A2 – A4), and the moon
(pages xxxiv – xxxv), which can be used instead of the above formulas if very high precision is not required (the tables
cause additional rounding errors).
Instruments with an artificial horizon can exhibit additional errors caused by acceleration forces acting on the bubble or
pendulum and preventing it from aligning itself with the direction of the gravitational force. Such acceleration forces can
be random (vessel movements) or systematic (coriolis force). The coriolis force is important to air navigation and
requires a special correction formula. In the vicinity of mountains, ore deposits, and other local irregularities of the
earth's crust, the gravitational force itself can be slightly deflected from its normal direction.