1

Numerical
Computation

Prepared by:
Richard Mitchell
Humber College
1.1 - THE NUMBER TYPES
1.1-PLACE VALUE
Whole Numbers
213 - Nearest Whole Number
51 700 - Nearest Hundreds
710 010 - Nearest Tens
52 - Nearest Whole Number
517 000 - Nearest Thousands

Decimal Numbers
0.105 - 3 DP’s/Nearest Thousandths
0.21 - 2 DP’s/Nearest Hundredths
11.051 - 3 DP’s/Nearest Thousandths
20.005 - 3 DP’s/Nearest Thousandths
7.00 - 2 DP’s/Nearest Hundredths
1.1-EXACT NUMBERS
Counted Quantities have no uncertainty.
4 wheels (exactly counted) 501 roses (exactly counted)
17 letters (exactly counted) 2 010 washers (exactly counted)
81 700 cars (exactly counted) 24 hours in a day (exactly counted)

Whole Numbers and Fractions have no uncertainty.

(Only when not measured and not in Decimal Form)

0 1 2 3 10 516 21 019 20 000

1 2
3/ 4
2 3

Defined Numbers have no uncertainty.

1 inch = 25.4 mm (exactly measured by definition)
1.1-APPROXIMATE NUMBERS
Measured Quantities have some degree of uncertainty.
(Last SD and/or Decimal Place is often estimated visually on a scale or meter).
217 m 1.75 cm 13.0 inches 1 000 m/s
50 112 Hz 2.02 lbs 0.00051 km 0.0020 mph

Decimal Numbers have some degree of uncertainty.

(Both measured and non-measured)
0.217 2 157.0 cm 3.00 12.05 ft

Estimates have some degree of uncertainty.

approximately 3 500 people about 2 710 cars were built

Decimal Form of Fractions and Irrational Numbers.

2/3 (exact form) equals 0.6667 (approximate decimal form)
∏ (exact form) equals 3.1428571 (approximate decimal form)
3 (exact form) equals 1.732 (approximate decimal form)
1.1-SIGNIFICANT DIGITS
Whole Numbers
217 - 3 SD’s 274 Ō00 - 4 SD’s
210 015 - 6 SD’s 31 Ō00 - 3 SD’s
1 000 201 - 7 SD’s 8 003 - 4 SD’s
24 900 - 3 SD’s 5 101 110 - 6 SD’s
310 400 - 4 SD’s 4 Ō00 - 2 SD’s

Decimal Numbers
0.152 - 3 SD’s 0.0177 - 3 SD’s
0.0005 - 1 SD 0.000583 - 3 SD’s
11.25 - 4 SD’s 2.008 - 4 SD’s
2.0500 - 5 SD’s 15.05 - 4 SD’s
1.1-SIGNIFICANT DIGITS SUMMARY
1.1-ACCURACY vs PRECISION
Accuracy is Determined by Significant Digits
1.255 - 4 SD’s of Accuracy
23 800 - 3 SD’s of Accuracy
0.0050 - 2 SD’s of Accuracy
125 - 3 SD’s of Accuracy
310.03 - 5 SD’s of Accuracy
0.002 - 1 SD of Accuracy

Precision is Determined by Place Value

1.255 - 3 DP’s of Precision (Nearest Thousandths)
23 800 - Nearest Hundred’s of Precision
0.0050 - 4 DP’s of Precision (Nearest Ten Thousandths)
125 - Nearest Whole Number of Precision
310.03 - 2 DP’s of Precision (Nearest Hundredths)
0.002 - 3 DP’s of Precision (Nearest Thousandths)
1.1-ROUNDING SUMMARY
Example 8

Example 9
1.2 - NUMERICAL
OPERATIONS
1.2-RULE of PRECISION
A d d t h e f o llo w i n g a p p r o x i m a t e n u m b e r s .
3 2 .4 c m + 5 .8 2 5 c m
1 Decimal Place 3 Decimal Places
(Approximate) (Approximate)

= 3 8 .2 2 5 c m USE: The Rule of PRECISION (1 Decimal Place)

When adding or subtracting approximate numbers,
keep as many decimal places in your answer as the
number having the fewest decimal places in the
= 3 8 .2 c m question. Use the least precise place value if there
are no decimal places in the question.

A N S : 3 8 .2 c m (1 D e c im a l P la c e )
1.2-EXAMPLE 13
A sports stadium contains about 3500 people. It starts to rain,
and 372 people (exactly) leave. How many are left in the stadium ?
(Subtract the following approximate and exact numbers).

3500 - 372
Nearest Hundreds Nearest Ones
(Approximate) (Exact)

USE: The Rule of PRECISION (Nearest Hundreds Place)

= 3 1 2 8 p e o p le When adding or subtracting approximate numbers, keep
as many decimal places in your answer as the number
having the fewest decimal places in the question. Use the
least precise place value if there are no decimal places in
= 3 1 0 0 p e o p le the question.

When using exact numbers, treat them as if

they had more Significant Digits than any of
the approximate numbers in the question.
A N S : 3 1 0 0 p e o p le (N e a re st H u n d re d s)
1.2-RULE of ACCURACY
M u ltip ly th e f o llo w in g a p p r o x im a te n u m b e r s .
1 2 3 .5 6 x 2 .2 1
5 Significant Digits 3 Significant Digits
(Approximate) (Approximate)

= 2 7 3 .0 6 7 6 USE: The Rule of ACCURACY (3 Significant Digits)

When multiplying or dividing approximate numbers,
keep as many significant digits in your answer as the
number having the fewest significant digits in the
question.
= 273

A N S: 273 (3 S ig n ific a n t D ig its )

1.2-RULE of ACCURACY
If a certain tire weighs 32.2 kg (approximate) when mounted,
how much will four (exactly counted) such tires weigh ?
(Multiply the following approximate and exact numbers).

3 2 .2 k g x 4
3 Significant Digits Do not count these as
(Approximate) Significant Digits.

= 1 2 8 .8 k g USE: The Rule of ACCURACY (3 Significant Digits)

When multiplying or dividing approximate numbers,
keep as many significant digits in your answer as the
number having the fewest significant digits in the
question.
= 129 kg
When using exact numbers, treat them as if
they had more Significant Digits than any of
the approximate numbers in the question.
A N S: 129 kg (3 S ig n ific a n t D ig its )
1.2-RULE of ACCURACY
D i v i d e t h e f o llo w i n g a p p r o x i m a t e n u m b e r s .
8 4 6 .2  4 .7 5
4 Significant Digits 3 Significant Digits
(Approximate) (Approximate)

= 1 7 8 .1 4 7 3 6 8 4 USE: The Rule of ACCURACY (3 Significant Digits)

When multiplying or dividing approximate numbers,
keep as many significant digits in your answer as the
number having the fewest significant digits in the
question.
= 178

A N S: 178 (3 S ig n ific a n t D ig its )

1.2-RULE of ACCURACY
D iv id e 8 4 6 .2 in to th ree eq u a l p arts.
(D iv id e th e fo llo w in g ap p ro x im ate an d ex act n u m b ers) .

8 4 6 .2  3
4 Significant Digits Do not count these as
(Approximate) Significant Digits.

= 2 8 2 .0 6 6 6 6 6 6 USE: The Rule of ACCURACY (3 Significant Digits)

When multiplying or dividing approximate numbers,
keep as many significant digits in your answer as the
number having the fewest significant digits in the
= 2 8 2 .1 question.

When using exact numbers, treat them as if

they had more Significant Digits than any of
the approximate numbers in the question.

A N S : 2 8 2 .1 (4 S ig n ific a n t D ig its )
1.2-POWERS
Powers
(1.45)5 ( approx .number )
 6.409 734 063
 6.41 ( 3 SD ' s )

(3.85) 3 ( approx .number )

 0.017 523 377
 0.0175 ( 3 SD ' s )

80.67 ( approx .number )

 4.027 822 200
 4.0 ( 2 SD ' s )

2
83 ( exact number )
 4 ( exact answer )
1.2-ROOTS
Roots
3
64 ( exact number )
4
Because 43  64

4
81 ( exact number )
3
Because 34  81

3
8 ( exact number )
 2
Because (-2)3  8

5
28.4 ( approx. number )
 1.952 826 537
 1.95 ( 3 SD ' s )
1.3 - ORDER OF OPERATIONS
1.3-EXAMPLE extra
Solve the following formula (exact numbers) and round off your
answer according to the Rules of Accuracy and/or Precision.

3
 118  423 
 
 136 
3
 541 
 
 11.66190379 

  46.39036728
3 USE: The Rule of EXACT NUMBERS.
Exact numbers produce ‘exact answers’
and do not need to be rounded off.

 99 835.14033 (exactly)
A n s : 9 9 8 3 5 .1 4 0 3 3 ( e x a c t ly )
1.3-EXAMPLE 37
Solve the following formula (approximate numbers) and round off
your answer according to the Rules of Accuracy and/or Precision.

 118.8 cm  4.23 cm
2 2 3
   (10.54973546 cm)3
 136 cm 2

 1 174.153 047 cm 3

 123.03 cm 2 3
   1 170 cm 3
(3 Significant Digits)
 136 cm  2

USE: The Rule of ACCURACY (3 Significant Digits).

Be sure to keep all of the digits used in each calculation
3 and only round off at the end.
 123.03 cm 2
 
 11.66190379 cm  A ns: 1 170 cm 3 ( 3 S .D .'s )
 1.4 - SCIENTIFIC AND
ENGINEERING NOTATION
1.4-EXAMPLES
Large Numbers
346 = 3.46 x 102 (3 SD’s)
2 700 = 2.7 x 103 (2 SD’s)
5 101 000 = 5.101 x 106 (4 SD’s)
31Ō 000 = 3.10 x 105 (3 SD’s)

Small Numbers
0.0000931 = 9.31 x 10-5 (3 SD’s)

0.008300 = 8.300 x 10-3 (4 SD’s)

0.00000950 = 9.50 x 10-6 (3 SD’s)

1.4-CALCULATOR SKILLS
Normal Mode
(30 000) x (215 000) = 6 450 000 000
(30 000) + (215 000) = 245 000 USE: The Rule of EXACT NUMBERS
(500 000) + (300 000) = when using exact numbers.

Exponential Mode (EXP or EE or key)

(3.00 x 104) x (2.15 x 105) = 6 450 000 000
(3.00 x 104) + (2.15 x 105) = 245 000

Scientific Mode (Mode or FSE key)

(3.00 x 104) x (2.15 x 105) = 6.45 x 109 USE: The Rule of ACCURACY
when using approximate
(3.00 x 104) + (2.15 x 105) = 2.45 x 105 numbers (Significant Digits).
1.4-EXAMPLES
C o n v e r t t h e f o llo w i n g n u m b e r s i n t o E n g i n e e r i n g N o t a t i o n .

2 1 8 4 0  2 1 .8 4 0 x 1 0 3 RULE: Digits from the decimal

part of the number are grouped
5 4 8 0 0 0  5 4 8 .0 0 0 x 1 0 3 into sets of three.

7 2 5 6 0 0 0 0  7 2 .5 6 0 0 0 0 x 1 0 6

RULE: For numbers less than 1,

3
0 .8 7 2 1 7 = 0 .8 7 2 1 7 = 8 7 2 .1 7 x 1 0 separate the digits following the
decimal point into groups of three.

0 .0 0 0 7 3 6 4 9 2  0 .0 0 0 7 3 6 4 9 2 = 7 3 6 .4 9 2 x 1 0  6
0 .0 0 0 0 0 0 0 4 7 2  0 .0 0 0 0 0 0 0 4 7 2  4 7 .2 x 1 0  9
1.4-SUMMARY
Scientific Notation

Engineering Notation
 1.5 - UNITS OF
MEASUREMENT
1.5-EXAMPLE 47 OPTIONAL FORMULA
Multiply by the conversion factor that will
cancel the units you wish to eliminate.
654.5 ft. x (0.3048 m / 1 ft.)
C o n v e rt 6 5 4 .5 fe e t (ft.) to m e te rs (m ).

ft m
=
ft m Conversion Factor: 1 ft. = 0.3048 m

654.5 ft. x m (654.5)  (0.3048)

= x
1 ft. 0.3048 m (1)

(1)•(x)=(654.5)•(0.3048) x  199.4916 m

A n s : 1 9 9 .5 m ( 4 S . D . 's )
1.5-EXAMPLE 48
C o n v e r t 1 3 4 a c r e s to h e c ta r e s (h a ).

acres ha
=
acres ha Conversion Factor: 1 ha = 2.471 acres

134 acres x ha (134)  (1)

= x
2.471 acres 1 ha (2.471)

(134)•(1)=(2.471)•(x) x  54.22905706 ha

A n s : 5 4 .2 h a ( 3 S .D .'s )
1.5-EXAMPLE 51
C o n v e rt 7 2 9 2 5 m e tre s (m ) t o k i lo m e t r e s (k m ).

metres km
=
metres km Conversion Factor: 1 km = 1000 m

72 925 metres x km (72 925)•(1)

= x
1 000 metres 1 km (1 000)

(72 925)•(1)=(1 000)•(x) x  72.925 km

A n s : 7 2 .9 2 5 k m ( 5 S .D .'s )
1.5-EXAMPLE 52
C o n v e r t 2 .7 5 x 1 0 5 d y n e s t o n e w t o n s ( N ).

dynes N
=
dynes N Conversion Factor: 1 N = 1.0 x 105 dynes

2.75 x 105 dynes xN (2.75 x 105 )•(1)

= x
5
1.0 x 10 dynes 1N (1.0 x 105 )

(2.75 x 105 )•(1)=(1.0 x 105 )•(x) x  2.75 N

A n s : 2 .7 5 N ( 3 S .D .'s )
1.5-EXAMPLE 53
C o n v e r t 2 .8 4 c u b i c f e e t ( c u .f t . ) t o U .S . g a llo n s ( g a l. ) .

cu.ft. gal.
=
cu.ft. gal. Conversion Factor: 1 cu.ft. = 7.481 gal.(U.S.)

2.84 cu.ft. x gal.(U.S.) (2.84)•(7.481)

= x
1 cu.ft. 7.481 gal.(U.S.) (1)

(2.84)•(7.481)=(1)•(x) x  21.24604

A n s : 2 1 .2 g a l. ( U .S .) ( 3 S .D .'s )
 1.6 - SUBSTITUTING INTO
EQUATIONS AND FORMULAS
1.6-EXAMPLE 57
S u b s t i t u t e t h e v a lu e s o f a = 5 , b = 3 a n d c = 6 i n t o t h e e q u a t i o n .

3(a )  (b)
x
(c )

3(5)  (3)

(6)

18

6

A ns: 3
1.6-EXAMPLE 58
A tensile load of 4500 lb. is applied to a bar that is 5. 2 yd. long and has
cross-sectional area of 1 1.6 cm 2 (Fig.1-19). The elongation is 0. 38 mm.
Using Eq.A54 , find the modulus of elasticit y E in pounds per square inch.

PL 842400 lb.
E E
ae 0.02689759873 sq.in.

4500 lb x 5.2 yd.

E Convert all units to
ns:31318780.85
AEpounds
3 1and0 0inches
0 0 0. 0 lblb./sq.in.
./sq .in . (2 S D 's)
11.6 cm2 x 0.38 mm

4500 lb. x 187.2 in.

E
1.797892126 in.2 x 0.01496062992 in.
1.7 - PERCENTAGE
1.7-FRACTIONS-DECIMALS-
PERCENTS
1.7-EXAMPLES
C o n v e r t in g F r a c t io n s t o D e c im a l s t o P e r c e n t s a n d b a c k a g a in .

1 75 3
 0.25  25% 75%  0.75  
4 100 4

1 75 3
1  1.25  125% 575%  5.75  5 5
4 100 4

3 5 1
27  27.6  2760% 0.05%  0.0005  
5 10000 2000

1 1 225 9
 0.04  4% 2 %  0.0225  
25 4 10000 400
1.7-STRATEGY A
Strategy for solving simple percent word problems.
1.7-STRATEGY B
Alternate strategy for solving simple percent word problems.

RATE (%) PART

(Change) FORMULA (Change)
RATE (%) PART (Change)

100% of BASE

BASE
(Original)
1.7-EXAMPLE 59
W hat (h o w m u c h ) is 3 5 .0 p e rc e n t o f 8 0 .0 ?

PART (Change)

RATE (%) PART (Change) (35.0)  (80.0)

 x
100% of BASE (100)

35.0 % x x  28

100% of 80.0

(100)  ( x)  (35.0)  (80.0)

A n s : x  2 8 .0 ( 3 S . D . 's )
1.7-EXAMPLE 60
F in d (h o w m u c h is ) 3 .7 4 % o f 5 7 1 0 .

PART (Change)

RATE (%) PART (Change) (3.74)  (5710)

 x
100% of BASE (100)

3.74 % x x  213.554

100% of 5710

(100)  ( x)  (3.74)  (5710)

A ns: x  214 ( 3 S . D . 's )
1.7-EXAMPLE 61
1 2 % o f w h a t n u m b e r is 7 8 ?

BASE (Original)

RATE (%) PART (Change) (100)  (78)

 x
100% of BASE (12)

12 % 78 x  650

100% of x

(12)  ( x)  (100)  (78)

A ns: x  650 ( 2 S . D . 's )
1.7-EXAMPLE 62
1 4 0 is 2 5 % o f w h a t n u m b e r ?

BASE (Original)

RATE (%) PART (Change) (100)  (140)

 x
100% of BASE (25)

25 % 140 x  560

100% of x

(25)  ( x)  (100)  (140)

A ns: x  560 ( 2 S . D . 's )
1.7-EXAMPLE 63
4 2 .0 is w h a t p e rc e n t o f 4 0 5 ?

RATE (%)

RATE (%) PART (Change) (100)  (42.0)

 x
100% of BASE (405)

x% 42.0 ______
 x  10. 370 %
100% of 405

( x)  (405)  (100)  (42.0)

A n s : x  1 0 .4 % ( 3 S . D . 's )
1.7-EXAMPLE 64
W h a t p e rc e n t o f 1 .4 5 is 0 .3 5 7 ?

RATE (%)

RATE (%) PART (Change) (100)  (0.357)

 x
100% of BASE (1.45)

x% 0.357 x  24.62068966 %

100% of 1.45

( x)  (1.45)  (100)  (0.357)

A n s : x  2 4 .6 % ( 3 S . D . 's )
1.7-STRATEGY
S tr a te g y f o r s o lv in g m o r e c o m p le x p e r c e n t w o r d p r o b le m s .
Type I: Percent Change
New Value-Original Value
Percent Change= x 100
Original Value

Type II: Percent Efficiency

Output
Percent Efficiency= x 100
Input

Type III: Percent Error

Measured Value-Correct Value
Percent Error= x 100
Correct Value

Type IV: Percent Concentration

Amount of Ingredient A
Percent Concentration= x 100
Total Amount of Mixture
1.7-EXAMPLE 65
The price of a chocolate bar rose from $1.55 to $1.75. Find the
percentage change in price.

Type I: Percent Change

New Value-Original Value
Percent Change= x 100
Original Value

($1.75-$1.55)
% Change= x 100 x  12.9032258%
($1.55)

($0.20)
% Change= x 100
($1.55)

% Change=0.129032258 x 100 A n s : x  1 2 .9 % ( 3 S . D . 's )

1.7-EXAMPLE 66
1
F in d th e co st o f a $ 1 5 6 .0 0 su it after th e p rice in creases b y 2 %.
2

Type I: Percent Change

 Percent Change 
New Value=(Original Value)+  •Original Value 
 100 

 2.5 
New Value=($156.00)+  •$156.00 
 100 
New Value=($156.00)+  0.025•$156.00 

New Value=($156.00)+  $3.90 

New Value=$159.90 A n s : x  $ 1 5 9 .9 0 (E x a c t)
1.7-EXAMPLE 67
A certain electric m o to r co n su m es 8 6 5 W an d h as an o u tp u t
o f 1 .1 2 h p . F in d th e efficien cy o f th e m o to r (1 h p  7 4 6 W ) .

Type II: Percent Efficiency

Output
Percent Efficiency= x 100
Input

 746 W 
836 W Output=1.12 hp• 
 1 hp 
 =836 W
Percent Efficiency= x 100
865 W
Percent Efficiency=0.966473988 x 100

Percent Efficiency=96.6473988 %
A n s : x  9 6 .6 % ( 3 S .D .'s )
1.7-EXAMPLE 68
A lab o rato ry w eig h t th at is certified to b e 5 0 0 .0 g is p laced o n
a scale. T h e scale read in g is 5 0 7 .0 g . W h at is th e p ercen t erro r ?

Type III: Percent Error

Measured Value-Correct Value
Percent Error= x 100
Correct Value

(507.0 g-500.0 g)
Percent Error= x 100 x=1.4 %
(500.0 g)

(7.0 g)
Percent Error= x 100
(500.0 g)

Percent Error=0.014 x 100 A n s : x  1 .4 0 0 % H i g h ( 4 S .D .'s )

1.7-EXAMPLE 69
A certain fu el m ix tu re co n tain s 1 8 .9 litres o f alco h o l an d 8 4 .7 litres
o f g aso lin e. F in d th e p ercen tag e o f g as o lin e in th e m ix tu re.

Type IV: Percent Concentration

Amount of Ingredient A
Percent Concentration= x 100
Total Amount of Mixture

84.7litres
Percent Concentration= x 100
(18.9litres+84.7litres)

84.7litres
Percent Concentration= x 100 x =81.75676 %
103.6litres

Percent Concentration=0.8175676 x 100

A n s : x  8 1 .8 % H i g h ( 3 S .D .'s )
 Copyright © 2016 John Wiley & Sons Canada, Ltd.

All rights reserved. Reproduction or translation of this work beyond that permitted by
Access Copyright (The Canadian Copyright Licensing Agency) is unlawful. Requests
for further information should be addressed to the Permissions Department, John
Wiley & Sons Canada, Ltd. The purchaser may make back-up copies for his or her
own use only and not for distribution or resale. The author and the publisher assume
no responsibility for errors, omissions, or damages caused by the use of these
programs or from the use of the information contained herein.