UC:  Plano  de  Dissertação/Plano  de  
Projecto/Plano  de  Estágio  
 
Tema  2:  Amostragem  e  Recolha  de  
Informação  
 
Theme  2:  Sampling  and  Data  Collec=on  
2016/2017  
   
Maria  Eduarda  Silva:  [email protected]  
Pedro  Campos:  [email protected]  
 
1  
 Overview  
•  Data  Collec=on:  Introduc=on  
•  Sampling  
–  Probabilis=c  
–  Non  probabilis=c  
•  Sample  survey/ques=onnaires  
–  Sources  of  informa=on  
–  How  to  make  a  ques=onnaire  
 
2  
 Research  Process  (Where  are  we  now?)  
 
Mo=va=on  
Formula=on  and  problem  clarifica=on  
Literature  Review  
Approach  and  strategy  
Data  Collec@on,  Sampling,  Surveys,  
Ques@onnaires  
Data  Analysis  (quan=ta=ve  and  Qualita=ve)  
Final  Report  
 
 

3  
 Research  Process  
 
•  PROBLEM  STATEMENT,  PURPOSES,  BENEFITS    
•  THEORY,  ASSUMPTIONS,  BACKGROUND  LITERATURE    
•  VARIABLES  AND  HYPOTHESES    
•     DATA  COLLECTION  
•     DATA  ANALYSIS    
•  FINAL  REPORT  and  CONCLUSIONS  
 

4  
 Inference  

Source:  HowMed  hap://howmed.net/community-­‐medicine/tests-­‐of-­‐sta=s=cal-­‐significance/  

5  
 
 Data  collec@on  (ques@ons)  
 
–  How  will  I  choose  the  sample?    
 
–  Am  I  interested  in  representa=veness?    
 
–  If  so,  of  whom  or  what,  and  with  what  degree  of  accuracy  or  level  of  
confidence?  

–  How  will  I  get  the  data  I  need  to  test  my  hypothesis?    
 
–  What  tools  or  devices  will  I  use  to  make  or  record  observa=ons?    
 

 
 

6  
 Data  Collec@on  (examples)  
•  Census  
•  Survey  
•  Observa=onal  study:  recording  well-­‐defined  
events    
•  Retrospec=ve  studies:  obtaining  relevant  data  
from  management  informa=on  systems.    
•  Prospec=ve  studies/Experiments/Clinical  trials  
           

7  
 Data  Collec@on  
Can  you  observe  (sample)  all  the  popula=on?  
Usually  not!  
 
–  Impossible  
 
–  Imprac=cal  
 
 
8  
 Data  collec@on    
•  Observe  the  whole  popula=on:  
 
       CENSUS  
 
Too many Resources (time, money) and too
much workload!  
Na=onal  Census  every  10  years  (in  many  
countries)!  

9  
Portuguese Census
•  22  000  people  
involved  
•  1200  training  
courses  
•  17000  training  
hours  
•  More  than  10  
million  People  
•  3  500  000  families  
•  4  500  000  
households    
•  2  900  000  buildings  
 Sampling  
•  A sample is “a smaller (but hopefully
representative) collection of units from a
population used to determine truths about that
population” (Field, 2005)

•  Why to sample?
–  Resources (time, money) and workload
–  Gives results with known accuracy that can be
calculated mathematically

11  
 Sampling  
Steps  of  sampling  design    
p   Define  the  target  popula=on  
p   Iden=fy  the  sampling  frame  
p   Define  sampling  unit  
p   Choose  a  sampling  technique  
p   Define  sample  dimension  
p   Select  sample  elements  
p   Collect  data  
p Review  sampling  process  

12  
Sampling:  Define  the  target  popula=on  

–  What  is  the  target  popula=on?  

–  Who  do  you  want  to  generalize  to?  
– All doctors
– School children
– Pet owners
– Women aged 15-45 years
– Other

13  
 Sampling:  Iden=fy  the  sampling  frame  

•  Subset  of  popula=on  you  get  access  to  

•  A  list  of  all  those  within  a  popula=on  who  can  
be  sampled,  and  may  include  individuals,  
households  or  ins=tu=ons.  
•  Examples:  
–  electoral  register  or  a  telephone  directory  for  an  
opinion  poll  
–  a  street  map  for  a  door-­‐to-­‐door  survey;  

14  
 Sampling  Frame  
•  Ideally,  the  sampling  frame  perfectly  coincides  
with  the  target  popula=on.    
•   Some=mes  the  frame  will  be  larger  or  
smaller,  depending  on  prac=cal  ways  of  
gelng  in  touch  with  each  member  of  the  
sample.  
•  The  sampling  frame  must  be  representa=ve  of  
the  popula=on  
•  Selec=on/coverage  bias  
15  
 Sampling  
  Overlape  
Target   between  the  
Popula=on   two  pops  

Sampling  
Frame  

Sample  

16  
 Sampling  
•  Sampling  unit-­‐  the  elements  of  the  sampling  
frame  that  are  chosen  for  the  sample  using  a  
sampling  procedure.  Examples:    
–  Persons  
–  Households  
–  Companies  
•  Sample:  set  of  sampling  units  –  you  may  or  
not  observe  all  of  them  

17  
 Sampling  techniques  

•  Probability  Sampling  
Scheme  in  which  every  unit  in  the  popula=on  has  a  
known  probability  of  being  selected.  
When  every  element  of  the  popula=on  does  have  
the  same  probability  of  being  selected  is  known  as  
a  equal  probability  selec0on,  EPS  design  
•  Non  Probability  Sampling  
Scheme  in  which  some  elements  have  no  chance  of  
selec=on  

18  
 Sampling  techniques  

Probability  /  Random  Sampling  techniques  

•  Simple  random  sampling  
•  Sistema=c  sampling  
•  Stra=fied  Sampling  
•  Cluster  sampling  
•  Mul=stage  sampling  

19  
 Simple  Random  Sampling  
In  a  simple  random  sample,  every  group  of  
individuals  in  the  popula=on  has  an  equal  
chance  of  becoming  the  sample.  A  simple  
random  sample  is  the  equivalent  of  pulng  
everyone's  name  from  the  popula=on  into  a  hat  
and  drawing  your  sample  from  that.  

20  
 Simple  Random  Sampling  
•  Applicable  when  popula=on  is  small,  homogeneous  &  readily  
available  
•  All  subsets  of  the  frame  are  given  an  equal  probability.  Each  
element  of  the  frame  thus  has  an  equal  probability  of  
selec=on.  
•  It  provides  for  greatest  number  of  possible  samples.  This  is  
done  by  assigning  a  number  to  each  unit  in  the  sampling  
frame.  
•  A  table  of  random  number  or  loaery  system  is  used  to  
determine  which  units  are  to  be  selected  

21  
 Simple  Random  Sampling  
•  Es=mates  are  easy  to  calculate.  
 
Disadvantages    
•  If  sampling  frame  large,  this  method  imprac=cable.  
•  Minority  subgroups  of  interest  in  popula=on  may  not  
be  present  in  sample  in  sufficient  numbers  for  study.  

22  
 Systema=c  Sampling  
•  Arrange  the  target  popula=on  according  to  
some  ordering  scheme  and  then  selec=ng  
elements  at  regular  intervals  through  that  
ordered  list.    
 
•  Random  start  followed  with  the  selec=on  of  
every  kth  element,  k=(popula=on  size/sample  
size).    
 

23  
 Systema=c  Sampling  
•  Star=ng  randomly  chosen  from  within  the  first  to  the  
kth  element  in  the  list.    
•  Example:  select  every  10th  name  from  the  telephone  
directory.    
•  It  is  not  'simple  random  sampling'  because  different  
subsets  of  the  same  size  have  different  selec=on  
probabili=es  -­‐  e.g.  the  set  {4,6,8,…,998}  has  a  one-­‐in-­‐
two  probability  of  selec=on,  but  the  set  {4,5,7,…998}  
has  zero  probability  of  selec=on.  

24  
 Systema=c  Sampling  
•  ADVANTAGES:  
–  Sample  easy  to  select  in  lists  and  datasets  
–  Suitable  sampling  frame  can  be  iden=fied  easily  
–  Sample  evenly  spread  over  en=re  reference  popula=on  
 
•  DISADVANTAGES:  
–  Sample  may  be  biased  if  hidden  periodicity  in  popula=on  
coincides  with  that  of  selec=on.  
–  Difficult  to  assess  precision  of  es=mate  from  one  survey.  

25  
 Stra=fied  Sampling  
•  par==on  the  popula=on  into  groups  (strata)  
•  obtain  a  simple  random  sample  from  each  
group  (stratum)  
•  collect  data  on  each  sampling  unit  that  was  
randomly  sampled  from  each  group  (stratum)    
•  Stra@fied  sampling  works  best  when  a  
heterogeneous  popula=on  is  split  into  fairly  
homogeneous  groups.        

26  
Draw  a  sample  from  each  stratum  

27  
 Cluster  Sampling  
•  divide  the  popula=on  into  groups  (clusters)  
(geographically)  
•  obtain  a  simple  random  sample  from  all  possible  
clusters.  
•  obtain  data  on  every  sampling  unit  in  each  of  the  
randomly  selected  clusters.  
 
NOTE:  
 
•  Clusters  are  microcosms,  rather  than  subsec=ons,  of  
the  popula=on.  

28  
 Mul=stage  Sampling  
The   above   methods   are   combined   in   different  
stages.  For  example,  the  Labour  Force  Survey  is  
a   mix   of   Simple   and   Stra=fied   Sampling   in  
different  stages.  
 
Stage  1:  stra=fied  sample  (Regions  NUTS  2)  
Stage   2:   simple   random   sample   (Households   in  
small  territorial  units)  
 

29  
 Sampling  techniques  

Non-­‐Random  Sampling  techniques  

•  Convenience  sampling  
•  Snowball  Sampling  
•  Quota  Sampling  
 
These  methods  do  not  allow  the  computa=on  of  
sampling  error  
 
30  
 Convenience  Sampling  
Data  collec=on  from  popula=on  members  who  are  conveniently    
available  to  par=cipate  in  study.  Facebook  polls  or  ques=ons  
 

31  
 Convenience  Sampling  
•  May  prove  to  be  effec=ve  during    
•  During  explora=on  stage  of  the  research  area  
•  When  conduc=ng  pilot  data  collec=on  in  order  to  
iden=fy  and  address  shortcomings  associated  with  
ques=onnaire  design.  

32  
 Snowball  Sampling  
•  Snowball  sampling  is  where  research  par=cipants  
recruit  other  par=cipants  for  a  test  or  study.  It  is  used  
where  poten=al  par=cipants  are  hard  to  find.    

•  Iden=fy  poten=al  subjects  in  the  popula=on.  Owen,  

only  one  or  two  subjects  can  be  found  ini=ally.  
•  Ask  those  subjects  to  recruit  other  people  and  then  ask  
those  people  to  recruit.  Par=cipants  should  be  made  
aware  that  they  do  not  have  to  provide  any  other  
names.  

33  
 Snowball  Sampling  
•  Advantages:  
–  It  allows  for  studies  to  take  place  where  otherwise  it  
might  be  impossible  to  conduct  because  of  a  lack  of  
par=cipants.  
–  Snowball  sampling  may  help  you  discover  
characteris=cs  about  a  popula=on  that  you  weren’t  
aware  existed.  For  example,  the  casual  illegal  
downloader  vs.  the  for-­‐profit  downloader.  
•  Disadvantages:  
–  It  us  usually  impossible  to  determine  the  sampling  
error  or  make  inferences  about  popula=ons  based  on  
the  obtained  sample.  

34  
 Quota  Sampling  
•  A  sampling  method  of  gathering  representa=ve  data  from  a  
group-­‐  ensures  that  sample  group  represents  certain  
characteris=cs  of  the  popula=on  chosen  by  the  researcher.  
•  Quota  sampling:    
–  controlled  -­‐  restric=ons  in  order  to  limit  researcher’s  choice  of  
samples  
–  uncontrolled-­‐  resembles  convenience  sampling    

•  Difference  to  stra=fied  sampling:      “researchers  use  non-­‐

random  sampling  methods  to  gather  data  from  one  
stratum  un=l  the  required  quota  fixed  by  the  researcher  is  
fulfilled”  

35  
 Prac=ce  1  (Sampling)  
Find  the  alterna=ve  solu=ons  to  select  data  in  the  
following  situa=ons  (for  each  one  define  target  
popula0on,  sampling  frame,  sampling  unit  and  
sampling  method):  
1.  The  Sunday  Times  is  trying  to  define  the  
archetypal  reader  of  the  Monday  supplement  
(about  music  and  culture);  
2.  The  WRS.com  company  (food  retail  business)  
needs  to  perform  a  market  segmenta=on  of  
their  clients,  based  on  their  loyalty  card.  
36  
 Determinants  for  sample  size  

•  Variability  of  the  characteris=c  being  studied  

•  Precision  and  Confidence  
•  Sampling  distribu=on  of  the  es=mator  
•  Cost  
•  Others  
l e  
ex amp
  an  
s   s ee
Let ’
37  
Sources  of  Informa@on  
 
Everything  starts  with  
the  data!  
 

38  
 Sources  of  Informa@on  
 
Sources  of  
informa=on  

Register-­‐based   Survey-­‐based  

Content   Direct  observa=on   Undirect  

Classic   observa=on  
analysis   (Par=cipant/non  
par=cipant)   (SURVEY)    

39  
 Data  collec@on  
Sources  of  Informa@on  
 
Sources  of  
informa=on  

Secondary  Data     Primary  Data    

Register-­‐based   Survey-­‐based  

Content   Direct  observa=on   Undirect  

Classic   observa=on  
analysis   (Par=cipant/non  
par=cipant)   (SURVEY)    

40  
 What  is  a  ques@onnaire?  

•  Ques=onnaires  are  the  most  frequently  used  

data  collec=on  method  in  several  different  
fields  
•  Ques=onnaires  help  gather  informa=on  on  
knowledge,  altudes,  opinions,  behaviors,  
facts,  and  other  useful  informa=on.  

41  
 How  to  make  a  ques@onnaire?  

•  To  build  a  ques=onnaire  you  are  required  to:  

–  know  exactly  what  is  being  looked  for;  
–  ensure  that  the  ques=ons  have  the  same  
interpreta=on  in  all  respondents;  
–  that  all  aspects  of  the  issues  have  been  well  
addressed,  etc.  

42  
 How  to  make  a  ques@onnaire?  Rules  

•  The  first  ques=ons  of  a  ques=onnaire  are  very  

important.  They  are  indica=ng  people  surveyed  
the  overall  style  of  the  ques=onnaire,  the  
response  type  that  is  expected  of  them  and  the  
topic  that  will  be  addressed.  
•  It  is  also  from  these  first  ques=ons  the  
interviewer-­‐respondent  rela=onship  is  defined,  
specially  if  one  feels  that  his/her  privacy  is  being  
disturbed.  

43  
 How  to  make  a  ques@onnaire?  Rules  

•  Avoid  using  direct  statements  

•  Avoid  responses  that  require  memory  
•  Avoid  ques=ons  that  require  file  queries  

44  
 How  to  make  a  ques@onnaire?  Rules  

•  Whenever  possible  introduce  filters  or  

ques=ons  that  prepare  responses  (serve  to  
filter  the  people  for  whom  certain  ques=ons  
make  no  sense  or  are  not  applicable).  
•  Separate  the  interviewer's  comments  from  
the  respondent's  answers  
•  Allow  some  missing  values  (do  not  know,  no  
answer,  not  applicable)  

45  
 How  to  make  a  ques@onnaire?  Rules  

•  It  is  usually  preferable  to  begin  with  issues  

that  encourage  interest  in  the  respondent  and  
not  the  scare.  
•  The  ques=ons  of  a  ques=onnaire  can  be  
closed,  open  and  semi-­‐open  

46  
 How  to  make  a  ques@onnaire?  Rules  

•  One  should  not  use  double  issues,  that  is,  we  

must  not  introduce  more  than  one  idea  in  
each  ques=on  
•  A  ques=onnaire  must  not  only  contain  open  
ques=ons  or  only  closed  ques=ons.  One  shall  
combine  the  two  types  of  ques=ons  to  not  
make  a  boring  ques=onnaire.  

47  
 How  to  make  a  ques@onnaire?  Rules  

•  Before  we  elaborate  some  issues  that  may  

cause  the  embarrassment  of  the  respondent,  
such  as,  for  example,  ques=ons  about  religion,  
consump=on  of  certain  products,  etc.  
–  One  shall  promote  a  short  introduc=on  to  the  
respondent,  as  many  people  may  be  afraid  to  give  
wrong  answers  or  confess  their  ignorance.  
–  Therefore,  a  rule  is  to  address  these  issues  as  
follows:  "...  in  your  personal  case  could  tell  me  ...";  
"I  would  like  to  know  your  opinion  ...".  

48  
 Pretes@ng  a  Ques@onnaire  

•  It  is  to  test  the  ques=onnaire  before  it  is  used  

in  the  end,  applying  some  copies  (pilot  tests)  
to  a  part  of  the  popula=on.  
•  With  pilot  tests  we  can  detect:  
–  redundant  issues  
–  badly  formulated  ques=ons  
–  Unanswered  Ques=ons  

49  
 Pretes@ng  a  Ques@onnaire  

•  In  addi=on,  you  can  take  care  /  improve  the  

following  aspects:  
–  The  order  of  ques=ons  
–  The  language,  spoken  or  wriaen,  which  is  used  
–  The  reac=on  of  the  ques=ons  asked  
–  The  size  and  presenta=on  of  the  ques=onnaire  
–  The  pretest  should  be  applied  to  the  individuals  
whose  typological  characteriza=on  is  close  to  the  
target  audience  
50  
 Assessing  a  Ques@onnaire  

•  Development  of  a  valid  and  reliable  

ques=onnaire  involves  several  steps  taking  
considerable  =me.    
–  Background  
–  Conceptualiza=on  
–  Format  (and  data  analysis)  
–  Validity  
–  Reliability  

51  
Assessing  a  Ques@onnaire  

•  Source:  Tips  for  Developing  and  Tes@ng  Ques@onnaires/Instruments  

•  Rama  B.  Radhakrishna,  available  at:  
•  haps://joe.org/joe/2007february/a2.php  
52  
 Assessing  a  Ques@onnaire  

•  Step  1-­‐Background  
Purpose,  objec=ves,  research  ques=ons,  and  
hypothesis  of  the  proposed  research  are  
examined.    
Define  Target  Popula=on  and  sampling  frame  

53  
 Assessing  a  Ques@onnaire  

•  Step  2-­‐Ques@onnaire  Conceptualiza@on  

Generate  variables,  statements,  ques=ons.  
Major  variables  (independent,  dependent,  and  
moderator  variables)  are  iden=fied  and  defined  
in  this  step.  

54  
 Assessing  a  Ques@onnaire  

•  Step  3-­‐  Format  and  Data  Analysis  

Define  appropriate  scales  of  measurement,  
layout,  ques=on  ordering.  
Preview  data  analysis.    
Understanding  the  rela=onship  between  the  
level  of  measurement  and  the  appropriateness  
of  data  analysis  is  important.    

55  
 Assessing  a  Ques@onnaire  

•  Step  4-­‐  Validity  

Validity  is  the  amount  of  systema=c  or  built-­‐in  error  
in  measurement.  In  other  words,  is  the  
ques=onnaire  measuring  what  it  intended  to  
measure?  
–  Is  the  ques=onnaire  comprehensive  enough  to  
collect  all  the  informa=on  needed  to  address  the  
purpose  and  goals  of  the  study?  
–  Addressing  these  ques=ons  coupled  with  carrying  out  
a  readability  test  enhances  ques=onnaire  validity.    

56  
 Assessing  a  Ques@onnaire  

•  Step  5-­‐  Reliability  

Reliability  refers  to  random  error  in  measurement.  
Reliability  indicates  the  accuracy  or  precision  of  the  
measuring  instrument  Norland-­‐Tilburg  (1990)  
Use  a  pilot  test  to  measure  reliability  (measure  
internal  consistency  through  the  Cronbach  alpha).    
Reliability  types  (test-­‐retest,  split  half,  alternate  form,  internal  
consistency)  depends  on  the  nature  of  data  (nominal,  ordinal,  
interval/ra=o).  may  be  appropriate  to  use.    

57  
Reliability  and  Validity  

Source:  hap://www.socialresearchmethods.net/kb/
relandval.php  
58  
 Measurement  Scales  

Retrieved  from:  hap://www.socialresearchmethods.net/kb/measlevl.php  

59  
 Measurement  Scales  
most  commonly  used  

Retrieved  from:  hap://www.socialresearchmethods.net/kb/measlevl.php  

60  
 Measurement  Scales  

•  Self  Ra@ng  Scales  

–  Graphic  Ra=ng  Scale  
–  Itemized  Ra=ng  Scales  
•  Likert  Scale  
•  Seman=c  Differen=al  Scale  
•  Stapel’s  Scale  
•  Mul=  Dimensional  Scaling  
•  Thurston  Scales  
•  Guaman  Scales/Scalogram  Analysis  
•  The  Q  Sort  technique  
Retrieved  from:  hap://www.managementstudyguide.com/altude-­‐scales.htm  
61  
 Measurement  Scales  

•  Graphic  Ra@ng  Scale  

The  respondents  rate  the  objects  by  placing  a  
mark  at  the  appropriate  posi=on  on  a  line  that  
runs  from  one  extreme  of  the  criterion  variable  
to  another.  Example  
BRAND 1
This is also known as continuous rating scale. The customer can occupy any position. Here one attribute is taken ex-quality of any brand of icecream.

5  
0   1   7  
(neither  good  nor  
(poor  quality)   (bad  quality)   (good  quality)  
bad)  

62  
 Measurement  Scales  

•  Itemized  Ra@ng  Scales  

Different  from  con=nuous  ra=ng  scales,  these  
scales  have  a  number  of  short  explana=ons  
associated  with  each  category.  They  are  widely  
used  in  Marke=ng  Research.    
The  most  common  are  -­‐  Likert,  Seman=c,  Staple  
and  Mul=ple  Dimension.  Others  are  -­‐  Thurston  
and  Guaman.category  ques=ons.    

63  
 Measurement  Scales  

•  Likert  Scale  
Each statement is assigned a numerical score ranging from 1 to
5. It can also be scaled as -2 to +2.

-­‐2   -­‐1   0   1   2  

1  
5   4   3   2  
Strongly  
Strongly  agree   Agree   Indifferent   Disagree  
disagree  

64  
 Measurement  Scales  

•  Seman@c  Differen@al  Scale  

Ex.  a  seven  point  scale  where  the  end  points  of  
the  scale  are  associated  with  bipolar  (antonym)  
labels.  
Suppose we want to know personality of a particular person. How would you describe him/her?

1   7  
Unpleasant   2   3   4   5   6   Pleasant  
Submissive   Dominant  

65  
 Prac@ce  2  (ques@onnaires)  

Buying  in  pharmacies  and  parapharmacies.  Iden@fy  the  

errors  /  reorder  the  ques@ons/  suggest  changes  in  the  
following  ques@onnaire:  
1)  What  is  your  age  and  salary?  Age___  Salary  ___  
2)  What  kind  of  products  do  you  usually  buy  in  paraphamarcies?  
__________________________  
3)  What  kind  of  products  do  you  usually  buy  in  phamarcies  
                 __________________________?  
4)  Do  you  usually  buy  in  paraphamarcies?    Yes__    No__  
5)          What  are  the  advtanges  of  Parapharmacies?  

66  
 New  Ways  of  collec@ng  data  

The  “tradi@onals”  
PAPI  (Paper  Assisted  Personal  Interview)  
CAPI  (Computer  Assisted  Personal  Interview)  
CATI  (Computer  Assisted  Telephone  Interviewing)    
 
The  new  ones  
CAWI  (Computer  Assisted  Web  Interviewing)    
Social  Media  
Website  Ac@vity  
Tracking  customers  in  stores  
 
67  
 New  Ways  of  collec@ng  data  

…more  on  the  new  ones  

Mobile  Surveys  
Online  Communi@es  
Eye  tracking  
Research  Gamifica@on  
Crowdsoursing  
Wearables  based  research  
…  
 

Retrieved from :
http://www.greenbookblog.org/2014/09/29/the-top-20-
emerging-methods-in-market-research-a-grit-sneak-peek/
68  
 References  
 

•  Andrew  Hill,  Maria  Manuela  Magalhães  Hill  (2008),  Inves=gação  por  Ques=onário  ,  
2ª  Ed,  Edições  Sílabo.  
•  Elizabeth  Reis,  Raúl  Moreira  (1993),  Pesquisa  de  Mercados,  Sílabo  Gestão,  1.ª  
Edição.  
•  Louis  Cohen,  Lawrence  Manion  and  Keith  Morrison,  (2005),  Research  Methods  
in  Educa=on  ,  Fiwh  edi=on  
•   Norland-­‐Tilburg,  E.  V.  (1990).  Controlling  error  in  evalua=on  instruments.  Journal  
of  Extension,  [On-­‐line],  28(2).  Available  at  
h>p://www.joe.org/joe/1990summer/>2.html  
•  William  M.K.  Trochim,,  (2006)  .  The  Research  Methods  Knowledge  Base,  2nd  
Edi=on.  Internet  WWW  page,  at  URL:  <
hap://www.socialresearchmethods.net/kb/>  (version  current  as  of  October  20,  
2006).    
 

69