BRM Unit 2

Download as pptx, pdf, or txt
Download as pptx, pdf, or txt
You are on page 1of 103

Research Problem Identification & Research design Determination

Formulation and Definition of Business Research Problem, Formulation of Research Hypothesis, Business Research Design,

Problem Definition
The process of defining and developing a decision statement and the steps involved in translating it into more precise research terminology, including a set of research objectives.
A decision statement is a written expression of the key question(s) that a research user wishes to answer. It is the reason that research is being considered.

Problem Complexity
Easier: 1. Situation is recurring/routine 2. A dramatic change occurs 3. Symptoms are isolated 4. Symptoms are consistent Harder: 1. Situation appears new 2. Change(s) in situation is subtle 3. Symptoms are scattered 4. Symptoms are ambiguous
4

Situation Frequency
Many business situations are cyclical. Cyclical business situations lead to recurring business problems. These problems can even become routine. In these cases, it is easy to define problems and identify the types of research that are needed. In some cases, problems are so routine that they can be solved without any additional research.
5

Dramatic Changes
When a sudden change in the business situation takes place, it can be easier to define the problem.

In contrast, when changes are very subtle and take effect over a long period of time, it can be more difficult to define the actual decision and research problems.

Widespread
The more scattered any symptoms are, the more difficult it is to put them together into some coherent problem statement. In contrast, firms may sometimes face situations in which multiple symptoms exist, but they are all pointing to some specific business area. In contrast, when the problems are more widespread, it can be very difficult to develop useful research questions.
7

Widespread
For instance, an automobile manufacturing company may exhibit symptoms such as increased complaints about a cars handling, increased warranty costs due to repairs, higher labor costs due to inefficiency, and lower performance ratings by consumer advocates such as Consumer Reports. All of these symptoms point to production as a likely problem area. This may lead to research questions that deal with supplier-manufacturer relationships, job performance, job satisfaction, supervisory support, and performance.
8

Symptom Ambiguity
an environmental scan of a business situation may lead to many symptoms, none of which seem to point in a clear and logical direction. In this case, the problem area remains vague and the alternative directions are difficult to ascertain.
A retail store may face a situation in which sales and traffic are up, but margins are down. They may have decreased employee turnover, but lower job satisfaction. In addition, there may be several issues that arise with their suppliers, 9 none of which is clearly positive or negative.

The ProblemDefinition Process

10

Problem means GAPS


A problem occurs when there is a difference between the current conditions and a more preferable set of conditions. In other words, a gap exists between the way things are now and a way that things could be better. The gap can come about in a number of ways:
1)- Business performance is worse than expected business performance. For instance, sales, profits, & margins could be below targets set by management.

11

Problem means GAPS


2. Actual business performance is less than possible business performance. Realization of this gap first requires that management have some idea of what is possible. This may form a research problem in and of itself. 3. Expected business performance is greater than possible business performance. Sometimes, management has unrealistic views of possible performance levelseither too high or too low. One key problem with new product introductions involves identifying realistic possibilities for sales.
12

The Problem-Definition Process Steps


1. Understand the business situationidentify key symptoms 2. Identify key problem(s) from symptoms 3. Write managerial decision statement and corresponding research objectives 4. Determine the unit of analysis 5. Determine the relevant variables 6. Write research questions and/or research hypotheses
13

Understand the Business Decision


A situation analysis involves the gathering of background information to familiarize researchers and managers with the decisionmaking environment.

The situation analysis begins with an interview between the researcher and management.
14

Understand the Business Decision


Interview Process
This process is critical and the researcher should be granted access to all individuals who have specific knowledge of or insight into this situation. Researchers may often be tempted to accept the first plausible problem statement offered by management.
There are many ways to discover problems and spot opportunities.
15

Understand the Business Decision


Interview Process
1. Develop many alternative problem statements 2. Think about potential solutions to the problem. 3. Use interrogative techniques to generate lists of potential questions. (Asking multiple what, where, who, when, why, and how questions.) 4. Be open-minded. Analogies can be useful 5. in thinking more creatively.
16

Identifying Symptoms
Interviews with key decision makers also can be one of the best ways to identify key problem symptoms. Once symptoms are identified, then the researcher must probe to identify possible causes of these changes. Probing is an interview technique that tries to draw deeper and more elaborate explanations from the discussion.
17

Identifying Symptoms
Identifying Symptoms

18

Identifying Symptoms
Identifying the Relevant Issues from the Symptoms
For instance, when a firm has a problem with advertising effectiveness, the possible causes of this problem may be low brand awareness, the wrong brand image, use of the wrong media, or perhaps too small a budget.

19

Writing Managerial Decision Statements and Corresponding Research Objectives


The situation analysis ends once researchers have a clear idea of the managerial objectives from the research effort. Decision statements capture these objectives in a way that invites multiple solutions. Multiple solutions are encouraged by using plural nouns to describe solutions.

20

Writing Managerial Decision Statements and Corresponding Research Objectives


In other words, a decision statement that says in what ways a problem can be solved is better than one that says in what way a problem can be solved.

21

Writing Managerial Decision Statements and Corresponding Research Objectives


The researcher should reach a consensus agreement with the decision maker regarding the overall decision statement(s) and research objectives. If the decision maker agrees that the statement captures the situation well and understands how the research objectives, if accomplished, will help address the situation, then the researcher can proceed

22

Determine the Unit of Analysis


The unit of analysis for a study indicates what or who should provide the data and at what level of aggregation. Researchers specify whether an investigation will collect data about individuals (such as customers, employees, and owners), households (families, extended families, and so forth), organizations (businesses and business units), departments (sales, finance, and so forth), geographical areas, or objects (products, advertisements, and so forth).
23

Determine Relevant Variables


A variable is anything that varies or changes from one instance to another. Variables can exhibit differences in value, usually in magnitude or strength, or in direction. In research, a variable is either observed or manipulated, in which case it is an experimental variable.

What things should be studied to address a decision statement? Researchers answer this question by identifying key variables.
24

Determine Relevant Variables


Types Of Variables
A continuous variable is one that can take on a range of values that correspond to some quantitative amount. Consumer attitude toward different airlines is a variable that would generally be captured by numbers, with higher numbers indicating a more positive attitude than lower numbers.
More examples are Sales volume, profits, and margin
25

Determine Relevant Variables


Types Of Variables
A categorical variable is one that indicates membership in some group. The term classificatory variable is sometimes also used and is generally interchangeable with categorical variable. Categorical variables sometimes represent quantities that take on only a small number of values (one, two, or three).
For example, people can be categorized as either male or female. A variable representing biological sex describes this important difference.
26

Determine Relevant Variables


Types Of Variables
A dependent variable is a process outcome or a variable that is predicted and/or explained by other variables.
An independent variable is a variable that is expected to influence the dependent variable in some way. Such variables are independent in the sense that they are determined outside of the process being studied.
Average customer loyalty may be a dependent variable that is influenced by an independent variable such as perceptions of restaurant food quality, service quality & customer satisfaction.
27

Research Questions & Hypotheses


Is advertising copy 1 better than advertising copy 2?Too Vague And General Which advertisement has a higher day after recall score?- A More Specific Research Question
Management is often faced with a go/no go decision. In this case, the research involves a managerial action standard that specifies a specific performance criterion for decision making. If the criterion to be measured (sales or attitude changes) turns out to be higher than some predetermined level, management will do A; if it is lower, management will do B.
28

The Research Proposal


The research proposal is a written statement of the research design. It always includes a statement explaining the purpose of the study (in the form of research objectives or deliverables) and a definition of the problem, often in the form of a decision statement. A good proposal systematically outlines the particular research methodology, a schedule of costs and deadlines and details procedures that will be used during each stage of the research process, The research proposal becomes the primary communication document between the researcher and the research user. 29

Exploratory Research Design

30

Qualitative Research
Qualitative business research is research that addresses business objectives through techniques that allow the researcher to provide elaborate interpretations of market phenomena without depending on numerical measurement.

31

Qualitative Research
Qualitative research is less structured than most quantitative approaches. It does not rely on self response questionnaires containing structured response formats. Instead, it is more researcherdependent. The researcher interprets the data to extract its meaning and converts it to information.

32

Use of Qualitative Research


Generally, the less specific the research objective, the more likely that qualitative research tools will be appropriate. 1*When it is difficult to develop specific and actionable problem statements or research objectives. For instance, if after several interviews with the research client the researcher still cant determine exactly what needs to be measured, then qualitative research approaches may help with problem definition.

To gain further insight and crystallize the research problem


33

Use of Qualitative Research


2* When the research objective is to develop an understanding of some phenomena in great detail and in much depth. Qualitative research tools are aimed at discovering the primary themes indicating human motivations and the documentation of activities is usually very complete. 3* When a fresh approach to studying some problem is needed. This is particularly the case when quantitative research has yielded less than satisfying results. Qualitative tools can yield unique insights, many of which may lead the organization in new directions. 34

Qualitative versus Quantitative Research


Quantitative business research can be defined as business research that addresses research objectives through empirical assessments that involve numerical measurement and analysis approaches. Qualitative research is more apt to stand on its own in the sense that it requires less interpretation.
35

Qualitative versus Quantitative Research

36

Qualitative versus Quantitative Research


Exploratory research designs produce qualitative data. The focus of qualitative research is on stories, visual portrayals, meaningful characterizations, interpretations, and other expressive descriptions. In some situations the outcome of exploratory research is a testable research hypothesis. The results of these tests help decision making by suggesting a specific course of action.
37

Qualitative versus Quantitative Research


Qualitative research can be performed in many ways using many techniques. Orientations to qualitative research are very much influenced by the different fields of study involved in research. The major categories of qualitative research include
Phenomenologyoriginating in philosophy & psychology Ethnographyoriginating in anthropology Grounded theoryoriginating in sociology Case studiesoriginating in psychology and in business research
38

Common Techniques Used in Qualitative Research


Focus Group Interview Depth Interviews Conversations Semi-Structured Interviews Word Association/ Sentence Completion Observation Collages Thematic Apperception/ Cartoon Tests
39

Focus Group Interview


A focus group interview is an unstructured, freeflowing interview with a small group of people, usually between six and ten. Focus groups are led by a trained moderator who follows a flexible format encouraging dialogue among respondents. Common focus group topics include employee programs, employee satisfaction, brand meanings, problems with products, advertising themes, or new-product concepts.
40

Focus Group Interview

41

Focus Group Interview Advantages


Focus Groups offer several advantages: 1. Relatively fast 2. Easy to execute 3. Allow respondents to piggyback off each others ideas 4. Provide multiple perspectives 5. Flexibility to allow more detailed descriptions 6. High degree of scrutiny
42

Focus Group Interview Advantages


Flexibility If a researcher is investigating a target group to determine who consumes a particular beverage or why a consumer purchases a certain brand, situational factors must be included in any interpretations of respondent comments Piggybacking and Multiple Perspectives one respondent stimulates thought among the others and, as this process continues, increasingly creative insights are possible.
43

Focus Group Interview Advantages


Speed and Ease In an emergency situation, three or four group sessions can be conducted, analyzed, and reported in a week or so. The large number of research firms that conduct focus group interviews makes it easy to find someone to host and conduct the research.
Scrutiny A focus group interview allows closer scrutiny in several ways. Either through live video or a two-way mirror, some check on the eventual interpretations is provided through the ability to actually watch the research being conducted.
44

Focus Group Interview Disadvantages


First, focus groups require objective, sensitive, and effective moderators. It is very difficult for a moderator to remain completely objective about most topics.
Second, some unique sampling problems arise with focus groups. Researchers often select focus group participants because they have similar backgrounds and experiences or because screening indicates that the participants are more articulate or gregarious than the typical consumer

45

Focus Group Interview Disadvantages


Third, although not so much an issue with online formats where respondents can remain anonymous, traditional face-to-face focus groups may not be useful for discussing sensitive topics. A focus group is a social setting and usually involves people with little to no familiarity with each other.

46

Depth Interviews
A depth interview is a one-on-one interview between a professional researcher and a research respondent. Depth interviews are much the same as a psychological, clinical interview, but with a different purpose.

Laddering is a term used for a particular approach to probing, asking respondents to compare differences between brands at different levels
47

Depth Interviews
Each depth interview may last more than an hour. Thus, it is a time-consuming process if multiple interviews are conducted. It provide more insight into a particular individual than do focus groups. In addition, since the setting isnt really social, respondents are more likely to discuss sensitive topics than are those in a focus group. Depth interviews are particularly advantageous when some unique or unusual behavior is being studied.
48

Conversations
Conversation is an informal data-gathering approach in which the researcher engages a respondent in a discussion of the relevant subject matter.

This approach is almost completely unstructured and the researcher enters the conversation with few expectations. The goal is to have the respondent produce a dialogue about his or her lived experiences. Meaning will be extracted from the resulting dialogue.
49

Conversations
A conversational approach is advantageous because each interview is usually inexpensive to conduct. Respondents often need not be paid. This approaches, however, are prone to produce little relevant information since little effort is made to steer the conversation.

50

Semi Structured Interviews


Semi-structured interviews usually come in written form and ask respondents for short essay responses to specific open-ended questions. Respondents are free to write as much or as little as they want.
The semi-structured interview began with a freeassociation task: List the first five things that come into your mind when you think of a car salesman. This was followed up with a probing question: Describe the way a typical car salesman looks. 51

Free-Association/Sentence Completion Method


Free-association techniques simply record a respondents first cognitive reactions (top-of-mind) to some stimulus. Respondents view an ambiguous figure and are asked to say the first thing that comes to their mind. Free-association techniques allow researchers to map a respondents thoughts or memory.

52

Free-Association/Sentence Completion Method


For example: People who drink energy drinks are _______________________________________. A man who drinks an energy drink is _______________________________________. Imported energy drinks are most liked by _______________________________________.
53

Observation
Observation can be a very important qualitative tool. The participant-observer approach typifies how observation can be used to explore various issues. Meaning is extracted from field notes.

Field notes are the researchers descriptions of what actually happens in the field. These notes then become the text from which meaning is extracted.
54

Observation
Observation can either be very inexpensive, such as when a research associate sits and simply observes behavior, or it can be very expensive, as in most participant-observer studies.

55

Collages
Business researchers sometimes have respondents prepare a collage to represent their experiences. The collages are then analyzed for meaning much in the same manner as text dialogues are analyzed. Computer software can even be applied to help develop potential grounded theories from the visual representations.

56

Collages
Harley-Davidson commissioned research in which collages depicting feelings about HarleyDavidson were compared based on whether the respondent was a Harley owner or an owner of a competitors brand.
Like sentence completion and word association, collages are often used within some other approach, such as a focus group or a depth interview. Collages offer the advantage of flexibility but are also very much subject to the researchers interpretations.
57

Projective Research Techniques


An indirect means of questioning enabling respondents to project beliefs and feelings onto a third party, an inanimate object, or a task situation.
Projective techniques usually encourage respondents to describe a situation in their own words with little prompting by the interviewer.
Individuals are expected to interpret the situation within the context of their own experiences, attitudes, and personalities and to express opinions and emotions that may be hidden from others and possibly themselves 58

Thematic Apperception Test (Tat)


A thematic apperception test (TAT), sometimes called the picture interpretation technique, presents subjects with an ambiguous picture(s) and asks the subject to tell what is happening in the picture(s) now and what might happen next.

59

Thematic Apperception Test (Tat)


Hence, themes (thematic) are elicited on the basis of the perceptual-interpretive (apperception) use of the pictures. The researcher then analyzes the contents of the stories that the subjects relate. A TAT represents a projective research technique.
Frequently, the TAT consists of a series of pictures with some continuity so that stories may be constructed in a variety of settings.
60

Misuses of Exploratory and Qualitative Research

61

When only one researcher interprets the meaning of what a single person said in a depth interview or similar technique, one should be very cautious before major business decisions are made based on these results. Is the result replicable?
Replication means that the same results and conclusions will be drawn if the study is repeated by different researchers with different respondents following the same methods
62

Scientific Decision Processes


Objectivity and replicability are two characteristics of scientific inquiry. Are focus groups objective and replicable? Would three different researchers all interpret focus group data identically? How should a facial expression or nod of the head be interpreted? Have subjects fully grasped the idea or concept behind a nonexistent product?

63

Scientific Decision Processes


Many of these questions are reduced to a matter of opinion that may vary from researcher to researcher and from one respondent group to another. Therefore, a focus group, or a depth interview, or TAT alone does not best represent a complete scientific inquiry.

64

Scientific Decision Processes


But is a scientific decision approach always used or needed? In practice, many business decisions are based solely on the results of focus group interviews or some other exploratory result. The primary reasons for this are (1) time, (2) money, and (3) emotion.

65

Scientific Decision Processes


Time Sometimes, researchers simply are not
given enough time to follow up on exploratory research results. Companies feel an increasingly urgent need to get new products to the market faster. Thus, a seemingly good idea generated in a focus group is simply not tested with a more conclusive study.

66

Scientific Decision Processes


The risk of delaying a decision may be seen as greater than the risk of proceeding without completing the scientific process. Thus, although the researcher may warn against it, there may be logical reasons for such action. Thus, there is less likelihood of good results from the decision than if the research process had involved further testing

67

Scientific Decision Processes


MONEY Similarly, researchers sometimes do not
follow up on exploratory research results because they believe the cost is too high. Realize that tens of thousands of dollars may have already been spent on qualitative research. Managers who are unfamiliar with research will be very tempted to wonder, Why do I need yet another study? and What did I spend all that money for?
68

Scientific Decision Processes


Thus, they choose to proceed based only on exploratory results. If the decision itself does not involve a great deal of risk or if it can be reversed easily, the best course of action may be to proceed to implementation instead of investing more money in confirmatory research. Remember, research shouldnt be performed if it will cost more than it will return.
69

Scientific Decision Processes


EMOTION Time, money, and emotion are all
related. Decision makers sometimes become so anxious to have something resolved, or they get so excited about some novel discovery resulting from a focus group interview, that they may act rashly. Perhaps some of the ideas produced during the motivational research era sounded so enticing that decision makers got caught up in the emotion of the moment and proceeded without the proper amount of testing.
70

Secondary Data

71

Secondary Data
Research projects often begin with secondary data, which are gathered and recorded by someone else prior to (and for purposes other than) the current project. Secondary data usually are historical and already assembled. They require no access to respondents or subjects.

72

Secondary Data
Data
(also called data transformation) is the process of changing the original form of data to a format more suitable for achieving a stated research objective. For example, sales for food products may be reported in pounds, cases, or dollars. An estimate of dollars per pound may be used to convert dollar volume data to pounds or another suitable format
73

conversion

Secondary Data
Data Mining refers to the use of powerful
computers to dig through volumes of data to discover patterns about an organizations customers and products. For example, neural networks are a form of artificial intelligence in which a computer is programmed to mimic the way that human brains process information.

74

Secondary Data
Cross-checks of data from multiple sources,
should be made to determine the similarity of independent projects. When the data are not consistent, researchers should attempt to identify reasons for the differences or to determine which data are most likely to be correct.

75

General Objective For Secondary Research

76

General Objective For Secondary Research


Fact-Finding
The simplest form of secondary-data research is fact-finding. Fact-finding can serve more complex purposes as well. In the digital age we live in, the use of music as a means to notify users of a call is commonplace. The Research Snapshot on the next page gives some of the amazing growth facts predicted in this industry.
77

General Objective For Secondary Research


Fact-Finding
Identification Of Consumer Behavior For A Product Category Trend Analysis

Environmental Scanning

78

General Objective For Secondary Research


Model Building
The second general objective for secondary research, model building, is more complicated than simple fact-finding. Model building involves specifying relationships between two or more variables, perhaps extending to the development of descriptive or predictive equations
79

General Objective For Secondary Research


Model Building
Estimating Market Potential For Geographic Areas

Forecasting Sales
Analysis Of Trade Areas And Sites

Analysis Of Trade Areas And Sites


80

General Objective For Secondary Research


Database Marketing and Customer Relationship Management
CRM systems are a decision support system that manage the interactions between an organization and its customers. A CRM maintains customer databases containing customers names, addresses, phone numbers, past purchases, responses to past promotional offers, and other relevant data such as demographic and financial data.
81

Sources of Secondary Data


Most organizations routinely gather, record, and store Internal Data to help them solve future problems. If the data are properly coded into a modular database in the accounting system, the researcher may be able to conduct more detailed analysis using the decision support system. Other useful sources of internal data include salespeoples call reports, customer complaints, service records, warranty card returns, and other records2
82

Sources of Secondary Data


External data are generated or recorded by an entity other than the researchers organization. The government, newspapers and journals, trade associations, and other organizations create or produce information. Traditionally, this information has been in published form, perhaps available from a public library, trade association, or government agency.
83

General Objective For Secondary Research


Database Marketing and Customer Relationship Management
Database marketing is the practice of using CRM databases to develop one-to-one relationships and precisely targeted promotional efforts with individual customers.

84

Descriptive Research Design


(Survey & Observation)

85

Survey Research
The purpose of survey research is to collect primary datadata gathered and assembled specifically for the project at hand. Often research entails respondentsto provide answers to written or spoken questions. These interviews or questionnaires collect data through the mail, on the telephone, online, or face-to-face.

Thus, a survey (aka Sample Survey) is defined as a method of collecting primary data based on communication with a representative sample of individuals.
86

Survey Research
The type of information gathered in a survey varies

considerably depending on its objectives. Typically, surveys attempt to describe what is happening or to learn the reasons for a particular business activity.

Identifying characteristics of target markets, measuring customer attitudes, and describing consumer purchase patterns are all common business research objectives.
87

Survey Research
Because most survey research is descriptive research, the term survey is most often associated with quantitative findings. Although most surveys are conducted to quantify certain factual information, some aspects of surveys may also be qualitative. In

new-product development, a survey often has a qualitative objective of refining product concepts. Stylistic, aesthetic, or functional changes may be made on the basis of respondents suggestions.
88

Advantages of Surveys
Surveys provide a quick, inexpensive, efficient, and accurate means of assessing information about a population. During the last two decades, survey

research techniques and standards have become quite scientific and accurate. When properly conducted, surveys offer managers many advantages.

89

Errors in Survey Research


Random Sampling Error
A statistical fluctuation that occurs because of chance variation in the elements selected for a sample.

Systematic Error
Error resulting from some imperfect aspect of the research design that causes respondent error or from a mistake in the execution of the research.
90

91

Errors in Survey Research


Respondent Error
A category of sample bias resulting from some respondent action or inaction such as nonresponse or response bias.

Non-response Error
The statistical differences between a survey that includes only those who responded and a perfect survey that would also include those who failed to respond.
92

Errors in Survey Research


Response Bias
occurs when respondents tend to answer questions with a certain slant. People may consciously or unconsciously misrepresent the truth.

Administrative Error
An error caused by the improper administration or execution of the research task.

93

Errors in Survey Research


Data-processing Error
A category of administrative error that occurs because of incorrect data entry, incorrect computer programming, or other procedural errors during data analysis.

Sample Selection Error


An administrative error caused by improper sample design or sampling procedure execution.
94

Classifying Survey Research Methods


Structured/Unstructured and Disguised/ Undisguised Questionnaires
Temporal Classification

95

Classifying Survey Research Methods


Structured Question
A question that imposes a limit on the number of allowable responses.

An unstructured question does not restrict the


respondents answers. An open-ended, unstructured question such as Why do you shop at Wal-Mart? allows the respondent considerable freedom in answering.
96

Classifying Survey Research Methods


Undisguised Questions Straightforward questions that assume the respondent is willing to answer
Disguised questions Indirect questions that assume the purpose of the study must be hidden from the respondent
97

Classifying Survey Research Methods


Temporal Classification

Cross-sectional Study
A study in which various segments of a population are sampled and data are collected at a single moment in time. The typical method of analyzing a cross-sectional survey is to divide the sample into appropriate subgroups. (e.g., less than 5 years, 59 years,1014 years, and 15 years or more)
98

Classifying Survey Research Methods


Temporal Classification Longitudinal Study
A survey of respondents at different times, thus allowing analysis of response continuity and changes over time. Respondents are questioned at multiple points in time.
consumer panel- A longitudinal survey of the same sample of individuals or households to record their attitudes, behavior, or purchasing habits over time.
99

Survey Research in Practice


Interviews as Interactive Communication
When most people think of interviewing, they envision two people engaged in a face-to-face dialogue or a conversation on the telephone. Electronic interactive media allow researchers to reach a large audience, personalize individual messages, and interact using digital technology. Survey respondents today are not passive audience members. They are actively involved in a twoway communication using electronic interactive media
100

Survey Research in Practice


Non-interactive Media
The traditional questionnaire received by mail and completed by the respondent does not allow a dialogue or an exchange of information providing immediate feedback. Hence, self-administered questionnaires printed on paper are non-interactive.

101

Survey Research in Practice


Interviews as Interactive Communication Personal Interviews
To conduct interviews, the researcher may communicate with individuals in person by going door to- door or intercepting them in shopping malls, or interviews may take place over the telephone.

102

103

You might also like