INDE / ENVH 549 A: Research Methods in Human Factors

Group Project
Autumn 2021

You are tasked with developing an online learning system that refers to the use of networked computing and
communications technologies to support learning (e.g., Smartphone application, social media, digital learning, Google
glass, and virtual reality).

• Identify your online learning system.

• Identify the subpopulation of online learning users your system will focus on. Some examples of subpopulations
include the children, elderly, the visually impaired, and students with ADHD.
• Develop a system that can help them learn efficiently, safely, and with joy.

Some examples:
• GRE preparation applications for non-native English speakers.
• Virtual reality environment for the elderly.
• Redesigning ALEKS for persons with disabilities.

The actual design needs to be based on research and justified using scientific literature (e.g., journal articles). Consider
what is needed for successful implementation. For this project, you will conduct a cognitive task analysis and use the
findings to generate input for an actual system design. You will then develop a rapid prototype of the system and identify
how you would do usability testing to assess the feasibility of the system (e.g., Will the display be too distracting? Are the
alerting cues at a reasonable level to be heard, but not too loud that they will startle?).

The following assignments are to be uploaded to canvas on the due date noted.
1. Cognitive Task Analysis: 11/7 midnight
2. Design Requirements: 11/21 midnight
3. Construct Prototype: 12/2 midnight
4. Final Presentation: 12/7, 12/9 in class
5. Final Paper: 12/12 midnight

1. Cognitive task analysis (10 pts)

Due date: 11/7 midnight
In class, we discuss the challenges of observing cognitive tasks. However, it is still possible to capture the essence of what
someone is doing. You would have read the paper “Applied cognitive task analysis (ACTA): a practitioner's toolkit for
understanding cognitive task demands" by Laura G. Militello and Robert J. B. Hutton before starting this exercise.

1.1 Task Diagram

Define the scope of the task that you have identified earlier. Using the instructions outlined in the Militello & Hutton
paper, create a task diagram to show how your target group would typically perform the task. Note that the figure in the
review paper is a bit weak, so be creative. Ultimately you should break the task into at least 15 subtasks. For each task,
provide a sentence or two descriptions of what you mean by that task.

1.2 Knowledge Audit

Go through the knowledge audit exercise for five of the most difficult tasks. Go through all the basic probes and create a
table similar to Table 1 in the Militello and Hutton paper for each of your selected subtasks. You may have to adjust the
wording of the probes so that they make sense in your context. This step should help you identify the cognitive gaps and
difficulties for the at-risk group to learn through a cyberlearning system.

1.3 Simulation Interview

Page 1 of 4
Find a video (e.g., youtube) of instructions on how to use some features of existing devices/software related to the task
that you selected. For example, an instruction video of how to use iTunes to add music:
https://www.youtube.com/watch?v=kb0g5DBYTjs Identify the video that you watched and note how some of the more
complex tasks are explained. Define your subject matter experts (SME). Are these the same (or similar to) the tasks that
you identified in the Knowledge audit in step 2? What errors could your study participants make while performing these
tasks?

1.4 Lessons Learned

Think about what you learned from this exercise. Summarize what you learned in bulleted points. Then create a list of
recommendations for changing designs that would improve usability, productivity, and safety for all users.

2. Design Requirements (5 pts)

Due date: 11/21 midnight
It is important to define the goals, variables, and constraints before you start to design a new system or product.

2.1 Define Purpose

Define the purpose of your application and the general tasks it is meant to support. Use the Cognitive Task Analysis to
create a specific design goal to enhance the targeted user group’s learning performance.
• Develop a simple, one-sentence description of your design goal. Your design goal should be quantifiable and
falsifiable (e.g., testable).
• Identify the overall purpose and 2-3 general tasks that your product should be able to achieve. Describe how people
might use it and what activities must be supported.
• Identify which steps in your cognitive analysis would be changed or improved by your product.
• Enumerate your design constraints. These should also be quantifiable (where possible) and include such issues as
learning time, operating performance, cost, and productivity.

2.2 Study Protocol

Design an experiment that would provide an evaluation of your product. This includes defining a falsifiable test that
could be conducted after your product is built that would determine whether or not your design goals were met. You do
not need to conduct the test, but rather just identify what you could do to evaluate it. For example, if you were going to
do an experiment to evaluate your product, how would you construct the experiment? What criteria would you use for
judging the performance and success in completing the tasks?

3. Construct Rapid Prototype (10 pts)

Due date: 12/2 midnight
Rapid-prototyping to demonstrate your design alternative. The objective of this exercise is to develop a prototype
application that takes your earlier concept and makes it concrete. Make sure to document each step, review your results
and identify how your prototype might be expanded or enhanced. In this step, you will create a prototype that conveys
your vision for how your application can support your target group.

3.1 Development requirements of your prototype.

In this task, you will document and define the development requirements for your prototype.
1. Based on a design defined in the Design Requirements, identify ONE function that is most promising.
2. Define and describe a use scenario/use case for this function. This scenario would elaborate on the design
requirements by including a description of the users, the cues that initiate the interaction, the context (e.g., time of
day, time constraints), the information requirements, cognitive resources (e.g., the information users need to
bring to the situation and the working memory demands), and interaction outcome.

Page 2 of 4
 3. Define the detailed activities and information flow. This information should be described in an information flow
diagram, where the ellipses represent activities and arrows linking activities that represent the information flow.
Be sure to include every activity the user performs, every activity performed by the system and any data
requirements (e.g., user input, real time data, web-based information, or database).
Think through these steps and select a function that is simple enough to represent with no more than 10-15 activities. You
should keep your prototype simple.

3.2 Develop paper prototypes (low-fidelity prototypes)

Use the system and prototype requirements you developed in 3.1 Development Requirement to create your prototype.
Start with a paper prototype. For example, you may draw the screens on paper and explore how you would navigate
from screen to screen, or from scene to scene. Test out the paper prototypes with people in your group noting where the
design works well and where it may fail. Document design changes that you identified with the paper prototype. Starting
with a paper prototype will save you much time when you build it on the computer. The paper prototype does not need
to be pretty—it is a tool for you to try out basic design ideas.

3.3 Learn to and develop virtual prototypes (high-fidelity prototypes)

There are many ways to develop virtual prototype. You can even use a presentation software package (e.g, PowerPoint,
or Keynote) to create a prototype with hyperlinks. In presentation format, each slide would be a different screen in your
prototype and the hyperlinks can be attached to buttons on each screen enabling you to navigate between screens by
clicking on the buttons when you are in the presentation view. In PowerPoint, there are some useful function under the
slide show menu (e.g., “action buttons", “action settings" and “custom animation"). PowerPoint also links to Visual Basic
programming environment. These functions can be used to create surprisingly complex prototypes. Use a web-based
search engine to identify a good tutorial for rapid prototyping. Every group member should spend time going through
the tutorial. The following is an example for power point: https://www.youtube.com/watch?v=bJh4QUzMPyQ. Here is an
example for a mobile app: https://www.youtube.com/watch?v=Yt2troF-Eyc Please feel free to find a better tutorial. In
your report, identify which software was used. Then, using the tool that you learned, include your paper and virtual
prototypes to the report.

3.4 Describe your prototype

After you have completed the three steps, summarize your results and identify what changes or enhancements you
would like to make to your prototype. Also, summarize any problems you encountered in translating the requirements
into the prototype. Document any changes in the requirements that you made in creating the prototype.

4. Final Project Report Requirements (15 pts)

Due date: 12/12 midnight

You are expected to create a report with the following:

• A cover sheet with the names of the group members.

• Documentation of all steps of this process.
• A total of approximately 10-15 pages. Font size should be 11-12 point, Times Roman or Arial, Single-spaced with
one-inch margins.
• Include figures and tables as appropriate.
• Include your rapid prototype.

4.1 Grading protocol (100 points)

4.1.1 Writing (10 points)
Please make sure to read and RE-read your manuscript, checking for spelling and grammatical errors. Have SEVERAL
PEOPLE review. It should sound like it was written by one person, telling a cohesive story. That is, make sure
contributions from various group members are well integrated.

Page 3 of 4
4.1.2 Aesthetic layout (10 points)
The paper is organized as described and includes appropriate use of section headings. Tables and graphs professionally
crafted, labeled, and cited appropriately in the text. References used are properly cited in the manuscript.

4.1.3 Story (20 points)

The paper should include clear and concise descriptions of what was done and why. It should include all updates I
recommended. The paper should be organized with the following sections:
1. Introduction. Includes a literature review (with journal article citations), problem statement, and justification for
creating a system for your interest group.
2. Cognitive Task Analysis. Includes an explanation (and interpretation) of your CTA; how it helped you identify gaps
and where to proceed for your product design.
3. Prototype. Includes the methods and assumption you developed as you move forward with your prototype design. See
section 4.1.4 for details.
4. Experimental Methods. How you would test the prototype in an iterative manner. Note that this was part of Cognitive
Task Analysis (Part 1). However, you may want to consider revising now that you have developed a prototype.
5. Discussion and Conclusions. See section 4.1.5 for details.

4.1.4 Rapid prototype and prototype demonstration (20 points)

This will be part of your class presentation and should also be included as an appendix to your report. In the report,
present the following:
• The process you used to design this product/system.
• Justification for any design changes. Are your redsgins are appropriate for the user population?
• Demonstrate the use case(s) identified in the project.

4.1.5 Insightful interpretation of data in tables, charts, or appendices (20 points)

Provide useful design implications derived from analysis. The discussion and conclusions should clearly connect to data
and include the following:
• Limitations of your design, what could have done better, and what changes would you make
• Address questions/comments from the class presentation (some of the comments may be addressed in other
sections).
• Implication and uniqueness of your project
• Contribution of your project to academia and industries
• Support any recommendations with peer reviewed citations.

4.1.6 Successful completion of specified tables, charts, appendices (20 points)

All steps in assignment performed and documented. All suggested tables, graphs, and appendices complete and correct;
Numerical calculations correct.

Page 4 of 4