PROGRAM ANALYSIS

OpenCourseWare is a team of faculty and staff at the Massachusetts Institute of

Technology (MIT) that publishes and manages MOOCs (massive open source courses) from
their own curriculum degree programs. MOOCs have been around for several years and in
addition to MIT offering select curriculum online, other elite universities like Princeton, Brown,
and Columbia also offer curriculum online (Pappano, 2012). Currently, MIT offers over 2,250
courses online and the course being evaluated is Technologies for Creative Learning (for
reference, here is the direct course link: http://ocw.mit.edu/courses/media-arts-and-sciences/mas714j-technologies-for-creative-learning-fall-2009/index.htm).
The purpose of offering this course and other MOOCs by MIT relates to MITs mission
statement, to advance knowledge and educate students (Massachusetts Institute of Technology,
2016). The Technologies for Creative Learning course explores the design of educational
technologies and creative learning environments. The course utilizes assessment strategies such
as case studies, activities with new educational technologies, learning experience reflections, and
discussions of strategies underlying the design of the technologies and activities (Massachusetts
Institute of Technology, 2016). The version of Technologies for Creative Learning available
online was last taught in the Fall 2009 semester. Prior to the 2009 offering, the course was also
taught in the Fall 2004 semester.
The following are the stakeholders in the course: MIT, OpenCourseWare, Professor
Mitchel Resnick, and the MOOC students (who have recently completed the course) for this
specific course. When considering who the stakeholders in this program were, I first considered
stakeholders at the course level; the teacher and the students. I then considered the stakeholders
that make the course possible; the OpenCourseWare team and MIT. Although this may seem like

a small pool of stakeholders, student stakeholders have the potential to represent hundreds, if not
thousands, of stakeholders (I do not have exact numbers due to limited access to the course).
The political environment appears to be a rather sable one. MITs president, L. Rafael
Reif, stressed that the MOOCs are available for you meaning anyone the courses are
available in hopes that the available curriculum with be utilized to pursue student interests and
improve life (Massachusetts Institute of Technology, 2016). Echoing that same message, the
OpenCourseWare team operates as a 27 person unit to provide instructor developed curriculum,
this is case, Professor Mitchel Resnicks curriculum, to deliver the course content online. This
speaks very highly of the interpersonal dynamic within the organization, as there is a large
amount of work and collaboration to ensure the program is delivered for MOOC students.
When considering ethical challenges involved in the evolution of this program/course, the
biggest challenge is anticipated to be the un-biased involvement from the stakeholders;
specifically the Instructor (Professor Mitchel Resnick) and the OpenCourseWare team. When
considering the available version of the course online is from 2009, a student would expect the
course content to be relevant in todays world. A specific focus of the course is exploring
innovative and new technologies so as an evaluator, concern might stem from the subject
matter expert (Instructor/Professor Mitchel Resnick) confirming that such information is still
relevant and worthwhile today. As an evaluator, I must also consider the OpenCourseWare
teams approach to taking the course curriculum from the on-ground version to the MOOC
version and if they were successful at retaining the same course objectives/goals and
educational message.

EVALUATION MODEL TABLE

Evaluation Model
EXPERTISE AND
CONSUMER-ORIENTED
APPROACHES

PROGRAM-ORIENTED
EVALUATION
APPROACHES

DECISION-ORIENTED
EVALUATION
APPROACHES

PARTICIPANT-ORIENTED
EVALUATION
APPROACHES

Advantages
Expertise: Considered the most
mature type of formal and public
evaluation.
Consumer: Utilizes experts in
evaluation, lowering the chances of
bias in the evaluation.
The simplicity of the approach; it is
easily understood, easy to follow,
easy to implement, and produces
relevant information.
Tends to prompt program
directors/administrators to reflect
upon program objectives/goals.
Considered a mature approach and
still in frequent use.
Provides relevant information that
helps program
directors/administrators make
decisions effectively.
Tends to improve the validly of the
program evaluation by utilizing
relevant knowledge that
stakeholders have that an evaluator
does not about the program.
The participation of stakeholders
also tend to foster more excitement
and positivity about the evaluation
results.

Disadvantages
Expertise: May permit
evaluators to make judgement
that reflect more of a personal
bias.
Consumer: Potential to increase
product costs.
Potential for evaluators to
ignore other outcomes of the
program due to primary focus
on objectives.
Approach also tends to neglect
program contexts.
The focus on decisions is also
considered a disadvantage
because of its narrow focus.
Programs that lack decisive
leadership will not benefit from
this approach.
Larger potential for bias with
larger number of participants
included in the evaluation.
Feasibility of having most or all
of the stakeholders participate
in the evaluation.

Explain your choice of model for your program evaluation:

When considering the most effective model for my program evaluation, I have selected the combined of
the expertise and participant-oriented evaluation approaches. Both approaches seem to overlap in regard
to needing information from stakeholders to complete my evaluation. Firstly, I myself did not develop or
implement the MOOC and would rely on the instructional designer(s) and professor for information
regarding develop and implementation. And secondly, I did not complete the course as a student, so I
would also need student feedback to gather data regarding met course objectives, course experience,
course likes/dislikes, etc.

EVALUATIVE CRITERIA
For reference, the course being evaluated is Technologies for Creative Learning (for
reference, here is the direct course link: http://ocw.mit.edu/courses/media-arts-and-sciences/mas714j-technologies-for-creative-learning-fall-2009/index.htm). The purpose of this evaluation is
to ensure that the course is effective and relevant after being brought from a traditional
classroom format from 2009. To achieve this purpose, I would propose the following 5
evaluation questions:
1) Is the online course design effective in helping students achieve the course objectives?
2) Are the educational technologies referenced and utilized in the course are still relevant
and applicable today?
3) Do students who elect to take this course, find the course and its content engaging to
complete the entire course?
4) Does this course contribute to MITs mission of advancing student knowledge?
5) What type of update or improvement plan needs to be put into place to ensure the course
remains relevant over time?
These questions relate directly to the stated purpose of this evaluation, they will act as a
guide to ensure the appropriate pieces of information are gathered for a valid program
evaluation. Even though MIT and the OpenCourseWare team manages a wide range of MOOCs,
the MOOC program as a whole is note being evaluated at this time, only the Technologies for
Creative Learning course is being evaluated. The health of the MITs MOOC program is also
important, but for the purposes of this program evaluation, there is only one course of focus.
Because the focus of this evaluation is on one MOOC, the choice of evaluation questions

highlight the need to ensure course information is relevant and that students are able to achieve
course objectives in the online format the course was converted to.
When considering evaluation questions, the professor, the instructional designer for the
course, OpenCourseWare representative, and department chair (for whichever department the
course falls under I would assume it would be Education) would be involved. All of these
individuals would be directly involved with the course content, course goals, and delivery of the
course. Modeled after the selected approaches for this evaluation, the role of the stakeholders
will be expertise and participatory in nature in regard to determining the evaluative criteria. Their
input on evaluative criteria is important because these are the individuals who have developed,
designed, implemented, and managed the course over several years.

DATA COLLECTION DESIGN AND SAMPLING STRATEGY

Note for Instructor: I have posted my audio file in the Doc Sharing section of the course.

REPORTING STRATEGY TABLE

Stakeholder

Reporting Strategy

Implications

Massachusetts Institute
of Technology (MIT)

Written Executive
Summary

Low impact on the

institution as a whole.

MITs
OpenCourseWare Team

Written Executive
Summary

Fair impact on the

MOOC collection as
data may reveal possible
trends.

Courses Instructional
Designer

Written Final Report

and Verbal Presentation

Depending on if the
impact is considering
negative or positive,
negative impact would
create the need for the
instructional designer to
reconsider the design of
the course (pending on
what findings revealed
specifically about the
design).

Professor Mitchel
Resnick

Written Final Report

and Verbal Presentation

Depending on if the
impact is considering
negative or positive,
negative impact would
create the need for the
professor to reconsider
the course content and
possible redevelop
(pending on what
findings revealed
specifically about the
course content).

Stakeholder
Involvement
Limited Amount Because this evaluation
is of only one course
that doesnt greatly
impact operations of
MIT, a final written
report would be
sufficient in
communicating the
evaluation data.
Fair Amount Because
the OpenCourseWare
team are the individuals
who manage the course,
they would be included
on an overall summary
of the evaluation data
that may speak to trends
for other MOOCs.
High Amount Because
the ID is the individual
who designed the course
and its delivery, the
data collected from the
students would greatly
benefit them in looking
at what worked/what
didnt work in regard to
the course design. A
Written Final Report
and Verbal Presentation
would ensure this data is
communicated
effectively.
High Amount Because
the professor is the
individual who
facilitates the course,
the data collected from
the students would
benefit him in looking at
what content students
felt was relevant, etc. A
Written Final Report
and Verbal Presentation
would ensure this data is
communicated

Department Chair for

Course that Program is
In

Written Final Report

Fair impact (pending the

involvement of Chairs
in the MOOC program).
If content is found to be
outdated or ineffective,
it may prompt the Chair
to get involved.

Students

Short Written
Communication (Email)

Limited impact as these

students have already
completed the course.

effectively.
Fair Amount Because
the Chair has duties in
regard to their
departments
curriculum, they would
be included to ensure
data regarding the
course content is
communicated to in
case they need to step in
and work with the
professor/ID.
Limited Amount
Because these students
have completed the
program, they will more
than likely have limited
interest in our findings.
However, as
stakeholders, they
should be made aware
of the data findings.
These would be sent in
an email format or
another short written
format.

Values, Standards, and Criteria: Due to my (the evaluator) work in higher education and course design,
I will utilize an audit trail to minimize any bias during the evaluation. I will also follow such principles as
communicating approaches/methods accurately; representing data and findings accurately; conducting
evaluations in a respectful way; and allowing stakeholders access to the evaluation information
(Fitzpatrick, Sanders, & Worthen, 2010).
Potential ethical issues: At this stage, potential ethical concerns may be findings are ignored by the
stakeholder (MIT personnel); findings are misused the by stakeholder (MIT personnel); or findings are
misinterpreted by the stakeholder (MIT personnel) (Fitzpatrick, Sanders, & Worthen, 2010).

References
Fitzpatrick, J., Sanders, J., & Worthen, B. (2010). Program evaluation: Alternative approaches
and practical guidelines (4th ed.). Boston, MA: Pearson.
Massachusetts Institute of Technology. (2016). Technologies for Creative Learning. Retrieved
from http://ocw.mit.edu/courses/media-arts-and-sciences/mas-714j-technologies-forcreative-learning-fall-2009/index.htm
Massachusetts Institute of Technology. (2016). Our History. Retrieved from
http://ocw.mit.edu/about/our-history/
Massachusetts Institute of Technology. (2016). Presidents Message. Retrieved from
http://ocw.mit.edu/about/presidents-message/
Pappano, L. (2012). The Year of The MOOC. Retrieved from
http://www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-aremultiplying-at-a-rapid-pace.html?_r=0