Evaluation Metrics Project

Evaluation Metrics Project = Project overview =

The Open.Michigan team aims to design and implement an evaluation plan that illustrates the impact of the Open.Michigan initiative as it adds value to the University of Michigan's teaching and learning environment and broadens its global education mission. At this stage in this initiative, it is imperative we determine measurable outcomes for our work and efforts and establish accountability within the open educational movement. This evaluation plan is part of a larger strategic plan and assessment that will take place over the fall 2010 and winter 2011 terms. With the help of a temporary, part-time employee, we aim to implement the phases outlined below and refine our initial mission, value statement, goals and objectives and our anticipated outcomes and tools of measurement by winter 2011. Open.Michigan staff will work to refine this framework and present it in a usable state to the temporary employee so phases 1-3 can be completed in a timely manner in the course of the next months (October-December). We anticipate that data collection will begin in the latter months of 2010 and the early months of 2011 and our overall strategic plan will be implemented by the third fiscal quarter. Evaluation on this strategic plan will take place in the fourth fiscal quarter.

NOTE: This page now includes finalized documentation from the evaluation project. See Evaluation and Metrics Planning Documents for documentation including anticipated outcomes, attached measures of impact, initial outlines for the survey and follow up questions and discussion.

Presentations and Papers
Measuring Our Impact: the Open.Michigan initiative at the OCWC Global meeting in May 2011.

[[Media:OCWCPresentation1.ppt|Presentation]] [[Media:Measuring_our_Impact_OpenMichigan_Rodgers.doc|Paper]]

= Data and Resulting Documents, 2011-2012 =

Last updated: April 12, 2012

Several artifacts were created from this evaluation plan that include a strategic vision document, a consolidation of analytics data and other types of quantitative data. What follows are links to these documents that are being maintained by Open.Michigan staff. We thank our Research Assistant Ted McCarthy for putting this information together and maintaining it until April 30, 2011.

Note: This document is still in progress and must be updated by hand. The data may not reflect current numbers or status of certain activities. Please contact Open.Michigan (open.michigan AT umich.edu) if you would like access to the most up to date information about the growth, use and impact of Open.Michigan's activities.

Aggregate Data Summaries
The report summarizing the aggregate survey results can be found online

Introduction

In March 2011, Open.Michigan launched a campus wide survey to uncover the sharing habits and knowledge of open educational resources (OER) at the University of Michigan. This survey is part of an ongoing evaluation effort by the initiative. The goals of this evaluation are to measure Open.Michigan’s impact on the University of Michigan (U-M) campus and on the broader open education landscape, as well as its progress toward overall objectives and mission. A preliminary analysis of these survey results were examined in a presentation at the 2011 OpenCourseWare Consortium Global Meeting.

Since this presentation, the Open.Michigan team has assembled an aggregated view of responses based on general disciplines and types of education. These aggregates can be found on our wiki at: http://openmi.ch/INxNSu. These aggregations are intended to illustrate the habits, culture and perspective on sharing that might differ by college or schools across the University of Michigan. This overview provides a summary of the results of the aggregate surveys and examples of how our initiative has addressed the priorities and objectives that were formed after the initial analysis of the survey. Each aggregation of survey results is available in a separate document on the Open.Michigan wiki under the "Evaluation and Impact" section.

Aggregate survey results

The Open.Michigan initiative has been producing and promoting high quality Open Educational Resources (OER) from the University of Michigan (U-M) and its partner institutions since early 2008. As the initiative refines its global education mission, it is imperative that we understand the value the initiative adds to U-M and to the global learning community. The outcomes from this effort will help Open.Michigan continue to strengthen accountability within our own institution and across the open education movement. They will also provide other institutions with valuable data and insight onto how open knowledge sharing contributes to broader educational opportunities and collaboration across the world.

In 2011, the Open.Michigan team launched three university-wide surveys as part of an effort to measure the impact of the Open.Michigan initiative on the teaching and learning environments at the University of Michigan. These surveys gathered information about the motivations for resource use and sharing in academic settings in addition to the awareness of and support for OER development and use. Preliminary results of these surveys were delivered at the OpenCourseWare Consortium's Global Meeting in May, 2011.

Open.Michigan is currently housed in the Office of Enabling Technologies in the Medical School. Much of Open.Michigan's publication, outreach and consulting work supports the academic goals of the Medical School. This summary aggregates survey results from the Medical School faculty and students who completed the surveys and is the first in a series of survey summaries.

[[Media:UMMS_AggregateData.pdf|Medical School survey results]]

[[Media:ProfGradSch_AggregateData.pdf|Professional Schools survey results]]

[[Media:LSA_AggregateData.pdf|Literature, Sciences and Arts survey results]]

All survey summaries licensed under a Creative Commons Attribution 3.0 license.

Growth, Use of U-M Produced OER
Aggregate Data for Open.Michigan Evaluation 2010-2011

[[Media: Measuring_Our_Impact_Infosheet_Updated.docx |Measuring Our Impact Summary]] (from OCWC Global meeting, May 2011)

This document contains data pulled from Google Analytics, our OERbit platform and from statistics kept by Open.Michigan staff. Open.Michigan plans to develop a Drupal module for our OERbit platform that will give others regular access to updated Google Analytics data and automate some of the processes we use for capturing data such as published resources, objects and materials.

Outreach and Education
Awareness Building

This document includes most of our presentations, events, workshops and associated trainings in an attempt to analyze our engagement efforts with the University of Michigan community and the broader open education landscape, including interactions with our partner institutions. Materials downloads for some of these presentations can be found at: Presentation, poster, and diagram downloads

Survey Tools
Open.Michigan staff developed a series of surveys that were administered to the U-M community, including all students, all faculty and select groups of staff (instructional technologies, academic support staff). The following table represents our response rate for the survey to the University of Michigan community. Open.Michigan also surveyed visitors to our website and received 88 responses between April 4-28, 2011. These surveys were generated using Qualtrics and embedded logic. The question sets may not adequately reflect this logic. Some question sets, especially those about use or creation of OER was triggered based on respondent answers to questions indicating prior knowledge with OER.

Question sets: 

Faculty

Student

Staff

Self Learner

= Timeline and Methodology =

Phase 1: Defining the mission and objectives
Timeframe: Fall 2010

A methodology for defining the goals and objectives:

Phase 2: Conduct an Environmental Scan
Timeframe: Fall 2010

Targeted Audience

Direct scope:


 * 1) U-M audiences (authors and users): faculty, staff, administration, students
 * 2) Faculty, staff, administration, students at peer institutions (state, domestic, English-speaking)
 * 3) Self-learners accessing and using Open.Michigan's OER

Indirect scope


 * 1) International/National OER community
 * 2) Multinational educational sector (non-English speaking)

Evaluation Scan


 * 1) Partner and Peer Institutions
 * 2) Regional activities
 * 3) National climate
 * 4) International climate

Potentially include:


 * PEST analysis
 * SWOT analysis, see: 2010 Vision session notes 1

Phase 3: Retrospective Evaluation
Timeframe: Fall 2010

Retrospective Data Collection and Analysis


 * Retrospective evaluation of Open.Michigan: see Open.Michigan Data and Research Collection (private to Open.Michigan staff)
 * See: 2007-2010 CTools survey data on Open CourseWare.

Analysis of Peer Institution Evaluation Efforts

Including:


 * ISKME
 * MIT
 * CIC
 * Tufts
 * OLNet
 * OLI
 * TALL, Oxford
 * Wikiwijs

See: Preliminary Report of Research and Findings from 2011.

Abstract: This report is a preliminary review of the evaluations and metrics in current and past use by organizations that produce OER (Open Education Resources) and to a lesser extent OCW (Open Courseware), and their implications and recommendations for beginning a thorough review of Open.Michigan and its work as it relates to these items. The report is not intended to be a final analysis set in stone, but rather an open document availing itself to iterative refinement and addition – thus the “preliminary” label. The aim of this document is twofold: to take into account and provide a brief summary of evaluations and evaluative research methods performed by other institutes and organizations that provide, analyze, or in some other way align themselves with “open” content, and to provide a set of recommendations based on this research with which to guide an analysis of Open.Michigan. Though this definition will become more detailed over the course of the report, “analysis” of these programs and of Open.Michigan will for now be defined as referring to the production, impact, and external knowledge of the open materials created and distributed by these organizations, where “open” content refers to resources made freely available and openly licensed for adaptation and distribution online.

Formulation for future metrics

Goals


 * 1) Define parameters to be measured
 * 2) Define target values for those parameters
 * 3) Perform measurements
 * 4) Compare measured results to the pre-defined standard
 * 5) Make necessary changes

Phase 4: Strategy Formulation and Implementation
Timeframe: Fall 2010

See: Open.Michigan Strategic Vision

Phase 5: Evaluation and Control
See: Aggregate Data for Open.Michigan Evaluation Project 2010-2011 for initial results of this phase.

Timeframe: Winter 2011


 * Implement lessons from retrospective evaluation and strategy implementation

Qualitative and Quantitative Data-Gathering Tools and Techniques


 * Surveys (using Qualtrics)
 * Follow up interviews with select respondents
 * Google Analytics
 * Outputs from distribution channels (e.g. iTunes U, YouTube, Blip.tv, etc.)


 * CTools

Metrics and Evaluation
See: Evaluation and Metrics Planning Documents for outlines of the target audiences, survey questions as wells anticipated outcomes and mapped metrics.

Pre-Data Collection Steps


 * Clearly define the goals and objectives of the data collection [see above]
 * Reach understanding and agreement on operational definitions and methodology for the data collection plan [see outcomes]
 * Ensure data collection (and measurement) repeatability, reproducibility, accuracy, and stability

During Collection Steps


 * Data collection process follow through

Post-Data Collection Steps


 * Evaluation and control