Skip to: Navigation | Content | Sidebar | Footer

Evaluating Online Education: The Quality Scorecard


Online education options are not only increasing in popularity, but also encountering increased demands for evidence of academic quality. But what does it mean if a program is deemed to be of high quality? How can quality levels be measured? U.S. News recently issued its first ranking report focused specifically on online programs and provided details about some of the difficulties encountered in "making the best apples-to-apples comparisons" possible.

The discussion and related research efforts among educators to determine what “quality” means are ongoing. There are currently many models to choose from, including the 20+ resources collected by the University of Wisconsin-Madison's Distance Education Certificate Programs. There is no single agreed upon approach to assessing quality, but the Quality Scorecard for the Administration of Online Education Programs is one of the most recently development tools that strives to become the primary tool "used to demonstrate the overall quality of online education programs, no matter what size or type of institution."

An adaptation and expansion of the Institute for Higher Education Policy's 24 Benchmarks for Success in Internet-based Distance Education, the Quality Scorecard was made available by The Sloan Consortium last year. The evaluation instrument itself is the result of a project led by Dr. Kay Shelton that included surveys and feedback from an expert panel of more than 40 online education and administrators in higher education. The scorecard can be used by individual schools to identify strengths and weaknesses of existing online programs and to guide the development of new programs. While there is a fee involved in accessing PDF, print, and interactive online versions of the instrument, a full version is also openly available online.

 

Scorecard Components

 

The scorecard is in the format of a rubric with which a reviewer looks for evidence of 70 specific quality indicators organized into 9 categories. Each indicator is individually rated on a scale, which results in a total score indicating the program's status and need for improvement:

90-99% – Exemplary, little improvement needed

80-89% – Acceptable, some improvement recommended

70-79% – Marginal, significant improvement needed in multiple areas

60-69% – Inadequate, many areas of improvement needed

59%-below – Unacceptable

The categories on which a program’s quality is judged are summarized below. Some categories feature one indicator, while others have 10 or more.

  • Institutional Support: Governance, policies, and guidelines are in place at the institutional level, which address decision-making, student assessment, and intellectual property of course materials, and they define the value of online education to the institution and its stakeholders.
  • Technology Support: Technology planning and support mechanisms are established that cover issues of data security and back up, course delivery systems, and user training and assistance.
  • Course Development and Instructional Design: Standards for the process of course design and development, including specific types of materials that measure learning objectives and appropriate assessments, are employed.
  • Course Structure: Individual courses provide the organization of materials required for students to engage in and complete the courses, including elements such as grading policies, library access, communication and collaboration tools, and documents that are accessible via multiple platforms.
  • Teaching and Learning: The courses exhibit evidence of facilitated interaction among students and between students and instructor, as well as indicators of timely feedback, access to support resources, and strategies for developing online presence.
  • Social and Student Engagement: Course delivery methods include ways in which students can engage with each other as part of an online learning community.
  • Faculty Support: Instructors are not only prepared to teach online courses, but also have guidelines for what is expected of them and receive support throughout the process, including training and professional development opportunities.
  • Student Support: Evidence exists of evaluation of prospective student readiness for online learning, communication of student expectations and requirements, as well as resources in a wide range of areas (e.g. technical support, tutoring, ADA requirements, and advising) that support individual student efforts within a course and help them to connect to the larger program and institution.
  • Evaluation and Assessment: Processes are in place in which instructors evaluate student learning, students evaluate instructor performance and support services, and college-wide course and program level alignment and effectiveness can be assessed. Data collection and analysis are also featured in this category.
 

Potential Uses

 

What does the Quality Scorecard mean to you? This instrument can be implemented in multiple ways, and provides information that may be helpful for decision-making about online academic programs by many different stakeholders in online education:

  • Online Education Administrators: Designed with this primary group in mind, the scorecard can be used to assess current online programs to identify specific areas for improvement, and can be used to prepare for accreditation reviews.
  • Instructional Designers and Developers: Another important group that may benefit from the list of quality indicators, designers and developers can use the results of the checklist to revise existing courses and to guide the development of new courses. 
  • Instructors: Faculty members are featured in several categories of the scorecard. These indicators can help instructors understand expectations of online teaching, and guide their questions related to availability of training and support.
  • Students: The Quality Scorecard and other similar assessments are not intended for student use, but the results may be helpful to students who want to find out more about the programs they are already enrolled in or may be considering. Students should ask schools and programs about how they are measuring quality, whether it is with this evaluation tool or by another method.
  • Online K-12 Educators and Administrators: Eric Wignall's review of the Quality Scorecard included the suggestion that it could "with small edits, be used for secondary education programs, as well."

There are challenges ahead for the Quality Scorecard, as well as other quality measurement initiatives. Not all elements of review are completely objective measures. There are also some vague areas. One such area is the quality indicator that reads, "There is consistency in course development for student retention and quality." This kind of vagueness can be difficult to avoid and demonstrates how defining and assessing concepts such as quality can be complicated in nature. While the scorecard is designed for easy access and use, evaluations should only be conducted by experienced online education professionals.

The problems associated with measuring online education quality are not solved, but the Quality Scorecard does offer a way to assess multiple factors. What are your thoughts about this measure of online program quality? Let us know if you have experience with the Quality Scorecard or have suggestions for additional indicators that are important to you, but may not be addressed in the current rubric.

January 17th, 2012 written by Staff Writers

Facebook Comments

Bookmark the permalink.