Making academic program review a CLAS act
Bringing clarity to the academic program reviews
2023:4
A recurring theme in academic institutions is how to and how often to review the quality of academic offerings. Too often, reviews occur only when there is an external driver, such as licensure, authorization, or accreditation. If there are other periodic reviews, the structure is frequently an ad hoc mixture of curriculum, market trends, and external factors. In our experience in working with dozens of higher education institutions, program reviews range from trivial broad-brush examinations of curriculum to overly complex market studies.
In working with a number of institutions in several countries, it has become clear that program reviews need a consistent structure and tempo. The following model, which is termed “CLAS,” has been developed in a way to allow actional insights yet avoid the extremes of a trivial surface-level analysis or diving into the details of teaching that are best left to faculty.
Avoiding confusion with education market analysis
An academic offering, be it an entire degree program, a course within a program, or a stand-alone certification course, exists within a competitive context. The analysis of the competitive context is a key part of a program review but will look different depending on the type of program and its competitors.
For example, a degree program in many areas will have not only direct peer competitors but also a growing number of certificate programs that are substitutes. One study that we recently completed in healthcare identified over 100 certificates in healthcare-related topics. There is a similar plethora of offerings in information technology, management, and services occupations. Credential Engine, an aggregator of certificate and credential listings, lists over 967,000 credentials on offer in 2021 in the United States alone.
A labor market and competitor analysis is often helpful for a program review but should not be confused with the core purpose of a program review, which is more around the performance of the academic program at delivering intended outcomes or results. The skills, tools, data, and analysis methods needed for meaningful competitive market analysis are highly specific and best sourced from a specialized provider using specific data analytics skills.
Many of the program reviews we have supported have included a separate market analysis using the real-time labor market and trends datasets from Lightcast, which has an extensive labor market and education institution dataset. The data analytics skills needed to deliver a meaningful market analysis, however, are very different from the needed analysis in reviewing the effectiveness of the curriculum.
The CLAS Model
The CLAS program review model focuses on four areas: curriculum, learning interactions or Learningscapes™, learning assessment, and the scholarship of teaching and learning. The model is shown in the diagram below. The four areas clarify actionable strategies once the program review is complete.
The essence of program quality can be effectively examined by reviewing the key instructional elements of the CLAS (curriculum, learning interactions, and assessment/evaluation of learning). The CLAS model also includes a review of the scholarship of teaching and learning (SoTL) to gauge how an institution has focused on improving program design and delivery. Research can also be a part of the instructional elements in programs that contain activities such as a thesis, summative project, or dissertation but is distinct from SoTL.
CLAS described
The following discussion with provide a brief description of each of the areas of the CLAS model. Each area will be explored and discussed in future posts in the Student360.Report. Feel free to subscribe to that publication for updated content.
Curriculum
Developing an idea of what should be in a learning encounter can be described as a curriculum. Think of it as the architecture of an academic program or course. The curriculum serves as a plan for engagement in a teaching and learning process. As with any plan, it will not survive unadapted upon contact with the learning situation.
Typically, the elements are captured in five categories, as described below.
Purpose or aims. A program or course description that sets out the essence of the scope and rationale of the learning unit(s).
Learning objectives. Stylized statements of what will be learned. Objectives are often stated using language relating to knowledge, skills, or abilities that students will acquire.
Resources. Learning resources often include written materials – books, articles, readings – databases, software, etc.
Learning activities. An overview of how the learning will be delivered, such as through classroom lectures, assignments, group meetings, etc.
Assessment or evaluation. A plan for how the outcomes of learning will be determined.
There are many models for curriculum elements using rubrics, checklists, and components. In higher education, there are several well-known models of curriculum development. Those models include Backward Design, Understanding by Design, ADDIE, Community of Inquiry, Universal Design for Learning, etc. To name just two, Quality Matters and the Online Learning Consortium provide guidance for online course content.
The plethora of models illustrates many ways to develop the key elements in a curriculum. Almost any of the models can be used to unify and focus efforts. Ensuring consistency of approach to design is more important than picking the perfect model.
LearningScapes™
The idea behind the term LearningScapesTM is to describe the totality of how teaching and learning take place. The idea is similar to the servicescapes model first developed by Mary Jo Bitner in the 1990s, beginning with her article Servicescapes: The Impact of Physical Surroundings on Customers and Employees.
LearningScapesTM may seem deceptively simple but involves many layers to evolve beyond the servicescapes idea of the physical environment. A learning environment in today’s world includes the interaction of the teacher and student, the learning process, and an ever wider range of technologies available to facilitate learning. I have used this approach to examine the impact of place or space in change management. The presentation is titled Designing Place and Space as Elements of Effective Change and introduces the idea of 3PT for people, process, place, and technology.
As with curriculum approaches, many models guide the design of LearningScapesTM. Concepts such as problem-based learning, challenge-based learning, and experiential learning are often used to design learning interactions. A decision to adopt any one of these approaches will drive elements of curriculum design. It is important to ensure that the curriculum aligns with the approach to learning design.
Assessment
With the issuance of ISO 29992:2018, Assessment of outcomes of learning services — Guidance, additional clarity has been brought to the definition of assessment with the following definition:
“assessment
test, examination, observation, or other process, designed to measure an examinee’s knowledge, competence, or performance in a defined area against specific reference points or standards (educational or professional).” (ISO 29992:2018, § 3.1).
The first line of confusion avoidance is to determine the level of focus. Significant differences exist in evaluating the success of a course, an academic program, or an academic department or institution. Effective assessment plays an important but different role at each level.
Once the focus level is determined, different ideas can guide the tools and techniques used. Typically, differences develop at a program level between internal and external and at a course level with direct vs. indirect techniques. Assessment is also seen as formative or summative.
A recent edition of the differences was developed in the Student360.Report in the article The AI imperative - A clear need for systematic assessment.
SoTL - The Scholarship of Teaching and Learning
Reasearch supportive of an academic program has two primary dimensions. In research-based institutions, a valid and helpful focus is on advancing knowledge in the program's subject matter. Research output is often measured in numbers and types of publications in preferred journals.
A second type of research focuses on developing effective teaching and learning methods. This area was identified as one of four domains of scholarship by Ernest Boyer, who was the president of the Carnegie Foundation for the Advancement of Teaching at the time. In Scholarship Reconsidered: Priorities of the Professoriate – both the 1990 original and the 2015 update – a distinction is made between and among research based on whether it seeks to make an original discovery, apply existing knowledge, integrate existing concepts, or support more effective teaching and learning.
The Scholarship of Teaching and Learning, or SoTL, is often seen as underserved yet challenging. SoTL is an essential part of whether teaching approaches can improve and, very importantly, address the different needs of new generations of learners, new technologies, and new ways of learning. Perhaps we need no further examples than the turbulent changes and adaptations occurring during the COVID-19 pandemic. Institutions with a strong culture and practice of SoTL pre-pandemic would logically be better positioned to manage the various pivots and challenges.
Centers for teaching and learning often address the need for scholarship in this area. Institutions and programs are well advised to clearly integrate the work of these centers, where available, into instructional design efforts. The International Society for the Scholarship of Teaching and Learning is an umbrella organization providing support and thought leadership for the subject. Overall, SoTL should be seen as a key process and quality improvement method in education.
Using the CLAS framework
The CLAS framework provides a set of four containers or areas to ensure a full, complete, and systematic review process for academic programs. Embedded in this program review model are a curriculum review, analysis of the learning approaches used, clarity on the role and outcomes of assessment, and ways to use the information to improve.
The CLAS framework can be applied at an academic program level, to a set of courses, or even to an individual course. The elements will be changed depending on the level of analysis.
The CLAS framework is best not to be confused with the various types of market needs analysis or program success monitoring. Those areas are important companion analyses to a program review but require different data and analytical skills.
About the Ex4EDU.Report
This report is offered as a free-of-charge contribution by Lone Tree Academics, LLC, an educational services company focused on enabling excellence in higher education. Our services are focused on excellence-based practices for developing institution and program strategies, curriculum design, and engaging LearningScapes™.
For more information, visit www.ltacademics.com