After four years of collecting and analyzing data, the Queen’s Learning Outcomes Assessment (LOA) project is about to publish and share its findings with other universities across North America.

The project implementation began back in September 2013. That’s when nearly 2,000 first year students took a set of standardized assessment tests during their classes, and allowed exams and assignments from some of their courses to be studied. At the same time, a group of fourth year students were also sampled from the same departments using the same tests. Once the research team had these baseline result in hand, they then tested the first year group again in their second, third and fourth years. The goal was to measure and track four important student learning outcomes as they progressed through Queen’s, namely, critical thinking, problem solving, communication and lifelong learning.

“We started out simply wanting to find out whether or not we could demonstrate in a reliable and valuable way the growth of those skills over four years with our students, and if yes, by how much” says Jill Scott, Vice-Provost (Teaching and Learning). “We learned that without a doubt our students gain in skills and that Queen’s students were well placed compared against other schools in other jurisdictions such as the United States. Queen’s students started higher and ended higher which was a very positive result.”

But this wasn’t the only important focus of the LOA research project which was funded by and developed in collaboration with the Higher Education Quality Council of Ontario (HEQCO). It also aimed to study the effectiveness of the testing tools themselves.

“There are a number of different highly-standardized measurement tools out there and we wanted to see how they work and what each one can tell us about the student skills they are designed to measure,” says Dr. Brian Frank, Associate Dean (Teaching and Learning) and co-Principal Investigator along with Dr. Scott. “This part of the research project is valuable as it will support our assessment work and help us improve the educational experience, as well as helping other post-secondary institutions looking to incorporate some efficient and effective student learning assessments into their academic operations.”

From the start the project team used four main assessment tools, the Critical Thinking Assessment Test (CAT), the Collegiate Learning Assessment (CLA+), and the Transferable Learning Orientations Survey (TLO). The fourth tool was a set of validated rubrics called VALUE, which stands for Valid Assessment of Learning in Undergraduate Education, developed by the American Association of Colleges and Universities. These rubrics were different than the tests in that they applied to a selection of assignments the students completed as part of their regular classwork. Once the assignment or assessment was marked in the regular way, the project team brought in specially trained students to rate them again to see how the students performed against the VALUE Rubrics in such areas as critical thinking and problem solving. The results were then shared with the course instructors so they could see where their students gained a lot, as well as how their assignment aligned to the learning outcomes they wanted their students to achieve.

“We learned a lot of about what works in each of these tools and what doesn’t. We also found that one of the key factors to take into account is student motivation, as their enthusiasm and participation in these types of assessments can vary greatly. For example their desire to take part in the standardized tests drops steadily as they progress through university which makes it difficult to gather comprehensive data,” says Dr. Scott.

As a result, the next phase of this research project will not attempt to track the same large group of students as they progress through Queen’s. Rather it will focus on a cross section of students in first and fourth year.

“We believe the assessment and credentialing of skills and core competencies is critical for, and is an essential component of, a high quality post-secondary education system in Ontario,” says Harvey Weingarten, HEQCO President and CEO. “That is why we are working with forward-thinking institutions across the province like Queen’s to conduct innovative research to identify reliable and valid tools that can be used to measure and improve the teaching of skills and competencies postsecondary institutions say are the hallmark of a high quality higher education.”

Phase two is also being supported by HEQCO and will again be firmly focused on building on the success of the VALUE Rubrics unveiled in phase one. This time, the project will focus on encouraging interested faculty to work with assessment facilitators to design their courses and assignments from the start to align with the learning outcomes rubrics. One standardized test will be used in parallel for comparison.

“We found both our faculty and students cared a lot about ensuring learning outcomes are embedded in their courses and work. It’s why we will be putting a lot more effort and energy into promoting and studying the rubrics as there is evidence they provide more motivation to students and better information for our instructors,” says Dr. Frank.

In another difference, the project will also be taking a new approach by creating clusters of courses in related disciplines and putting them together to create an assessment network so faculty can work together to learn about the rubrics and how to apply them.

“Phase two is already underway and we are recruiting instructors across campus to take part in this groundbreaking research,” says Dr. Scott. “There is no doubt Queen’s is a leader in studying learning outcome assessments in Canada and we have very compelling results on all fronts, in terms of what our students learned and how a project like this works, which we are excited to share and continue to study.”

The phase one results will be published in early 2018 by HEQCO. You can also learn more about the project at Queen’s and ways to take part by visiting the Learning Outcomes Assessment Project website.