‘Transformative’: the threshold learning outcomes for science
Madeleine Schultz
A * , Daniel C. Southam B * , Mark Buntine B , Kay Colthorpe C , Susan Howitt D , Elizabeth Johnson E , Susan Jones F , Jo-Anne Kelder F , Sally Kift G , Wendy A. Loughlin H , Glennys A. O’Brien I , Simon Pyke J , John Rice K , Susan Rowland L and Robyn Yucel M
A
B
C
D
E
F
G
H
I
J
K
L
M
Handling Editor: Amir Karton
Abstract
The Science Threshold Learning Outcomes (TLOs) are a consensus set of academic standards for Australian university Science education. They were developed by Prof. Brian Yates and Prof. Sue Jones, supported by Dr Jo-Anne Kelder, during 2010–2011. The co-authors of this paper are key figures in Australian Science education, and in this manuscript, we have used a reflective semi-structured interview approach to describe the process of developing the Science TLOs and consider their subsequent effect on tertiary science education in Australia. This manuscript documents Sue and Brian’s impact on science curriculum through the lenses of leadership, community and practice. We have a twofold aim: first to draw lessons for harnessing consensus in scientific communities on the value and purpose of a tertiary science education, and second to celebrate the success of these influential and impactful leaders in our community. We demonstrate how work to develop the TLOs has transformed tertiary science education in Australia.
Keywords: employability, higher education policy, leadership, Learning and Teaching Academic Standards project, science education, student‐centred, threshold learning outcomes, undergraduate curriculum.
Preface
This work is a participatory retrospective derived from transcripts of interviews conducted between the corresponding authors and the remaining authors. Illustrative block quotations are provided throughout this paper as critical reflections on the contextual factors that drove thought and action at the time or since.1
Introduction
The higher education landscape in Australia has evolved extensively from the 1940s, when it was an exclusive domain with a small, relatively homogeneous student profile compared to the present. The establishment of new higher education institutions and abolition of student fees during the 1960s and 1970s drove an increase in participation rates that increased further during the 1980s and 1990s as a number of providers merged and reformed as comprehensive universities. Fig. 1 illustrates key developments on this timeline and the corresponding increase in number of students in higher education.2 Increased demand for, and steady growth in, providers of higher education can be observed up to the year 2000, when the sector was opened up to private providers, and a dramatic increase in student numbers occurred.
Headcount of higher education students in Australia from 1949 to 20212 highlighting the impact of sectorwide policy changes on enrolments and the place of the Learning and Teaching Academic Standards (LTAS) project and commencement of the Tertiary Education Quality and Standards Agency (TEQSA) among these landmarks.
The 2008 Bradley review3 examined how a holistic further and higher education sector would meet the future needs of the Australian community and economy. Pressure in the employment market and changes in Australia’s economic profile were captured in the review. The review had wide-ranging recommendations, including a focus on resourcing, governance and regulation. In particular, it recommended that participation rates, especially for students who traditionally would not go to university, had to increase to meet these demands. The resulting policy paper4 anticipated a series of impacts on legal acts and frameworks relating to financial support for universities,5 standards6 and the structure of all certificates, diplomas and degrees.7 Thus, this review was a precursor to the need for academic standards at a disciplinary level.
The main consequence for universities was the removal of caps or quotas for Commonwealth-supported places for Australian students, and the introduction of a demand-driven system. Concomitantly, there was and remains a strong political perception that science would have a central role in finding solutions to societal challenges, and that a scientifically capable workforce would be crucial.8
As student numbers increased, the diversity of the student population expanded and private providers entered the market, the Bradley Review3 asserted a need to assure the quality of tertiary education. This led to the implementation of the Higher Education Standards Framework6 and establishment of the Tertiary Education Quality Standards Agency (TEQSA), which began work in 2011 and assumed regulatory power from 1 January 2012.9 This body and the enabling legislation identified a need to establish reference points for quality assurance of curriculum across the higher education sector.
So there’s government regulatory standards now with the Higher Education Standards Framework. There’s the Australian Qualifications Framework, and that requires that students know, understand and be able to do a range of things which are often ignored. [Interview 11]
The Learning and Teaching Academic Standards (LTAS) project,10 commissioned in 2009 by the Commonwealth Department of Education, Employment and Workplace Relations and conducted through the Australian Learning and Teaching Council, was initiated in response to the anticipated establishment of TEQSA, which had created some concern about what impacts such a body might have on universities.11 The focus was to intentionally craft a series of standards that could be used in benchmarking and accreditation activities, allowing TEQSA to verify that quality assurance processes met its legislative standards. LTAS established an approach that was trialled in six initial disciplines before being extended to nine cognate disciplines, including science, during 2010 and 2011. The aim common to all discipline scholars was to develop discipline-specific Threshold Learning Outcomes (TLOs) that would not only permit benchmaking and accreditation but also act as the basis for institutional curriculum decision-making and inform students about the overarching goals of their studies.12
I still remember Christine Ewan [LTAS Project Director, Australian Learning and Teaching Council] saying you’re either on the train, or on the tracks and their whole […] messaging of that national project was TEQSA is coming and the fear was big, big toothy tiger. If we don’t say what we expect of a degree, they’ll tell us and we don’t want that. [Interview 3]
So for the threshold learning outcomes, the idea was to try to get this tight set of outcomes statements articulating what students need to know, understand and be able to do as a result of discipline learning via a consensus generation process facilitated by the Discipline Scholars working with the broad disciplinary community… [Interview 11]
The approach of the broader LTAS project mirrored the government intention that the ‘discipline communities will “own” and take responsibility for implementing academic standards’ [p. 324] and thus set forth to create projects for cognate disciplines, each headed by Discipline Scholars. Sally Kift reflected in her interview about the group of people appointed as discipline scholars.
The Discipline Scholars were leaders in their fields – academic discipline leaders – but also each having a strong learning and teaching background or bent as well. So we all applied, were all interviewed and then subsequently appointed. And I just have to say it was glory days. It was just an astounding experience – a remarkable group of colleagues who met regularly to discuss process and learn from each other, led by Christine Ewan getting us all organised with her immense learning and teaching leadership expertise.
After competitive tender, the Australian Council of Deans of Science (ACDS) endorsed Prof. Brian Yates and Prof. Susan Jones as co-Discipline Scholars for Science, and they appointed Jo-Anne Kelder as Project Officer. The group embarked on a nationwide tour to harness the collective wisdom of science academics and develop a consensus on what a student should know, understand and be able to do on completion of an undergraduate science degree. The details of the consultation process are available elsewhere,13 and several co-authors of the present paper took part.
Through the consultation process, Sue and Brian drew together the set of five statements (each with second tier elucidations) that form the Science TLOs (STLOs)14:
STLO 1. Understanding science: demonstrate a coherent understanding of science
STLO 2. Scientific knowledge: exhibit depth and breadth of scientific knowledge
STLO 3. Inquiry and problem solving: critically analyse and solve scientific problems
STLO 4. Communication: be effective communicators of science
STLO 5. Personal and professional responsibility: be accountable for their own learning and scientific work
These outcomes have since been applied in curriculum and assessment design, and were further elaborated in sub-disciplines including Chemistry as described below. The Chemistry TLOs have been adopted as the basis for Royal Australian Chemistry Institute (RACI) Accreditation.15 Fig. 2 illustrates the relationships of the organisations and outputs of this process.
Methodology
The research adopted a participatory design16 to engage with community leaders who had lived experience of the national development and implementation of the Science Learning and Teaching Academic Standards.14 There were two phases to the research design. In Phase 1, participants were asked to reflect through a semi-structured interview17 on Brian and Sue’s co-leadership of these processes and the impact on science curriculum since. In Phase 2, participants were invited to contribute to co-authorship of the present paper through refinement of analysis, interpretation and narrative surrounding their experience.
Volunteer participants for a semi-structured interview were recruited from a purposive sample18 of known leaders involved in the original Science Learning and Teaching Academic Standards or related extension projects. Thirteen recruits consented to an interview and to be co-authors on this paper. The interviews typically lasted between 20 and 30 min, and were audio recorded after informed consent was obtained. The interview questions were designed to elucidate critical reflection on experience1 as follows:
What was your role in the Science Learning and Teaching Academic Standards (LTAS) project and the development of threshold learning outcomes (TLOs) during the early 2010s?
Can you describe working with Sue and Brian and their contributions to the project?
In what ways have you continued involvement with LTAS and TLOs since then?
What is your view on the impact of the TLOs on tertiary science education in Australia?
These questions served to guide discussion of participants’ experience, perceived value of the TLOs and personal development as a result of participating in TLO development and use, as well as reflection on the role Brian and Sue played. The audio recordings were transcribed verbatim and the transcript was sent to the participant for member checking to ensure trustworthiness.19 Participants could withdraw consent up until return of the transcript or remain anonymous until submission of this manuscript. Those participants named as authors herein did so with full informed consent and recognition that their contributions might be reidentifiable.
The transcripts of interviews were initially parsed for themes through a hybrid coding approach.20 A priori deductive codes derived from literature relating to transformational leadership in higher education21,22 were compared with inductive codes identified from the transcripts and comments during analysis to arrive at a preliminary codebook. The codes were confirmed by unattended semantic mapping23 of the transcripts (using Otter AI, see https://otter.ai) and the relations between emergent terms were visualised in a concept cluster map. These codes were applied to the transcripts broken into coarse units of analysis by identifying natural breaks in the conversation. The related quotations were used to develop initial narratives and were provided to participants, who helped refine them through the process of co-authorship.
Results
The note-keeping during interviews identified that the contributions to the TLOs of both Discipline Scholars Sue and Brian could not be easily disentangled. Thus, we do not differentiate between their contributions and impact. Preliminary coding reinforced the unsupervised concept map23 in Fig. 3.
Concept map of the main themes (spheres), codes (points) and their relationships (lines) derived from the verbatim transcriptions of the semi-structured interviews and generated by unsupervised semantic analysis.23
Our analysis identified four main themes, aligned to key concepts (bold below):
Students and their needs for a science curriculum that supports their development of knowledge and skills that are appropriately and adequately assessed.
People working on the project, including those appointed to author good practice guides or those invited to work on sub-disciplinary standards like chemistry.
Brian and Sue as leaders, their conduct of the project and the process to define the threshold learning outcomes, including in sub-disciplines like chemistry. In the discussion below, this grouping has been included with ‘People’.
Outcomes necessary to establish a new science curriculum and define educational standards, both for broad and narrow disciplines.
We explore the themes of students, people (incorporating Sue and Brian’s leadership) and outcomes in the discussion below. Representative quotes from interviews are heavily used to illustrate each theme. In most cases, these are presented without elucidation because they require no further explanation.
Discussion
Students
The political conditions in the higher education sector at the time the LTAS projects were initiated provided a timely opportunity to fundamentally redefine the way curricula are constructed and reviewed. Until this point, most science curricula focused on disciplinary content knowledge. Transferable skills were often not given explicit focus in the curriculum, with students thought to somehow pick them up ‘by osmosis’ throughout their degree.
There was a time when the university undergraduate science curriculum was designed to train scientists and it was fit for purpose, because most of the students prior to the massification of universities, prior to the 1990s, most students did come in with the reasonable expectation that they would practice science. And so the curriculum has kind of been stuck in that paradigm for a while […] it’s not fit for purpose any more. [Interview 6]
This paralleled a growing focus at the time on improving teaching quality in higher education24 with a greater emphasis on what the student does.25
Certainly, it’s the first time that I had ever been part of a project, at that scale, where the student-facing outcome was actually the thing that they were really trying to achieve. [Interview 10]
Likewise, the project was done at a time where graduate outcomes were gaining increasing resonance across the sector.
I think the first and obvious [impact of the TLOs] is that idea of describing what a graduate should know and be able to do, so lifting us up beyond knowledge and technical skills, even though that’s critically important, but looking to the transferable skills: what else could a graduate know and be able to do? [Interview 2]
At the same time, the federal government was pushing that university degrees, science degrees in particular, should be about more than just producing narrow discipline specialists, that students […] graduate with what we call the generic skills, the ability to actually use what you know, rather than to just know a lot of stuff. [Interview 7]
The TLOs, similarly to institutional graduate outcomes, require more than declarative knowledge, as a signal to students of what they should value and achieve as a consequence of studying.
Because you know, you’re trying to say to students we don’t want you to just rote-learn stuff. We want you to think about it. [Interview 4]
As participants, we noted the process of writing the TLOs was the first time students had been the focus of curriculum discussions. Sue Jones as co-Discipline Scholar reflected that the design of the TLOs aimed ultimately to support student learning.
Because there’s always the risk that if people start seeing them as simply things that have to be ticked off on the university level, it won’t work, which is one of the reasons we worked so hard to [focus on] how these things can actually help us design teaching that actually supports student learning. So the students know what they’re learning, why they’re learning and what they need to achieve. [Interview 1]
However, we also note that the student focus of curriculum and teaching can slip without a champion, especially during a crisis (e.g. COVID-19). The quote below underscores how curriculum, and our intentions for curriculum, need revisiting and reinvigorating on a regular basis.
So [prior to the TLOs] it was all about, you know, not even what students will be able to do, but what we have told students, and so it was very, very teacher-centric. And that was one of the big things, moving towards the Threshold Learning Outcomes was a really significant shift for us moving towards a much more student-focused construction. And that has largely survived. It’s not quite as robust as I would like it to be. There’s good reasons for that. The last 5 years has been fairly chaotic. And then you lose coherence in those kinds of activities, unless you’ve actually got somebody who’s banging the drum all the time. [Interview 10]
People
All interviewees had positive things to say about Brian and Sue personally, about interacting with them and about their skill in facilitating what was an enormous effort to bring the diverse science community together and reach consensus on a small set of TLOs. Among their positive characteristics, Sue and Brian share a student-centred focus that permeates their work. This ethos, which underpinned the process of writing the TLOs, was noticed by the participants.
Sue and Brian are actually […] as a working pair, they were just fabulous, because they brought really different things to the table. Sue was very focused on the student and what the daily life of the student is like and what they need. It’s not that Brian didn’t care about that, but he was the more intellectually focused – “How do we actually create this framework so that it can actually be useful for a really wide group of people?” [Interview 10]
Many participants reflected on the importance of the diverse group that was involved in drafting the Science TLOs. Brian and Sue were able to cultivate an environment that invited input from a wide spectrum of science academics. A significant part of their success was having excellent reference and advisory groups supported by a project officer who helped shape the project.
It was just a massively amazing experience of working with people who were not about ego. We’re about students, we’re about the quality of science education, because that’s the input that does quality science research. They all really lived the nexus between research and teaching. They all thought about it so deeply and broadly. And I think you could say the bringing together of all those people regularly to have those really important discussions – it meant that the LTAS science statement was very effective. [Interview 3]
There was an advisory group of just four discipline experts. I didn’t work with them, but I canvassed the people to go on these reference groups, the ones that you knew were passionate and inspired and would do a good job. [Interview 7]
And it was the right group of people, they picked the right people[…] And it worked. It would have come out with a different outcome if it had been a group of people that weren’t as cohesive. And I think that, again, there’s an element there of [Sue and Brian’s] capability. [Interview 10]
As can be seen in the quotes below, interviewees perceived the overall success of the LTAS project for Science to be partly due to the skilful way that Brian and Sue collected and collated information from the community.
It created a national community that brought together what we call the sub-disciplines. So you had biochemists and biologists and mathematicians and chemists and all of them, earth sciences, they all were in the room together. That was amazing. Really, really, really galvanised a lot of people, broke down some silos, and actually, I think transformed people’s understanding of the value of learning outcome statements as a scaffold for designing curriculum, an element in your curriculum design. [Interview 3]
Brian and Sue were […] the power they had for creating this from nothing, they were very consultative, they listened, they took issues on board, they heard points that are raised, and then they were able to show and demonstrate how those points were encapsulated within various TLO statements. [Interview 8]
A lot of other people – maybe they would want to push a particular ideological line or had ideas of their own about how these things should work. That limits the amount of participation you get. Brian and Sue are much more consultative. But along with that, they’re not just consultative, they do have, I think, a terrific way of providing an intellectual and conceptual framework for what they want people to do that people can understand and get behind. [Interview 7]
I always found on discussing these things [teaching and learning] with them to be very rewarding. They’re quite visionary. They’re quite passionate people – I mean Brian in his quiet way was – he has a deep passion for this stuff. And always had good ideas, better ideas than mine in a lot of cases, like the extent to which they went around consulting people. The number of workshops around the country – it was a punishing schedule that they did. [Interview 7]
So I think that one of the things that was particularly powerful – not the right word, but useful in that sense, was actually the range of people that Brian and Sue actually did engage in that advisory, that reference group. [Interview 10]
The character of the people going around to gather the information and the commentary was very conducive to getting good input from whoever wanted to make points. It was a very welcoming situation. And that’s what we really needed, especially as this was effectively about the teaching and learning part of our science practices. At that time, it was mostly women [involved in teaching and learning]. We were mostly lecturers and maybe senior lecturers, you know. Very few of us had gone any further than that at that time. So it was really good to have people who could open up that discussion and treat us all with having valuable commentary to make. [Interview 13]
Outcomes
The process of consensus building was attributed as part of the reason for the success of the LTAS Science project and the enduring nature of the Science TLOs.
Because of the work that Brian and Sue did in the first place, there was a lot of consensus building. And so one of the things that it’s pretty difficult to refute is that, you know, we’ve been to every institution in Australia, we’ve canvassed academics from a wide range of disciplines, and this is the distillation of what those people thought – it’s pretty hard to actually come back and go, yeah, actually, I disagree. So I think it was the extent of the consultation, and the depth of the consultation is probably the reason why it’s such a solid piece of work. [Interview 10]
The outcomes of the LTAS Science project extend far beyond the set of statements of TLOs. It forced the community to engage deeply with the vexed question of what makes a science graduate, and whether a person who is highly skilled in some aspects but missing others qualifies for a BSc.
I think it really challenged people’s assumption that a science degree was for the people who become scientists. So it shifted from saying, “We need to cater to the ‘high distinction’ students who are going to be our PhD and our Honours students” […] at a minimum, this is what we want them to be able to do. Again, acknowledging that some of their brilliant ones won’t be able to do the communication one quite as well as we might want. [Interview 3]
I think it’s extremely valuable to be able to hold these TLOs up. People will often say: “The students don’t know how to x”, right? “Students don’t know how to solve a problem.” “The students don’t know how to ask a question.” One of the great powers of these outcomes is to be able to hold them up and go, “Yeah, that’s right, they don’t, and this is something you need to teach them.”[…] So we have a responsibility. And I agree that there’s been a lot more acceptance over the last 10 years of the idea that a curriculum should explicitly help students do inquiry, should ask them to communicate, should ask them to take responsibility and reflect on their place in the world as scientists, which was not there, I would say, in any great detail, or was certainly not considered to be serious science before these outcomes came out. [Interview 5]
And so we’re starting to talk a lot more about program outcomes, program level students and what we want our graduates to be like, to do – and that’s a little related to the university ones, which can be broad or vague, and relate much more strongly to the TLOs, the Science TLOs. [Interview 9]
The following narrative addresses how the TLOs have been used in practice, how they have been adapted for use in chemistry and educational research derived from a focus beyond disciplinary content knowledge.
The most significant benefit from TLOs was a nationwide conversation on their implementation into practice. Each interviewee described some examples of how they perceive the importance of and implement the TLOs in their own teaching practice.
I think the TLOs legitimise the idea that there’s more than just teaching students the facts. [The TLO project] is a slow-burn change agent, but it’s given people permission to do something more than just require students to memorise a bunch of facts. That’s no longer acceptable. And that’s very positive. To me, helping people learn to teach communication in science is very interesting, as is helping students learn to communicate. [Interview 5]
[I use the TLOs] to value-add to my curriculum design, so I use them in lots of different ways actually. [We implemented] a particular task around feedback – students giving peer feedback, and I thought, “Oh, this would be really cool because it really shows how I embedded some of those communication and also the nature of science into my course design.” So these students can develop their feedback literacy but also develop their communication. [Interview 9]
Once the TLOs had been established, it was important to build best practice around their application in teaching and assessment.13 This was supported through the publication of Good Practice Guides for each TLO, modelled on similar guides that had been published in the discipline of Law.26 There was a tender process for authors of Good Practice Guides, and several of our interviewees were authors of one of these.
The author reflections on the writing of the Good Practice Guides for the TLOs captured the influential stage of formalisation of the TLOs in terms of a step change that became transformative. During the development of the Good Practice Guides, Sue and Brian were instrumental in leading interdisciplinary author meetings and maturing the conversation on integration of the five TLOS with each other.
The first Good Practice Guide27 on the ontological and epistemological origins of scientific knowledge and practice demonstrated practical ways to introduce this important element into curriculum.
I realised that was a missing thing, that students were graduating from a science degree with not much understanding of “What is science?”, “What is not science?”, “How do you tell science from pseudoscience?”, “How do you tell good science from bad science?”, “You know, do we have certainty in science?”, “Do we prove our theories?” All of those kinds of more meta perspectives on science, it’s coming through loud and clear that that was something that students are kind of missing. [Interview 6]
We believed at the time, and I still do, that we teach and assess [TLO 1] extremely badly. That students, because they’ve had that type of assessment throughout their career, they tend to think that science is black and white. And in fact, in reflecting assignments, we’ve had a lot of students saying, “I like science, because it’s always the right answer.” And so then part of the main thing in that course was to try and get the idea that scientists are people and they make mistakes, or they’re passionate so they do something that’s, you know, a bit over the top just because they really want something to be true. Some things aren’t known yet. [Interview 4]
The importance of scientific knowledge and practice, through disciplinary content28 and inquiry,29 remains central to the notion of science curriculum encapsulated in the TLOs. The predominance of these outcomes coming into equilibrium with previously hidden elements of curriculum allowed for development of communities of practice, supported by the ACDS.
…emphasising where you actually got these critical thinking and problem solving-type skills. So it really, really, I think, created quite a vision for people of what the [ACDS Teaching and Learning] Centre should be doing and helped the Centre along, that it actually had such a well-defined and worthy goal. [Interview 7]
For example, communication30 presented a challenge of conflicting goals – is the outcome to prepare students to communicate science to scientists or to the general public?
Where does the threshold lie? I think for a student as an undergraduate, there are a couple of key things. Aristotle’s rhetorical triangle, ethos, pathos, logos – I think that’s one of the thresholds. The idea that it’s not just about the facts, and it’s not just about the emotion, and it’s not just about presenting it logically. You have to have all three. I think another threshold is around the idea that you should work to what your audience needs, as opposed to what you want to communicate. And then a third is thinking about the idea that [you] need to use different genres, depending on your audience and your purpose. I think those are the intellectual thresholds for students. [Interview 5]
Similarly, personal and professional responsibility31 was a new element brought to science curriculum, which was possibly hidden within curriculum and now had greater emphasis through the TLOs.
…personal professional responsibility, it was actually something that was perhaps not so called out and transparent and overt in the curriculum and in the people’s approaches at the time, because […] it’s independent and self-directed, I think we were all sort of doing that. But working effectively, responsibly and safely was something that was emerging and getting more prominent in certain individual team contexts. [Interview 12]
Publication of the Science TLOs inspired development of learning outcome statements in a wide range of science sub-disciplines and allied disciplines (collated on the ACDS website32). Collaborative development of these statements drew heavily on the consultation approach used for the Science TLOs, further amplifying the national conversation. The resulting statements largely retained the core of the Science TLOs as demonstrated by development of the Chemistry discipline outcome statement. The Science TLO project also influenced and inspired follow-on curriculum projects, for example Les Kirkup’s national fellowship work on inquiry-oriented learning, which drew on the Science TLO statement and the activity that grew up under its umbrella.33
The last great frontier – one that remains to this day – is assessment of the TLOs.
The bit we didn’t get down to as much was assessment and defining assessment. But we talked a lot about good practice. [Interview 2]
…what for me has always been that the tricky part, which is assessment, so it’s very easy to say, a student will know, will be able to do X, Y, Z, but how do we check that? [Interview 11]
…we need tasks that are authentic to what students will use that degree for. [Interview 6]
Within Chemistry, an OLT project ‘Assessing the Assessments’ was funded that aimed specifically to address the ability of assessment tasks to allow students to demonstrate the TLOs. Again, a consultative process was adopted and many people who had been involved in developing and refining the Chemistry TLOs (described below) took part. A tool was developed and applied, which can be applied to any assessment task to test its fitness for purpose in addressing and assessing each Chemistry TLO.34,35
Many members of the Australian Chemistry community took an early and active interest in the development of the Science TLOs. This was partly because Brian himself is a chemist, but also because other chemists working in teaching and learning engaged with the Science TLO process, including Mark Buntine, then President-elect of the RACI. He chaired a working party for Chemistry that was formed as part of the LTAS project during 2010. This group facilitated a discussion workshop in early 2011, resulting in the first draft of the Chemistry TLOs being published together with the Science TLOs as an Appendix to the Science Learning and Teaching Academic Standards Statement in 2011.14
By definition, the Science TLOs are written in a generic way that is discipline-agnostic. They need to be. They should be. But a lot of people needed some discipline examples to see how it could work. I was listening to [Brian and Sue], the power that they brought to that project was listening and incorporating feedback and not dismissing anything. And I took out of that the need to give it a discipline-specific, concrete example. [Interview 8]
The Chemistry TLOs were then further articulated, including a third tier of detail for TLOs 2.1 and 3.3 (Table 1), through a process facilitated through the Chemistry Discipline Network (ChemNet).36 This process paralleled that used for the Science TLOs, with extensive consultation within the discipline through a series of workshops and national meetings supported financially by the RACI and ChemNet.37 Slight revisions have been undertaken over the past 10 years and the current version of the Chemistry TLOs, being used for accreditation by the RACI, is provided in Table 1.38
| On completion of a bachelor degree with a major in chemistry, graduates will: | |||
|---|---|---|---|
| CTLO 1. Understanding chemistry. Demonstrate a coherent understanding of science by: | 1.1 | Recognising the creative endeavour involved in acquiring knowledge, and the testable and contestable nature of the principles of chemistry | |
| 1.2 | Recognising that chemistry plays an essential role in society and underpins many industrial, technological and medical advances | ||
| 1.3 | Knowing and being able to articulate aspects of the place and importance of chemistry in the local and global community | ||
| CTLO 2. Scientific knowledge. Exhibit depth and breadth of scientific knowledge by: | 2.1 | Demonstrating and applying the principles and concepts of chemistry | |
| 2.2 | Recognising that chemistry is a broad discipline that impacts on, and is influenced by, other scientific fields | ||
| CTLO 3. Inquiry, problem solving and critical thinking. Investigate and solve problems in the chemical sciences by: | 3.1 | Synthesising and critically evaluating information from a range of sources using traditional and emerging technologies and methods | |
| 3.2 | Formulating hypotheses, proposals and predictions and designing and undertaking experiments | ||
| 3.3 | Applying recognised methods and appropriate techniques and tools, and being able to adapt these techniques when necessary | ||
| 3.4 | Collecting, recording and interpreting data and incorporating qualitative and quantitative evidence into scientifically defensible arguments | ||
| CTLO 4. Communication. Be effective communicators of chemistry by: | 4.1 | Presenting information, articulating arguments and conclusions, in a variety of modes, to diverse audiences, and for a range of purposes | |
| 4.2 | Appropriately documenting the essential details of procedures taken, key observations, results and conclusions | ||
| CTLO 5. Personal and social responsibility. Take personal, professional and social responsibility by: | 5.1 | Demonstrating a capacity for self-directed learning | |
| 5.2 | Demonstrating a capacity for working responsibly and safely | ||
| 5.3 | Recognising the relevant and required ethical conduct and behaviour within which chemistry is practised | ||
| 5.4 | Demonstrating a capacity for working effectively in a cooperative environment | ||
One important outcome of the adoption of the Chemistry TLOs as the basis for the RACI accreditation process is that they have effectively become a tertiary Chemistry curriculum framework.15
My experience has been with the chemistry TLOs – coming into institutions where I’ve been and done accreditations – that they influence course learning outcomes, and therefore they influence subject learning outcomes. It’s been a bigger influence in chemistry, because it’s tied to the accreditation now. [Interview 13]
Educational research centred on the deliberate application of the TLOs in practice has allowed sharing of scholarly best practice in new and creative ways – in part as a consequence of the approach Sue and Brian took.39 Importantly, the TLO development process and the TLOs themselves provided space and inspiration for science academics to develop their own careers and research streams.
I remember going to going to the workshop, and I remember Brian and Sue just asking questions, and they never really gave an opinion. It was my first introduction to a consultative process in science. It was very inspirational to me. At the time, I was thinking about how you could be this kind of researcher, because it was a research process. It was quite illuminating for me, looking at the way they worked. [Interview 5]
All of our work around communication stems from the TLOs. I’ve built an entire research stream around this communication question. I’ve seen other colleagues who’ve decided to build their research career around inquiry and problem solving, for example, or around understanding what it means to be a scientist. Different areas of the TLOs pique different people’s interest. I think it’s sparked a lot of scholarship of teaching and learning careers more generally. [Interview 5]
Jones40 discusses the challenges and possibilities of assessing the science knowledge of university students (TLO 2). The perils and pitfalls associated with assessing student learning outcomes in science education are explored. Colthorpe, Zimbardi et al. focus on peer feedback in a ‘journal club’ for undergraduate science students, which aims to develop oral communication and critical evaluation skills (TLOs 3 and 4). The authors utilise threshold learning outcomes to guide the design of the journal club and enhance student learning.41 They subsequently explore the progressive development of scientific literacy through inquiry-based biomedical science curricula, highlighting the role of assessment in achieving learning outcomes.42 Mercer-Mapstone and Matthews43 investigate student perceptions of communication skills in undergraduate science education and how they align with the desired learning outcomes (TLO 4). Lastly, Schultz44 presents a position paper on teaching and assessing ethics and social responsibility in undergraduate science education, emphasising the importance of incorporating ethical learning outcomes into the curriculum (TLO 5). Overall, this research demonstrates the influence of TLOs in guiding the design, assessment and evaluation of science education at the university level.
Impact of the TLOs
The interview partners have all worked at high levels within the Australian academic system and had a consistent view of the importance of the Science TLOs.
From my point of view, they were and are truly transformational. I think the first and obvious one is that idea of describing what a graduate should know and be able to do, so lifting us up beyond knowledge and technical skills – even though that’s critically important – but looking to the transferable skills: what else could a graduate know and be able to do? Other things it did: it galvanised and made space for a whole generation of teaching and learning leaders. [Interview 2]
I’d say it would be profound […] the fact that the teaching and learning leadership is looking to refresh them is because they’re still regarded as being valuable and relevant. So I’d go as far as saying that Brian and Sue’s work was transformative in the impact that it’s had on our higher education science curriculum. [Interview 8]
I think that what they provided for us was a language to discuss the things within and across institutions in a way which we hadn’t really before. That was a really big impact. Even if we don’t necessarily talk about a specific TLO that we’re trying to achieve, the fact that I can still discuss how students need to develop communication skills around different audiences and in different modes to my colleagues, having articulated that through the TLOs allows me to use that language to describe for other colleagues that might not necessarily engage with formalised TLOs. [Interview 9]
Oh, transformative I would say; I mean, it was the first time an exercise like that had been undertaken […] they’ve had a remarkable impact would be my sense. And to example the how, what and when, as a Deputy Vice Chancellor, I would routinely send out compilation copies of all of the discipline standard statements to the disciplines when they were considering curriculum and doing reviews, and say, please make sure that you’ve had regard to these, noting the commonality across disciplines also… [Interview 11]
I think what it did was crystallise what we had been doing well, and then actually making it important too, because I think the communication and the professional responsibility piece was something that was a bit embryonic. And people are starting to get it – that we can’t just all sit and do science, we need to actually make that current and relevant, and how that works. If people aren’t actually overtly saying, “Hey, I’m doing this Threshold Learning Outcome”, the culture shift has been there. The threshold learning outcomes have actually shifted the culture of the way people in science work. [Interview 12]
Since the inception of the Science and Chemistry TLOs, integration and alignment of the TLOs with accreditation processes has improved. This has served to further strengthen partnerships between higher education institutions and professional bodies, such as the RACI, to ensure degree programs curricula are of a high-quality and professional standard.
Detailed evidence of the impact of the Science TLOs on the Bachelor of Science degree at Australian public universities was recently collected45 and it was found that most institutions are using the Science TLOs for quality assurance, curriculum design and renewal, and to prompt deep discussion among staff. This provides quantitative confirmation of what our interviewees reported.
Revising the TLOs
The Science TLOs are now 12 years old and have been influential within the Australian higher education sector throughout this time, as described above. However, as was the case for each of the disciplines, the intention when they were developed by Sue and Brian was that they should be a living document that would keep pace with, or foreshadow, developments in curricula and university strategic directions.
So from a good practice perspective, and thinking flexibly, each standards statement was articulated with a genuine desire to ensure they were not anchored in time, so that they could be applied flexibly and adapted as disciplines evolved. But it was always the intention, I think, to come back and redo them. [Interview 11]
In 2022, the ACDS initiated a project to review and update the TLOs. There has been uniformly positive feedback about the value of keeping the TLOs up to date and recognition that change is timely. For example, science degrees have a greater emphasis on employability skills and workplace health and safety than previously and it is appropriate to recognise this. Indigenous knowledge is also now included in an increasing number of science degrees.
I think it’s really healthy that at the moment, the Science TLOs are actually undergoing a revision, because they’re getting on, in 2023. They’re over 10 years old. The educational climate has changed, our student base has changed. The fundamental concept of learning outcomes remains absolutely critical to producing top quality graduates where we can assure the quality of their degrees. I think it’s really healthy that the TLOs are being tweaked. [Interview 1]
I think the bigger issue is […] that we were just silent, a bit over a decade, about Indigenous knowledge systems, cultural capability and understanding of non-Western ideas of science. All of that stuff is absent. [Interview 8]
Brian and Sue have embraced this update enthusiastically and emphasised the value of consultation and collegiality in the development of the original TLOs. They have also provided useful background on the original project and the choices made about what to include and how to express it. Although it was not possible to reproduce the extensive meetings with universities that they organised in the development of the original TLOs, consultation has included science education leaders and discipline groups. It is a testament to the strength of the original project that the basic structure of the five TLOs remains unchanged; what has changed is the second tier that explicate each of the five. We hope that the new TLOs will continue to provide a structure and guidance for curriculum and professional development.
Conclusion
There is no doubt that the LTAS project and the Science TLOs have had an enormous impact on tertiary science education in Australia. We have documented how a focus on students and the outcomes needed for their success in science was novel for the time, and placed science at the forefront of curricular change in the intervening period. We observed that the process to create these outcomes by harnessing a consensus through active listening and cyclic iteration was a tangible reason for the success of the TLOs, their implementation into curriculum and consequential impacts on student learning. The community that grew around this project has been sustained through support from the ACDS and outlets for scholarly work. On their retirements, Sue and Brian leave a powerful legacy that will continue to affect university students in decades to come. We hope this recount of Brian and Sue’s leadership that nucleated a scholarly community of practice serves as a useful model for others in future.
Dedication
We dedicate this manuscript to the fond memory of our colleague Les Kirkup who is sorely missed.
Acknowledgements
We acknowledge the Traditional Owners and Custodians of countries where we work and live and pay our respects to the Elders – past and present – and through them to all Australian Aboriginal and Torres Strait Islander people.
References
1 Mezirow J. On critical reflection. Adult Educ Q 1998; 48(3): 185-198.
| Crossref | Google Scholar |
2 Department of Education. Higher Education Statistics. 2022. Available at https://www.education.gov.au/higher-education-statistics [Verified 6 July 2023].
5 Commonwealth of Australia. Higher Education Support Act 2003. 2023. Available at https://www.legislation.gov.au/Details/C2023C00057
6 Commonwealth of Australia. Higher Education Standards Framework (Threshold Standards) Act 2021. 2021. Available at https://www.legislation.gov.au/Details/F2022C00105
9 Commonwealth of Australia. Tertiary Education Quality and Standards Agency Act 2011. 2022. Available at https://www.legislation.gov.au/Details/C2022C00328
10 Australian Learning and Teaching Council. Learning and teaching academic standards project: final report. Sydney, NSW, Australia: Australian Learning and Teaching Council; 2011. Available at https://ltr.edu.au/resources/altc standards.finalreport.pdf
11 Gora J. Watch out! Here comes the TEQSA juggernaut. Aust Univ Rev 2010; 52(2): 76-78.
| Crossref | Google Scholar |
12 Ewan C. Disciplines setting standards: the Learning and Teaching Academic Standards (LTAS) Project. In: Proceedings of AuQF2010: Quality in Uncertain Times, Gold Coast, Qld, Australia, 30 June–2 July 2010. AUQA Occasional Publications Number 22. Melbourne, Vic., Australia: Australian Universities Quality Agency; 2010.
13 Jones SM, Yates BF, Kelder JA. Learning and teaching academic standards for science: where are we now. In: Proceedings of the Australian Conference on Science and Mathematics Education (18th annual uniserve science conference), 26–28 September 2012, Sydney, NSW, Australia. Sydney, NSW, Australia: UniServe Science; 2012. pp. 105–109. Available at https://openjournals.library.sydney.edu.au/IISME/article/view/5904/6662
15 Schultz M, Southam D, O’Brien G. Development, evaluation, and application of chemistry threshold learning outcomes – a curriculum framework for tertiary chemistry in Australia. Aust J Chem 2020; 73(10): 825-831.
| Crossref | Google Scholar |
16 Cornwall A, Jewkes R. What is participatory research? Soc Sci Med 1995; 41(12): 1667-1676.
| Crossref | Google Scholar |
18 Etikan I, Musa SA, Alkassim RS. Comparison of convenience sampling and purposive sampling. Am J Theor Appl Stat 2016; 5(1): 1-4.
| Crossref | Google Scholar |
19 Birt L, Scott S, Cavers D, Campbell C, Walter F. Member checking: a tool to enhance trustworthiness or merely a nod to validation? Qual Health Res 2016; 26(13): 1802-1811.
| Crossref | Google Scholar | PubMed |
21 Al-Husseini S, El Beltagi I, Moizer J. Transformational leadership and innovation: the mediating role of knowledge sharing amongst higher education faculty. Int J Leadersh Educ 2021; 24(5): 670-693.
| Crossref | Google Scholar |
22 Owusu-Agyeman Y. Transformational leadership and innovation in higher education: a participative process approach. Int J Leadersh Educ 2021; 24(5): 694-716.
| Crossref | Google Scholar |
23 Smith AE, Humphreys MS. Evaluation of unsupervised semantic mapping of natural language with Leximancer concept mapping. Behav Res Methods 2006; 38(2): 262-279.
| Crossref | Google Scholar | PubMed |
26 The Legal Education Associate Deans Network Australia. Resources: Good Practice Guides. 2023. Available at http://www.lawteachnetwork.org/resources.html [Verified 10 July 2023].
32 Australian Council of Deans of Science. Science Threshold Learning Outcomes. 2023. Available at https://www. acds.edu.au/teaching-learning/science-threshold-learning-outcomes-tlos/science-tlos/ [Verified 10 July 2023].
34 Schmid S, Schultz M, Priest SJ, O’Brien G, Pyke SM, Bridgeman A, et al. Assessing the assessments: development of a tool to evaluate assessment items in chemistry according to learning outcomes. In: Schultz M, Schmid S, Holme T, editors. Technology and Assessment Strategies for Improving Student Learning in Chemistry. Vol. 1235. Washington, DC, USA: American Chemical Society; 2016. pp. 225–244. 10.1021/bk-2016-1235.ch013
35 Schultz M, O’Brien G, Schmid S, Lawrie GA, Southam DC, Priest SJ, et al. Improving the assessment of transferable skills in chemistry through evaluation of current practice. In: Schultz M, Schmid S, Lawrie GA, editors. Research and Practice in Chemistry Education. Singapore: Springer Singapore; 2019. pp. 255–274. 10.1007/978-981-13-6998-8_15
36 Schultz M. A personal reflection on the formation of the Chemistry Discipline Network (Guest Editorial). Aust J Educ Chem 2012; 72: 5.
| Google Scholar |
37 Schultz M, O’Brien G. The Australian Chemistry Discipline Network: a supportive community of practice in a hard science. In: McDonald J, Cater-Steel A, editors. Implementing Communities of Practice in Higher Education. Singapore: Springer Singapore; 2017. pp. 501–530. 10.1007/978-981-10-2866-3_22
39 Kelder JA, Jones SM. The Science Learning and Teaching Academic Standards project: a discipline community’s response to regulatory change in Australian higher education. In: Thomas T, Levin E, Dawson P, Fraser K, Hadgraft R, editors. Research and Development in Higher Education. Vol. 38: Learning for Life and Work in a Complex World, 6–9 July 2015, Melbourne, Vic., Australia. Higher Education Research and Development Society of Australiasia; 2015.
40 Jones SM. Assessing the science knowledge of university students: perils, pitfalls and possibilities. J Learn Design 2014; 7(2): 16-27.
| Crossref | Google Scholar |
41 Colthorpe K, Chen X, Zimbardi K. Peer feedback enhances a ‘journal club’ for undergraduate science students that develops oral communication and critical evaluation skills. J Learn Design 2014; 7(2): 105-119.
| Crossref | Google Scholar |
42 Colthorpe K, Zimbardi K, Bugarcic A, Smith A. Progressive development of scientific literacy through assessment in inquiry-based biomedical science curricula. Int J Innov Sci Math Educ 2015; 23(5): 52-64.
| Google Scholar |
43 Mercer-Mapstone LD, Matthews KE. Student perceptions of communication skills in undergraduate science at an Australian research-intensive university. Assess Eval High Educ 2017; 42(1): 98-114.
| Crossref | Google Scholar |
44 Schultz M. Teaching and assessing ethics and social responsibility in undergraduate science: a position paper. J Learn Design 2014; 7(2): 136-147.
| Crossref | Google Scholar |
45 Jones S, Johnson E, Kelder JA. Discipline learning outcomes: design resource and quality assurance mechanism. Adv Scholarsh Research High Educ 2021; 2(1): 1-27.
| Crossref | Google Scholar |
