Measuring the unmeasured in educational programs: filling in the blanks through evaluation?

The indicators of performance put forward as measures of achievement at a state and territory level in Australia reflect to some extent the priorities of those jurisdictions. These are revealed in the annual reports of departments, usually under headings of targets and corresponding outcomes. It may seem reasonable to assume that these performance measures line up with stated objectives, and with what matters on the ground. But do they? This paper argues that while the aims of education are broad, the measures of education are narrow. Philosophically, a good education is one that has social, developmental, intellectual and economic aims. However, even though some of those broad aims are reflected in national, state and territory foundational documents and reports, they are not reflected in Australian measurement or reporting frameworks, which seem to suggest that it is neither practical nor cost-effective to collect such data. The purpose of this paper is to demonstrate that it is possible to measure educational success in ways that support the broader goals of education and schooling in Australia. To this end, the paper draws on two evaluation case studies in the field of education (run within schools), to highlight ways that the unmeasured aspects of educational activity can be measured. One case highlights the significance of social capital in a school-family partnership program, and the other demonstrates the psycho-social benefits of an alternative education program for ‘at risk’ children. These cases illustrate what can be measured and they provide useful data to fill in the blanks of what is not measured at a system level. However, the cases also raise bigger questions about what should be measured and reported as indicators of what matters to educational stakeholders. 205 Learning Communities International Journal of Learning in Social Contexts | Special Issue: Evaluation | Number 14 – September 2014


abstract
The indicators of performance put forward as measures of achievement at a state and territory level in Australia reflect to some extent the priorities of those jurisdictions. These are revealed in the annual reports of departments, usually under headings of targets and corresponding outcomes. It may seem reasonable to assume that these performance measures line up with stated objectives, and with what matters on the ground. But do they? This paper argues that while the aims of education are broad, the measures of education are narrow. Philosophically, a good education is one that has social, developmental, intellectual and economic aims. However, even though some of those broad aims are reflected in national, state and territory foundational documents and reports, they are not reflected in Australian measurement or reporting frameworks, which seem to suggest that it is neither practical nor cost-effective to collect such data.
The purpose of this paper is to demonstrate that it is possible to measure educational success in ways that support the broader goals of education and schooling in Australia. To this end, the paper draws on two evaluation case studies in the field of education (run within schools), to highlight ways that the unmeasured aspects of educational activity can be measured. One case highlights the significance of social capital in a school-family partnership program, and the other demonstrates the psycho-social benefits of an alternative education program for 'at risk' children. These cases illustrate what can be measured and they provide useful data to fill in the blanks of what is not measured at a system level. However, the cases also raise bigger questions about what should be measured and reported as indicators of what matters to educational stakeholders.

introduction
Over recent years there has been a lot of emphasis on measuring education. The introduction of the My School website in 2010 brought this into sharp focus and caused (and still does) something of a stir among those who saw the data measured being used for the production of league tables to unfairly rank schools on the basis of academic performance (Redden & Low, 2012). But what appears to emerge on the surface from the My School product actually comes from deeper roots that define and shape what education should be for and about. In this paper, I explore these foundations from the starting point of the 2008 Melbourne Declaration on Educational Goals for Young Australians, with an eye to uncovering the broader philosophical foundations of a 'good' education, and how these are played out in the Australian context. Next, I turn to the literature on issues of measurement in education and why some aspects of education are measured and some are not. I then consider how a good education might be considered from the perspective of four departments of education annual reports from 2013. What this will show is a fairly narrow interpretation of 'good education' with an even narrower set of measures. The Measurement Framework for Schooling in Australia 2012 states that the measures used are 'cost effective, practical to collect, and take account of the burden and impact that data collection may place on students, schools and schooling systems' (ACARA, 2012, p. 5). However, do the measures actually reflect policy makers' real (and narrow) priorities, and have less to do with practicalities or burden?
In order to counter the 'practicality' argument, I present two evaluation case studies that demonstrate in the context of alternative education programs, how other indicators can be identified. The first is a social circus program conducted in Tasmanian schools and the second is a family strengthening program conducted in a Northern Territory school. I contend that while these evaluations used qualitative methodologies, the identification of indicators allows for a valid quantitative approach to be taken that could easily measure other indicators of a good education. But these alternative indicators are seldom measured as confirmed by the examination of departmental annual reports presented here. But why is this so?
Background I begin with the broad philosophical question about what makes a 'good' education. Following this, the review turns specifically to the Australian context, showing how those philosophies are played out at a national level. Finally, attention is drawn to the specific goals and outcome measures for four state and territory departments of education.

What makes a 'good' education?
The philosophical and theoretical bases for educational strategic policy directions are diverse. The associated strategies and performance measures reflect sometimes divergent views about what education is really for. This paper does not allow for a detailed discussion of the theoretical and philosophical foundations of education. However, it may be helpful to briefly outline some of the key foundations on which departmental visions, goals and objectives, are based. What I am trying to get to is, as Biesta (2009) asks: 'what constitutes good education?' There is a social and societal rationale for education.
There are various social theories that underpin educational systems. Education has been seen as a vehicle for social control (Dewey, 1938;Payne, 1927) and for the promotion of citizenship (Gutmann, 2009;McCowan, 2010). Others have described education as transformative and emancipatory (Freire, 1970;Oakes et al., 2013). Education too, is seen as a process that builds 'social capital' (Coleman, 1988) and is a product of 'cultural capital' (Bourdieu, 1983) which in turn maintains class divisions (Reay, 2010).
There is a developmental rationale for education.
The international discourse around education and development suggests strongly that better education leads to increased levels of development (Hanushek & Woessmann, 2007;Keeley, 2007;OECD, 2012a). The empirical evidence that education and learning is related to a range of benefits including social equity (Field et al. 2007;OECD, 2012b), health (Ross & Mirowsky, 2010), justice and reduced criminal behaviour (Lochner, 2011;Machin et al. 2011), employment, economic and developmental (Hanushek & Woessmann, 2009;OECD, 2012a), family and individual outcomes (Schuller et al. 2004) is readily available in an array of literature. The hope of education is that it leads to a better life, particularly for those living on the margins of society. Leadbeater (2012, p. 23) suggests that education 'offers them a hope that their place in society will not be fixed by the place they were born' and that through education people can 'remake their lives'.

There is a knowledge and skills rationale for education.
There is a view that knowledge is an end in itself, that one of the primary aims of education is epistemic (Robertson, 2009), and that for educators it is reasonable to expect that it is 'possible, and desirable for people to know and do things, and to engage in and take seriously the fruits of rational inquiry, where such inquiry is understood to involve the pursuit of truth' (Siegel, 2010, p. 283). Such pursuit of knowledge forms a foundation for students to be able to make appropriate moral choices and therefore become good citizens (Feldman, 2009;Halstead, 2010).

There is an individual and economic rationale for education.
The focus on individualism has its roots in Greek philosophy and was further developed by Enlightenment philosophers such as Kant and Rousseau, who emphasised individual autonomy and individual freedom. (for a discussion of the historical development of philosophies of education see Carr, 2010). The arguments of liberalist education philosophers suggest that 'schools should encourage competition between individual students and prepare students to live independent lives in society, respecting their uniqueness and distinct capabilities' (Portelli & Menashy, 2010, p. 421). Individualism is also reflected in the economic theories of Adam Smith (1904) which in turn is reflected in what could be described as free market capitalism. The economics of education has come to the fore in recent decades. A notable contribution to the field was Becker's (1964) work Human Capital: a theoretical and empirical analysis, with special reference to education, which brought together ideas of return on investment in education and distribution of income on the basis of educational attainment. Internationally, policies are built around assumptions of economic growth flowing from education and training: 'skills have become the global currency of 21st-century economies' (OECD, 2012a, p. 10).

philosophies of education played out in australia
While elements of the above discussion can be found in literature that spans decades in Australia, a significant marker in education occurred in 2008 with the Melbourne Declaration on Educational Goals for Young Australians. It articulates two main objectives: Goal 1: Australian schooling promotes equity and excellence Goal 2: All young Australians become successful learners, confident and creative individuals, and active and informed citizens. (Ministerial Council on Education, 2008, p. 7) The goals articulated in the Declaration are also broadly consistent with a philosophy of education that goes beyond a focus on academic performance and transition to employment. They represent education as a vehicle for individual and social achievement, for an inclusive and respectful society that supports the development of knowledge and skills, but not to the exclusion of other personal and social imperatives. The definition of equity is not about conformance to the norm. Rather the goals express a need for diversity such that 'schooling contributes to a socially cohesive society that respects and appreciates cultural, social and religious diversity.' (p. 7) In short, the goals represent an array of educational, epistemic, moral and political aims (Brighouse, 2009;Robertson, 2009). The 2012 National Education Agreement, which builds on the Declaration, specifies five outcomes of education that in turn determine the key performance measures of education. These are: (a) all children are engaged in and benefiting from schooling; (b) young people are meeting basic literacy and numeracy standards, and overall levels of literacy and numeracy achievement are improving; (c) Australian students excel by international standards; (d) schooling promotes the social inclusion and reduces the educational disadvantage of children, especially Indigenous children; and (e) young people make a successful transition from school to work and further study. (Standing Council on Federal Financial Relations, 2012, p. 6) The Measurement Framework for Schooling in Australia (ACARA, 2012) articulates how these outcomes are to be measured. Essentially, this document distils the outcomes into three main areas: participation, achievement in the National Assessment Program and attainment. The specific indicators include enrolments, attendance, and participation in assessments, levels of literacy and numeracy, school completion and attainment and achievement of young people in other learning pathways. Equity 'measures are not separately listed in the Schedule of Key Performance Measures but are derived, for reporting purposes, by disaggregating the measures for participation, achievement and attainment where it is possible and appropriate to do so' (p. 6). This last point is important as it means the focus is not on equity and diversity but on the other outcomes listed, which may be seen to promote conformance to the norm. The Measurement framework makes no attempt to measure the broader goals of education as outlined earlier in the broad discussion about what makes a 'good' education or the goals of education as outlined in the Melbourne Declaration.

Measurement of education
Why do systems measure what they do in education? At the school level, the issues of measurement -as reflected in schools' student reports for example -are about ensuring that students learn what they are taught. Measurement is essential for student feedback, teacher professional development and informing parents about student progress. This kind of measurement underpins what Hattie (2009) describes as 'visible teaching and learning'. Even with this as a given, the issues of how assessments and tests should be used are controversial and contested. Some scholars (Anyon, 2010;Oakes, 2005;Oakes et al., 2013) argue that testing which 'tracks' -or streams -students has a deleterious effect on student learning and equity. The drive for accountability is another factor that determines what is measured. As an American proponent of educational accountability, Taft (2012) argues that: Aligned systems of academic standards, assessments and accountability are one essential component of an effective strategy for raising the bar for student learning … and giving all students an opportunity for success after high school. (p. 4) The language of transparency and accountability, effectiveness and efficiency underpins assumptions about why education should be measured. The 2012 Measurement Framework is premised on the 'accountability requirements established in the National Education Agreement and Schools Assistance Act 2008' (ACARA, 2012, p. 1). On the surface, the argument for accountability is one of equity. Zanderigo et al. (2012) contend that: The transparency and accountability mechanisms are aimed at improving outcomes and equity for all students by using nationally comparable school performance data to build a substantive evidence base to support future improvements. (p. 3) All other things being equal, there is some merit to this argument, provided that accountability measures do not lead to 'tracking' effects (Schütz et al., 2007). Making schools accountable for student performance -creating competition by providing funding incentives on the basis of differential outcomes -may well have the effect of driving consumer choice away from poor performing public schools to higher performing private schools where low income families are effectively excluded due to high fees. School behaviour may also change to take advantage of accountability systems. Figlio and Loeb (2011) suggest that schools' strategies may change in response to accountability measures, for example teaching to the test, and the use of strategies to exclude poor performing students at test times. As can be seen, all things are not necessarily equal and trying to disaggregate cause and effect in a complex system is very difficult, as Jenson (2013) revealed in his study of competition, autonomy, choice and markets in Queensland.
The above discussion suggests that accountability frameworks set the agenda for what is measured in education. Frameworks that promote measures such as attendance rates, academic performance and retention rates, result in those things being privileged. While indicators of equity are collected, in Australia at least, the incentives associated with accountability frameworks do not directly reward increases in social inclusion or ethnic diversity for example. The 2012 Review of Funding for Schooling (Gonski et al., 2012) argued that evidence 'shows that strengthening equity in education can be cost beneficial' (p. 108) and supported needs based funding models to promote equity. However, its Terms of Reference were focused on funding arrangements rather than measurement frameworks, even though it did discuss accountability. If equity was of primary concern in the measurement framework, surely equity outcomes and targets (not just equity indicators) would be incorporated into the table of measures listed under the Schedule of Key Performance Measures of the Framework (ACARA, 2012, p. 7).
In summary, Australia has a measurement framework that argues for the importance of equity but does little to promote it. What do count are measures related to student academic performance (in standardised test results) and participation (through attendance and retention measures). We would therefore expect to see those priorities reflected in the official positions of education departments. In the next section that proposition will be tested. Table 1 is a condensed summary of the vision, mission, goals, targets, strategies and measures of four jurisdictional education departments. The analysis is based on publicly available annual reports for the year 2012-13. The purpose of the table is to highlight the priorities of departments of education and how they line up with the priorities of a good education as discussed above. For all states, the vision/mission statements and supporting goals line up fairly closely with the measures associated with the direction presented in the Measurement Framework discussed above. There is a strong focus on skills, knowledge, innovation, economic prosperity, productivity, achievement and success.

Education Department objectives, outcomes and performance measures in four states
South Australia deviates somewhat from the other jurisdictions listed in that its vision includes a democratic, equitable and cohesive society. This vision indicates a broader understanding of what a good education is in terms of its societal goals. However, when it comes to measurement, the indicators are all about academic performance. Similarly, Western Australia includes the motto 'excellence and equity' but makes no mention of this in its performance measures. It adds cost per full time student to the mix of performance measures, which perhaps links to the goal of a 'capable and responsive organisation' but the link is not entirely direct. The Tasmanian mission statement does include the aim of 'contributing positively to the community' but none of its measures reflect that goal. The Northern Territory vision statement, targets and measures are perhaps the most congruent of all the jurisdictions presented. However, they are narrowly focussed on a limited number of academic/school related outcomes.
One of the key drivers for the indicators chosen appears to be the Measurement Framework for Schooling in Australia, which is clearly reflected in the reported performance measures for each of the jurisdictions. There is no attempt to measure the social or equity aims that are included in the mission or goals of South Australia, Western Australia or Tasmania. Further, for these jurisdictions there is no apparent logic that connects these aims to the performance measures. In summary, the performance measures appear to be narrowly focused on the knowledge and skills rationale for education, underpinned by a Human Capital theoretical position that sees economic benefit from schooling.
Given that three of the four jurisdictions seem to put education forward more holistically than that, it could be reasonable to ask why measures for these broader purposes and outcomes of education are not included. The Measurement Framework argues that equity measures can be derived from disaggregated indicators such as Indigenous status, gender, location, language background, socio-economic background and disability. However it could be argued that these disaggregated measures simply indicate how well equity groups achieve the performance measures. They do not in themselves indicate how equitable, inclusive, respectful, democratic or cohesive the education provided intrinsically is, or how it contributes to society in those ways.  It may be argued that these things are not measured because they are difficult to put a number on. This is the position that the Australian Bureau of Statistics takes in its Measures of Australia's Progress documents (ABS, 2013). However, it could be equally argued that perception measures could be used. I will return to this later, but for the moment I want to consider two cases of evaluations involving schools in two of the jurisdictions tabulated above, where the measure of outcomes extends beyond academic performance and attendance.

Findings from two alternative education programs
Two case studies highlight how measures other than attendance and academic performance can be measured. The activities were quite different and in different jurisdictions, but conducted by the same evaluator. The focus here is not on the outcomes of the evaluation but on the development of alternative measures.

Case study 1: Social circus in Tasmanian schools
As part of an evaluation of the Communities for Children (CfC) program in Burnie, Tasmania the evaluator was asked to conduct a detailed evaluation of two social circus activities. Social circus is a subset of community arts, which uses circus skills to promote artistic expression as a vehicle for a range of social and educational outcomes for participants. The activities in this case were designed to work with parents, carers and their children in an effort to engage them in an alternative educational activity that would build bridges between schools and families and offer opportunities for creative expression and learning success outside the classroom. The activities were evaluated during 2011. CfC was designed to support and build capacity in vulnerable families.
The two activities under investigation were designed to support families and students to connect with their schools and vice versa. There was also an expectation that parents would engage with their children in the activities. This was a prerequisite for the children to be able to participate. There were however, two major differences in the ways the activities were run. The first was conducted over six weeks at a hall away from school and involved a family support service. The second was conducted over four weeks at a primary school and did not include involvement of a family support service.
The outcomes of the two activities differed somewhat but there was substantial overlap. Outcomes common to both were categorised as: an experience to remember; interaction within family; physical activity; self-confidence; and skills and capacity.
In the first activity, which was more intensive, additional outcomes were recorded. They were categorised as: better behaviour from children; changed attitude; mutual support; rapport, relationship, social networks and trust.
The purpose of this case study is not to detail what these outcomes mean. Rather it is to highlight that they were measured. How were they measured? The tools used for both evaluations included: evaluator observations using photographs and video recordings; a reflective process with circus trainers (using a journal); a focus group with organisational stakeholders; a focus group with participants at conclusion and a survey of participants on completion. By combining the data from the various sources it was possible to demonstrate how the outcomes were perceived through the lens of participants, trainers, family support workers and the camera (as taken by the evaluator).

Case study 2: Family strengthening program in a Northern Territory school
Families and Schools Together (FAST) is a family strengthening program that helps build positive relationships between parents, schools and the broader community. At its core is an evidence-based eight week program designed to facilitate social support, greater parental self-efficacy, better child behaviour and improved educational outcomes. In the Northern Territory, the program has been run since 2002 in a variety of locations from remote communities to urban contexts. A 2013 evaluation of the program run in Alice Springs highlighted social capital outcomes that emerged from the program. The evaluation found that FAST contributed to a growing sense of community within the school. A recurring theme emerging from the data was the significance of improved social networks and support fostered during the eight week program.
The link between parent-school engagement and learning outcomes was not direct, but what could be surmised from the data was that better engagement led to better understanding of family needs, which in turn led to more responsive teaching, better targeted resources, and subsequently better learning outcomes. Over the longer term, for the school, FAST affirmed a culture of school-family-community engagement that permeated the school environment. The school's reputation in the community was enhanced. FAST Parents reported a sense of belonging and allegiance to the school that was reflected in their willingness to support the school as their school, not as the school.
So how were these findings measured? This was a largely qualitative study. Interviews were conducted, themes were analysed and quantitised 1 . The questions were built on a pre-existing theory of change which had been built up from previous evaluations. The themes (in order of importance) were: engagement and partnership; relationships with school; empowerment; social networks; support for struggling families; identity and confidence; attendance, performance and behaviour; and communication. There were some differences in the relative importance of some themes for different groups. For example, staff commented more on the importance of attendance, performance and behaviour than the other groups. Families commented more on 'support for struggling families' and the FAST team saw empowerment as more important.

Discussion
The intention in this paper is not to provide a detailed analysis of the case studies presented above. Nor is it to consider the impact of the measures identified above for commissioned evaluations. Rather the intention is to demonstrate that, based on the literature which describes a 'good education' beyond the fairly narrow indicators presented in Table 1, it is possible to measure other aspects of education that contribute to a broad range of outcomes.
Developing additional measures of a good education.
Essentially, the above case studies demonstrate the potential for outcomes such as engagement, partnerships, social connectedness, parental involvement, behaviour change, identity and confidence, social support and social capital to be measured. While the methods were qualitative, it would not be too difficult to change the instruments so that they assessed the indicators quantitatively. The value of the qualitative process is that it builds a theory that can be tested -which is consistent with grounded theory approaches in social research (Charmaz, 2006(Charmaz, , 2011. In the past I have developed tools that do this kind of testing quite successfully without the need for complex or overly long tools (see for example Guenther, 2011;Guenther & Arvier, 2010;Guenther & Boonstra, 2009;Guenther & Falk, 2000). Schools in general are quite good at eliciting perception data from parents, students and teachers. Further, there are plenty of precedents for the measurement of social characteristics such as social capital and social inclusion (Grootaert et al., 2004;Harpham, 2008;Narayan & Cassidy, 2001), social connectedness and conflict (Bond et al., 2007;Cornwell et al., 2008) to mention just a few. The point of this discussion is to confirm that qualitative measures such as those used in the evaluations cited earlier can be measured quantitatively as well. They could form part of a suite of alternative measures of a 'good' education that would better reflect the broader purposes of education beyond the prevalent individualistic, knowledge and skill based measures that comprise the Australian Measurement Framework.
It is reasonable to ask why the current measurement and reporting frameworks do not allow for a broader measurement of important aspects of educational goals such as social inclusion, equity, or civic participation. Is it because, as the 2012 Measurement Framework suggests, the measures of these elements of a good education are impractical? Or is it because the current measures reflect what is important for policy makers?

The importance of alternative measures
Why should any of this matter? After all, it could be argued that the vast majority of students happily conform to or embrace an education that fits neatly within a Human Capital Theory framework where returns from education are based on individuals investing in knowledge and skills for a productive economy. The challenge to this is that education should be for all, not just the majority. In my work as an evaluator 1. A term that Tashakkori and Teddlie (1998) used to describe the process of turning qualitative data into quantitative data. and researcher in education I have seen how marginalised students -such as those with learning disabilities, those from struggling families, those in contact with the criminal justice system, and Indigenous young people living in remote Australian communities -are further marginalised by measurement systems that fail to recognise their strengths, and which relegate them to alternative educational programs.
If, as the three departments of education shown in Table 1 profess, education is about equity, democracy, inclusion, respect and social cohesion, then these elements of a good education ought to be measured. It is one thing to have a Closing the Gap agenda -as Australian governments do have currently (Council of Australian Governments, 2011) -but it is quite another thing to measure performance of interventions and policies against the outcomes which it supports. (Appendix 1 in Atelier Learning Solutions, 2012 offers an extensive list of alternative indicators for National Partnership programs). This is a concern that has been identified often in evaluation reports and reviews. Sometimes it is put down to a lack of data or evidence (Wilson, 2014). Sometimes it is due to measures being associated with outputs rather than outcomes (Atelier Learning Solutions, 2012). I am arguing here that -in the case of education -the approach which disaggregates equity groups from the measures that matter (attendance, academic performance, attainment and retention) simply diverts attention from equity and has the potential to further marginalise those who are already marginalised.
Schools that promote ethnic and cultural diversity, that support students with learning or physical disabilities, that promote civic participation, and that foster social inclusion could very easily be identified by tweaking the current Index of Community Socio-educational Advantage (ACARA, 2013) so that it reflected equity or social inclusion indicators. Using this measure, schools which measure improved social inclusion and equity as outcomes (rather than needs) could be rewarded. The two case studies highlight -based on evaluation findings -how success can be reconceptualised and measured to support these goals.

conclusion
This paper has set out to lay a foundation for what makes a 'good' education in Australia. It shows-despite the multiple purposes of education-that in Australia the dominant rhetoric is that successful education is largely about individual knowledge and skills for economic benefits. This is reflected in the accountability and measurement frameworks that currently exist in Australia. This in turn is reflected in annual reports where performance measures are almost all related to academic performance, attendance and attainment. This is all well and good if, like the majority of Australians, we take these assumed purposes as a given. My concern in evaluation and research in education, however, has often been about minorities. The two evaluation case studies of alternative education programs demonstrate the array of alternative indicators that may be important beyond academic performance, such as equity and inclusion. While those evaluations drew on qualitative methods, there is ample precedent for the quantitative measure of similar outcomes.
Alternative measures do matter -and not just for equity groups. They matter because the important outcomes for many people taking part in alternative education programs are not necessarily academic ones. Even for those in standard programs, the significance of outcomes such as social cohesion, civic participation, identity formation, equity, democracy, and the development of strong and productive social networks should not be underestimated. Education without these underpinning values runs the risk of perpetuating and promoting social inequalities. In this paper what I have hopefully achieved is something like filling in the blanks of missing information in the patchwork of evidence that contributes to our understanding about what makes a good education in Australia.

acknowledgement
The work reported in this publication was supported by funding from the Australian Government Cooperative Research Centres Program through the Cooperative Research Centre for Remote Economic Participation (CRC-REP). The views expressed herein do not necessarily represent the views of the CRC REP or Ninti One Limited or its participants. Errors and omissions remain with the author.