Quality, Accountability and Action – Part 2

By Linda Baer, Ph.D.; Copyright 2015; reprinted by permission of ACICS

Assuring performance

Many who are looking for more public accountability point to performance as best evidence of quality. But how to effectively measure performance? One key measure is student learning outcomes. In addition, national and state efforts have identified several other measures including access, productivity and efficiency, degree completion, and economic returns from post-secondary education. (See Measuring Up: The National Report Card on Higher Education, National Center for Public Policy and Education. Read Here¹)

Historically, evaluation of the performance of post-secondary colleges and schools has been based in part on measured inputs (size of endowment, access to funding, mix of funding sources, investment in new buildings and facilities, pedigree of faculty, pedigree of students admitted, research grants and contracts received) and measured outcomes (student retention and persistence rates, graduation rates, rates of placement in jobs, and post-graduation income levels). Private, independent colleges and schools organized as for-profit institutions confront additional outcomes measures, including the rate at which students default on loans post-graduation (Cohort Default Rates) and the numerical relationship between the price of education and the post-completion earning performance of the completer (so-called Gainful Employment regulations.)²

All effective measurement requires a point of reference. Some measures are benchmarked against peer institutions, aspirational standards (such as published mission statements and strategic objectives), historical data, statewide data or national data.³

Other sources of performance evaluation in higher education are more focused on aggregate performance, rather than course level, program level or even campus level performance.

The National Report Card on Higher Education, authored by The National Center for Public Policy and Higher Education, focuses on five measures that apply to sets of institutions, an entire community (or state) or a set of communities.

These measures by implication integrate social and economic conditions into the performance evaluation of post-secondary education. The key indicators were selected because they are broad gauges for understanding success in key performance areas:

• Preparation: Percentage of 18- to 24-year-olds with a high school credential (1990 to 2006)
• Participation: Percentage of 18- to 24-year-olds enrolled in college (1991 to 2007)
• Affordability: Percentage of income (average of all income groups) needed to pay for college expenses at public four-year institutions (1999-2007)
• Completion: Certificates and degrees awarded per 1,000 state residents (age 18-44) without a college degree (1992 to 2007)
• Benefits: Percentage of 25- to 64-year-olds with a bachelor’s degree or higher (1990 to 2006)⁴

“Change Over Time” arrows in the Report Card’s Measuring Up 2008 compare each state’s current performance with its own previous performance in the 1990s. This year, however, a state’s “Change Over Time” is determined by its improvement or decline in performance on a key indicator in each performance category. States receive either an “up” or a “down” arrow: an “up” arrow indicates that the state has increased or remained stable on the key indicator; a “down” arrow indicates that the state has declined in the key indicator. Affordability is different from the other categories in that lower percentages indicate higher performance. The National Center does not establish benchmarks for improvement; however, the change–over-time performance of the top states is depicted graphically in the summary report cards. Many states, but not all, have improved on these key indicators.

Another source of information about the aggregate performance of colleges and universities is Complete College America (CCA). Established in 2009, Complete College America is a national nonprofit with a single mission: to work with states to significantly increase the number of Americans with quality career certificates or college degrees and to close attainment gaps for traditionally underrepresented populations. Thirty-four states are currently participating in Complete College America.⁵ (completecollege.org)

CCA has identified factors that appear to be material to institutional contributions to higher education success:

“Through research, advocacy, and technical assistance, we help states put in place the five GAME CHANGERS that will help all students succeed in college. The metrics include: performance funding, co-requisite remediation, full-time is fifteen, structured schedules, and guided pathways to success.”⁶

Shray has observed that a long standing debate persists regarding accreditation’s role in assuring to the government and the public that higher education institutions and programs are effective in achieving results, especially student learning outcomes:

“Currently, accreditation standards focus primarily on resource and process standards (e.g., faculty qualifications, facilities and support services) and do not require evidence that institutions have achieved results consistent with their missions, especially student learning outcomes. They also maintain that accreditation has a long way to go in establishing quality standards for student assessment to assure that institutions can and do provide valid and reliable evidence of student learning outcomes.”⁷

(Also see Ewell 2002 Assessment, Accountability and Improvement. National Institute for Learning Outcomes Assessment.) ⁸

learningoutcomeassessment.org

What is higher education doing about improving accountability?

Contemporary accountability conversations are progeny with a long history. What is accountability? Joseph Burke describes many faces of accountability:

“Accountability is the most advocated and least analyzed word in higher education. Everyone uses the term but usually with multiple meanings. Writer says it faces in every direction – upward, downward, inward, and outward. It looks bureaucratic, participative, political or market centered. It may appear two-faced, with sponsors and stakeholders demanding more services while supplying less support. Of critical concern is the fact that with the conflict over accountability, there is growing erosion of public trust and support for higher education as a public good for all Americans and not just a private benefit for college graduate.” ⁹

It is critically important to the future of higher education that there be a balance between federal, state and local interest.

Policies must continue to be reviewed and developed at the highest possible level where impartiality to all incumbent actors is manifest, protected and valued. At a time in higher education when support for innovation and entrepreneurialism is needed in order to improve and expand quality educational opportunity and success, policy and political bias toward one or more incumbent sectors of higher education stymies innovation thwarts financial investment and deliberately or by happenstance create barriers to entry. This fundamental challenge for policy makers and regulators should be kept front and center during policy formation.

Historical context

In the mid-1980s the National Governors Association published A Time for Results. States began to require colleges and universities to report on performance. Congress developed the Student-Right-To-Know Act, which mandated significant new disclosures regarding graduation rates and school safety. The joint state-federal State Postsecondary Review Entities (SPREs) were added to monitor and evaluate institutional performance. The SPREs did not last long as the government tone moved away from this form of accountability by the mid-1990s.¹⁰ (Carey 2-3)

In the mid-2000s, the Spellings Commission on the Future of Higher Education¹¹ provided a strong indictment of the sector. It focused on costs that were too high and graduation rates that were too low, especially among low-income and minority students. Learning outcomes remained a mystery.

Overall, the response to the Commission’s report was similar to previous calls for more accountability: the higher education community developed strong defenses against the criticisms, ultimately convincing policy makers that it was already accountable in many ways. The higher education community also argued effectively that “institutions are so diverse and unique that no single form of accountability could be used to assess all fairly.”¹² (Carey 1-3)

Those not swayed by the response to the Commission’s report argued that defensiveness is not effective, particularly from a multi-million or multi-billion dollar enterprise with great prominence and profile in the community that is routinely afforded great deference and respect. Persistent are demands for higher education to enumerate and explain how existing accountability systems work and why they are effective; and for institutions at all levels, public, private and for-profit, to stand for comparative review and evaluation if they have similar missions and are serving similar types of students.

Over the years, many panels, commissions, acts and proposals have called for stronger accountability in higher education. The premises behind accountability methods of the past were often flawed. Institutional performance was often based on historical information that lacked context or meaning.

Data, information, knowledge and decision-making

Access to reliable, authoritative data and evidence for performance and accountability analysis of higher education institutions is a fundamental issue.

Currently, a huge void exists in the availability to adequate, accurate data across institutions, states and at the national level. Only recently has there been investment in statewide data systems to better measure the performance and outcomes produced by higher education within a given state. (See NCHEMS report on state based data systems)¹³

A recent proposal in Congress demonstrates the high level attention is being paid to that void:

“The Evidence-Based Policy-Making Commission Act of 2014 would establish a commission to initially survey the types of federal data that are available to inform policymaking. It would then produce recommendations related to data the government needs but does not have – more specifically, asking whether and how to create a clearinghouse that would facilitate linkages between existing federal datasets that can help answer critical questions. In the higher education arena, this approach could be a huge win for students, families, taxpayers, and policymakers. The bill would … make lawmakers consider policy in a way they rarely do: with evidence.”¹⁴

The importance of this act is that it acknowledges that without definitional standards, a reliable clearinghouse and the ability to “facilitate the linkages between federal databased” the promise that can come from improved accountability will not be realized. As of January 2015, the legislation had not been enacted.

Today, technological advances in data collection, warehousing, mining and analysis have empowered institutions to make quantum leaps in the use of data to improve institutional performance and student success. By harnessing those data sources, and framing their application and analysis in the appropriate context, leaders of colleges and universities can deliver to various stakeholders accountability and performance results that are accessible, credible, and based on measurements that are replicable.

Dualities of expectations

Evaluating institutional effectiveness may forever remain part art and part science, but aspiring to make it more scientific is arguably honorable.

When demonstrable accountability has potentially existential consequences – that is, a lack of accountability can curtail or terminate access to critical resources – the need for defensible and reliable social science and empirical evidence, not just intuition, becomes mandatory.

In addition to the duality of evaluative methodology, higher education exists within an environment of conflict between autonomy and collegial governance. It must exhibit effectiveness to current and former students while acknowledging the demands of accountability by federal, regional, state and local governmental and community stakeholders.

One authority suggests that a meaningful discussion of accountability “should take into account who is accountable to whom, for what purposes, for whose benefit, by which means, and with what consequences.”¹⁵ (Lingenfelter, 2003). As previously noted, conflicts over accountability may produce opportunities for one set of institutions to gain advantage over another in the policy arena; avoiding this outcome will require scrupulous attention to process and impartiality by those managing the various policy venues.

Berdahl further describes conflicting and dynamic demands:
“Universities have generally had ambivalent relations with their surrounding societies: both involved and withdrawn; both serving and criticizing; both needing and being needed.” ¹⁶

Berdahl insists that post-secondary institutions must stay sufficiently safe from external pressures to safeguard their societal critique yet sufficiently responsive to external needs to sustain societal support. They must simultaneously serve and scrutinize the society that supports them. Burke indicates that the dual roles demand “both autonomy and accountability.”¹⁷ (Burke, 4)

Burke (Bogue and Hall)¹⁸ develops a helpful view of the tension between the civic and collegiate interests and cultures as well as the commercial and entrepreneurial culture that constitute today’s higher education environment. This tension is manifest as a series of accountability conflicts between institutional autonomy and external accountability:

• Institutional improvement versus external accountability
• Peer review versus external regulation
• Inputs and processes versus outputs and outcomes
• Reputation versus responsiveness
• Consultation versus evaluation
• Prestige versus performance
• Trust versus evidence
• Qualitative versus quantitative evidence.

Some of these conflicts may be considered false dichotomies. That is, many good institutions have the capacity to demonstrate external accountability for internally generated quality improvements; prestige should not be sacrificed because of having to prove performance; and gaining the trust of various external stakeholders should not grant an institution a waiver from producing evidence of performance.

Additionally, the application of accrediting compliance and greater institutional accountability may have unintended consequences, some would argue. Those include the protection of the status quo: incumbent providers benefit, while new providers who offer delivery modalities or systems of education that defy conventional evaluation are from stymied by regulation and requirements that favor the conventional and impede or resist innovation. The inherent effect of accreditation as inhibitor rather than facilitator of institutional innovation has been noted and verbalized during Congressional review of the reauthorization of the Higher Education Act.

But rather than arguing about the degree to which formalized quality assurance either facilitates or inhibits innovation at colleges and schools, the accrediting community has strong interest in how to assure the quality of education delivery modes and organizational structures that do not exist today.

If accreditation standards reasonably apply to colleges as we know them now, will the standards be enduring and applicable to those institutions that, in response to market and policy pressures, transform themselves? Or will the standards of quality have to be transformed as well?
Others in the accrediting community view the duality (innovation versus accreditation compliance) as superficial. A primary focus of all accreditation is the respectful but unrelenting push for quality and integrity. The force of that prerogative does not inherently preclude innovation or reinforce status quo; on the contrary, if done effectively, the only way an incumbent institution is going to fulfill evolving, dynamic accreditation requirements and remain operationally viable is through innovation, in some cases rising to the level of transformation.

Models for accountability

At least some of the conflicts over the meaning and value of accountability might be avoided if the various stakeholders could explore the subject with a common model or theory. An inventory of various accountability models is shown in Table 1.1:

Burke, the author of the inventory, observes that “past and present models of accountability suggest little progress in the process of reconciliation between the collegiate, civic, and commercial cultures. Higher education has featured at least six models of accountability: bureaucratic, professional, political, managerial, market, and managed market.” ²⁰

Each model has its own levers or drivers, agents or actors, and goals or purposes. The goals have shifted over time from efficiency to quality to productivity and, finally, to responsiveness to public priorities and market demands.

The techniques of accountability differ by model. Bureaucrats like rules. Professionals demand consultation. Policymakers prefer planning, although government officials still revert to regulation. Managers calculate costs and benefits; entrepreneurs respond to customer satisfaction and anticipate market demand. Each approach seems suited to different conditions.

The bureaucratic model demands stability, the professional requires autonomy, and the political necessitates consensus or at least majority consent. The managerial model works well in dynamic periods of considerable change. Both market models adjust capacity to demand, with government incentives shaping supply and demand to suit public priorities in managed markets.

Each model has positive or negative consequences based on performance or results. The bureaucratic rewards compliance with continuation and penalizes deviations with sanctions. Success in the professional model encourages consultation on decisions; failure results in neglect of professional advice. Financial incentives represent the positive and funding losses the negative in the political model, although performance reporting usually has only a positive or negative effect on the reputations of colleges or universities because it lacks a direct, formal connection to funding. The management approach involves either promotion or demotion, or possibly acclaim or disapproval. Markets produce profits or losses, and managed markets add incentives to the positive consequences.

Bureaucratic accountability centralizes governance. All the other models, with the possible exception of the political, demand decentralized decision making, although the political, managerial, and managed markets add varying degrees of policy directions. The political model can lead to intrusive regulations. Some of the accountability models rest on mature theories, but others represent practices outrunning theory. Each accountability model has generated accountability programs that accent the goals of efficiency, quality, productivity, market responsiveness, and public priorities.

Of course, the accountability systems in place are seldom as pure as the above categories may suggest. Each model has advantages and disadvantages, depending on application and timing. Bureaucratic accountability lives on in many states and constantly threatens a comeback in all organizations, public or private. Recent scandals in the stock market remind us that some regulation is required to prevent outrageous behavior. Conversely, uniform regulations do not work well in diverse and complex organizations such as colleges or universities. Professional accountability (or faculty participation) is essential to effective accountability systems in colleges and universities, but it can lead to gridlock in collegial decision making and to diminished responsiveness to public priorities or market needs. Policies and politics, management and markets are necessary parts of accountability in state higher education systems and in public and private colleges and universities. Yet each of these levers can divert higher education from its fundamental purposes in favor of momentary fads.”²¹

Conclusions

A number of perspectives on quality as applied to higher education have been inventoried and expressed in this report, all in support of strengthening the ability of colleges and schools to articulate their value to students, communities and society.

Though it may seem obvious, it is worth reiteration that quality and accountability driven by institutional impetus and managerial self-interest have perhaps the greatest value and durability.

Quality and effectiveness metrics as defined by the institution are not exclusive or inherently contradictory to indicators of accountability shared with the public, policy makers or funding sources. To the degree they overlap, there is a natural common ground on which to build more systemic and perpetual measurements of effectiveness.

While acknowledging the resistance of education performance to pure scientific analysis, harnessing the plethora of data, empirical evidence and other emerging resources is a high priority for accreditors and other students of quality assurance. At minimum, the emphasis on data-driven accountability and evidence-based practices will lead to better understanding and improved models of what is effective and what is sub-optimal. Furthermore, when the culture of evidence-based, data driven quality and accountability is integral to the operations of the institution, rather than imposed, the gains in performance will be more profound and recurring.

Yet questions and risks manifest as well. New technological tools and data solutions are having a profound effect on the ability of higher education to identify metrics for change, assess what makes a difference and focus on improving performance targets. If more robust, empirical, data-driven quality assurance standards and reviews drive substantial (or even transformative) change of institutional performance, will the broader community adapt to the new models of delivering education? And will accreditation’s current standards endure or will new ones be required that will defy recognition by those calling for more accountability?

Leadership will confront these issues and others as it attempts to identify trends and needs, lead change agendas, invest in what makes a difference, and remain authentic and courageous. Those attributes of strength will serve higher education well while confronting the challenges of a dynamic environment.

Dr. Linda Baer has served for more than 30 years in executive positions in higher education including Senior Program Officer in Postsecondary Success for the Bill & Melinda Gates Foundation, Senior Vice Chancellor for Academic and Student Affairs in the Minnesota State College and University System, Senior Vice President and Interim President at Bemidji State University. She recently served as the Interim Vice President for Academic Affairs at Minnesota State University, Mankato. Her ongoing focus is to inspire leaders to improve student success and transform institutions for the future. Recent publications have been on smart change, shared leadership, innovations in higher education, transforming in an age of disruptive change, and analytics as a tool to improve student success.

Citations

¹Measuring Up: The National Report Card on Higher Education, National Center for Public Policy and Education.
² ED.gov; http://collegecost.ed.gov/9010info.aspx.
³ACICS Accreditation Criteria, April 2015, Section 3-1-100, pg. 37.; www.acics.org
⁴National Report Card on Higher Education, 2008.
⁵Complete College America http://completecollege.org/about-cca/.
⁶Ibid.
⁷Shray, Vicki. Xxxp. 6
⁸Ewell, Peter. 2002 Assessment, Accountability and Improvement. National Institute for Learning Outcomes Assessment
⁹Burke, Joseph C. Many Faces of Accountability in Achieving Accountability in Higher Education, Jossey Bass, 2004. P.1
¹⁰Carey, 2007. 1-3.
¹¹Spellings Commission on the Future of Higher Education xx.
¹²Carey, 2007. 1-3.
¹³NCHEMs state-based data systems.
¹⁴Evidence-Based Policy-Making Act of 2014 http://www.edcentral.org/evidence-bill.
¹⁵Lingenfelter, 2003.
¹⁶Berdahl.
¹⁷Burke, Burke, Joseph C. Many Faces of Accountability in Achieving Accountability in Higher Education, Jossey Bass, 2004. 4.
¹⁸Bogue and Hall.
¹⁹Burke, Joseph C. Many Faces of Accountability in Achieving Accountability in Higher Education, Jossey Bass, 2004. pgs. 12-14.
²¹Burke, Joseph C. Many Faces of Accountability in Achieving Accountability in Higher Education, Jossey Bass, 2004.
²¹Burke, Joseph C. Many Faces of Accountability in Achieving Accountability in Higher Education, Jossey Bass, 2004. 10-12.

References

ACICS, How Effective Accreditation Supports Innovation in Postsecondary Education
Link Here

Baer, Duin and Bushway. 2015 change Agent Leadership. Planning for Higher Education.V43n3 April-June 2015.

Berdahl, R. “Academic Freedom, Autonomy and Accountability in British Universities.” Studies in Higher Education, 1990, 15(2), 169-180.

Bound, J., M. Lovenheim, and S. Turner. 2007. Understanding the Decrease in College Completion Rates and the Increased Time to the Baccalaureate Degree. Population Studies Center Research Report 07-626. Ann Arbor, MI: University of Michigan Institute for Social Research Population Studies Center. Retrieved April 1, 2015 from the World Wide Web: www.psc.irs.umich.edu/pubs/pdf/rr07-626.pdf/.

Bogue, E.G. and Hall, K.B. Quality and Accountability In Higher Education: Improving Policy, enhancing Performance. Westport, Conn: Praeger, 2003.

Burke, Joseph C. Many Faces of Accountability in Achieving Accountability in Higher Education, Jossey Bass, 2004.

Carey Kevin. 2007. Truth Without Action: The Myth of Higher-Education Accountability. Change. September-October 2007. http://www.changemag.org/Archives/Back%20Issues/September-October%202007/full-truth-without-action.html.

Complete College America http://completecollege.org/about-cca/.

McCann, Clare and Amy Laitinen. 2015. Bipartisan Bill for Evidence Could Improve Access to Higher Education Data. Retrieved from the World Wide Web May 12, 2015. http://www.edcentral.org/evidence-bill/.

Dolence & Norris, (1997). Transforming Higher Education. Society for College and University Planning.

Ewell, Peter. 2002 Assessment, Accountability and Improvement. National Institute for Learning Outcomes Assessment.

Frey, R. and K. Parker. (2012). Record Shares of Young Adults Have Finished Both High School and College. Washington D.C.: Pew Research Center. Retrieved April 1. 2015 from the World Wide Web. www.pewsocialtrends.org/2012/11/05/record-shares-of-young-adults-have-finished-both-high-school-and-college/.

Hess. F. (2011) Old School: College’s most Important Trend Is the Rise of the Adult Student. The Atlantic, September 28 Retrieved April 1. 2015 from the World Wide Web. www.theatlantic.com/business/archive/2011/09/old-school-colleges-most-important-trend-is-the-rise-of-the-adult-student/245823/.

Hurley, Daniel. (2015) AASCU Top 10 Higher Education State Policy Issues for 2015
https://mail.google.com/mail/u/0/#search/daniel+hurley+/14abfd1f1bdffde2.

Horizon Report 2014. New Media Center. NMC http://www.nmc.org/publications/2014-horizon-report-higher-ed,

Kanter, Rosabeth Moss (2001). Evolve! Succeeding in the Digital Culture of Tomorrow. Harvard Business School Press.

Kashner, J.B. (1990). Changing the corporate culture. In D.W. Steeples (Ed.), Managing change in higher education (pp. 19-28). New directions for higher education, Vol. 71. San Francisco: Jossey-Bass Publishers.

Lingenfelter, Paul, 2003.”Educational Accountability: Setting Standards Improving Performance” Change, 2003, 35(2), 19-23.

Merisotis. J. P. 2015. Higher Education Must Change to Reflect Shifting Student Demographics. The Hill, January 2015. Retrieved April 1, 2015, from the World-wide web. http://thehill.com/blogs/congress-blog/education/228795-higher-education-must-change-to-reflect-shifting-student.

National Report Card on Education, authored by The National Center for Public Policy and Higher Education 2008. Retrieved from World Wide Web. May 12, 2015.
http://measuringup2008.highereducation.org/print/NCPPHEMUNationalRpt.pdf Higher
NCHEMs report state-based data systems.

Norris, Donald. Robert Brodnick, Paul LeFrere, Joseph Gilmore, Linda Baer, Ann Hill Duin and Stephen Norris (2013) In Transforming in an Age of Disruptive Change in Planning for Higher Education V41 N2
http://www.academia.edu/2785351/Transforming_in_an_Age_of_Disruptive_Change.

Rowley, Daniel James Herman D. Lujan, & Michael D. Dolence (1998) Strategic Choices for the Academy: How Demand for Lifelong Learning Will Re-Create Higher Education Jossey-Bass.

Sandru, Ioana Maria Diana. 2008. Dimensions of Quality in Higher Education: Some Insights into Quality-based Performance Measurement. Synergy. Vol 4. No.2/2008. Retrieved form the World Wide Web. May 12, 2015.
http://synergy.ase.ro/issues/2008-vol4-no2/06-dimensions-of-quality-in-higher-education-some-insights-into-quality-based-performance-measurement.pdf.

Schray, Vickie. Assuring Quality in Higher Education: Recommendations for Improving Accreditation. https://www2.ed.gov/about/bdscomm/list/hiedfuture/reports/schray2.pdf.

The Test of Leadership. Charting the Future of Higher ducation Spellings Report on Higher Education. 2006.http://www2.ed.gov/about/bdscomm/list/hiedfuture/reports/final-report.pdf.

Suskie, Linda. 2015. Five Dimensions of Quality. Jossey-Bass.

World University Rankings 2014-2015 http://www.timeshighereducation.co.uk/world-university-rankings/2014-15/world-ranking.