About Academic Return on Investment
While academic return on investment (A-ROI) is an ideal form of cost effectiveness measure, school districts do not commonly use it. It is very likely, however, that school districts may need to migrate towards more rigorous cost effectiveness measures, such as A-ROI, as state and federal governments, grantors, and other entities in the enabling environment for school districts require more rigorous metrics. Given this possibility, this appendix describes A-ROI in greater detail.
Defining Academic Return on Investment (A-ROI)
A-ROI is a tool used to emphasize cost effectiveness in budget decisions and help decision makers make more informed choices between different potential uses of resources. The basic formula for A-ROI appears below.
A-ROI = ((Learning increase) x (Number of students helped))
Benefits of Academic Return on Investment
The key benefit of A-ROI is that it provides a clear measure of whether or not services are having their intended impact for a reasonable cost. More fundamentally, A-ROI helps districts evaluate expenditures in terms of whether or not tax dollars are used effectively.
Underlying Principles of Return on Investment
A-ROI helps incorporate the following of principles and policies into the budget process:
- Not all money spent with the intent to help children learn is necessarily effective.
- All money spent should lead to positive outcomes.
- Spending $1,000 to help a student learn is better than spending $2,000 for similar gains in learning.
- While some strategies may deliver a learning increase, these strategies also may be too expensive relative to other options to deliver similar increases.
- Money that isn’t being used effectively should be redirected to other purposes. Legal restrictions on funding may constrain the extent to which ineffective spending or activities can be curtailed, but A-ROI helps encourage decision makers to curtail them to the extent possible.
- A-ROI can help create alignment between district goals and financial resource utilization.
Calculating Academic Return on Investment
The most challenging part of the A-ROI calculation is the benefit produced by the service, often defined as the increase in student learning (e.g., years’ worth of growth that occur in a single year). Benefits can be estimated in the following ways:
- Run controlled tests. Especially germane to newer activities, the activity can be implemented as a “pilot” or “experiment” where the students benefiting from the activity are compared with a control group to estimate the level of benefit available.
- Look for natural experiments. Natural differences between how students are served within the district can provide an opportunity to observe the potential benefits from expanding an activity from one part of the district to others. However, a district must consider student demographic and other characteristics when observing a natural experiment to arrive at valid conclusions.
- Make formative assessments. Student progress (e.g., learning growth) can be evaluated after participating in the activity.
- Establish and monitor other salient measures. For example, an activity intended to prevent dropouts should have a noticeable impact on the dropout rate.
Costs are in the denominator of A-ROI. Costs should be “full costs” which include, at a minimum, the wages and fringe benefit costs of the employees performing the activities. If there are significant direct costs for equipment, outside contractors, etc., those costs should, ideally, also be included.
When to Use and Not to Use Academic Return on Investment
A-ROI is an ideal form of cost effectiveness evaluation and can be applied to a variety of different activities from specific professional development efforts, to small schools, to remediation and intervention. However, it is not applicable in all situations. Here are some guidelines for applying A-ROI judiciously.
- Apply A-ROI strategically. A district should not attempt to calculate A-ROI for all or even most of its services. It is likely that the time, cost, and effort needed to calculate A-ROI on such a broad scale would outweigh the benefit to decision making. Rather, A-ROI should be applied to programs that are high-profile, high-cost, or otherwise of special strategic significance.
- Apply A-ROI going forward, not backward. Gathering historical data to calculate A-ROI can be very challenging. Districts will likely find it more beneficial to design the systems necessary to capture the required information for A-ROI going forward and then analyze A-ROI information once it becomes available. While this will require the district to wait for solid A-ROI information up to a year after the data collection systems have been implemented, it will result in much more accurate and credible information and will conserve the district’s analytical resources.
- Plan ahead to use A-ROI. A-ROI analysis must be planned six to nine months ahead of when resource allocation decisions are made. Ad hoc A-ROI studies performed in response to the need to make budget cutbacks will not work. Calculating A-ROI needs to be a separate analysis step that occurs before resource allocation decisions are made.
- Do not apply A-ROI to services without direct academic benefit. Some services that a district provides are important but do not link directly to student learning. Support services like payroll, transportation, or food services are illustrations of this. Rather than attempting to estimate an indirect benefit to learning for these services, a district should use unit costs and performance measures, particularly measures of service outcomes, to evaluate cost effectiveness.