Tracking the Effectiveness of Charities

Competition for funds leads to rise in ‘performance-based’ evaluations

For decades, the Michigan Association of Rehabilitation Organizations and other non-profit groups have sought money from Congress by arguing that every new dollar produces corresponding social benefits.

But, laments David Price, the association’s executive director, “there was nothing to back that up,” a problem that has been intensifying as more and more federal and state lawmakers demand accountability from programs that receive government money.

Now, he says, “we realize that if we were going to say stuff like that, we’d better be able to prove it.”

A similar challenge confronts many charities these days: how to demonstrate that the work they do produces results that can be quantified and verified, not only for government officials but also for an increasingly skeptical universe of grant makers, private donors, and non-profit clients.

The method of choice for the Michigan Association of Rehabilitation Organizations and a growing number of charities is something called “outcome-based evaluation,” an approach that uses statistics, surveys, and other assessment tools to measure the results — not just the content — of a charitable program.

The Michigan association is now participating in a test project, along with about 50 local affiliates of Goodwill Industries International and rehabilitation groups elsewhere, to help create what has never existed before: national performance standards for non-profit organizations that provide rehabilitation and job-training services.

Declares Mr. Price, “We want to be able to say that we’re not just a feel-good thing.”

Many charities have a similar desire, but performance evaluation isn’t without risks or controversy. Conducting an accurate study of a program’s results can cost thousands of dollars and sometimes requires the expertise of outside consultants who may not have a thorough understanding of a charity’s mission. And if overused or poorly implemented, performance-based evaluation techniques can produce misleading results and undermine support for worthwhile programs.

“It’s not as simple as falling off a log,” says Gordon Raley, executive director of the National Assembly of National Voluntary Health and Social Welfare Organizations. “We stand the risk of losing some very good programs or having their impact diminished if we’re not very careful.”

Still, while acknowledging the dangers, more and more charity executives are embracing performance-based evaluation as a way to satisfy donors and to improve non-profit programs. Some groups are developing strict numerical methods to measure results, while others are using less-formal guidelines.

Some examples:

* Boys and Girls Clubs of America last month offered its 2,000 affiliates a variety of evaluation tools, including a questionnaire that youngsters can answer by computer. The answers are used to measure whether the charity’s programs are effective in nine areas — from developing youths’ “self-identity” and morality to helping children acquire skills that would be useful in their adult lives.

* The Y.W.C.A. of the U.S.A. this month is starting a nationwide assessment of youth-leadership programs and how they affect teen-age-pregnancy rates. The charity is sending a questionnaire to all its affiliates nationwide with the aim of identifying 20 that have especially promising youth-development programs. The Y.W.C.A. will use focus groups with young people and staff members, plus other methods, to evaluate those 20 programs, and will then find ways to spread effective techniques nationwide.

* Catholic Relief Services recently established rigorous evaluation and record-keeping standards for its health care, agriculture, and small-loan programs across the globe. The charity plans to use the information to improve programs and show donors that their money is being spent effectively.

* Big Brothers Big Sisters of America is putting the finishing touches on a charitywide effort to measure whether its volunteers make a difference in the lives of children and adolescents. The organization is gathering information through questionnaires given to volunteers, parents, children, and, in some cases, children’s teachers.

* Family Service America is designing a plan that it hopes its network of independent affiliates will use to measure the results of their social-service programs as well as those of the National Association of Homes and Services for Children, with which F.S.A. is merging.

Evaluation efforts in the non-profit arena are not new. For decades, government officials have pushed universities, hospitals, and other tax-exempt groups to document the effects of their work.

But the call for accountability is accelerating throughout the charity world.

United Way of America recently started a large-scale effort to teach charities how to measure the results of their programs, and it is encouraging local United Ways to require performance data from the charities they support.

“The increasing competition for charitable dollars leads to the need to make a case for what your organization does,” says Martha Taylor Greenway, a specialist in performance evaluation at the United Way in Atlanta. “And the best way to make that case is to talk about the benefits to the target population or to the community.”

While grant makers and private donors have long demanded greater accountability from charities, a variety of other forces also is driving the evaluation trend, experts say.

For one thing, corporate executives who sit on non-profit boards are urging charity managers to adopt sound business techniques, including independent assessments of their long-term plans and client services.

Government also is emerging as a key factor in the push for accountability. Millions of public dollars are flowing to non-profit groups that are assuming social-service duties because of changes to the welfare system. Government officials at all levels want proof that the money is producing measurable results.

In Maine, for example, state agencies negotiate specific performance goals with social-services charities they support, then require the groups to document the difference their work made.

Many experts believe that the accountability trend for non-profit groups doing business with government will intensify in coming years. “It’s almost certain that between the federal, state, and local governments and United Way, non-profit organizations are going to be at the very least encouraged, and very possibly pressed hard, to track outcomes,” says Harry Hatry, a scholar at the Urban Institute, a Washington think tank.

Yet proving the effectiveness of charitable programs isn’t easy, and crafting a solid evaluation program can pose huge challenges. Nowhere is that more evident than in the pioneering efforts of the National Results Council, which has attracted the Michigan Association of Rehabilitation Organizations and about 70 other disability and employment-training organizations in 27 states to participate in its pilot assessment program.

The council was founded in 1995 by Robert Walker, a veteran program-evaluation consultant, with support from the JM Family Foundation. It now has received money from sources that include the JM and Scaife Family Foundations, companies such as Pfizer and Philip Morris, and the U.S. Department of Education.

The council, whose fiscal 1998 budget is $485,000, is just now beginning to collect significant data from participants in the pilot project, and it is too early to draw conclusions from the information, says Bill Niederloh, the council’s chief executive. But the council’s aim, he says, is to develop a “common benchmark” for rehabilitation groups across the nation, using data collected from the organizations themselves. Ultimately the council hopes to distribute a scorecard that executives, donors, and clients of disability and job-training organizations can use to determine how well individual organizations meet those standards.

“If you can’t compare programs, then it’s very difficult to say you’re in the evaluation business,” Mr. Niederloh says. “There has been a hidden reluctance to ask the ultimate question: How successful are we as an industry and as an individual organization in applying a common, national standard?”

A sophisticated computer model developed by the council is supposed to adjust for local variables — a high local unemployment rate, a hot high-tech market with plenty of jobs for disabled people, or a caseload of severely handicapped clients — that can unfairly raise or lower an organization’s score.

Much of the data collected from organizations is basic — clients’ work history, level of disability, educational background, and whether there are financial circumstances, such as the availability of welfare money, that could discourage the person from finding a job.

But the program is designed to be more than a head count. It also asks rehabilitation groups to track what happens to clients after they complete an employment-training program: Did they find work? How many hours a week are they employed? Do they keep the job for at least 120 days? How much do they earn?

But for all the council’s high hopes, Mr. Niederloh suggests that it is facing formidable hurdles.

For one thing, none of the groups participating in the pilot project have yet agreed to make their scores public.

Kenneth Shaw, chairman of the National Results Council’s board and a local Goodwill executive in Florida, concedes that “there is some anxiety of having a scorecard when you’re in the bottom third without a chance to do some internal correction.” Most groups want a chance to do better before their scores are publicized, he says.

Rhonda R. Dolan, assistant director of employment and training services at Goodwill Industries International, says that public disclosure of participating Goodwills’ evaluation scores “is not going to be a short-term possibility.” Much work remains in insuring that the council’s numbers are accurate, meaningful, and clear to lay audiences and that they reflect the hurdles local groups face, she says.

Still, executives of some participating Goodwill affiliates, while noting that the national charity has always measured program results, say the pilot project already is proving to be useful.

“It’s an eye-opener for us,” says Beth Robertson, vice-president of employment and training at the Goodwill in Akron, Ohio. Ms. Robertson says her group decided to change the content of its Work-Adjustment Program, which helps prepare people without job experience or skills for employment, after collecting data for the National Results Council project. Goodwill managers discovered that many people weren’t employable even after they completed the work-adjustment program, Ms. Robertson says. “We are finding a lot of drug abuse and a need for counseling and behavior management,” she says. Now, the Goodwill is reshaping the program to accommodate those needs.

Resistance to publicizing evaluation results is not the council’s only obstacle. It also is grappling with how to insure that disability groups give a true accounting of how much they spend on programs.

“We struggle with the issue,” Mr. Niederloh says. Even though council officials try to verify data, “any system is subject to certain kinds of gaming,” he says.

Without accurate cost information that is comparable from one local charity to another, it is impossible to measure an organization’s efficiency or establish reliable national benchmarks.

While the National Results Council tries to work out its procedural bugs, many experts are voicing caution about the limits and drawbacks to program evaluation in the charitable arena, especially to approaches that use numerical indicators.

While counting the number of abused children seen by a non-profit counseling agency may be easy, measuring a counselor’s progress in raising a child’s self-confidence can be more difficult.

“It’s reasonable to ask programs to develop some quantifiable indicators,” says Jennifer Greene, an evaluation expert at Cornell University. “But we can’t stop there, because we truly miss a lot of what is likely to be important.”

Another concern is that government officials and grant makers may rely too heavily on accountability measures in making decisions on which charities to support.

Though she is a proponent of results-based evaluation, Ms. Greenway, of the United Way in Atlanta, says the approach “is not a silver bullet for weeding out good from bad.” She worries that state and local government officials, pressed to quickly disburse federal dollars to local charities to provide welfare services, will make rash decisions about a charity’s merit based on incomplete or misleading data.

“Often legislatures or elected officials who have little time to understand these issues have unrealistic expectations,” she says.

Many experts also say that reliance on a scorecard approach to accountability could induce a charity to skim off the easiest-to-serve clients, making itself look more effective than it is and leaving the tough cases for other charities.

Despite such limitations and pitfalls, the trend of marrying competition with compassion seems here to stay. As Mr. Price of the Michigan Association of Rehabilitation Organizations says, “Down the road, we’ll all have to hold ourselves up to standards, because in fact we’re all using public dollars.”