Making a Measurable Difference

Creator of rating tool seeks to improve charity effectiveness

When the Latin American Youth Center started a program two years ago to combat domestic violence, it had high expectations. For 10 weeks, men in the program heard from charity employees who advised them that violence wasn’t an appropriate way to express machismo or love.

The group-counseling sessions seemed to run without any major hitches.

But what the Washington charity learned from interviews with participants before and after the program’s completion was deeply disturbing. The male participants had become more violent, their behavior reinforced by their peers in the counseling group. After some soul searching, the organization was able to overhaul the program and make it more effective.

Influencing Giving Decisions

But such emphasis on frequent evaluations of programs is uncommon in the nonprofit world. A new effort gets under way this month to change that.

Steve Butz, a former charity employee turned software entrepreneur, is leading an effort to encourage social-service charities to assess their performance. It has attracted attention from prominent players in the nonprofit world — many of whom have witnessed philanthropy’s numerous failed attempts to spur better measurement of results.

His goal is not just to help social-service groups operate better but to stop donors from giving blindly to organizations, unaware if the groups are producing positive results. To that end, he has created a “social-investment rating tool” to give donors information about which charities are the most effective.

Mr. Butz, who grew up here, traces his interest in evaluating charities’ performance to a counseling job he held shortly after college, working with troubled teenagers. What Mr. Butz saw in his work bothered him deeply, not only the stories of young people struggling to stay out of jail, but the sense that the charity had little clue as to whether it was making a difference to the youths in its care.

“The measurements we were doing were either completely lacking or irrelevant,” he says. The charity took attendance, but no one examined whether participation was going up or down. “We would look only at how many kids made it through the program,” he says. “But that didn’t feel relevant to me.”

Such experiences set Mr. Butz, who left nonprofit work eight years ago to start the software company Social Solutions, on a search for ways to improve how social-service groups operate.

Mr. Butz says his rating tool could ultimately steer money away from mediocre charities to stronger groups, something that the crumbling economy has made all the more urgent.

Setting a Standard

Mr. Butz, 39, argues that existing ratings such as those done by the watchdog group Charity Navigator, which examines nonprofit groups based on financial information they provide to the Internal Revenue Service, tell donors nothing about whether their dollars are used to achieve results.

“Right now, you can find information about which organizations are doing the best job of raising money,” he says. “But you can learn little about which groups are doing the best job for the people they serve.”

Mr. Butz hopes his investment tool can become the “industry standard” for donors who want to ensure that their dollars are making a difference, leading them to approach their giving with the same care a venture capitalist uses to support start-up companies. The tool is based on 26 questions that measure charities according to how effectively they provide services and improve people’s lives.

He has assembled a group of 16 leaders in the nonprofit world — Diana Aviv, president of Independent Sector; Paul Brest, president of the William and Flora Hewlett Foundation; Robert Egger, president of the D.C. Central Kitchen; and Brian A. Gallagher, president of the United Way of America, among others — who will meet later this month to discuss how to improve the tool and whether it has a future.

Ideally, Mr. Butz would like to see GuideStar, the Web site that publishes financial information about charities, make the tool and its ratings available free to donors. Bob Ottenhoff, GuideStar’s president and a participant in this month’s meeting, says he is “very interested” in the idea.

He says that donors have a growing appetite for information about charities’ performance. A majority of wealthy individuals (58 percent) said they would give more if they could determine the impact of their donations, according to a 2006 survey by the Center on Philanthropy at Indiana University.

Previous Efforts

Many nonprofit experts say Mr. Butz’s goals are admirable but that it is too soon to know what, if any, difference the assessment tool will make. He is certainly not the first to try to rate charities based on their effectiveness.

Just in the past two years, two young former hedge-fund employees created a nonprofit group, GiveWell, to assess charities in different fields. The Center for High Impact Philanthropy at the University of Pennsylvania, the Urban Institute, and a charity called the Nonprofit Reporter are working on similar projects.

But while the nonprofit world has been searching for a way to rate the impact of charities for at least a decade, says Eric Kessler, an adviser to wealthy donors, so far no one has figured out how to do it in a large-scale, easily adoptable manner.

“If you could, in fact, put GuideStar on steroids and really address the impact side of the equation, so you’re not just making it easy to give but making it meaningful, that could have a huge impact on how philanthropy works,” says Mr. Kessler, founder of Arabella Philanthropic Investment Advisors. But he says that many previous efforts have come and gone.

Mr. Butz and his backers argue that the assessment tool he wants to use is less complicated than some past efforts.

He says that answering the tool’s questions should take only about 20 minutes. (A draft of the questions is online.)

The tool looks at, among other things, whether groups are collecting and analyzing data, whether they change course if their programs aren’t producing positive results, and whether they are consistently meeting their goals for helping people.

Informing Donors

One big unknown in Mr. Butz’s effort is how to use the tool to rate thousands of social-service groups and make the results available to donors. Mr. Butz says that charities might conduct the evaluations themselves, at least at first, something that would not require a big outlay of money.

But he would prefer to have an independent group get involved. David E.K. Hunter, a consultant and former director of evaluation and knowledge development at the Edna McConnell Clark Foundation, who helped Mr. Butz create the assessment tool, envisions hiring a team of professional evaluators, a costlier endeavor.

Mr. Butz plans to discuss that and many other questions at the meeting of advisers later this month. He has called the informal group the Alliance for Effective Social Investing.

Even after leaving the nonprofit world, Mr. Butz, as head of the company Social Solutions, continued to observe charities that couldn’t prove whether they were helping people. He grew concerned that donors were channeling money not only to weaker groups, but to organizations that were doing actual harm.

Mr. Butz cites the Latin American Youth Center, which uses his company’s performance-management software, to illustrate his point that charities can cause harm if they do not take care to track their work and modify it when necessary.

The center “had a pre- and post-assessment that caused them to stop that program and restructure it,” says Mr. Butz. “But the problem, of course, is that this isn’t how most nonprofits operate. Most of the time, they get a grant, and a year later they do an evaluation, and that can mean anything. Sometimes it’s a half-page summary of how many people finished the program. That’s a problem.”

Because of the youth center’s rigor, it was able to use the findings to reshape its domestic-violence program. The group brought in local experts on domestic abuse who taught staff members how to deliver their message in a serious way that was respectful of the cultural viewpoints of the Latino participants in the program.

“If we hadn’t had a system in place to look at the effectiveness of the program, we’d probably be doing it in the same form today,” says Isaac Castillo, director of learning and evaluation, who was hired three years ago to help the charity better assess its work.

Record-Keeping Burden

Mr. Butz’s effort to rate charities could prompt criticism from some nonprofit groups. Collecting the kind of exhaustive information necessary to perform well can be time consuming.

Nonprofit groups may also be concerned that Mr. Butz’s rating tool would do a better job of analyzing their data-collection abilities than their effectiveness.

Mr. Castillo, of the Latin American Youth Center, says he initially worried that larger organizations, which tend to be more sophisticated about collecting data, might score better.

But he says he was encouraged by how Mr. Butz’s rating tool tracks progress over time, giving charities a chance to improve.

“Organizations that can show an upward trend over time will carry as much weight to potential funders as someone who has a whole army of researchers in house,” he says.

Sean Stannard-Stockton, principal of Tactical Philanthropy, which advises wealthy donors, and an adviser for Mr. Butz’s effort, says he thinks it is important to ask all charities, no matter how small, to collect data on their programs’ performance.

“I manage a small business myself and I’m acutely aware that a small organization simply doesn’t have the data collection or analysis ability that a Goldman Sachs or a Morgan Stanley does,” he says. “Yet even small for-profit or nonprofit organizations should and can do data collection on a scale that’s appropriate to them.”

He also says that past efforts to evaluate charities’ effectiveness have faltered because they have tried to measure the value of a dollar spent on different types of groups, pitting soup kitchens against foster-care centers.

“One of the problems we’ve had in the past is trying to compare nonprofits in different program areas, and this bypasses that by saying, Let’s look at the organization itself,” he says.

Spurring Competition

If Mr. Butz’s assessment tool takes off, his software company could benefit. But Mr. Butz says that has never been his intention and that he would be happy if someone else took over the effort after the November meeting.

He also says a result of the meeting might be more competition for his company.

“There’s as much chance that we will have revved up the competition as anything else,” he says. “I would love that outcome too; I would love to get lots of people fired up about performance management.”

Mr. Butz also emphasizes that he’s not trying to stop donors from supporting charities that do not score well at first. He just wants donors to track their donations over time, the same way they would an investment in a risky company.

In fact, Mr. Butz says only about 2 percent of human-service organizations would score well across the board if the rating system was put in place today. He dubs those groups “blue-chip investments,” and says donors can give to them with complete faith that their dollars are achieving a positive result.

Donors should give to other charities, of course, but need to track the groups’ progress over time.

If the charities don’t improve over a period of years, the donors should consider dumping their investments. “Our message is not to stop investing in the 98 percent of organizations,” says Mr. Butz. “It’s just please know, Mr. $1-Million Check Writer or Ms. Foundation Head, that if you write the check, it comes with some amount of risk. And with that risk comes responsibility.”

ABOUT STEVE BUTZ, FOUNDER, THE ALLIANCE FOR EFFECTIVE SOCIAL INVESTING

Nonprofit employment: Mr. Butz started volunteering for the Chesapeake Bay Foundation while a student at Loyola College in Maryland. After graduation, he worked at two social-service organizations in his hometown of Baltimore, the Learning Bank and the Living Classrooms Foundation.

His company: In 2000, Mr. Butz founded Social Solutions, a software company that help charities collect and analyze data about the programs they operate.

Other nonprofit efforts: While working as a counselor to troubled teenagers at the Living Classrooms Foundation, Mr. Butz started an effort to reward high-achieving workers with a cash bonus of up to $10,000.

His interest in performance-based pay also led him to create the Superstar Foundation in June. The foundation will give cash awards to high-performing social-service workers.

Mr. Butz named the awards after his mother, who he says wanted to serve in the Peace Corps and work as a nurse but “got hunkered down with five kids.” He says: “I owe a lot of my social bent, or whatever you want to call it, my conscience, I don’t know, to my mom.”