Part III: Benefits, Reviews And Measurements: The Other Digital Divide

In this section, my tone changes noticeably from one of tentative researcher to one of the weary veteran. I make comment of various excellent sources which more than amply verify my assertions, but in truth this is a topic that I have been familiar with and engaged in for some time on a professional level.

Diverse Stakeholders, Diverse Interests

The question of measurable benefits to program participants must be approached from various angles. This is due in large part to the fact that a large portion of programmatically-oriented not-for-profits are not and have not ever performed the kind of reviews or evaluations that would clearly support such analysis. Moreover, a similar large portion are not capturing the types of data that would enable them to do so. This has not been a secret in some circles and it's a topic that's been pursued by several organizations throughout the not-for-profit sector. In the case of the Digital Divide, the issue was rather poignantly and immediately distinct from an early point. The many new partnerships that had been formed as a result of the Digital Divide (specifically wherein the partner was a technology company) had brought very disparate worlds together very quickly. Technology companies tend to do what they are good at, and sophisticated database technology now existed for very modest prices. Generally, all one needed to do was to upload the normalized data and get to work. Getting to work at "new-economy" speed, a young solutions developer might quickly find that this type of data seldom exists in any one place, and may not exist at all. This is due to many factors. One is that there are pretty divergent views on what constitutes a review or a measurement of success. Private grant funding institutions want to see one type of data, the corporate partner wants another, the program administrator might have an ad-hoc system that they've developed over time.

The funding institutions frequently get to call the shots on what they consider to be result metrics for particular programs and this would seem to be enlightened policy in light of the above assertions. These same criteria have a tendency to fall far from the sort of criteria that those who would be served actually require, which is also no secret in CCI and relief circles. In this event it's up to someone to help the funding organization redefine their metrics or what ends up happening is the administrator bends the program to the grant or they bend the results to the metrics. There is also the well-cited nuance that most grants do not offer funding for evaluation or programming, this cost must be borne elsewhere. The fact that many not-for-profits tend to become directed by grant availability rather than their mission statements is a tempting but irrelevant discussion.

Reviews conducted by programmatic organizations or ones that fund them are hopefully geared to quality of service or cost analysis as much as to PR and marketing, and the corporate partner wants quick and easy numbers that look good on financial reports: how many did we "effect" this quarter, is it up from last quarter? How would one determine this if one doesn't have a daily headcount? How would one define "effected" or what-have-you in something as subtle as an education program that involves flyers and posters? If a computer is given to one child in a household, is it reasonable to assume that other children in that household will be "effected"? Almost everyone up this food chain wants pictures of smiling children, happily engaged in the improvement of their lives and these are almost universally available.

Fundamental Disparities in Resources and Capability

Another basic problem is that most program administrators have little human resources to conduct the level of non-programmatic activity that persons in other sectors tend to take for granted--let alone the technology resources. Professional in other sectors tend to assume a that certain level of 'corporate' or systematized hierarchy exists right down to the ground, but this is frequently not the case. Many on-the-ground programs have a volunteer base and so a different relationship exists. As such, requests for action or information that would seem cut-and-dry for any other sector sometimes fall into the soft zone of community volunteers. For a person of corporate background, such "flakiness" is indicative of an organization that doesn't have it's act together when quite the opposite is frequently the case.

There is even a kind of protectiveness that develops in certain not-for-profits on behalf of their local partners, almost as if to shield them from inquiry. This is due in great part to the sensitivity that organizations with on-the-ground operations evolve. It's hard to understand in print, but once you actually go and see a program in action, you have an immediate, intuitive understanding of the benefits, plus a whole lot more. You not only are infected by the energy of people working to change their lives, but you get a sense of the level of sophistication or corporatization that you might expect. How might corporate professionals perform if the roles were reversed? Say perhaps if the field instructed the finance department to hang new sheetrock, install new appliances and put new carpeting down in their offices by Friday noon? I've been the heart-touched person who learns that a reporting requirement casually thrown into an email has kept numerous poor field-office souls up for nights and days ransacking paper files and hand-tabulating results. These persons are probably volunteers trying to help their communities. As with any community-oriented program, the picture looks very different depending on what side of the line you stand. For the parents of under serviced children, to the activists, volunteers and not-for-profits who service them, the magic happens one child at a time within whatever conditions prevail. It is certainly not about statistics and accountability as far as the child participant is concerned, not should it ever be. It should also be mentioned from this perspective, that local program staff or volunteers often rightfully feel a great degree of enfranchisement over goings-on in the program environment. This is known to be a positive response from community members who are encouraged to develop local solutions using local resources. From the ground-level perspective, if you were the person who had painted the walls, and gotten volunteers to help you install the technology, and other voluneers somehow pull together a CAC with funding from various sources, it can result in a barrage of inquiry on multiple levels which means one and only one thing to you: overhead.

It has often been suggested that not-for-profit institutions underwent a process of "liberalization" in the 1970's that undermined good administrative and executive practice (C. McCormac et al). While this may be the case for organizations which existed in that day, a great many new not-for-profits working across multiple sectors manifest the same challenges. This being an information society, the issue just keeps coming up. For those of us in the tech sector, we believe that we can reasonably assume many continuing benefits to not-for-profits as a result of continuing acceptance of technology and the methodologies that the PC "revolution" have inspired. Several organizations have taken on the challenge on various levels. Basically, the approaches take their models either from business in a standard project management approach or from the CCI or Comprehensive Community Initiative crowd who attempt to answer the issue of intangibility of certain benefits or from academia. My purposes require an academic answer, because the answer will in fact require answering several questions. We must satisfy the careful academic chain of causality. Can we define what level of access to technology is meaningful to children for example? Can we demonstrate that a certain group of children have had this level of access? Do computers actually help children in a way that can be measured in a way that gives us reason to believe that the indicators did not rise due to another factor? The latter is a hot debate indeed. Common sense and what we see on the ground indicates that it's not about parking kids in front of computers, it's about computers as a part of the curriculum in a supported environment. There's been a problem achieving consensus on what good curricular integration is because teachers are still struggling to catch up to their students and the technology was morphing faster, for a while there, than anyone could keep up with. We've done a great job of qualifying and characterizing, we've got best practices evolving nicely and anyone could agree that the early results point to astounding possibilities. The various stakeholders must still work through this question, however. Funders must become more sensitive to the requirement that locally-derived criteria by developed for the construction and evaluation of programs and to devote aspects of funding for evaluation. Program administrators must recognize and apply themselves to the many methodological resources that exist to them online and elsewhere, in the not-for-profit sector and elsewhere. Technologists must brace themselves for the initial impact of prevailing norms in not-for-profits, become warmed by the humanitarian glow and learn to work in a more grass-roots paradigm which assumes alternative means to achieve equivalent effect.

Recommendations for Change

With this considered, my recommendation is for a new approach to program inception which insures consensus amongst these top-level stakeholders by involving all of them in an inquiry phase and subsequent determination of funding and metrics. Specifically I propose a trial wherein a grant be initiated by either the grass roots, the umbrella or the granting entity, voted upon and accepted by all three, metrics and goals subsequently determined by all three. It may be presumed that the granting foundation will ultimately bear responsibility for the dispersion of funding, but by applying a rough set of parameters such as proposed by the Morino Institute (see below) with a flexible percentage, the stakeholders can then work within those rough guidelines as preliminary data is assessed, objectives determined, solutions identified, project requirements understood and metrics/test methodologies are developed. For development purposes, I would recommend the split of funding being between even thirds and perhaps a 50-30-20 percent split between pre-production (i.e. analysis, discovery, requirements/solutions), production (i.e. deployment, execution) and post-production (evaluation, testing). The difference between my wording and Morino's can be accounted for by understanding that their presumption deals with the organization implementing the program whereas mine seeks to deal more with how grants are conceived and administered. A natural division of primary responsibility may exist where the umbrella partner may assumes primary responsibility for project deployment, the grass-roots partner for project maintenance and the granter for evaluation. One of the many advantages of this system may be reduced overall cost of service to recipients as the natural "stopping points" built into this "projectized" format allows stakeholders to halt the development process in pre-production phase if flaws are discovered and before further expense is incurred. I should admit that this proposal may put significant burden upon the granting institution on the surface, but the supposition that this may require granters to become expert in the many fields over which they offer grants is unnecessary. The challenge for the granter, in my opinion, will be to find the time and traveling expense that such will entail and also to convert the interest of their present decision makers.

Sources and Recommended Links

The Morino Institute, Access to Outcomes stresses the oft-made point that locally-grown solutions often have the only chance of succeeding in distinct communities (. "No matter how impressive the technology or how well-intended the motives, technology initiatives imposed on a community by outsiders are often ineffective"), and offers numerous recommendations from working with community leaders on local or existing focal issues to community advantage. Partnerships with tech sector are necessary, in their view, to assist not-for-profits, and proposes that any initiative should be 1/3 technology, 1/3 training and 1/3 programming.

The Aspen Institute, New Approaches to Evaluating Community Initiatives Volume 1 Concepts, Methods, and Contexts Edited by James P. Connell, Anne C. Kubisch, Lisbeth B. Schorr, and Carol H. Weiss

The Canadian Centre for Philanthropy, A Review of Evaluation Resources for Nonprofit Organizations, March 1999, Sandra L. Bozzo Michael H. Hall

An attempt to categorize available review tools into three categories: program logic models, empowerment, participatory and collaborative models and the balanced scorecard. The categories explained in detail, strengths and weaknesses of each explained, resources for tools are listed. A good, if dated, literature review.

Urban Institute, INDEPENDENT SECTOR, Outcome Measurement In Nonprofit Organizations: Current Practices and Recommendations, Elaine Morley Elisa Vinson Harry P. Hatry, 2001 The smoking gun. An extensive survey of methodological application in the sector. Although the sample size of this study would not permit us to make generalized observations on national trends, it is nonetheless the most comprehensive survey of the issue that I could find. Given the preponderance of supportive material such as I list here and exists in massive quantities elsewhere, the issue could be considered to be common knowledge at this point.

The Non-Profit Pathfinder, Portal page maintained by the Benton Foundation and Independent Sector represents an ongoing response to the issue by building a forum and medium for dissemination & sharing of best practices.

The Measures Project, a great online resource.

The Pew Partnership For Civic Change, Coming Of Age In The Information Age July 2000 Jacqueline Dugery, Juan Sepúlveda Carole Hamner. Examines the results of focus groups conducted with not-for-profits which centered on data and information. What they find are gluts of marginally useful or externally oriented information but relative absences of information that would enhance programming either in quality or cost of service. This paper goes into great depth to explain why this remains a problem for not-for-profits and characterizes how focus group participants used technology in their own daily routine. Most gratifying to me personally is it's attention to the issue of legacy IT systems in not-for-profits, frequently mainframe systems that represent massive sunk cost (or would if they were to be replaces), which have historically been the backbone of their IT departments, or more commonly in this case still might dwell under the aegis of the Finance Department. The user acceptance issue is elucidated eloquently. I would add that the training requirements for these systems are very different from those of the PC "revolution" which creates a further challenge to change.

The James Irvine Foundation, Changing methods - and minds -on the use of evaluation in nonprofit organizations A look at WOW: Working On Workforce development project: Creating a Culture of Inquiry July 2001, Georgiana Hernández and Mary G. Visher. Notable for it's nearly explicit, almost comical treatment of the issue as the frustration of the reviewers seeps into the report of what sounds like a very helpful project.

The United Way of America, Outcome Measurement Resource Network represents a growing trend in some more established and well funded not-for-profits to share expertise and best-practices.

The Mitsubishi Electric America Foundation Creating and Sustaining Project Impact: Guidelines for Evaluation and Dissemination demonstrates a very straightforward, mainstream approach from the business world with a particular emphasis on iterative quality and course-corrections.

Annie E. Casey Foundation, Jobs Initiative Policy Report Benchmarking the Performance of Employment and Training Programs: A Pilot Effort of the Annie E. Casey Foundation's

Jobs Initiative Doug Welch, Abt Associates Inc.1 March 2001 is a report of a successful review and benchmarking of a workfare assistance programs in several national locations. In a very straightforward fashion, the report breaks operations and value flow through various stages of assistance and tracking of persons entering the job force after having been serviced. The report not only derives a very business like cost-of-services-per-unit, in this case individuals, but tracks benefits through time in terms of job retention rates.

Community Technology and Community Building: Early Results from the Camfield Estates-MIT Creating Community Connections Project Randal Pinkett Epistemology and Learning Group MIT Media Laboratory November 8-11, 2001. A gem of a study showing increased academic performance amongst students with online access as compared to those with no access. Taking place in multiple US urban locations, the study was comprised of 50% experiment classes and 50% control classes. All classes received the same curricula but the experiment classes had access to online resources including an educational website. Benefits were tracked across various social and community-oriented factors in what appears to be a highly successful study of a CCI technology project. Among the most interesting nuances of this study were the findings that many program participants, residents of a housing project, felt that much internet content did not appeal to them, or did not suit their interests. Pinkett goes on to postulate a "critical mass" threshold for certain communities or populations which if achieved would tend to reinforce and expand community technology penetration. An easy anecdote if email: if no-one that you know has access, to whom will you send email? Conversely, Pinkett argues that once a certain critical mass of an individual's social circle attains access, there will be a corresponding positive inducement to obtain access. I find Pinkett to have been particularly successful in creating metrics and requirements that seem truly community-based. This methodology forms a precedent for CCI evaluations at the very grass-roots level.

Forward to next page
Back to Contents
Back to top