Excerpt

The Government Performance and Results Act (GPRA), enacted by Congress in 1993, requires that all federal agencies evaluate and report on the results of their activities annually.

Evaluating federal research programs in response to GPRA is challenging because we do not know how to measure knowledge while it is being generated, and its practical use might not occur until many years after the research occurs and cannot be predicted. For example, today's global positioning system is the result of research conducted 50 years ago in atomic physics. In 1999, the National Academies Committee on Science, Engineering, and Public Policy (COSEPUP) addressed this issue for research programs in its report Evaluating Federal Research Programs: Research and the Government Performance and Results Act. That report indicated that federal research programs could be evaluated by a process it called expert review that makes use of three evaluation criteria: quality, relevance, and leadership. Expert review is more than traditional peer review by scholars in the field. It also includes the users of the research, whether they are in industry, nongovernment organizations, or public health organizations or are other members of the public who can evaluate the relevance of the research to agency goals.

This followup report, by the COSEPUP Panel on Research and the Government Performance and Results Act 2000, describes the panel's analysis of how federal agencies that support science and engineering research are responding to GPRA. The panel decided to focus its work on the five agencies that provide the majority of federal funding for research: National Science Foundation (NSF), National Institutes of Health (NIH), Department of Defense (DOD), Department of Energy (DOE), and National Aeronautics and Space Administration (NASA).

Contents

• The National Academies
• Panel on Research and the Government Perfromance and Results Act (GPRA) 2000
• Committee on Science, Engineering, and Public Policy
• Preface
• Acknowledgments
• Executive Summary
• 1. The Challenge of Evaluating Research
  • 1.1 Barriers to Evaluating Research and the Solution
  • 1.2 COSEPUP's Evaluation Criteria
  • 1.3 Organization of this Report
• 2. Agency Methods
  • 2.1 Expert Review
  • 2.2 Evaluation Criteria
  • 2.3 Human Resources
  • 2.4 Aggregation
  • 2.5 Validation and Verification
  • 2.6 Summary
• 3. Communication Issues
  • 3.1 Communication Between Agencies and Oversight Groups
  • 3.2 Communication by Agencies with User Groups and the Public
  • 3.3 Communication by Oversight Groups
  • 3.4 Communication Within Agencies
  • 3.5 The Issue of Timing
  • 3.6 Summary
• 4. Conclusions and Recommendations
  • 4.1 General Conclusions
  • 4.2 General Recommendations
  • 4.3 Specific Recommendations
  • 4.4 Summary
• Appendixes
  • A Panel and Staff Biographical Information
  • B White House and Congressional Correspondence
  • C Summaries of Agency Focus Group Presentations
  • D Summary of Workshop
  • E Executive Summary of Evaluating Federal Research Programs: Research and the Goverment Performance and Results Act
  • F Government Performance and Results Act
  • G Federal Agency GPRA Web Sites

COSEPUP is a joint committee of NAS, NAE, and IOM. It includes members of the councils of all three bodies. For more information on COSEPUP, see www​.nationalacademies.org/cosepup.

This material is based upon work supported by the National Science Foundation under Grant No. OIA-0073616, the National Research Council, the Department of Defense under Purchase Order SP4700-99-M-0510, the Department of Energy under Grant No. DE-FG02-00ER45803, the Department of Health and Human Services/National Institutes of Health under Contract No. N01-OD-4-2139, Task Order No. 60, and the National Aeronautics and Space Administration under Grant No. NASW-9937, Task Order No. 112. Additionally, this project, N01-OD-4-2139, Task Order No. 60, received support from the evaluation set-aside section 513, Public Health Service Act.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, the Department of Defense, the Department of Energy, the Department of Health and Human Services, or the National Aeronautics and Space Administration.

NOTICE: This volume was produced as part of a project approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences (NAS), the National Academy of Engineering (NAE), and the Institute of Medicine (IOM). It is a result of work done by a panel of the Committee on Science, Engineering, and Public Policy (COSEPUP). The members of the panel responsible for the report were chosen for their special competences and with regard for appropriate balance.