Program managers need to be aware of a potential disconnect between the desire to use mid-term reviews as an opportunity to “down-select” some of the projects and the desire to keep hands-off funded research projects. It is important to not micro-manage research from the funding organization. Micro-management of research from the funding agency is never a good approach to keep projects “on-track”.
When a review panel understands that a review will result in the termination (“down-select”) of some of the projects in a program, they will take a far different view of the project report than that taken by a NSF program, for example. In fact, the nature of the review will be similar to that of an original proposal being reviewed for initial funding. Typically observed is a concern with the level of direction that projects receive regarding what should be in a mid-term review and what reviewers need in order to offer advice on whether the program should continue the project or not.
In many instances, the mid-term report is written in a perfunctory manner, failing to address important questions that the panel reviewers needed in order to make judgments as to whether or not the project should be continued. In many cases, information that was lacking included:
- Honest discussion of what actually was already accomplished versus what was proposed;
- If objectives have changed, there should be a discussion of what has changed along with reasons for the changes;
- Information on papers in progress as well as listing of those already accepted for publication, if any; and
- Listing of all project participants, including students, and how they interact to accomplish the overall goals of the project – i.e., information that would show that the project is being managed appropriately and effectively.
Effective management and a reasonable management plan behind it are most often the primary causes of failed research projects involving more than one primary researcher. Collaboration is very difficult to achieve without such a plan. Most researchers have their own agendas and prefer to use funds to continue those agendas rather than support the needs of a larger collaboration.
One of the most frequently discussed issues during a review is the number and quality of publications made available for review. In projects that have been underway less than two years, it is probably inappropriate to require a listing of published journal articles. It normally takes at least a year or more for an article to be reviewed and published in a major scientific journal, so such a requirement for this stage of review is probably inappropriate. Such papers, when they do appear, appear to be based on work done prior to the funded project. It is often more useful in mid-term reviews to see a list of conferences attended and papers presented at international conferences as a measure of progress as well as a measure of international stature.
With appropriate guidance from a program, mid-term reviews of projects can be conducted fairly and effectively because such guidance ensures that the reviewers will have the information they need to give useful advice to the program.
