NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Robinson KA, Akinyede O, Dutta T, et al. Framework for Determining Research Gaps During Systematic Review: Evaluation [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2013 Feb.
Framework for Determining Research Gaps During Systematic Review: Evaluation [Internet].
Show detailsSeven EPCs evaluated the Research Gap Framework and submitted 8 evaluation forms (one EPC submitted completed evaluation forms from two different project teams). (Evaluation form is in Appendix B.)
We first provide a summary of the quantitative questions in a table. For each question asking for further details, such as a description of disadvantages, we include the text submitted with the EPCs and projects de-identified. For these questions we have added a column (JHU Response) that includes notes about changes to framework or instructions made in response to the comment(s) or a response, as appropriate. We have also indicated if the form was completed by a team applying the framework during a systematic review (SR) or applying the framework retrospectively during a future research needs project (FRN).
Summary of Responses to Evaluation Form (n, %)
| Question Number | Question Text | Number (n=8) No. (%) |
|---|---|---|
| Q5 | Stage sheet was completed | |
| Systematic review | 3 (37.5%) | |
| Future research needs document | 5 (62.5%) | |
| Other | 0 (0%) | |
| Q7 | Who completed research gap framework worksheet | |
| P.I. only | 4 (50%) | |
| Other investigator only | 1 (12.5%) | |
| Other investigator and Research staff member | 1(12.5%)) | |
| Research staff member only | 1(12.5%) | |
| Other: team feedback | 1 (12.5%) | |
| Q8 | EPC had previously completed gap identification | |
| No | 0 (0%) | |
| Yes | 8 (100%) | |
| Q10 | Advantages to using framework vs. previous gap identification method | |
| No | 0 (0%) | |
| Yes | 8 (100%) | |
| Q12 | Disadvantages to using framework vs. previous gap identification method: | |
| No | 1 (12.5%) | |
| Yes | 7 (87.5%) | |
| Q14 | Problems or issues when using framework vs. previous gap identification method: | |
| No | 2 (25%) | |
| Yes | 5 (62.5%) | |
| 1 (12.5%) no answer | ||
| Q16 | Suggestions to improve framework sheet efficiency/usefulness: | |
| No | ||
| Yes | 3 (37.5%) | |
| 5 (62.5%) | ||
| Q18 | Suggestions to improve framework sheet instructions: | |
| No | 5 (62.5%) | |
| Yes | 2 (25%) | |
| 1 (12.5%) no answer | ||
Q9Describe previous gap identification method
| Form | Stage Completed | Description of Gap Identification Method |
|---|---|---|
| A | FRN | <XXX>; A Future Research Needs report was also undertaken to systematically prioritize research gaps in the areas of <XXX>, and to develop a list of research questions to address the prioritized gaps based on the systematic review. |
| B | FRN | Previously, we would review the comparative effectiveness report to determine the number of studies and quality (strength) of evidence to determine the potential research gaps. High quantity + high quality (no gap); high quality + low quantity (no gap); |
| C | SR | All our reports have a section that identifies gaps. The earlier reviews tended not to be organized around PICOTS. |
| D | SR | Future Research Needs for the <XXX>. |
| E | FRN | Have had other Future Research Needs Projects (Different Investigators) |
| F | FRN | <XXX> Future Research Needs report <XXX> Future Research Needs report Numerous systematic reviews |
| G | SR | I'm not sure what you mean by “identified gaps from systematic reviews”. We regularly write a future research needs section but I'm not sure if these are the same thing. |
| H | FRN | Yes, this was our third FRN project in addition to the research gaps sections of prior reviews. |
Q11Describe advantages of using framework
| Form | Stage Completed | Description of Advantages |
|---|---|---|
| A | FRN | Facilitate the use of a systematic process to identify evidence gaps. |
| B | FRN | This framework provides standardized criteria to identify potential gaps in the literature, which was previously somewhat arbitrary. |
| C | SR | THE PICOTS framework assists writers in considering all areas. |
| D | SR | Systematic, transparent way; involvement of different stakeholder groups |
| E | FRN | The coding / having a list of reasons for the gaps is helpful; but we do not want this to be part of the protocol because we are not sure how to use it and what it adds to the process. |
| F | FRN | Requires you to be more systematic |
| G | SR | I can see advantages to using a framework such as this. Without a framework our approach has been fairly non-systematic and may be influenced by priorities of the research team or driven by what they see as the “most important” gaps. |
| H | FRN | Yes – a structured approach is helpful for constraining the content to researchable topics. It also helps to see where there are redundancies, and helps keep the research team focused on the scope of the project. |
Q13Describe disadvantages of using framework
| Form | Stage Completed | Description of Disadvantages | JHU Response |
|---|---|---|---|
| A | FRN | _____ | _____ |
| B | FRN | We found that applying this framework to all potential combinations of PICOS for the <XXX> FRN project yielded more than 1000 research gaps. This was due to the large number of populations, settings, and intervention/comparisons and the overall poor quality of the existing literature. The framework is much more practical when there is a manageable number of potential PICOS combinations (e.g. FRN for <XXX> project). | We have added text to the instructions suggesting that teams discuss prior to the use of the framework whether to, and how to, lump or split. For instance, it may be more manageable to abstract gaps by class of intervention and comparison. |
| C | SR | The overlap with GRADE is less helpful as it is not clear how the gap will assist in the judgments of the SOE. For example a gap in research design….not all issues can be addressed with trials….so not sure how to make this link. Also I think the list of reasons for developing a gap should be expanded. I found I used B2 very often to provide a reason for the gap….and the recommendations are widely varied. Thus some categories are not discriminating enough. | We did not consider the process of identifying gaps as a way to assist in making judgments about SOE. The framework was designed to leverage work completed, if SOE was assessed. We do not see how B2 could be made more specific. It is to be selected if the body of evidence was considered at high risk of bias (this may be for a number of reasons, but is aggregated across the studies). Use of the framework does not preclude providing more details. |
| D | SR | Involvement of different stakeholder groups may be not representative; information about ongoing studies may be incomplete ; no full representation of the NIH, other funding agencies; the role of industry is unclear | We have clarified at the beginning of the instructions the purpose for the framework – to identify and characterize gaps from systematic reviews. How to solicit stakeholder involvement and prioritize gaps is beyond the scope of this work. |
| E | FRN | The gaps are not clearly conveyed by the table. The statement in the instructions that “other elements will be apparent from the key question” does not seem to be accurate to us. At the FRN point it is almost too late; too difficult to use. It may have been more helpful during the CER. | We have added some discussion of this, including examples, under Characterization of Research Gaps in the instructions. We agree that there are different challenges in applying the framework retrospectively versus while completing a systematic review. We have added text in instructions providing some suggestions about how to proceed if doing so retrospectively. |
| F | FRN | Not all gaps are equally important, so it is not an efficient use of time to be required to complete this chart for every gap. Suggest that only the critical research gaps be prioritized for the chart. It may be too cumbersome for readers to understand. Many of the codes may need to be listed for each gap. It is not clear that using the codes, as opposed to a narrative description, will make the gaps easier or more efficient to understand. | It is unclear how one would determine the ‘most critical’ gaps without first systematically identifying and characterizing the gaps, such as through use of this framework or other method. The codes and worksheet were developed to aid in abstraction. The future research needs section of the systematic review, or future research needs document, would present the gaps. Our previous report provided a suggested presentation format (also another EPC has produced a report on how to present gaps). |
| G | SR | The key disadvantage I see is that it may replicate work already done. This may be less of an issue if it was done alongside preparing the results, as was suggested in your instructions. I did it after the review was complete so I found it fairly redundant, as much of this information was already in SOE or summary tables. I'm not sure that it highlighted any issues that were not already known, i.e., very few studies providing data for the same comparisons & outcomes. So it could add a lot of work without providing much additional insight. | We would hope that a team could leverage the work done in completing SOE but we take your point that it could also be redundant. We think this will depend on the team, the specific review, and the timing of applying the framework. |
| H | FRN | To some degree it can be overly constraining and it really doesn't work well for a review topic on which there is very little available. In this case, the overwhelming gap is that much more research needs to be done, period. Trying to specify at the level of the framework is not yet possible or appropriate. Also, the framework is not ideal for methodologic issues. | We agree that the framework may be too granular to use for questions for which, essentially, the entire question is a gap. We have added some text about these sorts of decisions to the instructions |
Q15Describe problems or issues faced when using framework
| Form | Stage Completed | Description of Problems or Issues | JHU Response |
|---|---|---|---|
| A | FRN | _____ | _____ |
| B | FRN | The only problem is the same as the disadvantage. | See above. |
| C | SR | I wasn't sure what would be helpful to you in the free text and notes. | We have added text addressing these sections on the worksheet. |
| D | SR | Complicated, does not address strength of existing evidence. | We feel that the strength of existing evidence is explicitly considered in the reasons for gaps. Further, we have tried to link the reasons for gaps with the various domains used in different systems to rate the strength of existing evidence. |
| E | FRN | Not sure the table format adds much value to the process. Seems like we would have to shoe horn items into the table and get little added value from the exercise. Not sure how to complete the PICOTS sections for the types of gaps we identified. | We are not sure of current process used by this EPC team (Q9), so do not have a basis for responding to how use of the framework to identify research gaps in a systematic manner might add value. We have added text to the instructions to clarify characterization of research gaps using PICOS elements in worksheet. |
| F | FRN | _____ | _____ |
| G | SR | For the review I used, we had many, many comparisons (22 drug-drug comparisons for two different conditions within each of 5 key questions) and many outcomes within each of the questions; for many comparisons and outcomes there were very few studies. Therefore, I found the framework rather cumbersome to use. The other challenge was when the outcomes weren't graded. Within the review I used, we only graded outcomes for 2 of the key questions, so for the outcomes (which were numerous) within the other key questions, we had no SOE assessments. So then the reason for gaps was usually A1 (no studies) or A2 (limited number of studies). | We have added to the instructions a discussion of lumping/splitting which, we think, would help in the situation described. We added to instructions decision about whether to review all questions and outcomes, even if not ‘graded’. |
| H | FRN | See question above – it worked for the review in question (<XXX>), but not for another review (<XXX>) that started from all insufficient. | We agree that the framework may be too granular to use for questions for which, essentially, the entire question is a gap. We have added some text about these sorts of decisions to the instructions. |
Q17Suggestions for improving usefulness and efficiency of framework
| Form | Stage Completed | Suggestions to Improve Framework | JHU Response |
|---|---|---|---|
| A | FRN | _____ | _____ |
| B | FRN | This framework is designed very well for specific projects that contain a manageable number of research gaps. However, for the <XXX> FRN project where literally every combination was determined to be a research gap. It would be impossible to ask expert Stakeholders to evaluate such a large number of research gaps and to then build consensus on prioritization. As exemplified in this project, I do not believe this tool is appropriate for use in all FRN projects and use should be determined on a case-by-case basis by discussion between the investigative team and the TOO. | We have clarified at the beginning of instructions where we envision this framework fitting within the work of a systematic review and future research needs project. We have added text describing decisions to be made about which questions and outcomes to consider (only those assessed for strength of evidence?) and dealing with multiple interventions/comparisons (lumping versus splitting). |
| C | SR | Might be easier to complete in an excel sheet where some responses can be constrained. | We agree. We completed abstraction for this project using forms in Distiller. We have added a note about this option in the instructions. |
| D | SR | Research and development framework (used by industry) can be applied using complete information about completed and ongoing studies ; electronic surveys of the representative groups of sponsoring organizations; policy makers, researchers, and consumers (NO “patients”); survey should address group specific interests (implications for funding, research methodology, policy, consumer interests) | We think these comments relate to other aspects of developing a research agenda and are beyond the scope of this project. |
| E | FRN | If we are going to have a table, it might be more useful to state the gap, then code the reason and the PICOTS issues. I am not sure how the table format is supposed to aid in either making conclusions or communicating them to readers. What is most important? Having gaps with the same reason? Having gaps related to a PICOTS element? Is the table supposed to help you summarize across gaps? | The worksheet was designed to aid in identification of research gaps. The future research needs section of the systematic review, or future research needs document, would present the gaps. Our previous report provided a suggested presentation format (also another EPC has produced a report on how to present gaps). |
| F | FRN | Do not think it is necessary to have a separate chart for each key question Instead of “serial number” suggest calling it “gap number” | The gaps are abstracted by question, and characterized by listing the elements of PICOS from the question where evidence is inadequate. Because of this explicit link to questions, each review question should have a worksheet. The alternative is to use the PICOS to flesh out the entire research question needed to address the gap. We have changed the column header to “gap number”. |
| G | SR | As I have alluded to above, it will likely be most efficient to incorporate it at an early stage in the review. It may also be most efficient to focus on some key comparisons or questions or outcomes. Since you mention using the SOE information, should it be based on or driven by the “graded” outcomes? | We have added text about applying the framework retrospectively versus during completion of a systematic review. We have added text to instructions suggesting team make decision about whether to limit consideration and abstraction of gaps to those questions and outcomes that were assessed for strength of evidence. |
| H | FRN | _____ | _____ |
Q19Suggestions for improving framework instructions
| Form | Stage Completed | Suggestions for Improving Instructions | JHU Response |
|---|---|---|---|
| A | FRN | _____ | _____ |
| B | FRN | The instructions were extremely clear. | Thank you. |
| C | SR | _____ | _____ |
| D | SR | Depends on the changes in the research and development framework | _____ |
| E | FRN | Provide one or two examples of a completed table. Describe how the table can or should be used and clarify what the purpose is. | We have provided, embedded in instructions, some examples to illustrate specific points. We have appended to end of instructions an example of a completed sheet. |
| F | FRN | _____ | _____ |
| G | SR | I found the instructions clear. As I mentioned above, many of the comparisons and outcomes were not graded, therefore the instructions “Work completed in grading the body of evidence should be used in completing this worksheet” are not relevant. What do we do when grading has not been done? | We have revised the text in this section to address this question. |
| H | FRN | _____ | _____ |
- Detailed Analysis of Evaluation of the Use of the Research Gaps Framework by Evi...Detailed Analysis of Evaluation of the Use of the Research Gaps Framework by Evidence-based Practice Centers (EPCs) - Framework for Determining Research Gaps During Systematic Review: Evaluation
Your browsing activity is empty.
Activity recording is turned off.
See more...