It must be masochism on my part, but I’m looking at writing yet another grant. This grant would go to the Heartland Institute for Clinical and Translational Research and would be for a pilot study.
I received notification of this grant by email, but I can’t find the call for proposals on the web. A list of awardees from the previous year is listed herefro1.
I’ll quote liberally from the email attachment.
The type of research supported by this program includes T1 translational research (i.e., early bench to bedside research involving human subjects such as Phase I drug trials or similar first-in-human projects), clinical research (i.e., studies using human subjects, families, and/or communities for testing new treatments or interventions, assessing new health outcomes, etc. See NIH definition of clinical research at end of this RFP under Appendix.), and/or T2 translational research (i.e., studies of effectiveness and/or on implementation and/or dissemination of clinical research findings into practice). Basic science and animal model research is not suitable for this RFP, although animal work will be considered if it is part of an overall project that primarily uses human materials.
The letter of intent is due on December 1 and the application is due on January 9, 2012. The limit on this grant is $30,000 and no indirect dollars are awarded. Part of the money can be used for salary, but note the following restrictions:
Faculty and research assistant salaries are permitted. However, no more than 10% of the PI’s time (salary) should be allocated to the budget, and if grant funds are used to provide release time, that time must be matched by departmental release time—e.g., if 10% of PI’s salary is charged to the grant, another 10% must by donated for a total of 20% effort dedicated to the proposed project. NOTE: The actual monetary value of donated effort must be documented in the letter from the investigator’s departmental chair. As noted above, the program encourages the development of proposals that promote collaboration between Frontiers investigators from different disciplines and/or from different institutions, and proposals that provide home departmental / institutional / organizational matching or contributed funds.
The proposal has a limit of five pages.
My tentative project title is “Monitoring trends in the peer-reviewed literature for best practices in risk adjustment.” Here’s a brief summary.
Almost all medical research relies on the use of appropriate statistical methods. But the translation of new statistical methods into research practice is an imperfect process that is poorly understood.The goal of this pilot project is to build the capacity to evaluate changes in research practices through a large carefully selected sample of research papers. We are asking for funding of a pilot study that will collect information on a convenience sample of 180 research publications involving case-control, cohort, and randomized trials across three broad medical areas (cardiology, nursing, and alternative medicine). A panel of reviewers will extract information from the convenience sample on the use of risk adjustment and compare these to best practices in risk adjustment. The pilot study will test sampling procedures, estimate resource requirements, and demonstrate the feasibility of monitoring trends in risk adjustment methods over a broad time frame. This information will help prepare an NIH grant to conduct a large scale study using a rigorous sampling process. This pilot study will also help establish a framework for examining the trends in the use of other best research practices in a representative sample of research publications.
Specific Aim #1. Explore search strategies across three medical disciplines (cardiology, nursing, and alternative medicine) using three common research designs (case-control, cohort, and radomized trials). The search strategy for the pilot has the modest goal of finding two relevant papers for each of the six decades from the 1950s forward. But the long range goal is to develop a strategy that can identify all available publications meeting certain medical and statstical criteria.
Specific Aim #2. Evaluate the effort required to abstract information about risk adjustment methods and assess the reliability of these abstraction efforts. Each paper will have information on the risk adjustment method evaluated by two independent quantitatively trained experts (e.g., statisticians, epidemiologists) with some experience in the medical discipline of the paper.
Specific Aim #3. Evaluate the appropriateness of a compensation system for abstracting information. Volunteers for this pilot study will be given $20 for each paper that they are asked to evaluate, and we will assess through time logs and their personal impressions whether this is an appropriate compensation.
The inspiration for this proposal comes from a study of research practices in schizophrenia that appeared in the British Medical Journal in 1997. Two researchers identified every published research article on schizophrenia appearing between 1947 and 1996. They identified over 2,600 publications and successfully summarized the results of the first two thousand papers in their list. With only two researchers, you can’t fault them for stopping and leaving the last 600 publications untouched. The researchers identified four major problems in schizophrenia research: sample sizes that were much too small, evaluation time frames that were much too short, too much focus on institutionalized patients, and too many different outcome measures. On the last point, the statistics are quite shocking. Researchers used 639 different outcome measures across these 2,000 studiens and 369 outcome measures were used in one publication and then never again.
While the findings of this study were quite helpful in identifying shortcomings in the research approach used in the study of schizophrenia, the effort required by these two researchers was something that would be difficult to reproduce. The goal of this pilot grant is to build the capacity for a distributed effort in reviewing the peer-reviewed literature for best statistical practices. Thie pilot study will focus on best practices in risk adjustment and will examine three medical specialties, cardiology, nursing, and alternative medicine. With the assistance of several medical librarians, we will identify a convenience sample of research studies in each of these specialities across a broad time spectrum (1950 to the present). In each of the decades from the 1950s to the 20000s, we will find, if possible, two examples of case-control designs, of cohort designs, and of randomized trials.
The studies must involve human patients and must have a binary endpoint as one of the primary outcomes. In cardiology, the endpoint must be either overall mortality or some cause specific mortality. In nursing the endpoint must be one of the nursing sensitive indicators identified by the American Nurses Association. For alternative medicine, any binary endpoint will be used.
For the pilot study only, the articles will be restricted to English language only. This does introduce some biases, but the goal of this research is not to produce a bias-free sample, but rather to evaluate the complexities involved in producing that sample.
The three medical specialities were chosen to evaluate the process of identifying articles in an area which is reasonably well defined (cardiology), somewhat broader in scope (nursing), and very diffuse (alternative medicine). There is a trade-off between having a broad base of research publications for any evaluation and having a narrow enough selection criteria to allow for an effective search strategy. By evaluating the search strategy in cardiology, nursing, and alternative medicine, we hope to gain some practical insights into these trade-offs.
The search will try to identify studies with three major research designs: a case-control design, a cohort design, and a randomized trial.
The search will also try to produce studies in each decade from the 1950s onward. The broad time frame is intended to examine the barriers to searching associated with older studies. Studies published before the advent of PubMed, for example, may require different approaches for searching.
The goal of the pilot study is to try to find at least two published studies for each of the six decades using one of the three research designs and in one of the three medical specialties. If the search is successful, we will have a convenience sample of 263*3=108 papers. It is quite possible that some combinations may not be found, especially in the earlier decades. The inability to find any publications of a paricular type in earlier decades will provide useful information about setting reasonable time frames for future large studies.
The researchers conducting searches to identify qualifying studies will share their search techniques with others on the team and try to develop a strategy that could be applied to a larger scale study where the effort was to identify all studies meeting a particular criteria and then sampling randomly from those studies.
Once the papers are identified, two copies will be shared with quantitatively oriented researchers (e.g., statisticians or epidemiologists) who have worked in the medical specialty. They will independently review whether the paper meets all the appropriate inclusion/exclusion criteria and discuss any discrepancies with the search team.
They will then abstract information from the papers about the use of risk adjustment. Risk adjustment is a methodology with substantial variation in its application and major inconsistencies with how it is reported. It is also a practice that has evolved over time, so there is great interest in comparing the publication of seminar theoretical papers on risk adjustment such as Rosenbaum and Rubin’s famous Biometrika paper in 1983 on propensity scores with their adoption as an applied tool in research studies. Finally, there is a need to identify the prevalence of use of methods of risk adjustment that have been shown to be flawed, such as stepwise selection in the risk adjustment model or binary categorization of risk factors.
Risk adjustment can take several forms: matching, covariate adjustmment, propensity scores, case-mix index.Researchers will identify whether risk adjustment was done, and then select the method that was used. A list of all commonly used methods will be available and the data entry would allow for the selection of more than one method (e.g., matching based on propensity scores) as well as an open ended field to identify any risk adjustment method not included in the list.
The researchers would also note the variables used in risk adjustment and provide an assessment as to whether the risk adjustment was incomplete or inadequate, either through the omission of critical variables in the risk adjustment model or the use of risk adjustment variables which were measured inaccurately or inappropriately.
Agreement between the two raters will be assessed using Cohen’s Kappa and papers with substantial disagreements will be reviewed by the principal investigator to see if the process of abstracting information from these studies needs to be defined more carefully or see if there are particular features of some papers that makes them difficult to get consistent results from multiple reviewers.
I will update this webpage as I progress in filling out the details of this grant application.
You can find an earlier version of this page on my original website.