NREPP is Dead! How Does SAMSHA Make NREPP or Its Successor Better?
March 17, 2018
March 17, 2018
Morning Zen Guest Blog Post ~ Dennis D. Embry, PhD ~
My breakfast cereal is science. My bedtime reading is science. I even read science on the beach. It’s a strange addiction.
Good science can be the difference between life and death. Ten years ago, I was diagnosed with a very serious melanoma. I was on www.pubmed.gov in seconds, once I had the diagnosis, I researched the heck out of it and changed my treatment as a consequence. This past December was ten years since that diagnosis, and I’m here with no melanoma.
On any given day, I probably check out at least several things in the National Library of Medicine (PubMed), PscycINFO, or both on prevention, intervention, and treatment of psychiatric disorders, addictions, violence, etc. In grad school, I did the literature searches for about 300 grad students and faculty. It was my version of the New York Times crossword puzzle, only better. I got smarter by looking things up, and some would say, a smart ass.
Good science about mental, emotional, and behavioral disorders can mean life and avoid tragedy. I know that only too well as a clinician working with children or adult survivors whose abuse histories would make one want to throw up. I know that because friends and colleagues were killed or injured at the Safeway store on January 8, 2011—after I came from teaching 18 SAMSHA sites how to prevent to prevent such tragedies.
On NREPP, it’s clear that many newer reviewed programs are far weaker in science than legacy programs and practices. The newer reviews provide no numerical or coded ranking, like a Consumer Reports rating. The NREPP legacy programs do have such numeric rankings. Still, there are deeper, structural issues on the NREPP site that are not evident to the casual user related to limitations in the database programming that is 2-3 decades old.
Consider an example limitation that is a relic of web construction and search-engine logic easily 20 years old. If one searches “randomized control,” there is one hit: “Resources for Enhancing Alzheimer’s Caregiver Health II (REACH II).” Or, search “longitudinal,” which yields no results. The same is true of “comparative effectiveness” trials or for the word “replication.” Does this mean that some 600 programs or practices on NREPP are based on junk science? No, rather something is faulty in the database engineering, and probably a historic cost-cutting procedure and limitation of database software of several decades ago.
If one goes to the National Library of Medicine (www.pubmed.gov), similar searches are much richer. A search for “prevention” AND “mental health” and “randomized” yields 3,055 citations to sort through. If you search randomized and some specific programs like “Good Behavior Game” or “Triple P” one has some heavy reading to do: There are 27 citations for the Good Behavior Game, and 102 for Triple P (Positive Parenting Program). If treatment is your gig, a search of “Contingency Management,” there 418 citations. Acceptance and Commitment Therapy (ACT) has 320 citations that included randomization. All four of these strategies are on the legacy side of NREPP, representing approximately 1,000 randomized trials in prevention and treatment. That’s hardly “junk science” as implied by critics during Congressman Murphy’s hearings several years ago.
There are a couple of issues here that must be remediated: 1) programming and search capacity, and 2) limitations on the number of studies allowed to be submitted for review, which was five on my last submission, despite many more experimental studies and replications that have only expanded since.
Are there flaky things on NREPP? Yes. Are there gold-standard, randomized-control, comparative effectiveness trials with long-term, preserved follow up on NREPP? Absolutely, but you would have to be a complete geeky scientist like me find them. What should SAMHSA assistant secretary and policy lab director do to really improve its ratings and recommendations for potent evidence-based practices?
First, design a good search engine would be a good start and linking all the sources for the submissions to www.pubmed.gov. Once that happens, then it becomes much easier for consumers of scientifically proven prevention, interventions and treatments of mental, emotional, and behavioral disorders to find the ones that really work. This alone will cost millions of dollars.
Second, adopt review standards like those found in the “Standards of Evidence” report 1 and report 2, published by the Society for Prevention Research or by the Institute of Medicine (IOM) reviews used in the 2009 Institute of Medicine Report on the Prevention of Mental, Emotional, and Behavioral Disorders Among Young People.
Third, review all the relevant studies for scientifically valid, specific practices—if SAMSHA leadership and the Administration are truly committed to good science, rather than limiting the reviews to a few studies or cherry-pick things promoted by high-paid lobbyists. The reviews must be independent, and not by SAMSHA employees lest it becomes colored by politics or lobbying.
Remember most of all: The United States of America has funded the very best science on the prevention, intervention, and treatment mental disorders in the world. And Heaven knows, we need to use that science for the general welfare of Americans of all ages.
* * * * * *
Dennis Embry, President/Senior Scientist at PAXIS Institute – Dennis D. Embry is a prominent prevention scientist in the United States and Canada, trained as clinician and developmental and child psychologist. He is president/senior scientist at PAXIS Institute in Tucson, Arizona. Dennis Embry serves on the scientific advisory board for the Children’s Mental Health Network and the U.S. Center for Mental Health Services Advisory Council.