Innovator Interview: Paul Guerin, University of New Mexico Institute for Social Research

Paul Guerin, Ph.D. is the Director of the Center for Applied Research and Analysis in the Institute for Social Research (ISR) at the University of New Mexico. He has worked at the ISR since 1992, where his primary areas of research have included the criminal justice system, the evaluation of substance abuse and criminal justice programs, and the evaluation of education programs. Dr. Guerin has been primarily involved in the Bernalillo County’s behavioral health initiative through the ISR’s role as an independent evaluator.

This interview has been edited for clarity and brevity.

What is your role with the County’s behavioral health initiative?

Paul Guerin: Normally when we’re hired to do evaluations, it happens after the program has been funded and running for years. Stakeholders want to know if their programs have been effective, and if there’s been an impact on the people they served. This initiative has been different. We [ISR] have been involved in the planning and implementation of the behavioral health initiative from the start. We were asked to think through what the County might fund, how to organize the funding mechanism, and how to set it up to learn what’s actually working. In this sense, I think what the County is doing is pretty innovative and thoughtful, as they’re really thinking through how the initiative will work before they actually fund it. While the County works through funding programs to provide these services, we’re evaluating the programs to look at how they’re implemented, whether they do what they say they do, and further down the road looking at the impact the programs had on the people they aim to serve.

What is different about the initiative?

The behavioral health initiative is an example of working smarter. It has been a very thoughtful process in my view. In developing the problem-based procurement for the initiative, we helped the County think through what the problem was and what the target population might be, and then let the respondents describe how they would work to solve it. This resulted in awards to eight different agencies which will provide a whole set of programs to address adverse childhood experiences. It’s been a really thoughtful process in developing, and awarding the RFP and this thoughtfulness will continue as we work to evaluate the selected programs.

What happens now that programs have been selected and funded?

There is an idea of how this might work, but we’re not exactly sure how it will play out. The problem-based procurement has been somewhat new for the County and for the eight programs that are being funded. Most agencies are accustomed to reporting on how many clients they serve, how long they serve them for, etc., but they are less accustomed to the type of evaluation that we plan to do that is more current and real-time. We want to provide back useful information on participant outcomes to allow providers to adjust their program as they move forward. This information will also be provided to the County so they can act really quickly and make systematic changes as they move forward.

This represents an exciting opportunity to figure out what works in relatively short order, but there are challenges. The process requires agencies to be thoughtful and responsive in terms of how they deliver their services and the information they collect. We are working through that process with them to ensure fidelity to the model, and that they’re collecting information at a variety of different levels, but it has been quite challenging.

What will we learn from this initiative?

Time will tell. In my 25 years at ISR, some places for whom we produced information don’t end up using it – either because they don’t understand what it means, how to use it, or it doesn’t tell them what they wanted to hear. With the behavioral health initiative, I think we have very large opportunities. We plan to run the most rigorous evaluations possible, including the use of randomized control trials. All of the eight programs are funded as 2-year pilot projects, and within this period we think we can complete process evaluations to ensure programs are doing what they said they’re doing. We may also be able to complete some outcomes studies that would look at effectiveness of these programs.

Normally government has difficulty in thinking long-term and sustaining this type of research, but we are encouraged by the way the County has planned the behavioral health initiative. Because the County is being thoughtful in releasing new funding, there is potential here to actually use some of the shorter-term learning to inform future funding and policy decisions. Some of the short-term outcomes we look at could actually be used to alter programs to produce better outcomes down the road.

Has the County’s use of data to drive decision-making evolved?

I have worked here at the Institute for 25 years, with ongoing contractual relationships at all levels of government including Bernalillo County. In the past, information has been a byproduct of what government did – they didn’t use it to help plan the delivery of services. Over time there has been an evolution in thinking to more completely figure out how to use information to understand if services have had an impact. In this regard, the behavioral health initiative is a really big step forward on part of the County. To more than use information at the back-end to help inform what they already did, but to actually think through the process of using information at the very front-end to help drive policy. It’s a big difference for the County.

Are there attempts to pursue more systems-wide initiatives within the County?

Bernalillo County was actually recently awarded a place to attend the Data-Driven Justice and Behavioral Health Institute held in Maryland. Many more counties applied than were admitted, and part of the reason we were awarded this opportunity is because the GPL was here and helping the County further their understanding of the need for data integration. Attending the Institute is another step forward for the County, which is working diligently to the extent they understand the issues. As part of this initiative, data integration will go beyond inter-county agencies and will span the medical and criminal justice systems.

What value did the Government Performance Lab bring?

Without them this wouldn’t have happened. Without the Government Performance Lab being involved in this process somewhat early on, we wouldn’t be where we are today. The first few months were slow, but through the GPL fellow’s persistence the County came to accept that this was a process they could use and would provide them with some benefits. Governments are almost like these immovable agencies – with layers of bureaucracy that drive them forward – especially in Bernalillo County. Despite that we’re here today; which is pretty far along. I think that’s pretty encouraging, but we need another GPL fellow around for five more years.

The Government Performance Lab is grateful for support from Bloomberg Philanthropies, the Corporation for National and Community Service Social Innovation Fund, the Dunham Fund, the Laura and John Arnold Foundation, the Pritzker Children’s Initiative, and the Rockefeller Foundation.