IMMAP opened less than two years ago, putting Little and his team of faculty, staff and students just a bit ahead of schedule.
The institute, part of the College of Education, runs advanced statistical analyses on research projects from researchers and organizations throughout the country. That is only a small part of its mission, though. IMMAP and its director, who is an internationally known quantitative specialist, are on a mission for social justice through accurate methodology and statistics. To get there, the team is inventing new statistical methods, finding new clients from around the world and training a new generation of specialists to perform the most advanced statistical methods available while breaking ground for more advancement.
“Doing methods right is a matter of social justice,” Little said. “If we don’t do it right, that means the results will be biased and policymakers and practitioners are going to be making decisions based on ill-advised applications of research methods. If we do it right, they’re going to have the best information.
“This is a decision science. If you’re going to be making decisions based on data, you need to make sure the data are presented in their best possible form.”
Starting IMMAPLittle’s doctorate is in developmental psychology, but most of his career has been spent on the data analysis side of research. He realized in graduate school the importance of producing good, unsullied analytics – but using only the most commonly accepted statistical techniques didn’t work for every data set. That approach, however, is what many researchers did then and still do. They learn a few techniques in graduate school and shoehorn their data into one of those techniques, even though it’s two decades later and technology has advanced to the point the once-imagined techniques are now viable.
Little embraced new techniques, developing some and perfecting others, and over time he made a name for himself as the specialist to call for statistics and methodology questions. For years, his colleagues would invite him to be the methodologist in their grant proposals; he could help design and execute the study so as to reduce statistical errors. The relationship worked for the researcher, who had access to Little’s expertise throughout the process, and for Little, who earned co-authorship on numerous papers, increasing his stature as an expert in the field.
“The way grants typically work is they would try to buy a percent of my time,” Little said. “After a certain point I’d get on two to three grant proposals, but then I would be basically out of percent time they could buy anymore. But there would be three or four other people who would like it if I would participate.”
To date, his name is on more than 250 publications with more than 300 unique co-authors; those publications have garnered nearly 16,000 citations in academic journals.
“Because of the quality of the methods, my work is getting out there,” Little said. “People are reading it, people are citing it.”
While a professor at the University of Kansas, administrators asked if he would direct a center for research methods and data analysis. Clients could access his expertise in methodology and innovations in statistical methods, what he called his “10,000-foot view,” but graduate students and interns could run the actual analyses.
“I realized if I were to direct such a center, I could be surrounded by people who could do the heavy lifting I don’t necessarily need to be paid to do,” he said. “Effectively, I can participate in many more research projects, provide input and insights that change the science and affect the quality of the work, but do it under the auspices of the institute.”
In 2013 College of Education Dean Scott Ridley contacted Little and invited him to Texas Tech to create IMMAP. He also created an academic concentration (research, evaluation, measurement and statistics, or REMS) to allow students to specialize in quantitative analysis. Many of his clients from Kansas followed him, and he’s picked up more in the last two years – enough to add a postdoctorate position and a few more interns.
Katy Roche, an associate professor at the Milken Institute School of Public Health at George Washington University, is one of Little’s clients. She is researching behavioral and mental health outcomes among Latino children of immigrant parents.
Roche cited Little’s unmatched experience in statistics and the qualified students and faculty members who are part of his center.
“Todd is uniquely qualified to provide guidance and advice around using state-of-the-art, advanced latent variable modeling techniques; imputing missing data using best practice methods; and developing study designs based on cost-effective approaches to data collection through the use of intentionally missing data,” she said. “I would never have the time to gain this knowledge and these skills, so I am most grateful for IMMAP’s capacity to do this for me.”
IMMAP’s client list is diverse, including individual professors from Texas Tech and dozens of other universities as well as the U.S. Navy, the Centers for Disease Control and Prevention, the Texas Department of Criminal Justice, the Noyce Foundation and other governmental and nonprofit agencies.
The need for good statisticsMost studies in major academic journals use outdated, less effective data analysis methods, Little said. Considering the reach data may have, and the possible effects as people interpret it, this cannot work.
Knowing this is why Dorothy Espelage, an educational psychologist at the University of Illinois-Urbana/Champaign who is a world-renowned researcher on bullying and power, came to IMMAP. She testified before Congress and other federal committees about bullying. Her data has an effect on national policy, so it needs to be correct – even more so when she comes up against bullying prevention programs not based on evidence.
She’s working now on a project that measures the effectiveness of the Second Step program in improving children’s social-emotional development, which may lead to decreased bullying. The grant is from the CDC and Committee for Children. She attributes her funding in part to Little’s expertise.
“It’s difficult to get funded without a strong statistical team like Todd’s,” Espelage said. “They keep up on the latest methods that I just cannot keep up with and run my large-scale, school-based studies that require a lot of hard work.”
The problem is bigger than researchers using bad methods, however. A couple of years ago a task force for the American Psychological Association found a shortage of quantitative training programs like the REMS program, which means not enough researchers know how to use advanced statistical techniques. Additionally, there are few places to learn new techniques once a researcher finishes graduate school.
To that end, in 2003 Little started Stats Camp, an intensive, weeklong seminar that invites the top quantitative specialists to teach courses on the most advanced statistical techniques available. He routinely brings in students and instructors from throughout the world, and in 2016 there will be Stats Camp Europe for the first time, with Little going to the Netherlands for a week.
He cannot stress enough the importance of good statistics. Little cites social justice every time the question arises: Researchers need to produce valid results, and to do so they need to use the best statistical methods for their question. To do otherwise misleads the public and policy- and decisionmakers.
“Too often researchers take their question and stuff it into the one or two analytical techniques they know, and it’s not the right way to do it,” Little said. “You have to tailor the technique to the question.”
IMMAP’s unique place in academiaThe academic world does not have many IMMAPs or Todd Littles. Some universities have a quantitative specialist who works with other professors on campus, either running their statistical analyses or training researchers on new techniques.
Others have what Little called service centers that can run statistics for a researcher. IMMAP provides this service, but it is a small part of the process.
The relationship between a researcher and IMMAP begins well before the time for data analysis. Little and his team help researchers refine and write the method section of the grant. This helps in two ways: Little has been part of hundreds of successful grants, so he knows what grant review panels look for; and he is the accepted expert on many quantitative techniques. He or one of his team also has invented many of the techniques this type of research uses.
“Our goal is to make sure the methods section and its execution are described in a pristine manner, such that a grant review panel would have no basis for criticizing the methods,” Little said. “In terms of resources, with IMMAP they can’t ding the resources. And in terms of personnel, we are the experts in the field who have invented most of the methods that we’re describing, so they can’t knock the personnel.
“The only thing they can pick on then is if this is a sufficiently good idea. Would it have the impact we want?”
Invention of new statistical methods also sets IMMAP apart from statistical service centers. Some statistical methods were around 30 years ago but not feasible because the necessary technology didn’t exist. Others, however, have been developed in concert with technology. That kind of innovation happens almost daily in IMMAP. “There are places typically run by statistics departments where you can get help,” Little said. “The uniqueness of IMMAP is that we are very involved in the substantive theories being tested; we understand the theory part and that is an attractive feature. Two, we’re not just applying techniques that have been learned, we’re innovating techniques. That’s a one-two punch I think is absolutely unique.” Finally, the experts at IMMAP are more than statisticians. They are quantitative specialists – well-versed in a statistician’s work but able to understand the theoretician’s questions and nature of research. They’re not looking at the data in a vacuum. In fact, all of the REMS students are required to be proficient in a substantive research area such as developmental or educational psychology.
Missing dataIMMAP and Little are leading the industry in addressing missing data, which all social scientists have in their work. Missing data occurs when, for example, a researcher gets only 400 out of 500 surveys back. Traditionally only the 400 returned surveys are used to analyze the question.
That, however, skews the data set. Those who did respond may feel more strongly about the question than the population as a whole, or the lack of response may represent a specific subset that differs from the population. Either way, the data set does not portray an accurate picture. The missing data must be accounted for.
“It can wreak havoc if you don’t treat it properly,” he said. “When you don’t try to correct for the reason it’s missing, you have selective data and your results are no longer valid.”
Little’s work on missing data has presented researchers with ways to address this issue. The methodologies he touts in accounting for biasing effects of missing data is so effective, he said, it will unequivocally increase the validity of the research findings. He has published numerous articles on the subject, and he is working on several projects that continue to advance the utility of modern missing data treatments.
IMMAP staff is developing a software package that facilitates the whole process of accounting for missing data. When it’s finished it will be open source freeware instead of a copyrighted program, giving all researchers access to it. Again, it’s a social justice thing for Little.
The REMS degreeCurrently about a dozen graduate students are working toward a REMS degree through IMMAP. Another half a dozen students from throughout the university are employed as graduate research assistants.
Several students are undergraduates who either are earning a minor in quantitative analysis or who want to find out what quantitative analysis is like before they get into graduate school. Little’s already had one student come to IMMAP when it just opened to check it out and choose to earn the minor concentration, and he thinks others will too, largely because this will practically assure admission into the graduate school of the student’s choosing.
“Most academics learn their statistics in graduate school, and then they go out and start their jobs, and if the statistics changed after graduate school, where are they going to learn it?” Little said. “There are very few programs like the REMS program where faculty members are dedicated quantitative specialists training students in quantitative methods. As quantitative specialists, their job is to stay on top of and invent new methods.”
RSS