Machine learning and crowdsourcing part of a high-tech approach to diagnosis.

Bioengineers, dermatologists and oncologists at Vanderbilt University Medical Center have teamed up to bring clarity to GVHD, particularly when it comes to measuring and interpreting GVHD skin lesions. Their latest research¹ suggests the process could be automated.

Experts agree that 30 to 70 percent of bone marrow and stem cell transplant recipients develop GVHD – but they do not always agree on the appearance of associated lesions. Machine learning could help standardize assessments and eliminate discrepancies.

Consistency is imperative to support research into new treatments, said Eric Tkaczyk, M.D., assistant professor of dermatology and biomedical engineering at Vanderbilt. “We have a lot of GVHD therapies that seem to work in our patients, but when we get to larger studies, they don’t pan out. Part of the problem is that we’re not able to measure the disease consistently.”

Automated Support

To quantify skin lesions for GVHD or dermatology studies, researchers typically must pore over hundreds or even thousands of images. It can be challenging to stay consistent between images and patients when defining borders (a process known as “segmentation”).

It’s also time-consuming, Tkaczyk said. “Instead, we’re trying to automate the process we currently do with our eyes. We’re using machine learning to recognize affected skin in photos and help enhance automation.”

Tkaczyk teamed up with Madan Jagasia, M.D., chief medical officer of the Vanderbilt-Ingram Cancer Center and Benoit Dawant, Ph.D., director of the Vanderbilt Institute for Surgery and Engineering to find a better way.

“We’re trying to automate the process we currently do with our eyes.”

In one of their earliest collaborations, the team created a dataset of over 400 3D patient skin photos broken up into 4,000 2D images. They asked experts to label a subset of the images to train a deep learning convolutional neural network (CNN). Their results showed the machine learning model could successfully segment GVHD lesions from images on its own.

The team has used a similar approach to monitor GVHD erythema and sort lesions by severity. More recently, Vanderbilt postdoctoral fellow Inga Saknite, Ph.D. used confocal microscopy to further analyze GVHD lesions. (Tkaczyk and Jagasia have also begun exploring ways to consistently collect and quantify other GVHD data, such as via handheld devices that monitor sclerosis.)

Crowdsourcing Expands Datasets

Challenges remain, including how to navigate color nuances. Said Tkaczyk, “The networks aren’t brilliant. They’re incredibly good memorizers. Our research shows they can recognize variants, but if we haven’t shown them a diverse population they aren’t going to know what to do with a new skin tone or lesion color.”

Crowdsourcing provides an opportunity to expand the training dataset and improve CNN algorithms. With help from Daniel Fabbri, Ph.D., assistant professor of biomedical informatics at Vanderbilt, the team asked both a board-certified dermatologist and untrained volunteers to identify GVHD lesions in patient images. Even with only a brief slide presentation to learn about GVHD, pixel-by-pixel match between the crowd workers and the expert across 410 2D images was 76 percent.

The findings are proof-of-concept that crowdsourcing can support machine learning, Tkacyzk said. “This places this group of crowd workers, as a collective, very much on par with expert evaluation for chronic GVHD.”

Diagnostic Algorithms

While the researchers continue to refine machine learning CNNs, Jagasia has also focused on diagnostic algorithms. He recently joined colleagues in the Chronic GVHD Consortium to refine the NIH response algorithm for GVHD.

“There was a real-world need to update interpretations of joint/fascia scoring, in particular. After reviewing over 200 cases we were able to determine what score ranges are actually clinically meaningful,” Jagasia said. “We are moving toward more precise, evidence-based methods for measuring GVHD and related therapeutic responses.”

Together, the researchers’ efforts provide innovative approaches to improving GVHD diagnoses. Tkaczyk noted there could be benefits for other fields too, such as dermatology.

“If we are able to create clear definitions and expert consensus to drive machine learning on subtle images of rashes, our approach could apply to many other visual medical problems, particularly those with a large degree of expert variation.”

Pathology and radiology could benefit, too. Already, Vanderbilt researchers have used machine learning to successfully classify diabetic nephropathy in patient samples, with similar precision to traditional methods.

About the Expert

Eric Tkaczyk, M.D.

Eric Tkaczyk, M.D., Ph.D., is director of the Vanderbilt Dermatology Translational Research Clinic, an assistant professor of dermatology and biomedical engineering at Vanderbilt University Medical Center, and an attending dermatologist at the Nashville VA Medical Center. His research interests include biophotonics and machine learning approaches to improve diagnosis and treatment of dermatologic and systemic diseases.


This work was supported by Career Development Award Number IK2 CX001785 from the United States (U.S.) Department of Veterans Affairs Clinical Sciences R&D (CSRD) Service (Tkaczyk), NCI K12 grant number CA0906525 from NIH (Tkaczyk), a Discovery Grant from Vanderbilt University (Dawant), and NIH grant number 1 UH2 CA203708-01 (Fabbri).