Adapting an artificial intelligence sexually transmitted diseases symptom checker tool for Mpox detection: the HeHealth experience

Introduction

Monkeypox (Mpox) was declared a public health emergency of international concern (PHEIC) on 23 July 2022, owing to the risk of human-to-human transmission, and the rapidly increasing number of cases being recorded worldwide.¹ An important observation of the ongoing global outbreak of Mpox is that the majority of the infected cases are among men who have sex with men, and this has led to stigmatising attitudes that conflate both Mpox acquisition with one’s sexual identity. This has led to concerns that individuals may be disincentivised to seek care due to concerns around stigma.^2,3

Artificial intelligence (AI) applications have shown promise in the management of pandemics and have been widely used to assist the identification, classification, and diagnosis of medical images, such as in the case of the last PHEIC, coronavirus disease 2019 (COVID-19).⁴ Others have also started to do so with Mpox.^5–9 In these studies, researchers created datasets through data augmentation, as well as using images of skin lesions caused by Mpox, chickenpox, measles, and normal controls collated from open-source data. Pre-trained, modified, and ensemble convolutional neural network (CNN) were then used to classify Mpox from other classes, obtaining accuracies of between 83.0% and 99.0%.¹⁰ A systematic review on the application of AI techniques for Mpox highlighted how AI has not only been used for the diagnostic testing or screening of Mpox, but also in the epidemiological modelling, drug and vaccine discovery, and media risk management in the context of Mpox.¹⁰

While our team had successfully deployed a tool that could screen for symptomatic sexually transmitted diseases (STDs) through visible dermatological clinical signs, we needed to rapidly adapt this tool to ensure that those who are at risk of acquiring Mpox had a similar option of assessing their risks for symptomatic Mpox without the fear of stigma, and guide them towards appropriate interventions where possible. We aimed to develop a robust digital screening test for symptomatic Mpox through AI and machine learning approaches.

We leveraged our existing AI symptom checker tool designed to detect STDs to do so. We sought to do so overfive stages. First, we engaged in formative research by having initial discussions with experts to better understand the nature of the current Mpox PHEIC. Second, given the lack of available data in the global outbreak, through the proposed system, the team aimed to intensify stakeholder engagements under several key directions such as: (1) to amplify discussions on the clinical presentation and risk factors of Mpox in the ongoing pandemic; (2) enable community stakeholders to address the stigma attached to Mpox; and (3) healthcare institutions to discuss possibilities of using data to train the tool. Third, we consolidated images to adapt and train our existing AI tool for the screening of Mpox. Fourth, we validated the Mpox symptom checker tool. Finally, we continued refining and validating our Mpox symptom checker tool in community and healthcare settings.

Materials and methods

Initial development of the AI tool to screen for sexually transmitted diseases

The team developed a smartphone app (HeHealth.ai) where app users can use their own smartphone cameras to take pictures of their own penises to screen for symptomatic STDs. Fig. 1 provides the symptom checker tool wireframe. The AI model was initially developed using 5000 cases and uses a modified CNN to output prediction scores across visually diagnosable penis pathologies. These 5000 cases comprised images of diseased (n = 2600) and non-diseased (n = 2400) penises. Images of diseased penises comprised a roughly equal number of five penile diseases, including herpes simplex virus eruptions, syphilitic chancres, penile candidiasis, penile cancer, and genital warts. Images of non-diseased penises included images of healthy penises, including those with penile pearly papules and Fordyce spots. The inclusion of non-diseased images for data augmentation of the disease class is important when building a robust model which can separate physiological variations (such as penile pearly papules) to differentiate from actual sexually transmitted diseases. The underlying AI predictions are conducted in two phases. In Phase 1, given the input image of the affected area, a computer vision-based screening model outputs prediction scores across visually diagnosable penis pathologies. In Phase 2, users’ answers to two simple questions (i.e. pain and discharge related) that are used to further fine tune the visual prediction results.

Fig. 1.

The AI symptom checker tool wireframe.

The interface then displayed the potential diagnoses that were output by the AI, sorted in the order of the AI’s predicted likelihood scores. AI predictions with low likelihood scores (<40% or <0.4) are not shown and are removed, and the resulting display included a primary possible condition and secondary possible condition with confidence levels (in percentages) to avoid presenting extraneous information. In addition, a heatmap showing the detected pathological locations are displayed. Fig. 2 provides an example of the heatmap. We subsequently conducted validation of the first 1000 cases, during which each image is reviewed by a venereologist situated in Singapore, Sri Lanka, or the United Kingdom. Following the review, corresponding users who submitted the images were given feedback from the venereologists that could be accessed through the app by a user notification. Given that personal information is not collected through this process, such feedback can only be obtained through a unique username and password created by the user to log into the system.

Fig. 2.

Example of screening heatmap shown to users.

Adaptation of the AI platform for the Mpox symptom checker tool

The overall timeline describing the development and subsequent refinement of the penis symptom checker for Mpox screening can be found in Fig. 3. The first phase (July to August 2022) comprised of the formative phase where initial discussions with medical, socio-behavioural, community and statistical experts were initiated to better understand the nature of the current Mpox PHEIC. The second phase (August 2022) comprised of engagement with other healthcare institutions and a search for publicly available databases for Mpox-related data (e.g. patient characteristics, images etc.) and the initial development of the tool (Table 1).^7,11–13 This phase leveraged the already-developed tool to identify STDs, which facilitated the process. The third phase (August to September 2022) that was iterative in nature involved the rapid consolidation of Mpox images to train the initial tool, and refinement of the tool’s user interface through feedback from stakeholders. The fourth phase (October 2022) involved a validation study to assess the accuracy of the symptom checker tool. The fifth and final phase (October 2022 onwards) involved the launch of the symptom checker tool and further training of the tool through the use of patient data obtained via the web app.

Fig. 3.

Timeline for the development of the Mpox symptom checker tool.

Table 1.List of stakeholders engaged to develop the Mpox tool.

Types of stakeholders	Insights
World Health Organization representative	Data and statistics regarding the risk prevalence
Centers for Disease Control	Data and stats regarding the risk prevalence
Industry partners	Research insights and validation
Community representatives	Communications strategy for Mpox risk and symptom checker tool
	Sources of stigma and patient care pathway for Mpox
	User interface and language used for key populations at risk of Mpox
General public	Mpox images

Results

HeHealth used 5000 images to train the initial AI tool for STDs identification. Of all the STDs, our tool can diagnosis for syphilis, herpes simplex virus, human papillomavirus/genital viral warts with accuracy of 86%, 93%, and 96%, respectively. From June 2022 to October 2022, a total of about 22,000 users had downloaded the HeHealth app, and about 21,000 images have been analysed using HeHealth AI technology.

In the same period, we then started engaging in formative research to engage in initial discussions with medical, socio-behavioural, community and statistical experts to better understand the nature of the current Mpox PHEIC. This process gave us insight into the issues around the nature of the PHEIC’s epidemiology, data sources, and other emerging issues around risk, communication, and Mpox-related stigma. We then sought to engage deeper with healthcare institutions and a search for publicly available databases for Mpox-related data. In this phase, we worked with healthcare institutions to identify sources of Mpox-related images, as well as supplemented the work by healthcare providers through the use of our symptom checker tool.

From July 2022 to October 2022, we also collated a total of 1000 Mpox-related images and 1000 images of penises without Mpox (600 without any visible skin pathologies or irregularities, and 400 with similar rashes or conditions including measles, scabies, eczema, psoriasis, smallpox, and chicken pox) have been used to train the Mpox symptom checker tool. After the initial training, the H2S algorithm had undergone internal validation in two phases. First, we calculated specificity of the tool using a dataset comprising 100 patient observations that had been assigned a H2S score of 80% and above. We then calculated sensitivity of the tool based on a dataset comprising 100 patient observations that had been assigned a score of 40% and below. Each dataset of 100 observations included 60 Mpox positive patients and 40 Mpox negative patients with similar rashes or conditions including measles, scabies, eczema, psoriasis, smallpox, and chicken pox. The cut-offs of 80% and above and 40% and below reflect ‘high likelihood’ and ‘unlikely’ results that are shown to participants when using the symptom checker, respectively. Based on these observations, we calculated a specificity of 87% and sensitivity of 90% for the H2S tool. HeHealth’s Mpox digital symptom checker test is currently undergoing further external validation through community and clinical partnerships (Fig. 4).

Fig. 4.

Mpox HeHealth Symptom (H2S) score calculator screenshot.

Discussion

Throughout the process of developing the innovation and field testing it, we noted several lessons learnt that arose due to several limitations or hurdles and conclude this article with recommendations for other innovators who are keen to work on similar projects. The first hurdle was about data privacy and security for app users. In our formative research conducted among potential users of the app, many of them raised issues of data privacy and security with us. This was a hurdle that the team had to overcome prior to the launch of the original STDs symptom checker tool, which would have had implications for patient confidentiality and trust in the use of the Mpox symptom checker tool. Users generally feel safe using the HeHealth tool as no personal identifiers are collected throughout the user journey. Any collected data including images are de-identified in ways that cannot be traced back to the users. All processes are conducted through the Health Insurance Portability and Accountability Act of 1996 (US) (HIPAA)-compliant processes and platforms.

The second hurdle came about due to an initial lack of data to train the AI model. Initially, health system and professionals were approached for the images. However, this was met with less success due to regulatory restrictions such as patient confidentiality. Furthermore, the lack of a central coordinating body for data consolidation (e.g. images of disease state) served as a key barrier to training the AI model early on. The third hurdle was due to technical challenges. Given that the current outbreak has been characterised by novel clinical presentation and patient characteristics,¹⁴ our team needed to be in constant dialogue with expert stakeholders, as well as to constantly monitor and review the scientific evidence surrounding the current outbreak of Mpox. Furthermore, unlike STDs, we had to train out AI tool to also review ulcers and lesions that appeared on other parts of the body that were not limited to the anogenital region.

The fourth and final hurdle arose due to concerns around the generalisability of our data. One concern arose during our earlier AI tool development was ensuring that our tool was inclusive of all skin colours. We ensured this was done through several means. First, we tried to build a dataset that represented multiple skin colours from a global population. At the same time, our core technology was built in a way skin results will not be affected by skin colour by training the AI to focus on colour-neutral skin abnormalities. Another concern was that pictures of visible skin lesions meant that symptomatic STDs that present in other ways including other dermatological conditions or genitourinary symptoms may get left out. Nevertheless, we used a combination of questionnaires they addressed issues such as the presence of discharge or pain to accompany the visual analysis of skin lesions, which may also help address and narrow down the various types of potential STDs that an individual may be experiencing.

Conclusion

Throughout the process of developing the original STDs symptom checker tool and the Mpox symptom checker tool, we took away several key lessons that we believe would be useful for researchers and other practitioners interested in embarking on similar innovations in health care. First, to engage a wide range of stakeholders early. In the context of an ongoing global pandemic, and with rapidly evolving data and clinical insights, we found that engaging not only the medical science experts, but also community representatives were important. Collaborators shared similar goals in helping to alleviate the impact of Mpox around the world and were willing to work together in achieving similar outcomes. Second, to have a multidisciplinary team. Our multidisciplinary team provided not just expertise in executing various aspects of work in the rollout of the symptom checker tool, but also provided strategic directions that led towards the prioritisation of key development milestones, including stakeholder engagement, building the AI tool, the validation study, and eventual rollout of the tool.

Third, to prioritise pragmatism over research ‘elegance’. Initial efforts to validate the initial tool to screen STDs had met several barriers; this included regulatory barriers in Singapore due to being classified as a medical device, as well as the requirement of the local university ethics board in obtaining approvals from the relevant medical devices authorities. While it is important that formal research be conducted to validate the accuracy of the tool, this was done at a later stage prior to rollout. Nevertheless, all processes were HIPAA-compliant and adhered to best clinical practices. Finally, big data comprises small data. Data for our AI tool were consolidated incrementally, which led to the eventual consolidation of about 10,000 Mpox and STD-related images. This was, to our knowledge, one of the largest and most comprehensive datasets globally. Instead of waiting on a wider body of data, researchers should consider engaging potential partners in the sharing of data in secure ways, so that an active rather than passive approach is undertaken.