Author + information
- Received May 28, 2018
- Revision received August 19, 2018
- Accepted August 21, 2018
- Published online November 12, 2018.
- Samit Bhatheja, MD, MPHa,
- Valentin Fuster, MD, PhDa,
- Surbhi Chamaria, MDa,
- Sarthak Kakkar, MSb,
- Rivelle Zlatopolsky, BSb,
- Jason Rogers, BAb,
- Emamuzo Otobo, MD, MPHb,
- Ashish Atreja, MD, MPHb,
- Samin K. Sharma, MDa and
- Annapoorna S. Kini, MDa,∗ (, )@MountSinaiHeart@MountSinaiNYC
- aZena and Michael A. Wiener Cardiovascular Institute, Icahn School of Medicine at Mount Sinai, New York, New York
- bAppLab, Icahn School of Medicine at Mount Sinai, New York, New York
- ↵∗Address for correspondence:
Dr. Annapoorna Kini, Mount Sinai Hospital, One Gustave L. Levy Place, Box 1030, New York, New York 10029.
Technological revolution in the field of medical education is here, and it is time to embrace it. Adoption of on-the-go learning style, portability of smartphones, and expression of concepts with interactive illustrations and their global reach have made application (app)-based learning an effective medium. An educational mobile app, BIFURCAID, was developed to simplify and teach complex coronary bifurcation intervention. This app has been downloaded worldwide. The survey results revealed its widespread acceptance and success. The authors believe that educational apps can have a significant impact on shaping the future of cardiovascular education in the 21st century. This experience with developing and testing the app could work as a template for other medical educators.
Technology has transformed the way we learn and medical education. Recently, advances in medicine significantly correlate with technological advancement (1). Creative expression in the form of highly interactive user interface and visualization of concepts in a stepwise manner using images have helped educators to express their knowledge in effective manners. An array of technologies is available for medical educators, including Internet-based simulation, social media platforms, and mobile application(s) (app) (2). The adoption rate of mobile devices and tablets continues to rise among the general population as well as health care providers (3). Approximately 85% of physicians possess a smartphone in the United States, which has the capability of running third-party software apps (4,5). There are approximately 100,000 mobile health apps available on the App Store (Apple, Cupertino, California) and Google Play Store (Google, Mountain View, California) (3). These apps include medical formulas and calculations, guidelines to weight management, and medication management (6).
This report describes the creation of an app for mobile cell phones and tablets to simplify learning the complex subject of coronary artery bifurcation intervention. Intervening in coronary artery bifurcation involves complex procedural techniques, which differ significantly depending on the baseline lesion. Based on the nature of the lesion (left main or non-left main) and Medina classification (most widely used classification of coronary bifurcation lesions based on presence, or absence of significant disease in the proximal main branch, the distal main branch, and the side branch), multiple permutations and combinations are possible (7). There is no single source available that completely covers all possible scenarios. Our institution is a large-volume percutaneous coronary intervention center, which is also a referral center for highly complex cases. This experience and knowledge gleaned from published studies helped us create a mobile app for this technically challenging procedure (7–16). Our objective was to share our knowledge and experience with converting medical knowledge into a simplified educational prototype in the form of a mobile app. This process can be extrapolated to create similar educational apps.
App Design Process
Conversion of medical knowledge into an app-based platform requires a multidisciplinary approach, which includes a medical educator, a technical team for app development, and a medical illustrator. Conversion of complex medical concepts into simplified language for technical coding was the most challenging part of the medical app design. The medical educator’s role is pivotal in orchestrating the development at each phase with clear communication strategies. How the knowledge was first constructed and then deconstructed to help facilitate development of the app is discussed. For ease of understanding, the app development process can be divided into 10 phases, as follows (Central Illustration).
This is the phase which requires brainstorming and choosing a complex medical subject which can be simplified for learning. The medical educator can select or reject a particular subject matter based on its educational value and communicability through an app-based platform. The authors chose procedural techniques of percutaneous intervention of coronary artery bifurcation as the central subject for app development because the intervention can be explained in a stepwise manner with illustrations. This is the phase of contemplation which can last from a few days to weeks to months.
This phase requires the educator to collect medical knowledge available on the central subject. Information includes searching published reports for peer-reviewed articles, medical books, and real-life experience from experts in that particular field. This phase allows the medical educator to gain extensive knowledge about the subject before developing educational content. This phase also includes developing a core team that includes a medical illustrator and a technical app development team. Medical illustrators would be preferred due to their experience in the field and would more likely be able to express thoughts through illustrations in an effective manner. An advanced animation creator, a videographer, and an editor may also be needed, depending on the nature of the app's content demands. A team with experience in medical app development would be preferred. This phase is the longest phase because it includes acquiring medical knowledge and developing a core team. The cost of making an app could be a limiting factor. This phase also includes making financial decisions and developing the core app development team, accordingly.
This part of app development is highly variable and negotiable. The cost of developing a mobile app can be divided into 3 major components: 1) cost of content creation; 2) cost of technical app development; and 3) cost of app launch and promotion (Figure 1). Content creation cost is highly variable, based on the experience of the creator and complexity of the content. Content generation may require multiple creators including a medical illustrator, a medical animator, a videographer, and a video editor, based on the type of content needed for the app. The cost of the technical app development consumes most of the app development budget. Most technical app development teams provide a projection of the entire process in hours and charge a fixed amount on an hourly basis. Hence, the more complex the work the more hours will be needed to accomplish it, and in turn, the higher the cost. Factors which typically influence the cost of technical app development are scope of the app, database complexity, number of platforms (e.g., Apple iOS, Google Android, Web, and so forth), versions (e.g., mobile version, tablet version, and so forth), licenses, and ongoing hosting costs. Furthermore, the technical team may be needed to maintain and update the app after its release. Some technical app development teams provide bundled plans, which include development as well as maintenance of the app. The cost of app promotion and release depends on the media used for promotion and the number of platforms the app is being released on. This component requires the smallest share of the app budget. Costs include acquiring Internet domains to release apps, registration for developer accounts with iOS (Apple) and Android (Google) to publish the app on these platforms. App promotion can be done at no cost through social media and Internet-based video services like YouTube (YouTube LLC, San Bruno, California). On the other hand, a professional app promotion company can be hired at a reasonable cost.
Algorithm development phase
If the complex medical subject has multiple concepts or steps, the educator converts currently available knowledge into an algorithm. Developing the algorithm is the educator's first attempt to bring the idea, the proposed app’s flow, and the scientific content to the paper. Microsoft Word software (Microsoft, Redmond, Washington) was used for the BIFURCAID algorithm. However, any simple text-based software (e.g., Pages software; Macintosh operating system, Apple, Cupertino, California) or notebook for handwritten notes can be used. Our method of algorithm development works best for subject matter with pre-existing extensive knowledge and guidelines. This is the most extensive part of app development, which requires multiple revisions and review by experts. For example, the app was divided into 2 major sections, the Left Main and the Non-Left Main section. Sections were further divided based on all possible options according to the Medina classification. Based on the type of Medina classification, each technique was explained in a stepwise manner.
Algorithms made in a previous phase are deconstructed into “maps” of the app in this phase. A map is a flowchart showing flow of the app subject matter, which helps the technical team to envision the complexity of the app. Once the flowchart is made, each subsection on the map is identified with a particular folder number. These folder numbers become very useful in providing content of the app to the technical coding team. For BIFURCAID, each of the sections (left main and non-left main) was divided into separate flowcharts. Each flowchart subsection was assigned a folder number. Excel (Microsoft) sheets with stepwise images of a particular bifurcation intervention technique and associated text with each image were created. The number on the Excel sheet was the same as the corresponding folder on the map. This helped the technical app development team to clearly identify the precise location of the image and corresponding text in the app flow. Because the app development team typically has little or no medical background, it is of paramount importance to provide them content in a systematic manner to decrease chances for errors in coding. The authors' experience with deconstruction of the BIFURCAID algorithm for left main bifurcation intervention sections is described in Online Figure 1, which shows each step of the process in detail. Excel software was used to communicate app content. However, any software which would help express images with associated text in a stepwise manner can be used. For example, PowerPoint (Microsoft) software can be used for this purpose by making separate image files for each folder on the flowchart. PowerPoint can express images with associated text in a stepwise fashion. The educator must ensure the fidelity of the image count and provide all the images separately, as the app usually requires a particular format and size of the images based on target screen sizes.
Image development phase
This phase starts when the educator has completed deconstruction of all the concepts and the steps have been completed. This phase helps the educator to visualize each permutation and combination of images, which may be needed to express the concepts. Creative expression of the images to explain the concept is possible with excellent communication between the educator and medical illustrator, which includes description of images and mockups (rough sketches made by the medical educator). Typically, the medical illustrator initially provides prototype images to the educator, which may require multiple revisions until the educator feels the images are able to explain the concept well. The authors decided to use a vector illustration platform to develop images because it provides freedom of changing orientation of interventional equipment in a stepwise fashion and can generate multiple images from a single base file. Vector illustration also has freedom of resolution as vector-based diagrams can be saved to any desired resolution and used for different media platforms.
Designing the logo
Logo design is a very important part of the image development phase. The logo is considered the “face” of an app and the most important object on an app store at first glance (17). The logo can have lasting impact on a user and can affect app downloads (18,19). Relevance to the subject matter and liberal use of vibrant colors can help the logo stand out. The authors used Sketch and Illustrator software (Adobe Systems, San Jose, California) to develop the BIFURCAID logo. An illustration of a bifurcating artery inside a lowercase letter “b” was used to keep the logo relevant to our central subject. Liberal use of red on a white background helped the logo to stand out (Figure 2).
Conceptualizing the user interface
This phase involves abstraction of colors, appearance, orientation (portrait or landscape), and navigation of the app. The medical educator helps the technical app development team with expression of medical concepts in app form. PowerPoint was used to communicate our desirable user interface design. User interface evolves further with creative input from an app development team. It is essential to include references when it comes to medical knowledge-related apps. References can be added as a separate section or added at each decision-making point as a hyperlink to make the app more intuitive in order to provide deeper learning opportunity at the learner’s fingertips. The technical app development team can provide initial mockups of the user interface including home screen, menu, and screens dedicated to teaching material. These mockups provide visual feedback to the medical educator about the app’s response to users’ interaction and input. The BIFURCAID user interface characteristics for app navigation are shown in Figure 3.
Technical phase of app development
This phase is about planning the app architecture and coding and is led by the technical app development team. A team which has worked on health care projects before would be preferable to a team with no previous experience. The technical app development team usually consists of 5 members, the: 1) technical manager; 2) user interface/user experience (UI/UX) designer; 3) software architect; 4) software developer; and 5) tester. The technical manager oversees the entire process of app development and ensures a proper and timely development process. The UI/UX designer works on designing the aesthetics of the app (user interface: color combinations of the buttons, icons, and background, and so forth) and making the app navigation intuitive (user experience). The software architect identifies appropriate programing languages and software to accomplish app development. The software developer writes the app codes. App codes are considered the technical building blocks of the app. The tester tests the fidelity of the app coding and suggests bug fixes (technical glitches in app coding) to the app developer for rectification before the app is available for testing by the client (medical educator). Constant communication between the technical team and the medical educator at regular intervals and at each phase is very important. Typically, the technical team provides a prototype of part of the app on computer-based software for the medical educator’s review. If the overall design and concept delivery of the prototype aligns with the educator’s intentions, the technical team proceeds with coding.
In this phase, the educator tests the app for integrity of scientific content and reviews the corrections made in the coding process to improve overall user experience. This phase also includes testing the usability and learnability of the app with the help of standardized questionnaires. This pilot study phase involves reaching out to experts in the field for their input on the scientific component as well as their responses to usability and learnability on a questionnaire. This questionnaire provides an overall score, which can be put forth to the industry standards of the country and provides invaluable information regarding how easy it is to navigate and use the app (usability) and how easy it is for the users to remember navigating the app during subsequent use (learnability). The survey methods section below further describes the process in detail.
Once the app proves its usability and learnability to the user and scientific integrity has been confirmed, the medical educator can decide to launch the app. This phase typically requires advertisement and outreach to potential users as the ultimate goal of the app to reach as many learners as possible for medical education, and their feedback can be invaluable for development of subsequent versions. It is also of paramount importance to work on the legal aspect of the app prior to its launch.
Updates are typically done to rectify technical glitches in the app (bug fixes), to improve user interface, or to update scientific content as knowledge in the field evolves, and if the medical educator would like to add more extensive steps, which could be an extension of a prior app. These variables are not time-dependent and can be updated as needed. Updating an app is a major component of the maintenance phase of the app. A post-app launch system usability survey can further provide insights into overall app design and usability. This can help improve app layout and user interactivity in future versions.
The System Usability Scale (SUS) is a very reliable tool (20,21), designed to obtain subjective feedback on perceived usability and user satisfaction. According to Nielsen (20), usability is defined as an attribute to assess how easy it is to interact with a user interface (22). Learnability is defined as the ease by which users are able to learn the technology by repeated use (23). This is a 10-item questionnaire with a 5-point Likert scale. Response options range from 1 (strongly disagree) to 5 (strongly agree). Items 1, 3, 5, 7, and 9 are positively worded, and items 2, 4, 6, 8, and 10 are negatively worded (Online Figure 2). The SUS is able to differentiate between good and bad usability features even with small sample sizes (n = <10 respondents). Ratings for SUS scores are as follows: 0 to 64 is unacceptable, 65 to 84 is acceptable, and 85 to 100 is excellent; a score of 81.3 represents the likelihood-to-recommend (LTR) threshold. Users are likely to recommend a product that has an average SUS score of 81.3, whereas users would not recommend a product that has an average SUS score of 67 (24). Using factor analysis, the SUS is able to provide additional information through 2 subscales: an 8-item “Usability” and 2-item “Learnability” scale (23,25).
As of January 2018, more than 2,170 users have downloaded the BIFURCAID app globally, including both the iOS (Apple) and the Android (Google) platforms. An e-mail containing questions about user location, frequency of use, and occupation type along with the standardized survey questions for system usability scoring was sent to all users. A total of 103 users responded to the survey. Most of the survey respondents were from the United States (46%). The majority of the survey respondents, mostly practicing interventional cardiologists (74%), used the app at least once a week. Most of them felt that BIFURCAID helped improve their knowledge (87%) and procedural practice (86%) of coronary bifurcation interventions (Figure 4).
Overall, users felt that BIFURCAID performed well. The group mean for overall SUS score was 81.3, an “Excellent” rating based on standard SUS responses (24). BIFURCAID met the industry benchmark SUS score of at least 80 for users to likely promote the product. The mean score for the “Usability” subscale was 82.3, and the mean score for the “Learnability” subscale was 77.3. A raw SUS score of 81.3 has a higher SUS score than 98.19% of all products. Internal reliability is good with a calculated Cronbach alpha of 0.806. With a mean SUS of 81.3 and an SD of 16 compared to a global SUS population benchmark of 68 and an SD of 12.5, a t-statistic was calculated. The sample SD was selected as the point of comparison, because it is more specific to this technology. The resulting t-score yielded a p value below 0.005 (0.001), allowing us to confirm the statistical significance of this pilot sample. BIFURCAID scored highest in the usability category of the SUS score, with an average score of 81.3.
Qualitative data were also collected with this survey to get user feedback. Most of the users liked easy navigation throughout the app, pictorial representation of each procedural step, and systematic approach to the complex interventional procedure (Online Figure 3). Users also recommended including videos, fluoroscopic images of difficult cases, and offline access to the app, which we have decided to update in a future version of the app. Survey results revealed only 15% of the users were fellows-in-training.
The need for medical education in a particular subject and the lack of comprehensive educational material should be the primary driving forces behind development of an educational app. There are many medical apps available on both major mobile platforms. According to U.S. Food and Drug Administration guidelines, some apps are considered medical devices and should undergo regulation (26). However, apps for education of medical professionals and reference guides do not require any regulatory process (27).
Users typically identify an app with reliable content by its makers. The app developed by the leading medical society or by leaders in a particular field is one way of knowing the app’s authenticity. Reliability of the content becomes questionable with a low level of the health professional’s involvement in app design (28–30). Adoption of a particular app by the user depends on its design, authenticity of content, and ease of learning.
We encountered several barriers during app development process. Sharing this experience might help other educators foresee and mitigate them. Major barriers that could considerably delay the app development process are listed below.
Lack of knowledge about basic and simple technical terms can create a significant barrier, which can discourage most educators from app creation and can considerably lengthen the contemplation phase. Additionally, technical barriers can create communication barriers, which is discussed separately below. We studied currently available literature, basic knowledge about the process of making an app from resources including PubMed, technological literature, and platform-specific guidelines published by technology companies.
App development can be an expensive process based on the content, amount of technological support, and maintenance requirements. Limited funds can adversely affect both app development and maintenance. Initial understanding of cost for app development and maintenance can help the educator budget and arrange funds prior to the commencement of the project. Other educators with experience in app development and technical support companies can help estimate the possible cost based on the scope of the app.
The app development team has a technical background and typically has little or no medical knowledge. On the other hand, the medical educator has typically little or no technical knowledge. Communication between these 2 key teams can be challenging due to differences in professional language. We alleviated this barrier by regularly (weekly or more often) communicating with the technical team and acquiring an initial understanding of basic technical terms and issues. The following barriers can potentially prevent widespread adoption of the app.
Need of the subject matter
If there is already a good digital resource available pertaining to the subject matter of the educator’s choice, it may not have widespread acceptance. Hence, careful selection of the subject matter is prudent.
A glitchy or counterintuitive interface can lead to loss of the user’s interest and would have low learning utility despite great subject matter. Survey for SUS scoring can help determine usability and learnability of the app. The educator can consider revising UI/UX design with the help of an app development team if scores are low.
Presentation and flow of the content
Due to the absence of the educator at the time of use of the app, flow and presentation of the content play crucial roles to keeping the user engaged. Illustrations and animations that are not of high quality (resolution) and are not self-explanatory can significantly affect the app’s learning value.
User’s knowledge about using the app
This barrier can be removed by keeping UI/UX design as simple and intuitive as possible. A video containing special features of the app and how to use them in the “About” page can also help minimize this barrier.
There are thousands of health-related apps available now. Medical education-oriented apps are also increasing in number. Strategically advertising the app with the help of social media, by promotion during scientific conferences, and by using digital educational portals can help the app stand out and, in turn, can improve widespread acceptance.
Need for active internet or cellular data to access
Large numbers of high-resolution illustrations, images, videos, and animations can substantially increase the size of the app. Both the major smart phone platforms iOS and Android impose app size limits and would not accept bulky apps. This forces app developers to decrease the bulk of the app by including user interface and essential components responsible for smooth app operation in the app downloaded by the user and keeping all high-resolution illustrations, images, videos, and animations on a secure data server. In many parts of the world, active and fast Internet service is not readily available. Additionally, if the user is traveling in the countryside or located in the basement of a building, cellular data reception may not be optimal. To mitigate these issues, either a low-resolution version of the app can be provided to the user to be used offline and can automatically turn to a high-resolution version if Internet access is available. The other option is to create an Internet version of the app, which can be used with fixed ethernet-based devices (i.e., desktops and laptops) in the absence of wireless internet access.
Testing the app with standardized tools like system usability scoring provides further insight into app user interface design and defines the need for further improvements. Feedback from users is invaluable and helps improve the app design and scientific content. Mobile and tablet platform-based apps allow the opportunity to update the app content with the latest developments in the field or subject. Global reach and its intuitiveness are 2 major virtues of app-based education and learning. The primary goal of the present app is to improve knowledge about coronary bifurcation intervention among fellows-in-training and early carrier interventional cardiologists. Survey results showed that only 15% of the respondents were fellows-in-training and indicated that we need to adjust the strategy to reach more fellows globally.
Global reach, portability, swift learning, highly interactive user interface, and illustrations make mobile apps very effective educational tools. Sharing this app development experience may help other medical educators communicate their knowledge in more innovative ways, which will eventually help furthering the field of medical education.
This project received no outside funding. Mr. Kakkar is the scientific co- founder of and holds inventor equity in Mont Sinai subsidiary company Rx.Health. Dr. Sharma is a member of the Speakers Bureaus of Abbott, Boston Scientific, Cardiovascular System Inc., Abiomed, and Trireme. Dr. Atreja is the scientific founder of and holds equity in Mount Sinai subsidiary company Rx.Health. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- operating system
- system usability score
- user interface
- user experience
- Received May 28, 2018.
- Revision received August 19, 2018.
- Accepted August 21, 2018.
- 2018 American College of Cardiology Foundation
- Masic I.,
- Pandza H.,
- Toromanovic S.,
- et al.
- Sandars J.,
- Patel R.,
- Goh P.S.,
- et al.
- Kelli H.M.,
- Witbrodt B.,
- Shah A.
- Reusche R.,
- Buchanan P.J.,
- Kozlow J.H.,
- et al.
- Chase J.
- Stevens D.J.,
- Jackson J.A.,
- Howes N.,
- et al.
- Medina A.,
- Suarez de Lezo J.,
- Pan M.
- Sawaya F.J.,
- Lefevre T.,
- Chevalier B.,
- et al.
- Ormiston J.A.,
- Webster M.W.,
- El Jack S.,
- et al.
- Chen S.L.,
- Zhang J.J.,
- Ye F.,
- et al.
- Iakovou I.,
- Ge L.,
- Colombo A.,
- et al.
- Panwar S.R.,
- Rajamanickam A.,
- Kini A.
- Wang M.,
- Li X.
- Hou K.-C.,
- Ho C.-H.
- ↵Nielsen J. Usability 101: Introduction to Usability [Internet]. Nngroup.com. 2017 [cited 9 March 2017]. Available at: https://www.nngroup.com/articles/usability-101-introduction-to-usability/. Accessed November 14, 2017.
- Lewis J.,
- Sauro J.
- Sauro J.
- ↵Sauro J. Measuring U: 10 Things to Know About the System Usability Scale (SUS) [Internet]. Measuringusability.com. 2017 [cited 5 March 2017]. Available at: http://www.measuringusability.com/blog/10-things-SUS.php. Accessed November 14, 2017.
- Bangor A.,
- Kortum P.,
- Miller J.
- ↵U.S. Food and Drug Administration. Draft Guidance for Industry and Food and Drug Administration Staff—Mobile Medical Applications. Available at: http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm263280.htm. Accessed December 12, 2017.