Using Visual Diagrams to Communicate Complex Mixed-Methods Research Design Procedures

Mixed Methods in Public Health Research in Taiwan:

Using Visual Diagrams to Communicate Complex Design Procedures


Michael d. Fetters, MD, MPH, MA


Scholars introduce modern mixed methods research (MMR) and its application in public health research in Taiwan. Specifically, they showcase a multi-phased Taiwan Cervical Cancer Screening mixed methods study using visual diagrams to communicate complex design procedures. While some previous researchers have incorporated quantitative and qualitative data in research, here we hope to provide significant clarity to guide those new to the MMR field. We have structured the paper in the following way. First, we provide a brief overview of mixed methods research. Second, we illustrate the compelling need for MMR from a public health perspective using cancer screenings as an example. Third, we introduce the Taiwan Cervical Cancer Screening Program as an exemplar of MMR application and the utility of visual diagrams. Study methodology can be applied to international researchers and scholars from interdisciplinary fields beyond public health.

Keywords: mixed methods research, Taiwan, public health, cervical cancer screening, visual diagram


This is an Accepted Manuscript of an article published by Health Care for Women International on 28/11/2018, available online: https://www.tandfonline.com/doi/full/10.1080/07399332.2018.1516769

Many public health problems are complex. To date, most researchers have attempted to solve these complex problems by using monomethod research approaches. Monomethod research approaches in essence use exclusively quantitative research approaches or exclusively qualitative approaches (Teddlie & Tashakkori, 2009).  Over the past three decades, the mixed methods research paradigm has emerged as a third alternative to quantitative and qualitative research paradigms (Creswell & Clark, 2018). Although modern mixed methods research was established in the late 1980’s and still has a relatively short history, researchers around the globe are embracing mixed methods research (Fetters, 2016).  By integrating both qualitative and quantitative procedures, mixed methods research offers the power of numbers (quantitative) and stories (qualitative) for investigating complex social, behavioral and health sciences (Pluye & Hong, 2013).  Researchers and scholars from public health fields and other social and behavioral science areas can learn the value of modern mixed methods methodology and its development, and how to use visual diagrams to communicate complex mixed methods design procedures.

By definition Mixed Methods Research (MMR) is “a research approach or methodology: (1) focusing on research questions that call for real-life contextual understandings, multi-level perspectives, and cultural influences; (2) employing rigorous quantitative research assessing magnitude and frequency of constructs and rigorous qualitative research exploring the meaning and understanding of constructs; (3) utilizing multiple methods (e.g., intervention trials and in-depth interviews); (4) intentionally integrating or combining these methods to draw on the strengths of each; and (5) framing the investigation within philosophical and theoretical positions” (Creswell, Klassen, Plano Clark, & Clegg Smith, 2011).  A hallmark of mixed methods research is a focus on integration. Fetters and Freshwater propose all mixed methods researchers should take the mixed methods challenge, that is, to consider how to integrate both the qualitative and quantitative data together to achieve a whole greater than the sum of the individual parts. They illustrate this concept numerically as 1 + 1 = 3 (Fetters & Freshwater, 2015). Simply stated, this means the whole (3) is greater than the sum of the respective qualitative (1) and quantitative (1) parts. By integrating, researchers can gain new insights, and more robust conclusions. Ironically, while mixed methods seem strange to many researchers, there are many examples of mixed methods thinking outside of the research world. For example, consumers on their favorite shopping website leave and read reviews for the products they purchased using a five-star quality rating system as well as qualitative comments. By looking at both the star rating (quantitative) and the comments (qualitative), consumers can make informed decisions about purchases.

Mixed Methods Research Internationally and in Taiwan

International literature on mixed methods research

The modern field of mixed methods research was established in the late 1980’s, and there are over 50 books published on this topic. A number of events signify the growth and vibrancy of the field. The Journal of Mixed Methods Research completed ten years of publication in 2016 (Fetters & Molina-Azorin, 2017), and the International Journal of Multiple Research Approaches re-launched in 2017 after publication from 2007 to 2015. The Mixed Methods International Research Association (MMIRA) convened the first MMIRA conference in Boston, in 2014 (Mertens, et al., 2016). The Japan Society of Mixed Methods Research became the first affiliate of the MMIRA in 2016. In 2017, the first MMIRA Chapter, the Caribbean Chapter of MMIRA was announced. These developments underscore the growth of mixed methods research internationally.

Rationale for Using MMR in Taiwan

As a rapidly aging society, the population of people in Taiwan with cancers and chronic diseases has reached unprecedented levels that continue to grow (Chen, You, Lin, Hsu, & Yang, 2002). Cancer has continued to be the leading cause of death in Taiwan for more than 30 years. Trends of cancer incidence in Taiwan have shown a significant and persistent increase of 1.7% per year from 2002 to 2012 at rates higher than other countries (Chiang, et al., 2016).

While Taiwan has universal health coverage for regular preventive health services including cancer screenings among middle-aged adults age 40 years and older, cancer screening utilization is disappointingly low. Lessons learned from the few existing studies examining selected cancer-screening utilization among community and worksite groups in Taiwan reveal continued and general need to encourage preventive health services utilization (Hou & Chen, 2004; Hou & Chen, 2005; Hou, Fernandez, Baumler, & Parcel, 2002; Hou, Fernandez, Baumler, Parcel, & Chen, 2003; Hou, Hou, & Hou, 2014; Hou & Hou, 2014). Previous cancer screening studies in Taiwan show that only 62% of married women and 31% of single women from a community sample reported a Pap test in the preceding 3 years (Hou, et al., 2003), and less than 30% of a worksite sample had ever had a fecal occult blood test (FOBT) for colorectal cancer screening (Hou & Chen, 2004). Cancer education and screening programs are critical to relieve the cancer burden as many cancers are potentially preventable via modified behavioral lifestyles or early detection via regular screenings. Culturally tailored and innovative screening interventions have been demonstrated effective to significantly encourage more non-adherent women enrolled in the intervention group than in the control group to receive a pap test within 6 months (50% vs. 32%) (Hou, et al., 2002), and encouraged 74% of a midlife workplace sample to return a completed FOBT within 4 weeks (Hou & Chen, 2004; Hou & Chen, 2005).

Implementing effective cancer screening programs and public health services are complex and require interdisciplinary collaboration of many health professionals including nurses, health educators, physicians, social workers, care managers, as well as various community partners. Culturally tailored and innovative screening interventions require deeper understanding of the complex interplay of social, interpersonal, and individual factors influencing the adoption of preventive and screening behaviors. Evaluation and optimization of the delivery of public health programs requires sophisticated research procedures capable of adequately grasping the complexity of the public health enterprise. Mixed methods research which uses qualitative and quantitative data collection procedures, is particularly appropriate for addressing the complex research problems in the field of public health.

Despite the relevance of mixed methods research procedures for investigating complex public health issues, there are limited mixed methods research articles about public health in Taiwan. We conducted a literature search combining the key terms of “mix methods,” “public health,” and “Taiwan” using both the EBSCO host databases and “national library database” in Taiwan with no limitation on time period. This search failed to reveal a single article in the National Library Database in Taiwan when using the terms “mix methods” and “public health”.  We then just used “mix methods” as the key search term. The search revealed five articles from the National Library Database in Taiwan and only one article in the EBSCO host databases conducted in Taiwan using mixed methods.

Using “mixed methods research” as the sole search term may not return some studies that used mixed methods research procedures in cases when the authors did not use the language of mixed methods research (Molina-Azorin & Fetters, 2016). Thus, it is possible that additional studies could be identified if we had used a different search strategy. However, it is unlikely such studies would have used state-of-the-art mixed methods procedures if they did not include mixed methods in the paper. Thus, like Ivankova & Kawamura (2010), we chose “mixed methods” as our key term as we sought studies where the authors were aware of the mixed methods research paradigm and intentionally used “mixed methods” in their study titles or abstract. As a final step, we examined carefully the actual content of all articles identified in the search as “mixed methods studies,” to ensure that all the studies actually met criteria or discussed issues or trends related to mixed methods research.

Based on our literature search in Chinese and English, the few existing mixed methods research articles identified in Taiwan discussed either mixed methodology as a future trend in social and educational research (Hsieh, 2007), or the qualitative-quantitative debates and mixed methods as a new third paradigm (Kuo, 2011). Existing literature in Taiwan mostly pertains to applications in educational research (Tsai & Chauo, 2008; Sung & Pan, 2010). There were two articles identified applying mixed methods in the public health field. One evaluated a health screening program for migrant women to Taiwan (Huang, Mathers, Chia, Shiu, & Kao, 2016), and the other focused on exploring staff understanding and attitudes towards a hospital-balanced scorecard implementation (Ma, Hsu, Huang, Tsai, & Ying, 2011).

Researchers in Taiwan have shown the potential to produce quality mixed methods research. This could be enhanced by better understanding of mixed methods design procedures. Thus, the purpose of this illustration of mixed methods research in public health in Taiwan that follows is to enhance understanding of mixed methods research procedures and to demonstrate the potential for these procedures to be applied much more broadly in public health research. Lessons learned can also be applied to other countries across the world.

An application of mixed methods procedures in public health research

To provide an example of the application of MMR in public health research, here we introduce the Taiwan Cervical Cancer Screening Program, a sophisticated multi-phase mixed methods study, published via a series peer-reviewed articles in Journal of Community Health, Healthcare for Women International, AWHONN Lifelines,Health Promotion Practices, California Journal of Health Promotion, Preventive Medicine, International Journal of Behavioral Medicine, and a book chapter (Hou, et al., 2002; Hou, et al., 2003; Hou & Lessick, 2002; Hou, Fernandez, & Parcel, 2004; Hou & Luh, 2005; Hou, 2005; Hou, 2006a; Hou, 2006b).  This study, “Love Yourself before You Take Care of Your Family”, was a hospital-based community outreach program implemented to increase cervical cancer screening among women in Taiwan. This program of research involved a comprehensive process for the development and evaluation of a theory- and evidence-based cancer screening intervention program for Chinese women. This program of mixed methods research involved three major phases: (1) Instrument development and needs assessment (Hou et al, 2003; Hou & Lessick, 2002; Hou & Luh, 2005); (2) Intervention development using a framework called Intervention Mapping (Hou et al, 2004; Hou, 2006a; Hou, 2006b); and (3) Program evaluation using a randomized-controlled trial (Hou et al, 2002; Hou, 2005).  The mixed methods design procedures are illustrated using visual diagrams in order to facilitate communication and understanding [Creswell & Clark, 2018).  The visual diagrams of Phase I to IV of the study help provide a new way to communicate and link the complex mixed methods research design phases, procedures, and products together for readers to better understand the methodological approaches (Figures 1-4).

Phase I Exploring Sequential Design for Instrument Item Development

Phase I explored, developed, and tested study instrument items using an exploratory sequential design. As illustrated in Figure 1, the study explored survey items by reviewing existing literature and theories, and then developing qualitative interview questions. The qualitative data collection involved one-on-one interviews with 14 never-been-tested Chinese women to identify key barriers to cervical cancer screenings. Four screening belief constructs (perceived benefits, barriers, norms, and perceived cancer risk), and screening-related knowledge were identified with corresponding items drafted. This quantitative measurement tool was then preliminarily tested with a sample 125 women in Taiwan. The preliminarily quantitative results showed that measurement scales were reliable, and all four screening beliefs and knowledge were significantly related to cervical cancer screening history (Figure 1).

Figure 1 – Exploratory sequential design – Exploring, developing, & testing instrument items

Phase II Intervention Development Using Intervention Mapping Framework

Phase II was designed with lessons learned from Phase I.  Researchers produced a three-month direct-mail campaign intervention program that was pretested in an 8-person focus group. As illustrated in Figure 2, this “Love Yourself before You Take Care of Your Family” cervical cancer screening project was developed using the Intervention Mapping framework (IM), an innovative process of designing theory- and evidence-based interventions. Intervention Mapping involves an inter-connected iterative six-step process to ensure that theory and evidence guiding (a) the identification of psychosocial and environmental determinants related to cancer screening behavior, and (b) the selection of the most appropriate methods and strategies to address the identified determinants (Figure 2).

Figure 2 – Intervention Development & Pre-testing

Phase III Evaluation Study Using Embedded Mixed Methods Research Design

Phase III was a randomized controlled trial (RCT) to evaluate the effectiveness of the intervention developed in Phase II. The primary outcome of the RCT evaluation study was receiving a cervical cancer screening (Pap smear test) after the intervention. A secondary outcome was change knowledge and beliefs concerning cervical cancer screening. Female family members of inpatients who were admitted into a major teaching hospital in Taichung, Taiwan were asked about their cervical cancer screening history. Women who had not had a cervical cancer screening in the past 12 months were identified as non-adherent and thus the were eligible to participate in the randomized controlled intervention trial (total n=424; with 212 women in each group).  As illustrated in Figure 3, this quantitative-dominant evaluation design embedded qualitative data collection (a) beforethe experiment to gather input on measures, as well as intervention ideas and feasibility; (d) duringthe experiment to document implementation process and fidelity; and (c) after the experiment to describe the impact (Figure 3).

Figure 3 – Evaluation Embedded Design (Randomized-Controlled Trial)

Results showed women in the intervention group reported a higher rate of receiving cervical cancer screening than women in the comparison group (50% vs. 32%; p=.002).  Baseline data from the randomized controlled intervention trial was further used to confirm the final validated study instrument. Reliabilities showed good internal consistencies for the perceived pros, cons, and susceptibilities scales (alpha ranged from .78 to .87). Factor analysis showed good construct validity revealing concordant patterns with the behavioral constructs used. This validated culturally sensitive and theory-based measurement tool was published with English-Chinese side-by-side to facilitate use by other researchers conducting similar studies (Hou & Luh, 2005). This multi-phase mixed-methods cervical cancer screening program among Chinese women was invited for inclusion in the CDC Chronic Disease Prevention Database to serve as a model to assist researchers and practitioners in planning, implementing, and evaluating health promotion programs in cancer screening (Figure 4).

Figure 4 – Multi-Phase Mixed Methods Research Design

This Taiwan cervical cancer screening project was published using the concept of staged integration (Fetters, Curry, & Creswell, 2013) via a series of papers from the single multi-phased mixed methods study, including a qualitative paper (Hou & Lessick, 2002), two quantitative papers (Hou, et al., 2003; Hou, 2006a); an instrument development paper (Hou & Luh, 2005), a mixed methods intervention development paper using the Intervention Mapping framework to link all three major phases together (Hou, et al, 2004), two evaluation papers highlighting the project outcomes (Hou, et al., 2002; Hou, 2005), and book chapter used to introduce comprehensively the overall project through use of figures, tables, and intervention material samples (Hou, 2006b).

In this paper, we have introduced this sophisticated multi-phase project that illustrates the complexity, rigor, power, and synergy for publication resulting from using a mixed methods approach. The mixed-methods diagrams have been found to greatly facilitate understanding of various study phases and designs [Phase I, II, III, and the multi-phase figures], and showcase how a series of multiple papers, including a methodological paper, can be published from a program of mixed methods research and integration via the reporting dimension.


As illustrated by the example presented here, mixed methods research offers the power of numbers and stories for investigating complex social and health problems. Understanding the context for using MMR in Taiwan from a public health perspective underscores the opportunity for applications of mixed methods in public health research. While a concise overview of mixed methods research, this paper illustrates a sophisticated mixed methods cervical cancer screening program of research based in Taiwan as an exemplar of MMR application. Visual diagrams greatly facilitate communication of complex mixed methods design procedures. As illustrated, multiple papers can be published from a mixed method program of research, and this work illustrates mixed methods research integration through the reporting dimension.  The methodology discussed can be applied to interdisciplinary fields of research and across the world.


  • Ethical approval: “All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.” 
  • Informed consent: “Informed consent was obtained from all individual participants included in the study.”
  • Disclosure of potential conflicts of interest: “The authors declare that they have no conflict of interest.”


Chen, C.-J., You, S.-L., Lin, L.-H., Hsu, W.-L. and Yang, Y.-W. (2002) Cancer epidemiology and control in Taiwan: a brief review. Japanese journal of clinical oncology, 32, S66-S81.

Chiang, C.-J., Lo, W.-C., Yang, Y.-W., You, S.-L., Chen, C.-J., & Lai, M.-S. (2016) Incidence and survival of adult cancer patients in Taiwan, 2002–2012. Journal of the Formosan Medical Association, 115,1076-1088

Creswell, J., & Clark V.L. (2018). Designing and conducting mixed methods research. (3rded.) Sage publications, Inc.  ISBN: 978-1-4833-4437-9

Creswell, J.W., Klassen, A.C., Plano Clark, V.L., & Smith, K.C. (2011). Best practices for mixed methods research in the health sciences. Bethesda, MD: National Institutes of Health.

Fetters M.D. “Haven’t we always been doing mixed methods research?”: Lessons learned from the development of the horseless carriage. Journal of Mixed Methods Research, 10(1), 3-11, 2016.

Fetters M.D., Curry L.A., & Creswell J.W. (2013). Achieving integration in mixed methods designs – principles and practices. Health Service Research, 48(6 Pt 2), 2134-56.

Fetters M.D., & Freshwater D. (2015). The 1 + 1 = 3 Integration Challenge. Journal of Mixed Methods Research, 9(2), 115-7.

Fetters MD and Molina-Azorin JF. (2017). The Journal of Mixed Methods Research Starts a New Decade: The Integration Trilogy and Its Dimensions. Journal of Mixed Methods Research. 11(3).

Hou S.-I.  (2005). Stage of adoption and impact of direct-mail communications with and without phone intervention on Chinese women’s cervical smear screening behavior. Preventive Medicine, 41, 49-756.

Hou, S.-I.(2006a).  Perceived spousal support and beliefs towards cervical smear screening among Chinese women. California Journal of Health Promotion, 4(3), 157-164.

Hou, S.-I. (2006b). Using Intervention Mapping in Program Development to Promote Cervical Cancer Screening among Chinese Women – From Needs Assessment, Intervention Development, to Program Evaluation.  In George Rolland (Ed.), New Research on Cervical Cancer.  (Chap 1. pp.1-46).  Hauppauge, NY: Nova Science Publishers, Inc. (ISBN 1-60021-300-6).

Hou, S.-I., & Chen, P.-H. (2004) Home-administered fecal occult blood test for colorectal cancer screening among worksites in Taiwan. Preventive Medicine, 38, 78-84.

Hou, S.-I., & Chen, P.-H. (2005) Cancer screening beliefs and reactions to an innovative colorectal cancer screening kit among Chinese worksite population. Methods of Information in Medicine, 44, 315-318.

Hou, S.-I., Fernandez, M. E., Baumler, E. & Parcel, G. S. (2002). Effectiveness of an intervention to increase Pap test screening among Chinese women in Taiwan. Journal of community health, 27, 277-290.

Hou, S.-I., Fernandez, M., Baumler, E., Parcel, G, & Chen P. (2003).Correlates of cervical cancer screening among women in Taiwan.  Health Care for Women International, 24(5), 384-398.

Hou, S.-I., Fernandez, M., & Parcel, G. (2004). Development of a cervical cancer educational program for Chinese women using Intervention Mapping. Health Promotion Practice, 5(1), 80-87.

Hou, S.-I., & Hou, P.-H. (2014) Developing and Validating a PHSU Belief Inventory (PHSU-BI) among Chinese Middle-aged White-and Blue-collar Workers. International Journal of Health, Wellness and Society, 4, 13-20.

Hou, S.-I., Hou., P.-H, & Hou., H.-S. (2014) The impact of a colorectal cancers worksite screening intervention on knowledge, screening beliefs, and uptakes among middle- and older-age employees in Taiwan. Journal of Cancer Education, 29, S12-13.

Hou, S.-I., &Lessick, M. (2002). Cervical cancer screening among Chinese women: Exploring the benefits and barriers of providing care. AWHONN Lifelines, 6(4), 349-354.

Hou, S.-I., & Luh, W. (2005).  Psychometric properties of the Cervical Smear Belief Inventory (CSBI) for Chinese women.  International Journal of Behavior Medicine, 12(3),180-191.

Huang, Y.-C., Mathers, N. J., Chia, S.-L., Shiu, M.-N., & Kao, S.-C. (2016). An evaluation of a screening programme for immigrant women to Taiwan. Family Medicine and Community Health, 4(2), 5-15.

Hsieh C.W. (2007). Educational Research Paradigm of Future: The Introduction of Mixed Methodology. Ping-Tong Education University Report, 175-194. [In Chinese: English abstract]

Ivankova N, & Kawamura Y. (2010). Emerging trends in the utilization of integrated designs in the social, behavior, and health sciences. In: Tashakkori A, Teddlie C eds, Handbook of mixed methods in social & behavioral research. 2th ed. Thousand Oaks, CA:Sage, 581-611.

Kuo C.Y. (2011). Mixed research and the qualitative-quantitative debat. Soochow Journal of Political Science, 29(1), 1-64. [In Chinese: English abstract]

Ma C.C., Hsu M.H., Huang S.K., Tsai Y.T., & Ying J.C. (2011). Using mixed methods to explore staff understanding, attitudes and problems in the iImplementation of balanced scorecard. Journal of Medical Management(Taiwan), 12(2), 98-113.[In Chinese: English abstract]

Mertens, D., Bazeley, P., Bowleg, L., Fielding, N., Maxwell, J., Molina-Azorin, J.F., & Niglas, K. (2016). The future of mixed methods: A five year projection to 2020.  Retrieved from: https://mmira.wildapricot.org/resources/Documents/MMIRA%20task%20force%20report%20Jan2016%20final.pdf

Molina-Azorin J.F., & Fetters M.D. (2016). Mixed methods research prevalence studies: Field-specific studies on the state of the art of mixed methods research. Journal of Mixed Methods Research, 10(2):123-128.

Pluye, P., & Hong, Q.N. (2013). Combining the power of stories and the power of numbers: Mixed methods research and mixed studies reviews. Annual Review of Public Health, 35, 29-45.  https://doi.org/10.1146/annurev-publhealth-032013-182440

Sung Y.T., & Pan P.Y.(2010).Applications of mixed methods research in educational studies. Jouranl of Educational Research, 55(4), 97-130. [In Chinese: English abstract]

Teddlie C., & Tashakkori, A. (2009). Foundations of Mixed Methods Research-Integrating quantitative and qualitative approaches in the Social and Behavioral Sciences. Sage Publications, Thousand Oaks, CA.

Tsai H.M., & Chauo C.W. (2008). The paradigmatic analysis of adult education research and discussing of the tendency of adult education research in Taiwan. Humanity of Social Science, 159-177. [In Chinese: English abstract]


Share This Book


Comments are closed.