Review Article

How Are Continuous Quality Improvement (CQI) Approaches Used in Evaluating Management Development Programs? a literature review

Leigh-ann Onnis, Marcia Hakendorf, Komla Tsey
APJHM 2018; 13(2):i10.  doi:10.24083/apjhm.2018.0010
 
James Cook University Cairns Campus - College of Business, Law & Governance. PO BOX 6811 Cairns Queensland 4870, Australia
CRANAplus - Professional Services, Adelaide, South Australia, Australia
Correspondence: leighann.onnis@jcu.edu.au

Abstract

Aim: The aim of the review was to examine the characteristics of studies that use CQI approaches to evaluate management development programs; and to synthesise the findings to understand how CQI approaches are being used to evaluate the effectiveness of management development programs.  
Method:
A scoping review of the literature was conducted in a manner consistent with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. The matches were screened by title and abstract using the inclusion criteria, leading to a full paper review of 48 papers. Of these, the 14 papers found to meet the inclusion criteria for the scoping review were independently reviewed and analysed by two of the authors.
Findings:
The review revealed the ways in which CQI approaches were used in evaluating management development programs highlighting the role of context, pre-determined competencies and participatory CQI approaches. Participatory CQI approaches including on-the-job application of learning provided opportunities for participants to learn through CQI activities associated with action learning and CQI feedback cycles.
Conclusions: 
The authors concluded that evaluations using participatory CQI approaches are better positioned to report more comprehensively on the benefits of management development programs when they include the competencies required to be successful in the context within which the manager is working.  Future directions for research in this area include an examination of the microsystem context to determine whether the required management competencies associated with remoteness differ from other contexts.

Keywords: CQI, Continuous improvement, Program evaluation, Management, Development


Introduction

More than a decade has passed since Drucker observed that large healthcare organisations are the most complex institutions in history, and that small healthcare centres are barely manageable.[1]  Yet, the complexity of health services, together with the increasing demands placed on health managers remains relevant.[1] With predicted health workforce shortages globally, strong leadership and management are essential, particularly in regions that traditionally experience workforce shortages, if they are to deliver quality health services. The WHO[2] policy recommendations about improving access to health workers in areas where workforce shortages are common (e.g. rural and remote areas) encourages all countries to strengthen leadership development programs and create supportive workplaces.[2] Furthermore, the WHO suggest that the geographical context requires specific interventions ‘because addressing rural and remote areas will also address the needs of underserved populations more broadly.’[2, p.9]  Similarly, CRANAplus [3] and the National Rural Health Alliance[4] report specific issues associated with the characteristics of remoteness. Therefore, a focus on improving management capability, particularly for managers working in remote areas where remoteness exacerbates the difficulties in accessing management education, training and support will contribute to health service improvements. [4-6]

The challenges for managers stem from widespread health system reform, health service restructuring, economic pressure from aging populations, increased demand for health services and funding reforms.[7,8]  In Australia, a review of health service management raised concerns about widespread skill deficiencies, particularly a need to develop skills in building and nurturing relationships.[8] This is not restricted to health services, with a major review (Karpin Report) into ways of improving management development in Australia into the 21st century raising awareness about the relationship between management capability and organisational performance. The Karpin Report highlighted the critical role of education and professional development, particularly, improved non-technical ‘soft’ skills (e.g. managing people, communication) in ensuring managers have the skills needed to be effective at any level of management. [9]

Management Development

Management development refers to a planned process of training, or specifically chosen capacity building activities, resulting in management capabilities benefitting the organisation. [10] Often, management development is self-directed with benefits experienced directly by the manager, and indirect benefits filtering through to health services. Often, evaluations reinforce individual benefits focusing on the participant’s experience and satisfaction, missing the opportunity to evaluate changes in their performance as a manager and the quality of health services. [11,12]

 

Continuous Quality Improvement (CQI) approaches

Generally, CQI is viewed as an opportunity to reflect on the success of an activity and how it could be improved. More specifically, program evaluation is a process of assessing ‘the total value of training: that is the cost-benefit and general outcomes, which benefit the organization as well as the value of the improved performance of those who have undertaken training.’[11, p.14]  Therefore, it is imperative that management development program evaluations measure improvements in individual performance and service quality. CQI approaches are often used to bridge gaps between best practice evidence and what happens in practice with a view to improving population level health outcomes. [13] Hence, CQI is used in identifying problems, developing solutions and evaluating changes to ensure that education and training programs meet the needs of the participant, provide cost-effective management development solutions for organisations, and lead to service improvements for customers. [13,14]

The CQI literature highlights the influence of context with some studies attributing variation of results to differences in the context for the CQI initiatives.[15-17] These contextual or within service factors are described as microsystems.[18] Microsystems are defined as ‘small groups of people who regularly work together to provide care.’[17, p.503]  In microsystems, context includes the characteristics of individuals, the organisation, the physical, and cultural environment (e.g. supportive clinical leadership, workforce stability).[16,17] There is limited understanding about how the drivers of CQI effectiveness in a microsystem interact with one another and/or with other contextual factors to achieve the desired impacts in primary health care services.[12,17]  Despite this, it is believed that lessons learned in one microsystem, provide valuable insights for other microsystems with similar characteristics.16  Hence, this scoping review contributes to the literature about how CQI approaches are used in evaluating programs for a subgroup  (managers) who have considerable influence over their particular microsystem.

This scoping review analysed and synthesised the existing literature to answer the question, How are CQI approaches used in evaluating the effectiveness of management development programs?  There were few health service specific management development programs reported in the literature, so a broad scope was necessary in an effort to identify literature that describes empirical studies with management development programs containing general management and/or soft management skills training; areas of need identified in the health service management literature.[2,8]  Therefore, the aim of this scoping review is to examine the characteristics of studies that use CQI approaches to evaluate management development programs, and to synthesise the findings to understand how CQI approaches are being used to evaluate the effectiveness of management development programs.

Method

A scoping review of the literature was conducted using accepted scoping review methods and in a manner consistent with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement (Figure 1). [19, 20, 21] The literature search used a combination of the search terms ‘evaluat*’, ‘manage*’, ‘program’, ‘training’ using One Search which searched multiple databases including: informit, CINAHL, EBSCOHost, OvidSP, OvidMP, PubMED, ProQuest and the Wiley online library.  The literature searches resulted in variable matches across search terms; however, a larger number of matches did not result in a higher number of articles that met the inclusion criteria. Next, a search using the same search terms was conducted on the Emerald Insight and the Cochrane Library databases. An additional search of systematic reviews was conducted using the Cochrane Library database, which identified two systematic literature reviews, both of which authors LO and MH agreed did not meet the inclusion criteria.

The matches were screened by title and abstract against the inclusion criteria:

  1. Peer reviewed
  2. Published in English
  3. Published between 1/10/1997-1/10/2017
  4. Used a CQI approach to evaluate a management development program that included general management and/or ‘soft skills’ management training

All matches were screened and 48 papers were included in the full paper review. These papers were read in full by author LO, with 14 papers selected for the scoping review (Figure 1).  While quality assessment does not usually form part of a scoping review, the requirement that the selected papers were published in peer reviewed journals was a proxy for research quality. [19,20] Authors LO and MH agreed that the 14 papers selected met the inclusion criteria for the scoping review.  The review commenced with an analysis of the broader characteristics of the publications using the key characteristics (Table 1 and Table 2) developed by author LO through an iterative process. To minimise researcher bias, authors LO and MH, independently reviewed and analysed the selected papers using the pre-identified key characteristics, and resolved the small proportion (approx 5%) of coding disagreements through a consensus-seeking procedure. Through reviewing the papers and discussing the coding together, interrater differences were addressed and resolved.

 

Figure 1: Scoping review process using the PRISMA statement

Results

Characteristics of the studies

An examination of the selected publications illustrated the diversity in characteristics of the management development programs (Table 1).  An analysis of the characteristics highlighted the similarities and differences in management programs from across the globe: United Kingdom (UK) and Ireland, Europe (Netherlands, Sweden, Serbia), Asia (China, Iran), Canada and the United States of America (USA). There were no Australian studies; however, one contained an Australian program customised for managers in China. The programs ranged from ten days to two years with 43% less than one year, 29% one year, 14% more than one year, and for 14% the duration was not reported. The studies were across four industries (health, hospitality, insurance and gaming), in both government organisations and private organisations. Only two were pilot studies; however, many of the studies were small (half had less than 60 participants). Many (43%) were accredited training programs, with the remainder being non-accredited training programs.  While many (43%) did not report how participants were selected; of those that reported the selection process, most frequently (43%) participants were nominated by senior management. Two studies provided training for all managers in the organisation, and for another, managers self-nominated.

Most frequently, the studies were mixed methods (43%), followed by quantitative (36%) and qualitative (21%).

Overall, the studies reported that the programs were developed to achieve three aims: provide managers with qualifications; improve management capacity, knowledge and/or skills; and to improve retention.  The evaluations of the management programs aimed to examine: the benefits of the project undertaken in the study (including the returns from the program); the effectiveness of the CQI evaluation approach used; the aspects of the programs that influenced its effectiveness; an improvement in management skills and capacity; and organisational level improvements. Notably, the aim of the evaluation was not always consistent with the aim of the program (e.g. Adams and Waddle [11] evaluated the returns, including the impact of outputs on profitability, for a program that aimed to provide managers with management qualifications).

Table 1: Characteristics of the Management Development Program

ID No.

Author

(date)

Program Name

Country

 

Industry

 

Sample size

 

 

Management Program Type

 

Program duration

Selection

Program Aim

Purpose of the study

On the job

Study Type

Predetermined

Competencies

Findings

ID1

Adams & Waddle (2002)

Whitbread Enjoy Learning

UK

Hospitality

NR

University

(post graduate)

NR

NR

Gain qualifications using a project driven approach

Explore how the value of the strategy was assessed

Y

Case Study

 

N

Benefits directly attributed to the work-based projects

ID2

Doyle

(2014)

Leader’s Edge

Ireland

 

Health

7

Accredited - level 8 module

(undergraduate)

7 months

NOM

Enhance manager capacity and bring about change

Evaluate a leadership training program

Y

Program

Evaluation

(Pilot)

N

Action learning is effective in developing leaders and supporting change

ID3

Wallis & Kennedy (2013)

Leadership for Resilience

USA

 

Nursing

25

 

Education Program

1 year

(4 x residentialretreats)

NOM

Promoting team-based approaches to improve nurse retention

Assess the effectiveness of the training program

Y

Program

Evaluation

Y

Effectiveness is influenced by leaders’ emotional intelligence and organisational culture

ID4

Omar et al. (2009)

NR

UK & Iran

 

Health

23

Education Program

7 courses (1-10 weeks) over 1 year

NOM

Tailored program for capacity building

Evaluation to guide future program development

N

Program

Evaluation

N

Training evaluations should assess learning and communicate results

ID5

Saleh et al. (2004)

NEPHLI

USA

 

Public Health

81

University modules

1 year

ALL

To improve the leadership skills

Evaluate program effectiveness against predetermined competencies

N

Program

Evaluation

 

Y

Participants’ skill level improved across all 15 competency areas

ID6

West et al. (2016)

Nurses Emerging as Leaders

USA

Nursing

75

Education Program

1 year

(8-hours every 4-6 weeks)

NOM

Preparing nurse leaders for role

transition and leadership

Evaluate participant competency improvements compared with non-participants

Y

Program

Evaluation

 

Y

The program improves succession planning by developing leaders who are prepared for leadership positions

ID7

Throgmor-ton et al.

(2016)

Physician Leadership Academy

USA

Health

21

Education Program

10 months

NR

To develop strong physician leaders in healthcare

Outline evaluation strategy and inaugural program outcomes

Y

Program

Evaluation

 

N

The program met targeted outcomes across all levels of evaluation

ID8

Berg & Karlsen

(2012)

Coaching

NR

Fortune 500 company

59

Education Program

5 x 2-day seminars

NR

Change behaviour to improve use of manager’s toolbox

Examine the effect of coaching on leadership development

Y

Case Study

N

Participants learned a variety of solutions from the manager’s toolkit

ID9

Supic et al. (2010)

Project for capacity building

Serbia

Health

107

University modules

1-2 years

NOM

Improve particular management skills

Identify improvements and explore predictors and relationships

Y

Cohort Study

Y

Training programs can improve competencies which improves competitive advantage

ID10

Steensma & Groen-eveld (2010)

NR

Nether-lands

Government

54

NR

NR

NR

CG

Improve growth, knowledge, and performance

Demonstrate the value of experimental designs in evaluation studies

N

Experimental with a control group

N

Demonstrated ‘good’ management behaviours; differences in knowledge acquisition but not in behaviour

ID11

Hayes

(2007)

Dimensions Leadership Program

Canada

Gaming

258

Pathway to certificate in management

2 week training sessions

ALL

Build existing skills and build stronger leaders

Evaluation of a leadership development initiative

N

Case Study

Y

Positive impact on the leadership competency; positive impact on KPIs

ID12

Fealy et al. (2015)

Clinical Leadership Development

Ireland

Nursing

70

Education Program (including mentoring / coaching)

6 months

NR

Individual and service level development improvements

Evaluation of leadership development programs

Y

Case Study

(Pilot)

N

Clinical leadership development can impact on service in distinct and identifiable ways

ID13

Yaping & Staton

(2002)

Health management Training Course

China

Health

233

Program formally recognised by the Hospital

Accreditation Committee

2 years (part-time)

NR

Improve understanding of management, skills and efficiency of the health sector

Evaluate the impact of training in management practice

Y

Program

Evaluation

 

N

Positively impacted on health management practices and made a significant contribution to management education

ID14

Holmberg et al.

(2016)

Leadership Development Program

Sweden

Insurance

107

Education Program

12 days (2-3 day residential seminars)

NOM

SS

CG

 

Increase participants’ leadership skills and capacities

Evaluate a leadership development program

Y

Program

Evaluation

 

Y

Outcomes were meaningfully operationalised for generic skills and health and wellbeing

ALL = All; CG = Control Group; EVAL = Evaluation study; N = No; NEPHLI = Northeast Public Health Leadership Institute leadership training; NOM = Nominated; NR = Not reported; SS = Self-selected; Y = Yes

Six studies used predetermined competencies and measured the participant’s progress using these competencies. Four of these studies (three in USA and one in Serbia) were conducted with health managers. The pre-determined competencies contained in the studies prescribed a variety of competencies presented as two skillsets: soft management skills and hard management skills (Table 2).

Table 2: Competencies used to evaluate management development programs

 

Soft management skills

Self-awareness and

Self-development


Emotional Intelligence (ID3)

Creating a leader in yourself (ID5, ID11)

Professional self-development (ID9)

Organising and time management (ID9)

Self-development and initiative (ID11)

People management

Leading people (ID5, ID11, ID14)

Supervision (ID9)

Motivation and guidance (ID9, ID14)

Creating positive atmosphere (ID9)

Delegation (ID14)

Managing change (ID5, ID9)

Working

with others


Collaboration (ID3, ID5, ID9)

Teamwork (ID3, ID5, ID11, ID14)

Relationship building (ID5, ID14)

Valuing diversity (ID5,ID11)

Integrity and building trust (ID11)

Communication

Oral and written communication (ID5, ID9, ID11)

Using media and forums to inform and educate (ID5)

Using visual representations of data (ID5)

Hard management skills

Managing

the Business

Knowledge of the business, policy, law (ID5, ID9, ID11, ID14)

Strategic planning (ID9, ID11)

Operational planning (ID9)

Resource mobilisation (ID5)

Evidence-based decision making (ID5, ID9, ID11)

Systems change (ID3)

 

 

Context

The scoping review sought to consider the influence of context. Unfortunately, it was not possible to consider geographical context as the selected publications did not provide sufficient information to determine whether the managers worked in urban, rural or remote locations. However, context was raised in six studies. Yapping and Stanton [22, p.166] reported that contextual issues may arise from cultural differences in management education explaining that ‘China has been slow to develop the concept of student participation.’ Omar et al. [23, p.10] reported that not collecting information about context from respondents ‘made it impossible to assess to what extent reported changes (or not) in practice were related to the training programme or to factors in the organizational environment.’ Wallis and Kennedy [24] found that context may influence the manager’s success in workplace application of skills. Similarly, Fealy et al. [25, p.331] emphasised the importance of evaluating ‘the expression of these competencies in context, i.e. in the everyday performance of the leader’s professional role’; and Steensma and Groeneveld [26, p.331] proposed that analysis ‘should lead to cumulative knowledge of causal connections between characteristics of persons, interventions, and contexts.’ Finally, Holmberg et al. [27, p.165] explained that their quasi-experimental design was ‘vulnerable to a range of contextual influences’. Hence, context creates both interconnectivity and complexity for program evaluations. 

 

CQI approaches

CQI approaches are being used to evaluate management development programs at the individual and organisational level.  There were a variety of CQI approaches, reported, including: action learning (29%); CQI feedback cycles (e.g. learn-apply-feedback-review, plan-develop-implement-evaluate) (14%); staged continuous learning approaches (7%); best practice (7%); pre/post training evaluation (14%); and post training evaluation (29%) (Table 3).  Of the ten programs that included an on-the-job component, seven evaluated at an individual and organisational level, using action learning (43%), CQI feedback cycles (29%); best practice (14%) and post training evaluation (14%). Further, the programs using a CQI project to demonstrate the application of skills used either action learning (67%) or a CQI feedback cycle (33%).

 

Table 3: CQI approaches for determining impact for management training programs.

 

ID No.

Author

(Date)

CQI Approach

Purpose of Evaluation

Participatory learning

Participatory evaluation

CQI Project

Study Type

Impact Level Evaluated

Impact Reported

Cost-benefit

Evaluation recommendations

ID1

Adams & Waddle (2002)

Action Learning

Evaluate the returns from the program

Y

N

Y

MMR

Individual

Organisational

Benefits were directly attributable to the work-based projects; financial savings

Y

Action Learning is a powerful tool for relevant knowledge to be brought to the workplace and helps personal learning and transformation

ID2

Doyle (2014)

Action Learning & Reflective Practice

Evaluate the returns from the program

Y

Y

Y

Qual

Individual

Organisational

Improved leadership skills. Projects had positive impacts

Y

The project must be sufficiently difficult to promote learning, the mix of participants, and organisational commitment is crucial.

ID3

Wallis & Kennedy (2013)

CQI Projects –plan, develop, implement and evaluate.

Can differences be attributed to the LR training?

Y

N

Y

MMR

Individual

Team

Organisation

Success at the team level was affected by success at the individual and organisational level

Y

Evaluation suggests that success of the program may be more directly related to selection of teams and to organisation’s commitment than to the ideas proposed by individuals.

ID4

Omar et al (2009)

Staged approach with continuous learning

To evaluate reaction, learning, application and organisational impact

Y

N

N

MMR

Individual

Organisational

Participant satisfaction with the training; 81% could perform their jobs better

N

Consistent use of evaluation over time for comparisons

ID5

Saleh et al.

(2004)

Post training evaluation

Evaluate program’s effectiveness against predetermined competencies

N

N

N

Quant

Individual (program level)

Good long-term outcomes (e.g. PhDs and occupations)

N

Public Health leadership training programs are effective in improving skills

ID6

West et al.

(2016)

Post training evaluation

Evaluation of program effectiveness

Y

N

N

Quant

Individual

Organisation

Improves morale, succession planning and personal satisfaction

 

N

Further evaluation is required to ensure the content remains current, and that individual and organisational needs are met.

ID7

Throgmorton et al.

(2016)

Best practice

Evaluation of program effectiveness

N

N

N

MMR

 

Individual

Organisational

Continued engagement post training

Y

Following participants over time would yield more information on long-term impact of leadership development programs.

ID8

Berg & Karlsen

(2012)

Learn-Apply-Feedback-Review Cycle

Setting the context and reflecting on the meaning of the experience.

Y

Y

N

Qual

Individual

Organisational

Participants can learn to solve real work challenges through coaching

Y

Future research should apply a comprehensive research design (e.g. control group).Supervisors and subordinates should be involved in the training process

ID9

Supic et al. (2010)

Pre/Post training evaluation

Identify improvements

Y

N

N

Quant

Individual level

The time in the management position influenced individual skill improvements; influenced by duration as a manager

N

Study can be improved with data other than self-reported and having a control group

ID10

Steensma & Groeneveld (2010)

Evaluation -pre, during and post program

Program effectiveness

Y

N

N

Quant

Individual

Organisational

No long term outcomes reported

N

The 4–levels method gives detailed insights in results

ID11

Hayes

(2007)

Action Plans to foster CQI

Examine the level impact of training

N

N

N

MMR

Individual

Organisational

No

N

Preplanning for the evaluation process was critical to ensure a comprehensive program evaluation

ID12

Fealy et al. (2015)

Action learning, Service Assessment Tool (SAT)

Evaluate the program’s service impact

Y

N

N

Qual

Individual

Organisational

Direct impact related to projects and indirect impact arising from the program participation

N

Service user data is needed to examine service impact. Time bound studies cannot establish long-term impact

ID13

Yaping & Stanton (2002)

Post training evaluation

To improve the training program

N

N

N

MMR

Individual

Impact limited by workplace factors

N

The findings will be of interest to other health service management programs but cannot be generalised

ID14

Holmberg et al.

(2016)

Pre/Post training evaluation

Increase workforce health /wellbeing

Y

N

N

Quant

Individual

Significant increase in self-reported LSE and PS; there may be a positive effect on health and wellbeing

N

Leadership evaluation programs can be evaluated within a framework of generic leadership skills and health-related outcomes supporting more theoretically anchored learning

Behaviour Response Inventory (BRI) (Schutte & Mellen 1999); Leadership Practices Inventory (LPI) (Koues & Posner 2007); Life Orientation Test – Revised (Scheirer et al 1994);
LSE = leadership self-efficacy; Mixed Methods Research (MMR); N = No; PS = Political skills; Qualitative Research (Qual); Quantitative Research (Quant); RIHEL = Imputed Regional Institute for Health and Environmental Leadership (RIHEL) programme change model; SOC = Sense of Coherence; Y = Yes.

 

Ten studies (71%) used participatory approaches for learning and skill development; yet, only two studies (14%) used participatory approaches in evaluating the program.  While five studies (36%) reported conducting a cost-benefit analysis, only four reported tangible organisational benefits, such as ‘One of the work-based project outcomes provided an immediately workable solution and in turn it is estimated has saved the WPLC £1.5 million in development and implementation costs’ [11, p.20] and ‘With all salary/benefit and program costs compared to these dollar savings, the 2012-2013 PLA yielded a 106 percent ROI.’ [28, p.404] 

Stratifying the selected publications by level of evaluation (individual, or individual and organisational), and then following the evaluation process from program content (e.g. participatory learning approaches) and on-the-job application of skills, through to outcomes (e.g. cost-benefit analysis) and CQI evaluation approach reveals how CQI approaches are being used (Figure 2).  Of interest to this review, three studies evaluated management development programs at both the organisational and individual level and included the implementation of a CQI project. These same three studies, as well as two others, evaluated cost-benefits as an outcome for the program.  Furthermore, two of the studies that evaluated for cost-benefits included a participatory CQI approach to evaluation (Figure 2).

Figure 2: CQI approaches to management development program evaluations

 

Discussion

The review revealed that the most compelling evidence for the effectiveness of management development programs arises from studies using participatory CQI approaches for evaluating on-the-job application of skills leading to organisational benefits. Further, the studies that included the development and implementation of a CQI project not only provided for real-world application of skills, they provided an opportunity to measure organisational impact, including cost-benefits.  The synthesis highlighted three factors to consider when using CQI approaches to evaluate management development programs: context, core competencies, and participatory CQI approaches to evaluation.

Context

The review suggests that both organisational and ethnically-based cultural contexts influence the implementation and evaluation of management development programs.[16,17,22- 24]  Some organisational cultures nurture real-world skill development, are open to change and create safe, supportive environments for managers to practice and refine their skills; however, some are not.[17,18,29] Therefore, evaluations that assess the application of skills must consider the cultural context of microsystems.[17,18,25]

The review contained a study where a participatory approach was not compatible with traditional Chinese culture highlighting the importance of considering the compatibility of participatory CQI approaches for programs that contain cross-cultural groups. Particularly, where western perspectives of management are presented to participants from non-westernised cultures. [22]   Also, evaluations should consider the cultural context when employees are asked to provide feedback about their manager’s performance as it may be contrary to culturally accepted behaviour (e.g. respect for a hierarchy, fear of losing their job).  Therefore, it is essential that management development programs are customised to ensure that they use contextually and culturally responsive participatory CQI evaluation approaches.  

Core competencies

Core competencies recommend the skills needed for a manager to perform at a level that meets organisational and customer expectations. Few of the selected studies used pre-determined competencies in their evaluations; despite many credentialing bodies and professional organisations having frameworks or models for their particular industry and/or profession. [30-32] Management competencies are categorised as ‘soft’ management skills or ‘hard’ management skills. [29,33]  Hard management skills are tangible; therefore, more easily evaluated. Hard business skills include: accounting, computer literacy, and technical knowledge to operate equipment. In contrast, soft skills are abstract and generally harder to evaluate; however, they are important skills for leaders and managers. [29,33]  Soft management skills include: self-awareness, communication, emotional intelligence, self-regulation, and social skills.[29,33]  Recently, there has been an increased focus on developing soft skills in managers. This focus on improving soft management skills for health managers is congruent with the competencies recommended by professional and credentialing bodies. [1,8,30] Common sense suggests that using health service management competency frameworks  for management development programs will contribute to improve quality health services.

Participatory CQI approaches

The review provided strong evidence about the benefit of participatory CQI approaches to learning and evaluation. Further, programs that contained on-the-job application of skills were better placed to demonstrate cost-benefits and return on investment (ROI).  Hence, evaluating programs using pre-determined competencies, to the level of cost-benefit provides a robust and economically sensible method of evaluation. To achieve this end, management development programs must be developed, implemented and evaluated in a manner that collects the data required for this level of analysis. The findings from this synthesis suggest that one way to do this is through participatory CQI approaches measuring outcomes against predetermined competencies, with an on-the-job component to assess the application of skills from multiple perspectives, over time, in a real-world context.

Future directions

The review revealed a dearth of information about management development programs specifically developed for geographically remote regions suggesting that an opportunity exists to explore the role that context plays for managers in this microsystem. Further, an examination of these microsystems could include investigation into whether specific competencies are required for managers working in remote health services; and whether management competencies differ by context (e.g. country, region, and/or ethnically-based culture).  The findings would have international relevance and could inform the development of a framework that specifies competencies required for managers in geographically remote regions.

Limitations

The inclusion criteria limited selection to articles published in English possibly excluding some relevant studies.  Also, the quality check for the review required articles to be peer reviewed, restricting the use of grey literature. This excluded articles from industry magazines and journals as they lacked the information needed to determine the validity and reliability of the information reported.

Conclusion

Evaluating management development programs beyond the level of participant satisfaction is costly and time consuming. However, to know the effectiveness of a program it is imperative to conduct evaluations that capture how well the program achieved the desired outcomes as well as the cost-benefit of the program.  This scoping review set out to examine the ways in which CQI approaches are used in evaluating the effectiveness of management development programs. The findings suggests that participatory CQI approaches to management development program implementation and evaluation can contribute to improvements in the quality of healthcare.[17]  Through a synthesis of the findings, the authors conclude that evaluations using participatory CQI approaches are better positioned to report more comprehensively on the benefits of management development programs when they include the competencies required to be successful in the context within which the manager is working.

 

Acknowledgements
This study was undertaken under the auspices of the Centre of Research Excellence: An Innovation Platform for Integrated Quality Improvement in Indigenous Primary Health Care (CRE-IQI, funded by the NHMRC ID 1078927). The views expressed in this publication are those of the authors and do not necessarily reflect the views of the funding agencies.  The authors also acknowledge in-kind support from James Cook University, the Cairns Institute and CRANAplus.

References

1.      Stefl ME. Common competencies for all healthcare managers: the Healthcare Leadership Alliance model. Journal of Healthcare Management 2008; 53: 360-374.

2.      World Health Organisation (WHO). Increasing access to health workers in remote and rural areas through improved retention. WHO; 2010. Available: <http://www.who.int/hrh/retention/guidelines/en/>  (Accessed 27/03/18)

3.      Malone G, Cliffe C. A Framework for Remote and Isolated Professional Practice. CRANAplus; August 2014. Available: <https://crana.org.au/resources/practice/remote-practice/> (Accessed 27/03/18)

4.      National Rural Health Alliance. Supporting Health Service Managers in Rural and Remote Australia. September 2004. Available: < http://ruralhealth.org.au/sites/default/files/position-papers/position-paper-04-09-21.pdf> (Accessed 27/03/18)

5.      Hegney D, McCarthy A, Rogers-Clark C, Gorman D. Retaining rural and remote area nurses. The Queensland, Australia experience. The Journal of Nursing Administration 2002; 32: 128-35.

6.      Onnis L, Pryce J. Health professionals working in remote Australia: a review of the literature. Asia Pacific Journal of Human Resources 2016; 54: 32-56.

7.      Isouard G, Martins JM. Health service managers in Australia: Progression and evolution. Asia Pacific Journal of Health Management 2014; 9: 35-52.

8.      Martins J, Isouard G. An evidence-based framework: Competencies and skills for managers in Australian health services. Asia Pacific Journal of Health Management 2015; 10: 8-23.

9.      Innovation & Business Skills Australia (IBSA). Karpin Report Revisited: Leadership and Management Challenges in Australia. South Australia: IBSA; 2011. Available: <https://store.ibsa.org.au/sites/default/files/media/Karpin%20Revisited,%20Leadership%20and%20Management%20Challenges%20in%20Australia.pdf> (Accessed 27/03/18)

10.    Becker K, Bish A. Management development experiences and expectations: informal vs formal learning. Education & Training 2017; 59: 565-78.

11.    Adams D, Waddle C. Evaluating the return from management development programmes: individual returns versus organizational benefits. International Journal of Contemporary Hospitality Management 2002; 14: 14-20.

12.    Schierhout G, Hains J, Si D, Kennedy C, Cox R, Kwedza R, O'Donoghue L, Fittock M, Brands J, Lonergan K, Dowden M, Bailie R. Evaluating the effectiveness of a multifaceted, multilevel continuous quality improvement program in primary health care: developing a realist theory of change. Implementation science 2013; 8: 119.

13.    Bailie R, Bailie J, Broughton E, Larkins S. Editorial: Continuous Quality Improvement (CQI)-Advancing Understanding of Design, Application, Impact, and Evaluation of CQI Approaches. Frontiers in Public Health 2017; 5.

14.    Dunne S, Lunn C, Kirwan M, Matthews A, Condell S. Planning and Selecting Evaluation Designs for Leadership Training: A Toolkit for Nurse Managers and Educators. Journal of Professional Nursing 2015; 31: 475-81.

15.    Foy R, Eccles MP, Jamtvedt G, Young J, Grimshaw JM, Baker R. What do we know about how to do audit  and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Services Research 2005; 5: 50.

16.    Gardner K, Bailie R, Si D, O’Donoghue L, Kennedy C, Liddle H, Cox R, Kwedza R, Fittock M, Hains J, Dowden M, Connors C, Burke H, Beaver C. Reorienting primary health care for addressing chronic conditions in remote Australia and the South Pacific: Review of evidence and lessons from an innovative quality improvement process. Australian Journal of Rural Health 2011; 19: 111-7.

17.    Kaplan HC, Brady PW, Dritz MC, Dritz MC, Hooper DK, Linam WM, Froehle CM, Margolis P. The Influence of Context on Quality Improvement Success in Health Care: A Systematic Review of the Literature. The Milbank Quarterly 2010; 88: 500-59.

18.    Larkins S, Woods CE, Matthews V, Thompson SC, Schierhout G, Mitropoulos M, Patrao T, Panzera A, Bailie, RS. Responses of Aboriginal and Torres Strait Islander primary health-care services to continuous quality improvement initiatives 2016; 3: 288.

19.    Weeks LC, Strudsholm T. A scoping review of research on complementary and alternative medicine (CAM) and the mass media: looking back, moving forward. BMC Complementary and Alternative Medicine 2008; 8: 43.

20.    Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal 2009; 26: 91-108.

21.    Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009; 339: 332-336.

22.    Yaping D, Stanton P. Evaluation of the health services management training course of Jiangsu, China. Australian Health Review 2002; 25: 161-70.

23.    Omar M, Gerein N, Tarin E, Butcher C, Pearson S, Heidari G. Training evaluation: a case study of training Iranian health managers. Human Resources for Health 2009; 7: 20.

24.    Wallis A, Kennedy KI. Leadership training to improve nurse retention. Journal of Nursing Management 2013; 21: 624-32.

25.    Fealy GM, McNamara MS, Casey M, O'Connor T, Patton D, Doyle L, Quinlan C. Service impact of a national clinical leadership development programme: findings from a qualitative study. Journal of Nursing Management 2015; 23: 324-32.

26.    Steensma H, Groeneveld K. Evaluating a training using the "four levels model". Journal of Workplace Learning 2010; 22: 319-31.

27.    Holmberg R, Larsson M, Bäckström M. Developing leadership skills and resilience in turbulent times. Journal of Management Development 2016; 35: 154-69.

28.    Throgmorton C, Mitchell T, Morley T, Snyder M. Evaluating a physician leadership development program – a mixed methods approach. Journal of Health Organization and Management 2016; 30: 390-407.

29.    Charoensap-Kelly P, Broussard L, Lindsly M, Troy M. Evaluation of a Soft Skills Training Program. Business and Professional Communication Quarterly 2016; 79: 154-79.

30.    Australasian College of Health Service Management (ACHSM). ACHSM Master Health Service Management Competency Framework. ACHSM; 2016. Available: <https://achsm.org.au/Documents/Education/Competency%20framework/2016_competency_framework_A4_full_brochure.pdf> (Accessed 28/03/18)

31.    West M, Smithgall L, Rosler G, Winn E. Evaluation of a nurse leadership development programme. Nursing Management 2016; 22: 26.

32.    Saleh SS, Williams D, Balougan M. Evaluating the Effectiveness of Public Health Leadership Training: The NEPHLI Experience. American Journal of Public Health 2004; 94: 1245-9.

33.    Ibrahim R, Boerhannoeddin A, Bakare KK. The effect of soft skills and training methodology on employee performance. European Journal of Training and Development 2017; 41: 388-406.