Professionalism of Information Management in Health Care: Development and Validation of the Construct and Its Measurement

Abstract

Background Against the background of a steadily increasing degree of digitalization in health care, a professional information management (IM) is required to successfully plan, implement, and evaluate information technology (IT). At its core, IM has to ensure a high quality of health data and health information systems to support patient care.

Objectives The goal of the present study was to define what constitutes professional IM as a construct as well as to propose a reliable and valid measurement instrument.

Methods To develop and validate the construct of professionalism of information management (PIM) and its measurement, a stepwise approach followed an established procedure from information systems and behavioral research. The procedure included an analysis of the pertaining literature and expert rounds on the construct and the instrument, two consecutive and comprehensive surveys at the national and international level, exploratory and confirmatory factor analyses as well as reliability and validity testing.

Results Professionalism of information management was developed as a construct consisting of the three dimensions of strategic, tactical, and operational IM as well as of the regularity and cyclical phases of IM procedures as the two elements of professionalism. The PIM instrument operationalized the construct providing items that incorporated IM procedures along the three dimensions and cyclical phases. These procedures had to be evaluated against their degree of regularity in the instrument. The instrument proved to be reliable and valid in two consecutive measurement phases and across three countries.

Conclusion It can be concluded that professionalism of information management is a meaningful construct that can be operationalized in a scientifically rigorous manner. Both science and practice can benefit from these developments in terms of improved self-assessment, benchmarking capabilities, and eventually, obtaining a better understanding of health IT maturity.

Keywords

information management - professionalism - health information technology - construct - validation

Introduction

In light of the ongoing progress toward the digitalization of health care delivery, researchers and information technology (IT) professionals have repeatedly stressed the increasing need for effective and efficient information management (IM) to provide safe and high quality care.[1] [2] [3] Efforts to digitize workflows are subject to a variety of barriers at the stage of IT initiation, implementation as well as institutionalization.[4] Risks can emerge from the misalignment between the IM strategy and the overall strategy of an organization as well as from the misalignment of workflows, i.e., poor workflows which are digitized without being optimized beforehand.[4] [5] Against this background, it becomes clear that a professional IM is of central importance for a successful transition from the paper to the digital world and for its advancement toward better patient care.

The literature on IM reflects the interest in this topic and suggests a variety of approaches. It is argued that a professional IM is required on various levels and its activities must be performed in a regular and repeatable fashion to leverage successful implementations and overcome barriers.[6] [7] [8] At this, IM should manage people, structures, processes, and strategies in a goal-oriented manner to ensure the high quality of the health information systems concerned as well as the provision of information and thereby the support of patient care.[1] [7] [9] [10] In addition to long-term planning and execution, IM is also concerned with the daily business.[1] [9] [11] [12]

Information management should be distinguished from IT management. They are two distinct areas but are frequently used synonymously. In addition, depending on the point of view, they can also be used in a hierarchical manner where one of the two fields is considered to be at a higher level.[13] [14] [15]

In the present paper, we speak of IM as the area where information, IT, and other pertinent resources must be best aligned with an organization's strategy. In detail, IM is planning, monitoring, and directing of information systems, information, and communication technologies as an overarching management task, with the aforementioned goal of ensuring the best possible use of information resources with regard to the organization's goals.[1] [13]

However, the question is what exactly does a professional IM look like? To be able to research, assess, and improve the professionalism of IM, it is necessary to better understand the concept behind it. Comprehending the essence professionalism of information management will lead to a reliable and valid assessment instrument that makes the current state measurable, visible, and comparable on a concise and aggregated level. Preferably, such instrument is a scale that results in scores reflecting the degree of professionalism of information management, i.e., PIM scores. PIM scores could be used for research and as national as well as international benchmarks. In particular, health IT maturity research could benefit from them as performance indicators to be associated with the successes and failures of health IT implementation and use. Finally, such scores can become a suitable self-assessment tool for practitioners as well.

Objectives

Against this background, the main objective of the present study is to examine and determine the construct of professionalism of information management in health care. Therefore, this study aims to specify and operationalize the construct of professionalism of information management. This construct should offer the possibility to provide a system of scores for reflecting the degree of professionalism that breaks down the construct into one key indicator as well as into scores of related professionalism of information management dimensions.

The score system will be developed focusing on hospitals representing health care delivery organizations that are usually large and complex enough to have a strong need for a fully developed IM. The model of MacKenzie et al[16] will hereby serve as a methodological backbone for a construct definition, measurement and validation to reach the objectives of this study. It is a particularly rigorous and well elaborated framework which had proved useful to develop the workflow composite score (WCS) that measures the degree of workflow support through IT in selected clinical workflows and, thus, the technical maturity.[17] [18] [19] The following research questions guided the construct identification, measurement, and validation process.

1. How can the professionalism of information management construct be specified?

2. How can the professionalism of information management construct be measured in a suitable inventory?

3. How reliable and valid is the professionalism of information management measurement?

Methods

Overview

Pursuant to the model of MacKenzie et al,[16] this study followed eight consecutive steps. [Table 1] shows the individual steps in detail, whereby the methodological questions and methods per individual step are listed separately in the second and third columns. The fourth column indicates the year of implementation. The first phase of the study could be divided into the steps one to three. The first data collection and construct revision took place in phase two, which embraced steps four to six. Phase three consisted of the steps seven and eight, which were dedicated to conducting the second survey and finally to test for validity and reliability.

Table 1 Steps undertaken to develop the construct measurement instrument[16]

Steps	Related questions	Methods	Year
1. Conceptualization and definition of the construct	• What is the construct and how does this construct differ from others? • What are the main attributes of the construct?	• Literature search and analysis • Expert group discussions based on the literature findings • Definition of a framework of dimensions based on the literature and discussions	2015
2. Development of measures—generate items of the construct	• What dimensions fully represent the conceptual construct?	• Operationalization of the construct • Definition of scales per item	2015
3. Development of measures—assessment of the content validity	• Do the dimensions capture all the relevant attributes of the construct (completeness)?	• Expert group discussion	2015
4. Scale evaluation and refinement—first quantitative survey	• Are the items understandable (comprehensibility)? • Is the use of the inventory practicable (feasibility)?	• First quantitative survey	2016
5. Model specification—specify the measurement model	• How are the dimensions associated with their respective item set as well as with one another?	• Model specification	2016
6. Scale purification and refinement	• How good is the measurement model/are the scales?	• Explorative factor analysis (EFA) • Expert group discussion	2016
7. Validation—data capture and computation—second quantitative survey	• How good is the revised model and the related parameters?	• Second quantitative survey • Confirmatory factor analysis (CFA) for the assessment of convergent and discriminatory validity	2016/17
8. Validation—examination of PIM construct reliability and validity	• How reliable is the PIM? • How valid is the PIM?	• Computation of the reliability (internal consistency) • Computation of the validity	2018/19

Abbreviations: CFA, confirmatory factor analysis; EFA, explorative factor analysis; PIM, professionalism of information management.

Source: Adapted from MacKenzie et al[16].

Conceptualization and Definition of the Construct (Step 1)

First, a comprehensive literature research was conducted to define the essence of IM on all levels, distinguishing it from other constructs (mainly IT governance) and identifying the elements of professionalism in conjunction with IM. The search was performed in the relevant databases (i.e., PubMed, ACM, AISeL). The keywords information management, IT governance, strategy, framework, ITIL, and COBIT as well as validity, reliability, and evaluation were used individually and in combination.

The literature search was complemented by a snowball search starting with the work of Winter et al[1] and the IT frameworks COBIT (control objectives for information and related technology) and ITIL (IT infrastructure library).[8] [20] [21] After the literature research, an internal expert panel of five medical informatics scientists came together to identify the key elements and key terms in the literature and hereupon constituted the professionalism of information management construct and its dimensions.

Development of Measures—Generate Items of the Construct (Step 2)

Once the construct had been defined, a framework for mapping this construct onto the respective items had to be designed. The framework consisted of the potential dimensions of IM and phases of IM. Based on this framework, descriptive and concise item sets including the corresponding categories and Likert scales for each dimension of IM were derived from the literature. For the development of scales and items, established survey instruments were used as the basis.[21] [22] This framework served as the method to operationalize the professionalism of information management construct.

Development of Measures—Assessment of the Content Validity (Step 3)

The content validation was performed through an on-site expert workshop. To this end, independent experts were asked to appraise the construct including the item set and scales and to adapt it, if necessary. The panel of experts consisted of six chief information officers (CIOs) and six medical informatics scientists. During this process, individual items were adapted.

Scale Evaluation and Refinement—First Quantitative Survey (Step 4)

After the items had been consented to, a first quantitative survey was conducted using an online questionnaire that was implemented with the online tool Unipark. A heterogeneous group of ten CIOs completed the questionnaire and gave detailed feedback in a pretest. Next, the questions were embedded in a comprehensive survey on IM in German hospitals. Following the pretest of the questionnaire, 1,284 CIOs of German hospitals were invited via e-mail. The survey was conducted from February to April 2016 and yielded 164 responses (response rate 12.7%).

Model Specification—Specify the Measurement Model (Step 5)

The dimensions of the professionalism of information management construct were operationalized with five and six items for each dimension, respectively. Once a set of items, which fulfilled the requirements of content validity, had been constructed, the next step was to define a measurement model that captured the expected relationships between the items and the dimensions (step 1). All of the items within the professionalism of information management dimensions should contribute equally to the respective dimension in the sense that no weighting scheme needs to be applied, an item can only belong to one dimension and the dimensions themselves can correlate.

Scale Purification and Refinement (Step 6)

In this step, the model was explored and refined employing statistical methods, in particular explorative factor analysis (EFA). The purpose was to assess if the data were mapped on the previously designed model specifications. This initial, unsupervised approach was chosen against the background that this instrument was newly developed. Then, the EFA was coupled with subsequent confirmatory factor analysis (CFA) (see Step 7-Validation—Data Capture and Computation—Second Quantitative Survey) on a new dataset. According to the literature, the sample sizes were sufficient for both, the EFA and CFA, especially considering the relatively simple factor model and the mid to high communalities and loadings in both solutions.[23] [24] [25] [26] [27] The following equation expresses the principle elements of the factor analysis, which is explained below, as these elements will be interpreted in the results.

y_pi  = λ_pq ƒ _qi  + ε_pi (1)

y_pi is the individual value i of the pth observed variable, ƒ_qi is the individual value i of the qth latent common factor, ε_pi is the individual value i of the pth latent unique factor (error variance), and λ_pq is the factor loading indicating the relationship between the pth observed variable and the qth latent common factor.[28] [29] Since our data were categorical and ordinal, we applied the underlying variable approach using polychoric correlation coefficients in the correlation matrices and estimated the model coefficients using the unweighted least squares procedure as is widely recommended in the methodological literature in these cases.[30] [31]

The model's sampling adequacy was evaluated based on the Kaiser-Meyer-Olkin criterion with an acceptance range of ≥0.7 and following Bartlett's test of sphericity.[32] [33] All of the analyses were conducted in R using the package psych. If the criteria were not met, the variables with low variance caused by all the factors or variables with high factor loadings on more than one factor were removed. The factors included in the EFA can be found in ([Table 2)].

Table 2 Component loading matrix for the explorative factor analysis (EFA) (n = 164)

	Item	Factor
		1	2	3
Strategic IM	Preparation and development of a project portfolio.	0.80
	Strategic monitoring in the form of targeted evaluations and collection of key figures.	0.72
	Long-term finance and investment planning.	0.71
	Preparation and further development of an information management strategy.	0.65
	Strategic control in the form of prioritization and initiation of projects.	0.60
Tactical IM	System analysis and evaluation (e.g., process modelling, evaluation of the current state).		0.84
	System specification (e.g., requirements definition, specifications, migration plan).		0.78
	System selection (e.g., market analysis, tendering, bid comparison).		0.71
	System implementation (e.g., implementation strategy and adaptation).		0.61
Operational IM	Management and monitoring of the technical performance (e.g., infrastructure, networks).			0.93
	Application support and maintenance.			0.89
	Running of the help desk and/or service desk.			0.83
	Training of clinical end users.			0.71

Abbreviation: IM, information management.

Note: Values below 0.3 are left blank.

To improve the wording, all the items were scrutinized by a panel of experts (five CIOs and seven medical informatics scientists). Pursuant to their comments, the items were supplemented, combined to one item or split into two. Finally, the item set consisted of 15 statements on the professionalism of information management dimensions.

Validation—Data Capture and Computation—Second Quantitative Survey (Step 7)

To further extend the validation beyond a mere German perspective, Swiss and Austrian CIOs were also contacted in addition to the German CIOs in the second round of data collection. A group of five CIOs completed the questionnaire and gave detailed feedback in a pretest (four from Germany, one from Austria). The final questionnaire was sent to 1,349 German CIOs, 135 Swiss CIOs and 185 Austrian CIOs via an online questionnaire using the online tool LimeSurvey (See [Appendix 1]). The surveys took place from December 2016 to mid-2017 as part of the IT Report Health Care.[34]

Appendix 1 Extract of the questionnaire from the second quantitative survey

Question	Response options
Are the following information management activities or procedures performed in your hospital?  I. Strategic tasks and processes
• Preparation and further development of an information management strategy.	Regularly/Irregularly/Not at all
• Strategic controlling in terms of IT project management including project portfolios.	Regularly/Irregularly/Not at all
• Long-term finance and investment planning.	Regularly/Irregularly/Not at all
• Strategic risk management (e.g., maintenance of emergency plans).	Regularly/Irregularly/Not at all
• Evaluation of IT efficiency benefits.	Regularly/Irregularly/Not at all
Are the following information management activities or procedures performed in your hospital?  II. Procurement and implementation
• System analysis and evaluation (e.g., process modelling, evaluation of the current state).	Regularly/Irregularly/Not at all
• System specification (e.g., requirements definition, specifications, migration plan).	Regularly/Irregularly/Not at all
• System selection (e.g., market analysis, tendering, bid comparison).	Regularly/Irregularly/Not at all
• System implementation (e.g., implementation strategy and adaptation).	Regularly/Irregularly/Not at all
• Further cooperation with manufacturers (for product development/enhancement).	Regularly/Irregularly/Not at all
Are the following information management activities or procedures performed in your hospital?  III. Operational tasks and processes
• Application support and maintenance.	Regularly/Irregularly/Not at all
• Management and monitoring of the technical performance (infrastructure and networks).	Regularly/Irregularly/Not at all
• Training of clinical end users.	Regularly/Irregularly/Not at all
• Evaluation of user satisfaction.	Regularly/Irregularly/Not at all
• Running of the help desk and/or service desk.	Regularly/Irregularly/Not at all
Do defined IT management processes exist in your institution in terms of IT governance (e.g., based on COBIT or ITIL)?	Yes/No
Does your institution have a strategic IT plan and to what extent is it integrated into the strategic hospital plan?	We have no strategic IT plan/We are developing a strategic IT plan/There is an IT plan, but it is not aligned with the hospital strategy/The IT plan is aligned with or an integral part of the hospital strategy.

Abbreviations: COBIT, control objectives for information and related technology; IT, information technology; ITIL, IT infrastructure library.

A total of 223 responses out of the original dataset (224 German, 16 Swiss and 16 Austrian participants) were complete (i.e., had no missing values) and could thus be used for the analysis and calculation of a CFA. A factor model was specified according to the results of the initial analysis (see step 6) and the CFA was conducted to assess the scale convergent and discriminatory validity based on this second dataset in R using the lavaan package. In accordance with the previous EFA procedure, parameter estimates were calculated based on polychoric correlation coefficients and a diagonally weighted least squares procedure with robust corrections to standard errors and test statistics. The model was also specified to allow for interfactor correlations. Model fit was evaluated drawing on the root mean square error of approximation, comparative fit index, Tucker-Lewis index and standardized root mean square residual.

Validation—Examination of Professionalism of Information Management Reliability and Validity (Step 8)

In addition to the CFA, Cronbach's α was calculated as a means to further test the scale in terms of its internal consistency. An α value greater than 0.7 was considered acceptable.[10] [35]

To calculate a professionalism of information management score from the survey results, the different items were transformed to a scale ranging from 0 to 100 to allow for aggregation. A mean value was calculated across the respective dimensions and across all dimensions. In addition, a mean value was calculated for the individual items to show the degrees of implementation. For testing the criterion validity of the measurement instrument, correlations between the professionalism of information management scores and two related criteria, which had been added as extra questions, were performed. These two criteria for the validation were (1) usage of an IT framework and (2) availability and integration of a strategic IT plan. In detail, the items were:

• Do defined IT management processes exist in your institution in terms of IT governance (e.g., based on COBIT or ITIL)?

• Does your institution have a strategic IT plan and to what extent is it integrated into the strategic hospital plan?

The criterion of IT frameworks can be regarded as an upstream criterion that could promote the professionalism of information management[5] [8] [36] while the integrated IM strategy corresponds to a downstream criterion that is developed as part of a professionalism of information management.

Results

Specification of the Construct: Professionalism of Information Management

The first question: “How can the professionalism of information management construct be specified?” was answered in steps 1 to 3. The professionalism of information management construct could not be explicitly found in the international IM literature. It rather stood in a network of more or less similar theoretical constructs ([Table 3]). There were strong references to the construct about the main terms (IT) governance, IM procedures, IT frameworks as well as phases (performing cycle) of IM. It was also possible to find influencing environmental determinants of professionalism of information management ([Table 3]). Following the literature search and subsequent expert discussions (steps 1 to 3), professionalism of information management was regarded as the regularity with which the sum of all IM activities (procedures) were performed and which ensured that IT properly contributed to the fulfilment of the hospital goal of patient care.[9]

Table 3 Constituent and environmental terms of the professionalism of information management construct

Constituting terms	Environmental terms
(IT) governance[1] [8] [20] [36] [37] [38]	Structures[1] [7] [21] [39] [40]
Procedures of IM[1] [9] [21]	Sociotechnical aspects[41] [42]
Levels of IM[1] [7] [9]
IT frameworks[8] [20] [21] [43] [44]
Phases of IM (performing cycle)[1] [21] [43]

Abbreviations: IM, information management; IT, information technology; PIM, professionalism of information management.

According to the literature ([Table 3]), governance embraced all activities to ensure the general conditions were established that determined the achievement of the corporate goals. It served as a framework for decision-making and executing tasks at various levels,[1] [36] with the intention to support IM activities leading to better IT performance and thus information as well as information system quality.[8] [21] [37] [38] [44] In other words, IT governance was a condition which IT needed to prosper and to be managed well.[8] [38] In contrast, IM planned, directed, and monitored these activities that were specified within the framework of governance.[8] [21]

Information management itself, which was performed within the scope of IT governance, was composed of actions or IM procedures which were performed on different levels. Therefore, IM embraced practices and activities to achieve the goals of the organization.[21]

Based on Winter et al.,[1] there were three different levels of IM on which the procedures could be executed following the need to cover the range from short-term to long-term actions, i.e., operational tasks to long-term planning and evaluation. This would also result in different levels of consequences because different views on IM requiring different methods and tools would be taken.[1] Based on this assumption, a distinction could be made between the levels of strategic, tactical, and operational IM. The strategic IM dealt with the long-term perspective on the entire information processing in an organization. The tactical IM was responsible for the functions and applications and was initiated as the next step following strategic IM. Thus, a strategic IM was compellingly necessary for a tactical one. The operational IM, finally, was in charge of operating the components of the information system and processes.[1] [7] [9] Exemplary processes at the strategic level were long-term finance and investment planning as well as preparation and further development of an IM strategy.

Pursuant to the literature analysis and the discussions in the expert panel, the levels of a strategic, tactical, and operational IM were regarded as good candidates for the dimensions of IM.

The IT frameworks COBIT for IT governance and ITIL for IT service management were found to provide information about the degree of professionalism of information management. Offering tools to support IM in adopting and implementing IT and IT innovations,[20] [43] [44] they addressed, amongst others, methods to enforce routinization. Hereby, routinization was understood as the way in which IM procedures were performed, i.e., periodic, unplanned (ad hoc,) or not at all.[8] [21] This meant, the more regularly a given process was performed, the more repeatable and thus more stable, safe, effective, and efficient it became.[8]

The literature analysis also showed that standardization and regularity were augmented by the notion of a cyclic way of performing these procedures, i.e., starting from planning, then implementing, and finally evaluating before planning again. This meant that any IM action was split into activities before, during, or after an intervention. Thus, the activities of any kind were part of a repetitive and systematic cycle[1] [21] [43] that also expressed the fact of whether an IM action was performed professionally.

Consequently, the professionalism of information management should be defined by the elements of (1) regularity of IM procedures and (2) the distinction of procedures belonging to different cyclic phases. This held true for IM procedures on each of the three levels of strategic, tactical, and operational IM.

These activities had to be distinguished from the structures and sociotechnical aspects that possibly exerted an influence on IM. Structures were necessary prerequisites or environmental aspects from which procedures were derived. Typical examples of structures were “the existence of an IT department” or “the positioning of a CIO and his responsibilities within the hierarchy of the organization.”[1] [7] Moreover, the implementation of procedures influenced people (or were influenced by people), which constituted the sociotechnical environment.[41] [42] These structures and sociotechnical aspects, although they are associated, were found to be distinct from the professionalism of information management construct. [Fig. 1] shows how the professionalism of information management construct emerged from the literature analysis and the expert panel discussions.

Fig. 1 Conceptualization and definition of the construct of the professionalism of information management (PIM) to a literature research.

Following the definition of the construct, a set of items was developed, whereby the items reflected activities on the strategic (five items), tactical (six items) or operational (six items) level, i.e. the IM dimensions, and could be assessed according to the elements of professionalism in terms of standardization and regularity, i.e. whether the activity took place regularly, irregularly or not at all. Furthermore, IM activities were included that reflected professionalism from the perspective of planning, implementing, and evaluating the cycle ([Table 4]).

Table 4 Items resonating the professionalism of information management (PIM)[1]

	Items	Primary phase
Strategic IM	Preparation and development of a project portfolio.	Planning
	Strategic monitoring via targeted evaluations and collection of key figures.	Evaluation
	Long-term finance and investment planning.	Planning
	Preparation and further development of an information management strategy.	Planning
	Strategic control in the form of the prioritization and initiation of projects.	Planning
Tactical IM	System analysis and evaluation (e.g., process modelling, evaluation of the current state).	Evaluation
	System specification (e.g., requirements definition, specifications, migration plan).	Implementation
	System selection (e.g., market analysis, tendering, bid comparison).	Implementation
	System implementation (e.g., implementation strategy and adaptation).	Implementation
	System evaluation (information gathering, preparation, and presentation).	Evaluation
	Project management (project planning, support, and completion).	Planning
Operational IM	Management and monitoring of the technical performance (e.g., infrastructure, networks).[a]	Implementation
	Application support and maintenance.[a]	Implementation
	Running of the help desk and/or service desk.[a]	Implementation
	Training of clinical end users.	Implementation
	IT-related accounting.	Implementation
	Contract management.[a]	Implementation

Abbreviations: IM, information management; IT, information technology.

^a There are overlaps with IT service management.

Measuring the Professionalism of Information Management Construct

Following steps 1 to 3 for the construct definition and development of measures, steps 4 to 6 were conducted, to further answer the second question: “How can the professionalism of information management construct be measured in a suitable inventory?” To this end, a first round of utilizing the items for data capturing and analyzing them by an exploratory factor analysis (EFA) was performed. The results of the EFA ([Table 2]) confirmed the dimensions of professionalism of information management as separate factors, i.e., IM on the strategic, tactical, and operational level. It also led to a slight adjustment of the item set due to high factor loadings on more than one factor or due to variables with low variance extracted caused by all factors. The measure of sampling adequacy for the EFA across all criteria was 0.86 and the total variance explained by all factors was 66.54%.

Reliability and Validity of Professionalism of Information Management Measurement

The third question “How reliable and valid is the professionalism of information management measurement?” was answered in steps 7 to 8. The second comprehensive survey based on a revised questionnaire and on an independent international dataset of answers from 223 CIOs confirmed the construct of professionalism of information management. [Fig. 2] shows the results of the CFA and confirmed the tripartite division of IM according to the strategic, tactical, and operational dimensions.

Fig. 2 Confirmatory factor analysis (CFA) path diagram including factor correlations, factor loadings, and error variances (n = 223).

The item “evaluation of user satisfaction” (introduced after the expert panel at step 6) was initially defined as an operational item, but it was loaded on the strategic IM and finally assigned to it. Despite the moderate to high correlations between the latent variables, psychometric properties proved to be good overall and indicated the satisfactory fit of our model to the data ([Table 5]).

Table 5 Goodness-of-fit statistics of the confirmatory factor analysis (CFA) (robust DWLS estimation)

df	χ 2	CFI	TLI	RMSEA	SRMR
87	173.202	0.967	0.960	0.067	0.075

Abbreviations: CFI, comparative fit index; df, degrees of freedom; DWLS, diagonally weighted least squares; RMSEA, root mean square error of approximation; SRMR, standardized root mean square residual; TLI, Tucker-Lewis index; χ ², chi square.

The professionalism of information management score system developed from the item set is shown in [Table 6] together with the corresponding reliability measure Cronbach's α. The professionalism of information management overall score reflects the mean of how professionally IM procedures were performed across all three IM dimensions. The scores are shown for the model version 1 (step 4) and version 2 (step 7). The professionalism of information management overall score of version 2 could be broken down into three countries, i.e., Germany, Austria, and Switzerland, as shown in [Table 6]. Furthermore, the professionalism of information management overall score was split into three IM dimensions, called dimension subscores, and the IM phases, called phase subscores.

Table 6 Cronbach's α, PIM score system, and descriptive statistics

		Items		α		Mean[b]		SD[b]
Model version[a]		1	2	1	2	1	2	1	2
PIM overall		13	15	0.88	0.88	43.2	64.8	18.7	19.2
Germany[c]		–	–	–	–	–	64.2	–	19.3
Austria		–	–	–	–	–	71.5	–	15.6
Switzerland		–	–	–	–	–	67.9	–	20.2
Dimension	Strategic IM	5	6	0.82	0.81	30.2	54.2	26.1	24.6
	Tactical IM	4	5	0.81	0.81	44.9	60.7	17.4	23.4
	Operational IM	4	4	0.91	0.74	57.8	85.7	24.7	20.1
Phase	Planning	4	4	0.79	0.81	33.2	64.8	27.1	28.4
	Implementation	7	8	0.85	0.81	52.1	74.2	18.5	18.9
	Evaluation	2	3	0.28	0.56	32.0	39.5	21.2	22.9

Abbreviations: α, Cronbach's Alpha; IM, information management; PIM, professionalism of information management; SD, standard deviation.

^a Model version 1 (n = 164)—first survey; Model version 2 PIM overall and operational IM (n = 223), strategic IM and tactical IM (n = 224), implementation (n = 223), planning and evaluation (n = 224) —second survey.

^b Value range 0 to 100.

^c Germany (n = 199), Austria (n = 11), Switzerland (n = 13).

Overall, the scales indicated good psychometric properties across both survey iterations and stayed stable after minor adjustments had been made to the scale sets, i.e., number and wording of the items. The α value of the “operational IM” scale decreased from survey one to two but remained at an acceptable level of above 0.70. The α value of “evaluation” was very low at 0.28 in the first round. By splitting an item, the reliability could be increased to 0.56.

There was a rise of the mean values in all professionalism of information management dimensions as well as in the overall score from survey one which took place in 2016 to survey two which took place primarily in 2017. Operational IM obtained the highest score values, while strategic IM was the lowest. This finding was consistent across the two surveys. The scores for the phases planning, implementation, and evaluation were the least high for evaluation, followed by planning. The highest score was achieved for implementation (individual medians for each professionalism of information management item see [Appendix 2]).

Appendix 2 Individual medians of professionalism of the information management items (n = 224)

Item	Dimension	Primary phase	Median	IQR
Application support and maintenance	Operational	Implementation	Regularly	Regularly
Management and monitoring of the technical performance (n = 223)	Operational	Implementation	Regularly	Regularly
Running of the help desk and/or service desk	Operational	Implementation	Regularly	Regularly
Training of clinical end users	Operational	Implementation	Regularly	Regularly to irregularly
System implementation	Tactical	Implementation	Regularly	Regularly to irregularly
Long-term finance and investment planning	Strategic	Planning	Regularly	Regularly to irregularly
System selection	Tactical	Implementation	Irregularly	Regularly to irregularly
Strategic risk management	Strategic	Planning	Irregularly	Regularly to irregularly
Strategic controlling in terms of IT project management including project portfolios	Strategic	Planning	Irregularly	Regularly to irregularly
System specification	Tactical	Implementation	Irregularly	Regularly to irregularly
Preparation and further development of an information management strategy	Strategic	Planning	Irregularly	Regularly to irregularly
Further cooperation with manufacturers	Tactical	Implementation	Irregularly	Irregularly
System analysis and evaluation	Tactical	Evaluation	Irregularly	Irregularly
Evaluation of user satisfaction	Strategic	Evaluation	Irregularly	Irregularly to not at all
Evaluation of IT efficiency benefits	Strategic	Evaluation	Not at all	Irregularly to not at all

Abbreviations: IQR, interquartile range; IT, information technology.

Validity was tested by correlating the overall professionalism of information management score as well as the subscores with two related criteria. The results of the correlation with the criterion “Do defined IT management processes exist in your institution in terms of IT governance (e.g. based on COBIT or ITIL)?” as well as with “Does your institution have a strategic IT plan and to what extent is it integrated into the strategic hospital plan?” are shown in [Table 7].

Table 7 Correlation results to the measurement of validity

	A strategic IT plan exists and is integrated into a strategic hospital plan[a]	n	Defined IT management processes in terms of IT governance (COBIT, ITIL) exist[b]	n
PIM overall	0.57[c]	223	0.36[c]	223
Strategic IM	0.56[c]	224	0.33[c]	224
Tactical IM	0.47[c]	224	0.30[c]	224
Operational IM	0.39[c]	223	0.24[c]	223
Planning	0.57[c]	224	0.32[c]	224
Implementation	0.50[c]	223	0.31[c]	223
Evaluation	0.42[c]	224	0.30[c]	224

Abbreviations: COBIT, control objectives for information and related technology; IM, information management; IT, information technology; ITIL, IT infrastructure library; PIM, professionalism of information management.

^a Correlation according to Spearman-Rho.

^b Correlation according to point-biserial.

^c The correlation is significant at the 0.01 level.

All correlations between these criteria and the professionalism of information management scores yielded significant results at a significance level of 0.01.

The criterion “strategic plan” correlated to a considerably high degree with the overall professionalism of information management score and was decreasingly associated along the three levels strategic, tactical, and operational IM, thus reflecting the distinction between the three levels of the dimension very well. The correlations with the criterion “use of IT frameworks” resembled the ones with the criterion “strategic plan,” but at a lower level of the correlation coefficients.

Discussion

Summary

This study shows the results of the development and validation process for a construct to express the degree of professionalism of information management in health care and its measurement instrument. The construct of professionalism of information management designed from an initial literature study is composed of the dimensions strategic, tactical, and operational IM and the elements of professionalism, i.e., the standardization and regularity of procedures as well as the distinction between the planning, implementation, and evaluation phase. All IM procedures in the three dimensions and the three phases are described regarding how standardized they are and how regularly they take place.

Validity of the Construct

The professionalism of information management construct reflects the degree with which an organization forces and realizes the implementation of an integrated strategic IT plan. This is demonstrated by the significant and considerably high positive correlation of the professionalism of information management overall score with the strategy criterion. This finding is supported by the literature which highlights the importance of strategic IM for tactical and operational IM.[1] [7] Similarly, the implementation of IT frameworks is related to the professionalism of information management construct as is indicated by the significant and positive correlation with professionalism of information management overall and the decreasing positive correlations from strategic to operational IM. However, this finding is less pronounced as in the case of the implementation of the strategic plan, which is mirrored here by all the scores. This result is backed by the literature that confirms that these instruments lead to formalized procedures and process empowerment.[8] [21]

Strength of the Construct—Development of a Construct Stable Over Time

Two consecutive comprehensive surveys and the resulting factor analyses (EFA and CFA) confirmed the structure of professionalism of information management regarding its dimensions. Reliability and validity showed good results for the construct in terms of its IM dimensions and phases. Thus, a model for the professionalism of information management construct that is stable over time could be developed. It embraces a conclusive explanatory model for the operationalization of professionalism of information management from which a measurement instrument was compiled. The professionalism of information management construct thus constitutes the basis of a reliable and valid instrument for assessing the degree of professionalism of information management that can be utilized in many practical ways, e.g., for benchmarking health care institutions—also across countries as well as for individual assessments of a single institution. The development of the professionalism of information management construct stands out in relation to other maturity models in health care, such as EMRAM (Electronic Medical Records Adoption Model),[47] due to its IM focus and in particular due to the transparency of what is measured and how it is measured (reliability and validity). Compared with other measurements of IM maturity such as COBIT or ITIL, professionalism of information management can be easily used for population-wide assessments. From a scientific point of view, it may also serve for gaining better insight into information system success and other phenomena that strongly depend on a professional IM.

Adjustments of the Measurement Instrument

The professionalism of information management measurement instrument differed between the two surveys as some items had to be changed (as described in “Scale Purification and Refinement (Step 6)”) in accordance with the experts' votes, such as the addition of an item on strategic risk management. Therefore, the instruments are not entirely identical. There were considerable and consistent differences in the professionalism of information management score magnitude between the first and second survey with higher values in the latter measurement. The aforementioned adjustment of the items could have led to this finding. These results can also reflect an actual increase from the first to the second measurement point. This cannot be totally ruled out as IM has gained more attention recently.[48]

Empirical Survey Results: PIM Score

The focus of this work was put on the development of the professionalism of information management construct and its related measurement tool resulting in a score system. However, the empirical findings of the two successive surveys in addition reveal facts about the status of IM and its professionalism. In particular, it can be seen that the professionalism of operational IM is the one that is most pronounced compared with tactical and strategic IM. This indicates severe deficits in these areas. Maybe these activities are not given high enough priority by the top management team of the organization. It could also be due to a lack of staff in the IT department and a high workload so that only the most urgent operational tasks can be performed. Similarly, it could be a matter of competencies. Not all IT managers are trained to take on responsibilities at the strategic level. The missing strategic orientation is associated with the findings that the planning and implementation scores were higher than the evaluation score. It indicates that projects are planned and implemented, but that sustainable monitoring and steering according to the evaluation results rarely take place.

Limitation

When interpreting the results, various limitations must be taken into account. This study is limited with regard to the response rates of the two surveys of 12.7 and 13.4% (included answers) that might have caused a nonresponse bias.

Furthermore, focusing on Austria, Germany, and Switzerland, the empirical evidence stems from German speaking countries. However, the professionalism of information management construct as well as the corresponding measurement tool draw extensively on the international literature and thus incorporate a perspective that ensures generalizability.

It could be argued that the three IM dimensions, i.e., strategic, tactical, and operational IM, are not distinct because there is a high intercorrelation between the three levels. However, we do not assume that these dimensions are independent either, which is why a low or nearly zero correlation would be surprising. Given that the CFA is not rotated oblique, a “medium-strong” correlation is acceptable.[49]

In terms of reliability, the operational IM showed a slight decrease between the first and second survey, possibly due to the adaptation of the questions. The α value of “evaluation” in the first round amounting to 0.28 was very low. By adding an item on user satisfaction, the value could be increased to 0.56. Further items should be added in future rounds, if applicable. Generally spoken, the reliability values, however, remained acceptable.

Future Work

In addition to specifying the professionalism of information management construct consisting of strategic, tactical, and operational IM as well as of the distinction between the planning, implementation, and evaluation phase, the literature discusses a further effect on professionalism of information management. The relevant literature suggests an influence on professionalism of information management exerted by the IT relevant structures and sociotechnical aspects, i.e., the resources and position of the CIO, professionalism of information management.[7] [50] Future approaches could, therefore, examine which effects structures have on the PIM scores, e.g., whether there are structures that can increase the professionalism of information management. It also becomes possible to investigate the extent to which professionalism of information management has an impact on IT outcomes, e.g., on IT innovations as well as IT quality and how it relates to the sociocultural aspects of innovation and change.

It is also worth considering not only to concentrate on CIOs but also to include the voice of the chief executive officers (CEOs) on this topic to obtain a more comprehensive picture. This point of view promises new insights because it is particularly the CEO who decides about IT in hospitals and health systems today.[51]

To implement the PIM score as a permanent measure of the professionalisms of IM, it should continue to be surveyed within the framework of benchmarking rounds. This allows a long-term view on the evolution of IM to be obtained. By means of a benchmark, it can provide scientific information and it directly reflects the results to practitioners.

Conclusion

Using an iterative process, we could define a construct as well as develop a reliable and valid instrument to measure it. The degree of professionalism of information management is defined by the dimensions of strategic, tactical, and operational IM as well as by the elements of professionalism, i.e., the standardization and regularity of the IM procedures and their allocation to the planning, implementing, and evaluating cycle. Thus, professionalism of information management as a construct and assessment tool can be used for various practical and research purposes, e.g., for national and international comparisons of IM capabilities or as predicting constructs of health IT maturity and information systems success. As a result, professionalism of information management can serve as a catalyst for best practice or the science-practice dialogue, in which it identifies the potential for IM improvements at the individual, organizational level as well as at the level of the health care system.

Annotations

Parts of this work were published at the International Conference on Wirtschaftsinformatik (WI 2017) in St. Gallen[52] (first study on the professionalism of IM and its dimensions. Hypothesis-based research model taking into account IT governance and IT entrepreneurship) and the Medical Informatics Europe (MIE 2018) in Gothenburg[53] (excerpt and slightly modified analysis of professionalism of information management in a reliable and valid way in German hospitals as well as the impact of hospital characteristics). However, none of the publications embrace the full stepwise development of the professionalism of information management construct and its measurement.

Die Autoren geben an, dass kein Interessenkonflikt besteht.

Acknowledgments

This study was funded by the Ministry of Science and Culture of the State of Lower Saxony, Germany (grant: ZN 3062) and the German Research Foundation – DFG (grant: 1605/7-1 and 1387/8-1). We would like to thank all the participating experts and CIOs for their support. In particular, we would like to thank the working group Methods and Tools for the Management of Hospital Information Systems (mwmKIS) of the German Association for Medical Informatics, Biometry and Epidemiology (GMDS), for the support especially in the first phase of construct and questionnaire development.

• References

• 1 Winter A, Haux R, Ammenwerth E, Brigl B, Hellrung N, Jahn F. Strategic Information Management in Hospitals. In: Winter A, Haux R, Ammenwerth E, Brigl B, Hellrung N, Jahn F, eds. Health Information Systems: Architectures and Strategies. 2nd Edition. London: Springer; 2011:237–282
• 2 Arvanitis S, Loukis EN. Investigating the effects of ICT on innovation and performance of European hospitals: an exploratory study. Eur J Health Econ 2016; 17 (04) 403-418
• 3 Kuperman G. Reflections on AMIA—looking to the future. J Am Med Inform Assoc 2013; 20 (e2): e367
• 4 Avgar AC, Litwin AS, Pronovost PJ. Drivers and barriers in health IT adoption: a proposed framework. Appl Clin Inform 2012; 3 (04) 488-500
• 5 Bradley RV, Byrd TA, Pridmore JL, Thrasher E, Pratt RME, Mbarika VWA. An empirical examination of antecedents and consequences of IT governance in US hospitals. J Inf Technol 2012; 27: 156-177
• 6 Haux R, Winter A, Ammenwerth E, Brigl B. Strategic Information Management in Hospitals – An Introduction to Hospital Information Systems. New York: Springer-Verlag; 2004
• 7 Seidel C. Strategisches Informations management. In: Schlegel H. , ed. Steuerung der IT im Klinikmanagement. Wiesbaden: Vieweg+Teubner Verlag; 2010: 29-52
• 8 Schlegel H. IT-Governance mit COBIT—Methodenunterstützung für das Management. In: Schlegel H. , ed. Steuerung der IT im Klinikmanagement. Wiesbaden: Vieweg+Teubner Verlag; 2010: 7-27
• 9 Winter A, Ammenwerth E, Bott OJ, et al. Strategic Information management plans: the basis for systematic information management in hospitals. Int J Med Inform. 2001;64:99–109
• 10 Otieno GO, Hinako T, Motohiro A, Daisuke K, Keiko N. Measuring effectiveness of electronic medical records systems: towards building a composite index for benchmarking hospitals. Int J Med Inform 2008; 77 (10) 657-669
• 11 Zarnekow R, Brenner W. Integriertes Informationsmanagement: Vom Plan, Build, Run zum Source, Make, Deliver. In: Zarnekow R, Brenner W, Grohmann HH. , eds. Informationsmanagement. Konzepte und Strategien für die Praxis. Heidelberg: dpunkt.verlag; 2004: 3-24
• 12 Wild J. Management-Konzeption und Unternehmensverfassung. In: Schmidt RB. , ed. Probleme der Unternehmensverfassung. Tübingen: Mohr; 1971
• 13 Krcmar H. Informationsmanagement. 6th revised ed. Berlin: Springer Gabler; 2015
• 14 Zarnekow R, Brenner W. Auf dem Weg zu einem produkt- und dienstleistungsorientierten IT-Management. Praxis Wirtschaftsinformatik 2003; 232: 7-16
• 15 Hofmann J, Schmidt W. Masterkurs IT-Management. 2nd updated and extended ed. Wiesbaden: Vieweg and Teubner Verlag; 2010
• 16 MacKenzie SB, Podsakoff PM, Podsakoff NP. Construct measurement and validation procedures in MIS and behavioral research: integrating new and existing techniques. Manage Inf Syst Q 2011; 35 (02) 293-334
• 17 Liebe JD, Hübner U, Straede MC, Thye J. Developing a Workflow Composite Score to Measure Clinical Information Logistics. A Top-down Approach. Methods Inf Med. 2015;54(05):424–433
• 18 Thye J, Hübner U, Straede M-C, Liebe JD. Development and evaluation of a three-dimensional multi-level model for visualising the workflow composite score in a health IT benchmark. JBEI. 2016;2(02):83–98
• 19 Thye J, Straede M-C, Liebe JD, Hübner U. IT-Benchmarking of Clinical Workflows: Concept, Implementation, and Evaluation. Stud Health Technol Inform. 2014;198:116–124
• 20 Thatcher M. IT governance in acute healthcare: a critical review of current literature. In: George C, Whitehouse D, Duquenoy P. , eds. eHealth: Legal, Ethical and Governance Challenges. Berlin: Springer-Verlag; 2013: 349-370
• 21 ISACA. COBIT 5: Rahmenwerk für Governance und Management der Unternehmens-IT. Rolling Meadows: Information Systems Audit and Control Association; 2012
• 22 Hübner U, Liebe JD, Straede MC, Thye J. IT-Report Gesundheitswesen – Schwerpunkt: IT-Unterstützung klinischer Prozesse. Osnabrück: Schriftenreihe des Niedersächsischen Ministeriums für Wirtschaft Arbeit und Verkehr; 2014
• 23 Wolf EJ, Harrington KM, Clark SL, Miller MW. Sample size requirements for structural equation models: an evaluation of power, bias, and solution propriety. Educ Psychol Meas 2013; 76 (06) 913-934
• 24 MacCallum RC, Widaman KF, Zhang S, Hong S. Sample size in factor analysis. Psychol Methods 1999; 4 (01) 84-99
• 25 Comrey AL, Lee HB. A first course in factor analysis. 2nd ed. Hillsdale: Lawrence Erlbaum Associates; 1992
• 26 Cattell RB. The scientific use of factor analysis in behavioral and life sciences. New York, NY: Plenum Press; 1978
• 27 Gorsuch RL. Factor analysis. Classic 2nd ed. New York, NY: Routledge; 2015
• 28 Brown TA. Confirmatory factor analysis for applied research. New York, NY: The Guilford Press; 2006
• 29 The Pennsylvania State University. Intro—basic confirmatory factor analysis. Social Science Research Institute [online]; 2019. . Available at: https://quantdev.ssri.psu.edu/tutorials/intro-basic-confirmatory-factor-analysis . Accessed May 23, 2019
• 30 Forero CG, Maydeu-Olivares A, Gallardo-Pujol D. Factor analysis with ordinal indicators: a Monte Carlo study comparing DWLS and ULS estimation. Struct Equ Modeling 2009; 16 (04) 625-641
• 31 Maydeu-Olivares A. Limited information estimation and testing of Thurstonian models for paired comparison data under multiple judgment sampling. Psychometrika 2001; 66 (02) 209-227
• 32 Tabachnick BG, Fidell LS. Using Multivariate Statistics. 5th ed. Boston: Allyn & Bacon/Pearson; 2007
• 33 Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate Data Analysis. 7th ed. Upper Saddle River, NJ: Prentice Hall; 2009
• 34 Hübner U, Esdar M, Hüsers J, et al. IT-Report Gesundheitswesen - Schwerpunkt: Wie reif ist die IT in deutschen Krankenhäusern? Osnabrück: Schriftenreihe der Hochschule Osnabrück; 2018
• 35 Paré G, Sicotte C. Information technology sophistication in health care: an instrument validation study among Canadian hospitals. Int J Med Inform 2001; 63 (03) 205-223
• 36 Weill P, Ross JWIT. Governance. Boston: Harvard Business School Press; 2004
• 37 Chapman P, Wieder B. IT governance as a higher order capability. Paper presented at: Pacific Asia Conference on Information Systems (PACIS 2015), Singapore 2015; 2015: 150
• 38 Arezki S, Elhissi Y. Toward an IT governance maturity self-assessment model using EFQM and CobiT. Paper presented at: the International Conference on Geoinformatics and Data Analysis (ICGDA 2018), Prague 2018; 2018: 198-202
• 39 Barney J. Firm resources and sustained competitive advantage. J Manage 1991; 17 (01) 99-120
• 40 Wernerfelt B. A resource-based view of the firm. Strateg Manage J 1984; 5 (02) 171-180
• 41 Esdar M, Hübner U, Liebe J-D, Hüsers J, Thye J. Understanding latent structures of clinical information logistics: A bottom-up approach for model building and validating the workflow composite score. Int J Med Inform. 2017;97:210–220
• 42 Leidner DE, Preston D, Chen D. An examination of the antecedents and consequences of organizational IT innovation in hospitals. J Strategic Inf Syst 2010; 19: 154-170
• 43 Grillmayer H. Best Practice in der Servicesteuerung—ITIL und ISO 20000. In: Schlegel H. , ed. Steuerung der IT im Klinikmanagement. Wiesbaden: Vieweg and Teubner Verlag; 2010: 111-130
• 44 Vejseli S, Rossmann A. The impact of IT governance on firm performance: a literature review. Paper presented at: the Pacific Asia Conference on Information Systems (PACIS 2017), Langkawi 2017; 2017: 41
• 45 Ammenwerth E, Haux R, Knaup-Gregori P, Winter A. IT-Projektmanagement im Gesundheitswesen. 2nd Edition. Stuttgart: Schattauer GmbH; 2015
• 46 Heinrich LJ, Riedl R, Stelzer D. Informationsmanagement. 10th ed. München: Oldenbourg Wissenschaftsverlag; 2011
• 47 HIMSS Analytics—Europe. Electronic Medical Record Adoption Model; 2017. Available at: https://www.himssanalytics.org/europe/electronic-medical-record-adoption-mode . Accessed March 8, 2020
• 48 Fenton SH, Low S, Abrams KJ, Butler-Henderson K. Health Information management: changing with time. Yearb Med Inform 2017; 26 (01) 72-77
• 49 Shevlin M, Adamson G. Alternative factor models and factorial invariance of the GHQ-12: a large sample analysis using confirmatory factor analysis. Psychol Assess 2005; 17 (02) 231-236
• 50 Hütter A, Arnitz T, Riedel R. On the Nature of Effective CIO/CEO Communication. Cham: Springer; 2017
• 51 Thye J, Hübner U, Hüsers J, Babitsch B. IT Decision Making in German Hospitals - Do CEOs Open the Black Box? Stud Health Technol Inform. 2017;243:112–116
• 52 Liebe JD, Thomas O, Jahn F, et al. Zwischen Schattendasein, Governance und Entrepreneurship - Eine empirische Bestandsaufnahme zum Professionalisierungsgrad des IT-Managements in deutschen Krankenhäusern. Proceedings 13th International Conference on Wirtschaftsinformatik (WI 2017), St. Gallen 2017. 2017:559–573
• 53 Liebe JD, Esdar M, Jahn F, Hübner U. Ready for HIT Innovations? Developing a Tool to Assess the Professionalism of Information Management in Hospitals. Stud Health Technol Inform. 2018;247:800–804