A model for post-implementation evaluation of health information systems: The case of the DHIS 2 in Kenya

Abstract

The Kenyan government through the Ministry of Health (MoH) in its quest to improve the health outcomes of its citizenry, has embarked on a number of initiatives. One of these initiatives under the umbrella of eHealth is the District Health Information Software(DHIS 2). DHIS 2 is ”an integrated, web-based, country owned and managed, national health information system that integrates quality data used at all levels to improve health service delivery”Douglis (2012) . The logic behind the emphasis of DHIS 2 is that with quality data, public health decision making will be more efficient and effective. Despite the importance of DHIS 2, no study has been done to evaluate the post-implementation of the system, particularly user perspective. DHIS 2 was a large investment and like any organisation, the MoH would be interested in knowing the impact the system has on its users. The objectives of this study were: to establish the user’s perspective in evaluating a health information system (HIS) and present a set of evaluation factors that can be used in evaluating a HIS, identify a suitable evaluation model and, test the evaluation model using DHIS 2. We reviewed literature on information systems, health information systems, the information system(IS) user and IS success theories and models. The IS-Impact model was used as the theoretical foundation for this study. The model comprises four dimensions, which are Individual Impact, Organizational Impact, Information Quality and System Quality. The questionnaire was the main data collection tool with focus group discussions(FGDs) being used as complementary tool. A total of 135 reponses from questionnaires and transcribed data from 2 FGDs were used for analysis. The results indicate that 89% of the respondents were operational or day-to-day DHIS 2 users with System Quality being the strongest contributor to the IS-Impact model. In Individual Impact, the individual performance of the respondents on behalf of their organisation, most respondents stated that DHIS 2 had an impact in their performance. Under Organisation Impact, the impact of DHIS 2 at a broader, organisational level, most respondents felt that DHIS 2 has an impact particulalrly in achieving the MOH’s objectives. In Information Quality, the quality of information that the system produces in reports and on screen, most respondents stated that it was high. However, respondents were divided on whether data from DHIS 2 was unique. Under System Quality where users were asked on their perception of DHIS 2 performance from a technical and design perspective, there were mixed views regarding system quality issues. The FGDs revealed that impact of DHIS 2 was due to comparison with the previous Excel-based system which was time-consuming and involved a lot of paperwork. On quality the FGDs revealed problems with lack of DHIS 2 integration with other systems and inconsistencies between data collection tools and the DHIS 2 data entry portal. The conceptual model was tested using tolerance, Variance Inflation Factor(VIF) and correlation co-efficients. The test results indicate that 30 measures are significant for measuring the impact of health information systems in Kenya. This demonstrates the validity of the model. Based on these findings, a model for post-implementation evaluation of health information systems is presented. It is recommended that the model needs to be extended further to include the context under which the system is based. For instance, comparative studies can be conducted to evaluate the success of the HIS in a rural setting as compared to an urban setting. Also, one of the limitations in this study was getting respondents who were in strategic management. This study’s respondents were operational users i.e., they interacted with the system on a day to day basis. Strategic users as compared to operational users interact less with the system but are important because they decide on the resources required to achieve organizational objectives. Also, decisions made in the strategic level have long-term implications. Therefore, this model can be extended further to include strategic users.