Assessment of eHealth Literacy in Healthcare Service Users: Construction and Validation of a Measurement Instrument

1. INTRODUCTION

Currently, 67% of the world's population uses the Internet, fundamentally transforming human interactions [1]. Digital connectivity is essential for daily life, providing access to critical services and educational opportunities [2].

Information and communication technologies (ICT) are essential for modern healthcare [3]. The Internet has become a primary source of health information for patients [4] [5] [6].

Health literacy refers to an individual's ability to access, process, and comprehend basic health information and services necessary for making informed health decisions [7]. E-health literacy builds upon this concept, emphasizing the ability to navigate, evaluate, and apply health information from digital sources to address or resolve health-related issues [8].

E-health literacy is strongly influenced by exposure to digital technologies, Internet access, and online health information sources [9] [10]. Higher e-health literacy is observed (AOR = 2.35; 95% CI: 1.67, 3.30) among individuals with Internet access [11], as well as those with access to smartphones or other mobile devices, and those with higher levels of education [12].

E-health literacy is associated with better self-care, improved medication adherence, and enhanced disease management and preventive behaviors [13]. Low health literacy is linked to adverse health outcomes and suboptimal utilization of health services [14]. Moreover, e-health literacy has a positive impact on health promotion, disease prevention, psychological well-being, social support, and healthcare service utilization [15].

A widely recognized and utilized instrument for assessing digital literacy is the eHealth Literacy Scale (eHEALS) [16]. The eHEALS was originally designed for the first generation of eHealth technologies (Web 1.0), with an emphasis on information retrieval [17] [18]. However, it lacks comprehensiveness in assessing eHealth literacy within the context of Web 2.0 [15]. The World Wide Web (WWW) has evolved from Web 1.0, which was characterized by static, information-centered content, to Web 2.0, which introduced interactivity and user-generated content, giving rise to social media platforms. It has continued to evolve through Web 3.0 and Web 4.0, and is expected to advance further toward Web 5.0 in the future [19].

The development of a new measurement instrument is justified for several reasons. First, due to the obsolescence of existing tools [20]; moreover, the interactive applications that characterize Web 2.0 and beyond require a broader range of skills [17]. Currently available instruments present limitations resulting from their age, which compromises their relevance in a context where digital environments have rapidly evolved. This is the case of the eHEALS [16], one of the most widely used scales for measuring eHealth literacy, which was developed prior to the rise of social media and mobile web technologies [21], and has proven insufficient to assess the new skill sets required for interacting with emerging technologies. In the Peruvian context and across different geographical settings, both surveyors and healthcare users have perceived shortcomings [22] [23] [24].

Second, since the onset of the COVID-19 pandemic [25], the digitalization of individuals and electronic environments became essential to ensure healthcare delivery and continuity of services [26]. In-person consultations were replaced by telemedicine in both primary care centers and hospitals [27]. Moreover, telemedicine and eHealth have demonstrated significant potential for improving health outcomes [26].

Social media can be used for various health-related purposes, with both healthcare organizations and patients maintaining an active presence, giving rise to new dynamics in the access, search, use, and dissemination of health information [28] [29] [30]. These changes demand tools that assess not only basic digital competencies, but also other dimensions such as interaction in digital environments, information quality, and informed decision-making. Therefore, the objective of this study was to develop and validate a contextualized and updated instrument to assess eHealth literacy among healthcare service users, thereby contributing to a better understanding and approach to health-related issues.

2. MATERIALS AND METHODS

2.3. General Description of Study

The study was conducted in the following stages: literature review, instrument construction, selection of expert judges, content validation, preliminary pilot test, field testing of the instrument, construct validity, and reliability assessment (Fig. 1). The entire study process took place from January to December 2024.

2.3.1. Literature Review

A literature search was conducted in PubMed using eHealth Literacy Scale' as the search term. The instrument was designed based on Norman's Lily model and the eHealth Literacy Scale (eHEALS) [8] [16].

2.3.2. Instrument Construction

A key aspect in instrument development is ensuring that all domains of the variable of interest are adequately represented [20]. In this study, we followed the 'Lily model' of eHealth literacy [8], which includes six types of literacy: Traditional Literacy, Information Literacy, Media Literacy, Health Literacy, Computer Literacy, and Scientific Literacy. Items were developed for each of these domains accordingly.

The initial instrument was designed following Deming’s plan-do-check-act (PDCA) cycle [42]. A total of 33 items were proposed, distributed across six domains: Traditional Literacy (4 items), Information Literacy (4 items), Media Literacy (6 items), Health Literacy (7 items), Computer Literacy (7 items), and Scientific Literacy (5 items). This set of items constituted the initial version of the instrument (Version 1).

2.3.3. Selection of Expert Judges

Professionals with postgraduate studies, as well as teaching and clinical experience, were considered. Experience in eHealth was considered desirable. Candidates were selected based on their published articles or their participation in an eHealth academic program. The introduction letter and collaboration request were sent via email, using addresses retrieved from their publications or through their social and academic networks. Upon acceptance, they received the validation documents.

A total of 36 candidates were invited via email, of whom 13 experts from Bolivia, Ecuador, Spain, Mexico, and Peru responded and participated in the validation process. Of the two judges from Ecuador, one was excluded as their evaluation was deemed inapplicable; however, their suggestions were taken into account. The final panel for validation consisted of 12 judges. The literature suggests that a panel of at least 10 experts provides a reliable estimate of an instrument’s content validity [43].

2.3.4. Content Validation

For content validity assessment, the criteria of appropriateness, relevance, and clarity were examined. These criteria were assessed using a Likert scale ranging from 1 to 4, with the following categories: Does not meet the criteria (1 point), Low level (2 points), Moderate level (3 points), and High level (4 points). Additionally, each item included options for suggestions and comments.

Content validity was assessed using Aiken's V coefficient, which quantifies the degree of consensus among judges. This coefficient ranges from 0 to 1, with higher values indicating greater agreement [44] [45]. A value above 0.8 was considered indicative of strong consensus [44] [45].

In addition to the statistical analysis from the first evaluation, necessary modifications were made in response to feedback, resulting in a new version. Before undergoing the preliminary pilot test, the instrument was reviewed and approved by experts in instrument design and validation (Version 2).

2.3.5. Preliminary Pilot Test

Before testing the questionnaire with the target respondents, it is recommended to conduct a pilot test on a small sample of 30 to 50 respondents [46] [47]. The pilot test can be both qualitative and quantitative [20]. This preliminary test helps identify ambiguous items and gather suggestions for potential improvements [46] [48].

For this purpose, we used the questionnaire in digital format (Version 2), developed in Google Forms®, and administered it to a sample with characteristics comparable to those of the target population.

A total of 53 healthcare service users (43 females, 10 males) were randomly selected, with a mean age of 35.9 years (SD: 10.9; Min: 18, Max: 64). The completion of the instrument took approximately 10 to 15 minutes. Additionally, participants' suggestions and observations regarding clarity and simplicity were collected and considered in developing the final version of the instrument.

In this test, priority was given to the qualitative component, which allowed for the identification of difficulties in comprehension, application time, and data collection feasibility. Based on these findings, the questionnaire was refined and improved, resulting in the final version (Version 3), which was used for the main validation. The final version consisted of 21 items (q1 to q21), rated on a Likert scale from 1 to 5, with the following scale: 1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, and 5 = strongly agree.

2.3.6. Field Testing of the Instrument

At this stage, the final version of the questionnaire (Version 3) was administered to a sample of the target population. A total of 1068 randomly selected healthcare service users of both sexes participated.

Data collection was conducted mainly through interviews and, to a lesser extent, through self-administered questionnaires. The instrument was administered across different services of a public healthcare facility in the district of 'Mi Perú' by two healthcare personnel who had received prior training. They conducted the application for at least 180 minutes per session. Data collection took place from September to October 2024.

2.3.7. Construct Validity and Reliability

The following procedures were performed on the data collected during the field test: the correlation of each questionnaire item was assessed. To determine whether to retain or remove items, an item-total analysis was conducted. Items with item-total correlations below 0.3 were removed [49] [50].

A polychoric correlation matrix of the items was generated, along with its graphical representation. Before conducting the Exploratory Factor Analysis (EFA), sample adequacy was assessed using the Kaiser-Meyer-Olkin (KMO) index and Bartlett's test of sphericity. The KMO index ranges from 0 to 1. A value above 0.7, along with a statistically significant Bartlett's test of sphericity (p<0.05), is recommended to proceed with EFA [51].

In the EFA, the optimal number of factors to extract was determined using the Kaiser rule (Eigenvalue >1) and the scree plot, the most commonly used methods [52]. Additionally, parallel analysis was performed, as it is considered highly reliable [53].

The questionnaire used in this study comprised ordinal variables (Likert-type items). Ordinal variables do not satisfy the assumptions of linearity and normality [39]. When normality is unlikely or when univariate skewness and kurtosis are excessively high, more robust correlation methods, such as polychoric correlation, are recommended [39]. Assuming a normal distribution of latent variables, polychoric correlations provide more precise estimates that are less affected by measurement error, outperforming Pearson and Spearman correlations in the analysis of ordinal data [54] [55].

For this study, EFA was performed on the polychoric correlation matrix, using Weighted Least Squares (WLS) extraction with Oblimin rotation. WLS with polychoric correlations generally outperforms Maximum Likelihood (MLE) with Pearson correlations. While MLE tends to underestimate true factor loadings, WLS may overestimate them [54].

Instrument reliability was assessed using Cronbach's alpha coefficient. Cronbach's alpha is commonly used to assess internal consistency [56] [57] [58]. An alpha value between 0.70 and 0.95 is considered acceptable for reliability assessment [56].

3. RESULT

3.4. Exploratory Factor Analysis

The scree plot visualization based on the eigenvalue-greater-than-one rule suggested a three-factor solution (Fig. 2). For the sample adequacy measures of the 18 items, we calculated the Kaiser-Meyer-Olkin (KMO) measure and Bartlett's test of sphericity (KMO = 0.92, Bartlett's p < 0.001) Table 1.

Table 1.

Sample Adequacy Measures for Exploratory Factor Analysis									Interpretation
Kaiser-Meyer-Olkin factor adequacy
Overall MSA= 0.92									Adequate
MSA for each item:
Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8	Q9
0.96	0.95	0.93	0.95	0.96	0.66	0.91	0.90	0.95
Q10	Q11	Q12	Q13	Q14	Q15	Q16	Q17	Q18
0.94	0.96	0.97	0.95	0.86	0.95	0.89	0.80	0.76
Bartlett test of homogeneity of variances
Bartlett's K-squared = 11750.31, df = 153, p-value < 0.001									Adequate for EFA

Table 2.

	WLS1	WLS4	WLS3	WLS2	h2	u2	com
Q1	0.167	0.457	0.288	-0.017	0.575	0.425	1.99
Q2	0.218	0.379	0.056	0.093	0.395	0.605	1.79
Q3	-0.025	0.866	-0.026	0.028	0.718	0.282	1.01
Q4	0.104	0.563	0.148	0.018	0.527	0.473	1.21
Q5	0.183	0.552	0.128	0.052	0.617	0.383	1.36
Q6	-0.086	0.249	-0.310	0.310	0.159	0.841	3.06
Q7	0.882	-0.026	0.016	0.011	0.768	0.232	1.00
Q8	0.947	0.009	-0.018	-0.008	0.885	0.115	1.00
Q9	0.723	-0.099	0.136	0.075	0.590	0.410	1.13
Q10	0.181	0.154	0.150	0.048	0.186	0.814	3.08
Q11	0.533	0.315	-0.026	-0.008	0.601	0.399	1.63
Q12	0.445	0.327	0.016	0.024	0.542	0.458	1.85
Q13	0.686	0.108	-0.076	0.011	0.539	0.461	1.07
Q14	-0.077	0.023	0.797	-0.057	0.563	0.437	1.03
Q15	0.205	0.300	0.511	0.076	0.777	0.223	2.04
Q16	0.055	-0.011	0.766	0.142	0.729	0.271	1.08
Q17	0.032	-0.024	0.032	0.900	0.840	0.160	1.01
Q18	-0.038	0.018	-0.015	0.831	0.670	0.330	1.01

(*) Extraction method: Weighted Least Squares (WLS), Rotation method: Oblimin, h2: Communality, u2: Uniqueness, com: Complexity.

Using the Weighted Least Squares (WLS) extraction method with Oblimin rotation, we identified four factors. Items were grouped as follows: WLS1 (Q7, Q8, Q9, Q10, Q11, Q12, Q13), WLS4 (Q1, Q2, Q3, Q4, Q5), WLS3 (Q14, Q15, Q16), and WLS2 (Q6, Q17, Q18) Table 2.

These four factors collectively accounted for 59.3% of the total variance (WLS1: 22.4%; WLS4: 15.7%; WLS3: 11.5%; and WLS2: 9.8%) Table 3.

4. DISCUSSION

The aim of this study was to develop and validate an instrument for measuring eHealth literacy among health service users. The results suggest that the instrument has a four-dimensional factorial structure.

In exploratory factor analysis (EFA), commonly used methods for determining the number of factors include retaining those with an eigenvalue greater than one or using the scree plot [52]. The Kaiser rule (Eigenvalue > 1) is considered unreliable and outdated [60] [61].

Items with factor loadings below 0.30 are considered inadequate, as they contribute less than 10% of the variance in the measured latent construct [62]. The rules for retaining an item within a factor are generally arbitrary, and there is no mathematical test indicating that values must exceed |0.3| or |0.6|; instead, a strong rationale should be provided for the chosen threshold [60]. More complex, often simulation-based approaches exist that offer a more accurate estimation of the number of latent factors, and in this context, parallel analysis is considered the gold standard for factor retention [53] [61].

In this study, the Kaiser rule suggested three factors, while according to the graph curve it suggests four factors. We decided to retain four factors, which were then extracted using the Weighted Least Squares (WLS) method with Oblimin rotation.

The resulting factors were defined as follows: WLS1 (Q7, Q8, Q9, Q10, Q11, Q12, Q13), WLS4 (Q1, Q2, Q3, Q4, Q5), WLS3 (Q14, Q15, Q16), and WLS2 (Q6, Q17, Q18), collectively explaining 59.3% of the total variance. These factors were subsequently labeled as “Digital Literacy” (WLS1), “Digital Self-Management” (WLS4), “Digital Skill” (WLS3), and “Scientific Empowerment” (WLS2), respectively.

There is no established rule to guide the selection of an orthogonal or oblique rotation technique [51]. Among the various rotation methods available, Varimax is the most widely used orthogonal method [39]. In this study, we employed Oblimin rotation. This technique minimizes the cross-products of loadings to simplify factors, and it is also the preferred method when working with large datasets [63].

Factor rotation is intended to achieve a more interpretable solution [39]. Moreover, factor rotation does not alter the amount of total variance explained but facilitates the analytical process by more clearly associating each variable with a single factor and enhancing its explanatory power [51].

In this study, the four extracted factors included an adequate number of indicators. However, one item (Q6) showed a positive factor loading of 0.31 on Factor WLS2, while simultaneously presenting a negative loading of -0.31 on Factor WLS3.

When alternative extraction methods were applied [64]—such as Minimum Residuals (MINRES), Weighted Least Squares Mean and Variance Adjusted (WLSMV), and Unweighted Least Squares (ULS), all compatible with polychoric matrices—all items remained within the four-factor structure. Nonetheless, item Q6 had a higher positive loading (0.311) on the fourth factor, along with items Q17 and Q18 (while on the third factor, item Q6 showed a negative loading of -0.308). Therefore, we decided to retain item Q6 within Factor WLS2.

Several authors have noted that at least three measured variables are required for the statistical identification of a factor. Although more indicators are preferable, it is recommended to have four to six indicators per factor [39] [65]. A factor with fewer than three items is typically considered weak and unstable; however, with a very large dataset, it may be possible to reduce the number of items while still maintaining a strong factor [66]. The quality of the items is also considered as indicators of the factor [67].

There is a strong factor loading for each item associated with the four extracted factors. A factor loading greater than 0.30 generally indicates a moderate correlation between the item and the factor. Additionally, the higher the communality value (h²), the more the extracted factors explain the item's variance [68]. The remaining variance of the item is considered unique variance (u²), which does not contribute to the measurement of the common factors [65].

In the study, with the exception of three items belonging to three factors [WLS4 (Q2), WLS2 (Q6), and WLS1 (Q10)], the h² values were greater than 0.5 (0.50 < h² ≤ 0.99).

Regarding reliability, when working with the 21 items, the Cronbach's Alpha coefficient obtained was 0.85. However, for reliability within each factor, only one factor had a value of 0.83, while the remaining three factors had values lower than 0.70. The three eliminated items (q1, q7, and q11) had a significant impact on reliability.

When eliminating items, it is essential to conduct another EFA and report which variables were removed, as well as the criteria used to make the decision [64]. It is known that many items may be lost during the theoretical and psychometric analysis [48]. Therefore, the EFA results presented correspond to the data from the 18 items.

The new Cronbach's Alpha value for the complete 18-item scale was 0.86 (95% CI: 0.85-0.87), and 0.9 for McDonald's ω coefficient. Furthermore, the coefficients for each factor also improved [WLS1 (0.81, 95% CI: 0.80-0.83), WLS4 (0.72, 95% CI: 0.70-0.75), WLS3 (0.67, 95% CI: 0.64-0.71), and WLS2 (0.60, 95% CI: 0.56-0.64)]. After eliminating an item, the coefficients remained ≥ 0.84; however, items Q6, Q10, and Q14 had low item-total correlations of 0.14, 0.32, and 0.30, respectively. Items with very low adjusted item-total correlations (< 0.30) are less desirable and may indicate the need for their possible removal from the scale [62].

To meet the minimum required number of items per factor [39] [65], we decided to retain these items with low item-total correlations, but minor modifications in wording were made without substantially altering the content. For minor modifications, a formal assessment of the psychometric properties of the instrument may not be necessary, or a small-scale pilot test may suffice [69]. In our case, the change was made with the involvement of one of the experts. The way respondents interpret the meaning of the questions strongly influences their answers; therefore, it is important that the questions are brief, clear, and unambiguous [62] [70].

Regarding the Cronbach's alpha coefficient, there is a wide range of qualitative descriptors based on the value range [71], with acceptable alpha values considered between 0.70 and 0.95 [56], although values between 0.80 and 0.95 are preferred for the psychometric quality of scales [62]. A low alpha coefficient may be due to a low number of questions, poor inter-item correlations, or heterogeneous constructs, while a high value (>0.90) may suggest redundancies [56].

Although the alpha coefficient is often used as a measure of internal consistency and reliability [56] [57] [58], it has its limitations, as it can either underestimate or overestimate; in this regard, the omega coefficient (ω) provides a better estimate of an instrument's reliability than the alpha coefficient [57].

The alpha coefficient is lower than omega, and therefore, it can be trusted as the lower bound of reliability; moreover, the difference between alpha and omega has no practical consequences when the factor loadings average 0.70 [72]. For now, there is insufficient reason to completely abandon the use of alpha and replace it with omega, but rather to use both coefficients as complementary estimators [73].

We consider that the 18-item instrument has a reasonable length and meets the criteria for validity and reliability. There is no general rule for determining the number of items in a questionnaire; however, the questionnaire should contain enough items to measure the construct of interest, but it should not be so long that respondents experience fatigue or loss of motivation [46].

E-Health literacy considers various aspects. In this regard, the items grouped into the four factors obtained, namely “Digital Literacy,” “Digital Self-management,” “Digital Skill,” and “Scientific Empowerment,” are relevant and consistent with the “lily model” of eHealth literacy, which includes six types of literacy [8].

Digital literacy refers to the ability to find and use health information through technology [74], while digital self-management can help individuals manage their health issues [75]. On the other hand, digital skill involves the ability to use computers and adapt to new technologies [8], whereas scientific empowerment refers to patient participation in co-creating knowledge with the aim of achieving autonomy [76].

Within the limitations and strengths, one limitation is that the study population was limited to primary healthcare users from a single district; however, some of these users also visit at least three hospitals in the Callao Region. Additionally, neighboring districts have similar sociodemographic characteristics and also host a significant number of migrants from other parts of the country, so the results can be at least extrapolated to the Peruvian population. Furthermore, the neighboring districts have similar sociodemographic characteristics and also host a significant number of migrants from other regions of the country, so the results can be extrapolated at least to the Peruvian population.

It should also be considered as a limitation that the study did not include confirmatory factor analysis (CFA), which would have allowed for a more complete analysis of the psychometric properties of the instrument. CFA is planned for future studies, along with the possibility of validation in other contexts.

On the other hand, we consider the sample size as a strength, as it was much larger than the minimum recommended sample, which was crucial for the EFA based on a polychoric matrix. An EFA based on a polychoric correlation matrix requires a larger sample compared to a Pearson correlation matrix to achieve the same level of precision and stability [65].

CONCLUSION

The results of this study support that the eHealth literacy assessment scale for healthcare service users shows adequate levels of validity and reliability. The findings suggest that the questionnaire could be considered as an alternative for use both at the primary care level and in the hospital setting. However, additional studies are recommended to expand the psychometric analysis.

It is also relevant to explore its performance in diverse populations and different cultural contexts, as well as its usefulness in interventions aimed at improving eHealth literacy. These actions will contribute to consolidating the instrument's robustness and optimizing its applicability in both clinical and community settings.

REFERENCES