Subscribe to RSS
DOI: 10.1055/s-0041-1728658
Examining the Relationship between Altmetric Score and Traditional Bibliometrics in the Ophthalmology Literature for 2013 and 2016 Cohorts
Abstract
Background In this study, we reviewed a select sample of ophthalmology literature to determine if there was a correlation between Altimetric and traditional citation-based and impact factor metrics. We hypothesized that Altmetric score would more closely correlate with impact factor and citations in 2016.
Methods Journal Citation Reports for the year 2013 was used to find the 15 highest impact factor ophthalmology journals in 2013. Then Elsevier's Scopus was used to identify the 10 most cited articles from each journal for the years 2013 and 2016. Metrics for all identified articles were collected using the Altmetric Bookmarklet, and date of Twitter account creation was noted for journals with such an account. Altmetric scores, impact factor, and citation counts were tabulated for each article. Pearson's correlation coefficient (r) determined correlation of independent variables (number of citations or impact factor) with dependent variable (Altmetric score). For our Twitter analysis, account age was the independent variable and calculated correlation coefficients (r) were the dependent variable. Proportion of variance was determined with a coefficient of determination (R 2).
Results This study included 300 articles, evenly split between 2013 and 2016. Within the 2013 cohort, three journals had significant positive correlations between citation count and Altmetric score. For the 2016 cohort, both Altmetric score and citation count (r = 0.583, p < 0.001) and Altmetric score and impact factor (r = 0.183, p = 0.025) revealed significant positive correlations. In 2016, two journals were found to have significant correlations between Altmetric score and citation number. Neither year revealed a significant correlation between the age of a journal's Twitter profile and the relationship between Altmetric score and citation count. In each year, Twitter accounted for the highest number of mentions.
Conclusion The findings suggest that correlation between Altmetric score and traditional quality metric scores may be increasing. Altmetric score was correlated with impact factor and number of citations in 2016 but not 2013. At this time, Altmetrics are best used as an adjunct that is complementary but not an alternative to traditional bibliometrics for assessing academic productivity and impact.
#
Introduction
Bibliometrics is a method of quantifying and analyzing the impact of academic publications and ultimately is used for determining academic productivity, scholarly effort, journal prestige and impact, and individual promotion and tenure. Traditionally, bibliometrics include quantity indicators, performance indicators, and structural indicators of an article.[1] Citations give credit for an idea or fact to an earlier article, and the number of times an article is cited is generally understood as a marker of impact within an academic field.[1] Citation analysis, then, involves the ranking of an article based on the number of citations it receives to identify published work with the greatest intellectual influence.[2] Impact factor is a marker of impact for journals that compares the number of citations for a given year to the number of articles published. More citations per article equate to a higher impact factor.[1] These metrics are considered quality indicators and can have tremendous impact in funding decisions for individual researchers and entire departments.[3]
Traditional bibliometrics allow the application of a standardized, scientifically rigorous methodology to the determination of the quality of individual publications, researchers, and journals in aggregate, as well as to assess the productivity of departments, research teams, universities, and even entire countries.[4] The recent advent of newer information technologies and platforms, however, has created opportunities for dissemination of information among different audiences and across multiple platforms and formats.[5] Traditional bibliometrics fail to capture the reach of this interconnected information web and the larger impact that publications may have for a broader audience. Furthermore, citation time window has been shown to have a large role in determination of elite, highly cited articles. The list of highly cited articles at 5 years from publication is quite different from the list of highly cited articles at 30 years.[4] It is obvious there is much that traditional bibliometrics leaves out.
Recently, researchers have begun to question why markers for research impact should not include broader impact among the general public.[3] [5] [6] Altmetrics, so named with the intention of being alternative metrics of publication impact, include mentions of articles on various social media sites, as well as Wikipedia, patent applications, and news outlets.[7] The term “Altmetric” is sometimes used to refer generally to all alternative metrics of article reach, but here we use the term to refer to the article score provided by Altmetric.com. Rather than supplanting bibliometrics and peer review as markers of research quality, alternative metrics are intended to be used along with these traditional means as part of a “basket of metrics” for quantifying impact in a particular field.[5] The objective of this study is to examine correlation between bibliometric and Altmetric scores for a subset of articles in ophthalmology literature, to answer the questions of whether Altmetric data correlates with bibliometrics, and in what ways it may be weaker, stronger, or complementary. It was hypothesized that there would be a stronger correlation between Altmetric scores and traditional bibliometrics in the later cohort as a reflection of social media's expanded influence across time compared with articles published previously.[8] In this study, we reviewed a select sample of ophthalmology literature to determine if there was a correlation between Altimetric and traditional citation based and impact factor metrics. To our knowledge, this is the first such study in the English language ophthalmic literature.
#
Methods
Using Journal Citation Reports for the year 2013, the 15 ophthalmology journals with the highest impact factors were selected ([Table 1]). Then, using Elsevier's Scopus,[9] the 10 most cited articles from each journal were identified for the years 2013 and 2016. These particular years were selected to ensure article metrics had sufficient time to mature after publication, as well as to allow comparison of two distinct snapshots of a changing social media landscape. The Altmetric Bookmarklet was used to collect metrics for all identified articles, and journals with an active Twitter account were identified and date of creation of the account was noted.
Altmetric scores are calculated by an automated algorithm. This algorithm is designed to take into account two aspects of a share: quality and quantity. To capture quality, different sources of attention are given different weights: a mention in a news source carries more weight than a mention in a blog, which carries more weight than a mention on Twitter, which carries more weight than a mention on Facebook or Reddit. These relative weights are intended to capture the likelihood that that mention will bring attention to the source. For example, a news story is likely to bring more attention than a post on Facebook. The Altmetric algorithm also takes quantity into account, as well as other factors, so the score is not simply a sum of all mentions by each media type.[10]
Pearson's correlation coefficient (r) determined correlation of independent variables (number of citations or impact factor) with dependent variable (Altmetric score). A significant p-value was predefined as <0.05 and Microsoft Excel was used for all statistical analyses. For our Twitter analysis, account age was the independent variable and calculated correlation coefficients (r) were the dependent variable. Proportion of variance was determined with a coefficient of determination (R 2). These statistical analysis methods were derived from prior research examining the correlation between Altmetric score and traditional bibliometrics in the general surgery,[11] pediatric surgery,[12] dental implantology,[13] and urology literature.[14]
#
Results
The 15 journals with the highest impact factor based on the Journal Citation Reports in the year 2013 were selected for analysis. Subsequently, the 10 articles accruing the most citations were obtained from each journal in 2013 and 2016 ([Table 1]). In total, 300 articles were reviewed and included in this analysis. The cohort of publications from 2013 accrued 16,753 citations with a median of 96 (21–441). For the same cohort, the summed Altmetric score of all articles was 1,495 with a median of 2.5 (0–399).
As would be expected, there was a significant positive relationship between article citation count and journal impact factor in 2013 (r = 0.394, R 2 = 0.155, p < 0.001). Altmetric score, however, did not have a significant correlation with either citation count (r = 0.076, R 2 = 0.00057, p = 0.358) or impact factor (r = 0.086, R 2 = 0.0074, p = 0.295) for the same cohort ([Fig. 1A, B]). Within each journal, the presence of a correlation between Altmetric score and article citation number was studied ([Table 2]). Three journals had significant positive correlations between these two variables: American Journal of Ophthalmology (r = 0.684, p = 0.029), Retina-The Journal of Retinal and Vitreous Disease (r = 0.683, p = 0.029), and British Journal of Ophthalmology (r = 0.841, p = 0.002). No other journals in 2013 had a significant correlation between citation count and Altmetric score.
To establish a frame of comparison across time in regard to social media usage in the ophthalmology literature, an identical number of the highest cited articles were reviewed from the same journals in the year 2016. This 2016 cohort accrued a sum of 9,736 citations with a median of 48.5 (15–432). The 2016 total citations represented a 41.9% decline from the 2013 total citations that is expected given that the 2016 cohort has had less time to accrue citations. In addition, the summed Altmetric score for all articles in this cohort was 6,638 with a median of 3 (0–1,340). The 2016 cumulative Altmetric score represents a 344% increase in relation to the 2013 total. Nearly 42% of the cumulative Altmetric score in 2016 was composed from one journal—Archives of Ophthalmology (now JAMA Ophthalmology).
Like the 2013 group, review of the 2016 cohort demonstrated that citation count and impact factor were positively correlated (r = 0.540, R 2 = 0.2912, p < 0.001). Distinct from the 2013 cohort, in 2016 analysis of both Altmetric score and citation count (r = 0.583, R 2 = 0.3398, p < 0.001) and Altmetric score and impact factor (r = 0.183, R 2 = 0.0333, p = 0.025) revealed significant positive correlations ([Fig. 1C, D]). Studying the journals individually, in 2016 two journals were found to have significant correlations between Altmetric score and citation number. These included positive correlations for the journals Ophthalmology (r = 0.748, p = 0.013) and Experimental Eye Research (r = 0.719, p = 0.019) ([Table 2]).
Interestingly, less than half (7 of 15) of the ophthalmology journals studied had an established Twitter account at the time this study was performed ([Table 1]). The average age of established Twitter profiles was 7.27 years with the oldest account belonging to Archives of Ophthalmology (now JAMA Ophthalmology) and the youngest to the Journal of Cataract and Refractive Surgery.
The correlation between the age of a journal's Twitter profile and the relationship between Altmetric score and citation count was determined for both years. Neither year revealed a significant correlation: 2013 (r = 0.070, p = 0.882) and 2016 (r = 0.673, p = 0.098) ([Fig. 2]). Reviewing the individual components that derive the composite Altmetric score, in each year Twitter accounted for the highest number of mentions. The second and third highest mentions were on news outlets and Facebook, respectively ([Table 3]). Across the individual factors contributing to the Altmetric score, the majority of sources experienced increases in mentions from 2013 to 2016 with the exception of policy sources, Wikipedia, patents, and research highlight platforms.
#
Discussion
To our knowledge, this is the first study in the ophthalmology literature to compare Altmetrics with traditional bibliometrics. While journal impact factor and article citation count showed positive correlations for both the 2013 and 2016 cohorts, Altmetric score only showed a positive correlation with article citation count and journal impact factor for the 2016 cohort. Given that the combined Altmetric score for all articles was more than four times higher in 2016 than 2013 (6,638 and 1,495 respectively), we can infer an increase in use of social media for publication dissemination among the ophthalmology literature. As this change is seen across only two snapshots in time, further research is needed to better characterize this trend and help elucidate the underlying causes. One explanation may include society-wide increases in social media participation or journal integration with social media between 2013 and 2016.[8]
Twitter mentions were by far the most common type of social media involvement in both the 2013 and 2016 cohorts. However, age of journal Twitter account showed no significant correlation with the relationship of Altmetric score and citation count for either 2013 or 2016. Given that much of the content on Twitter is generated by individual users, it is not particularly surprising that article Twitter mentions did not correlate with journal account age. Unfortunately, this does not shed any light on what factors other than journal Twitter account age, if any, correlate with a higher level of mentions on Twitter.
When analyzed individually, only three journals from 2013 (American Journal of Ophthalmology, Retina-The Journal of Retinal and Vitreous Disease, and British Journal of Ophthalmology) and two journals from 2016 (Ophthalmology and Experimental Eye Research) had significant correlations between Altmetric scores and article citation number. Thus, in spite of higher overall Altmetric scores in 2016, and a significant correlation between Altmetric score and both citation count and impact factor in 2016 but not in 2013, fewer journals showed a significant correlation between Altmetric score and article citation number. This may be due to several factors including fewer citations accrued for the 2016 articles. It certainly seems to suggest that no journal studied has managed to consistently get attention via alternative sources in an identical manner to the journal's citations via traditional avenues. For this reason, Altmetric scores and traditional bibliometrics at this time are not tightly correlated across the spectrum of individual journals in the ophthalmology literature.
Comparison with similar studies conducted in other fields including the general surgery,[11] pediatric surgery,[12] dental implantology,[13] and urology literature[14] provides some illuminating conclusions. The analyses of the general surgery, pediatric surgery, and urology literature all demonstrated significant correlations between Altmetric score and either article citation count alone or both article citation count and impact factor. The analysis of dental implantology literature showed no correlation between Altmetric score and citation count or journal impact factor. Only the pediatric surgery literature analysis showed an increased correlation between Altmetric score and article citation number with increasing age of journal Twitter accounts. Taken together, these findings support the conclusion that different medical subspecialties exhibit different patterns of use and uptake of the types of social media investigated by Altmetric score.
Upon examination of the Altmetric data for 2013 and 2016, several notable changes are present. The most drastic change is in total Twitter mentions, increasing from 284 in 2013 to 1,608 in 2016, an increase of 466.2%. Of note, news outlet mentions made the second biggest increase, from 121 to 658 (443.8%). Facebook mentions likewise showed an increase, but a comparatively more subdued one from 64 in 2013 to 204 in 2016 (218.8%). Notably, mentions by policy sources, Wikipedia, and patent applications all showed marked decreases in number of mentions from the 2013 to 2016 cohorts. It has been noted that there exists variation among alternative metric collection programs, with Altmetric reporting the most news sources, blogs, and tweets, but missing some Mendeley readers (as compared with PlumX) and missing some Wikipedia mentions (as compared with CED).[15] However, increasing prevalence and penetrance of social media use is at least partly responsible for the increases in Twitter and Facebook mentions.
We recognize the limitations of our study. First, it is important to point out that the conclusions are drawn based on an intentionally limited sample size. Nevertheless, we believe that the sample is representative of the most widely read scientific ophthalmic journals. Second, only the most highly cited studies from the highest impact factor journals were included, so the conclusions of our study may not be broadly generalizable to the remainder of the ophthalmology literature. On that note, our sample size is also limited to ophthalmology literature, and cannot be used to draw definitive conclusions about trends within medical or scientific literature more broadly. Third, because our data are only obtained from ophthalmology journals, they do not take into account ophthalmology articles published in nonophthalmology journals like New England Journal of Medicine or Nature. Fourth, in choosing to compare two cohorts 3 years apart and 4 or 7 years old at the time of analysis, we intended to give citation numbers sufficient time to mature, but correlations between variables may have been different if different years were chosen.[4] While we were unable to find a correlation between Twitter account age and Altmetric scores, other metrics like Twitter account activity may be better suited to this purpose. Fifth, there are limitations inherent to the metrics used. While Altmetrics tend to level off fairly quickly after article publication, citations can take much longer to mature, with significant differences seen in cohorts of articles examined before and after 5 years of age.[4] Unfortunately, the inherent subjectivity in any rating mechanism means that any method will not perfectly capture what it sets out to capture. When we accept that neither Altmetrics nor bibliometrics perfectly measure “impact,” we can see that comparison of the two will be limited by the degree of imprecision inherent in each methodology. For example, questions remain regarding how to best measure social media impact. Number of total tweets about an article varies in significance depending on how many are retweets and how many individual users are in the conversation. Different Altmetric aggregators (Altmetric, CrossRef, Plum, Mendeley) vary in ability to track data, and in weight given to different types of data for determining total score.[15] [16] Finally, being merely descriptive and not experimental, our study format does not allow us to answer questions regarding underlying trends and forces shaping our results.
#
Conclusions
While journal impact factor and article citation count showed positive correlations for both the 2013 and 2016 cohorts, Altmetric score only showed a positive correlation with article citation count and journal impact factor for the 2016 cohort. We hypothesize that some possible causes are greater uptake of social media among readers of ophthalmology literature, primarily driven by a greater number of mentions on Twitter, and further supported by a higher combined Altmetric score for the 2016 cohort. When compared with similar projects done in other medical specialties, our results differed in that we found a statistically significant correlation only for the 2016 cohort. Reasons for this interspecialty heterogeneity are unclear and may warrant further investigation.
The question remains: how do Altmetrics compare with traditional markers of publication impact? A complete answer to that question will take time as the role of social media in society in general and in the field of ophthalmology evolves. It will additionally require more in-depth research to further illuminate the details of the challenging concept of article quality, and how Altmetrics and bibliometrics attempt to approximate this concept. Although traditional bibliometrics will likely remain the main measure of journal and article impact within ophthalmology, Altmetrics are becoming an important complementary marker of article influence in society more generally. Authors and academic institutions should recognize the increasing prevalence of alternative metrics of article reach and adjust promotions committee decisions and compensation structures accordingly.
#
#
Conflicts of Interest
None.
-
References
- 1 Durieux V, Gevenois PA. Bibliometric indicators: quality measurements of scientific publication. Radiology 2010; 255 (02) 342-351
- 2 Powell AGMT, Bevan V, Brown C, Lewis WG. Altmetric versus bibliometric perspective regarding publication impact and force. World J Surg 2018; 42 (09) 2745-2756
- 3 Bornmann L, Leydesdorff L. Scientometrics in a changing research landscape: bibliometrics has become an integral part of research quality evaluation and has been changing the practice of research. EMBO Rep 2014; 15 (12) 1228-1232
- 4 Wang J. Citation time window choice for research impact evaluation. Scientometrics 2013; 94: 851-872
- 5 Konkiel S. Altmetrics: diversifying the understanding of influential scholarship. Palgrave Commun 2016; 2: 16057
- 6 O'Connor EM, Nason GJ, O'Kelly F, Manecksha RP, Loeb S. Newsworthiness vs scientific impact: are the most highly cited urology papers the most widely disseminated in the media?. BJU Int 2017; 120 (03) 441-454
- 7 Altmetric website. https://www.altmetric.com . Accessed February 13, 2021
- 8 Social MFS. Pew Research Center. https://www.pewresearch.org/internet/fact-sheet/social-media/ . Published June 12, 2019. Accessed February 13, 2021
-
9
Source S.
https://www.scopus.com/sources . Data gathered December 10, 2019. Accessed February 13, 2021
- 10 How are outputs scored? Altmetric.com. https://help.altmetric.com/support/solutions/articles/6000232839-how-are-outputs-scored- . Updated September 17, 2020. Accessed February 13, 2021
- 11 Mullins CH, Boyd CJ, Corey BL. Examining the correlation between Altmetric score and citations in the general surgery literature. J Surg Res 2020; 248: 159-164
- 12 Chang J, Desai N, Gosain A. Correlation between Altmetric score and citations in pediatric surgery core journals. J Surg Res 2019; 243 (243) 52-58
- 13 Warren VT, Patel B, Boyd CJ. Analyzing the relationship between Altmetric score and literature citations in the implantology literature. Clin Implant Dent Relat Res 2020; 22 (01) 54-58
- 14 Nocera AP, Boyd CJ, Boudreau H, Hakim O, Rais-Bahrami S. Examining the correlation between Altmetric score and citations in the urology literature. Urology 2019; 134: 45-50
- 15 Ortega JL. Reliability and Accuracy of Altmetric Providers: A Comparison Among Altmetric, PlumX and Crossref Event Data. 2017 https://doi.org/10.13140/RG.2.2.32553.90720
- 16 Zahedi Z, Costas R. General discussion of data quality challenges in social media metrics: extensive comparison of four major Altmetric data aggregators. PLoS One 2018; 13 (05) e0197326
Address for correspondence
Publication History
Received: 21 October 2020
Accepted: 22 February 2021
Article published online:
30 June 2021
© 2021. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)
Thieme Medical Publishers, Inc.
333 Seventh Avenue, 18th Floor, New York, NY 10001, USA
-
References
- 1 Durieux V, Gevenois PA. Bibliometric indicators: quality measurements of scientific publication. Radiology 2010; 255 (02) 342-351
- 2 Powell AGMT, Bevan V, Brown C, Lewis WG. Altmetric versus bibliometric perspective regarding publication impact and force. World J Surg 2018; 42 (09) 2745-2756
- 3 Bornmann L, Leydesdorff L. Scientometrics in a changing research landscape: bibliometrics has become an integral part of research quality evaluation and has been changing the practice of research. EMBO Rep 2014; 15 (12) 1228-1232
- 4 Wang J. Citation time window choice for research impact evaluation. Scientometrics 2013; 94: 851-872
- 5 Konkiel S. Altmetrics: diversifying the understanding of influential scholarship. Palgrave Commun 2016; 2: 16057
- 6 O'Connor EM, Nason GJ, O'Kelly F, Manecksha RP, Loeb S. Newsworthiness vs scientific impact: are the most highly cited urology papers the most widely disseminated in the media?. BJU Int 2017; 120 (03) 441-454
- 7 Altmetric website. https://www.altmetric.com . Accessed February 13, 2021
- 8 Social MFS. Pew Research Center. https://www.pewresearch.org/internet/fact-sheet/social-media/ . Published June 12, 2019. Accessed February 13, 2021
-
9
Source S.
https://www.scopus.com/sources . Data gathered December 10, 2019. Accessed February 13, 2021
- 10 How are outputs scored? Altmetric.com. https://help.altmetric.com/support/solutions/articles/6000232839-how-are-outputs-scored- . Updated September 17, 2020. Accessed February 13, 2021
- 11 Mullins CH, Boyd CJ, Corey BL. Examining the correlation between Altmetric score and citations in the general surgery literature. J Surg Res 2020; 248: 159-164
- 12 Chang J, Desai N, Gosain A. Correlation between Altmetric score and citations in pediatric surgery core journals. J Surg Res 2019; 243 (243) 52-58
- 13 Warren VT, Patel B, Boyd CJ. Analyzing the relationship between Altmetric score and literature citations in the implantology literature. Clin Implant Dent Relat Res 2020; 22 (01) 54-58
- 14 Nocera AP, Boyd CJ, Boudreau H, Hakim O, Rais-Bahrami S. Examining the correlation between Altmetric score and citations in the urology literature. Urology 2019; 134: 45-50
- 15 Ortega JL. Reliability and Accuracy of Altmetric Providers: A Comparison Among Altmetric, PlumX and Crossref Event Data. 2017 https://doi.org/10.13140/RG.2.2.32553.90720
- 16 Zahedi Z, Costas R. General discussion of data quality challenges in social media metrics: extensive comparison of four major Altmetric data aggregators. PLoS One 2018; 13 (05) e0197326