Friday 3 April 2015

Study analyzes the use of social networks in the assessment of scientific impact | SciELO in Perspective

Source: http://blog.scielo.org/en/2015/03/13/study-analyzes-the-use-of-social-networks-in-the-assessment-of-scientific-impact/

Study analyzes the use of social networks in the assessment of scientific
impact

March 13, 2015 4:00 pm , Leave
a Comment , SciELO

Photo: Jason
Howie.

In recent years, the use of social networks in science communication has
been increasing on a large scale, and specific platforms have been created for
interaction and information sharing among researchers. Despite the increasing
interest of the academic community on social networks as a tool for scientific
communication, little is known about the usage profile of these tools, and how
traditional measures of scientific impact based on citations (offline indexes,
impact off-line) correlate with the new impact measure (online index, impact
online).

A paper presented at the 47^th Hawaii International Conference on
System Sciences (HICSS) in 2014¹ by researchers from the University
of St. Gallen in Switzerland examined whether and how scientific impact can be
measured through social media data analysis, and how this approach is related
to the traditional metrics. According to the authors, network centrality
measures² based on analysis of social media had not been considered
in the context of the impact assessment. The results of the exploratory work,
performed at a single institution with a small number of researchers, indicate
that these measures correlate with traditional impact metrics and can be used
to complement them.

Social network sites (SNS) are defined as “Web-based services that allow
individuals to (1) construct a public or semi-public profile within a bounded
system; (2) articulate a list of other users with whom they share a connection,
and (3) view and traverse their list of connections and those made by others
within the system”. Examples of SNS are Academia.edu, ResearchGate, and
Mendeley since their sites are directed to the scientific community and, in
addition to the aforementioned functions, they also allow uploading and sharing
articles, endorse the work of colleagues or find related literature.

Traditionally, scientific impact is measured by bibliographic metrics such
as publications and citations in refereed journals. Among these are the Impact
Factor (Web of Knowledge), the Scimago Journal Rank (Scopus, Elsevier), and the
h-Index, the first two focusing on publication and the last on the individual
researcher. These metrics incorporate numerous flaws and limitations, as has
been widely debated in the scientific community, including in this blog.
Citations measures effectively reflect the value of the work and its ability to
pass through the editorial process and peer review. However, the impact of a
publication also refers to the degree of influence it exerts and in this case,
citations are only part of the measure of this influence among the scientific
community and society. In certain disciplines, other forms of publication as
books (in arts and humanities), reports and technical manuals (engineering),
presentations and conference proceedings (mathematics, computer science)
outweigh the journal articles, but are not detected by traditional
bibliometrics. In addition, these metrics foster self-citation culture and
citation cartels, neglecting its context, which is, how and why certain
articles are cited.

Despite the limitations, the traditional metrics have advantages, too: they
allow comparisons between journals, disciplines and institutions and are easy
to calculate and understand. However, it is agreed that scientific impact
cannot be measured only by citations. The authors demonstrate through this
study that the emergence of the Internet as a space for scientific
communication complements the analysis of scientific impact, enriching and
diversifying its assessment.

There are currently under development alternative scientific impact metrics
based on social media such as, for example, Altmetric, which appears very
promising. Social networks store a lot of information and promote linkages
between researcher communities and general stakeholders. The analysis of these
interactions allows us to evaluate the impact of publications in a broader
aspect than traditional metrics. Moreover, altmetrics and webometrics can be
applied at the article, journal or individual researcher levels. Altmetric was
adopted by SciELO Brazil to monitor the performance of articles on social networks.

The Altmetrics Manifesto makes a compilation of the goals and scope of this
initiative. Its authors define impact as being formed by four pillars: use
(access and download); peer-review (experts’ opinion); citations; and the
altmetrics component (storage, links, bookmarks and sharing). The main
advantage, they say, is the celerity with which an article is assessed by means
of social media. Rather than waiting two years or more to count citations, in
only one week sharing via Twitter, Facebook, LinkedIn and other SNS may provide
the specific impact of an article, and not of the journal where it was
published.

The study by Hoffmann and colleagues included 55 scholars of a public
administration school at the University of St. Gallen, in Switzerland. The authors
gained access to the ResearchGate profile of the researchers of this
institution between September 2012 and February 2013 and evaluated three
indicators in order to analyze the relationship between online (social media)
and offline scientific impact (citations):

Seniority: formal position,
honors, awards, participation in editorial boards.
Impact of publications:
classical bibliometric measures as h-Index, journal Impact Factor, besides
online impact on SNS, such as shares, downloads and bookmarks.
Centralization of the
network: measure the extent that a researcher is connected to other
members of the scientific community, an indication of its prominence and
influence.

The results can be summarized as follows:

When using ResearchGate to
networking, researchers tend to follow more colleagues at their own
institution rather than establishing new contacts;
In scholars networks,
assistant professors occupy more central positions (i.e., establishing
connections with more colleagues), followed by full professors, doctors
and post-docs;
The online activity (online
communication) of scholars strongly correlates with measures of
centrality, but not with offline impact or seniority;
Online and offline impact
measures are strongly related. However, while off-line measures are
related to seniority, online impact is related to the centrality of
positions that actors assume in the network.
Seniority is correlated with
measures of centrality, i.e. the offline social capital is also expressed
by the online universe.

Performance evaluation of scientific research, institutions and journals
until recently relied almost exclusively on bibliometric measures, mainly
citations. Insofar as new technology tools have emerged, alternative metrics
have been proposed to assess scientific impact. This study points out the
limitations of traditional metrics such as disregarding aspects of the
interactions between the actors of the networks, and the formation of social
capital for individual members. As social media facilitate the analysis of the
researchers’ networking, one would expect that measures of centrality of the
network could provide data on evaluation of scientific impact.

This study led to the conclusion that the scholars at the Swiss University
use social networks more in the Facebook than in the Twitter approach, i.e.,
they do not follow many of their colleagues, preferring to interact with their
offline contact community, such as colleagues from the same institution, or
collaborators. Thus, social networks are intended to corroborate, rather than
to establish contacts.

As expected, scholars at the beginning of their careers – and therefore
younger researchers – occupy the central positions of social networks and are
more active in online communities, possibly because of their career projection
desire and to establish strong collaborative networks, which may result in a
greater impact in the future. Regarding the impact of publications, the
almetrics score correlates with seniority, rather than with traditional offline
metrics as h-Index.

This study contributes to the debate – just started – about scientific
impact assessment and altmetrics. In this small sample, online impact
correlates better with seniority than with offline impact, and online impact is
closely related to the centrality of the network. Thus, social networks based
metrics could help to elucidate the dynamics of online and offline scientific
impact measures.

According to the manifesto’s authors “Altmetrics are in their early stages;
many questions are unanswered. But given the crisis facing existing filters and
the rapid evolution of scholarly communication, the speed, richness, and
breadth of altmetrics make them worth investing in” (PRIEM et al 2010).

Notes

¹HOFFMANN, C.P., LUTZ, C., and MECKEL, M. Impact Factor 2.0:
Applying Social Network Analysis to Scientific Impact Assessment. In: 47th
Hawaii International Conference on System Science, Hilton Waikoloa Village,
2014. DOI: 10.1109/HICSS.2014.202

² Centrality, in network analysis, refers to indicators which
identify the most important vertices within a graph. Applications include
identifying the most influential person(s) in a social network, key
infrastructure nodes in the Internet or urban networks, among others.

Source: Centrality. Wikipedia. [viewed 26 January 2015]. Available
from: http://en.wikipedia.org/wiki/Centrality

External links

Academia.edu – <http://www.academia.edu>

Altmetric – <http://www.altmetric.com/>

Mendeley – <http://www.mendeley.com/>

ResearchGate – <http://www.researchgate.net/>

References

Altmetrics, Alternative metrics and Alternative measurements: new
perspectives on the visibility and impact of scientific research. SciELO
in Perspective. [viewed 31 January 2015]. Available from: http://blog.scielo.org/en/2013/08/14/altmetrics-alternative-metrics-and-alternative-measurements-new-perspectives-on-the-visibility-and-impact-of-scientific-research/

Article downloads: An alternative indicator of national research impact
and cross-sector knowledge exchange – Originally published on the Elsevier
newsletter “Research Trends Issue 36″. SciELO in Perspective. [viewed 26
January 2015]. Available from: http://blog.scielo.org/en/2014/03/24/article-downloads-an-alternative-indicator-of-national-research-impact-and-cross-sector-knowledge-exchange-originally-published-on-the-elsevier-newsletter-reseatch-trends-issue-36/

BIK, H.M, and GOLSTEIN, M.C. An Introduction to Social Media for
Scientists. PLoS Biol. 2013, vol. 11, nº 4.
DOI: 10.1371/journal.pbio.1001535

BOYD, D., and ELLISON, N.B. Social Network Sites: Definition,
History, and Scholarship. Journal of Computer-Mediated Communication. 2007, vol. 13, nº 1, pp. 210–230. DOI: 10.1111/j.1083-6101.2007.00393.x

Declaration recommends eliminate the use of Impact factor for research
evaluation. SciELO in Perspective. [viewed 23 January 2015]. Available
from: http://blog.scielo.org/en/2013/07/16/declaration-recommends-eliminate-the-use-of-impact-factor-for-research-evaluation/

HOFFMANN, C.P., LUTZ, C., and MECKEL, M. Impact Factor 2.0:
Applying Social Network Analysis to Scientific Impact Assessment. In: 47th
Hawaii International Conference on System Science, Hilton Waikoloa Village,
2014. DOI: 10.1109/HICSS.2014.202

Interview with Euan Adie, CEO of altmetric.com. SciELO in
Perspective. [viewed 31 January 2015]. Available from: http://blog.scielo.org/en/2013/08/29/interview-with-euan-adie-ceo-of-altmetric-com/

Interview with Vincent Larivière. SciELO in Perspective. [viewed 26
January 2015]. Available from: http://blog.scielo.org/en/2013/08/16/interview-with-vincent-lariviere/

Paper investigates: is your most cited work your best work?. SciELO
in Perspective. [viewed 26 January 2015]. Available from: http://blog.scielo.org/en/2014/11/24/paper-investigates-is-your-most-cited-work-your-best-work/

PRIEM, J. Scholarship: Beyond the paper. Nature. 2013, vol. 495, nº 7442, pp. 437–440. DOI: 10.1038/495437a

PRIEM, J., and et al. Altmetrics: a manifesto. Altmetrics.org,
2010, pp. 1–5. Available from: http://altmetrics.org/manifesto/

Rise of the Rest: The Growing Impact of Non-Elite Journals – Originally
published on Google Scholar Blog on October 8, 2014. SciELO in
Perspective. [viewed 26 January 2015]. Available from: http://blog.scielo.org/en/2014/10/13/rise-of-the-rest-the-growing-impact-of-non-elite-journals-originally-published-on-google-scholar-blog-on-october-8-2014/

Study proposes a taxonomy of motives to cite articles in scientific
publications. SciELO in Perspective. [viewed 26 January 2015]. Available
from: http://blog.scielo.org/en/2014/11/07/study-proposes-a-taxonomy-of-motives-to-cite-articles-in-scientific-publications/

What can alternative metrics – or altmetrics – offer us?. SciELO in
Perspective. [viewed 31 January 2015]. Available from: http://blog.scielo.org/en/2014/08/07/what-can-alternative-metrics-or-altmetrics-offer-us/

How to cite this post [ISO
690/2010]:

Study analyzes the use of social networks in the
assessment of scientific impact. SciELO in Perspective. [viewed 03 April
2015]. Available from:
http://blog.scielo.org/en/2015/03/13/study-analyzes-the-use-of-social-networks-in-the-assessment-of-scientific-impact/