Source: http://blog.scholasticahq.com/post/112788830243/data-is-king-tracking-internal-performance#.VTrlT5OgssS
If you’re like most editors, you’re always looking for new ways to
optimize your journal’s peer review process. Of course, in order to know
why bottlenecks are occurring in your workflow and come up with
solutions to stop them you have to figure out when and where they’re
happening first.
Many journals have begun to focus on tracking journal metrics to get a
granular view of their peer review processes - from simple stats like
average annual submission rate, to the average number of days it takes
individual editors to send authors manuscript decisions. Tracking
metrics can help journals stay abreast of how they are performing
externally in terms of volume, quality, and scope of submissions, and
internally in terms of the speed of journal-wide and editor and reviewer
specific performance.
“The big saying out there right now is data is king,” said Christine
Dymek, senior managing editor at leading journal management consultancy
Kaufman Wills Fusting & Company.
According to Dymek, who consults journal editors on peer review best
practices, all journals should produce analytics reports to go over
during regular team meetings.
“I think that it’s absolutely something journals need to do and audit
annually, if not bi-annually,” said Dymek. “It’s a good way to check
progress to see where you stand and to set goals.”
In Academic Journal Management Best Practices: Tales from the Trenches,
a recent Scholastica eBook, Dymek exaplains the core metrics she
encourages journals to track. Below we roundup those main metrics. For
editors using journal management software with built-in analytics, you
will likely be able to see all of this information from your account. If
you are not using journal management software, Dymek said not be
overwhelmed by having to manually track multiple stats. She encourages
journals to start out small choosing one or two metrics that matter most
to them.
track and look to improve is the average amount of time it takes them to
make decisions on submissions, from the time a manuscript is first
received. It’s important for journals to ensure that their peer review
process is moving forward and that they aren’t accruing backlogs of
submissions that need to be assigned to editors or that editors need to
assign to reviewers. Dymek encourages the journals she works with to
agree on a benchmark for the number of days in which decisions should be
made on new submissions, so that editors have a shared goal to work
towards. Once that benchmark is in place, journals can determine if
they’re on or missing their mark by tracking their average time to
decision on a bi-annual or annual basis.
In addition to tracking journal-wide time to decision, Dymek said
journals could also benefit from tracking how long it is taking their
individual editors to move manuscripts through peer review, to determine
if everyone is working at the same pace or if one editor is struggling
to keep up with the manuscripts assigned to him.
“There’s a lot of benefit to that,” said Dymek. “You can take a look
at who your high and low performing editors are and ask - why are the
low performing editors moving at a slower rate?” Dymek said
editor-specific time to decision metrics can quickly reveal gaps in
editor support that need to be addressed, such as insufficient software
training, or simply whether or not journals need to modify their
manuscript assignment process to account for editors with too much on
their plate.
submissions they accept and reject on a bi-annual or annual basis and
where rejection decisions are being made in their peer review process.
In her experience, Dymek said she’s seen that many journals can benefit
from assessing their average number of desk rejections in particular and
what those numbers say about their process.
“Desk rejects is a really important topic now,” she said. “The big
talk in academic publishing has been the increased strain on reviewers.
You want to make sure you’re only sending reviewers submissions that
have potential.”
Dymek said, on one hand, journals should make sure they aren’t making
too many desk rejections. If you notice a rise in desk rejections over
time, you may want to meet with your editors to discuss the change and
whether or not you’re putting enough manuscripts through peer review.
However, on the other hand, if you find that a high number of
manuscripts are being rejected during first-round peer review you may
need to consider whether your editors are screening manuscripts well
enough before sending them out to reviewers.
Your journal’s acceptance rate can also reveal a lot about your peer
review process and the quality of your submissions. If you find that you
have a very high acceptance rate or that your acceptance rate is
growing, it’s a good idea to compare acceptance rate by submissions rate
every six months or year to see if a decline in submissions has caused
your editors to begin accepting more manuscripts than usual. If that’s
the case, you should look to acquire more quality submissions to ensure
that your journal remains selective.
ensuring that they have enough peer reviewers to reach out to and that
their reviewers are completing assignments in a timely manner.
“It’s very important to make sure that the reviewer pool you have is
accurate and up-to-date, and that you only have people in it who
actually want to review,” said Dymek.
To gauge the quality of their reviewer database, she encourages journals to track how often reviewers decline review requests.
“If you have someone that’s been in a reviewer database for three
years and is yet to accept an assignment, you need to step back and ask
if it is really necessary to have that person in the database,” she
said.
Dymek said tracking the amount of time it takes each of your
reviewers to complete manuscript assignments is another indicator of
whether or not you are reaching out to the right people. You may find
that your reviewers continue to accept assignments, but that their
average time to decision is growing. If that’s the case, it may be a
sign that it’s time to find new reviewers and to give your go-to
referees a needed break.
To proactively ensure that your journal is not burning out reviewers,
Dymek said it’s also important to keep track of how often you are
assigning manuscripts to specific reviewers. She advises journals to
continually seek new reviewers so that they can alternate the people
they reach out to often.
Scholastica Blog — Data is King: Tracking Internal Performance...
Data is King: Tracking Internal Performance Metrics at Your Journal
If you’re like most editors, you’re always looking for new ways to
optimize your journal’s peer review process. Of course, in order to know
why bottlenecks are occurring in your workflow and come up with
solutions to stop them you have to figure out when and where they’re
happening first.
Many journals have begun to focus on tracking journal metrics to get a
granular view of their peer review processes - from simple stats like
average annual submission rate, to the average number of days it takes
individual editors to send authors manuscript decisions. Tracking
metrics can help journals stay abreast of how they are performing
externally in terms of volume, quality, and scope of submissions, and
internally in terms of the speed of journal-wide and editor and reviewer
specific performance.
“The big saying out there right now is data is king,” said Christine
Dymek, senior managing editor at leading journal management consultancy
Kaufman Wills Fusting & Company.
According to Dymek, who consults journal editors on peer review best
practices, all journals should produce analytics reports to go over
during regular team meetings.
“I think that it’s absolutely something journals need to do and audit
annually, if not bi-annually,” said Dymek. “It’s a good way to check
progress to see where you stand and to set goals.”
In Academic Journal Management Best Practices: Tales from the Trenches,
a recent Scholastica eBook, Dymek exaplains the core metrics she
encourages journals to track. Below we roundup those main metrics. For
editors using journal management software with built-in analytics, you
will likely be able to see all of this information from your account. If
you are not using journal management software, Dymek said not be
overwhelmed by having to manually track multiple stats. She encourages
journals to start out small choosing one or two metrics that matter most
to them.
Time to Manuscript Decision
According to Christine Dymek, one metric that all journals shouldtrack and look to improve is the average amount of time it takes them to
make decisions on submissions, from the time a manuscript is first
received. It’s important for journals to ensure that their peer review
process is moving forward and that they aren’t accruing backlogs of
submissions that need to be assigned to editors or that editors need to
assign to reviewers. Dymek encourages the journals she works with to
agree on a benchmark for the number of days in which decisions should be
made on new submissions, so that editors have a shared goal to work
towards. Once that benchmark is in place, journals can determine if
they’re on or missing their mark by tracking their average time to
decision on a bi-annual or annual basis.
In addition to tracking journal-wide time to decision, Dymek said
journals could also benefit from tracking how long it is taking their
individual editors to move manuscripts through peer review, to determine
if everyone is working at the same pace or if one editor is struggling
to keep up with the manuscripts assigned to him.
“There’s a lot of benefit to that,” said Dymek. “You can take a look
at who your high and low performing editors are and ask - why are the
low performing editors moving at a slower rate?” Dymek said
editor-specific time to decision metrics can quickly reveal gaps in
editor support that need to be addressed, such as insufficient software
training, or simply whether or not journals need to modify their
manuscript assignment process to account for editors with too much on
their plate.
Acceptance and Rejection Rate
Another core metric journals can track is the average number ofsubmissions they accept and reject on a bi-annual or annual basis and
where rejection decisions are being made in their peer review process.
In her experience, Dymek said she’s seen that many journals can benefit
from assessing their average number of desk rejections in particular and
what those numbers say about their process.
“Desk rejects is a really important topic now,” she said. “The big
talk in academic publishing has been the increased strain on reviewers.
You want to make sure you’re only sending reviewers submissions that
have potential.”
Dymek said, on one hand, journals should make sure they aren’t making
too many desk rejections. If you notice a rise in desk rejections over
time, you may want to meet with your editors to discuss the change and
whether or not you’re putting enough manuscripts through peer review.
However, on the other hand, if you find that a high number of
manuscripts are being rejected during first-round peer review you may
need to consider whether your editors are screening manuscripts well
enough before sending them out to reviewers.
Your journal’s acceptance rate can also reveal a lot about your peer
review process and the quality of your submissions. If you find that you
have a very high acceptance rate or that your acceptance rate is
growing, it’s a good idea to compare acceptance rate by submissions rate
every six months or year to see if a decline in submissions has caused
your editors to begin accepting more manuscripts than usual. If that’s
the case, you should look to acquire more quality submissions to ensure
that your journal remains selective.
Manuscripts Per Reviewer and Average Time to Review
One of the hardest parts of peer review for all journal editors isensuring that they have enough peer reviewers to reach out to and that
their reviewers are completing assignments in a timely manner.
“It’s very important to make sure that the reviewer pool you have is
accurate and up-to-date, and that you only have people in it who
actually want to review,” said Dymek.
To gauge the quality of their reviewer database, she encourages journals to track how often reviewers decline review requests.
“If you have someone that’s been in a reviewer database for three
years and is yet to accept an assignment, you need to step back and ask
if it is really necessary to have that person in the database,” she
said.
Dymek said tracking the amount of time it takes each of your
reviewers to complete manuscript assignments is another indicator of
whether or not you are reaching out to the right people. You may find
that your reviewers continue to accept assignments, but that their
average time to decision is growing. If that’s the case, it may be a
sign that it’s time to find new reviewers and to give your go-to
referees a needed break.
To proactively ensure that your journal is not burning out reviewers,
Dymek said it’s also important to keep track of how often you are
assigning manuscripts to specific reviewers. She advises journals to
continually seek new reviewers so that they can alternate the people
they reach out to often.
Scholastica Blog — Data is King: Tracking Internal Performance...
No comments:
Post a Comment