Repository navigation
Reviewers List #78
Description
I would like to propose to either make the reviewers list private (or at least not on the website on the page it currently is) and/or to make all reviewers add themselves, perhaps?
Most of the people I ask to review and who do a very good job (of course) are not on the list and thus do not get the same level of recognition. Having a public list without having it updated doesn't seem to add much except to say "look at all these people"? Or am I missing something?
Activity
khinsen commentedon May 13, 2020
The list of reviewers on the Web site is more precisely the list of people who have volunteered to be on that list. It's mainly of interest to editors looking for a suitable reviewer. We could probably easily (meaning automatically) compile a list of people who actually reviewed ReScience papers from the article metadata.
I agree that the current presentation is a bit misleading. I don't think we should make anything private, and we don't even have any private space for ReScience, but we can probably do a better job with the presentation.
oliviaguest commentedon May 13, 2020
Great. So let's move to maybe two lists: a list with active reviewers and a list with people who volunteer(ed)?
rougier commentedon May 14, 2020
The idea of the public list of reviewers was to give some recognition to reviewers. I used to add stars behind each name that point to the review they made but in the end it was too much work. Also, once people have registered, it's hard to know if they are reactive or not. For some of them, we do not even have an email address and the only wait to notify them is through the GitHub interface. If they don't receive such notifications through mail and do not connect to GitHub on a regular basis, they won't receive anything.
I agree for the two lists but I'm not sure how to display them.
oliviaguest commentedon May 14, 2020
I think ideally we should have a list of who has reviewed on the website. We can also have a "private" list (or not and have it on website too?) of who can review — by private I mean one that can just be a .csv on GitHub and doesn't need to be public-facing on the website. Just an idea, but either way: I think it would be nice to do the former list.
rougier commentedon May 15, 2020
I agree and if I remember well, @ctb initiated such a list in .csv format and of course, I don't remember where it is now. But most probably it is outdated.
oliviaguest commentedon May 15, 2020
@khinsen how easy is it to make the "people who have reviewed" list? can I help?
khinsen commentedon May 16, 2020
I can't really say how easy it is because I have no experience with Web scraping, but here's an outline of what it takes:
article.mdfiles for the reviewer names.article.yamlfor reviewer names.oliviaguest commentedon May 16, 2020
Oh, I thought you were suggesting using the GitHub API. I misunderstood:
What do we want to do that's a bit less time-/working memory-demanding? Because I can't offer anything that involved — I don't have time to write a scraper, right now (sorry).
Do we want to take the current list offline from the website itself? I personally believe yes, because I think it disincentivises people (in the current set-up) from reviewing who think it's an in-group thing only (based on private communications).
khinsen commentedon May 18, 2020
Maybe there is a GitHub API that would make this easier. I know nothing about GitHub APIs!
As for making the reviewer less prominent (but keeping it public in one of our GitHub repositories), I agree. Pinging @rougier @benoit-girard for their opinion!
oliviaguest commentedon May 18, 2020
NBD but I assume the GitHub API is how bots like @whedon — see: docs for all the features, if you haven't already — work.
ctb commentedon May 18, 2020
I don't remember what I did back when with respect to a list, but I do have GitHub API experience - see http://ivory.idyll.org/blog/2019-github-project-reporting.html
@oliviaguest I would be happy to whip up a Python approach, if it suits, and then iterate with you. I'm not very fast at the iterations tho so if this is time sensitive I might not be a good choice.
ctb commentedon May 18, 2020
(by "iterate" I mean ask for ~5 minutes of your attention periodically to go, oh yeah, that's useful, vs what the heck is that? gut checks, not serious time investments.)
oliviaguest commentedon May 18, 2020
Sure — thanks. BTW I think if so it should be all EiCs involved?
khinsen commentedon May 18, 2020
Just use this issue for follow-ups! And thanks @ctb for offering to help!
ctb commentedon May 20, 2020
digging into it. Some notes --
Inclusive (but still some FP) list of article md files from archives is:
articles-list.txt
ctb commentedon May 21, 2020
Reviewers for old stuff, from ReScience-Archives -
reviewers-archives.txt
ctb commentedon May 23, 2020
for
I find the following reviewers:
there are a bunch of None in here, looks like reviewers aren't always indicated in the article.yaml.
khinsen commentedon May 23, 2020
Normally we have two reviewers, so there are two reviewer slots in the metadata file. For the ten-year challenge we decided to simplify to a single reviewer, and then the second slot is None. I hope that's the only explanation required for those Nones!
ctb commentedon May 23, 2020
no reviewers found for:
"ReScience (R)evolution" - 10.5281_zenodo.3069619/article.yaml
"ICLR Reproducibility Challenge 2019" - 10.5281_zenodo.3158244/article.yaml
which seems ok?
There are also several anonymous reviewers.
ctb commentedon May 23, 2020
SO. Where do we want to take this? Do we want to find github IDs for reviewers? Is there other metadata parsing and indexing foo that should be done? Are people interested in the code? (It's pretty simple stuff, now that I've got it worked out :)
oliviaguest commentedon May 23, 2020
I think for me the answer to this question is probably what will answer your question:
rougier commentedon May 25, 2020
T oasnwer @ctb question, editorial and article for the Ten Years Reproducibility Challenge have only 1 reveiwer. For the NeurIPS challenge, reviewers are anonymous (through open review).
ctb commentedon May 28, 2020
When we are speaking of "the web site" list of reviewers, this is what that refers to, right? https://github.com/rescience/rescience.github.io/blob/sources/04-board.md
khinsen commentedon May 28, 2020
@ctb Yes, that's all there is for now.
karmendra8386 commentedon Oct 7, 2026
I would like to volunteer as a reviewer. Karmendra Pandey (https://github.com/karmendra8386) — Machine Learning, Artificial Intelligence — Python. ORCID: 0009-0000-3461-9889. Practice Architect in AI & ML at TEKsystems (19+ years); 27 published peer reviews on PREreview with DOIs; open-source contributor (Strands Agents SDK).