Transparent peer review to be extended to all of Nature's research papers

The difference is that as a scientific reviewer you are not hiding a physical location and what you need is plausible deniability, which would still exist. In addition to this, actively attempting to deanonymise your reviewers is on the level of scientific misconduct that your employer and professional organisation should consider taking disciplinary action against you. I am not arguing that this makes it entirely safe to publish anonymised reviews and that we will not affect reviewer behaviour (maybe for the better in some cases, as "one-sentence reviews" will be something in the public record), but it is in stark contrast to the example that you bring up.

Translate into $RANDOM_LANG and then back to English. The perfect prose obfuscator.

We are struggling badly with review quality in natural language processing though. Most likely due to the unprecedented expansion of the field over the last ten or so years. Reviewers are suffering with review loads far exceeding what one reasonably can manage mentally (used to be two to three papers per reviewer and now five would be considered rather generous). Authors and area chairs suffer from worse quality reviews due to reviewer inexperience and overload, not to mention how good reviewer/author correspondence with author and area chair comments frequently being ignored by the reviewers. To me, the last holdout of good peer review in the field is Transactions of the Association for Computational Linguistics (TACL), but there the acceptance bar is sky high compared to ACL Rolling Review (ARR) for better or worse.

The ACL leadership and senior members of the field are very much aware of this and are trying their best (ARR being an attempt to improve the situation, but I am unsure how much better it really is compared to the old system of conference reviewing now that we are a few years in). But there appear to be no easy fixes for a complicated, distributed system such as peer review. Every discussion I have with said leadership and other senior members always ends with us agreeing on the problems and likewise agreeing that despite considerable mental effort we are failing to come up with solutions.

Returning to the main topic. Nature is worthy of praise for making their peer review transparent and I say this as a massive Nature critic. It is a move I loved seeing from NeurIPS (then NIPS) and ICLR over a decade ago, as it helps younger researchers see what good (and bad) communication looks like and that even papers they know now are greatly appreciated received a fair amount of criticism (sometimes unwarranted). I have argued for ACL to introduce the same thing for nearly a decade at this point, but we still do not and I have never heard a solid argument as to why not (best argument was the technological effort, but OpenReview, with all its flaws, makes this even easier than with Softconf; not that it would have been that hard with Softconf either).

> Anyone who has published a "scientific paper" in the last decade or so either "distrusts science" or is more likely than not a mid-wit at best.

That is very unhelpful, to say the least. The amount of noise has increased, but it does not mean that the scientists who know their subject disappeared. They are still around and not any less bright than their predecessors were 30 years ago.

Absolutely, the quickest way to lose faith in "The Science" is actually to do Science in a formal research institute....

> The fact remains that distrust of science continues to grow.

Of course it does, billion-dollar interests who have a vested interest in attacking it have a well-funded propaganda arm, and as we've been discovering over the past century - angry, loud, and frequently repeated bullshit with a profit motive floods any signal out of the room.

What's really sad is people who have legitimate concerns or desires for improvement hitching their horse-cart to the former.

No amount of peer review or replication is going to convince someone whose fortune is built on peddling snake oil.

> Ignoring the problem is not going to fix it,

Here's a solution. Hold the people making these attacks to the same level of rigor as what they are attacking. Stop giving proven liars a loudspeaker. The day the press will do that is the day some meaningful progress to fix that damage may be made.

Until you do that, we're going to continue getting shit outcomes.

> Why should anyone trust science?

When people talk about those who distrust science, they aren't referring to the carefully sceptical. They're talking about people who come to a conclusion and then reject any evidence against it.

> Why should anyone trust science? Skepticism should always be the default position.

Skepticism is not anti-scientific. Hell, distrusting results and theories is not anti-scientific. Distrusting the scientific method is. There is a difference.

> Putting it on a pedestal to be worshipped is what led us all into this mess.

Scientists did not ask for this. Amongst all high profile politicians, those who whine about science becoming political are those who made it so, by taking contrarian positions to rile up their voter base. Most people who make this point are not being honest. If you are, you should make specific arguments rather than rehash propaganda.

Nothing is ever “proven”. There is simply more or less support for a theory or proposition.

Replication and meta-analysis are an important part of this.

Most scientists are in fact very conservative with how they claim their results - less so university PR departments and “study shows” clickbaiters.

> I'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be?

It is not, and it cannot be. It is unrealistic to expect a referee working in their free time to confirm studies that often cost millions of dollars. This is a fundamental misunderstanding of what peer review is and why it is useful in popular or heavy vulgarised science.

Politicians, journalists, and university press offices are guilty of this, and they are those abusing peer review to give some studies more weight than they deserve.

> public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus

The public has lost trust in science because 10 to 30% of it is scientifically illiterate [1]. (Tens of millions of American adults are literally illiterate [2].)

That's what lets activists and politicians cherrypick bad science that supports their position or cast a scientific consensus as unquestionable.

[1] https://carsey.unh.edu/publication/conspiracy-vs-science-sur...

[2] https://nces.ed.gov/pubs2019/2019179/index.asp

One of the weaknesses to such a system is human nature. We want to believe, which leads to farces like Piltdown man, which was a farce by a man called Charles Dawson (not Darwin, not the natural selection guy, though I did a double take at first), against the Geological Society and society at large. The farce wasn’t definitively disproven until 41 years later, which is quite the downside risk of the gentlemen’s club research group gatekeeping scholarly research, but the more things change, the more they stay the same. This is a good thing which seems like it’s in response to grumblings about irreproducibility and bias, but those issues will likely always be with us and must be considered anew each time an experiment is designed, and each time a print run is cut.

I’m having trouble finding a downside besides vote buying or voting rings, now that which way one has voted is now attributable. Can you think of any risks under the new system?