       |
|
|
 |
Broadcast Sunday 26 October 2003
with Robyn Williams The difficulties of getting a research paper published Summary: Associate Professor David Pannell from the University of Western Australia describes the frustrations he has experienced when attempting to have a research paper accepted by a scientific journal. Transcript: Robyn Williams: On the 25th April 50 years ago a paper was published in the journal Nature, titled A Structure for Deoxyribose Nucleic Acid. DNA to you and me. It had been sent in by James Watson and Francis Crick and was received on April 2nd. Three weeks was the short wait, straight in, in other words. The publication changed history, as we know. It’s the cherished ambition of all scientists, and most academics in general, to make such an impact. Their careers depend on it. Yet nearly a quarter of scientific papers published disappear without trace. They are never mentioned by anyone else. It’s worse in the social sciences and the arts. Of papers published in social sciences, 48% are never cited by anyone else. In the arts and humanities the figure is a whopping 93%. So what’s the point of publishing? And why does y our employer, usually a university, care so much about your turning up in even an obscure journal when your appearance in the media, radio and television, would make more impact? Yet being on radio or television can even be counterproductive as a career move. Dave Pannell is Associate Professor in the School of Agricultural and Resource Economics at the University of Western Australia, and here are some of his experiences with the perils of publishing. David Pannell: Last week I received a letter from a scientific journal. This was an exciting event because I was waiting to see whether they would publish a research paper that I sent them about six months ago. I opened the letter, hoping for the best, but fearing the worst. The news was bad. One reviewer had a reasonably positive attitude to the paper but the other was very negative. Some of the comments of the second reviewer are really annoying me. I can’t believe that they’ve really read the paper properly. It’s frustrating and quite disheartening, especially when I think about how much work went into the research and into writing the paper. But deep down I know this is all perfectly normal. Publishing scientific research was not meant to be easy. It’s important of course for all serious researchers. It provides scrutiny, recognition, rewards and a vehicle for wide communication, but on the downside, the process is slow, heartless and somewhat random. So I’ve decided that it’s time for an expose of the scientific reviewing process. First I should tell you briefly how the process works. There are hundreds of journals, each specialising in a particular research field. Once you’ve written a research paper, you choose a journal that you judge might publish it, and send the paper to the editor. The editor sends it to two or three referees or reviewers. Editors usually attempt to select referees with some expertise in the topic of the paper. In most journals, the reviewers don’t get paid anything and the authors don’t get paid either. Indeed, many journals charge authors ‘page charges’ for the privilege of getting published. You can only submit to one journal at a time. A common practice is to initially submit to a relatively high prestige journal with low probability of success, and if your paper is rejected, you work down the hierarchy of journals until the paper is no longer rejected. Many journals use a system of ‘double-blind’ reviewing, in which the identities of both authors and reviewers are kept secret by the editor. The intention is that reviewers should feel free to speak their minds without fear, and that the work should be judged on its merits, independent of the identity of the authors. While these are reasonable objectives, it doesn’t always work out. Sometimes the power of being anonymous goes to a reviewer’s head, resulting in needless aggression or negativity. Sometimes the anonymity of the author is hard to maintain, particularly in highly specialised areas with relatively few active researchers. When reviewing a paper myself, I find that I can usually have a good guess at who the author is. So some journals use a system of ‘single-blind’ reviewing, revealing the authors to the reviewers but not the other way around. Let’s talk now about some of the problems that a researcher will encounter when trying to publish in a journal. The first is randomness. The sheer unpredictability of what reviewers will say still take my breath away at times. It can be particularly entertaining when reviewers of the same paper are diametrically opposed in their views. Here is an example. I submitted a paper to an Australian journal some time ago and got these two responses from the two reviewers. Reviewer 1: ‘This paper falls into the class of papers that can be denominated as “measurement without theory”.’ Reviewer 2: ‘This paper, as it stands, is purely theoretical’. Not surprisingly, the paper was rejected. Sometimes the entertainment comes from comparing the attitudes of referees for one journal with those of a subsequent journal. I once wrote a paper on the economics of agricultural insurance. I submitted it to an American journal, and it was rejected with highly critical responses by the reviewers. I modified the paper slightly and submitted it to an equivalent Canadian journal. It was accepted with minimal changes and with some enthusiasm. Subsequently that paper shared the award of the Canadian Agricultural Economics Society for the best journal article of the year. Although the initial rejection was disappointing, I can take heart in the experiences of far more eminent researchers than myself. There are some great examples from my field of economics. For example, in 1923, an economist named Bertil Ohlin submitted to the Economic Journal a paper that introduced the factor proportions theorem to international economics. The theorem eventually earned him a Nobel Prize, but the editor of the journal, John Maynard Keynes, returned the manuscript with a blunt rejection note, saying, ‘This amounts to nothing and should be refused.’ The second problem is bad or unreasonable reviewers. If a reviewer says your research papers is insufficiently interesting or original to deserve publication, it is usually hard to argue. Such judgements are subjective and personal. Sometimes though, reviewers write what is obviously nonsense. This happened to me with a paper I wrote about herbicides. Reviewer 1 quite liked the paper and made some constructive suggestions, but Reviewer 2 obviously hated the paper, primarily because ‘I don’t think its main point is sufficiently worth making to be the primary focus of a paper, particularly one which has some technical problems.’ Despite the reviewer’s comment, no genuine technical problems were identified. I revised the paper, and wrote a letter to the editor explaining that the supposed technical errors were not technical errors at all, and resubmitted it. He sent it off for reviewing again. Referee 1 was happy but the second reviewer still hated it. He or she made no comment about the supposed technical errors that had been identified the previous time but introduced a brand new set of supposed technical problems, again which have no foundation. I revised the paper again, wrote another lengthy rebuttal letter to the editor, and resubmitted again. After resubmission, the second reviewer again made no acknowledgement that the so-called errors in the paper were actually non-errors, conveniently ignored the whole issue, while re-asserting disdain for the paper in very general terms. A third reviewer brought in to adjudicate was, unfortunately, not impressed with the paper and eventually it was rejected. Subsequently it was published in another journal with little trouble. This was a particularly upsetting experience. I put an enormous amount of work into the two rounds of revisions to try and keep Reviewer 2 happy, but I could only feel that the reviewer was being extremely unreasonable and almost dishonest. The other feeling I had was one of powerlessness. Even though I felt I was so obviously in the right and the reviewer was so obviously wrong, I felt like my punches were landing on smoke. I had to be accountable for every tiny issue raised by the reviewer, but the reviewer could blithely ignore his or her past erroneous comments as if they had never happened, and still win the battle. Of course, sometimes the process works, in the sense that the paper is correctly judged to be not of sufficient merit or real problems with the paper are identified, potentially saving the author from serious embarrassment. The painful examples I’ve described did at least teach me a lot about the reviewing process, and thicken my hide for subsequent encounters. And one certainly does need a thick hide. I have found that many referees are needlessly hard and heartless. Too many seem to take the view that their role is to demolish a paper if possible. Even when reviewers are trying to be constructive, it is very easy for an author to become disheartened at the poor reception their work has received. I believe it is very important for reviewers to consciously highlight positive aspects of papers, not just negative. As some therapy after that experience, I wrote a little verse called ‘I’m the referee’. Here’s an extract: This power’s a revelation I’m so glad it’s come to me I can be a total bastard with Complete impunity. I used to be a psychopath But never more will be I can deal with my frustrations now that I’m a referee. I sent it to the editor, but he didn’t respond. That story shows how much persistence it can sometimes take. And it’s not just me. Some of the most cited and celebrated papers of all time have only seen the light of day because their authors did not give up in the face of repeated rejections. A great example is George Akerlof’s wonderful 1970 paper on the economics of information. It’s called ‘The market for “lemons”.’ It’s about used cars. That paper was rejected by The American Economic Review, with the comment that the journal did not publish such trivial stuff; then it was rejected by The Journal of Political Economy, who asserted the opposite: the paper was too general to be true; then it was rejected by The Review of Economic Studies again on the grounds that it was trivial; and finally The Quarterly Journal of Economics accepted the paper with some enthusiasm. Subsequently the paper has gone on to be one of the most widely cited papers in all of economics. Akerlof’s understandable response to the trauma of publishing ‘The market for lemons’ was to be discouraged about the topic and to underestimate the importance of the topic and its potential for further work. He later said, ‘Its early rocky reception did have an effect on my own work. It was not until three years later when I spent six months on sabbatical in England that I realised that quite a few people had read the paper and even liked it. I believe I would have done follow up work on ‘The market for lemons’ sooner if I had not been made to feel lucky just to have it published at all.’ The third common problem with trying to publish in a research journal is the slowness of the process. Partly because their role is honorary, and partly because researchers are busy people, reviewers are not always quick to respond. I recently went back and checked how long the process took for my most recent 27 submissions to journals of agricultural economics, natural resource economics and agricultural science. The total time lag between submission and the receipt of the editor’s final decision varied from 3 months to 2.5 years. Only a quarter of my papers were accepted or rejected within 7 months, and the average for all 27 papers was 12 months. There are other lags in the system as well: the author usually has to make changes to the paper to satisfy the referees, the editor has to look at the final version and make a decision, and then if it is accepted, it goes into a queue before being printed. For my recent papers the average total lag from initial submission to finally seeing the paper in print was almost 20 months. It is worth acknowledging that there are often very legitimate and understandable reasons for the failure of reviewers to provide timely and high quality reviews. Editors naturally see it as desirable for reviewers to have a wide breadth of knowledge and considerable experience in publishing. But such people are inevitably very busy with their other work responsibilities, and it is certainly understandable that many of them don’t rush to undertake a task for which they get no professional recognition and usually no monetary reward, although some journals are now offering small financial payments to referees, more as a token than as a reward. They are also likely to receive a large number of papers to review. Authors can probably help themselves by recognising these realities and making sure that their paper is as short as they can make it, well written and presented, and is being submitted to an appropriate journal. Publishing in research journals requires fortitude, resilience and persistence. So if you are a researcher, you shouldn’t be too discouraged about the negative response of reviewers, because even the best researchers have suffered similar problems. You should not give up on a paper that you continue to believe has merit, despite being rejected, because some of the most celebrated papers have suffered through several rounds of rejection before finally being accepted. Of course do consider the possibility that your paper is in fact not worth persisting with, but just don’t be too quick to leap to this conclusion. Robyn Williams: Dave Pannell on publish or perish. There are about 20,000 refereed journals across all fields of scholarship, he tells me, and they publish more than 2-million articles a year. Some of them are read by more than just the author. Professor Pannell is in the School of Agricultural and Resource Economics at the University of Western Australia. Next week, Ockham’s Razor comes from Adelaide, where Neil Underwood is contemplating humility as a healing tool in mental health care. I’m Robyn Williams. Guests: More information: Story Index |
||
|
|
|||||||||||||
