Journals, Conferences, and Referees, or why "modern" "professional" Science is organized in a completely screwed up Victorian amateur way ----------------Warren D. Smith March 2001--------------------------- All science is organized around "journals" and "conferences". In principle, the (laudable) goal is to generate and disseminate scientific information to us all. The decision about which articles and/or talks get into which journals and conferences is made by unbiased expert "referees". Sounds great, but let's examine how this actually works in practice. REFEREES First of all, 80% of all referees are idiots and 100% are amateurs. By "idiot" I mean, either way too lazy to do a good job, or incompetent, or too biased due to some goofy agenda to judge the work unbiasedly (in roughly descending order of frequency), and in any of these cases, their effect is essentially that of tossing a coin. In the remaining 20% of cases, let us say that the referee is good and makes the right decision. In that case, the effect of any one referee is a 60-40 biased coin toss. In that case, if you have 3 independent referees (which is common, although in some cases they use fewer) the probability is .4^3 = .064 > 1/16 that a unanimous wrong decision is made. Thus, you can publish essentially ANY garbage if you simply keep re-submitting and have enough patience (and political skills probably help, e.g. being friendly with editors). Note that on average 1 out of every 16 garbage submissions leads to a unanimous acceptance decision, so if you submit and re-submit a total of up to 8 times, the probability is >50% you'll get it published. It is considered unethical to keep resubmitting, and especially unethical to submit simultaneously to more than 1 journal, since these tactics overload the (already busted) system. This ethical notion makes sense to me. Unfortunately there is essentially no way to detect or (especially) enforce these ethics and I believe (and have heard several scientists simply tell they do it and I should) these unethical methods are simply standard methodology for many scientists. Indeed I have encountered several cases where apparently somebody slipped up in their tricks so that almost-verbatim copies of the SAME article got published in more than 1 journal roughly at the same publication time (which definitely is unethical; every published article is supposed to be NEW original research, aside from articles that clearly are reviews). On the other hand, if you wrote an excellent paper, then the probability is still quite considerable that it will be rejected. Each rejection or acceptance can easily require years due to the presence of super-slow referees (remember, all referees are amateurs, i.e. unpaid volunteer do-gooders with zero accountability, indeed whose identities are kept secret!). Indeed in maybe 50% of all submissions, the journal simply apparently loses the paper (or something) and no rejection or acceptance ever comes back to the author. Ever. You get a slip acknowledging receipt of the article, but that is all you ever get. Note the expected number of submissions for an excellent paper before it gets accepted unanimously is 1/(.6^3) = 4.6. Thus if you are ethical you could easily wait 10 years even before even being able to publish an excellent paper. If you do not have the luck, patience, organization (and political skills?) necessary for this - merely the ethics - then you aren't going to be very successful at publishing even if your science is superb. The entire referee system is a non-working anachronism from the Victorian Age of Noble Rich Dilettante Do-Gooders. It is sort of like the asinine notions of "Amateurism" in athletics which are FINALLY being scrapped (e.g. the Olympics) after over 100 years. As science became bigger and became an enterprise of numerous professionals rather than few amateurs, this system became incapable of doing the job, but everybody refused to admit it. The resulting cost to society has been, and continues to be, absolutely immense. MERE ANECDOTES? All the above numerical figures are based on my experience as a professional scientist. However, my opponents could attack them as merely my biased special anecdotal claims. To respond to that, consider the following study [PLACE CITE HERE LATER]: The authors of the study took 18 random already-published scientific papers, changed the authors and titles, and sent them in to the same journal as submissions, in all cases within a few years of their publication date (thus the articles were still reasonably up to date). Result: 16 of the 18 got rejected. (In 0 cases was the plagiarism detected.) I think this study totally supports my "anecdotal" conclusions from my personal experience, don't you? Case closed. JOURNALS Many are exceedingly overpriced. Often they adopt two-tier pricing systems where "libraries" have to pay an enormous rate (such as $20000 per year per journal!?) while certain individuals pay a far smaller rate (otherwise none would buy it). This combined with the vast proliferation of journals means few libraries can afford the cost. The result is the failure of the whole mission of journals in the first place. The outrageous prices (which also vary by vast amounts from journal to journal; Elsevier and Kluwer journals have particularly outrageous prices) seem particularly hard to understand since many scientist-authors now do their own computer typesetting, and the editing and refereeing costs are usually free (done by volunteer amateurs). Many journals have reams of "for show" editors who are selected because putting their names on the masthead makes the journal "look prestigious" - but those "editors" actually do no work. Example: Turing Award Winner and famous scientist Robert Tarjan told me he was a for-show editor on numerous journals and he'd finally decided to tell them all to remove his name - although he was primarily concentrating on getting the journals which actually made him do work, to remove his name first! If you send your submission to some of those prestigious "editors" you can be in for an especially high probability your submission ends up in limbo land - never accepted or rejected! (In my experience, this happened 100% of the time I submitted to a non-chief editor, since at that young and naive stage in my life I had not realized that only the CHIEF editor is for real and should actually be submitted to, despite what the journal submission policy and masthead may say.) CONFERENCES Since journals are so screwed up (and they seem especially screwed up in fast-growing fields such as Computer Science, as opposed to say, physics, perhaps because everybody has been too rushed to get a semblance of professionalism going) many subfields of science (especially computer science) have instead become reliant on conferences. Conferences usually have far shakier standards of "refereeing" than journals (which already are pretty shaky!) and often lie by pretending to have more refereeing than they in fact do have (in an effort to appear more "prestigious"). Example: A too-typical "scientific" conference was VIDEA'95 organised by the Wessex Institute of Technology. (VIDEA = "Visualization and Intelligent Design in Engineering and Architecture", '95 is the 1995 incarnation.) The fact that in this case the emperor had no clothes was exposed by Werner Purgathofer, Eduard Groeller, and Martin Feda (Institute of Computer Graphics, Technical University of Vienna). Previously, one of them had accepted the role of "member of the program committee" for VIDEA'93 and noted that (a) he received exactly zero abstracts and zero papers to review, and (b) was never informed about any program committee meetings nor of any reviewing results. So for VIDEA'95, these 3 generated and submitted 4 bogus papers with varying degrees of hilarity (in one case the "call for papers" was submitted AS a paper, in another they "took a dictionary of information processing words and selected randomly some 40 phrases from there and joined them together to a fantastically technical sounding text." All four were then "reviewed and provisionally accepted"! Their conclusion was: "VIDEA accepts EVERYTHING!" At that point the only remaining requirement for their 4 nonsense papers to be published in the Elsevier proceedings ("high-quality books") was that the authors pay a "registration fee" (these fees often are $600 per paper). Oho, now we see the motivation... Wessex Institute of Technology also does 25 other conferences which Purgathofer, Groeller, and Feda now suspect are also a joke. Personally I think this kind of thing is quite typical of many of the high-volume low-quality conferences out there. At the other end of the scale, let us consider the allegedly highest quality conferences out there, which in Theoretical Computer Science means the SFOCS "Symposium on Foundations of Computer Science" and STOC "Symposium on Theory of Computing" annual conferences. I've heard a professional computer scientist say "Journal publications do not matter. Only FOCS and STOC papers matter." Careers can rest on publications in these conferences, which supposedly have a high rejection rate, and are refereed, thus assuring high quality. Oh? Really? Actually, for many years, the only "refereeing" you'd get back was a number from 1 to 5, as opposed to a referee's report. Indeed often this was condensed into 1 bit, rather than 1 number! Oh yeah. We can be sure a lot of consideration went into that. And indeed various well known completely wrong FOCS and STOC papers were published, although it was much rarer for anybody to ADMIT that (the first ever example I saw where it actually was admitted was again a paper by the admirable Tarjan, who in the next conference published a retraction [R.E. Tarjan, C.J. Van Wyk: Correction to ``A Linear-Time Algorithm for Triangulating Simple Polygons''. SFOCS 1987 page 486] perhaps in a failed effort to convince the rest of the conference authors to be equally responsible - since apparently this was the first such retraction in about 20 years of these conferences). In fact, once when I was a referee and wrote a long referee report for a paper I cared about, the editor (Valerie King) REFUSED even to GIVE my report to the paper authors, saying "I don't want a report. I want a number from 1 to 5." I then had to bypass her and send it to them directly. I complained to the conference's editor-in-chief about this but nothing came of it. Also, once, after my paper was rejected (this was back when I was a grad student) with of course zero explanation, I luckily managed to get some verbal explanation of the reason by talking to a Committee Member. It was: "Oh, well, somebody said this was all basically done before by Lovasz, so we rejected it." So then I said "But - Lovasz was the professor who advised me to submit this paper!" A typical scenario - the judgements are based largely on vague rumors. Another amusing case demonstrating the importance of being in the politically "in group" and the shallowness of the acceptance considerations was by my friend Bruce Maggs. He was in his advisor's (Leiserson, MIT) office one day. Advisor: "Well Bruce, I just was at the Program Committee Meeting for SPAA and there was considerable debate about whether to accept your paper. The going was tough, but finally, I managed to ram it through. You are In." Bruce: "That's great doc, and I appreciate it a lot. But... I never submitted a paper." Nowadays these 2 conferences actually do send back actual reports (sometimes) although sometimes you still get nothing back, and more often than not the reports are just 1 useless hastily written paragraph. Indeed I recently submitted 3 papers to one of these conferences, result: 2 rejects and 1 accepted. I wrote to the editor saying in all 3 cases the judgement should have been reversed(!) documenting my reasons. I concluded the reasons for the errors were a combination of stupidity, low quality, zero referee accountability, and political biases both pro and con (e.g. in one case the single referee, whose identity was obvious, found it very convenient for his career if my paper criticizing a certain approach, would just vanish. In the case where my paper which contained numerous errors and unsupportable hype, was accepted, it was politically inconvenient for it to be rejected for reasons of a similar but opposite kind.). JUNKETS As far as I can see, conferences are not good ways to disseminate information. Often authors are forced to talk for 20 minutes at most - and that is in the GOOD conferences like FOCS & STOC; in some conferences I have seen time limits below 10 minutes. This is a joke. In some conferences there are hundreds of such micro-talks. What percentage are you going to get anything from? One is supposed to fly 1000s of miles, consuming 100s of gallons of irreplaceable petroleum (more than the annual per capita consumption of average people in the world - each such flight is thus costing some average world inhabitant over a year worth of the economic benefits of gasoline) and the benefit that results is: a 10-minute talk??! You get far more information by reading the paper in the proceedings rather than seeing the talk. This is the communication age: The telephone; internet; Email; computerized search tools to find just the paper you need to find. In such an age, it is scientifically and morally unsupportable to have most of these conferences. The real reasons many of them exist is NOT their scientific value, but (a) insane holdovers from past ages, (b) bean counting based promotion decisions causing pressure to "publish", (c) opportunity by scientists to abuse their funders or employers to get free junkets to faraway places in the guise of "working". (d) The conference organizers pocket hefty fees (over $600 per participant easily). The airlines and hoteliers are also happy of course... Only people who should be unhappy are the granting agencies and employers (and taxpayers), but they are manned by scientists who are in on the junkets too, so they don't complain and the corrupt system cruises on. There have been cases [e.g. the "Winter brain conference"] where conferences held at ski resort areas have been cancelled due to lack of snow. My colleague Kevin Lang actually witnessed a medical conference at such a ski resort. Rather than giving their talks, the MDs had pre-recorded the talks on videotapes (allegedly this allowed them to present a better quality talk in less time, plus it got recorded that way - sounds like a good idea I guess), and they then played the videos at the lecturns. Lang witnessed a case where an absent lecturer was lecturing via videotape-player to an empty room - while all the MDs involved presumably cavorted on the ski slopes. SCIENTIFIC CAREERS Are often advanced or stalled by boards who mainly utilize "bean counts" (publication counts) as their decision making tool, as opposed to actually trying to READ one's papers (that would require work). Thus the pressure to publish can be immense, leading to a giant proliferation of garbage publications, a trend toward more papers with fewer ideas, overloading of the busted system, and encouraging non-ethics. I believe bean-count is anti-correlated to ethics. The fact that most of these boards are mainly populated by coprolites causes any consideration of the idea that maybe, just maybe, something is screwed up about the current system of Journals, Conferences, Referees, bean counts, Zero Accountability, and 100% Amateurism, to be dismissed. That in turn prevents any reform of the system. HOW TO FIX THE SYSTEM: 1. Referee anonymity and consequent lack of accountability and responsibility has to be got rid of. Papers should be placed on a web site like the xxx.lanl.gov site BUT submissions, and comments, ref reports, and author responses to said reports, and rejections, should non-eraseably accumulate as attachments to it on the site. This also will make the tactic of multiple unethical submissions, visible. 2. I think if anything the anonymity should work in the other direction: refs should be named, author-identities should be hidden from them. 3. Boards should actually (gasp!) read the scientist's work, which I believe they do not do, in general. (There even have been cases of scientists who enjoyed numerous promotions, including being granted tenure, who in fact had zero publications, but listed a lot of fake publications on their resumes. Evidently nobody actually went to the journal "cites" in the resumes to actually read those "articles.") If Boards plan to rely on bean counting, they could be replaced by my (non-scientist) sister, for a lot cheaper! This could be forced (or at least we could try) by making them make a report public reviewing that work. 4. Boards generally should be forced to operate in a non-secret mode (and all votes by them should be revealed vote-by-vote) and the evaluations and salaries of those they judge have to be made public so that info exists so that feedback and restoration-to-sanity forces become a possibility to try to at least have the possibility to get rid of incompetent board members, etc. All my attempts to make 1-4 happen have been failures, however. The whole way refereeing is run (amateur volunteerism with zero responsibility) is a ludicrous way to run science. The whole journal system is obsolete and massively overpriced and fortunately may FINALLY be BEGINNING to collapse under its own weight, being replaced by web sites. Note that web sites featuring a plethora of unrefereed scientific papers actually in many ways have papers of HIGHER quality than do refereed journals: (a) the papers can be searched by computer to find what you want, (b) they can include data and programs no journal would publish (for space reasons) (c) they could in future include actual software calculators, e.g. the reader fills in the blank with a number and out pop more numbers... (d) the cites could be implemented as hyperlinks (including cites OF this paper by future papers, could be traversed both directions) (e) accumulating positive and negative Comments and author Responses could be attached, ultimately having a referee-like effect far more accurate and severe than most actual refereeing ever is (f) If the paper is bogus, this could eventually be revealed by an attached Comment, preventing the paper from sitting around permanently as a trap waiting to destroy the work of some poor gullible graduate student or waste the time of some other reader. Meanwhile conferences show no sign of slackening - to the contrary.