The Worst Peer Review Ever!


On 17 Septermber 2007 an editor of the Journal of Vertebrate Paleontology sent me the following reviewers' comments on a manuscript that was subsequently published in the JVP (Bennett, S. C., 2008, Ontogeny and Archaeopteryx, JVP, 28:535-542, ). Note that to fully appreciate the review from Reviewer #1 you really should have read through that paper, but it is not necessary to get a feel for things. Okay, so let's start with the review from Reviewer #3:


Reviewer: #3

Recommendation: Accept

Comments: "This is the first ms. I've revewed with so little to change. On page 9, line 20, "data was" should be changed to "data were" (data are plural; datum is singular), and on p. 4 the software (SPSS) needs a fuller reference with its own bibliographical entry, unless JVP's editors disagree. The study is well done. Dr. Bennett stated the problem clearly, clearly stated why the study needed to be done, tested hypotheses explicitly, and clearly explained how the results supported specific hypotheses. His interpretations of results are logical and clearly stated. The figures, tables, and captions are exactly as they need to be, and there is a one-to-one correspondence between in-text citation and bibliographical listing of references Dr. Bennett has been publishing for many years, so it is no great surprise that he writes with clarity, concision, and exquisite grammar and mechanics, etc. This study is of especial importance for JVP because it involves a test of competing hypotheses that were both published in JVP."


Wow! That is the kind of review I like to receive, though I am chagrined that I missed a "data" treated as singular in my submitted manuscript. Now let's look at the review from Reviewer #2:


Reviewer: #2

Recommendation: Minor Revision

Comments: "The paper carries out regression analyses on bivariate plots of the linear measurements of osteological characters of Alligators taken from the literature and uses these to address the application of this method for determining whether specimens should be assigned to a single or multiple species. The impetus for the paper is provided by a previous publication by Christiansen (JVP – 2006) who challenged the validity of this regression method for determining number of species within a collection of specimens. Christiansen showed that you can get a lower correlation coefficient between pairs of skeletal elements within a single species than in a plot of multiple closely related species and criticised previous work conducted by the author of this paper on Rhamphorynchus and other Pterosaurs, and studies by Senter and Robins (2003) and Houck et al. (1990) on Archaeopteryx. Christiansen argued that size differences evident in the Archaeopteryx data might suggest multiple species, whereas Senter & Robins and Houck et al. concluded a single species. The current paper argues that Christiansen's analyses and conclusions (using Larus & Panthera) are not comparable to the Rhamphorynchus and Archaeopteryx samples, largely because ontogenetic factors were overlooked. I agree with the author (Bennett) here and find this a well written manuscript describing the regression approach, its rationale and how it 'should' be applied in a very clear and comprehensive manner. It systematically and carefully addresses Christiansen's criticisms and conclusions, and highlights very well the importance of ontogeny in analyses of this type (the aim of the paper)."


Reviewer #2 then went on to make numerous suggestions for minor changes and improvements to the presentation of the manuscript, many of which I accepted, but which need not be repeated here, but note that Reviewer #2 did not suggest any changes to the substance of the manuscript. Again this is the sort of review I like to receive. Okay, now with the stage properly set, let us turn to the review from Reviewer #1.


Reviewer: #1

Recommendation: Reject

Comments: "Once it was common in paleontology to construct regression plots on singular characters to infer phylogenetic relationships. Decades ago, this approach was gradually replaced by a much more advanced type of method; cladistics and the use of homology testing and parsimony. Yet even today a manifestly outdated and incorrect use of allometry and multivariate statistics is still observed in part of the paleontological community, in particular the bizarre bird-dinosaur debate.

It is not easy to establish what the current paper is about. It purports to criticize a previous study on concluding that allometry is an incorrect approach to phylogeny. Any systematist knows this already and would never use regression plots in such a manner. Yet, the current study appears to endorse this flawed approach. This becomes almost mysterious when the author has to admit that his samples of different species are non-distinguishable! This fatal flaw is of course heavily exacerbated when dealing with fossils of unknown phylogenetic affinity. Thus, the results from this paper appear, if anything, to corroborate the study it supposedly criticizes. However, this is largely irrelevant; the problem lies in an improper use of regression and multivariate statistics to infer phylogeny. It would appear that the author is unfamiliar with proper approaches to phylogeny (not taxonomy as is repeated over and over again throughout the ms).

When dealing with fossils the biological species concept is unavailable, (and the species question is, in itself, irrelevant to the current problem). Thus, for decades the morphological species concept has been endorsed, in modern terms by identification of synapomorphies in a parsimony framework, and applying ad hoc tests of homology and similarity (and statistical non-independence for morphometric characters), and post hoc congruence. Allometry may serve as a character in such analyses, nothing more. Fossil samples of dubious phylogenetic affinity cannot possibly be lumped into one species simply by high correlation coefficients. The question of ontogeny is irrelevant to the current discussion.

It [sic] the author wishes to study ontogeny, which is by default correlated shape changes, and, in particular, if he wishes to use this in analyses of relationships, there are far more advanced methods than simple allometry or PCA, neither of which studies shape changes. Yet, these methods are not even mentioned. Warp analyses furthermore have the advantage of returning variables which are potentially phylogenetically informative, and may be subjected to the traditional tests of homology, similarity, statistical independence, and congruence. Allometry slopes may, at best, be included as one of many potentially informative characters in a parsimony analysis, and most often are simply mapped onto a phylogeny post hoc. PCA may be employed in cluster analyses to test for groupings (total evidence). Such phenetic trees are, however, manifestly not phylogenetic in nature. This debate between phylogenetic systematists and numerical taxonomists has been pretty much over for decades. These methods have no bearing in phylogenetic systematics.

Proper methods for phylogeny analyses have been formulated, tested, debated, and expanded, and have proven their worth in countless studies in the relevant literature (e.g., Systematic Biology, Cladistics), both on real samples and on simulated datasets. It is unsettling to still encounter studies that ignore decades of scientific advancement within the very difficult concept of relationship analysis."


Reviewer #1 seems to have completely misunderstood my manuscript and criticized me for things that have nothing to do with what I wrote, while offering not even the tiniest shred of constructive criticism. In my manuscript, I was not interested in inferring phylogenetic relationships, not interested in the bird-dinosaur debate, and not interested in studying ontogeny. I was simply interested in countering Christiansen's incorrect assertion that high correlation coefficients found in statistical analyses of length measurement data of Rhamphorhynchus and Archaeopteryx are inconsistent with the interpretation that the samples consist of a single species, and by extension I was interested in whether the sample of Archaeopteryx consists of a single species or multiple closely-related species. Furthermore, in my manuscript, when I wrote 'taxonomy" it was because I was concerned with alpha taxonomy, and I did not write 'phylogeny" because I was not in any way concerned with phylogenetic reconstruction. Reviewer #1 has produced the worst peer review I have ever seen, and I challenge them to contact me and defend, explain, or repudiate their review—and they can even do so anonymously if they are spineless.





 

Back