HealthWatch members

RE: Results must mean something musn't they?

JamesMay — Mon, 08 Dec 2008 14:10:42 GMT

Les - If I have understood what you mean...
I wasn 't intending to be quite so nihilist/existentialist about the meaninglessness of their results. I believe a result is meaningful if it describes something real about the way the world works. I was suggesting that there may be a flaw in their study which means that actually the results do not reflect reality. This seems the most likely to me because of the suspicious basic science behind acupuncture, and because of the suprising nature of these results.

Regarding the bigger questions of meaning. If there is no meaning and we invent value, then there is no high ground from which to critique error - which seems to me to be essential. eg - false study results - why should they be valued less highly than true ones? Truth is a value which may not be shared by all, but which Healthwatch believes should be pursued, and falsehood opposed. This is not merely our invention or else it would be as arbitrary as the choice of homeopathy. We oppose homeopathy because it is false, and we value truth above falsity.

James[/u]

RE: Results must mean something musn't they?

LesRose — Sun, 07 Dec 2008 22:06:55 GMT

When I read this originally the authors' conclusions jarred and I could not spot their error. You have hit the nail on the head. Indeed such is the desire to assign meaning to everything, that meanings are commonly inferred spuriously. I think this is closely related to religion. Greek philosophy (to my limited understanding) was based on causes - everything that happened must have a cause, and if no material cause could be found then the cause must be spiritual. For me, the most liberating experience of my advancing years is to discover that life itself has no intrinsic meaning. It still has value though, depending on what we choose to do with our lives, and thereby we create our own meaning.

Results must mean something musn't they?

JamesMay — Sun, 07 Dec 2008 11:24:04 GMT

Dear all,

I was just reading through the Human Reproduction paper on acupuncture and came across this sentence:

"There are two possible explanations for our interesting results: real acupuncture may be associated with a lower pregnancy rate or placebo acupuncture may lead to a higher pregnancy rate."

There is a sometimes explicit tendency amongst empiricists to say that philosophy should be guided by science. I would argue that it cannot be so. Science must be guided by philosophy. This statement is driven by a desire to demonstrate that these results must mean something. However, the logical possibilities here include the possibility that the results do not mean anything but are mistaken and demonstrate nothing. Indeed in this case I suspect this is the more probable explanation.

James

Placebo labels for CAM products - could it work?

davidabender — Thu, 27 Mar 2008 15:21:52 GMT

Placebo labels for CAM products—could it work?

A HealthWatch member e-mailed the Newsletter with this interesting suggestion for how to regulate and label complementary and alternative treatments.

I believe that regulation based on the ‘Evidence basis’ is impossible with CAM treatments. It would be better to enforce clear labeling and promotion of all products and therapies perceived to be in the CAM area (including ‘Food Supplements’) that have not met the modern efficacy criteria with the bold writing, ‘This product/therapy is a PLACEBO and has no proven health benefit’. Any unsafe product could be banned (if a product contained Comfrey for instance) with ease. All CAM products and therapies would be classified at a stroke as a single group ‘perceived health enhancing treatments unlicensed as medicines or treatments’. Then, if a CAM product or therapy wished to become licensed to make claims then it would have no alternative but submit itself to the modern scientific route to achieve that status using appropriate criteria.

I believe funding of studies regarding safety and efficacy of CAM is a waste of resources both financial and human. The absence of patents is just one problem. Meagre profit margins on many CAM therapies and products could never fund the ever increasing demands from regulators of medicinal products. Whilst overall revenue in CAM worldwide may be billions of pounds there are plenty of witchdoctors making only a crust! Wouldn’t it be better if we accepted that most CAM therapies and products are fundamentally placebo and regulate and research accordingly? Perhaps we should concentrate on finding out what makes many such placebos seem so effective to their users before embarking on any more fruitless research into CAM using the evidence based paradigm.

More importantly, authorities should accept that CAM practitioners, therapies and products need to be regulated by reference to a different set of criteria from those of powerful modern medicines. The recently introduced Traditional Herbal Medicines Directive has already spawned a bureaucracy. My suggestion would be to abolish the THMD, class Homoeopathy as CAM and use the money saved to bring CAM into a simple framework of law based on safety and self regulation.

Les Rose, consultant clinical scientist and fellow of the Institute of Clinical Research, comments, “This raises some interesting and radical ideas, but I doubt if it’s workable. Suppliers of CAMs would continue to protest that they do have evidence, so a regulator will still be forced to test the evidence. If legislation along these lines came in, it would create a continuous burden of challenges. I suppose we could insist that to avoid a ‘placebo’ label manufacturers will have to make a case and pay a fee, thus financing the process.”

RE: OSCARS for referees

johng — Mon, 10 Mar 2008 10:34:20 GMT

I hope I am not breaking the rules of this forum by putting into this alleged "reply" box what is really a further comment about the pressures on referees of medical scientific papers. I am stimulated to put the question about publishing the names of referees because today's Independent (http://www.independent.co.uk/news/science/drug-giant-pfizer-tries-to-force-medical-journal-to-reveal-anonymous-sources-793711.html) reports that Pfizer is trying to force the NEJM to reveal the names of referees who gave advice to the Editor about papers on 12 studies on the drugs Celebrex and Bextra. This is to help Pfizer in their defence against claim for damages allegedly caused by these drugs.

I am against authors of research paper being told who reviewed their paper, because I think that it is in the public interest that the Editor gets a frank and honest opinion on the papers. However the editorial staff of some journals (including the BMJ) insist on "open" reviewing, saying in this way the reviewer is rendered "accountable" for the opinions expressed. However they are not so keen that the names and views of the editorial staff should be made public. Isn't that strange?

Where (if anywhere) should this "openness" stop. Should the public have instant access to the minutes of Cabinet meetings? Should individal members of juries in criminal trials be accountable for the way they voted, and their reasons for doing so?

I think not, because there is a danger that attempts at openness leads to more secrecy (eg. confidential covering letters saying "Also I believe that the data in this paper are fraudulent").

RE: OSCARS for referees

johng — Sat, 08 Mar 2008 17:33:53 GMT

The "reply" by David is a good summary of the function of the Editor plus the referees in mananging copy submitted to a peer-reviewed journal, but Editors and referees have different contributions.

I think it does not help to lump the two components together.

The Editor has the primary responsibility for the quality of the journal. His name is on the front page of the journal, and he deserves the credit, or blame, if the published material is good or bad. Of course he in turn is dependent on researchers to provide material from which he can select the best for publication. So to he must treat contributors well, which means he should be courteous, prompt to reply, constructive in criticism, and fair in judgement, otherwise potential contributors will send their ms. elsewhere. He must also be courteous to the referees to whom he sends papers for review, because he is asking a lot of an expert in a particular field to advise the Editor on points mentioned by David B, which is skilled work for which the referee gets very little reward.

The referee has a quite different task, and a much narrower responsibility. The editor has paid him (or her) the complement of trusting him (or her, but let me use "him" for both genders) to provide a balanced and constructive review of the paper. A referee who simply replies to the Editor "This paper looks OK to me" is of little use to the editor. If this is the reply he gets the editor himself may be to blame.

A good editor will indicate the points on which he needs expert advice. For example he may write "This paper comes from a scientist who has yet to learn the format our journal requires, but his data are interesting and novel so I hope they can be published. Specifically he claims that his is the first study of xxx, but I remember reading a similar study ten years ago on a much larger group of volunteers which came to a quite different conclusion. Please can you check if my memory is correct, and suggest the appropriate references he should have made to previous research. Also I do not think the abstract should call the results highly significant in view of the rather marginal statistical analysis."

A reviewer will respond more helpfully who receives a request that shows the editor has himself read the paper and thought about it, and which indicates the specific points on which he wants advice. I am not myself willing to spend a lot of time reviewing a paper if I see no evidence that the editor has himself read it!

When the referees' reports are in the editor has the task of replying to the referees to thank them, summarise advice received from other referees, and usually send a copy of the reply he has sent to the author concerning acceptance or rejection of the paper, or a statement about the weaknesses of the original ms. that would need to be rectified to justify publication.

All this involves a lot of work for both editor and referees, but it is very importent that it is done and done well. The big journals have many in-house staff paid to do the work - some journals even pay the referees!
It is tragic if good research is not published, or bad research is published, because editors and/or referees have not done their job properly due to lazyness or ignorance, but alas this does happen.

It also sometimes happens due to conflict of interest, but that is a topic I will return to sometime I hope.

RE: OSCARS for referees

davidabender — Sat, 08 Mar 2008 15:29:15 GMT

Peer review cannot usually detect fraud unless there is audit of the data as well. I do not think this is the purpose of peer review anyway. I think the main purposes of peer review are (not in any particular order):

1) To determine whether the paper advances our knowledge and understanding of the subject. Is it reporting original findings or significant confirmation of previously reported findings, or is it repeating previous work to no good purpose? I recall a series of studies on tryptophan metabolism some years ago in which each paper had essentially the same table of data, but with new results in the first line and the last line of the old table deleted. Even worse, I remember chasing multiple papers about the dopa decarboxylase inhibitor Benserazide, only to find that each paper - published more or less simultaneously - reported the same data, but with various permutations of the authors' names and keywords in the title.

2) To ensure that the description of the methods is clear enough for a fellow scientist to repeat the experiments. On one occasion I rejected a paper I was sent to referee for a pathology journal because the authors stated that they had fed their animals on the diet described by X & Y. The relevant journal was in another library. I went there, only to find that X & Y had used the diet described by P & Q, in a journal that was in another library. Altogether I visited four libraries, some distance apart, on a cold January day, only to find that what might have been the ultimate description of the diet was in a journal that none of our libraries held. Since I could not discover how the animals were fed, in a study of the effects of a low protein diet on something or other, I recommended rejection of the paper.

3) To ensure that the results are reported appropriately. For example, if mean and sd are given for determination of nutrients in wild mushrooms, was this multiple determinations on the same sample of each mushroom (unacceptable) or were multiple samples of each mushroom taken form different sites, and if so, how many? It took several weeks of correspondence with the authors to get the answer to this simple question. On another occasion I was sent papers from different psychiatry journals by the same authors who were reporting nutrient intake in people with senile dementia, based on 24 hour recall (in people who cannot remember whether they have just eaten, never mind what) - and then expressing average calorie intakes (1500 - 2000) to two decimal places! In this case I suspect (and hope) that the papers had been written by two PhD students without showing them to their supervisors, who were also listed as authors, and who were an eminent psychiatrist and an eminent nutritionist, both of whom I knew and respected.

4) To ensure that the experiment was designed appropriately, with adequate controls and replication. My saddest experience here was with work related to my own at the time. I had been a referee for the author's original grant application, which described a sound study. I subsequently met the author, and indeed he gave a good seminar in my department. Then I received the paper to referee and it reported that the results "approached significance" (in other words there was no significant effect) - but this may well have been because there were only 5 rats in each group - if he had repeated the experiment he might have found a result worth reporting.

5) To ensure that the introduction and discussion include reference to (all) the relevant literature. Sometimes, of course, the referee will angrily tell the editor that the paper is not fit to publish because it does not include a key paper (usually one written by the referee himself) published in a very obscure journal.

6) To ensure that the authors' interpretation of their results seems sensible and in accordance with what is already known. This, of course, is where referees may well stifle novel thinking, if a novel hypothesis upsets their comfortable world view. Nature rejected the original paper by Krebs and Johnson describing the citric acid cycle (and Krebs had a copy of the rejection letter framed next to the photograph of him receiving the Nobel Prize).

It is now common for journals to ask authors to suggest possible referees when submitting papers - a prudent editor will ask others as well, since obviously the author's friends are likely to agree with him. This is not always so. Several times as editor I have sent papers to referees suggested by the authors only to receive advice to reject or require major revisions.

Some journals also invite authors to suggest referees who should not be approached because of a possible conflict of interest.

Authors have to trust journal editors, and both editors and authors have to trust referees. Referees are sent papers in confidence, and it is assumed that they will act honorably, and not steal the data and publish it as their own.

It may surprise people who are not editors to know how many papers are rejected. Nature aims very high and is reported to reject about 90% of papers submitted, but even much lower down the citation index a 50% rejection rate is not uncommon. I am an executive editor of the Journal of Science of Food and Agriculture. When I am sent a paper I have the option to reject it immediately, which I do quite frequently, either because the results are trivial or for some other reason do not advance our knowledge and understanding, or the topic falls outside the (quite broad) scope of the journal. If I do not reject immediately I then assign the paper to an associate editor with appropriate expertise. Quite often when I do so I include a note that in my opinion the paper does not make a significant contribution, but it is outside my area of expertise and I would value the associate editor's more expert opinion. The associate editor then has the option to reject the paper immediately or send it to referees. The referees then recommend rejection, major revision (often requiring more experiments to be done), minor revision or (extremely rarely) immediate acceptance.

Immediate rejection by the editor can always be couched in kind terms - e.g. "the journal receives many more papers than it can publish", or one that I use quite often when sent papers about possible medical uses of herb extracts, which do not fit in the remit of JSFA "I am sure that your paper will reach a more appreciative and appropriate audience in a journal devoted to materia medica and medical herbalism".

Refereeing papers can be hard work. If there are only minor flaws then the referee's report is easy to write, but if there are major flaws (as is quite common) then it takes a great deal of work to chase up relevant important literature that has not been cited, sometimes recalculate the authors' data, etc, then write a report that tells the author that his paper does not add anything to the sum total of human knowledge, understanding and happiness in terms that will not be too disheartening. I sometimes agonise for several days over how to tell an author "your paper is crap" in kind terms!

OSCARS for referees

davidabender — Tue, 04 Mar 2008 12:37:12 GMT

Topic: OSCARS for referees (R)

Posted by John Garrow

This is my first attempt ever to contribute to an on-line forum. For several reasons I have been stimulated to raise the above topic, in this forum. What should be the status of reviewers (henceforward known as R) of scientific papers submitted for publication to a so-called “peer-reviewed” journal?

Why this forum? Because HW is largely about improving public understanding of the scientific component of healthcare. The source of this understanding is evidence published in peer-reviewed journals (although these insights may be severely distorted when reported by the media, but that is a different topic). Therefore HW should be concerned about the respect shown to the R who advise the editors if a paper submitted to the journal should be accepted, or rejected, or (in most cases) accepted subject to certain specific revisions.

Why now? Mandy, the excellent editor of the HW Newsletter, sent to all committee members copies of a piece in the Daily Telegraph by one Terence Kealey. He alleges that peer review is supposed to combat fraud, but can just as easily hold back radical discoveries. The full text is at
http://www.telegraph.co.uk/earth/main.jhtml?xml=/earth/2008/02/19/scipeers119.xml.
There may or may not be an item in the next issue of the Newsletter about the peer review process, but meanwhile I hope readers of this Forum will post opinions about the good or bad characteristics of R. Dr Kealey thinks they are on balance an evil influence, but an alternative view is that (some, at least) deserve to win an OSCAR. In this instance the award is for an R who is an Outstanding Source for Criticism And Revision.

RE: The Limits of Science

JamesMay — Mon, 14 Jan 2008 22:22:00 GMT

I wonder if I can change tack to try to inspire/provoke some response from anyone who has the time/inclination...

The question of the limits of science is really a question of whether naturalism is a true description of reality. Naturalism is the belief that nature is all there is, and that nature is a self-explanatory, deterministic and materialistic system. It may be that no one who reads this would describe themselves as a naturalist - but I doubt it.

The task of demonstrating that naturalism is not true is in principle easy. All that is needed is to demonstrate one thing that cannot be explained by natural causes, because then there would be something that was outside of nature. My first example was morality - see previous entry.

My next example is rational thought: Thought is rational if it corresponds to the real world, and is held for good reason. Naturalism has the problem that ultimately the mind is to be explained by non-rational factors - ie matter in motion, and that the feeling of knowledge is just that - a feeling, but not a genuine insight into reality. Rational thought is therefore excluded by Naturalism.

This problem is compounded by the lack of scientific progress into understanding what thought is or how the mind works. Further still, morality and justice depend on human beings being responsible for their actions - which assumes that human beings exist in reality, and that they are responsible, both of which would be denied by naturalism.

Naturalism as a philosophy is therefore bankrupt in my view.

An alternative therapist might make the error of falsely conflating the skepticism of Healthwatch with Naturalism as a philosophy. Given that to some eyes Naturalism is so manifestly in error, the temptation is to throw the baby out with the bath water, by dispensing with the study of nature along with the belief that nature is all there is.

Deliberately provocative I know - feel free to pick holes at your leisure.

James

RE: The Limits of Science

JamesMay — Wed, 05 Dec 2007 11:12:44 GMT

Les,

Fair enough - quite abstract I agree - though important.

How about the moral question? Isn't morality a natural phenomenon? I guess it depends how you define 'natural'. Can something be natural but not physical? Anyhow science is descriptive, not prescriptive. I can see no way that science can tell us what we should and shouldn't do. Indeed I think it has a track record of being bad at this. (That doesn't mean that science can't inform the decision - but it is neutral - not pushing us one way or the other).

Dawkins seems to recognise this sometimes and not others. At times he admits that life is meaningless and futile (by his own lights), and that morality is an illusion. But then he spends a lot of time criticising people for being immoral/evil... Where does he get those ideas from? It's not from science - by his own admission! (In my view)

James