To test whether my belief that homeopathy is not an evidence-based rational system of medicine is reasonable or just prejudice on my part, I did a quick experiment using Google Scholar. (This was done Jan 17, 2008)

A "layman's meta-analysis" of sorts. Not supremely scientific alas (although perhaps somewhat reminiscent of a GCSE science project ;) ), but the best I could do early in the morning with only vending-machine coffee to keep my brain from sleep ;-) 

The Setup
The basics of the test were as follows:

1. Search for homeopathic trials in Google Scholar (using a neutral search term) - I used "homeopathy trial"

2. Look at the conclusions of the first 10 results to come back (I don't know how Google Scholar prioritises results, so hopefully there's no bias either way in picking the first 10).

I'm taking the first 10 results regardless of what specific remedy or ailment is being tested,  I have no way of knowing if there's certain ailments homeopathy is better or worse for, and no way of knowing what ailments will come up, so I think this is as fair as I can make it. If there's any duplicates, I'll ignore them and take the next one.

3. Categorise each result into 3 broad categories:

    Negative - no statistically significant evidence that homeopathic medicine was more effective than placebo

    Positive - statistically significant evidence that homeopathic medicine was more effective than placebo

    Neutral - no conclusive result either way, or indication of problems with the trial that could make the results meaningless

4. Having sorted out the results into the categories, I'll look at the results and see if my belief that homeopathy isn't supported by reasonable evidence is fair or just "out of hand" dismissiveness on my part.


Having set up the experiment, I entered the search term into Google Scholar and took the first 10 results and categorised them based on their stated conclusions:


POSITIVE - statistically significant evidence that homeopathic medicine was more effective than placebo

http://pediatrics.aappublications.org/cgi/content/abstract/93/5/719
(small scale single trial, concluded more research needed)

http://www.pidj.org/pt/re/pidj/abstract ... 29!8091!-1
(small scale single trial, concluded more research needed)

NEGATIVE - no statistically significant evidence that homeopathic medicine was more effective than placebo

http://archsurg.highwire.org/cgi/conten ... 33/11/1187
(meta-analysis, fairly large scale, conclusively says no effect)

http://www.clinicalpain.com/pt/re/clnjp ... 28!8091!-1
(medium scale, conclusively says no effect)

http://thorax.bmj.com/cgi/content/abstract/58/4/317
(small scale, no significant effect)

http://www.jrsm.org/cgi/content/abstract/96/2/60
(small scale, no significant effect)

NEUTRAL - no conclusive result either way, or indication of problems with the trial that could make the results meaningless

http://www.sciencedirect.com/science?_o ... 7efe96eb7d
(small scale, no significant effect, but "further research is justified")

http://www.sciencedirect.com/science?_o ... 5e704b40fd
(slight positive effect in one specific area, no significant effect in others, limited number of patients observed and the posology employed could have interfered with the significance of the results.)

http://rheumatology.oxfordjournals.org/ ... t/39/7/714
(semi-positive. medium scale, positive results in that homeopathic gel was as good or better than the conventional one, but "The presence of a clinically relevant difference between treatment groups cannot be excluded"))

http://www.sciencedirect.com/science?_o ... cd1f237978
(semi-negative. no significant effect but mentions "complexity theory" as a possible factor in future trials)

Which gives us the final scores as follows:

RESULTS:

Positive (in favour of homeopathy) - 2

Negative (indicating homeopathy is ineffective - 4

Neutral (not conclusive either way) - 4

Conclusion:   In conclusion, I think that, with 40% of the results strongly indicating homeopathy is ineffective, and with only 20% demonstrating a statistically significant effect in favour of homeopathy, I think my belief that homeopathy is ineffective is reasonable and my dismissal of homeopathy isn't merely "out of hand".

Of the cases that showed a positive effect for homeopathic treatment, none of them indicated a really strong positive effect, just a statistically significant one.  Both of the cases showing a positive result for homeopathy concluded that "more research was needed".

In light of the fact 40% of cases were in the neutral category, I'd conclude that more research is needed to give a stronger answer as to whether homeopathy is effective, but from the evidence so far, it seems it isn't.

What do you reckon? Was this a fair test to check if I was prejudiced, or was it a total waste of time given that I chose which category to put each result into? ;)