To state the obvious first, any poll in which the person answering the question makes the decision to become a part of the poll (i.e. "self-selected") is bogus. It immediately becomes "unscientific" and therefore meaningless, at least in terms of helping anyone understand anything. There's a Washington Post story today about how inventive people with time on their hands can manipulate the whole "user rating" system on a web site. This T-shirt photo shows the item that became the best selling piece of clothing on Amazon.com because of manipulation by users. The next time you see a TV station using a poll from their web site remember this story. Is candidate "X" really ahead...or just ahead in convincing supporters to click on his name? Do Alabama residents really believe in UFOs? Or is a local UFO group supporting its avocation by manipulating the system. At the very least, TV stations should include a disclaimer...though I understand why they don't. How dumb does it sound for an anchor to use a poll as a"news story" and then tell viewers the poll results are "not scientific"..i.e., essentially meaningless?
Now Tim, I'm gonna' gently goad you here on this post - for a couple of reasons.
ReplyDeleteFirst, what grade did you make in college statistics? I failed it twice... before I made an "A".
And professionally, having read numerous research articles ranging from social research to scientific medicine, I've read more methodologies than I care to recall.
Second, you write "...the person answering the question makes the decision to become a part of the poll." People may - and do - refuse to participate in polls. Their voluntary inclusion or exclusion does NOT automatically invalidate any findings.
The creation of scientifically valid polls require certain actions and must undergo scrutiny by many others in the process. Outliers, randomization, interpretation, analyses, methodologies, categorization, pool data, variability, hypothesis testing, regression, etc. are all terms used in statistics.
Are we talking about anecdotal, observation or controlled evidence?
The principle reason why online polls are NOT scientifically valid are because they have almost none of the hallmarks used in any scientifically valid study. Contrary to what you write - vis-à-vis, "...any poll in which the person answering the question makes the decision to become a part of the poll is bogus." - it seems you may be confusing controlled studies with observational studies or perhaps associating anecdotal studies with controlled.
In observational studies, subjects are self-selected, whereas in controlled studies subjects are selected in advance by the researchers, variables are accounted for, and control groups used.
Randomization is another factor in polling - or data gathering - the topic of discussion herein.
That's not to say that observation studies are not useful, for they are immensely useful. For example, "the evidence that smoking causes lung cancer in humans, although largely observational, is compelling."
Observational studies - which I think may be the type to which you refer - can show association "when the effects of plausible confounding variables are taken into account by appropriate statistical techniques, such as comparing smaller groups that are relatively homogeneous with respect to the factor."
"Deciding whether associations are causal is not a matter of statistics, but a matter of good scientific judgment.” Questions of control, randomization, investigator control, observation and all figure in.
Now, let's get personal.
In your blog entry Monday, April 20, 2009, MMMM* # 37 - Purposeful Clicks (ref: http://timlennoxonline.blogspot.com/2009/04/mmmm-37-purposefull-clicks.html), you touted Google Analytics service on your blog site, writing "I subscribe to Google Analytics, and it provides me with a wealth of information about the thousands of people who come by for a visit."To what extent would you purport that their results are valid or even scientific? What inferences would you draw from the data supplied by them? You indicated some reliance upon that data, and indeed it can be.
However, (I write with an eye toward understanding), it seems to me that on one hand while you purport to excoriate "any poll" (specifically, the online type) writing "How dumb does it sound for an anchor to use a poll as a "news story" and then tell viewers the poll results are "not scientific"..i.e., essentially meaningless?" you hail Google Analytics for what they give you... which is - as you write - "essentially meaningless?"Yeah, I guess it'd chap my hide to have someone telling me something was useful when it was "essentially meaningless," while they happily use such information.
On a contemplative curiously excoriating note, which side of your mouth are you using today?