True Experimental Design

R O1 X O2
R O1 -- O2

There once was a research design
That look, on first glance, oh so fine
Yet it is stupidity
To view its validity
As chaste due to Rs on each line

For one thing, mere randomization
That's done to gain equalization
Of Es versus Cs
Gives no guarantees
That you'll have good gen'ralization

Results may, for instance, show all
That Es topped the Cs (with p small)
But if that full troop
Ain't your "target" group
Wide claims will lead to your downfall

Or X may work well for just some
While others to it are quite numb
The treatment's true fate
Depends on the trait
One needs if gains from X shall come

The one who rates subjects may know
Which Ss got treatment -- and so
Effect Rosenthal
May ruin it all
And truth from your study won't flow

If treatment is novel, each E
May first say "It's super for me!"
But as time goes by
E may think and sigh . . .
"No value in X do I see."

Or just the reverse can occur
With X thought at first to deter
But once acclimated
Es may be elated
And to X high marks they'll confer

Should hist'ry and X interact
'Tis true there is no artifact
But if replicated
You'll be decimated
By findings that prove inexact

The IV or DV may be
Reported not sufficiently
If later one tries to
Your study to redo
New findings may be what you'll see

If two or more studies are done
With Ss not used in just one
A treatment . . . its "grade"
May strengthen or fade
If only one study is done

A pretest or posttest may make
The E-group to X wide awake
O leads to creation
Of sensitization
And sans O the treatment won't take

The time from the X till O2
Is worthy of careful review
For if it's revised
When new plans devised
Fresh findings may seem quite askew

The subjects may be told or know
Observers are watching . . . and so
They'll work with great skill
But only until
The watchers stop watching and go

Or what if the subjects perceive
What YOU hope and want and believe
Unconscious or not
Like robots they'll trot
And data you want you'll receive

I hate now to sound so paternal
But we must move past threats "external"
More pitfalls there are
To rip, maim, and scar
And keep your work out of a journal

Should either X or O be thought
To deal with a construct that's "caught"
When such is not true
With "fit" that's askew
One's main claims won't be what they ought

Moreover, one should not assume
That stat work can't add to the gloom
Assumptions may be
False prima facie
And that makes the truth not illume

Or what if there's meager precision
Strong Xs you'll miss for poor "vision"
The Error Type II
Will hide truth from view
And you'll make an inapt decision

But what of those "internal" threats?
Are all tamed, like nicely trained pets?
If that rings as true
You've got learnin' to do
For Rs catch just some in their nets

For instance, take treatment diffusion
Which can make for lots of confusion
Should E give to C
All X that there be
You'll come to a faulty conclusion

Or what if the folks in the C-group
Think that they were put in a "B" group
They're demoralized
And you victimized
By "findings" as murky as pea soup

The converse may also take place
Cs saying, "We will win the race!"
Each tries like a kid
(John Henry once did!)
Yet this can one's findings debase

And what if there's subject attrition
Related to the X condition
It may just appear
That X was "in gear"
When that's just a sad apparition

Of course, there is further the chance
That someone may try to enhance
The plight of those in
The C-group wherein
Such help is a foul circumstance

And lastly consider the stew
You're in if on post Es construe
On DV their standing
As not so commanding
'Cause X brought respect for O2

By "lastly" I sure don't intend
To mean that one need not attend
To problems that have now
No label, yet somehow
They could wreck one's work in the end

The moral, I hope, is quite clear
The word "true" doth make it appear
That problems of import
Are all on the backcourt
When really they may lurk quite near!!!

--

Schuyler W. Huck

(1992)