Thursday, March 26, 2009

Week 12: True/Quasi Experiments

Lauer and Asher – True Experiments

-experimental research actively supplies treatments or condition such as sentence-combining, planning strategies, or word processing to determine cause-and-effect relationships between treatments and later behavior.

True Experiments:

-actively intervenes systematically to change or withhold or apply a treatment to determine what its effect is on criterion variables.

-uses randomization by which subjects are allocated to treatment and control groups.

-give hypotheses in the form of questions or positive statements

-require a null hypothesis

-must have a randomized selection of subjects. There can be no expected differences among the subjects on any variable.

-have criterion variables to compare the control group and the treatment group with


-DO NOT randomize subjects because of restrictions and use already established, intact groups

-have at least one pretest or prior set of observations on the subjects in order to determine whether the groups are initially equal or unequal on specific variables tested in the pretest

-have research design hypotheses that account for ineffective treatments and threats to internal validity.

-are either strong or weak. Strong have proven equality in pretests that determine the two groups are very similar, despite not being randomized. Weak quasi-experiments are unequal in at least one of the variables.

-must show whether or not groups are initially equal or unequal

-still must show interrater reliability and stability of variables

-risk internal validity because of the lack of randomization. However, it is better to leave in outlier subjects than to remove them from the study. The results are more generalizable this way, even if the study is a weak quasi-experiment.

John M. Carroll, et al.: “The Minimal Manual”

Detailed account is given of the Minimal Manual strategy that was developed prior to this research study. A manual was designed to be simple, stripped of verbiage, and user-centered. It was task-oriented, not jargon and information loaded. Description is given about the errors that users made and the adjustments that were made to the manual. However, no description of subject selection is given. It is impossible to tell if the subjects were randomized. No hypotheses are given.

Two experiments were conducted after this preliminary test. In the first, 19 subjects were observed, 9 using a tradition user’s manual, 10 in the Minimal Manual group. Subjects are not randomized, thus, at best, this can only be a quasi-experiment. However, the subjects they used were not already intact groups, thus there is a poor design from the beginning. Interrater reliability is not given and criterion variables are unclear. This experiment is poorly designed, or, at least, poorly described.

The second experiment has four groups that either Learn by Doing (LBD) or Learn by the Book (LBB) and test the Minimal Manual against the traditional manual. Each group, though ,was given different tasks. This automatically makes for a problematic experiment. Also, 32 subjects were tested, slightly smaller than the suggested 40 (four times the number of variables being tested), which hurts the design. Subjects are certainly not randomized, nor did they come from intact groups. Data collection is vague and it is hard to determine interrater reliability.

Elaine M. Notarantonio & Jerry L. Cohen – “The Effects of Open and Dominant Communication Styles on Perceptions of the Sales Interaction”

A history is given of the communication studies as they relate to business sales. This is provided in order to show that variables come from previous studies and are to be further examined in this experiment. A hypothesis is given that says a salesperson can manipulate his or her style to provide for maximum sales effectiveness. An explanation is given between Open and Dominant salespersons—the Open person is gregarious, frank, conversational, unsecretive, and shares personal information. The Dominant is competitive, confident, enthusiastic, and forceful.

The subjects were business students, but they were not randomized, thus making this a quasi-experiment. A pretest is given to determine if there is reliability in the test. 10 people were given a questionnaire of sorts that asks them measure openness and dominance. Results matched hypotheses. Subjects were put into 4 different conditions, run with 6 – 8 people at a time. They are never shown to be equal or unequal groups, making this, likely, a weak quasi-experiment. Six specific variables are looked at—1) general attitude toward salespeople, 2) perceptions of the product being sold, 3) interaction between the salesperson and the customer in the tape, 4) general buying behavior of the respondent, 5) probablility of purchase of the product in the tape, and 6) perceptions of the sales person being depicted in the tape. Because there are over 80 respondents and only 6 variables, this meets the ratio Lauer and Asher are looking for. Overall, this appears to be a somewhat effective weak quasi-experiment.

Barry Kroll – “Explaining How to Play a Game: The Development of Informative Writing Skills”

A literature review explains the variables that have been discovered in regards to children explaining how to play commonplace games. Four (4) variables are explained: 1) the amount of relevant information included in the instructions, 2) the kinds of rules that kids of various ages could explain well, 3) the extent to which students provided orienting details, and 4) the degree to which writers used elements of an abstract and formal approach when explaining the game. 133 participants were used, thus meeting L&A’s variable to participant ratio. Subjects were NOT randomly selected, thus making this, at best, a quasi-experiment. They came from English classes, determined to be “normal” students, were native English speakers, and understood the basic rules of the game they needed to explain.

There are 10 elements to the game that need to be explained. Researchers made each student take a test about the game they explained in order to determine that they understood it in the first place. Only those who scored 90% or better were allowed to be counted in the research. This eliminated 16 participants. The testing of the multiple choice test is suspect. The researchers had college seniors answer a multiple choice test about the rules of the game, but they had never heard of the game. When these seniors only scored 13% on the test, it was assumed that the test was valid.


  1. This comment has been removed by the author.

  2. Curtis:
    I like how you briefly described all of the items missing from the study (ex. Detailed account is given of the Minimal Manual strategy that was developed prior to this research study. However, no description of subject selection is given. It is impossible to tell if the subjects were randomized. No hypotheses are given.) I looked back at my own blog and saw that when I was commenting in a similar vein, it takes me forever to say it. I appreciate how you get to the point. In terms of what you are saying, I agree that Carroll may have thought he was doing a cause and effect/quasi-experiment study, but he did not use intact groups or establish interrater reliability. It was like he skipped the heart of the study in favor of the results. Notarantonio & Cohen also had issues with a very weak quasi-experiment in that they never proved why they had established the groups they did (neither equal nor unequal), though they did test variables that met L&A's ratio. Lastly, Kroll used the most ELABORATE mechanisms to prove their point - too long to list here, but then chose to basically skew the results by not telling the college students the rules of the game, and then when they tested low, surmising that that was as it should be. I agree that that was odd. Basically, none of the examples seemed to be that great to me.