Thursday, March 26, 2009

Week 12: True/Quasi Experiments

Lauer and Asher – True Experiments

-experimental research actively supplies treatments or condition such as sentence-combining, planning strategies, or word processing to determine cause-and-effect relationships between treatments and later behavior.

True Experiments:

-actively intervenes systematically to change or withhold or apply a treatment to determine what its effect is on criterion variables.

-uses randomization by which subjects are allocated to treatment and control groups.

-give hypotheses in the form of questions or positive statements

-require a null hypothesis

-must have a randomized selection of subjects. There can be no expected differences among the subjects on any variable.

-have criterion variables to compare the control group and the treatment group with


-DO NOT randomize subjects because of restrictions and use already established, intact groups

-have at least one pretest or prior set of observations on the subjects in order to determine whether the groups are initially equal or unequal on specific variables tested in the pretest

-have research design hypotheses that account for ineffective treatments and threats to internal validity.

-are either strong or weak. Strong have proven equality in pretests that determine the two groups are very similar, despite not being randomized. Weak quasi-experiments are unequal in at least one of the variables.

-must show whether or not groups are initially equal or unequal

-still must show interrater reliability and stability of variables

-risk internal validity because of the lack of randomization. However, it is better to leave in outlier subjects than to remove them from the study. The results are more generalizable this way, even if the study is a weak quasi-experiment.

John M. Carroll, et al.: “The Minimal Manual”

Detailed account is given of the Minimal Manual strategy that was developed prior to this research study. A manual was designed to be simple, stripped of verbiage, and user-centered. It was task-oriented, not jargon and information loaded. Description is given about the errors that users made and the adjustments that were made to the manual. However, no description of subject selection is given. It is impossible to tell if the subjects were randomized. No hypotheses are given.

Two experiments were conducted after this preliminary test. In the first, 19 subjects were observed, 9 using a tradition user’s manual, 10 in the Minimal Manual group. Subjects are not randomized, thus, at best, this can only be a quasi-experiment. However, the subjects they used were not already intact groups, thus there is a poor design from the beginning. Interrater reliability is not given and criterion variables are unclear. This experiment is poorly designed, or, at least, poorly described.

The second experiment has four groups that either Learn by Doing (LBD) or Learn by the Book (LBB) and test the Minimal Manual against the traditional manual. Each group, though ,was given different tasks. This automatically makes for a problematic experiment. Also, 32 subjects were tested, slightly smaller than the suggested 40 (four times the number of variables being tested), which hurts the design. Subjects are certainly not randomized, nor did they come from intact groups. Data collection is vague and it is hard to determine interrater reliability.

Elaine M. Notarantonio & Jerry L. Cohen – “The Effects of Open and Dominant Communication Styles on Perceptions of the Sales Interaction”

A history is given of the communication studies as they relate to business sales. This is provided in order to show that variables come from previous studies and are to be further examined in this experiment. A hypothesis is given that says a salesperson can manipulate his or her style to provide for maximum sales effectiveness. An explanation is given between Open and Dominant salespersons—the Open person is gregarious, frank, conversational, unsecretive, and shares personal information. The Dominant is competitive, confident, enthusiastic, and forceful.

The subjects were business students, but they were not randomized, thus making this a quasi-experiment. A pretest is given to determine if there is reliability in the test. 10 people were given a questionnaire of sorts that asks them measure openness and dominance. Results matched hypotheses. Subjects were put into 4 different conditions, run with 6 – 8 people at a time. They are never shown to be equal or unequal groups, making this, likely, a weak quasi-experiment. Six specific variables are looked at—1) general attitude toward salespeople, 2) perceptions of the product being sold, 3) interaction between the salesperson and the customer in the tape, 4) general buying behavior of the respondent, 5) probablility of purchase of the product in the tape, and 6) perceptions of the sales person being depicted in the tape. Because there are over 80 respondents and only 6 variables, this meets the ratio Lauer and Asher are looking for. Overall, this appears to be a somewhat effective weak quasi-experiment.

Barry Kroll – “Explaining How to Play a Game: The Development of Informative Writing Skills”

A literature review explains the variables that have been discovered in regards to children explaining how to play commonplace games. Four (4) variables are explained: 1) the amount of relevant information included in the instructions, 2) the kinds of rules that kids of various ages could explain well, 3) the extent to which students provided orienting details, and 4) the degree to which writers used elements of an abstract and formal approach when explaining the game. 133 participants were used, thus meeting L&A’s variable to participant ratio. Subjects were NOT randomly selected, thus making this, at best, a quasi-experiment. They came from English classes, determined to be “normal” students, were native English speakers, and understood the basic rules of the game they needed to explain.

There are 10 elements to the game that need to be explained. Researchers made each student take a test about the game they explained in order to determine that they understood it in the first place. Only those who scored 90% or better were allowed to be counted in the research. This eliminated 16 participants. The testing of the multiple choice test is suspect. The researchers had college seniors answer a multiple choice test about the rules of the game, but they had never heard of the game. When these seniors only scored 13% on the test, it was assumed that the test was valid.

Week 9: Quantitative Descriptive Studies

Lauer and Asher: Quantitative Descriptive Studies

-Go beyond ethnographies and case studies to further define variables, quantify them (either roughly or accurately) and interrelate them.

-Correlate variables by various statistical means to look for strong, weak, or no existing relationships

-Report statistical analyses on variables

-Are descriptive, not experimental research, because no control groups are created and no treatments are given.

-Have larger number of subjects required than for case studies and ethnographies—this is because variables will be quantified. At least 10 times as many subjects as variables.

-Use subjects selected based on their appropriateness to the variables and their availability

-Divide variables into independent and dependent variables

-Have alternative hypotheses to test the variable against

Brenton Faber: “Popularizing Nanoscience: The Public Rhetoric of Nanotechnology, 1986 – 1999.”

Begins with literature review to give grounding to social construction of arguments and a scientist’s ability to persuade and lend credibility to his work through socially constructed rhetoric. He then shows how popular media is affecting the way the general public and scientists alike perceive science. Hypotheses are given that 1) the introduction of nanoscience would be a social and rhetorical process, that 2) the introduction of nanoscience would create a persona—a presentation of the author in the text—and insertthe work within an existing understanding of science, and that 3) the public image of nanoscale science and technology would emerge transitionally.

History of nanotechnology is given. Faber shows how the media has used terms such as “Buckyballs” and a discussion of cryogenics, cures for cancer, self-repairing highways, bulletproof clothing as thin as a jacket, and affordable energy have “popularized” nanotechnology.

Subject selection is described (though labeled ‘data collection’ in the article). Articles were chosen from a library database and were limited to newspapers, general interest magazines, and popularized scientific publications. Data was analyzed based on theme and rheme and article topic. While this was thorough, there appears to be a noticeable interrater reliability issue here. Faber is the only one determining the “39 representations of nanotechnology in 262 occurrences” and he alone determines the social-rhetorical nature of these articles.

Discussion: I am left wondering what variables were precisely being examined in this study. Other than showing us that nanotechnology is being presented in popular media, I don’t understand what else Faber gives us here. What I would like to know is the how this rhetorical phenomenon within the field affects the way the field is perceived. This is never really decided and what little is given is based on Faber’s opinion. Faber needs to work through his subject selection and determine why they are important and how they relate to the variables that he wishes to explore.

Steven Golen: “A Factor Analysis of Barriers to Effective Listening”

Golen begins by explaining some of the listening barriers that were discovered in Watson and Smetlzer’s study. He suggests that these barriers (the variables) need further study and he claims to factor analyze them in more detail. He also increased the number of barriers from 14 to 25. Other studies are mentioned where barriers are discovered but Golen claims that none of these studies identifies factors or dimensions of the barriers.

Golen makes a very presumptuous statement when he says: “only one instructor taught the class; therefore, all the students received the same instruction.” Subjects were chosen from business communication lectures. Because the purpose of the study was to investigate listening barriers amongst business college students, this seems like a good subject selection. 25 barriers were examined, 279 subjects were chosen. This meets Lauer and Asher’s variable: subjects ratio requirement.

Literature review is briefly given, letting us know where the 25 variables came from—the most frequent barriers occurring in several studies.

The data analysis is a bit confusing to me. I’m unsure what the “loan on a factor” refers to. I believe Golen is trying to say that in order to make managing the data more efficient (easy), worthless variables needed to be eliminated, so they got rid of several after the fact. I just don’t understand how he came to that conclusion. It is found that gender influences two of the six independent variables (barriers) and thus instruction may need to be adjusted to meet that need. Overall, it appears that this meets Lauer and Asher’s requirements, but I believe more description about the instruction students are given about listening is important. It is hard to determine where subjects are at when going into the study because of this.