Best Years Home
About Mike
More Free Columns
Links Page
Friends of Best Years
Trading Post
Old Stuff
Quick Reference
Abstract Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 6 Chapter 7 Chapter 8 Bibliography

CHAPTER V

PRELIMINARY EMPIRICAL SUPPORT FOR PERSON-BASED RESPONSE

 

Overview

The empirical research project presented in the next chapter (for clarity, referred to as the "major study" in this chapter) is the culmination of a number of other projects. These preliminary works, that have honed the research questions and statistical design to follow, are presented briefly in this chapter.

 

"Evaluating Teacher Comments: A Classroom Assessment:"

West Texas A&M University, November 1994

 

In this assessment of my classroom commenting practices, I presented surveys to two classes of beginning composition. Students were asked to rate ten statements about my written comments on their first essays of the semester using the same Likert scale described later in the major study (for a look at the survey questions, see Appendix G). Class A returned 20 forms, a response of 80%, while class B returned 22 forms, an 85% response rate. Overall, the students affirmed my commitment to such person-based issues as praising student successes, caring about student content, and creating student confidence. Also the study helped refine the statements I would use in later surveys; six of the ten are included in the major study. In addition, this exercise suggested that student grades do make a difference in student perceptions of their teachers (the numbers were not large enough to create statistical significance). While both Class A and B had about the same number of D essays, Class B had four times more A papers (8 rather than 2). Class B averaged 88.6% on "agree" statements, while Class A averaged 82.5%. Class B had only one "disagree" statement, while Class A had seven. The biggest difference on a single statement was on statement one: "My teachers' comments made me believe I can do better on the next essay." Ninety-five percent of Class B agreed with this statement, while only 80% of Class A did. From a practitioner's perspective, this suggested that students with lower grades would require extra encouragement to produce the same effect as with their more successful peers, and from a researcher's perspective, I began to see that students' grades might become a confounding variable for measuring student motivation (an hypothesis confirmed in the major study).

 

"Evaluating Teacher Comments: A Focus Group:"

West Texas A&M University, April 1995

 

Lawrence Frey and others describe a focus group as "a relatively open discussion about a specific product or program among a small group of people (usually five to seven) led by a facilitator, who introduces topics, encourages participation, and probes for information in a flexible, interactive way to elicit subjects' genuine views" (350).

Usually, focus groups are not chosen randomly, their statements/questions are not operationalized, and the results are not subject to statistically significant quantitative analysis. However, as chapter two points out, they offer useful qualitative data that can give depth to a quantitative study. In this first of two focus group studies, I enlisted the help of a small (about 12 students present) introductory literature class. In the spring of the year, we went outside, sat on the grass, and talked about written comments they received from former teachers (these could be high school or college).

When I asked for a show of hands of students who had teachers who wrote three or more sentences of comments on their papers, I received no response. The majority told me that most often they received no comments, only marks identifying things like a run-on sentence or misspelled word. How did this make them feel? "It sucked; I hated it," said one student. "Half the time, I didn't have a clue of what she meant," added another. Many of the comments students received offered no explanation. "My teacher would write `you should say it this way,'" said a student. Most pupils seem to interpret words like these to mean "do it my way simply because I like it better." This frustrates the student writer. "How can I read her mind?" complained one young man.

Students felt that the tone of teacher comments was mostly negative. "Some never say anything good," said a girl. "Don't use red ink," added a boy. "Are there other things that can make comments more positive?" I wondered out loud. "Use short words of praise," advised a girl. "Yeah, like `Wow," `Yes,' `Well put,' or `I agree' said others. One student told of a teacher who praised his use of supporting quotes. Another said she was complimented for good organization. A male student thought that teachers should let students know that they are human too. When I asked for an example of how this could be done in a written comment, he was unsure.

I concluded by asking what we had not covered. What one piece of advice would students like to give prospective teachers about their written comments? "Say something about what I have to say," said a female student. "Sometimes I feel that teachers are not really listening, only grading." I asked for additional thoughts, and, after a long pause, I received one more piece of advice: "If a teacher doesn't say anything good or bad, we will do the same thing on the next paper," offered a student.

Although this study lacked the sophistication that came with more training in focus group strategies, including the fact that one's current students might feel uneasy in the presence of their teacher and thus be less than candid in their responses, the results were useful. These students voiced concerns of such postmodern issues as clarity, excessive criticism, silencing of student voice, and the teacher's role as a real reader, even hinting at the concept of consubstanciality (teachers are human too). Finally, the responses about number of comments became an operationalized statement in the later quantitative surveys, with results that verified these student's concerns. It seems that students in general desire more rather than less teacher comments.

 

"Words That Teach: An Analysis of Teacher Comments on the

Essays of Beginning College Composition Students:"

West Texas A&M University, May 1995

 

The major contribution of this study, which analyzed the attitudes of 50 first-semester composition students, was the sophistication of quantitative analysis. This was my first attempt to correlate student motivation with various person-based strategies, and the first to run a multiple regression test to see how these strategies collectively predict student motivation. Also, the 41-item survey (see Appendix H) contains most of the statements that, with some refining, appear in the project reported later in the major study. In addition, the survey vividly demonstrated how teacher ambiguity between text-based and person-based response has resulted in the same ambiguity in their students. When asked the most important purpose of writing, the number one answer was "communicating with an audience" (48%), and "persuading an audience" was picked number one as the second most important purpose (34%), statistics which suggest a postmodern, person-based understanding of audience. However, many students listed "constructing error-free sentences as the most important purpose of writing (34%), and the second most important (30%), statistics which suggest a text-based understanding. And while 54% of students said that teacher comments reminded them that they addressed a real audience, less than 50% described teacher comments "as a running conversation between the teacher and me in which the teacher responded to the ideas in my essay."

There were obvious weaknesses with this preliminary study. One, the sample was very small with only 50 respondents. Two, the survey was administered by the students' own teachers, which threatens bias and, therefore, internal validity. Three, the teachers (teaching assistants) knew the purpose of the study, which introduces the same threat. And finally, the survey was conducted late in the semester. By this time, less successful students may have dropped the course, which might account for overly optimistic survey results.

 

"Teacher Responding Strategies: A Focus Group"

Texas Tech University, March 1996

 

In this study, a colleague and I convened a focus group consisting of four students plus ourselves. The students were chosen from previous classes because we anticipated that these individuals would be both candid and loquacious about their opinions. Of the five invited, four actually attended. Of these, three had received a grade of B in the first semester of college composition, and one had earned a C. Two were male and two were female. At the time of the focus group, all were second-semester freshmen, 19-years-old, and enrolled in a composition course. None were in sections that we taught. David was a pre-med major who had taken advanced honors English and advanced technical writing in high school. He presented himself as confident about his own writing and professed himself to be insensitive to teachers' comments. John, undecided in his college major, described his previous experience in English classes as "normal" and described writing as "hard." Emily, a Mexican-American and journalism major, was the only minority student in the group. She also had taken honors English in high school and seemed comfortable with her writing. Janet, an interior design major, had taken honors English throughout high school, but was the least confident about her own writing. Both she and David remarked about difficulties they were having with their present composition courses.

Aside from open-ended questions at the beginning and end (designed to make sure the students were able to voice their own agendas), interviewer questions sought to examine the following variables:

The focus group met on a weekday in the early evening, and began with pizza and soft drinks provided by the interviewers. Students were told the entire session would take only one hour, so the discussion began exactly 15 minutes after they arrived. During the 45-minute session, each interviewer asked predetermined questions. Based on students' replies, we echoed, probed, and reworded questions to encourage full responses. Students also were encouraged to respond spontaneously to the thoughts of their peers. I started the session with some general remarks and a brief free-association exercise. Then I led the questioning for 25 minutes, focusing on each of the commenting variables. My research associate took over for 15 minutes with questions relating to her research interest, and I then wrapped up the session with 5 minutes of open-ended queries. The focus group was audio-taped, transcribed and then coded for analysis. Following are some of the results.

Table 5.1 shows the type and frequency of affirming remarks made by each student about particular teacher commenting variables, whether phrased in a positive or negative manner. A plus sign ("+" in the table) means that the student spoke favorably about the teachers' use of this particular commenting strategy. A minus sign ("-" in the table) means that the student found it unhelpful when a teacher did not use this strategy (for instance, David said four times that he found it helpful when his teachers commented at the draft stage of writing, and, similarly, he mentioned four times that he found it difficult when teachers did not do this). The columns and rows of the table are both tallied, showing the combined count of student comments on each variable.

In the few cases where students remarked on a strategy that was neither helpful when the teacher employed it nor unhelpful when the teacher did not, but was, in fact, never helpful, the response is noted in parentheses. For instance, David said on two occasions that teacher comments in general were never helpful (something he later contradicted in other remarks). These comments (there were only a total of four of them) are listed but not tallied on the table.

As the table shows, students spoke most often about the need for positive feedback (26 student responses). The second-place category was a tie between comments on drafts and comments on content (each with 24 responses). Students mentioned concerns linked to teacher clarity 21 times, and the number of teacher responses (about anything) and the appropriation of student voice 20 times. Clarity and number of comments often commingled. Students said in general that more comments were better than less, and they particularly disliked one-word comments because of their inherent ambiguity.

 

 

 

Table 5.1: Types and Frequencies of Affirmative Student Responses to Teacher Commenting Strategies ("+" = positively phrased; " -" = negatively phrased)

 

 

 

 

David

 

John

 

Emily

 

Janet

 

Total +,-

 

Total all

 

Comments in general

 

+5, -1, (2)

 

+2, -0

 

+1, -0

 

+1, -0

 

+09, -01

 

10

 

Clarity

 

+1, -1

 

+0, -4

 

+0, -5

 

+3, -7

 

+04, -17

 

21

 

Content

 

+6, -0

 

+7, -0

 

+0, -0

 

+9, -2

 

+22, -02

 

24

 

Mechanics

 

+4, -0

 

+2, -0

 

+3, -0

 

+1, -0

 

+10, -00

 

10

 

Positive

 

+7, -0

 

+2, -1

 

+2, -4

 

+5, -5

 

+16, -10

 

26

 

Confidence

 

+0, -0

 

+1, -1

 

+0, -0

 

+2, -0

 

+03, -01

 

04

 

Draft

 

+4, -4

 

+5, -1

 

+3, -0

 

+4, -3

 

+16, -08

 

24

 

Number of comments

 

+0, -2

 

+4, -4

 

+1, -4, (2)

 

+2, -3

 

+07, -13

 

20

 

Organiza-

Tion

 

+1, -0

 

+2, -0

 

+0, -0

 

+1, -1

 

+04, -01

 

05

 

Appropria-

Tion

 

+0, -5

 

+0, -4

 

+0, -7

 

+0, -4

 

+00, -20

 

20

 

total +,-

 

+28, -13

 

+25, -15

 

+10, -20

 

+28, -25

 

+94, -53

 

164

 

total all

 

41

 

40

 

30

 

53

 

 

 

164

 

The results of the focus group suggest support for the variables under study. These students said they wanted teachers to be clear, to respond to content, to praise them for things well done, to strengthen writer confidence, and to refrain from appropriating their own voice. Generally speaking, students found teacher comments helpful, and said they want more of them, especially on their drafts.

One limitation of the data in Table 5.1, however, is the possibility of misleading counts of affirmative responses, because some of the same topics arose in discussions led by each researcher. Thus our two-researcher approach may have exacerbated the already present danger of trying to quantify students' affirmations of any predetermined agenda. It is noteworthy, however, that overall there were exceptionally few instances of the students disagreeing in principle with the benefit of any of these commenting strategies.

A particularly animated discussion arose about the value of comments on drafts, and yielded an important insight. Earlier in the focus group, when the discussion had centered on graded papers, the students expressed a clear need for positive feedback, and a sensitivity to criticism. Emily put it like this:

I think the comments that get me mad straight off just say "change this, change that, this isn't good." You know, everything negative. If they could look at it like "I like this, but why don't you change it a little bit like this," something like that, but when professors, teachers, just say "it sucked, change it" OK, it kind of intimidates you.

 

Janet agreed, saying "for me it is better if they put more positive comments [than negative ones], even if there aren't as many." Yet later when our discussion returned to the issue of criticism, but this time in the context of drafts, the students did an about-face. "I think they should bash the draft," said David, bringing both laughter and agreement from his peers. Janet added, "I really do [agree with David] because that would help a whole lot. Then you would know what to change. If they don't bash . . . a certain thing, then you can say, 'that's probably good.'" They remarked that comments on a draft helped them anticipate what grade they would eventually receive. Similarly, David described the disadvantage of comments on a final draft, saying "the only time it could help is the next paper, but usually the next paper is a totally different thing." Janet pointed out that she probably wouldn't refer back to earlier comments, because "you're going to have that other one stashed, and you're not gonna go through the trouble of finding it."

Regardless of at which stage they were given, more comments were generally preferred to fewer, and students especially appreciated those that revealed the teacher's attention to the writer's ideas. Janet spoke enthusiastically about receiving comments from her teacher that showed he was "totally into the paper when [he was] reading [it]." Students expressed their frustration about a teacher jotting a single word either in the margin or at the end of the paper. These single word comments, such as "fragment," or "good," lead to confusion, because students struggle to understand the implications of the cryptic comment. As Emily said, "you're like, well, is it good or is it bad?"

Of course, any qualitative study, including this one, cannot be bold in generalizing to a larger population. The small number of students and the potential bias of interviewer/researchers both threaten external validity. Similarly, quantitative analysis of these student responses does not necessarily reflect the true hierarchy of all student concerns. Other students may be more concerned with mechanics than content or with organization than positive reinforcement. Hence, this study needs follow-up research to test these hypotheses with a large enough sample to show statistical significance.

 

"Teacher Commenting Strategies: How Our Students

Perceive Us" Texas Tech University, April 1996

 

My final preliminary study served as a pilot study to refine the survey instrument used in the major project. Also conducted with a colleague, this study used as a sample students in four sections (74 students) of Advanced College Rhetoric (the second

 

semester composition course) who were selected on the basis of their instructors' willingness to take part in the project (a nonrandom, convenience sample). The 47-item questionnaire (see Appendix I) includes all the statements that appear in the final study with the exception of eight propositions measuring student motivation, which used a seven-point semantic scale taken from Dianne Christophel's study of the relationships between teacher immediacy behaviors and student motivation (see Appendix IX, 32-40).

The research questions are the same as those in the major study and thus will not be restated here. Similarly, the statistical analyses are reflective of those in the later study, and since that project has greater validity (due to a better, more random sample), I will not report the results of its preliminary counterpart in this summary (for a chart of basic correlations see Table 5.2).

Most instructive in this study were the failure of two variables to show correlation. Both teachers' use of draft comments (as opposed to comments on final papers) and student motivation did not show strong covariance with the other variables (person-based response strategies did not seem to increase student motivation, and teacher practice of commenting on student drafts did not seem to affect either students' sense of motivation or the usefulness of comments). Regarding the use of draft comments, it may be, as suggested in Chapter IV, that responding to student process work is only effective if one responds to students who are themselves in process. Thus the statements in the survey that measure the degree to which teachers communicate faith in their students as developing writers may be a better predictor of process pedagogy than

 

Table 5.2 Correlation of commenting strategy variables (perfect correlation = 1.00).

 

 

Vars.

 

Clarity

 

Reader

 

Praise

 

Voice

 

Confi.

 

Comp.

 

Useful

 

Motiv.

 

Draft

 

Clarity

 

1.00

 

.74**

 

.55**

 

.65**

 

.68**

 

.43**

 

.44**

 

.24

 

.40**

 

Reader

 

.74**

 

1.00

 

.63**

 

.69**

 

.76**

 

.49**

 

.58**

 

.24

 

.30

 

Praise

 

.55**

 

.63**

 

1.00

 

.66**

 

.58**

 

.59**

 

.50**

 

.22

 

.27

 

Voice

 

.65**

 

.69**

 

.66**

 

1.00.

 

.74**

 

.60**

 

.53**

 

.24

 

.37*

 

Confi.

 

.68**

 

.76**

 

.58**

 

.74**

 

1.00

 

.53**

 

.60**

 

.32*

 

.35*

 

Comp.

 

.44**

 

.49**

 

.59**

 

.60**

 

.53**

 

1.00

 

.65**

 

.52**

 

.35*

 

Useful

 

.44**

 

.58**

 

.50**

 

.53**

 

.60**

 

.65**

 

1.00

 

.59**

 

.15

 

Motiv.

 

.24

 

.24

 

.22

 

.24

 

.32*

 

.52**

 

.59**

 

1.00

 

.27

 

Draft

 

.40**

 

.30

 

.27

 

.37*

 

.35*

 

.35*

 

.15

 

.27

 

1.00

Clarity =teacher writes clear comments

Reader =teacher presents self as real reader

Praise =teacher praises things done well

Voice =teacher encourages student voice

Confi =teacher inspires student confidence

(*p<.01 **P<.001)

Comp =student perceives teacher as competent

Useful =student sees teacher comments as useful

Motiv =students are motivated by comments

Draft =teacher comments on preliminary drafts

 

the practice of commenting on process drafts. Ideally, person-based teacher/responders will do both. They will both require process work and treat their students as the same.

Regarding the finding that student motivation did not correlate with person-based strategies, it is possible that while students may perceive comments as useful (the variable that did correlate with all commenting strategies), they may not also be motivated by them. However, a second possibility is more likely. This variable was measured on a different scale than the others (a seven-item semantic scale--see Appendix IX, questions 32-40). It is likely that students misunderstood this scale and thus the results are skewed. In fact, the major research project shows this to be the case. When presented with the same Likert scale, students reveal a level of motivation that correlates strongly with all the other variables.

 

Conclusion

The research questions, survey statements, and methods of statistical analyses reported in the next chapter did not just happen. They are the legacy of the teacher assessment, qualitative, and quantitative studies presented in this chapter. Without these preliminary attempts, the major project would lack both reliability and validity. With them, the next chapter is able to offer substantial empirical support for person-based response.

On to Chapter VI

Respond or view responses to my dissertation.