Tuning in to Soundwriting

Sound and Access: Attuned to Disability in the Writing Classroom

by Dev K. Bose, Sean Zdenek, Prairie Markussen, Heidi Wallace, & Angelia Giannone

3. Access, Transformed

Accessibility is not very well-integrated into theories of multimodal and sonic composition. Put simply, scholars who write about sound tend to assume that everyone can hear and speak well (or well enough to participate in the standard podcast assignment). While many scholars offer compelling insights about composing with sound (e.g., Rodrigue et al., 2016), they may have nothing to say about accessibility or disability. Or accessibility may be reduced to a footnote (McKee, 2006).

Accessibility tends to be an add-on in multimodal composition classes. We typically instruct students to design their compositions and then, once their works are "finished," to make them accessible. While a script that will become the basis for an accessible transcript and a caption file might develop along the way, captioning is still almost always treated as a separate intellectual and practical process. Students typically create their videos with the expectation that they are designing for a hearing audience. They may also be instructed to caption their videos, but captioning is not usually treated as part of the design process itself. Rather, it is treated as a separate accommodation, a finishing touch, an add-on. Large-scale professional captioning is always performed by third parties who receive the completed work from the producers and caption it independently. The separation of producers from captioners, programs from captions, is at the heart of one important critique of captioning practices (access Udo & Fels, 2010). But even when the designer is also the captioner, as in a classroom context, captioning may nevertheless be cut off from the creative work of making multimodal compositions. In other words, captioning is usually understood as a simple act of copying, an objective practice of unreflective transcription, so easy a machine can presumably do it (consider YouTube's autocaptioning technology).

Can captioning be more than a mechanical and uncreative act? We think so. We are interested in what it means (and how it sounds and feels) to start with accessibility, to hold it at the center of our pedagogies. Instead of a checklist or add-on, we imagine accessibility as a mindset, a way of thinking more broadly about the complexities of access and diversity. Transforming sound into writing (in writing contexts, a process called transduction) can be quite challenging, full of uncertainty, subjectivity, and multiplicity. Even when it feels straightforward ("just copy down what the speaker is saying"), the processes of captioning are not straightforward but rhetorical through and through as choices are always made, sometimes automatically, about how to present speech in writing. In the case of nonspeech sounds, captioners have immense control over which sounds should be captioned (because not all sounds are captioned) and how to caption them. The caption track is not an objective record or reflection of the soundscape so much as a transformation of it, a new text—a mode for new or different experiences, a retuning of meaning, affect, and time. One of us has written a book that explores these complexities (Zdenek, 2015). Sean Zdenek's Reading Sounds: Closed-Captioned Media and Popular Culture offers a humanistic rationale for captioning, an argument for how captions contextualize, clarify, formalize, equalize, linearize, time-shift, and distill sounds.

When we transform access in the classroom from a marginal to a central concern, the complexities of and relationships between sound and writing resonate. How do we embody sound in writing, especially when "we are so limited in how we use language" (Waterman, 2017, p. 129)? How "might we produce sonorously resonant words for (silent or sonorous) reproduction by the reader" (p. 118)? How do we "engrave the sound itself into the page" (Henderson, 2017, p. 147)? These questions, though written from the perspective of sound, poetics, and performance studies, reflect the concerns of both soundwriting scholars and caption studies scholars under the larger interdisciplinary umbrella of sound studies. By foregrounding captioning (and access more broadly) in our pedagogies, we address the central questions that motivate sound studies. By working through problems of access and universal design, we broaden notions of audience and diversity.

In what follows, we describe how three of us have begun developing a college-level curriculum for analyzing captions as a means of teaching lessons about close reading and cultural analysis. Drawing on a handful of films and television shows, we argue that this curriculum can build students' literacy by promoting metacognitive (i.e., self-reflective) awareness. To gauge the effectiveness of this curriculum, instructors were surveyed on how they use captioning in the courses they teach. The survey was part of an IRB-approved study on disability and accommodations in writing classrooms that one of us (Dev Bose) has been conducting. That long-term study focuses on the utilization of technology for deep augmentation of learning styles and is critical of disability myths (Wood et al., 2017).3 3 To provide more background: Since 2014, Dev Bose has been studying the effects of collaborative technology on accessibility in digital platforms at the University of Arizona. Participants include 8 to 14 instructors of record, 4 to 8 administrators in a writing program (including staff), and 5 to 30 undergraduate students currently or previously enrolled in first-year and advanced writing courses. Courses surveyed include face-to-face, hybrid, and online first-year composition and advanced professional and technical writing. Some of this chapter's' coauthors—Prairie Markussen, Heidi Wallace, and Angelia Giannone—are graduate teaching associates who participated in class discussions and interviews. They also recorded discussions in a graduate-level teaching practicum taught by Dev, an undergraduate digital storytelling course taught by Angelia, and first-year composition courses taught by Prairie and Heidi.

Teachers in our study introduced captioning to their students as a simple, straightforward process akin to close reading (or listening). To paraphrase instructions provided to students: Just make sure that what viewers see on the screen reflects precisely what is being heard at the same time it is being heard. In classroom discussions, however, students began to grapple with the idea that captioning is a more complex rhetorical process. The following audio clip is an example from coauthor Angelia Giannone's digital storytelling class, in which students are introduced to the importance of captioning by watching a video clip and then writing out their transcriptions.

Captioning Explored: Transcript

Angelia: Did everyone get a good caption? Do we want to rewatch that clip again? No? Okay. Does anyone want to share their captions? I think that there’s a lot of really interesting stuff in that clip.

Speaker 1: Do we just write down [inaudible]?

Angelia: For the caption?

Speaker 1: Yeah, I mean [inaudible].

Angelia: It's like however you think that, that information should be conveyed. So how do you describe the tone of the music through words? Right? Sometimes we see this in movie captions, so there'll be, like, intense drums beating in the background or something like that. Right? Or it'll be, like, happy pop music—pop music playing in the background or something like that. So what, what sorts of words do you think you can use to capture the feeling of the song?

Speaker 2: So we just do this about the music and not the video?

Angelia: It's about everything in the video. So imagine that you're captioning the video. You can pretend like the captions that are there aren't there because there was obviously a lot of information from the video that wasn't captioned, but it's kind of up to you guys to determine if that information is important. So imagine that you have no hearing, no sound. What would you want to know from the captions? Yes, Lucas.

Speaker 2: I think it's important that they didn't caption their initial question, just the response. So, like, if the person didn't hear, they wouldn't know what they're asking, just what their answer was—

Angelia: [affirmatively] Mmhmm.

Speaker 2: They never [captioned], like, "We're just taking pictures of things we think are beautiful.""

Angelia: Yeah. Other than that first caption, right? So, we see it in the beginning, but then it's not there again. Do we think that's an important thing to caption each time they say, "I'm here to caption something I think is beautiful."

Speaker 3: Just once maybe, and then have everyone's reaction [inaudible].

Angelia: Okay. Do we think that the reaction would be more powerful or more meaningful without the added captions? Imagine if you're watching this with no sound and you just see people's reactions.

Speaker 3: [inaudible] What do you hear, like, what did that person say to you? [inaudible] are you happy? They'd want to know what they said.

Angelia: You'd want to know. Yeah, okay. What about facial expressions? Did anyone write anything in their captions about facial expressions or nonverbal things? Does anyone want to share?

Speaker 4: I wrote that this might change their day to a whole other level. Like, this sentence might just change their day if they're having a bad day.

Angelia: Okay! So you're capturing kind of like the result of it, sort of? How it might make the interviewee feel? [inaudible]

Speaker 3: When the compliment was given, they tended to look away from the camera as soon as it happened, so.

Angelia: Yeah, right? There's kind of that, like, immediate moment of embarrassment where someone you don't know compliments you and you're kind of like— [audio quickly fades out]

As discussion continued, students' ideas evolved rather quickly. Questions emerged as students audited a rich variety of sounds including scripted dialogue, music, and sound effects: What exactly should captions say? Should captioners write everything down or only the sounds that are most significant (access Zdenek, 2011)? If the latter, how does one define "significance"? Discussion concluded with evidence of rhetorical decision-making and the need for metacognition.

There was a similar trend in Dev's teaching practicum course, which is designed for first-year graduate student instructors enrolled in the University of Arizona Writing Program. Prior to the following audio clip, Dev played a video clip twice—first with captions, then without. In what follows, he asks students to compare their perceptions.

What Not to Exclude in Captions: Transcript

Dev: Um, I don't know. So just now, we watched it without sound but with the captions. And before, like in the beginning, I had shown you guys no captions but with the sound—like, before, like, the exercise. What do you guys think? Like, is it funnier? Is it—how is it different than, like, to see it with no sound but with captions? Is there anything missing or is there perhaps something to be gained? What do you guys think?

Speaker 1: I think there's emphasis that's missing in tone, right? [other students expressing agreement in background]

Dev: Yes.

Speaker 1: I mean, like, with the Indian conversation, you don't even need tone to understand that it's an uncomfortable conversation. But in the interaction between the dudes, like, we know the context, but it's not totally clear just how exasperated they are with him or how contrite he is. You know, there’s no tonal—

Dev: You can't read that within the subtitles because there's, like, no tonal emphasis.

Speaker 1: Yeah.

Dev: Yeah, okay, I can see that.

[brief crosstalk]

Speaker 2: Oh, you go first.

Speaker 1: I was going to talk about something a little different. So, if you had anything—

Speaker 2: Oh, I was just saying the impact of humor, kind of. Because you can't see their timing as much with that conversation.

Dev: Yeah.

Speaker 2: Like, do certain jokes, especially for Denise, her landing of jokes— [audio quickly fades out]

Notice that when the first student responds, she focuses on rich variances in emphasis and tone that appear to be absent, or at least implicit, when captions do not exist. At the same time, she recognizes the opposite might also be true: that situational context is inferred from character dialogue and that captions aren't typically necessary for interpreting emphasis and tone. Another student observes the impact of humor and timing and how they are less prevalent for viewers who depend on captioning. Put simply, students considered how rhetorical decision-making is the result of thinking through what to caption, how to caption, and why.

This observation correlates with coauthor Sean's argument that captions "flatten" meaning (Zdenek, 2015, p. 44). Speech sounds are "homogenized"—specifically, manner-of-speaking identifiers are rarely used in captioning, so all speech sounds end up "sounding" the same. As students noted, captioning is not as straightforward as it first appears, especially when trying to account for humor, tone, and other nuances of spoken language within the constraints of space and time that captioners must navigate.

More broadly, the practice of captioning raises important questions about the nature of and audiences for soundwriting. We realized that we needed a clearer definition of soundwriting. In this recording, coauthors Dev, Prairie, and Heidi discuss some procedural definitions of soundwriting.

Recapping Soundwriting Pedagogy: Transcript

Dev: I'm curious: so one of the things that I'm interested in, I want to know how each of us define soundwriting pedagogy. What does it mean for each of us? And I'll just start. For me, I believe soundwriting is when the instructor uses sound of any kind during his or her teaching of writing, such as audio feedback on essays or podcasts, that students are supposed to listen and respond. I'm particularly interested in how accessible this type of instruction can be as well as to what extent the material learned during this instruction may be applied outside the classroom.

Prairie: Yeah, I think I agree with most of that. I define it, like Dev says, kind of as any use of audio in the classroom or any use of audio in feedback, which I think, Dev, you also just mentioned. And for me—this kind of gets into the next question—but for me, it's primarily about using podcasts, using the audio feedback, and then sometimes using music, but I'll kind of wait on that. We'll get into that more.

Heidi: Yeah, I'm really interested in podcasts as a text, something that can be analyzed just like anything, like a short story or poetry or even, you know, pieces of news or something like that. And I think that certain podcasts definitely cater and present themselves as a text, especially the ones that are highly edited and formed into a certain kind of narrative.

Prairie: Yeah. And just to piggyback off of that, not just a narrative, but an argument.

Heidi: Absolutely.

Prairie: And I've been thinking about this in terms of the public argument assignment that's coming up in our classrooms and the fact that it's going to be National Poetry Month starting April 1st and trying to think about how to use slam poetry, how to use performance poetry in the classroom to show an argument, specifically. So I think that can be really useful too.

Dev: I think so too.

Our definitions of soundwriting are built around listening in specific contexts, including giving audio feedback and recording podcasts using multiple modes and formats that are accessible to the widest population. Along these lines, we consider podcasts as applied examples of soundwriting, especially those which are highly edited, given the rhetorical nature of choosing how and when to make certain revisions over others. We also conclude that certain types of sound-based narratives, such as slam poetry, function as arguments.

Within these contexts, we argue that spaces need to be built around the bodies inhabiting them. Without an acknowledgement of these bodies, these spaces become structurally uninhabitable and exclusionary. As a rhetorical exercise, captioning can powerfully capture and transform sounds. This promotes multimodal listening, a practice that considers how sounds shape (and are shaped by) contexts, objects, and experiences (Ceraso, 2018).

We also considered several classroom applications of soundwriting in terms of accessibility:

Soundwriting Applications: Transcript

Heidi: Yeah. I will often assign a podcast ahead of time, and then we go over that podcast by—I'll play certain clips that I've prepared beforehand that I want to bring their attention to, and then we discuss it just as if we would, you know, a short story where we read one paragraph and then we discuss that paragraph, in the sense, I'll just play five minutes of a clip, and then they will take note of different moments and aspects of the podcast and analyze it accordingly. Yeah.

Prairie: I do something similar to that with things like the TED Radio Hour or Latino USA, Snap Judgment, Fresh Air sometimes. And the way that I use the podcast is I usually have my students listen to maybe about 20 minutes of it—which, to be honest, sometimes can be a bit much. They get a little bit bored or they're not able to actively listen like they could maybe in the first five minutes. But what I do is I use it to show how you can construct, maybe not necessarily an argument, but you can link multiple sources through one common theme or topic. And I think it's good for them to hear the way that that's done, and to sort of put themselves—I ask them to put themselves in the role of narrator. So, like, they are the narrator, and so how is the narrator smoothing transitions over? Yeah. How is the narrator introducing new sources? So I think that's useful as opposed—it goes alongside with giving them written examples, but I think it's useful for them to hear it in that way as well.

And then I also give my students the option of receiving audio feedback on their assignments. So they still will get, like, the in-text comments through Turnitin, but they will also either get summary written comments or summary audio comments, and they get to choose. So I don't force it on anybody if they don't—because I've had some students who don't like it; they think it's not helpful. But most students appreciate the tone of the feedback, that they can hear my tone, they can sense if I was being kind of funny or stern or whatever it is. They can sense that better than in a written text.

Dev: Have you ever— [audio quickly fades out]

In that recording, coauthor Heidi explains how she uses podcasts as listening assignments given ahead of time as homework. She plays short video and audio clips in the classroom in order to stimulate discussion through close analysis. She also provides transcripts and captions for each clip. These visuals emphasize the rhetoric of communicative delivery, especially when pauses, hesitations, music lyrics, or sound effects are included in the transcripts. The transcripts, then, function in multiple ways in the classroom, emphasizing nuances that students may have missed while listening and providing a visual accompaniment to those with visually oriented attention. The point of the podcast assignment is to reach a level of inclusivity for various kinds of learners. Coauthor Prairie notes that time constraints are essential when using podcasts due to wavering attention beyond five-minute segments; this instructor also indicates that multiple stories can be linked through "one common theme or topic." Students place themselves in the role of narrator by analyzing how to smoothly transition between topics and introduce new sources. Additionally, Prairie gives the option of audio feedback to her students, either in addition to or instead of written feedback. Aligning with current findings related to audio feedback in writing classes (McKittrick et al., 2014), Prairie indicates the value of such feedback in terms of students' awareness of tone that may not have been perceived in written feedback alone. At the same time, students are always given the option to choose what kind of feedback they would like to receive so that those who prefer text over audio are provided with the same level of instruction.

In this section, we have highlighted a handful of representative soundwriting pedagogies, including captioning and audio feedback on student writing. Central to our arguments has been the conceptualization of the many ways accessibility can be foregrounded in our pedagogical practices. To that end, we have argued for the importance of designing sound pedagogies with universal design principles proactively rather than retroactively in mind, the necessity for teachers to clearly define soundwriting within their specific classroom contexts, and the importance of recognizing how sound can be harnessed to guide students toward making powerful rhetorical decisions. At its core, this section has endeavored to galvanize teachers to consider the myriad abilities students possess and to move away from teaching to and for default bodies.