The terms “multimodality” and, more recently, “translinguality” are now circulating in the discourse of contemporary composition teaching and scholarship. We trace the emergence of these terms in that discourse as a response to events “on the ground”: the development, increasingly global reach, and use of new communication technologies and networks for these; the increasing, and increasingly undeniable, traffic among peoples and languages; and the consequent recognition by teachers and scholars of composition that the assumption of a monolingual and monomodal norm for composition—as communicative practice and terrain of study—is no longer appropriate, if indeed it ever was.1
Click the above image to view a timeline of the terms "multilingual" and "multimodal" in CCC.x
What seem apparent to us both are the following: (1) these relatively recent changes bring into awareness features of all communicative practice that ideologies posing the “norm” of a single, uniform (“standard”) language or mode (hereafter referenced as “SL/MN”) elide, (2) these same changes bring to awareness the presence of communicative practices in the past that SL/MN ideology has suppressed, and (3) currently emergent communicative practices are themselves materially different from past and other communicative practices in ways that challenge both SL/MN ideologies and the practices now identified (ideologically) as SL/MN.
In other words, the various terms and neologistic variants listed in the title of this section represent challenges both to beliefs about the modality and language of all communicative practice [sic] and to communicative practices themselves.
We resist, in short, any understanding that statistically standard language practices are singular either in their linguistic or modal forms, and we resist the understanding that statistically standard is the linguistic or modal equivalent of normal. This ideological formation is two sided and doubly dangerous.3
Definitions and inflections of the terms “multimodality” and “translinguality” in composition scholarship and teaching represent different responses to changes in belief and communicative practice. To illustrate; early on in this project, Cindy cautioned about the conduct of the project itself:
My only concern [. . .] is the limitations of the alphabetic in doing this job well. [. . .] In fact, I suspect that the success of this piece—on my end, at least—will depend on my ability to focus on specific examples/situations that illustrate these limits, or, at least, that illustrate why people (other than academics!) feel so compelled to turn to multiple modalities to make meaning and why academics (especially those who specialize in semiotics) ought to blessed well pay attention to these efforts and take them seriously instead of ignoring/dismissing/diminishing them as somehow less intellectual, less effective, less... (fill in the blank).
So, this piece may well need some online accompaniment—in fact, I think it would be cool to experiment, for instance, with what each of us can—and cannot—say using the different modalities and even perhaps render parts of the argument in multiple ways and using multiple modalities.4
Cindy’s caution draws on at least two definitions of multimodality: (1) as a set of material practices to which people (especially people other than academics) turn to make meaning, and (2) as the belief that such practices might allow composers (the authors) to break out of the limitations of SL/MN—a belief obviously at odds with dominant SL/MN ideology.
We see a concern about treating multimodality as a fixed set of practices in the following exchange. The exchange starts with a caution regarding fetishizing practices, then turns to the strategic advantages and limitations of specific terms:
How do we exploit the shift in perspectives that encounters with unfamiliar language/modal forms can produce without then fetishizing these at the cost of retaining dominant restricted understandings of the familiar? How do we learn to recognize the “strange”/“new” in the “familiar”/“old” and the “familiar”/“old” in the seemingly “new” or “strange”?
Here’s how I had framed this point in an earlier version of our dialogue. I cut this later not just for reasons of length but to avoid making a fetish of my use of the untranslated Molière:
Here’s a dilemma, however: the interest in translinguality and multimodality can be attributed, at least in part, to the increasing impossibility of ignoring changes in communicative technologies and practices, including growing encounters with a diversity of languages and technologies of communication (e.g., digital technologies). So, for example, just as (commonly) the effort to learn an additional language gives learners a greater sense of their “home” language as a language (vs. being just “language”), and hence a sense of choices and options, one powerful effect of peoples’ encounter with new communication technologies is a greater sense of their extant communication technologies as technologies: like M. Jourdain in Moliere’s play Le Bourgeois gentilhomme, who confesses « Par ma foi ! il y a plus de quarante ans que je dis de la prose sans que j'en susse rien, et je vous suis le plus obligé du monde de m'avoir appris cela.» Likewise, we may learn to recognize the multimodality of all communicative efforts by first being trained to recognize modality as a feature of communicative efforts taking unfamiliar form, e.g., those deploying digital technologies. The temptation I’ve already discussed is to then identify modality/multilinguality strictly with use of these forms/languages and not with what seem more “ordinary” communicative efforts—e.g., with my introduction of French language text in this paragraph (“how multilingual”) and not with any of the text that preceded it. How do we exploit the shift in perspectives that encounters with unfamiliar language/modal forms can produce without then fetishizing these at the cost of retaining dominant restricted understandings of the familiar? How do we learn to recognize the “strange”/”new” in the “familiar”/”old” and the “familiar”/”old” in the seemingly “new” or “strange”?
There’s a parallel here to the need, and difficulty, of movement from a pluralization model to a model of what Lu calls “transcultural literacy,” which I think applies to questions of both translinguality and transmodality: a movement whose need is signaled by efforts to distinguish “multilingual” from “translingual” through the change in prefix. Historically it seems clear that 1) a pluralization model is a significant advance over mono-cultural/-lingual/modal models, and that 2) such a model has preceded the introduction of any “trans-cultural/lingual/modal” framework. However, there is the difficulty of moving beyond that pluralization model (which retains key features of what it is intended to displace), and at least the theoretical possibility of adopting a “trans-” framework without first adopting a pluralization model. So do we pursue this theoretical possibility (or is it really only hypothetical at this point)? Or if not, how do we move past/beyond/or avoid the limitations of the pluralization model? Another possible way of putting this is to consider how we keep the focus on work across boundaries of language and modality rather than seeing our task as one of selecting from a menu of languages and modalities?
Bingo! And not only recognize these unfamiliar forms, but try them out/experiment with them to see what they offer, tell us, show us.
The “multi-” prefix works against this in seeming to require an additive model of change: counting the number of varieties, whether of languages or modalities, and identifying how they are configured (e.g., meshed or switched between) hence the introduction of the “trans-” prefix as an alternative meant to focus on cross-language and mode work and the need for negotiation (and the difficulty people have of understanding this as anything other than a peculiar way of invoking the enumerative framework for grasping difference).
I have no problem with “transmodal” as long as we include a discussion about how it is connected with multimodal both in terms of awareness and production practices, and the discussion is situated historically, and we specify what particular kinds of work we are hoping to suggest with “trans.”5
I think your point about needing both awareness and production practices corresponds to my comments [. . .] about needing both a change in dispositions and practices. Which makes me wonder if we need to separate these out for analytic or pedagogical purposes: e.g., multimodality as the means toward transmodality as the goal, albeit with the usual cautions about means becoming ends? Another possible way of putting this is to consider how we keep the focus on work across boundaries of language and modality rather than seeing our task as one of selecting from a menu of languages and modalities?6
This is a great question. I'd rather tackle the problem head on (getting beyond the “piling up” suggested by the plurality model—linked, I suppose to what Brandt talks about with her “accumulating” model). But how, then to avoid the idea of “selecting from a menu of languages and modalities?” is harder!
Bruce, reviewing the literature (!) on translingualism, brings out a somewhat different notion of translinguality and transmodality as in fact “dispositions”:
[C]hallenges to monolingualist ideology recognize the degree to which we are all always multilingual: that, in Pennycook’s phrase, for example, English is a language “always in translation.” These challenges would seem to call for a shift in dispositions rather than engagement in specific practices the dominant has trained us to recognize as multilingual/translations. But instead, the still dominant definition of multilingual resurfaces, leading to the celebration of what we’ve learned to recognize as multilingual and dismissing of what we’ve been taught to think of as monolingual.
The parallel in discussions of multimodality seems to be a tendency to adopt a celebratory stance toward practices that dominant ideology has trained us to recognize as multimodal and to push to the background or dismiss as unduly restricted those practices that this same ideology has trained us to recognize as, well, monomodal. As in questions of language, specific practices are removed from history and treated, instead, as in themselves having specific significance and effects across contexts.7
Here Bruce insists on a distinction between specific material practices, on the one hand, and, on the other, beliefs about/dispositions towards those practices, suggesting that the very notions of monomodality and monolinguality are misleading, manifestations of SL/MN ideology rather than (actual) practice, hence Bruce insists that:
what’s needed [. . .] is a way to grasp how specific practices are multimodal despite the blindness to that multimodal character that dominant culture’s training has led us to—and I don’t think we can say that the medium in itself controls this (e.g., the alphabet) but, rather, the ways we’ve been trained to grasp things like the alphabet.8
(music parallel: Western music notational practice can and has seemed to limit both what is recognized as music and the components comprising music [. . .], most obviously in restricting the pitch relations recognized to those of the 12-tone system; but this limitation is not so much the effect of the notational system itself as it is an effect of trained dispositions toward that system, leading to restricted ways of putting it to use and modifying it as needed.)9
But as Cindy observes in her response, these distinctions aren’t so easy to maintain:
[P]art of what is happening with multimedia / multimodality / transmedia / transmodality is tied to/situated within digital composing environments where people have access to composing tools that allow for different forms of hybrid mediation. As the engineers says, “When the only tool in your tool belt is a hammer, every problem looks like a nail.” So in composing environments, if your only tool is a pen or a piece of paper, or a word processor—the common sense approach (given dominant ideology in departments of English) often includes “writing” that happens primarily with words (although it includes, as always, other modalities).10
Here Cindy foregrounds the effect of material social environments on dispositions, rather than treating these as discrete from them. This same exchange and dynamic between disposition and material social environment surfaces more forcefully in the following excerpt from our exchange:
Collaborations always compose their own unique insights. During the time that Bruce and I and Tim exchanged emails on this project, I had to confront some of the challenges associated with fetishizing either print texts or digital texts, with thinking of any given text as monomodal or multimodal. As we talked about how and why SL/MN habits held such sway, I had to keep reminding myself that all texts are—already and always—multimodal (c.f. Kress, 2000; Prior et al, 2007; Jewitt, 2005). So, if we are interested in the semiotic channels that convey meaning in a text, It is our cultural dispositions that deserve our close attention: how (and why) a particular culture (or group of people) reads a particular kind (or genre) of text in a certain way at any given point in history, how (and why) certain modes seem to be foregrounded or seem to disappear in specific texts.
With this realization, I keep coming back to the example of the typical first-year English paper, which we have referenced at several points in this project as part of our argument. We know that such papers, like all texts, are multimodal, that they convey meaning through a number of semiotic channels. They are composed of words, certainly, but also of lots of visual information (among other data) carried in the arrangement and presentation of those words. So, the most interesting part about first-year compositions (what we often call “student papers” is not their multimodality, which has always and already been present, but rather our cultural dispositions (Bourdieu, 1977) toward such texts and how these dispositions, these habits, focalize many teachers’ attention and understanding almost solely on one semiotic mode, in particular, the printed word.
In part, this habit of seeing student compositions in terms of words alone (Yancey, 2004) has been historically sedimented. For much of the 20th century in most colleges and universities in the U.S., in most English Departments, our own professional activities and the historical values placed on the printed word in our culture have conditioned our dispositions toward student compositions, which reflect these values. Certainly in departments of English, printed books have been venerated as the sine qua non of intellectual achievement (MLA Task Force, 2006) and as the key focus of many faculty members” studies (Berlin, 1996; Williams, 1977). These factors—which in turn shape the assignments and instruction we offer as a profession—have also helped habituate us to see and understand student compositions in certain naturalized ways. In many ways, student papers are partial reflections of the printed works that we ourselves value and study in literary and non-fiction texts, in books and scholarly papers, in tenure and promotion decisions, in libraries and book stores.
What we commonly refer to as student “papers,” then, are a genre in a visible key (as well as in other ways) (Yancey, 2004): for instance, they are printed in a manner that mimics the text in books: they are printed on paper, generally, with titles, and blocked paragraphs, and subheadings. Sometimes, they have tables of contexts and footnotes and lists of works cited (which themselves often reference other printed materials). Often these texts come to us in cardboard folders (that resemble book covers), and they bear an author’s name. Through these textual features, among others, students’ academic essays acquire a culturally, ideologically, economically, historically rendered provenance lent to them by book culture. They resemble in these features the printed texts we, ourselves, value: the works of great authors, religious texts, books by philosophers, journals and monographs; the intellectual works, in sum, associated with book culture as it has been historically constituted. This is a culture to which we, as teachers of English studies, are devoted, in which we ourselves are invested.
So it’s not the student essays—the papers themselves—that are monomodal. In fact, the visual information these texts convey is front and center, on display right along with the alphabetic information. What is so very remarkable, in fact, is the effectiveness and efficiency with which the visual information has been rendered invisible and un-remarkable, ignored as common-sense—precisely because the ideological roots of such texts are naturalized.
It is our understanding of these texts and our habituated approaches to them that are so often monomodal, not the texts themselves. The visual meaning and provenance of these texts have become so naturalized that such information goes unremarked as a topic or feature of study; it is disappeared while we focus our courses primarily, often solely, on the alphabetic. And while there are now exceptions in all areas of English studies (e.g., courses on graphic novels and comic books, courses on the images of Blake, courses on 16th and 17th century emblematic texts, courses on film adaptions of novels), these courses are remarkable because of the attention they pay to modalities other than the alphabetic. In composition studies, we let it go unacknowledged when we choose primarily, or only, printed books to talk about in a class and ignore the visual information these texts contain; when we assign primarily, or only, written papers that look a lot like book texts and talk primarily, or only, about the ideas they express through words; or when we approve another dissertation prospectus without ever asking the author to talk about the visual information the text will convey.
I suspect that one reason digital media texts are so fascinating to English studies teachers is that the unfamiliar, unexpected forms and features and approaches of such texts are—at this point in history—still surprising, sometimes genre-busting in ways that calls into awareness the usual cultural dispositions with which we approach related texts. In the moment of encountering such unfamiliar texts, we are surprised, and we glimpse multiple modalities at work in making meaning—even though our attention to these modalities, this work, is usually suppressed by our ideologically shaped understandings of what we consider “normal” texts of this kind. Thus, I remember a time I was surprised in reading student essays for a writing class to discover an essay that contained an embedded audio file. The unexpected element in this essay (especially, but not only, because it succeeded so well) made me re-consider the text itself: among other things, I wondered about the different ways in which spoken words and printed words conveyed meaning, the conventional genre of student essays and why the text could never be reproduced on paper like other essays, why I hadn’t previously invited students to use audio as a resource, what visual information both the printed words and the representation of the audio file conveyed and why.
Of course a related set of challenges, too, adhere to the monolingual way in which we are habituated to understand texts in English, even when they clearly contain Spanish and French or German words that go unremarked. Or the way we understand a Spanish text to be “foreign” even when so many of its words are linguistically familiar to people who think they read or speak only English.
In the opening, Bruce, in cautioning against fetishizing specific language practices, insists on distinguishing between a translingual disposition and a specific language practice, positing that:
One can acknowledge the legitimacy of the “translingual” position while engaged in practices that appear monolingual (and vice versa), and one can acknowledge the legitimacy of the transmodal position while likewise being engaged in practices that appear from dominant perspectives to be monomodal (and vice versa).11
Which prompts Cindy’s important demurral and qualification:
Well, yes! At the same time, I want to work within the profession to encourage more teachers not only to recognize or “acknowledge the legitimacy of the transmodal position,” but also to encourage/experiment with/try more transmodal production, to experiment with different semiotic ways of composing meaning—and to help students do so as well.
We see a similar dynamic at work in the following exchange, which is initiated with Bruce expressing concern about the power of analytic categories to “overwhelm and limit our understanding of the phenomena being studied/taught.” Here, however, the issue is how the effort to break past limits of analytic categories—language and modality—can lead to a flattening of important distinctions: to allow a focus on continuity to obscure important differences. Bruce begins by pointing to problematic distinctions produced through categories:12
The most obvious example in language study is the categorization of languages and language varieties. On the one hand, it seems useful, for analytical and political purposes, to identify boundaries distinguishing one language/language variety from another. On the other hand, for other analytical and political purposes, those boundaries seem highly problematic (see Gal and Irvine; Parakrama). The equivalent is true of the category “language” itself as a demarcation of a far more complex ecology of practices. Recall here David Olson’s (1995) observations not only that there are “aspects of speech [that] are not represented in a writing system” but also that “writing systems create the categories in terms of which we become conscious of speech,” leading us to “introspect our language along lines laid down by our scripts” (p. 122, paraphrasing Whorf).13
Following Olson’s warning (cited above), it seems ultimately problematic to distinguish between language and modality. Dominant conceptions of language offer a highly attenuated, restricted sense of all that goes on in the activity of “language acts” (a.k.a. communicative acts). Kress (2000) acknowledges this in calling language multimodal (p. 186), vs. thinking of language as itself a discrete mode. Conversely, it seems appropriate to recognize modalities as a feature of language. From this, it no longer makes sense to treat language, whether as writing or speech or both, as apart from the “multimodal” (see Calvet 21-22).14
Cindy responds with another demurral and qualification:
Well, yes and no. I think it is quite true that all language use is multimodal. I’m not sure that all environments for linguistic exchange are created equal in regards to the modal mixing they accommodate. For instance, while print texts have always mixed some modalities of expression (words and visual information, for instance), digital environments allow for different kinds/varieties of mixing. Here, I'm thinking of the ways in which print text and video/audio texts can be juxtaposed/combined in a single composing environment. So, while multimodal/transmodal texts have always been present in our lives, I think it might be justified to say that new production tools and environments and social relations offer very different ways of accomplishing multimodality than printed works on paper-based pages.15
To which Bruce adds:
So while all language practice is multimodal (using the terms language, practice, and multimodal as “mass” nouns), language practices are not multimodal in the same ways, and the differences among/between them are significant. A radio play is not the same as a live theater performance or a television broadcast, even though they’re all (in quite different ways) multimodal, and the differences are quite significant from the production, distribution, and reception ends.
How might we make productive sense of these exchanges in forwarding specific definitions? Tentatively, we conclude the following: First, we see the need to remind ourselves to distinguish between analytic categories and practices to which they are applied, the latter of which, as fluid phenomena, can never be fully represented by the categories invoked. Instead, categories serve as lenses that inevitably distort as they clarify. This appears to be the thrust behind Bruce’s caution against consigning specific practices to the monolingual/monomodal dustbin: their seeming monolingual/monomodal character may be more the effect of our mode of analysis than an accurate representation of their actual status as practices.16
And behind the thrust of that caution as well is Bruce’s sense that conventional ways of describing, admiring, and also dismissing Western so-called “classical” music, especially J. S. Bach’s, as purely “mathematical,” fail to account for the full range of what that music is, does, and can be and do, but also that such accounts have the power to limit listeners’ experience of that music as one of only “numbers.”
Here the emphasis on dispositions toward modality and linguality has force: we need to be wary of the power of monolingual-ist, monomodal-ist dispositions to distort our sense of the practices under consideration. This danger manifests in two ways: the tendency to view practices not marked as either multimodal or multilingual as SL/MN; conversely, the tendency to conflate practices marked as either multimodal or translingual with multimodal/translingual dispositions, when their non-SL/MN character may be more apparent than real.17
Second, and paradoxically, we also need to recognize the effect of specific material social environments on dispositions toward language(s) and modalities. As we’ve already suggested, the emergence of changes to communicative practices—most obviously, the development of digital communication technologies and global communicative networks; less obviously, the increasing traffic of (exchanges and changes to) peoples and language practices, reinforced and changed as well by global communication technologies—has contributed to the increasing visibility of and questions about language and modality.
The “new” communicative practices, as they are often described—those that dominant dispositions lead us to recognize as different—also force a re-evaluation of and change to the communicative practices those dominant dispositions have led us to see and experience as simply natural, the norm.18
We see this articulated in the following exchange:
[We need to think] of our work less as discovery of the new and more as the recovery and recuperation of alternative dispositions toward meaning making practices, including both those our dominant training has led us to recognize as monolingual or monomodal and those that training leads us to think of as multi- or trans-lingual/modal.
[B]ut at the same time, we can't dehistoricize/remove such discussions completely from the context of massively extended computer networks/the increase of digital tools for composing/the practices of multimedia composition online that have, in part, given rise to the contemporary interest in multimedia composing.
So we need both to recover/recuperate and to consider significant changes/gaps between old and new. Hard to do without either fetishizing new or overlooking those gaps (yielding to the temptation to see only continuities and overlook differences).
In light of all this, rather than understanding modality and linguality in terms of fixed (“defined”) categories and practices, we pose the following questions of definition as more productive in bringing out the dialectical relations between dispositions and practices with language and modality:
What are the material social conditions of composing possibility for the deployment of language and modality (including available and competing dispositions toward and training with these)?
How are modality and language deployed (or might they be deployed) in this composition? To what end? Demanding, or expecting, what kinds of work? How does such deployment work on and with the conditions of its composition, distribution, and reception?
In what ways do our current analytical categories of modality and language need to be revised to accommodate differences in the ways this composition engages these?
Here, I’d love to show a timeline of when these terms emerged on the web or a heat map that would illustrate when and where these terms emerged, geographically and historically.
For instance a search of CCC titles and abstracts identifies 50 instances of the term “multimodal”, dating from 1991 to the present, and 34 instances of “multilingual” dating from 1990 forward.
Tim says: Our multimodal/multilingual timeline was built using the TimelineJS framework, an open source timeline generator developed by the Knight Lab. Our CCC dataset is available in JSON format, and we hope that others might extend the project. You can view the source files at Github.
Of course, SL/MN practices—and representations of these practices—have never been limited to one modality, one medium. Consider, for example, “The Flemish School”, created by Richard Brookshaw in the 18th century, reproduced by Egbert van Heemskerck, and now displayed online by the British Museum (Copyright British Museum).
A more contemporary example of the inadequacy of SL/MN can be found here Xuan Wang's “I am not a qualified dialect rapper”.
Yes, see John Trimbur and Karen Press's observation that "multimodality itself is not new, nor is it a break from the past. Multimodality is new as a term, a conceptual terrain that surfaced at a particular historical conjuncture, goaded by the need to understand dramatic changes in the means of communication."
Tim says: This hypertext piece is a response to Cindy’s call, an experiment in forms and modalities and intersections.
Cindy says: Maybe a link to the term “ensembles” here as a way of opening up the word “trans” and showing how it might intersect, in terms of modality with “ensemble”??
Bruce replies: My own sense is that there is never not an ensemble—it’s just that we’re trained not to recognize this. Christopher Small’s (1998) concept of “musicking” might be pertinent here:
“To music is to take part, in any capacity, in a musical performance, whether by performing, by listening, by rehearsing or practicing, by providing material for performance (what is called composing), or by dancing. We might at times even extend its meaning to what the person is doing who takes the tickets at the door or the hefty men who shift the piano and the drums or the roadies who set up the instruments and carry out the sound checks or the cleaners who clean up after everyone else has gone. They, too, are all contributing to the nature of the event that is a musical performance.” (p. 9)
Small, Christopher. (1998). Musicking: The meanings of performing and listening. Middletown, CT: Wesleyan University Press.
We take the term disposition from the work of Pierre Bourdieu (1977) in Outline of a Theory of Practice.
Disposition, according to Bourdieu, is “the result of an organizing action, with a meaning close to that of words such as structure; it also designates a way of being, a habitual state (especially of the body) and, in particular, a predisposition, tendency, propensity, or inclination.” (p. 214, emphasis ours)
For Bourdieu, disposition is closely linked to habitus. He notes,
“[Habitus is] transposable dispositions, structured structures predisposed to function as structuring structures, that is, as principles of the generation and structuring of practices and representations which can be objectively ‘regulated’ and ‘regular’ without in any way being the product of obedience to rules, objectively adapted to their goals without presupposing a conscious aiming at ends or an express mastery of the operations necessary to attain them, and, being all this, collectively orchestrated without being the product of the orchestrating action of a conductor.” (p. 78)
A good example of a caution here is Steve Bernhardt’s (1986) “Seeing the Text,” an early piece which suggested the importance of paying attention to the many visual elements (and the rhetorical information) that are present in texts that many people considered alphabetic.
This sounds like a footnote to add for sure, and thanks for the reference.
Cindy says: Here, I'd like to show examples of print texts across history that have always been multimodal: illuminated manuscripts, illustrated letters, etc. Like these:
It is important to note that even those texts that appear not to be multimodal are in fact multimodal: dense clusters of black serif-font printed letters on white background (like this page). Bringing attention to the modalities these texts deploy is crucial to debunking the notion of multimodality as something other or different from the norm, rather than being the unrecognized norm. This, I think, is Bernhardt's point. Black and white are colors passing as non-colors, and Times font is an image with its own provenance masquerading, instead, as a transparent, unmediated feature of print.
For instance, consider the art of “bending” notes as demonstrated in the Berklee School of Music’s “Master The Art of Bending Notes”.
Note that the conventional scoring of notes or tablature for chords cannot completely/adequately/wholly represent a bent note, which is individually composed by a specific musician in the activity of playing a particular instrument.
On the effects of Western musical notation on conceptions of music, see John Shepherd’s (1993) “Difference and Power in Music” and(1987) “Music and Male Hegemony,” and Horner’s (2011) discussion of “Idealizing the Medium” in “On the Study of Music as Material Social Practice.”
Shepherd, John. (1993). Difference and power in music. In Ruth A. Solie (Ed.), Musicology and difference: Gender and sexuality in music scholarship (pp. 46-65). Berkeley: University of California Press.
Shepherd, John. (1987). Music and male hegemony. In Richard Leppert and Susan McClary (Eds.), Music in society: The politics of composition, performance and reception (pp. 151-72). New York, NY: Cambridge University Press.
Horner, Bruce. (1998). On the study of music as material practice. Journal of Musicology 16, 182-85.
Are there copyright issues here? Another example might be oddly enough John Lennon’s “writings” which are a conglomeration of drawings and verbal writings—though again, copyright issues might interfere.
I wonder if something simpler might work: the felt need, ultimately, especially in scientific “writing,” to include graphics (charts, tables, images) to characterize the phenomena or relations described, vs. verbal characterizations of these? I've read enough of the latter to scream out (in comments) “PUT THIS IN A CHART PLEASE!”
Here, I'd like to show examples of what we frequently call technical writing, although such texts almost always include visual import like this one or this.
YES. I'd have to dig out my old poetry examples to bring some of this out, but attempts at linguistic transcription are telling in this regard. For poetry, see esp. Marlene Nourbese Phillip's She Tries Her Tongue.
Good references here are Wysocki and Johnson-Eilola’s (1999) “Blinded by the Letter” and Steve Bernhardt's (1986) “Seeing the Text”. Both pieces talk about the ways in which modalities of expression interact with cultural contexts and expectations in ways that naturalize the effects of our communicative practices. Print, for instance, and the analytic approaches and expectations that came to be associated with print during the seventeenth, eighteenth, and nineteenth centuries helped to create a conservative disposition/adherence to print as a means of academic/scholarly expression, but also to blind us to many of the multimodal tactics we always/already employed in printed texts: line breaks, font, placement, white space, font size, bolding, etc.
There is an analogy here from the study of music: whereas traditionally the Western music score and system of music notation was understood to represent nothing other than the aural, musicologists have come to recognize the ways in which the score works also as a visual entity (exploited in “augenmusik”) directed at performers enjoying the view of the score, and, likewise, the performance of music—including the most traditional performance traditions of Western classical music—cannot be categorized as purely aural, or visual, or tactile (recall Barthes here), or purely anything. Hence musicologists have had to:
come up with the neologism “musicking” (see Small) to name the conglomeration of practices that operate in any “musical event” (analogous to the concept of the “literacy event”),
learn to pay attention to “listening” practices to grasp differences in the experiences of different listeners/viewers/performers with (ostensibly) the “same” piece of music or performance of it (reference), and
learn to attend to features even of “aurality” of significance that traditional Western systems of musical notation have difficulty representing, style of “attack” (e.g., staccato vs. legato), and timbre, not to mention the full spectrum of pitch relations.
Likewise, distinctions between types of music, and the legitimacy of the category “music” itself (especially to name a distinct category of cultural activity), are vulnerable to radical challenge, as studies in ethnomusicology and “popular” music have demonstrated.
...concern about conflating analytic categories and actual practices... and our cultural and historical context...
The distinction between traditional notions and practices of multilinguality is a case in point: use, or mixing, of different languages does not in itself signal a break with monolingualist dispositions. Rather, interjecting the occasional French or Spanish locution into a predominantly English text may in fact reinforce such dispositions by highlighting (and capitalizing on) a monolingualist notion of languages as discrete.
Likewise, predominantly alphabetic print verbal compositions that deploy the occasional image or attached audio clip may simply reinforce an “additive” or ornamental disposition toward modality. Given our own early training as written-language specialists, we have risked such a situation in this very piece although we have tried hard to avoid it by calling on our experience with other kinds of non-alphabetic texts. For discussion of a richly ambiguous example of a composition that deploys both multiple languages and images, see the discussion of student work in Canagarajah's (2009) “Multilingual Strategies.”
For a composition that, to our minds, helps us read with an awareness of transmodal contributions (while addressing transmodality, albeit not in such terms), see McCloud (1994). On the complex strategies by which writers have resisted monolingualism—including the strategy of writing the “national” language by writers not “authorized” (because of social positioning) to write that language, see Yildiz (2012).
Yildiz, Yasemin. (2012). Beyond the mother tongue: The postmonolingual condition. New York, NY: Fordham University Press.
McCloud, Scott. (1994). Understanding Comics. New York, NY: William Morrow.
A good reference here is Diana George’s (2002) “From Analysis to Design.” As George writes, “For many years, in fact, the research paper section was literally the only place in composition textbooks where we might encounter any reference to page design, layout, or font choices...” (p. 25).
George, Diana. (2002). From analysis to design: Visual communication in the teaching of writing. College Composition and Communication 54(1), 11–39.
Here’s how I had framed this point in an earlier version of our dialogue. I cut this later not just for reasons of length but to avoid making a fetish of my use of the untranslated Molière:
Collaborations always compose their own unique insights. During the time that Bruce and I and Tim exchanged emails on this project, I had to confront some of the challenges associated with fetishizing either print texts or digital texts, with thinking of any given text as monomodal or multimodal. As we talked about how and why SL/MN habits held such sway, I had to keep reminding myself that all texts are—already and always—multimodal (c.f. Kress, 2000; Prior et al, 2007; Jewitt, 2005). So, if we are interested in the semiotic channels that convey meaning in a text, It is our cultural dispositions that deserve our close attention: how (and why) a particular culture (or group of people) reads a particular kind (or genre) of text in a certain way at any given point in history, how (and why) certain modes seem to be foregrounded or seem to disappear in specific texts.
Tim says: Our multimodal/multilingual timeline was built using the .gif){kind=link}
