Journal of Linguistics and Language Teaching
Volume 14 (2023) Issue 1
pp. 11-33
Words, Pictures, and Arguments:
A Relevance-Theoretic Synthesis1
Gerald Delahunty (Fort Collins (CO), USA)
“[H]e had merely shuddered when the politician’s name was mentioned, an eloquent enough comment–more expressive, indeed, than mere words” (McCall Smith 2004: 197).
Abstract
Whether visual representation can function in arguments is a controversial issue. Those who claim they cannot, claim that only propositions may function thus and that as visuals cannot represent propositions, they cannot function in arguments. The current paper, invoking recent developments in Relevance Theory, demonstrates that visuals, specifically photographs, can represent propositions and can therefore function as and in arguments. The paper demonstrates that visuals also communicate more than propositions in that they provide evidence for a range of ‘impressions’ that support a ‘credal attitude’ toward the document in which they occur.
Keywords: Multimodal discourse, argument, Relevance Theory, Amnesty International
1 Introduction
1.1 Pictures, Propositions, and Arguments
"A picture is worth a thousand words." (English proverb)
Davidson (2013: 465) rightly asks:
How many facts or propositions are conveyed by a photograph? None, an infinity, or one great un-statable fact? Bad question. A picture is not worth a thousand words, or any other number. Words are the wrong currency to exchange for a picture.
It is often claimed that only propositions may function in arguments, for example, by Dutilh Novaes (2022), who claims that
[a] n argument can be defined as a complex symbolic structure where some parts, known as the premises, offer support to another part, the conclusion (§1).
So if premises and conclusions must be propositions, and if Davidson is correct, pictures should not be able to function in arguments. However, there are voices claiming that modalities other than language can indeed contribute to arguments. Bateman (2018: 302) proposes an abductive 2 framework for theorizing how non-linguistic modes may be "an ingredient in an argument" (ibid.). Other voices 3, especially from the Relevance Theory (RT) wing of pragmatics, claim that visuals may express propositions 4. Sperber & Wilson (2015: 141) characterize the information made manifest to an addressee as "an array of propositions which may vary in size” from “a single proposition” to a “vast array of propositions” (p. 34) regardless of whether the ostensive stimulus is verbal or not5. If this is so, then pictures may function as (elements of) arguments.
This paper proposes an analysis of an Amnesty International webpage6 that provides information about a violation of human rights and urges viewers to take action by sending an appeal to government officials calling on them to redress the violation. Amnesty produces reports, webpages, and emails regarding such violative situations which synthesize text, i.e., strictly linguistic expression, with photographs and other visuals. I refer to such composites as documents. In this paper, I show how such multimodal documents are to be interpreted as persuasive arguments designed to “give reasons in support of a standpoint” the acceptance of which “engender[s] a disposition to act” (Nettel & Roque 2012b: 59). I demonstrate that the photograph in the document is integral to the argument made by the webpage, not just an embellishment. As visuals of various sorts are increasingly used in social media arguments, it is crucial that we understand and can explain how they contribute, and contribute effectively and importantly, in multimodal argument. The rich theoretical apparatus of Relevance Theory permits a fine-grained analysis of the roles of text and photograph in the webpage and provides an explanation of how each mode singly and in interaction with the other enacts Amnesty’s argument. What I say about the webpage can be extended to an analysis of other Amnesty multimodal documents, and to similar documents created by other individuals and institutions.
1.2 The Amnesty Webpage
Figure 1: Screenshots of Amnesty’s respective web page below (https://act.amnestyusa.org/page/21064/action/1?locale=en-US; 29-06-2023)
1.3 The Amnesty Webpage Genre
The webpage above consists of a photograph preceded and followed by written text in various fonts and highlighting. Embedded within it is the text of an appeal letter addressed to high-ranking officials with the power to redress the exigency represented elsewhere in the document and re-presented in the appeal. Not captured in the screenshot is a 'SEND NOW' button against a yellow background, which appears on the same webpage but is not positioned directly beneath the displayed text.
The webpage genre typically consists of the following “stages” (Bateman 2006: 178):
The Amnesty logo on yellow background
A general directive to support human rights
A directive specific to the current rights violation
A photograph
A text describing the current concern
An appeal letter
A request for webpage viewer’s personal information
A directive in yellow background to SEND NOW
The text of the webpage typically consists of:
A general characterization of a problematic situation and its cause.
More specific details of the problem.
The need to solve the problem and a general characterization of the solution.
Bald-on-record directive to current webpage readers to tell those powerful enough to solve the problem for which they are responsible by sending YOUR MESSAGE via the appeal letter.
And the typical appeal letter consists of:
The headline in bold font directing appeal addressees to solve the problem.
A standard letter greeting to addressees.
A general characterization of a problematic situation and its cause.
More specific details of the problem.
An appeal to addressees to redress the situation as the people with the power to do so.
The webpage is an instance of a stable (with minor variations), multi-modal genre that has evolved over the life of Amnesty International. It is an excellent target for the current analysis because the photograph has no writing within it so that, while it might communicate several partial propositions on its own, it depends upon its generic and linguistic context for its intended interpretation as an element in the webpage argument. This is in contrast with other RT analyses of multi-modal documents whose pictorials, e.g. comic book panels and political cartoons, communicate as intended without benefit of surrounding text (Forceville 2014, Forceville & Clark 2014). In what follows, I provide an RT analysis of the Amnesty webpage that accounts for how visuals may function as propositions in arguments, beginning with a brief review of the pertinent RT concepts.
2 Relevance Theory (RT)
Relevance Theory (Sperber & Wilson 1995 [1986], Wilson & Sperber 2004, Wilson 2016, and many more7) is arguably the most influential current pragmatic approach to discourse interpretation. It is a cognitively based, post-Gricean theory of communication. Its fundamental concept is that communication is ostensive-inferential, a characterization that entails specific principles and supporting concepts. I briefly describe those pertaining to the current analysis:
Manifestness
Fundamental to RT’s conception of cognitive processing is manifestness:
A fact is manifest to an individual at a given time if and only if he is capable at that time of representing it mentally and accepting its representation as true or probably true. (Sperber & Wilson 1995 [1986]: 39)A fact is manifest to an individual at a given time if and only if he is capable at that time of representing it mentally and accepting its representation as true or probably true. (Sperber & Wilson 1995 [1986]: 39)
Cognitive Environment
The cognitive environment of an individual is the set of facts that are manifest to them at a particular time.
Positive Cognitive Effects
Positive cognitive effects include contextual implications (inferences derivable from input and context together but from neither alone), the addition of new information to the interpreter’s store of information, and the strengthening or weakening of their commitment to assumptions in that store.
Relevance of an Input to an Individual
Because RT is a cognitive theory of communication, it models the inferential process of individual interpreters. Relevance is relevance to an individual at a particular time, regardless of whether the stimulus is designed for a single specific interpreter or for a mass audience.
Other things being equal, the greater the positive cognitive effects achieved by processing an input, the greater the relevance of the input to the individual at that time.
Other things being equal, the greater the processing effort expended, the lower the relevance of the input to the individual at that time. (Wilson & Sperber 2004: 609)
Ostension
An action is ostensive if it “makes manifest an intention to make something manifest … Showing someone something is a case of ostension. So too … is human intentional communication” (Sperber & Wilson 1995 [1986]: 49). Ostensive communication is defined by two intentions:
The informative intention: To make manifest or more manifest to the audience a set of assumptions (Sperber & Wilson 1995 [1986]: 58).
The communicative intention: To make it mutually manifest to audience and communicator that the communicator has this informative intention (Sperber & Wilson 1995 [1986]: 61).
The Communicative Principle of Relevance
Every ostensive stimulus conveys a presumption of its own optimal relevance:
a. The ostensive stimulus is relevant enough to be worth the audience's processing effort.
b. It is the most relevant one compatible with the communicator's abilities and preferences.
Explicatures
RT distinguishes information (partially) explicitly communicated from information implicitly communicated, i.e. explicatures from implicatures. Verbal communication requires the deployment of linguistic information, thereby encoding a logical form. This form represents an incomplete proposition, which, along with contextual information is inferentially elaborated into an explicature, a fully propositional logical form. The logical form representing the proposition explicitly communicated is the basic explicature. Because linguistic forms may include more or less coded information, basic explicatures can be more or less explicit. The basic explicature may be embedded under various predicates such as evidentials, speaker's stance, and those derived from Grice’s Maxims (Grice 1975, 1989) and Speech Act felicity conditions. These are higher order explicatures.
Implicatures
Implicatures are propositions inferred from explicatures and contextual information; they are pragmatically inferred, not coded.
Interpretive Processes
Explicatures are arrived at by inferential processes: disambiguation, reference assignment, saturation, free enrichment, and ad hoc concept construction.
Implicatures are arrived at by pragmatic inferencing based on the explicatures and contextual information. They may function as implicated premises or implicated conclusions derived from those premises. Implicatures may be more or less strongly or weakly communicated. Strongly communicated implicatures are those that the interpreter must construct for the utterance producer's communicative intention to be fulfilled. Weakly communicated implicatures include a range of inferences suggested by the utterance and consistent with its explicatures and other implicatures, which the interpreter might choose amongst but which are not essential for the recovery of the utterer's ostensive intention:
A proposition may be more or less strongly implicated. It is STRONGLY IMPLICATED (or is a STRONG IMPLICATURE) if its recovery is essential in order to arrive at an interpretation that satisfies the addressee's expectations of relevance. It is WEAKLY IMPLICATED if its recovery helps with the construction of such an interpretation, but is not itself essential because the utterance suggests a range of similar possible implicatures, any one of which would do (Wilson & Sperber (2004: 620); highlighting in original).
Relevance-guided Comprehension Procedure
The communicative principle of relevance licenses an addressee to expect an utterance to be optimally relevant, and in interpreting that utterance, he or she will automatically:
a. Follow a path of least effort in computing cognitive effects / constructing an interpretation of an utterance (in particular, in resolving ambiguities and referential indeterminacies, adjusting lexical meaning, supplying contextual assumptions, deriving explicatures and implicatures, etc.).
b. Stop when [their] expectations of relevance are satisfied or abandoned (Wilson 2016: 86)
Subtasks in the Overall Comprehension Process
Instantiating the overall comprehension procedure entails:
a. Constructing an appropriate hypothesis about explicit content (EXPLICATURES) via decoding, disambiguation, reference resolution, and other pragmatic enrichment processes.
b. Constructing an appropriate hypothesis about intended contextual assumptions (IMPLICATED PREMISES).
Constructing an appropriate hypothesis about intended contextual implications (IMPLICATED CONCLUSIONS). (Wilson & Sperber 2004: 615).
These apply in parallel rather than sequentially and may involve mutual adjustments.
3 Argument
What argument is is a matter of longstanding debate, which has been made considerably more interesting by the relatively recent consideration of the roles, if any, played in it by visuals. One set of related issues has to do with whether argument may include elements of persuasion or is purely a matter of logic and epistemology, a relationship amongst truth-bearing entities such that the truth of some licenses the inference that others may or must also be true. Roque (2012), in a special issue of Argumentation, offers a variety of opinions on the natures of argument and persuasion and on whether the two should be kept separate or may be conjoined in theoretically and analytically insightful ways. Epistemologically committed theorists claim that arguments are to be evaluated by their reasonableness and tend to regard persuasion as emotional and manipulative, and so to be eschewed. Rhetoricians, however, generally assume that the goal of argument is persuasion and that persuasion need be neither emotional nor manipulative and that a case may be made for 'persuasive argumentation', a blending that allows arguers to influence both beliefs and behavior (Nettel & Roque 2012a,b, Rohan, Sasamoto & Jackson 2018, Roque 2017, Tseronis & Forceville 2017.)
Roque (2017) addresses the roles of visuals in argument. If we regard visuals as primarily emotional and therefore persuasive and manipulative, as many analysts do (Roque 2017 for references), then they can have no legitimate place in argumentation, narrowly construed. However, if we consider that argument may be either logical or persuasive or both, then we have opened a theoretical space in which to consider what roles visuals might play in multi-modal persuasive argumentation. The photograph in the Amnesty webpage (Section 1.1) contributes both reason and emotion to the webpage argument.
4 Multi-Modal Discourse
I begin by briefly discussing the three elements essential to the study of multi-modal discourse identified in Bateman (2018): a relevant community of practice, its media, and its genres. I argue for an RT analysis of the interaction of the webpage text and photograph and demonstrate that photographs, and by extension, other types of visual representations, can communicate propositions and thus enter into logical argument. I extend this analysis to demonstrate that RT can also account for attitudinal and affective aspects of argument.
4.1 Community of Practice
The concept of a community of practice derives from the work of Jean Lave & Etienne Wenger (1991) and Wenger (1998) on social learning. Such a community shares concerns, communicative media, ways of deploying elements of those media, and ways of interpreting those deployments. Communities of practice subsume speech communities, which Morgan (2014: 1) defines as "groups that share values and attitudes about language use, varieties and practices." Membership in such communities need not be fixed nor require that all members share the community discourses and other practices to the same degree or with the same authority.
Amnesty International can be viewed as a community of practice. Its members share a concern for human and humanitarian rights and for the redress of their violations. Amnesty staff and members enact a variety of roles: organization director, information gatherer, information curator, web designer, web-page reader, member, email recipient and appeal sender, amongst others. All engage with activist human rights discourses in various genres and modalities, though ordinarily, webpage, email, and report readers may not know (or care) which staffers are responsible for which discourse elements. From an RT perspective, knowledge of the AI community practices would be included in members’ encyclopedic information, which varies in depth and precision from person to person.
4.2 Semiotic Modes
A community of practice must choose its communicative media and decide how these media are to be deployed and interpreted. The Amnesty documents integrate logo, photos, and other visuals, e.g. maps, with written English text presented in a variety of colors, font types and sizes, all of which are elements of Amnesty genres.
4.3 Multimodality
Multimodality would appear to imply the existence and combinatory potential of discrete modes, and textbook and online definitions provide definitive lists of the individual constituent modalities, though these lists are not always in agreement.8 Researchers are not so sanguine. Forceville (2014: 51), for example, doubts whether a definitive list is possible and whether any of the proposed modes is discrete. Nonetheless, he provides a provisional, practical list: written language, spoken language, visuals, music, sound, gestures, olfaction, touch.
4.4 Genre
Genres were traditionally regarded as categories of texts, especially of literary texts. More broadly, Bateman (2006: 178) regards genres as “recurrent staged social activities”, a characterization about whose implication of sequential ordering and interpretation I will have more to say below. More recently, Bateman describes genres as
socially established communicative activities which employ conventionalized strategies in order to achieve their effects. (2018: 300)
According to Gumperz, genres invoke
schemata or frames, embodying presuppositions associated with ideologies and principles of communicative conduct that in a way bracket the talk, and that thereby affect the way in which we assess or interpret what transpires in the course of an encounter. (2018: 313)
The strategies, frames, presuppositions, and ideologies represented in the Amnesty webpage are elements of human rights discourses; the communicative conduct includes multimedia creation, distribution, and interpretation of reports, webpages, and emails. These documents are typically, though not universally, assessed as reliable; and they are typically, though not universally, evaluated as sincere attempts to redress human rights and other violations.
Some genres are rigid and stable over long periods of time, e.g. Shakespearean sonnets. Others are considerably more synchronically variable and diachronically unstable, e.g. blogs. Amnesty’s documents are mass-communication genres developed over the course of Amnesty's history for their various purposes and in response to reactions to them and to the affordances of new communication technologies, as well as to the innovations created by other media users. These genres articulate Amnesty's arguments for persuading governments, diplomats, and Amnesty members and volunteers to act for the redress of human rights violations and to donate to the organization.
Genres are culture-specific devices which, from an RT perspective, are mentally represented as encyclopedic information. They contribute contextual assumptions to the inferential phase of utterance interpretation by influencing the audience's expectations about the format of the discourse, about the arrangements of the discourse elements, about the kinds of cognitive effects to be licensed, and about how explicitly and strongly these effects will be communicated (Unger 2006, chap. 10).9 These RT-internal concepts will be discussed below.
Forceville, ChF in the text quoted below, proposes that "the very first thing a would-be addressee of a mass-communicative message does is assess the genre to which the message belongs. He (sic) then activates", from his encyclopedic knowledge, “the genre conventions that allow him to derive pertinent” inferences (2014: 68). And more specifically,
[g]enre is an element of context whose importance cannot be overstated (italics in original). Genre-attribution moreover occurs mostly subconsciously and in milliseconds, and is in my view the single most important element in the addressee's “cognitive environment” steering his strategy of interpretation of any pictorial or multimodal message [reference omitted]. Whereas context is endless, and ever-changing, genre-attribution is quite stable and reliable. Indeed, I submit that "genre" more than any other contextual factor helps constrain what the relevance theorist Robyn Carston [2010: 265] calls "free pragmatic processes: [ ... ] pragmatic processes that contribute to what a speaker [or picture maker, ChF] is taken to have explicitly communicated but which are not triggered or required by any linguistic [visual, ChF] property or feature of the utterance [picture, ChF]". (Forceville 2014: 63-64)
The Amnesty documents are all instances of mass-communication genres that enact directive speech acts whose intent is to persuade their addressees to solve problems. Problems and their solutions entail evaluation. A situation deemed problematic must be shown to be so; that is, it must be evaluated as such by appeal to relevant criteria. Complementarily, the solution must be evaluated by additional relevant criteria. Evaluation requires a justification (Forst 2012), that is, an argument that the evaluated problem and its proposed solution meet appropriate criteria. Arguments are designed to persuade their audiences to believe in specific propositions and to act in specific ways and may be based on the traditional rhetorical concerns: truth and reasoning (logos), emotions (pathos), and the audience's assumption that the arguer is reliable (ethos) (Cooren 2015: 102, 119), subject to the interpreter’s epistemic (and other) vigilance and the producer’s anticipation of the interpreter’s vigilance (Sperber et al. 2010, Sperber 2010). 10 11
Tseronis & Forceville (2017: 3-4) describe argument as dialectical, governed by conventions, e.g. those governing debates at a university versus those in a parliament. For an argument to be effective, its rhetoric must be appropriate in its context for its audience. The Amnesty documents are designed to engage their audiences by ‘naming and shaming’ those deemed responsible for human rights violations and capable of their redress and persuadable by appeal to the logic of human rights and their violations, to human feelings, to the pressures imposed by large numbers of appeals, and to Amnesty's reputation for reliability, as discussed in the following sections.
5 The Amnesty Webpage Genre
5.1 The Webpage
In what follows, I analyze the argument made by the text and photo in the webpage regarding the siege of Eastern Ghouta by Syrian and Russian forces during the Syrian civil war. The structure of this webpage is characteristic of Amnesty webpages in general. At its top is the Amnesty logo against its signature yellow background; directly beneath this, in small capital letters, is the general imperative, TAKE ACTION FOR HUMAN RIGHTS; followed by the more specific imperative in large and bolded capitals to help the people under attack in Eastern Ghouta. English imperative constructions are typically analyzed as having an underlying second person unpronounced subject pronoun, whose reference is contextually determined. The subjects of these two imperatives are inferred to refer to the reader of the webpage at a specific time and place of access.
The photo beneath the imperatives is spread across the page. It is above further text in much smaller font in sentence case, provides information about the problematic situation in Eastern Ghouta, and exhorts readers to act. Beneath this is the text of an appeal addressed to Presidents Putin and Assad describing the situation in Eastern Ghouta and "calling on" them to end the siege and allow aid into the area.12 Beneath the appeal is an invitation to MAKE YOUR VOICE HEARD by entering your name and address on an electronic form and clicking the SEND NOW button. These imperatives, like those above, are addressed to the webpage reader.
Amnesty appeals are typically addressed to heads of state and other government functionaries, often identified as "Message Recipients." They briefly characterize the violative situation and urge the addressees to redress that violation, which they typically characterize as contravening human or humanitarian rights. They re-direct the message of the webpage by re-contextualizing, re-genrecizing, and re-entextualizing it for specific addressees rather than the more indefinite webpage reader. The webpage and appeal belong to the category of "hortatory discourse" as described by Longacre (1992) and related to philanthropic fund-raising appeals (Mann & Thompson 1992; Delahunty 2018).
5.2 Applying RT to the Amnesty webpage
As Forceville (2014) predicts, I, and anyone familiar with Amnesty documents, readily recognize the AI webpage genre. I have very specific knowledge of its structure and functions and my "expectations of [its] relevance" are "precise and predictable enough to guide [me] toward [Amnesty's intended] meaning" (Wilson & Sperber 2004: 607). I assume that it was produced and disseminated with the intention of informing its readers of the exigency of the situation in Eastern Ghouta and of its producers' desire that readers act for the redress of the human rights violations that occasioned the exigency. Simultaneously, the webpage readers assume that its producers intend to inform them of this intention. That is, the webpage fulfills the informative and communicative intentions articulated in Sperber & Wilson (1995 [1986]) and is therefore an instance of "ostensive communication."13
5.3 Just the Photo
As a preliminary to the analysis of the photo in its context, imagine that you are out for your daily walk and find a photo on your path. What information can and can't you glean from just the photo? If the photo were the one in the Eastern Ghouta webpage and you were asked to describe it to someone who cannot see it, you would probably say that:
(a) a man is holding a boy
(b) the man and boy are in front of a burning car
(c) the burning car is on what appears to be debris
(d) the debris may be from damaged buildings in the background.
There are a great many other things you might say, such as that while the boy is looking directly at the viewer of the picture, the man is facing toward the viewer’s left and seems to be moving in that direction, possibly to take the boy out of danger.14 While you might include many such observations in your description if you omitted items (a-d) then you would have seriously misrepresented the photo.
The American philosopher Charles Saunders Pierce identified a great many relations between signs and their denotations / referents (Atkin 2022). Of these distinctions, those amongst symbols, indexes, and icons are most relevant to our purposes here and are those most invoked in linguistics research. Symbols are signs whose relation to their referents is arbitrary and conventional, as words typically are to their meanings. Indexes are related to their referents by physical or temporal correlation or co-occurrence. The clearest cases are effects and their causes: an effect, e.g. smoke, may be an index of its cause, fire, by virtue of their causal physical relationship. A perceiver of smoke may infer the presence of fire on the abductive basis that fire may cause smoke. Icons represent their referents by virtue of their similarity, which may vary by degree. Photographs are icons par excellence, as they are (assumed to be) veridical representations of their referents.
Even though we take the photo to be an icon and thus to veridically represent its object, we cannot say much more about that object beyond those items discussed above. We cannot say where the photo was taken, who the man and boy were, what caused the car fire or the damage to the buildings, or why and whither the man was carrying the boy. You might surmise, for example, that the photo is of a scene in the aftermath of an earthquake. We know that photographs cannot capture the entirety of a scene and may be cropped for various reasons, so we do not know what the photo omits: what was to the left and right sides in the original scene? What was above and what was below?
Nor would you be able to say how the photo came to be on your path. Perhaps it was accidentally dropped? Perhaps it was deliberately placed there so that social scientists hidden in the bushes could observe your reactions to it? You cannot tell these things because the picture has no text in or around it and its location on the path provides no information that would allow you to contextualize it for a fuller interpretation. You would not be able to construct a discourse meaning for it (Bateman 2018: 303). One thing you could not assume is that the photo represents 'one great unstatable fact': you would have no way of knowing, for example, whether the photo represented a fact or a fiction. You could not regard the photograph in this context as ostensively displayed and therefore could not regard it as optimally relevant in RT terms.
5.4 The Photo in Context
Now consider the photo back in its webpage context. We recognize the combination of Amnesty’s logo, text, and photo as an instance of the AI webpage genre seeking to persuade us to do something. The AI logo allows us to more specifically identify the genre as asking us to act to redress a situation violating human or humanitarian rights. We assume that the text and photo were deliberately combined, disseminated, and displayed with the goal of influencing its viewers' minds and actions in the ways just described. We assume that the photo and text were ostensively communicated. We also assume that the text provides context relevant to the interpretation of the photo, rather than, say, that the photo is an advertisement interpolated into the webpage for a particularly shocking form of extreme tourism. The context provided by the text and other genre elements allow us to hypothesize a discourse meaning for the photo. We also assume that the photo provides context relevant to the interpretation of the text.
5.5 Photo and Text
We infer, when we contextualize the photo with information from the text, that the photo iconically (Atkin 2013) and faithfully resembles (Sperber & Wilson 1995) and 'shows' (Sperber & Wilson 2015, Forceville & Clark 2014, Wharton 2009) a scene in Eastern Ghouta, not, for example, a scene from a movie. We also infer that the fire in the car and the damage and debris were caused by Syrian and Russian bombing, not by an earthquake. We might also infer, because the man appears to be wearing a jacket that might be part of a fireman's uniform, that he may be an emergency responder carrying a child to safety. And we might infer that the child is one of the injured thousands mentioned in the text and the man is carrying him - perhaps because he is unable to walk - to a hospital or first aid station - or to whatever kind of medical facility has survived the bombing. We might more weakly infer that the photo represents a situation typical of all wars, of adults carrying children to whatever safety might be available. We infer all this (and more) based not just on the textual context, but also on our trust in Amnesty's veracity and reliability, influenced no doubt by "negativity" (Tversky & Kahneman 1974), “myside”, or confirmation biases (Mercier 2017), tempered by epistemic vigilance (Sperber et al. 2010).15
5.6 Text and Photo
The following sections discuss the information we can infer when we contextualize the text with the photo; what the photo adds to the text in the creation of the overall interpretation of the webpage; and how the photo functions in the argument that the situation in Eastern Ghouta is 'catastrophic' because of daily Syrian and Russian bombing attacks and their denial of humanitarian aid.
5.7 Photo, Salience, Ground, and Strength of Communication
The photo takes up approximately a third of the webpage and, except for the yellow background of the Amnesty logo and the yellow background to the SEND NOW button, is the only element of the page with color; consequently it is highly salient and therefore an important element of the document. It foregrounds and thus draws the viewer's immediate attention to the man and boy. However, they are positioned to the right of center thus allowing the burning car to be foregrounded as well. The foregrounded elements are particularly salient, though the burning car may be more so than the man and boy. Because of their salience, the foregrounded elements are strongly communicated. That is, if a viewer were to overlook them in interpreting the photo, they would have missed the most important aspects of what the producer intended the photo to communicate. The flames from the burning car contrast intensely with the backgrounded elements, which are in varying shades of grey and thus communicate less strongly what they represent (Wilson & Sperber 2004: 620), in this case, circumstantial, especially causal, information about the state of the buildings and debris on the ground, which is important, though perhaps not essential, to the interpretation of the photo.
5.8 Deictics: Temporal and Spatial
Like any utterance, the photo indexes temporal and spatial deictic information; that is, the photographic icon is anchored (Bateman 2018: 303) in a specific time and place. Because it is not time and date stamped, the photo communicates its temporal information implicitly. As is usual in discussions of temporal deixis, I distinguish between utterance time and time of reference. The utterance time is the time at which the photo was published in the AI webpage, or alternatively, the time at which a reader accesses the webpage; the reference time is the time interpreters assume that the producer wants them to attribute to the scene in the photo. This is to be inferred from the contextualization of the photo with information from the associated text. It is something along the lines of: this is the kind of thing that happens (frequently) during the siege and bombing of Eastern Ghouta at the time at which the AI webpage on the situation in Eastern Ghouta was published.16 We can make a similar distinction between the utterance and reference place. The utterance place is just the place where the photo was published and/or accessed; the reference place is inferred from the interaction between the photo and the text to be a place in Eastern Ghouta.
5.9 The Photo in the Argument
How does the photo contribute to the argument to convince the webpage reader of the truth of Amnesty’s claims and to persuade them to send the appeal? The text describes, 'tells' (Sperber & Wilson 2015), the situation in general terms. The photo is a synecdochic and metonymic representation of the situation in Eastern Ghouta and, we infer, a specific instance of the situation more generally described by the text. It is to be interpreted as ostensively 'showing' (Sperber & Wilson 2015) what it represents, and by virtue of showing by photo, we infer that we are seeing a specific man, carrying a specific boy past a specific burning car, across a specific debris field, with specific damaged buildings in the background. It is thus a case of "determinate showing" (Sperber & Wilson 2015: 124). Nonetheless, we don't know the names of the man or boy, or exactly where they were when the photo was taken. And we do not know exactly when it was taken. However, we must assume that the photo was not integrated into the webpage merely to represent17 unknown people, places and times.
5.10 Enrichment: Ad-hoc Concepts
One way to theorize our further interpretation of the photo is to analyze it in its context in the way that RT proposes that word meanings in context are to be analyzed (Carston 2002: 40; 2010 and passim; Wilson & Carston 2007). A word's meaning may be inferentially narrowed or broadened (or both) as its interpreter hypothesizes the meaning intended by its producer. For example, the rather broad lexical meaning of hot is inferentially narrowed in context to something like hot enough to burn the drinker's mouth in That coffee is hot issued as a warning. That is, the interpreter will enrich the lexical meaning of hot to the ad-hoc concept hot*18, its optimally relevant, contextualized meaning. We can treat the individuals in the photo like words: the individuals in the photo, while pictorially specific, are otherwise unknown. However, in the context of the AI webpage genre, and especially of the text, the man is narrowed to a man in Eastern Ghouta; similar contextual narrowing applies to the boy, the burning car, and the damaged buildings. The context provides information interpreters can use, indeed are expected to use, to further narrow our interpretation of the photo: the buildings were damaged by Russian and Syrian bombing; the car was set alight by that bombing; the man is carrying the boy to whatever safety and medical attention might be available. We can readily imagine other such inferences an interpreter might derive from the contextualization of the photo by the webpage genre and text.
Just as word meanings can be narrowed in context they may also be broadened, e.g., for metaphorical interpretation (Sperber & Wilson 2015), the meaning of the photo can likewise be broadened. It is a photo of a scene in bombed and besieged Eastern Ghouta, but it may be interpreted as a synecdoche and metonym for all of the scenes of devastation and desperation in Eastern Ghouta, and perhaps in all wars. Interpreters will infer at least that the photo represents a scene in Eastern Ghouta and is representative of all such scenes in Eastern Ghouta; if they failed to do that, they would have failed to construct an adequate hypothesis about the producer's communicative intention. However, they may derive any number of further inferences, but they do this on their own dime, so to speak.
5.11 Explicatures and Implicatures
Explicatures, as noted above, are the logical forms of "explicitly communicated assumption[s]", that is, logical forms (Sperber & Wilson 1995 [1986]: 182), which have been contextually enriched. I take the logical forms represented by the photo to be versions of the items (a-d) listed in section 5.3 above. These are inferentially enriched with information from the webpage as its viewer searches for an optimally relevant interpretation.19 I suggest that the contextually enriched explicatures of the photo are, at a minimum:
A specific man is carrying a specific boy past a specific burning car which is situated on specific building debris in front of specific severely damaged buildings in Eastern Ghouta.
The Syrian and Russian bombing caused the damage to the buildings and the car-fire.
The man is a rescue worker; the boy has been injured, perhaps by the bombing, perhaps by falling masonry (note the dirt on his face) caused by the bombing.
The man, boy, car, buildings and their relationships are specific but representative instances of the effects of the Syrian and Russian siege and bombing of Eastern Ghouta.
Higher level explicatures might include:
The Amnesty staffers responsible for the webpage believe the explicatures.
They want the viewers to believe the explicatures.
They want the viewers to integrate the photo explicatures with the interpretation of the text.
They want the viewers to act as the document demands.
If photos license explicatures, then, per RT, they must also license implicatures – implicated premises and conclusions. The following are some implicatures that might be derived from the photo and its context:
Implicated premises:
The bombing that caused the car to burn must have occurred very shortly before the photo was taken.
The bombing may be continuing.
The Russian and Syrian governments are responsible for the bombing.
Implicated conclusions:
The situation must be very dangerous.
The man and boy, and other people in Eastern Ghouta, are in continuing danger.
Many people may be being injured or killed by the bombing and siege.
Dangerous situations should be made safe.
To make this dangerous situation safe, the siege and bombing must be stopped.
Viewers should urge the Russian and Syrian governments to stop the bombing.
Viewers should send the appeal letter to Presidents Putin and Assad.
In fact, the webpage text explicitly articulates the consequences of the siege and bombing on the people of Eastern Ghouta:
[T]hey are now being attacked daily, killed and maimed by their government. Children and elderly people are dying of malnutrition and lack of medication.
The Amnesty webpage genre thus integrates explicatures and implicatures of the photo with explicatures and implicatures of the text into a coherent argument, designed to persuade webpage readers to act to redress the situation. The webpage is also explicit about how its readers should act. They should "Tell the Syrian and Russian Governments to end the attacks and lift the siege on Eastern Ghouta immediately" by sending the appeal. The webpage will have succeeded in its persuasive purpose if its reader does click the SEND NOW button at the bottom of the page.
5.12 Impressions
Although verbalizing the elements of the photo as explicatures and implicatures allows us to articulate its individual and contextualized contributions to the document, there remains a strong feeling that the photo contributes much more and much differently than is verbalizable. Bateman (2018: 303-304) says that "semiotic modes may deliver material that does more than just contributing to dicents", that is, more than just Peircean modality-independent propositions. Sperber & Wilson (2015) suggest adding ‘impressions’ to the theoretical apparatus of Relevance Theory:
[A]n impression is a change in the manifestness in an array of propositions which all bear on our understanding of the same phenomenon, answering the same question, or deciding the same issue. (ibid.: 138)
"However fleetingly", impressions may become
manifest or more manifest they become more likely to be attended to, and more likely to be taken as true [and] therefore more likely to influence the interpretive decisions of their audiences. (ibid.: 137)
An impression's array of propositions results from
changes in patterns of activation, none of which would properly speaking amount to the fixation of a distinct credal representation, but the totality of which would correspond to the formation of an impression. (ibid.: 137)
So, an impression
does not simply pick out a weak credal attitude; it picks out a certain type of vague information basic for such an attitude. (ibid.: 138)
Its communicated meaning "cannot be paraphrased without loss" (Sperber & Wilson 2015: 122).
Sperber & Wilson's (2015) descriptions of impression suggest a way to characterize the residue of information communicated by a contextualized photo beyond its explicatures and implicatures. It allows us to have our analytic cake and to eat it: pictures can be cashed in for words, contra Davidson, but they always also communicate an excess of information that is not and perhaps cannot be verbalized, and on which Sperber & Wilson's notion of impression gives us some analytic purchase. This residue is indeterminately shown because its "intended import cannot be rendered as a proposition at all" (Sperber & Wilson 2015: 124), though the photo may communicate a range of rhetorical appeals, particularly combining ethos and pathos. It contributes to ethos, a credal attitude, because we take it to be a genuine icon of an Eastern Ghouta situation or event; it contributes to pathos by virtue of its representation of the plight of the man and boy due to devastation and danger it represents, and by contributing to these, contributes to the fulfillment of AI’s communicative intentions.
6 Summary and Conclusions
The discussion above demonstrated the explanatory value of applying RT to the analysis of multi-modal arguments. It showed how an Amnesty International webpage provides evidence for two sets of assumptions, one derived from the text, the other from the photo and how each set is enriched by its contextualization by the other. Some of the assumptions are explicatures, both basic and higher. The latter are evaluations of the situations described and speech acts exhorting audiences to behave in particular ways. Other assumptions are implicatures, both implicated premises and conclusions, also including exhortations to the viewers to act. The document, and especially the photo, also provides evidence for a range of ‘impressions’ that support the ‘credal attitude’ the Amnesty producers of the webpage hope will be adopted by its audiences, who will thereby by persuaded to send the appeal to its intended recipients. The analysis provides a model that is applicable to the analysis of multi-modal arguments in general.
References
Atkin, Albert. (2022). Peirce’s Theory of Signs. In: Zalta, Edward N. & Uri Nodelman (Eds.) The Stanford Encyclopedia of Philosophy (Fall 2022 Edition) (https://plato.stanford.edu/archives/fall2022/entries/peirce-semiotics/; 29-06-2023).
Atkin, Albert. (2013). Peirce's Theory of Signs. In: Zalta, Edward N.(Ed.): The Stanford Encyclopedia of Philosophy (Summer 2013 Edition) (https://plato.stanford.edu/archives/sum2013/entries/peirce-semiotics/; 29-06-2023).
Bateman, J. A. (2006). Introduction to the special issue on genre. In: Linguistics and Human Sciences. 2 (2): 177-183.
Bateman, J. A. (2018). Position paper on argument and multimodality: Untangling the connections. In: International Review of Pragmatics. 19: 294-308.
Carston, Robyn. (2002). Thoughts and utterances: The pragmatics of explicit communication. Malden, MA: Blackwell.
Carston, Robyn. (2010). Explicit communication and "free" pragmatic enrichment. In Belén Soria & Esther Romero (eds.). Explicit communication: Robyn Carston's pragmatics. Basingstoke, UK: Palgrave MacMillan, 217-285.
Cooren, François. (2015). Organizational discourse: Communication and constitution. Polity Press.
Davidson, Donald. (2013). What metaphors mean. In: Ezcurdia, Maite & Robert J. Stainton (Eds.): The semantics-pragmatics boundary in philosophy. Ontario: Broadview Press, 453-465.
Delahunty, Gerald (2018). Amnesty International (AI) and philanthropic fundraising (PF) appeals: A comparative move analysis. In: Goetsch, Hans (Ed.): The meaning of language: Proceedings of the 26th Scandinavian Conference of Linguistics. Newcastle upon Tyne. UK: Cambridge Scholars Press.
Delahunty, Gerald. Language, text, and ideology in Amnesty International appeal letters. (In preparation).
Douven, Igor (2017). Abduction, In: Zalta: Edward N.: The Stanford Encyclopedia of Philosophy (Summer 2017 Edition) (https://plato.stanford.edu/archives/sum2017/entries/abduction/; 29-06-2023).
Dutilh Novaes, Catarina (2022). Argument and Argumentation. In: Edward N. Zalta & Uri Nodelman (eds.): The Stanford Encyclopedia of Philosophy (Fall 2022 Edition). (https://plato.stanford.edu/archives/fall2022/entries/argument/; 29-06-2023).
Fillmore, Ann & Johnny Cook. 2023. Multimodal communication. In: Critical reading, critical writing: A handbook to understanding college composition. Curated/composed by the English faculty at Howard Community College.
Forceville, Charles. (2014). Relevance Theory as a model for analysing visual and multimodal communication. In: David Machin (Ed.).Visual Communication: Handbooks in Communication Science. Volume 4: 51-70.
Forceville, Charles & Billy Clark (2014). Can pictures have explicatures? In: Linguagem em Discurso, 451-472 (http://dx.doi.org/10.1590/1982-4017-140301-0114; 29-06-2023).
Forst, R. (2012). The right to justification: Elements of a constructivist theory of justice. Translated by J. Flynn. New York: Columbia University Press.
Grice, H. Paul. (1975). Logic and conversation. In: Cole, Peter & Jerry Morgan (Eds.). Syntax and semantics 3: Speech acts. New York: Academic Press, 41-58.
Grice, H. Paul. (1989). Studies in the ways of words. Cambridge, UK: Cambridge University Press.
Gumperz, John J. (2018). Interactional sociolinguistics: A personal perspective. In: Tannen, Deborah, Heidi E. Hamilton & Deborah Schiffrin (Eds.). The handbook of discourse analysis. 2nd ed. Oxford UK: Wiley Blackwell.
Hobbs, J.R. (2004). Abduction in natural language understanding. In: Horn, L. R. & G. Ward (Eds.). The handbook of pragmatics. Malden, MA: Blackwell, 724-741.
Jewitt, Carey. (Ed.) (2009). The Routledge handbook of multimodal analysis. London & New York: Routledge.
Jewitt, Carey. (2013). Multimodal methods for researching digital technologies. In: Price, Sara, Carey Jewitt & Barry Brown (Eds.). The SAGE handbook of digital technology research. London, UK: SAGE Publications.
Jewitt, Carey (Ed.) (2017). The Routledge handbook of multimodal analysis. (2nd ed.) London & New York: Routledge.
Larson, B. N. (2018). Bridging rhetoric and pragmatics with relevance theory. In: Strassheim, J. & H. Nasu (eds). Relevance and irrelevance: Theories, factors, and challenges. De Gruyter, 69-96.
Lave, Jean & Etienne Wenger (1991). Situated learning: Legitimate peripheral participation. Cambridge UK: Cambridge University Press.
Longacre, Robert E. (1992). The discourse strategy of an appeal letter. In: Mann, Willian C. & Sandra A. Thompson (Eds.) 1992: 109-130.
Mann, William C. & Sandra A. Thompson (Eds). (1992). Discourse description: Diverse linguistic analyses of a fund-raising text. Amsterdam: John Benjamins Publishing.
McCall Smith, Alexander. (2005). The Sunday philosophy club. New York: Anchor Books.
Mercier, H. (2017). Confirmation bias – Myside bias. In: Pohl, R. F. (Ed.): Cognitive illusions: Intriguing phenomena in thinking, judgment and memory. Routledge/Taylor & Francis Group, 99-114.
Morgan, Marcyliena. (2014). Speech communities. Cambridge, UK: Cambridge University Press.
Nettel, Ana Laura & Georges Roque (Eds.) (2012a). Introduction. In: Roque 2012: 1-17.
Nettel, Ana Laura & Georges Roque (2012b). Persuasive argumentation versus manipulation. In: Roque 2012: 55-69.
Oswald, Steve. (2016). Rhetoric and cognition: Pragmatic constraints on argument processing. In: Cruz, M. Padilla (Ed.): Relevance theory: Recent developments, current challenges and future directions. Amsterdam: John Benjamins, 261-285.
Rohan, O., R. Sasamoto & R. Jackson (2018). Argumentation, relevance theory, and persuasion. In: International Review of Pragmatics 10: 219-242.
Roque, Georges (2012) (Ed.) Special issue on persuasion and argumentation. Argumentation 26.
Roque, Georges (2017). Rhetoric, argumentation, and persuasion in a multimodal perspective. In: Tseronis, A. & C. Forceville (Eds.): Multimodal argumentation and rhetoric in media genres. Amsterdam: John Benjamins.
Solin, A. (2011). Genre. In: Zienkowski, J. & J.-O. Ӧstman 6 J. Verschueren (Eds.). Discursive pragmatics: Handbook of pragmatics highlights 8: 119-134.
Sperber, D., F. Clément, C. Heintz, O. Mascaro, H. Mercier, G. Origgi, & D. Wilson. (2010). Epistemic vigilance. Mind & Language. 25 (4), 359-393. (https://doi.org/10.1111/j.1468-0017.2010.01394.x; 29-06-2023).
Sperber, D. & D. Wilson. 2015. Beyond speaker’s meaning. In: Croatian Journal of Philosophy. 15 (44): 117-149.
Sperber, D. & D. Wilson (1995 [1986]). Relevance: Communication and cognition. (2nd ed.). Oxford: Blackwell.
Swales, J. M. (1990). Genre analysis: English in academic and research settings. Cambridge University Press
Tseronis, A. & C. Forceville. (2017). Argumentation and rhetoric in visual and multimodal communication. In: Tseronis, A. & C. Forceville (eds.): Multimodal argumentation and rhetoric in media genres. Amsterdam: John Benjamins, 1-23.
Tversky, A & D. Kahneman (1974). Judgment under uncertainty: Heuristics and biases. Science 185: 1124-1131.
Unger, Christoph (2006). Genre, relevance and global coherence: The pragmatics of discourse type. London, UK: Palgrave Macmillan.
Unsworth, Len & Chris Cléirigh (2014). Multimodality and reading: The construction of meaning through image-text interaction. In: Jewitt, C. (Ed.), Routledge handbook of multimodal analysis (2nd ed.). Milton Park, UK: Routledge, 176-188.
Wenger, Étienne (1998). Communities of practice. Cambridge, UK: Cambridge University Press.
Wharton, Tim (2009). Pragmatics and non-verbal communication. Cambridge, UK: Cambridge University Press.
Wilson, D. (2016). Relevance theory. In: Huang, Y. (Ed.). The Oxford handbook of pragmatics. Oxford: Oxford University Press, 79-100.
Wilson, Deirdre & Robyn Carston (2007). A unitary approach to lexical pragmatics: Relevance, inference and ad hoc concepts. In: Burton-Roberts, Noel (Ed.). Pragmatics. Basingstoke: Palgrave.
Wilson, Deirdre & Dan Sperber (2004). Relevance theory. In: Horn, L. R. & G. Ward. (Eds). Handbook of pragmatics. Oxford, UK: Wiley Blackwell, 607-632.
Yagoda, B. (2018). The cognitive biases tricking your brain: Science suggests we’re hardwired to delude ourselves. Can we do anything about it? in: The Atlantic. (https://www.theatlantic.com/magazine/archive/2018/09/cognitive-bias/565775/; 29-06-2023)
______________________________
1 My thanks go to Luciana Marques and Andreea Calude for their valuable comments on earlier versions of this paper.
2 Abduction is reasoning to the best explanation (Douven 2017).
3 Unsworth & Cléirigh (2014) provide a model of text / image interaction based on systemic-functional approaches to discourse. For other approaches: Jewitt (2009, 2013, 2017).
4 Hobbs (2004) explicitly compares abductive reasoning in discourse interpretation with Relevance Theory analyses, saying that what can be done in one framework can also be done in the other, but that his abductive approach "has a more compelling justification" though "the two frameworks are almost entirely compatible" (ibid.: 2739-2740).
5 Consider the following interpretation of a picture and its evaluation by a novelist (McCall Smith 2005: 20-21):
She found herself gazing at the label of a bottle of olive oil which Cat had placed in a prominent position on a shelf near the table. It was painted in that nineteenth-century rural style which the Italians use to demonstrate the integrity of agricultural products. This was not from a factory, the illustration proclaimed; this was from a real farm, where women like those shown on the bottle pressed the oil from their own olives, where there were large, sweet-smelling white oxen and, in the background, a mustachioed farmer with a hoe. These were decent people, who believed in evil, and in the Virgin, and in a whole bevy of saints. But of course, they did not exist anymore, and the olive oil probably came from North Africa and was rebottled by cynical Neapolitan businessmen who only paid lip service to the Virgin, when their mothers were within earshot.
6 The webpage is available at: https://act.amnestyusa.org/page/21064/action/1?locale= en-US; 29-06-2023).
7 See the RT Online Bibliographic Service athttps://personal.ua.es/ francisco.yus/rt./html ; 28-06-2023).
8 Typical online lists claim that there are five modes: the linguistic, the visual, the aural, the spatial, and the gestural (e.g. Fillmore and Cook 2023). As the linguistic mode is defined as written or spoken, it is unclear, at best, how it is to be distinguished from the aural and the visual modes. These five modes recur, especially in materials aimed at students and document designers, and each may be further subdivided.
9 Solin (2011: 119-134) offers a brief reprise of the history of the term genre and the concepts it has denoted, particularly "as [maleable GD] forms of semiotic practice [that] are socially based" (ibid.: 119; footnote omitted). For rhetorical move and step approaches to the analysis of genres: Swales 1990.
10 Larson (2018: 70) provides a model for “bridging rhetoric and pragmatics with relevance theory.”
11 It is not the goal of this paper to model how multimodal argument like that of the webpage might or might not be effectively persuasive. For a relevance theoretic model of rhetorical effectiveness: Oswald 2016.
12 While the lettering varies in size and prominence, it is consistently sans serif, presumably because this font is easier than serif fonts to read online.
13 Sperber & Wilson 1995 [1986]. For recent synopses of RT relevant to the issues discussed here: Forceville 2014, Forceville & Clark 2014, and Rohan, Sasamoto & Jackson 2018.
14 Andreea Calude (2023, personal communication) noted that we might infer that there appears to be trust between the two people (the child does not seem uncomfortable or afraid of being in the man's arms).
15 Yagoda (2018) provides a readily accessible review of these and myriad other biases. (https://www.theatlantic.com/magazine/archive/2018/09/cognitive-bias/565775/; 29-06-, 2023).
16 AI’s report on the siege was published in August 2015 (https://www.amnesty.org/en/documents/mde24/2079/2015/en/; 29-06-2023)
17 "Represent" is glossed in Sperber & Wilson (2015: 137; italics in original) as to be "broadly understood as meaning fulfill the function of making some information available for processing".
18 RT uses one or more asterisks to indicate ad hoc concepts.
19 Forceville & Clark (2014: 470-477) are more cautious in extending the term explicature to the explicit information derivable from pictures. Forceville (2020), though citing Yus' (2016) argument that "visual content can … lead to visual explicatures" (citation omitted, emphasis in original), adopts the "stronger position … that at least some (parts of) visuals can trigger explicatures … [and] can singlehandedly present information of which we would say: 'this is true' or 'this is false'" (ibid.: 87-88).
Author:
Gerald Delahunty, PhD
Professor of Linguistics and English
Director of Language Programs
English Department
Colorado State University
Email: gerald.delahunty@colostate.edu
Fort Collins, Colorado, USA