A Multimodal Approach to Digital Publications. Presentation at HDH 2015

On the 6th of October I will present at the second edition of "Humanidades Digitales Hispanicas" about a "Multimodal Approach to Digital Publications". The presentation tries to outline basic problems which still hinders sustainable development of digital publications from a viewpoint of Multimodal Analysis and tries to indicate two conceptual tools from this field which might help to improve modeling and infrastructure efforts for digital publications. The slides are at:

A Multimodal Approach to Digital Publications - (Source)

The abstract with which I applied:

In my presentation I will introduce a new approach to research on digital publications, especially in the context of e-Science and Digital Humanities. The content of this presentation reflects an ongoing research project which aims at defining an OAI-ORE based model for multimodal digital publications.

Problem: The State of Digital Publication Forms

Research is discursive practice. This claim is applicable in two ways: first, research is a social activity and second, knowledge is a phenomena which resides and is created in discourse. Publications are objects that used to support well both aspects because they partly suspended space and time in communication. Globally, this quality of publications improved the conditions for knowledge dissemination. Structurally, it improved the conditions for the creation of more complex meaning and reliable knowledge. Publications take discourse out of the flow of space and time and re-materialize it into an object which moves across space and time instead. However, materiality dramatically changed shape in digital world and there are activities allover to change the shape of publications likewise.

Approaches like Enhanced Publications (Sierman, Schmidt, and Ludwig 2009; Holl 2012), Scientific Publication Packages (Hunter 2008) or Semantic Publications (David Shotton 2009; D. Shotton et al. 2009; Bourne et al. 2012), Nano Publications (Mons and Velterop 2009; Groth, Gibson, and Velterop 2010) among many others propose to make publications modular, formally annotated, distributed, multi-media, including data and software, a living entity, automatically reproducible and much more. In contrast to these abundant activities which started at least in 1995 with the ACM Publishing Plan (Denning and Rous 1995) its success seems quiet limited. This is especially true for the field of the Humanities. The PDF is still the most common digital publishing strategy (David Shotton 2009). Where publications are more sophisticated its richness is bound to the portal where it is produced or presented, like it is the case for Scalar (ANVC), myExperiment (“MyExperiment”) or Authorea (“Authorea”).

Digital publications are often ideologically bound to certain domains and to the End of Theory debate (Anderson 2008). Contrasting examples from the field of DH (ANVC; “Journal of Digital Humanities”) explore new possibilities of multimedia writing but do not offer a generalizable model. Hence, new publications lack sustainability in terms of concepts and preservation. Accordingly, there are many models and few tools for their production. Their potentials remain unexploited and the reality of digital discourse that takes place is rarely objectifiable.

Approach: Multimodal Analysis and Systemic-Functional-Grammar

The benefit of publications is twofold as I claimed at the beginning. However, digital publications tend to focus on the communication and dissemination aspect of publications and neglect or take for granted aspects of meaning making and knowledge – the ideological background. Naturally, these aspects are of special importance for the field of Humanities which study and communicate in distinct types of meaning material. It is this part which suggest changes in the way knowledge and thus publications function in society. The question of digital publications is not how to do old things better due to digital technology. The question is how to do things good in a changing world of digital technologies. Thus, what is missing in the debate around digital publications is a theoretical framework which is capable to contribute both to the sociocultural re-evaluation of knowledge and publications as well as to the formalization and implementations of digital publications. In this respect I propose the field of and Multimodal Analysis.

Multimodal Analysis is a research field which grew out of Social Semiotics, a branch in linguistics initiated by Michael Halliday (1978; 1985). One of the main claims of Social Semiotics stresses that signs are socially constructed and thus that there is no categorical realm of signs, for example language. Multimodal Analysis is following this claim by investigating semiotic and grammatical structures in linguistic and non-linguistic resources like: visuals, audios, colour, space, rhythm and so forth. Furthermore, Multimodal Analysis investigates how these resources relate to each other grammatically. In doing so Multimodal Analysis offers sophisticated insights on structural and semantic aspects of discourse beyond language. Additionally, the historical orientation in a social approach to language makes it possible to evaluate how the configuration of signs and grammar as a condition for meaning and knowledge change over space and time.

By stretching Multimodal Analysis it is possible to make two claims:

  1. computation is one but not the only theme in and for digital publications as many existing models suggest. Instead, the omnipresence of computation provokes a proliferation of strategies (interfaces) to create knowledge;
  2. the dominant function of publications in a digital world will shift from dissemination to the creation of complex knowledge which is hardly organisable outside of such a framework like publications.

Benefit: How Multimodal Analysis improves the Conditions for Digital Publications

The design of digital publications may benefit from Multimodal Analysis in several ways: Its grammatical and functional contents are very useful to derive semantics for a formal model of digital publications. The sociocultural perspective is suitable to approach digital publications in a sustainable way. This will promise to improve conditions for long term preservation and for authoring tools. Multimodal Analysis and the realization of multimodal digital publications will help to advance digital literacies, thus the capacities to write and read multimodal texts in a publicly understandable that means rule oriented way. Finally, it will support the adoption of digital technologies and computational methods in the Humanities because they are contextualized in a way which is integrative and not exclusive.

In the presentation I will give a short overview about examples of digital publications for the purpose to expose its common problems. I will substantiate the claim that multimodal analysis is a promising approach to frame modeling tasks for digital publications. Finally, I will indicate how certain high-level concepts of multimodal analysis can be transformed to fit the needs of modeling digital publications. Publications which are capable to include data, code and media of varying type and to organize them in distinct ways.