Explicit Art Historical Image Referencing on a Big Scale

• Preliminaries
• Chapter one: Textual image annotation
• Chapter two: Image annotation to image detail – the image footnote
• Chapter three: Image detail to image detail – the image cross-reference
• Last chapter: Big data

---

Referencing of image details has its difficulties in the internet. A technology for doing this is proposed along with a short analysis of rhetoric forms and social implications.

Preliminaries

It is one of the most important insights of the theory and history of media that knowledge itself depends heavily on the media technologies in use in their respective disciplines. Friedrich Kittler (1943-2011) defines his very German theory as, “Media determine the situation […].” [Fn 01] Scholars of the history of art history like Heinrich Dilly [Fn 02] or Donald Preziosi [Fn 03] have made it clear that the media in use always also lay out the ground for what is possible to know about an artistic image.

Let me give just two examples. When August Wilhelm von Schlegel in 1817 described Fra Angelico's Coronation of Mary [Fn 04], he used engraved drawings by Wilhelm Ternite to do so and naturally these largely consisted of contours. The famous Florentine artist, noted for his use of color, consequently suffered from a misjudgment about his skills in showing bodily detail and, because drawings prefer the elegant line to the anatomical shadow of hypodermic muscles, was judged to be much more ethereal than good color reproductions today suggest he was. Secondly, while Heinrich Wölfflin’s pairs of opposed precepts in art, for example the linear and the painterly, [Fn 05] from 1917 work perfectly well and are overwhelmingly convincing to an observer exposed to pairs of images, as in a double slide projection, they fail to do so when he is confronted with a different arrangement of images, for example only one of the examples at a time. [Fn 06] Linear works of art then suddenly gain painterly qualities and paintings reveal their passion for contour.

Media determine the situation, and after engravings and double slide projections, PowerPoint with data projectors arrived. Now the internet is the leading medium, introducing endless interlinkage of material, even to a discipline that is as amiably technologically reserved as art history.

So which epistemic structures will we see emerging from the net? Deleuzian rhizomatics? Zuckerbergian surveillance? As we do not know we have to try.

But there are particular problems. Because they are continuous images are intrinsically analogue. As W. J. T. Mitchell wrote, referring to Nelson Goodman, “The image is syntactically and semantically dense in that no mark may be isolated as a unique, distinctive character (like a letter in an alphabet).” [Fn 07] As long as images are treated as monolithic, this makes interlinkage easy, giving rise to the plethora of image databases in the internet. But it makes it hard to reference a detail within the image. This may be the reason why the decomposition of an image is normally a disciplinary no-go in art history – or a joke, as in Ursus Wehrli's “Kunst aufräumen” (Cleaning Up Art). [Fn 08] So there are two obstacles to the explicit referencing of image details, a technical one and one of scholarly habits. To this day, the usual way to refer to image details is by ekphrasis, by verbally describing the visual. And art history has taken this to beautiful heights, but now we must have technical solutions that approximate text interlinkage in the World Wide Web in order to stay up to date with the latest media, since they determine, as we know, the situation. What we have to expect is that these media will develop a will of their own, an autonomy and epistemic independence that will doubtlessly impregnate our knowledge about the artistic image.

I would like to now describe our [Fn 09] approach to this problem. Since 2006 we have worked on the technology and methodology of image detail interlinking in the internet. This work is documented at the websites www.hyperimage.eu and www.meta-image.de.

At www.hyperimage.eu by following the path "english version > examples" you will find most of the examples I refer to in the following.

Allow me to make a few remarks before I describe the results of our project. All observing, marking and linking of image details involves the use of human intelligence. Although there are clever image segmentation algorithms available, these are by far not clever enough for art historical purposes. In our work, all image marking and interlinking is done with an editor we have built. Image network browsing is done in the web. In the standard configuration the images come from a repository such as prometheus, [Fn 10] the distributed digital image archive for research and education, but also from other repositories and local storage. All software is open source and freely available from the Sourceforge platform.

Chapter one: Textual image annotation

Our first project that treated images as the underpinnings of huge amounts of detailed information was a medieval world map, the thirteenth-century Ebstorf Map, which, measuring 3.5 by 3.5 square meters, holds about 1600 text fragments annotating and commenting on image details of about the same number. Long before the book became the leading medium, images and text were in close companionship, inducing a wild thinking instead of strict logical chains of arguments that later became typical for the book era.

The document structure, done with HyperImage, is a simple image annotation, pointing at a detail reveals the corresponding text (see: www.hyperimage.eu > english > Examples > EbsKart):

For scholarly purposes we made the image markup zoomable:

The idea is simple enough, but the effect is powerful, particularly when pieces of art do not physically fit into the book universe. The alienation of historic distance is lifted. We now can read as we are used to reading. Give it a try at the url indicated above.

Chapter two: Image annotation to image detail – the image footnote

The situation becomes more internet-like when dealing with a large number of image files. Anna Oppermann (1940-1993), documenta and Biennale participant, did huge image ensembles consisting of thousands of pieces. She pictured her own pictures, growing image universes. From her we learned how to deal with big networks of images.

The source of an image reference is a polygon that zooms with the image; the target is the image that the artist herself pictured (see: www.hyperimage.eu > english > Examples > HyperKunsthalle):

Selecting the canvas inside the polygon yields

And finally selecting the marked bottle shows it as initially pictured by the artist:

The whole structure is tree-like, hierarchical. Compiling an image index is an obvious solution. The index of the bottle consists of all the places from which it is referenced:

Selecting one of the findings carries the viewer to it, to the inside of the image where the polygon surrounding the indexed detail makes itself visible.

Now we know the explicit structure of Oppermann's interlinking. It is, by the way, complex in the sense of mathematical network theory. [Fn 11] The artistic wonder of the growth and stability of Oppermann's works is now revealed: it is the very nature of those complexities that scale-free networks also possess. [Fn 12] There is a deep resemblance between this type of art and the internet, both stemming from the last quarter of the twentieth century. [Fn 13] The network as an epistemic structure is beginning to emerge.

Chapter three: Image detail to image detail – the image cross-reference

The venerable business of iconology connects image motifs. This is what our software now allows to be done explicitly, and more accurately than by ekphrasis. Aby Warburg did so by using woolen thread with the Menomosyne Atlas, we do so with address pointers, indexed and published in the web. [Fn 14] Hubertus Kohle from Munich's Art History Department analyzes [Fn 15] the pictorial communication of the conflicting parties during the French Revolution with symbols like the tree of freedom mutated to expressions of horror, iconological entities crisscrossing through the caricatures of that time, mutually influencing each other in a complex, interwoven way, network-like. The linear argument is replaced by a web of references, shown here some of the stations of just one thread (see: www.hyperimage.eu > english > Examples > HyperMedia).

Every colour indicates a distinct thread of iconological mutations, meandering through the image corpus. They intersect again and again, weaving a web of context.

Last chapter: Big data

What happens if the image corpus is not fixed, if interlinking becomes a collective activity? The answer to this is waiting in the network: meaning is being created by linking. The meaning of a term in the World Wide Web is the place where it resides within its network of references.

Niklas Luhman confided to us his Zettelkasten, his card index box, which was to him what the World Wide Web is to us: “Every note is an element that gains its qualities only by virtue of the network of references and cross-references in the system.” [Fn 16] For him, the note was the node. And “[c]ontrary to the structure of updatable references, the importance of what is noted concretely is small.” [Fn 17] This was, following Philippe-Alain Michaud, not very far from Warburg’s methodology. “The enigmatic formula ‘iconology of the interspace’ Warburg used in his diary – an iconology that did not concern the meaning of the figures […] but the varied relations that they took on in a complex, autonomous dispositif, which could not be reduced to the order of the discourse – suggests that he thought of mnemosyne as topographic from the beginning.” [Fn 18]

Since references are explicit in the Web, a machine can extract this meaning as a ranked index of terms. The top ranked in this kind of index are being used much more frequently than the lower ranked ones, reinforcing at the same time their status as defining items. By this recursive and self-amplifying mechanism, meaning is constituted.

This resonates very well with the term Sinn, or meaning, which Niklas Luhman introduced into his theory. Meaning is the medium in which selections are made, events becoming actual instead of remaining virtual, triggering new selections to actualize virtualities. And the emergence of meaning takes place in a similar fashion in language. Think of the network of all the selections as being laid out explicitly and you arrive at something similar to the World Wide Web with its interlinked documents.

So with explicitly interlinked image detail references, a new way of constituting meaning is becoming possible. By analyzing the web of references, it should be possible to create meaning that is personally and socially constructed – and machine readable.

Actually, this is not fundamentally different from the constitution of meaning in a society, where traditionally more important scholars also had greater influence on the definition of terms than lesser ones, and so became even more influential. But there is one difference, and this difference will bring about new structures. What and who are the most important – and thus the most influential – nodes of a network evolve and emerge from the network itself.

Iconology could turn into an emergent social phenomenon.

By analyzing the pictorial references in the web and the activities of the users, it will be possible to find out what the varied meanings of an image are, dynamically changing from person to person, from time to time, from place to place, or in its broadest sense, what the collective meaning given by a vast number of users to a particular object is.

We could, for example, find out what stylistic distinctions users make, whether they follow the established epochs of art history. We could find out which ones might belong to the central ‘canon’ of art history, which ones the most innovative, with the broadest scope of interrelations. Maybe it could even be possible to predict the art history's central interests, may be even the topic of next year's annual conference. [Fn 19]

Using software like HyperImage for art history could mean a step into ‘big data’. [Fn 20] The social science strand in art history could profit from today's data-centric methodologies and reveal insights that were impossible until now and might greatly differ from traditional understandings. [Fn 21]

---

Fußnoten

[01] Friedrich Kittler, Grammophon Film Typewriter (Berlin: Brinkmann & Bose, 1986), 3: "Medien bestimmen unsere Lage, die (trotzdem oder deshalb) eine Beschreibung verdient." [zurück]

[02] Heinrich Dilly, Lichtbildprojektionen – Prothesen der Kunstbetrachtung, in Kunstwissenschaft und Kunstvermittlung, in ed. Irene Below (Gießen: Anabas, 1975), 153-72. [zurück]

[03] Donald Preziosi, Rethinking Art History (New Haven & London: Yale University Press, 1998). [zurück]

[04] August Wilhelm von Schlegel, Mariä Krönung und die Wunder des heiligen Dominicus, nach Johann von Fiesole, in funfzehn Blättern; gezeichnet von Wilhelm Ternite. Nebst einer Nachricht vom Leben des Mahlers und Erklärung des Gemähldes, (Paris: Griechisch-lateinisch-deutsche Buchhandlung, 1817). [zurück]

[05] Heinrich Wölfflin, Kunstgeschichtliche Grundbegriffe in Das Problem der Stilentwicklung in der neueren Kunst ed. Heinrich Wölfflin. (Munich: Bruckmann, 1917), 20-79: Das Lineare und das Malerische. [zurück]

[06] I am indebted to Holger Kuhn for sharing such insights with me. [zurück]

[07] Cited in William J. T. Mitchell: Iconology – Image, Text, Ideology (Chicago, London: Univ. of Chicago Press, 1986), 67. [zurück]

[08] http://kunstaufraeumen.ch/shop/shop.htm [zurück]

[09] That is, Leuphana University Lüneburg (Institute for Aesthetics and Culture of Digital Media), Humboldt University Berlin (Computer and Media Service and Institut für Kunst- und Bildgeschichte) and University Cologne (prometheus e. V.). [zurück]

[10] http://www.prometheus-bildarchiv.de [zurück]

[11] Martin Warnke and Carmen Wedemeyer, Documenting Artistic Networks: Anna Oppermann’s Ensembles are Complex Networks! in Leonardo – Journal of the International Society for the Arts, Sciences and Technology 44, no. 3 (2011), 258-259. [zurück]

[12] Albert-László Barabási, Linked (New York: Plume, 2003); Albert-László Barabási and Eric Bonabeau, Scale-Free Networks, Scientific American (May 2003): 50-59. [zurück]

[13] See Claus Pias' foreword to Uwe M.Schneede and Martin Warnke, (eds.) Anna Oppermann in der Hamburger Kunsthalle (Hamburg: Hamburger Kunsthalle, 2004). [zurück]

[14] Martin Warnke, God Is in the Details, or The Filing Box Answers in Imagery in the 21st Centur, eds. Oliver Grau and Thomas Veigl, (Cambridge, Massachusetts: The MIT Press, 2011), 339–374. For our Mnemosyne excerpt see http://hyperimage.leuphana.de/warnke/HI_Mnemosyne/ [zurück]

[15] Rolf Reichardt and Hubertus Kohle, Visualizing the Revolution (London: Reaktion Books Ltd., 2008). [zurück]

[16] Niklas Luhmann, Biographie, Attituüden, Zettelkasten in Short Cuts, Niklas Luhmann, ed. Peter Gente, et al. (Frankfurt: Zweitausendeins, 2000). Translation by the author. [zurück]

[17] Niklas Luhmann, Kommunikation mit Zettelkästen. Ein Erfahrungsbericht in Öffentlicher Meinung und sozialer Wandel, ed. Horst Baier et al. (Opladen: Westdeutscher Verlag, 1981), 227. Translation by the author. [zurück]

[18] Philippe-Alain Michaud, Zwischenreich. Mnemosyne, ou l’expressivité sans sujet in Les Cahiers du Musée national d’art moderne, Nr. 70, Winter (1999–2000): 42-61. http://trivium.revues.org/index373.html Retrieved 2 August 2009. Translated by the author. [zurück]

[19] See for example, Albert-Laszlo Barabasi, Bursts: The Hidden Pattern Behind Everything We Do (New York: Dutton, 2010). [zurück]

[20] http://wiki.myexperiment.org/images/Wikiomics.pdf [zurück]

[21] See the pioneering works by Maximilian Schich, Rezeption und Tradierung als komplexes Netzwerk. Der CENSUS und visuelle Dokumente zu den Thermen in Rom (2009). http://archiv.ub.uni-heidelberg.de/artdok/volltexte/2009/700/ [zurück]

---

Born in 1955, Martin Warnke studied mathematics and physics at FU Berlin and at Hamburg University. He graduated in theoretical physics and started work at Lüneburg University, where he did his Habilitation in 2008. By this he acquired the venia legendi for computer science and digital media. In 2010 he beacame the managing head of the Institute for Culture and Aestetics of Digital Media (ICAM), where he works as professor.

Furthermore he chaired the section "Computers and Society" in the German "Gesellschaft für Informatik" until 2008, as well as the project “HyperImage” together with Humboldt University. He now chairs the project "Meta-Image" in cooperation with Humboldt-University and Cologne University.