Dragging, rotating, fading, rearranging – motion picture books and interactive picture books invite users to engage in a wide range of interactions. The BMBF-funded project “BeWeB-3D – Digital Motion Picture Books” at the Berlin State Library tested how these complex objects can be digitised. museum4punkt0 picks up on this and asks: How can we make the leap from historical activity books to today’s browser games? A conversation with Christian Mathieu, a project member of BeWeB-3D, as well as Silke Krohn and Katrin Glinka from museum4punkt0.
Christian Mathieu, what are motion books?
Christian Mathieu: These are books that invite readers to interact mechanically beyond conventional book use – that is, opening and turning pages. They have a long history dating back to 13th-century Europe. We find the category of picture-book games particularly fascinating. From the 17th century onwards, a type of motion book emerged that offered an enormous range of interactive possibilities. In Germany in particular, the production of these books flourished in the late 19th and early 20th centuries. Leading children’s book authors and illustrators, such as Lothar Meggendorfer from Munich, produced highly complex, internationally influential works during this period.
How is it that the State Library possesses such an extensive collection of these?
C.M.: Since the 1950s, the State Library has devoted considerable attention to children’s and young adult books – a literary genre that has often been neglected in academic collections. Consequently, it is now one of the few academic institutions to possess a very large, systematically organised collection. The collection comprises historical and modern children’s and young adult books from all over the world, magazines, original illustrations for children’s books, picture sheets and posters – and, indeed, the aforementioned activity books and movement books.
How did your BeWeB-3D project come about in this context?
C.M.: The idea arose during a lecture by Carola Pohlmann, Head of the Children’s and Young Adult Books Department, as part of the series ‘Materiality of Writing’. The aim of the series is to productively confront academic perspectives on our collections with practical questions. During the discussion, it emerged that digitising such books would be challenging.
How do these challenges manifest themselves?
C.M.: There are hardly any standards for digitising interactive, three-dimensional objects. For so-called “flat material” – that is, conventional books, magazines or archival materials – libraries and archives have already developed procedures over the last 20 years. 3D digitisation, on the other hand, is already difficult for immobile objects, but far more so for movable ones: How do you make the interactive materiality of motion books digitally tangible? We don’t just want to convey an impression of what the book looks like. People should also be able to interact with it and use it as a research object. This includes being able to record and cite any sequence of movements. Our project therefore began as a concept study.
So that you can then extend your findings to other fields?
C.M.: Our observations are also relevant to other types of objects – such as artist’s books with moving elements, as well as altar retables, picture frames with foldable elements, and screens with motifs that must be unfolded in a specific way. To explore this scope, we focused on the case study of pop-up books, as this type of motion book already covers a very broad spectrum of interaction mechanisms.
How did you digitally capture the unique materiality of the books?
C.M.: It was important for us to evaluate various 3D scanning methods – ranging from highly sophisticated techniques such as computed tomography to strip-light and laser scanning. At the same time, we also utilised structure-from-motion methods.
Computed tomography?
C.M.: Many motion books conceal the lever arms and joints of their mechanisms between the pages. To see exactly how they work, one would have to take the pages apart. For conservation reasons, however, this is virtually impossible. CT scans allow us to view the hidden mechanisms non-destructively. They are, however, labour-intensive and costly.
Strip light and laser scanning are also very challenging measurement methods, which sometimes reach their technical limits with our books. For example, laser scans are only suitable to a limited extent. You have to get very close to the book objects because they contain very small components. Laser scans often simply cannot provide valid information from such a short distance.
Is that why you experimented with structure-from-motion?
C.M.: Yes. The idea behind Structure-from-Motion is that a spatial image is created from a large number of calibrated photographs. Each photo is taken from a predefined angle. There is free software available online for this – e.g. VisualSfM, CaptureReality and Agisoft PhotoScan76. Ultimately, you can even photograph the motion books in 360 degrees using a mobile phone camera.
A second consideration was to test a method that is also accessible to other institutions. In our study, we collaborated with the Centre for Digital Cultural Heritage in Museums (ZEDIKUM) at the Berlin State Museums. ZEDIKUM possesses core expertise and the necessary infrastructure in the fields of 3D digitisation and visualisation. Just like the State Library, it is part of the Prussian Cultural Heritage Foundation – so the service centre and the research collections are, so to speak, under ‘one roof’. There are likely to be very few institutions worldwide that can operate from such an ideal setup. That is why we investigated a method that delivers valid results, whilst also keeping the proverbial ball as low as possible.
What exactly did your approach involve?
C.M.: Structure-from-motion scans were used to capture the static dimensions of the book. The next step was to render the dynamic pages of the books. We used computer game technology for this. Physics engines allow the physical properties of a work to be simulated. And game engines enable users to interact with the digital copy.
How close does this get to the original?
C.M.: This approach has great potential. However, we are not claiming that it creates a 100 per cent authentic replica. Transposing the dynamic properties of a work requires an enormous amount of effort. Ultimately, one would have to measure paper thicknesses and replicate the material properties of a page – how it turns, how it creases – in order to meticulously transfer every detail into the digital realm. This means you can approximate authenticity, but you won’t fully achieve the materiality of interaction. In the case of a motion book, every conceivable form of interaction would actually have to be represented digitally, including unexpected ones: you would actually have to allow the possibility of crumpling the book digitally, tearing it up and throwing it into the corner.
museum4punkt0 builds on your work and focuses primarily on use in a non-academic context. Why this topic?
Katrin Glinka: museum4punkt0 explores the potential of digital formats for cultural mediation. This includes the question of how digitised cultural heritage can be used in computer games and, conversely, how museum content can be conveyed through games. We are also looking at how cultural heritage, once digitised, can be used in various contexts. In doing so, we wanted to draw on existing material from the broad spectrum of collections held by the Prussian Cultural Heritage Foundation.
Silke Krohn: The State Library’s concept study was particularly interesting for several reasons, as there are points of intersection between our field of work and the motion book and games. On the one hand, this is of course the fact that game engines were used to digitise the books. On the other hand, BeweB-3D raises the question of whether motion books and games should be seen as more closely related to one another than has been the case so far.
How much of a book is actually in a computer game?
C.M.: Books have appeared as content elements in computer games with great regularity, from early on right up to the present day. In BeweB-3D, however, we are interested in a different aspect: to what extent did the paper mechanics of motion books influence various forms of interaction in early computer games or even arcade games?
So the points of intersection in game mechanics are of interest?
C.M.: Yes. The basis for our considerations is the approach of media scholar Espen J. Aarseth. He speaks of ‘cybertext’, the content of which is only constituted following mechanical interaction (beyond eye movements and conventional page-turning). A computer game can also be viewed as a cybertext: it is only through your interaction and gameplay decisions that it becomes clear when and how the plot unfolds. Pop-up books also exhibit characteristics of cybertexts. Many books are not simply about the passive reception of something dictated by the author. Instead, the focus shifts to the decisions made by the players: this is particularly evident in pop-up stages. Admittedly, basic modes of interaction are dictated by the materiality: unfolding the stage, placing figures by inserting them. However, players themselves decide where and how to arrange the ‘theatre crew’ on stage and which figures to choose, thereby altering the plot. Our thesis is that the computer game – at least in its early form – was influenced by the motion book.
And is museum4punkt0 now exemplifying this transition from book to game?
K.G.: In a way, yes. We are exploring how old principles of interaction from books and digital gameplay elements can be combined in a new game that suits today’s habits of engagement. The process itself is also of great interest – the way in which the historical material is, so to speak, ‘translated’ for today.
So the focus is on the question: How can we use the historical material to connect with today’s gaming habits?
S.K.: For children today, who have a very different media experience, the books from back then might not be enough if you simply transfer them 1:1 into a computer game. Gaming habits and expectations of a digital game are different from those of a physical book. An important aspect of our work is therefore: How do you build this bridge without disregarding the source material too much?
How are you incorporating the book’s mechanics into the game?
S.K.: The specific format hasn’t been fully decided yet. First, we’re identifying the expectations of our target audience – children aged eight to 12. To this end, we’ve, for example, held a workshop over the past few weeks with pupils in years four to six at a school in Berlin. The aim was to identify their gaming behaviour and preferences regarding game types, characters and design aesthetics. We also brought along facsimiles of the historical game books.
During the test, we noticed that the children found the historical book objects quite exciting depending on their mechanics – for example, Meggendorfer’s Stadtpark, which can be set up and rearranged entirely in three dimensions. Even a child who initially said they couldn’t get into books didn’t want to give one of the books back to me at the end. In these initial observations, which we intend to supplement in further test settings, a trend has already emerged: the children have started to think for themselves about which mechanisms they would combine to keep the books exciting. These are good starting points for us to translate the source material into a digital game.
K.G.: Exactly. So we’re not just interested in turning a single motion and interactive picture book into a digital game. At the same time, there’s no point in designing a game that incorporates every single mechanism and feature of existing motion books.
Apart from the unique materiality: what else do you want to convey through the books?
S.K.: We’re currently exploring various ideas. The initial feedback from our workshop was interesting here too: for instance, the children were fascinated by the context in which the books were created – such as the circumstances under which children played in past centuries and the toys they used. They were less engaged with the content of the books. There are many facets that could be explored. We are also aware that we cannot implement everything in a prototype browser game. What we ultimately decide to focus on will become clear as part of our iterative process.
Why did you choose the format of a browser game?
K.G.: At museum4punkt0, we bring together a relatively large amount of educational content within the physical museum building. So, on-site VR installations, location-specific AR apps that offer additional layers of information and educational content directly in front of the originals, and so on. It was important for us to also test a format that people are familiar with from their leisure activities at home. At the same time, a game based on current web technologies could certainly also be used as a gaming station within the museum building. The flexibility in its use was a key factor.
C.M.: This approach was, in turn, exciting for us and for the further development of BeweB-3D. In the first phase, our project was designed as a concept study to evaluate different methods of digitisation. In the next step, we aim not only to produce fully developed digitised materials for academic use, but also to explore the aspect of the relationship between games and picture books in greater depth.
In cooperation with the Computer Game Museum Berlin, we are therefore planning an exhibition that will explore the connection between the interactive picture book and the computer game. The game prototype from museum4punkt0 will also be featured. With a different focus at each venue, we intend to stage the exhibition at two locations: at the State Library, we will trace the history of the interactive picture book right up to the dawn of the computer game. The Computer Game Museum will highlight aspects in which the motion picture was constitutive for digital games. In my view, this museum connection is important. As a further form of 20th-century narration, computer games are also a medium with which museums, libraries and archives must engage.
The text first appeared on the blog of the museum4punkt0 project.
