A Linked Open Data model for describing comic book sequences: Exploring semantic enrichment opportunities with graphic medicine

Abstract

Applying a Linked Open Data (LOD) approach to modeling the visual structure and content of comic books and graphic novels enables the description of these works to be enhanced through the process of semantic enrichment. This strategy may be particularly impactful for graphic medicine, a non-exclusive genre of comics that communicate medical and healthcare information, including personal stories of illness. However, the metadata for these works may lack references to healthcare-related vocabulary, thesauri or ontology that would more precisely describe their contents. This material may include pages and panels that illustrate specific medical topics, such as symptoms, side-effects or treatments — subjects that often overlap with other healthcare challenges, including mental health and illness. Exploring an LOD approach for describing comics content may potentially enhance the discoverability of this material and its ability to be remixed and reused, and better connected to other information resources.

The medium of comics is a dynamic art form that has not only evolved with technology,¹ but also matured alongside the reader. Once relegated to dime store spinner racks, the modern comic book evolved as young readers moved on from classic superhero stories to mature tales that often placed their favorite characters within new dilemmas and moral contexts.² As the stories evolved, so did the format, and the floppy, newsprint comic book transitioned from drugstore to bookstore as the hardcover graphic novel.³ Today comics are celebrated within popular culture, easily found on library shelves, and the focus of critical analysis and scholarship across a variety of disciplines.⁴ The typical comic story has also evolved beyond traditional tales of action and adventure to incorporate increasingly complex narratives, like the illustration of historical events (e.g. Maus by Art Spiegelman, Kent state by Derf Backderf) and personal memoirs, including stories of mental health and illness (e.g. Marbles by Ellen Forney, RX by Rachel Lindsay). These stories and events are reframed and made accessible through the graphic medium of comics which distills often complex, challenging topics and histories to a simplified cartoon world of juxtaposed images and text, in which the reader mentally participates in the flow of the narrative, utilizing their own knowledge and experience to bring closure to the space between pages and panels.⁵

A current and relevant area of modern comic book and graphic novel storytelling is the intersection of comics and healthcare, or graphic medicine.⁶ This genre is non-exclusive⁷ and includes any comic that communicates medical and healthcare information, or conveys stories of health, illness and recovery.⁸ These works may not only help educate, but can also increase understanding and empathy between patients, caregivers and healthcare providers,⁹ while enabling better insight into the patient experience.¹⁰ With regards to mental health and illness, the visual medium of comics may provide a unique window into the mind of a person experiencing depression, anxiety, mania, etc.,¹¹ and may be destigmatizing when compared to other media such as television or movies, which often broadcast negative portrayals of mental illness that can influence the perception of caregivers.¹²

While comic book and graphic novel data can be found in many bibliographic catalogs, databases and indexes, the metadata for works that can be categorized as graphic medicine often lacks references to healthcare-related knowledge organization structures (including taxonomy, controlled vocabularies, thesauri and ontologies) that would more precisely describe their contents. These distinct pages and panels may illustrate specific disorders, symptoms, side-effects, or other medical events and challenges — topics that often overlap and are shared with other illnesses, such as panic, anxiety and depression.

The implementation of a Linked Open Data (LOD) approach to describing graphic medicine may help enrich the description and discoverability of this distinct comics content, while also enhancing its potential to be remixed, reused and connected with other information resources. This article introduces a Linked Data (LD) model for describing the visual structure of comics content, and explores the application of that model in an abbreviated set of examples from States of mind, a graphic novel and personal memoir of bipolar disorder.

Comics community cataloging, indexing and other data projects

The comic book was not a publication format that received dutiful cataloging or preservation by knowledge institutions in its early years, and the responsibility of cataloging and indexing comics has traditionally been the activity of comics readership.¹³ In the 1970s and 1980s, comics historian George Olshevsky and others worked to create indexes for major Marvel and DC Comics publications, providing the first detailed indexes of modern superhero stories from the major publishers.¹⁴ In the 1990s this activity would move online with projects like the Grand Comics Database (GCD), an internet-based nonprofit and open database supported by a team of international volunteers.¹⁵ Today the tradition of detailed hobbyist cataloging and indexing continues, and is joined by the efforts of libraries and universities that have made their datasets freely available, like the Comics as Data North America (CaDNA) project from the Michigan State University (MSU) Comics Art Library,¹⁶ which includes the connection of that data to knowledge platforms like Wikidata.¹⁷ Additionally, databases like eDBtheque¹⁸ from the University of LaRochelle explore applying new technologies such as deep learning and natural language processing to the automated extraction, indexing and semantic annotation of digital comics content.¹⁹ And cross-media projects like the Japanese Visual Media Graph (JVMG)²⁰ propose aggregating enthusiast data covering similar materials, including manga and visual novels, and making it more easily accessible to researchers — highlighting the significance of detailed, community created data in describing popular visual media.

Visual vocabulary of comics structure and content

The medium of comics communicates stories and ideas using sequential art, the juxtaposition of images, text and other pictorial elements on a page.²¹ This common visual vocabulary of comics storytelling includes the use of panels to frame scenes within a story, balloons to convey speech, thought, or sound, and captions to incorporate narrative content. The negative space that forms around these panels is referred to as the gutter, which functions to not only visually separate panels, but also to help shape the perception of time and space within a sequence.²² Within this empty space, the reader mentally participates in construction of the narrative by applying their own experience to bring closure between two sequences.²³ The concept of closure — perceiving the whole story through parts — is critical to reading and understanding comics. As Scott McCloud states in Understanding comics, ‘If visual iconography is the vocabulary of comics, closure is its grammar,’²⁴ or rather, more illustratively: comics sequences have ‘Closure for blood, gutters for veins…’²⁵

While the visual iconography of comics vocabulary can be reduced to common components, the precise identification of these elements is more difficult: a panel may have no border; a balloon may exist without a panel; the gutter may ‘bleed’ to the edge; art may ‘splash’ across multiple pages;²⁶ and so on. And while most comics are plotted with text intended to be read in a specific order, not all comics art has a specific sequence, and creators have little control over how the reader will interpret the page or where they might begin reading.²⁷

Comics metadata vocabulary

The visual vocabulary of comics can be transformed into digital data, or encoded, using several metadata schema: ComicsML is an XML-based markup language for online comics;²⁸ the Advanced Comic Book Format (ACBF) is a distribution and interchange format for digital comics;²⁹ the Comic Book Markup Language (CBML) is an XML vocabulary for encoding comics based on the Text Encoding Initiative Guidelines (TEI);³⁰ A Comics Ontology is an OWL ontology focused on narrative elements;³¹ and the Manga Metadata Framework supports the description of digital manga.³² Additionally, comics related terms were added to Schema.org in 2015 as part of the bib.schema.org extension. However, with the exception of Schema.org, these metadata vocabularies are either not intended to or do not appear to currently meet the technical requirements for publishing Linked Data (LD), such as providing dereferencable URIs under a persistent namespace, or various representations of a resource using content negotiation.

Comic Book Ontology and Linked Open Data

The Comic Book Ontology (CBO) is an OWL ontology and RDF metadata vocabulary for describing comic books and comic book collections.³³ It is open source, open licensed (CC BY 4.0) and publicly maintained on GitHub where it is open to feedback and contributions. CBO is not a replacement for existing metadata schema like ComicsML, CBML or Schema.org, rather as an LD vocabulary it can be used alongside other schema to create new records, or complement and enhance existing data through the process of semantic enrichment — a strategy that libraries, archives and museums have used to successfully enhance the discoverability and reuse of their data.³⁴

Linked Data (LD) refers to machine-readable structured data that is linked together through the use of standard protocols and common vocabularies.³⁵ LD utilizes an abstract data model called the Resource Description Framework (RDF) expressed as sentence-like ‘triples’ composed of a subject, predicate and object (e.g. Mary — subject — knows — predicate — Bob — subject). These RDF triples, or statements, compose the edges and nodes of a graph data structure, and when identifiers (e.g. URIs or web addresses) are used instead of text, machines (i.e. semantic web applications) can build connections between resources, discovering additional information through an interlinked, networked knowledge graph.

Linked Open Data (LOD) builds on the principles of LD by encouraging the utilization and publication of open licensed datasets and vocabularies,³⁶ or ontologies — knowledge organization structures that formally describe the classes and properties within a domain and the relationships between those concepts (including relationships to other ontology). When structured data is made freely available on the open web using non-proprietary, freely-licensed protocols and vocabularies, the effective knowledge graph for a given subject is free to expand and be queried semantically across open datasets, and across knowledge domains.

LD and RDF can be encoded using a variety of data formats (e.g. XML, JSON-LD, etc.), or embedded in HTML using microformats or RDFa. This article takes a ‘no-code’ approach, visualizing the examples presented as a graph diagram. However, all related code and technical examples are available in a companion GitHub repository: https://github.com/comicmeta/LOD-MentalHealth.

CBO Sequence data model

The CBO sequence model (cbo:Sequence) maps the vocabulary of comics structure and content to a set of classes: stories (cbo:Story), pages (cbo:Page), panels (cbo:Panel), captions (cbo:Caption), and balloons (cbo:Balloon) — distilling the visual language of comics to a simplified data model that can be mapped to RDF and aligned with other vocabulary, like Schema.org (schema:CreativeWork, schema:ComicStory). This updated model (as of v0.16) acknowledges both the literary and visual components of a complete comics sequence as it unfolds across pages, panels, and serialized stories. Additionally, the updated sequence model (see Fig. 1) attempts to conceptualize the grammar of comics by introducing a class for the gutter (cbo:Gutter). Similar to the visual element it represents, this class can also be used to group visual elements, providing a placeholder for describing otherwise invisible data related to closure (e.g. varying reader perspectives of a sequence).

Fig. 1. The Comic Book Ontology (CBO) Sequence model mapped to an excerpt from the story “Cut of the cards,” appearing in Dear lonely hearts (Comic Media, 1953), issue 2. Public domain. https://digitalcomicmuseum.com/preview/index.php?did=18196.

There are no strict constraints defined between these classes (i.e. there are no cardinality requirements or strict hierarchical rules) and the model requires minimal semantic and ontological commitment, acknowledging that there is often great visual variety in the expression and medium of comics (a term which may refer to publications, stories, artwork, etc.), and that comic book characters and stories often appear within or are adapted to other media, such as manga, movies and video-games (i.e. a comic can belong to multiple classes: a book, page of artwork, scene from a video-game, etc.).

A graphic medicine example

Comics have a long history in the application of sequential art for functions like storyboarding or illustrating instructional content.³⁷ Graphic medicine is not only an extension of this application, but also an expansion in how comics are analyzed and interpreted. In addition to the casual comics reader, there are at least three potentially differing reader perspectives when considering works of graphic medicine: patient, caregiver and healthcare provider.³⁸ These readers may interpret sequences quite differently — the knowledge they bring to the process of closure providing unique dimensions of experience and insight between panels.

The graphic novel States of mind by Emilie and Patrice Guillon is a personal memoir of bipolar disorder — a mood disorder that includes emotionally-high periods called mania, and low periods of depression.³⁹ This work is a unique example of graphic medicine, as both creators embody different reader perspectives: Emilie as patient, and father and co-creator Patrice as caregiver. The writers periodically break the narrative and provide a window into the often challenging and deeply personal creative process, as they pause to discuss and recall the events illustrated in the story, with occasionally differing recollections.

Example LOD graph using the CBO sequence data model

In the following example, Library of Congress Subject Headings (LCSH)⁴⁰ and BioPortal (a repository of biomedical ontologies)⁴¹ were used for term discovery, with preference given to the Medical Dictionary for Regulatory Activities Terminology (MedDRA) and the National Cancer Institute Thesaurus (NCIT), both of which are open licensed, and/or freely available for non-commercial use. This article uses a shorthand when referring to these and other LOD resources that replaces the full URI (e.g. https://www.comicmeta.org/cbo) with a prefix (e.g. cbo).

While there are numerous sequences within States of mind relevant to mental health, there are several key examples LOD may enhance: a) illustration of medical topics like a specific condition or disease, such as bipolar disorder (lcsh:sh85080541); b) symptoms like stress (meddra:19942209) or panic attacks (meddra:10033664); c) medications and side-effects, like the antipsychotic Tercian (ncit:C79120) and loss of appetite (ncit:C113630); and d) medical events or experiences, like an involuntary commitment (meddra:10080523). The latter is a poignant example of varying reader perspectives — a patient may recall the loss of freedom or a sense of fear (meddra:10016275), while a caregiver might mentally fill the space between these panels with a memory or feeling of guilt (ncit:C95555).

Above is a graph diagram (see Fig. 2) of RDF statements from the preceding summary. This representative knowledge graph incorporates identifiers from the GCD (licensed under CC BY-SA 4.0), which includes an index for the work and entries for each sequence or chapter.⁴² The CBO sequence model is applied to extend and enrich the index by adding nodes for pages and panels, which are then connected (using the schema:about property) to healthcare-related LOD resources. If these nodes were expanded, they would then incorporate additional statements into the graph, such as bipolar disorder and variants like manic depression from the LCSH entry, as well as Tercian and the synonym Cyamemazine from NCIT. Additionally, the diagram demonstrates use of the gutter concept in an effort to capture reader perspective, although future vocabulary development may uncover a more expressive or effective approach.

Fig. 2. A graph diagram visualizing select content elements from the graphic novel States of mind that illustrate topics relevant to mental health and illness. Nodes in blue and red represent data from external sources, including healthcare-related vocabulary and ontology.

Conclusion

This article presents a Linked Data (LD) model for describing comics content, and a method for connecting that content to healthcare-related Linked Open Data (LOD) resources. This approach may be helpful for enhancing the discoverability of graphic medicine by building upon the tradition and practice of comics indexing to link specific content with medical vocabulary and ontology. Additionally, the LOD approach presented is multilingual, respectful of copyright and attribution, and applicable for non-traditionally published work, including crowd-funded and self-published materials. However, data validation is required, and future study may explore incorporating objective review by domain experts.

While graphic medicine does not replace mental healthcare, enhancing and extending the description of this work may potentially help support the discovery and reuse of accessible and destigmatizing comics materials in advancing the broader conversation around mental health and wellness.