Glam Manual

A text layer holds strings of text that form the foundation of documents. Most projects will only have one text layer, though it is possible to create more than one. (One such rare occasion might be when the same text has been given in two different languages, and both versions of the text need to be annotated.)

There are no constraints on what texts can contain, except one: many of Glam’s core interfaces assume that one line in the text will roughly correspond to something like a sentence. Texts should be edited accordingly, and very long lines should be avoided if possible.

A token layer contains tokens, which are defined as non-overlapping, non-empty, continuous substrings of a text layer which the token layer depends on. Tokens provide basic units for further linguistic analysis, and will most commonly capture something like a morpheme or a word. It therefore may be desirable to have multiple token layers in some situations.

A span layer contains spans and depends on a token layer. Spans are anchored to tokens from the token layer it depends on: each span must be linked to at least one token from that token layer. (Associated tokens do not need to be contiguous.) Spans may also have a value, which is either a plain text value or an entry in a vocabulary (see below).

Many kinds of span layers will have spans which always span exactly one token—​these might express common information such as grammatical category, or part of speech. Other span layers will have much wider spans, such as a translation span layer, whose spans would cover entire sentences.

A relation layer depends on a single span layer and holds edges between spans. These edges are directed (one span is the source, the other is the target), and must always connect exactly two spans from the span layer. The relation also has a value like spans do: either a plain text value, or an item from a vocabulary. Many projects will not need relation layers, but they are critical for expressing some kinds of data such as coreference links and syntax trees.

A vocabulary is simply a list of items which have at minimum a form. A vocabulary may also specify additional fields that each item should have, which will be able to hold arbitrary text as information. Vocabularies may be closed or open: if a vocabulary is closed, then only administrators will be able to add more items, and if it is open, then regular users will be able to extend the vocabulary with new items during regular use.

Unlike the other layer types, vocabularies are global and are not associated with a project. Vocabularies come in two different varieties: if a vocabulary is to be used with a token layer, then it is an alloformic vocabulary, and if it is to be used with a span or relation layer, it is a basic vocabulary.

In this kind of vocabulary, spans and relations whose values refer to a vocabulary item will have exactly that vocabulary item’s form as its value. This is suitable for annotations like grammatical categories, POS tags, lemmas, and morphological features.

Alloformic vocabularies are used with token layers, and help address the fact that formal units of linguistic analysis such as words of morphemes often do not surface in texts in their citation form: for example, the Turkish plural marker may surface as either -lar or -ler depending on vowel harmony, but we would like them both to correspond to a single vocabulary item for a morphemic analysis. An alloformic vocabulary therefore still requires that each item have a canonical form which is to serve as a citation form, but it relaxes the requirement that an instantiation of this item have exactly the canonical form.

The text editing interface (at /document/:id?tab=text) is used for editing the raw text that will be annotated in the document.

A text box allows for entry, and the default "save" action will save the text as it is entered. No tokens will be created—​just the text will be saved.

You will often want tokens that represent a morpheme rather than a whole word. To conveniently enter data like this, the text editing interface offers a "save with morpheme tokenization" action. Text that is entered and saved with this action will create tokens when the text is saved with explicit guidance from you in the form of - separators between morphemes. For example, consider the following raw text:

When saved with morpheme tokenization mode, two things will happen: first, the - separators will be removed from the text’s final form; second, tokens will be generated that will have boundaries at - separators as well as other heuristic boundaries, such as whitespace characters. The text above would therefore look like so, where the numbers under each character indicates the token it belongs to:

If you want to include a literal - in your text, simply repeat it:

The token editing interface (at /document/:id?tab=token) allows you to create and edit tokens. A token is visually represented in both this interface and the text interface with a black rounded box around its text.

Currently, only whitespace tokenization is supported for bulk token editing. It is recommended that you use text editing with morpheme tokenization if this is not suitable for you. In the future, it will be possible to use external tokenizers, such as SpaCy’s.

Simply click on a token’s box to delete it.

Highlight a totally untokenized span of text, either by clicking and dragging your mouse over it or double clicking on a whole word. Creation will only succeed if all characters do not belong to any existing token.

To shift the boundaries of an existing token, hover over the token with your mouse and use the following keyboard commands:

The interlinear interface allows properly configured span layers to be edited. For each token, each token-level span layer will have a single field, and for each sentence, each sentence-level span layer will also have a single field. Changes in the spans' values will be propagated to other users as soon as your cursor leaves a field.

Just as you might expect the data for a document to be spread out across different tables for a SQL approach, in XTDB, we make a separate document for every different kind of data that makes up a project or a document. Every token, for example, will have its own document. Each document has attributes, which you can again think of as analogous to a SQL column.

Let’s consider an example. A token document has the following attributes:

Putting that all together (ignoring layers):

The token represented by the second document captures the substring "Hello" in the text.

The data structures we just described are highly relational, and as such we need to take care to never allow an invalid state to occur. Every attribute which refers to another document with an ID must always be valid, or bad things will start happening. Consider, for example, what must happen when we delete a text object. If a text no longer exists, then we must be sure to delete all of the token objects that depend on it, and so on for dependents of token objects. And importantly, we must make sure that this deletion is an all-or-nothing affair: it would be bad if the deletion process succeeded only on some documents, or if database reads occurred while the deletion was in progress, yielding a view that would have invalid joins.

XTDB has an elegant solution for this in the form of the transaction function. First of all, recall that all writes happen with XTDB’s submit-tx function, which takes a vector of transaction operations which are executed in order. This alone isn’t enough, since XTDB’s primitive :xtdb.api/delete transaction operation isn’t sufficiently expressive for the detailed behavior our data model requires. For example, when a token is deleted, a span that contains that token should be deleted if and only if the token was the only one contained in the span. This is where the transaction function comes in: it allows us to define our own transaction operations that have the freedom of arbitrary code and the guarantees associated with being executed within a transaction by XTDB.

To make using transaction functions as convenient as possible, we define new transaction functions using glam.xtdb.easy/deftx. Consider an example: glam.xtdb.token/delete:

This looks a lot like a normal function definition, but there are a few things to note about this. First of all, the first argument to the function must always be node, which will be bound to the XTDB node executing the function. Second, deftx actually defines two things. There is an internal version of the function that is trailed with **, and invoking this prepares an XTDB transaction which contains only the invocation of this transaction function. The normal version of this function without the ** at the end does the same but also invokes the transaction immediately. Having these two variants allows us to use the asterisk-less version whenever we’re interested in that operation as our top-level goal, and the asterisk-ful version whenever we want to use that operation as a part of another operation. In the example above, we can see that this transaction function relies on another transaction function, glam.xtdb.span/remove-token**.

Let’s go through an example to make all of that concrete, continuing with glam.xtdb.token/delete. First, let’s observe that this function is stored in the database:

This code should look very familiar—​it’s what we saw above! (Don’t worry about the minor differences on variable names—​that’s an implementation detail that you can safely ignore.) Now let’s see what the result of invoking delete** is:

As you can see, not much happens: all this does is to prepare the transaction function’s invocation by formatting it with its arguments. Nothing has happened to the database. Invoking (glam.xtdb.token/delete node 5), on the other hand, would have submitted this to the database, equivalent to (xtdb.api/submit-tx node (glam.xtdb.token/delete** node 5)).

In most places in Glam, we will refer to an entity simply with its ID. This is true, for example, in all XTDB documents: :token/text has as its associated value the ID of the associated text. However, in some places, you will see idents, which are two-item vectors where the first value is a keyword like :text/id and the second value is the ID. This is called an ident, and it is another way of referring to an entity which encodes its type. We use idents because Pathom (the library we use for graph query processing) works in terms of them. For this reason, you will see that many namespaces have an xt→pathom function which will convert IDs to idents.

All namespaces under glam.xtdb are devoted to a particular data type, with the following exceptions: