TMQL Tutorial (Preview)

TMQL Tutorial (Preview)

Robert Barta rho@bond.edu.au

This preview of TMQL may give first insights how a developer will experience and use the language.

Introduction

The need for a dedicated query language for Topic Maps is not as uncontested as one might think. Serious work on TMQL has started roughly a year ago, kicked off by the ISO/IEC SC34/WG3 committee decision to create an industrial-grade language with a low entrance barrier.

While the editors are still trying to fix odds and ends in the current TMQL specification, the main concepts are sufficiently stable now to justify asking the public for feedback. In this, please note that for the purpose of presentation here we have sometimes chosen a particular syntax some of which will appeal to a particular audience while others may find it at least eyebrow raising. As the specification is in flux (I so much like euphemistic terms), what follows may well be out of sync with the current TMQL draft. Still, it tries to draw a realistic picture how the language may appear to a developer.

The following TMQL preview builds on an existing presentation, but it breaks somewhat with the TM tradition to use 'operas' as running use case. Instead, we will use the more general theme 'music', and will assume that we have a TM data store which contains various albums, (female or male) musicians and music groups (which are all artists and have persons as members). All these topics are connected via various associations, such as is-produced-by or is-part-of.

Setting Off

If you have used SQL before, then you will not be completely puzzled by the following query:

SELECT $album
WHERE
   is-produced-by ($album: production, tom-waits: producer)

This query (technically a query expression) will return all albums where Tom Waits is known to be a producer. tom-waits is an identifier of a topic which we happen to know to uniquely pinpoint a topic about that person in the map we query. The query processor will try to find an association of type is-produced-by and will check whether the topic tom-waits is playing the role producer there. If so, it will bind the variable $album to the topic playing the role production in that same association. It will so work through the whole map and will collect all these variable bindings. Finally, the query processor will return a list of these variable bindings. We leave it open for the moment, what exactly is returned to the application, so whether that gets topic identifiers, names of the topics as strings or actually data structures representing the topic.

If we wanted to make the query more watertight to return albums only (and not something else produced), then we will have to add another constraint to the WHERE clause:

SELECT $album
WHERE
   is-produced-by ($album: production, tom-waits: producer),
   $album : album

It is worth noting, that is-a honors the (transitive) subclass-superclass relationship. If a particular production were an instance of collector-box and that (directly or indirectly) is a subclass of album such productions would also be returned.

If we are not fixated on Tom Waits and would instead want a list of all albums together with their producers, we can extend the wishlist in the SELECT clause:

SELECT $album, $producer
WHERE
   is-produced-by ($album: production, $producer: producer),
   $album : album

Controlling What is Returned

All queries so far return topics into the application in the form of data structures. If the application were interested in a name of such a topic, it would have to navigate using an API which is outside the scope of TMQL. If it is clear that the application needs only the name (and not the whole topic), we can tweak the SELECT expression again by adding a path expression:

SELECT $album / bn
WHERE
   $album : album

To fix the first problem, we could choose to only accept names in a particular scope. This is achieved by appending a filter to the path expression:

SELECT $album / bn [ @ en ]
WHERE
   $album : album

SELECT $album / bn [ @ en ] `
WHERE
   $album : album

Path expressions are also a convenient way to impose a sorting order on the list of tuples we return:

SELECT $album, $producer
WHERE
   is-produced-by ($album: production, $producer: producer)
ORDER BY
   $album / bn [ @ en ] `

The ordering can also include more than one ordering criterion, like in

SELECT $album, $producer
WHERE
   is-produced-by ($album: production, $producer: producer)
ORDER BY
   $producer / bn [ @ en ] ` desc, $album `

As you would expect, it is also possible to make the list of tuples unique, i.e. to take care that no two tuples are the same:

SELECT $album
ORDER BY
   $album / bn `
UNIQUE

SELECT $album
ORDER BY
   $album / bn `
OFFSET 10 LIMIT 20

Identifying Things

You may correctly argue that identifying topics with their (internal) map identifier (TMDM calls this for some reason 'source locators') is not an immensely robust idea if that identifier may change any second.

In the likely case that there is a subject indicator (a resource which helps to indirectly identify a subject) for a subject, no one can stop you to use that instead. You only have to tell the processor that you are doing so:

SELECT $album
WHERE
   is-produced-by ($album: production, s'http://www.u2.com/ : producer)

If you are a lucky owner of a subject locator (so a URI which refers exactly to the subject in question), then you can use that instead. The album 'How to Dismantle an Atomic Bomb' from U2, for instance, has the ASIN code B0006399FS:

SELECT $producer
WHERE
   is-produced-by (i'urn:x-ASIN:B0006399FS : production, $producer: producer)

Controlling Variable Bindings

As we have seen above, variables can be bound to values. Used naively, this can lead to incorrect queries. As an example let us find all two albums which share the same producer. In a first attempt we write:

SELECT $album1, $album2
WHERE
   is-produced-by ($album1: production, $producer: producer),
   is-produced-by ($album2: production, $producer: producer)

This does not work for us if we want different albums. The usual escape hatch is to have something like this:

SELECT $album1, $album2
WHERE
   is-produced-by ($album1: production, $producer: producer),
   is-produced-by ($album2: production, $producer: producer),
   $album1 != $album2

TMQL has a peculiar way to fine-control when variables are allowed to match anything or when they must be bound to something different:

SELECT $album, $album'
WHERE
   is-produced-by ($album : production, $producer: producer),
   is-produced-by ($album': production, $producer: producer)

Association Templates

In the queries above we made use of association templates. Writing inside a query

   is-produced-by ($album : production, $producer: producer)

To allow for such associations with additional roles to match, TMQL allows to append an ellipsis:

SELECT $album
WHERE
   is-produced-by ($album : production, $whoever: producer, ...)

Association templates also have more implicit meaning than is obvious at first. If, for example, the map contained an association of type is-remastered-by which also connects an album with a producer and is-remastered-by is a subtype of is-produced-by, then also such associations would match the template.

Honoring subclassing also applies to roles and their types. Had we in our queried map an association of type is-remastered-by, but the role (type) for the album is not production, but a subclass remastering, such association would also match the association template.

If you would not care about the role type, you can also omit it for some players:

SELECT $album
WHERE
   is-produced-by ($album : production, $whoever, ...)

Query Context

You may have wondered how the query processor knows which map to query. This can be defined in the query context which allows applications to pass all sorts of information into a query. If a map is passed in, it will be - by default - bound to a variable %_. So in all cases we actually could have written more explicit

SELECT $album
FROM %_
WHERE
   ...

Using parameters in the query context makes queries more flexible and portable, but sometimes one wants to name the map explicitely. The expression

SELECT $album
FROM http://example.org/map.xtm
WHERE
   ...

Another feature which may make it into the final standard is to define a list of scoping topics as part of the query context. As an example, the application may choose to pass in [ en, fr, de ] as such a list. Then the query

SELECT $album / bn `
WHERE
   $album : album

Path Expressions

The textual overhead of the SQLish style which we have used so far may not be convenient if queries are trivial. Especially for web applications where pages have to be filled with lots of content from a TM backend a much shorter notation is more adequate.

To return all albums from the map bound to %_, we can simply write

%_ // album

%_ // album / bn [ @ en ]

%_ // album [ . / bn [ @ en ] ]

Such conditions can be arbitrarily complex, so that it is no problem to reformulate our earlier query which returns only the english names of Tom Waits albums:

%_ // album [ . -> production [ * is-produced-by ] / producer = tom-waits ] / bn [ @ en ]

              . -> production [ * is-produced-by ] / producer = tom-waits

Trivializing the process somewhat, this list is now compared to the single topic which is identified by tom-waits. If that topic is part of the list we got just before, then the condition is satisfied and the album is kept as candidate. If not, the next album is tried. The last stage of the path expression finds again the english names for each of the remaining albums.

Chocolate, Vanilla, Caramel

The different language flavours, SQLish and path expressions, can - as we have already seen - be mixed. Not so obvious is the fact that both styles are (almost?) equivalent in terms of expressitivity; every SELECT query expression can be transformed into an equivalent path expression. It is up to the developer to choose the most appropriate combination on a case by case basis.

Both styles allow to return sequences of tuples of things into the application. This may be exactly along your way of thinking in most cases, but there are several application scenarios where you need more complex structures to be returned into the application.

One of them is if you need to integrate a TM backend into one of these shiny XML application servers (such as Tamino, Cocoon or AxKit), or if you - more generally - write taglibs to be used in SAX processors or as XSLT extensions. Here you need a query to return XML content.

While one might argue, that SELECT query expressions together with some templating system to generate the XML content can achieve exactly this, such an approach has severe downsides. Firstly, most templating systems are text oriented, so the XML stream would have to be deserialized (parsed) before being post-processed by an application. This is not fast.

But more importantly, in the case of nested XML documents - and nesting may be the whole point of XML anyway - an the template processor would have to iterate and reissue many queries using the TMQL processor. This results in intense interaction and this is not fast either.

With TMQL you can create XML content directly without fuss using a third flavour, which - you may have guessed it - is otherwise equivalent to the other styles. This flavour, FLWR, is inspired by XQuery and uses RETURN clauses to specify the output:

return
   <albums>{
     for $a in %_ // album return
         <album>{$a / bn [@ en ]}</album>
   }
   </albums>

Such body is defined by a nested RETURN clause. It contains an element <album> which is supposed to contain text content. That can be constant, or - like in our example - can be specified using a TMQL path expression. To signal that, you will have to wrap this inside {} brackets.

TMQL query expressions following the FLWR structure can also return lists as the following example demonstrates:

for $album in %_ // album
return
    ($album / bn [ @ en ]`, $album / oc [ * homepage ]`)

Often FLWR expressions are easier to read than SQL-style expressions, because you can read them from the top to the end (and not the other way round as in SQL). To prove the point that the styles are mostly interchangeable, we reformulate the query to find all album pairs sharing one and the same producer.

for $album in %_ // album
    for $album' in %_ // album
    where
       exists
          is-produced-by ($album : production, $producer: producer)
          and
          is-produced-by ($album': production, $producer: producer)
    return
          ($album, $album')

As a side note: You may have noticed, that the syntax of the WHERE clause inside a SELECT expression is somewhat different from that inside FLWR expressions. There is no technical reason for that, it is simply that we (the editors) did not yet agree whether to use a more orthodox style (using AND, OR, ...) or the style which swapped over from tolog (using comma and |, ...). This will be resolved hopefully soon; two syntaxes for the same thing is not good. No.

The structure of FLWR expressions also makes it possible to return something else: topic maps. Just like SQL takes tables and returns a table as result, and XQuery takes XML documents and returns XML content, also TMQL can take one or more topic maps and can return TM content. At this time of writing, we have no clear understanding yet which notation to use for this.

While existing ones (XTM, LTM, AsTMa=) might be obvious candidates a newly developed one might integrate more smoothly. But this is another story...

Nested Queries

Nesting of query expressions is quite natural and could potentially be used in quite a few places. To demonstrate how much the language can be stretched, here an experimental (so not yet agreed on) syntax to use nesting within a SELECT clause.

To lists all groups and their members in the map, the SELECT clause may contain a whole subquery:

SELECT $group /bn `,
       " members are: " + join (",",
                                {
                                 SELECT $person / bn `
                                 WHERE
                                     is-part-of  ($person: member, $group: whole)
                                 })
WHERE $group : group

You can also use subqueries inside strings. The following example would return a single string where all names of all albums are concatenated:

return
    "{
      SELECT $album / bn
      WHERE $album : album
      }"

Using Exist and All Quantification

On some occasions you will have to test whether particular things exists in a map or whether certain things have a relevant property. For illustration, let us ask for all music groups in our map which have at least one female group member

for $group in %_ // group
where
    some $person in $group -> whole / member
         satisfy $person : female
return
    ($group)

Conversely, we might be interested to find all boy groups, well, at least those groups where all members are male:

for $group in %_ // group
where
    every $person in $group -> whole / member
         satisfies $person : male
return
    ($group)

Functions

Not too surprisingly, the language also sports functions. They can be declared anywhere before a query expression and can then be invoked inside that:

declare function get_female_members ($g) return {
   for $m in $g / member
   where
       $m : female
   return ($m)
}
SELECT $group / bn, count(get_female_members ($g))
WHERE
   $group : group

Adhoc Ontologies

One of the more controversial issues is the adoption of features which would actually be expected to be in a constraint language. Hereby, we can give the query processor more information about the application domain, either by enriching the vocabulary with further topics, add new associations, or introduce associations, based on those already existing in the map.

As example let us consider that our map contains associations of type has-created where we have recorded which artist (single or group) has created which album, such as "U2 as creator has created an opus 'Rattle and Hum'", or "'Leonard Cohen as creator has created an opus 'Various Positions'".

It lies in the nature of associations has-created that if a group has created something, then we might also deduce that every member of the group also has created that thing. Naturally, we do not want to change our map and add all this redundant information to it. Instead we would like add a rule to the purpose of "if an individual belongs to a group and the group has created something, so has the individual".

using MYRULEZ [tolog] for {
  has-created-indirect ($group, $opus) :- {
     has-created ($group: creator, $opus: opus)
     |
     is-part-of  ($person: member, $group: whole),
     has-created ($group: creator, $opus: opus)
     }
}
SELECT $creator
WHERE
     MYRULEZ:has-created-indirect ($creator, $whatever)

What happens here is that - before the actual query - we introduced an adhoc ontology and gave it a name MYRULEZ. That name can be used as prefix from then on. In this ontological part we added two rules, following the style introduced in tolog. Without going into any details these rules introduce a new association type has-created-indirect which is derived from knowledge existing in our map. The second rule, for example, covers the situation that an individual belonging to a group is also a creator of the thing in question, albeit only indirectly.

The obvious benefit of such rules is to keep redundancy in maps itself low, while not burdening the query itself; it would have been possible to hard-code it there

SELECT $creator
WHERE
    has-created ($creator: creator, $opus: opus)
    |
    (
     is-part-of  ($creator: member, $group: whole),
     has-created ($group: creator, $opus: opus)
    )

On open issue is whether TMQL should adopt the tolog-ish way of thinking as the only way to express such additional domain knowledge or whether TMQL is kept promiscuous in the sense that other constraint languages (the upcoming TMCL, or OWL, for example) could potentially be used. The advantage then would be that the inferencing involved can be adapted to the problem at hand: inferencing with tolog is certainly of a higher computational complexity than that associated with OWL. On the other hand, comitting to a predicate structure like the above may allow a tighter integration in TMQL implementations.

Another venue open to us is to factor out the ontological information completely into a separate document:

using wine [owl] for http://www.w3.org/2001/sw/WebOnt/guide-src/wine.rdf
SELECT $winery
WHERE
    $winery is-a wine:Winery

using sales [python] for urn:x-in-house-database:sales-numbers
SELECT $artist, sales:total ($artist / bn `, 'uk', '2005')
WHERE
    $artist is-a artist

Implementors

Some potential implementors may have had a glimpse at the current TMQL draft. With its about 90 syntax rules and 33 pages (inclusive issues and editor notes) it about the same complexity as, say SPARQL. Once the ISO committee has made some decisions on various syntax issues we may also reduce these numbers somewhat.

What is not so obvious at first sight is that the different language flavours do not add any computational complexity. Both, SELECT and FLWR expressions can be mapped into path expressions, so that most of this syntactical variations are swallowed by the parser anyway. This is also true for the majority of shorthand notations introduced; while the numbers may drop over time, each of them can be dealt with a single line of code. The only exception to above rule is the processing for FLWR RETURN clauses, but that one is fairly trivial.

While path expressions might be an obvious implementation target, you may also want to map everything into FLWR expressions. Maybe just because you happen to have this shiny new XML database which understands XQuery and you decide to map TMQL onto XQuery.

Feedback

Designing a language for oneself is not always easy, even in cases where you completely agree with yourself. Doing this is a group or as part of a committee does not make things necessary easier, although discussions with other people help a lot. Creating a new language for a large group of developers with wildly varying backgrounds involves a lot of second guessing.

So if you would like to avoid that a lack of feedback is misinterpreted as utter, unconditional bliss and happiness, then please let us know what you think. I promise that there will be no involvement of the Russian mafia.