Packaging System

1 Introduction

XSLT, XQuery and XProc are amazing programming languages. But they lack a large choice of libraries, and when such libraries do exist, this is a challenge to install. There is no automatic install process, the rules are different for each processor, library authors do not follow the same rules regarding the info they provide, the cataloging, the way they reference third-party libraries, etc.

All those problems (well, most of them) can be addressed by a packaging system that is broadly adopted by processor vendors and library authors. The cornerstone of such a system is the packaging format: a description of the information to be provided by the library authors and how to provide and structure them.

2 Concepts

A package is a set of files fulfilling a common purpose. A package could for instance provide XML schemas for a particular document type, alongside a set of XSLT stylesheets to format the same document type to HTML or XSL-FO. Or a package can simply provide an XQuery library module containing a set of functions (directly supporting the notion of library). Those files composing a package (stylesheets, schemas, pipelines, queries, etc.) are called its components.

A package has a unique name (a URI) as well as a convenient short name, also known as its abbrev (an NCName), used in contexts where a URI is not suitable (or simply not convenient) and where the uniqueness of the name is not necessary. A package has a set of dependencies too. A dependency is another package, identified by its full name, a package depends on.

A package provides all those infos as well as its components as a single file. This is then very convenient to organize packages, publish them, give them to a processor to install them, etc.

A component can be of one out of several types: an XSLT stylesheet, an XML schema, a RELAX NG schema, an XProc pipeline, but also any other type a particular implementation chose to support. It is identified by a public URI. This public URI is used to access the component from the outside of the package. Each component type has its own URI space, within which a component's URI has to be unique (so two different components can have the same public URI if they are of different types, but this is not recommended).

All the components composing the package, alongside an additional package descriptor, are used to create a ZIP file. This file is the physical package, and can be used to distribute the package to the users. The package descriptor is a plain XML file, named expath-pkg.xml at the root of the ZIP file, and containing information about the package (like its name and its version number) and about the components it provides and how to reference them.

3 Package descriptor

The package descriptor is an XML file, named expath-pkg.xml and located at the root of the ZIP file. It describe the whole package, and all of its components. Alongside this descriptor, the root of the ZIP file contains a directory named after the module abbrev, which contains the components and any other file the package needs. All the relative URIs used to identify components are relative to the package directory. See A Package descriptor schema for a normative schema.

<package name = uri
         abbrev = string
         version = string
         spec = string>
   title,
   home?,
   dependency*,
   (xslt
   |xquery
   |xproc
   |xsd
   |rng
   |rnc)*
</package>

<dependency package = uri
            processor = string
            versions = string
            semver = string
            semver-min = string
            semver-max = string>
   empty
</dependency>

<xslt>
   import-uri,
   file
</xslt>

<xquery>
   (namespace
   |import-uri),
   file
</xquery>

<xproc>
   import-uri,
   file
</xproc>

<xsd>
   (namespace
   |import-uri),
   file
</xsd>

<rng>
   import-uri,
   file
</rng>

<rnc>
   import-uri,
   file
</rnc>

<schematron>
   import-uri,
   file
</schematron>

<nvdl>
   import-uri,
   file
</nvdl>

4 Processor behaviour

A processor is any program that use packaged components. For instance, an XSLT processor uses XSLT stylesheets (as well as XML schemas for an XSLT 2.0 SA processor), an XML database can use XQuery modules, XSLT stylesheets or any other kind of component, etc. But processors include also IDEs, editors, and any program that could want to use packaged components and that supports this specification.

The installed package list is implementation-defined. Each implementation (for a specific processor) can define its own way to install and remove packages, as long as it properly documents it. A processor should use, when appropriate, the on-disk repository layout as defined below.

Whether or not such a repository exists (or several repositories), the implementation must define an installed packages list. How this is done is outside the scope of this spec. An XML IDE could provide a way to select packages to activate for a specific scenario, or a web server container could activate packages on a per-web application basis.

When a reference to a file of a specific kind is done via an absolute URI, a processor must look up for this URI in the corresponding URI space in the repository. How the repository is set to the processor is implementation-defined (a processor can also use a list of repositories, and enable or disable some libraries in any implementation-defined way.)

The URI space to use is defined by the nature of the reference. An XSLT href attribute on xsl:import will use the xslt URI space, while it will use the xsd space for xsl:import-schema.

<p:xslt>
   <p:input port="stylesheet">
      <p:document href="..." pkg:kind="xslt"/>
   </p:input>
   ...
</p:xslt>

5 Versioning and dependency management

Every package has a name (a URI) and a version number. It can also have a set of dependencies on other packages, identified by their names. Each dependency can also define a specific version (or range of versions) of the package it depends on. In addition to depending on other packages, a package can also depend on a specific processor (or on one among a set of specifc processors). This section uses the following example package to illustrate those concepts:

<package name="http://example.org/app" version="1.0" ...>

   <dependency package="http://external.org/library"/>
   <dependency package="http://partner.com/lib-2" semver="2"/>
   <dependency processor="saxon" version="pe"/>

   <!-- rest of package descriptor skipped -->
   ...

</package>

At first glance, we can see that this package (which is named http://example.org/app and has the version number 1.0), depends on the package http://external.org/library (no specific version), on the package http://partner.com/lib-2 (version 2, whatever minor revision numbers) and on the Saxon processor, Professional Edition. The rest of this section explains in details the rules behind those dependencies, how to represent them, and their semantics.

The remainder of this section will use the term primary for the current package, described by this package decriptor, and secondary for a package it depends on.

<dependency processor="saxon"/>

6 On-disk repository layout

This section defines a standard structure for on-disk repositories. An implementation can choose not to support this kind of repository and to define its own one (or even to not define it publicly, just to provide the ability to install and remove packages, in a clearly documented way.) However, there are several advantages to support this structure, the most obvious one is to be able to benefit from existing tools to manage such repositories as well as existing libraries to access those repositories.

The repository is a directory on the file system, within which the packages are installed. Each package results in a subdirectory in the repository, where the content of the XAR file is unzipped. Next to those package directories, there is also a subdirectory named .expath-pkg/, which contains administrative information about the packages installed in this repository. The package directories should be named after the package abbreviation (the abbrev attribute in the package descriptor) and the version number, separated by a dash. A package directory name must not contain the space character.

The repository maintains a list of installed packages. For each of them, this list contains its name (the name URI), its subdirectory name, and its version number. This list is maintained within two files: .expath-pkg/packages.xml and .expath-pkg/packages.txt. Both contain the same information, the former in an XML format and the later as plain text. The XML format looks like the following (see B Package list schema for a normative schema):

<packages xmlns="http://expath.org/ns/repo/packages">
   <package name="http://example.org/some/package"
            dir="some-package-2.3"
            version="2.3"/>
   <package name="http://example.com/some-library"
            dir="some-lib-0.1"
            version="0.1"/>
   <package ...
</packages>

The text file is a line-separated file (either line termination can be used: \r, \n or \n\r, though the standard Internet line termination is preferred: \n\r). Each line represent a package, and is a space-separated record. The first field is the package directory (the name if the subdirectory within the repository where the package has been unzipped), the second one is the package name (the name URI), and the last one is the version number. The two first fields cannot contain any space character. The version number is the only one that can contain a space character (though this is not recommended). Every field is separated by exactly one space character (the Unicode character with code point #x20). This file then looks like:

some-package-2.3 http://example.org/some/package 2.3
some-lib-0.1 http://example.com/some-library 0.1
...

The overall repository thus looks like:

[repository-dir]/
   .expath-pkg/
      packages.txt
      packages.xml
   some-package-2.3/
      query.xq
      style.xsl
   some-lib-0.1/
      ...

In addition to this structure, a specific processor can store information in subdirectories (both in the repository level or within each package directories), provided those directories start with a dot, and are named unambiguously with the name of the processor. For instance [repository]/.saxon/, or [repository]/[some-pkg]/.exist/.

7 Example

This section provides a non-normative example to illustrate the concepts defined here. Instead of using a hello world example, it describes the packaging of the existing FunctX library [FunctX]. This library consists of a standard XQuery 1.0 library module and a standard XSLT 2.0 stylesheet (both provide the same set of functions to either XQuery or XSLT, but this is not relevant to packaging.)

The first thing to do is to create a ZIP file with both of those components, alongside a package descriptor. The constraints are: 1) the package descriptor is named expath-pkg.xml at the root of the package, 2) the library content is in a directory at the root of the package (aka the package directory), and 3) the name of this directory must be the value of the abbrev attribute, and must be a valid NCName. The structure (the content) of the package directory is completely free. In our case, let's just put both component files directly in the package directory, and define the module name as functx:

expath-pkg.xml
functx/
   functx.xql
   functx.xsl

The XQuery library module's target namespace is defined by the module itself. For the XSLT stylesheet, we have to define its public URI, used to identify it within an xsl:import (or any other means, for instance within XProc or an IDE scenarii system). Let's define it as http://www.functx.com/functx.xsl. The package descriptor thus looks like the following:

<package xmlns="http://expath.org/ns/pkg"
         spec="1.0"
         name="http://www.functx.com"
         abbrev="functx"
         version="1.0">
   <title>FunctX library</title>
   <xquery>
      <namespace>http://www.functx.com</namespace>
      <file>functx.xql</file>
   </xquery>
   <xslt>
      <import-uri>http://www.functx.com/functx.xsl</import-uri>
      <file>functx.xsl</file>
   </xslt>
</package>

We just have to create a ZIP file with this structure and content. The convention is to call this file functx-1.0.xar (that is, [abbrev]-[version].xar). That's all for the package itself. If the target processor supports the on-disk repository layout, here is what the repository could look like after the package has been installed:

[repository-dir]/
   .expath-pkg/
      packages
      packages.xml
   functx-1.0/
      expath-pkg.xml
      functx/
         functx.xql
         functx.xsl

The content of .expath-pkg/packages.xml is:

<packages xmlns="http://expath.org/ns/repo/packages">
   <package name="http://www.functx.com"
            dir="functx-1.0"
            version="1.0"/>
</packages>

The content of .expath-pkg/packages.txt is:

functx-1.0 http://www.functx.com 1.0

The package directory functx-1.0/ has been named after the abbrev and the version number. Its content is simply the content of the XAR file, unzipped. A user can then import the FunctX library from either an XQuery module or an XSLT stylesheet by using respectively an import statement:

import module namespace fx = "http://www.functx.com";

or an import instruction:

<xsl:import href="http://www.functx.com/functx.xsl"/>

8 Extensibility

The package format defined in this specification is a complete system to package libraries, for the right definition of a library. But a specfic kind of library can require more information. For instance, a web application is also a set of components, but requiring more configuration. An additional descriptor format for such applications could be defined and added to this packaging framework to define a specialized kind of packages. By building upon this packaging format, the new format would benefit from the existing mechanism to map public URIs to components.

Another extensibility point is the definition of additional component types. The component types defined here are the standard types, but an implementation may support additional implementation-defined types.

More generally, an implementation may define any extension element in the package descriptor to achieve its purposes, providing that the new elements it defines are neither in the EXPath Packaging namespace nor in the null namespace.