The presentation of this document has been augmented to identify changes from a previous version. Three kinds of changes are highlighted: new, added text, changed text, and deleted text.


W3C

Binary Module 1.0

EXPath Proposed Module 3 December 2013

This version:
http://expath.org/spec/binary/1.0
Latest version:
http://expath.org/spec/binary
Previous versions:
http://expath.org/spec/binary/20131113
http://expath.org/spec/binary/20130920
http://expath.org/spec/binary/20130731
http://expath.org/spec/binary/20130312
Editors:
Jirka Kosek <jirka@kosek.cz>
John Lumley <john@saxonica.com>

This document is also available in these non-normative formats: XML and Revision Markup.


Abstract

This proposal provides an API for XPath 2.0 to handle binary data. It defines extension functions to process data from binary files, including extracting subparts, searching, basic binary operations and conversion between binary and structured forms. It has been designed to be compatible with XQuery 1.0 and XSLT 2.0, as well as any other XPath 2.0 usage.

The module homepage, with more information, is on the EXPath website at http://expath.org/modules/binary/.

Table of Contents

1 Status of this document
2 Introduction
    2.1 Namespace conventions
    2.2 Error management
    2.3 Binary type
    2.4 Test suite
3 Use cases
    3.1 Example – finding JPEG size
    3.2 Example – reading and writing variable length ASN.1 integers
4 Loading and saving binary data
5 Defining 'constants' and conversions
    5.1 bin:hex
    5.2 bin:bin
    5.3 bin:octal
    5.4 bin:to-octets
    5.5 bin:from-octets
6 Basic operations
    6.1 bin:length
    6.2 bin:part
    6.3 bin:join
    6.4 bin:insert-before
    6.5 bin:pad-left
    6.6 bin:pad-right
    6.7 bin:find
7 Text decoding and encoding
    7.1 bin:decode-string
    7.2 bin:encode-string
8 Packing and unpacking of encoded numeric values
    8.1 Number 'endianness'
    8.2 Integer representation
    8.3 Representation of floating point numbers
    8.4 bin:pack-double
    8.5 bin:pack-float
    8.6 bin:pack-integer
    8.7 bin:unpack-double
    8.8 bin:unpack-float
    8.9 bin:unpack-integer
    8.10 bin:unpack-unsigned-integer
9 Bitwise operations
    9.1 bin:or
    9.2 bin:xor
    9.3 bin:and
    9.4 bin:not
    9.5 bin:shift

Appendices

A References
B Summary of error conditions


1 Status of this document

This document is in a final draft stage. Comments are welcomed at public-expath@w3.org mailing list (archive) a final specification.

2 Introduction

2.1 Namespace conventions

The module defined by this document defines several functions, all contained in the namespace http://expath.org/ns/binary. In this document, the bin prefix, when used, is bound to this namespace URI.

Error codes are defined in the same namespace (http://expath.org/ns/binary), and in this document are displayed with the same prefix, bin.

Binary file I/O uses facilities defined in [EXPath File], which defines functions in the namespace http://expath.org/ns/file. In this document, the file prefix, when used, is bound to this namespace URI.

2.2 Error management

Error conditions are identified by a code (a QName.) When such an error condition is reached in the evaluation of an expression, a dynamic error is thrown, with the corresponding error code (as if the standard XPath function error() had been called.)

2.3 Binary type

The principal binary type within this module is xs:base64Binary.

Conversion to and from xs:hexBinary can be performed by casting with xs:hexBinary() and xs:base64Binary().

Note:

As these types are normally implemented as wrappers around byte array structures containing the data, and differ only when being serialized to or parsed from text, such casting in-process should not involve data copying.

An item of type xs:base64Binary can be empty, i.e. contain no data, (in the same way that items of type xs:string can contain no characters.) Where 'data' arguments to functions that return binary data are optional (i.e. $arg as type?) and any of those optional arguments is set to the empty sequence, in general an empty sequence is returned, rather than an empty item of type xs:base64Binary.

2.4 Test suite

A suite of test-cases for all the functions defined in this module, in [QT3] format, is defined at [Test-suite].

3 Use cases

Development of this specification was driven by requirements which some XML developers regularly encounter in examining or generating data which is presented in binary, or other non-textual forms. Some typical use cases include:

3.1 Example – finding JPEG size

As an example, the following code reads the binary form of a JPEG image file, searches for the 'Start of Frame/DCT' segment, and unpacks the relevant binary sections to integers of height and width:

<xsl:variable name="binary" select="file:read-binary(@href)" as="xs:base64Binary"/>
<xsl:variable name="location" select="bin:find($binary,0,bin:hex('FFC0'))"/>
<size width="{bin:unpack-unsigned-integer($binary,$location+5,2,'most-significant-first')}"
      height="{bin:unpack-unsigned-integer($binary,$location+7,2,'most-significant-first')}"/>
               
      => <size width="377" height="327"/>

(The 'most-significant-first'() argument ensures the numeric conversion is 'big-endian', which is the format in JPEG.)

3.2 Example – reading and writing variable length ASN.1 integers

[ASN.1] defines several formats for identifying and encoding arbitrary-sized telecommunications data as streams of octets. Many of these forms specify the length of data as part of their encoding. For example, in the Basic Encoding Rules, an integer is represented as the following series of octets:

  • Type - 1 octet - in this case the value 0x02

  • Length – >=1 octet – the number of octets in the integer value. The length field itself can be variable in length - to accomodate VERY large integers (requiring more than 127 octets to represent, e.g. 2048-bit crypto keys.)

  • Payload – >=0 octets – the octets of the integer value in most-significant-first order.

To generate such a representation for an integer from XSLT/XPath, the following code might be used:

 <xsl:function name="bin:int-octets" as="xs:integer*">
    <xsl:param name="value" as="xs:integer"/>
    <xsl:sequence
            select="if($value ne 0) then (bin:int-octets($value idiv 256),$value mod 256) else ()"/>
 </xsl:function>
 <xsl:function name="bin:encode-ASN-integer" as="xs:base64Binary">
     <xsl:param name="int" as="xs:integer"/>
     <xsl:variable name="octets" select="bin:int-octets($int)"/>
     <xsl:variable name="length-octets"
         select="let $l := count($octets) return
         (if($l le 127) then $l 
         else (let $lo := bin:int-octets($l) return (128+count($lo),$lo)))"/>
     <xsl:sequence select="bin:from-octets((2,$length-octets,$octets))"/>
 </xsl:function>

The function bin:int-octets() returns a sequence of all the 'significant' octets of the integer (i.e. eliminating leading 'zeroes') in most-significant order. Examples of the encoding are:

 bin:encode-ASN-integer(0) => "AgA="
 bin:encode-ASN-integer(1234) => "AgIE0g=="
 bin:encode-ASN-integer(123456789123456789123456789123456789) => "Ag8XxuPAMviQRa10ZoQEXxU="
               
 bin:encode-ASN-integer(123456789.. 900 digits... 123456789) => "AoIBdgaTo....EBF8V"

The first example requires no octets to encode zero, hence its octets are 2,0. Both the second and third examples can be represented in less than 128 octets (2 and 15 respectively), so length is encoded as a single octet. The first three octets of the result for the last example, which encodes a 900-digit integer, are: 2,130,1 indicating that the data is represented by (130-128) * 256 + 1 = 513 octets and the length required two octets to encode.

Decoding is a matter of compound use of the integer decoding function:

 <xsl:function name="bin:decode-ASN-integer" as="xs:integer">
     <xsl:param name="in" as="xs:base64Binary"/>
     <xsl:sequence
         select="let $lo := bin:unpack-unsigned-integer($in,1,1,'BE') return (
         if($lo le 127) then bin:unpack-unsigned-integer($in,2,$lo,'BE') 
            else (let $lo2 := $lo - 128, $lo3 := bin:unpack-unsigned-integer($in,2,$lo2,'BE') return
            bin:unpack-unsigned-integer($in,2+$lo2,$lo3,'BE')))"
      />
 </xsl:function>               

(all numbers in ASN are 'big-endian') and the examples from above reverse:

 bin:decode-ASN-integer(xs:base64Binary("AgA=")) => 0
 bin:decode-ASN-integer(xs:base64Binary("AgIE0g==")) => 1234
 bin:encode-ASN-integer(xs:base64Binary("Ag8XxuPAMviQRa10ZoQEXxU=")) 
     => 123456789123456789123456789123456789              
 bin:encode-ASN-integer(xs:base64Binary("AoIBdgaTo....EBF8V")) 
     => 123456789.. 900 digits... 123456789                

4 Loading and saving binary data

This module defines no specific functions for reading and writing binary data from files. The EXPath File Module [EXPath File] provides three suitable functions:

There may be an argument for a positioned file:read-binary($file as xs:string,$offset as xs:integer), for access into large files without total read.

5 Defining 'constants' and conversions

Users of the package may need to define binary 'constants' within their code or examine the basic octets. The following functions support these:

5.1 bin:hex

Summary

Returns the binary form of the set of octets written as a sequence of (ASCII) hex digits ([0-9A-Fa-f]).

Signature

bin:hex($in as xs:string?) as xs:base64Binary?

Rules

$in will be effectively zero-padded from the left to generate an integral number of octets, i.e. an even number of hexadecimal digits. If $in is an empty string, then the result will be a xs:base64Binary with no embedded data.

Byte order in the result follows (per-octet) character order in the string.

If the value of $in is the empty sequence, the function returns an empty sequence.

Error Conditions

[bin:non-numeric-character] is raised if $in cannot be parsed as a hexadecimal number.

Notes

When the input string has an even number of characters, this function behaves similarly to the double cast xs:base64Binary(xs:hexBinary($string)).

Examples
bin:hex('11223F4E') => "ESI/Tg=="
bin:hex('1223F4E') => "ASI/Tg=="

5.2 bin:bin

Summary

Returns the binary form of the set of octets written as a sequence of (8-wise) (ASCII) binary digits ([01]).

Signature

bin:bin($in as xs:string?) as xs:base64Binary?

Rules

$in will be effectively zero-padded from the left to generate an integral number of octets. If $in is an empty string, then the result will be a xs:base64Binary with no embedded data.

Byte order in the result follows (per-octet) character order in the string.

If the value of $in is the empty sequence, the function returns an empty sequence.

Error Conditions

[bin:non-numeric-character] is raised if $in cannot be parsed as a binary number.

Examples
bin:bin('1101000111010101') => "0dU="
bin:bin('1000111010101') => "EdU="

5.3 bin:octal

Summary

Returns the binary form of the set of octets written as a sequence of (ASCII) octal digits ([0-7]).

Signature

bin:octal($in as xs:string?) as xs:base64Binary?

Rules

$in will be effectively zero-padded from the left to generate an integral number of octets. If $in is an empty string, then the result will be a xs:base64Binary with no embedded data.

Byte order in the result follows (per-octet) character order in the string.

If the value of $in is the empty sequence, the function returns an empty sequence.

Error Conditions

[bin:non-numeric-character] is raised if $in cannot be parsed as an octal number.

Examples
bin:octal('11223047') => "JSYn"

5.4 bin:to-octets

Summary

Returns binary data as a sequence of octets.

Signature

bin:to-octets($in as xs:base64Binary) as xs:integer*

Rules

If $in is a zero length binary data then the empty sequence is returned.

Octets are returned as integers from 0 to 255.

5.5 bin:from-octets

Summary

Converts a sequence of octets into binary data.

Signature

bin:from-octets($in as xs:integer*) as xs:base64Binary

Rules

Octets are integers from 0 to 255.

If the value of $in is the empty sequence, the function returns zero-sized binary data.

Error Conditions

[bin:octet-out-of-range] is raised if one of the octets lies outside the range 0 - 255.

6 Basic operations

6.1 bin:length

Summary

The bin:length function returns the size of binary data in octets.

Signature

bin:length($in as xs:base64Binary) as xs:integer

Rules

Returns the size of binary data in octets.

6.2 bin:part

Summary

The bin:part function returns a specified part of binary data.

Signatures

bin:part($in as xs:base64Binary?, $offset as xs:integer) as xs:base64Binary?
bin:part($in as xs:base64Binary?,
$offset as xs:integer,
$size as xs:integer) as xs:base64Binary?

Rules

Returns a section of binary data starting at the $offset octet. If $size is defined, the size of the returned binary data is $size octets. If $size is absent, all remaining data from $offset is returned.

The $offset is zero based.

The values of $offset and $size must be non-negative integers.

It is a dynamic error if $offset + $size is larger than the size of the binary data in $in.

If the value of $in is the empty sequence, the function returns an empty sequence.

Error Conditions

[bin:index-out-of-range] is raised if $offset is negative or $offset + $size is larger than the size of the binary data of $in.

[bin:negative-offset] is raised if $offset is negative.

[bin:negative-size] is raised if $size is negative.

[bin:offset-beyond-end] is raised if $offset + $size extends beyond the binary data of $in.

Notes

Note that fn:subsequence() and fn:substring() [fo11] both use xs:double for offset and size – this is a legacy from XPath 1.0.

Examples

Testing whether $data variable starts with binary content consistent with a PDF file:

bin:part($data, 0, 4) eq bin:hex("25504446")

25504446 is the magic number for PDF files: it is the US-ASCII encoded hexadecimal value for %PDF. 7.2 bin:encode-string can be used to convert a string to its binary representation.

6.3 bin:join

Summary

Returns the binary data created by concatenating the binary data items in a sequence.

Signature

bin:join($in as xs:base64Binary*) as xs:base64Binary

Rules

The function returns an xs:base64Binary created by concatenating the items in the sequence $in, in order.

If the value of $in is the empty sequence, the function returns a binary item containing no data bytes.

6.4 bin:insert-before

Summary

The bin:insert-before function inserts additional binary data at a given point in other binary data.

Signature

bin:insert-before($in as xs:base64Binary?,
$offset as xs:integer,
$extra as xs:base64Binary?) as xs:base64Binary?

Rules

Returns binary data consisting sequentially of the data from $in upto and including the $offset - 1 octet, followed by all the data from $extra, and then the remaining data from $in.

The $offset is zero based.

The value of $offset must be a non-negative integer.

It is a dynamic error if $offset is larger than the size of the binary data in $in.

If the value of $in is the empty sequence, the function returns an empty sequence.

If the value of $extra is the empty sequence, the function returns $in.

If $offset eq 0 the result is the binary concatenation of $extra and $in, i.e. equivalent to bin:join(($extra,$in)).

Error Conditions

[bin:index-out-of-range] is raised if $offset is negative or $offset is larger than the size of the binary data of $in.

[bin:negative-offset] is raised if $offset is negative.

[bin:offset-beyond-end] is raised if $offset extends beyond the binary data of $in.

Notes

Note that under non-error conditions and when $offset gt 0 the function is equivalent to:

bin:join((bin:part($in,0,$offset - 1),$extra,bin:part($in,$offset)))

6.5 bin:pad-left

Summary

Returns the binary data created by padding $in with $size octets from the left. The padding octet values are $octet or zero if omitted.

Signatures

bin:pad-left($in as xs:base64Binary?, $size as xs:integer) as xs:base64Binary?
bin:pad-left($in as xs:base64Binary?,
$size as xs:integer,
$octet as xs:integer) as xs:base64Binary?

Rules

The function returns an xs:base64Binary created by padding the input with $size octets in front of the input. If $octet is specified, the padding octets each have that value, otherwise they are blankinitialized to 0.

$size must be a non-negative integer.

If the value of $in is the empty sequence, the function returns an empty sequence.

Error Conditions

[bin:negative-size] is raised if $size is negative.

[bin:octet-out-of-range] is raised if $octet lies outside the range 0 - 255.

Notes

Padding with a non-zero octet value can also be accomplished by the XPath expressions:

bin:join((bin:from-octets((1 to $pad-length) ! $pad-octet), $in)) [XPath 3.0]
bin:join((bin:from-octets(for $ i in (1 to $pad-length) return $pad-octet), $in)) [XPath 2.0]

6.6 bin:pad-right

Summary

Returns the binary data created by padding $in with $size blank octets from the right. The padding octet values are $octet or zero if omitted.

Signatures

bin:pad-right($in as xs:base64Binary?, $size as xs:integer) as xs:base64Binary?
bin:pad-right($in as xs:base64Binary?,
$size as xs:integer,
$octet as xs:integer) as xs:base64Binary?

Rules

The function returns an xs:base64Binary created by padding the input with $size blank octets after the input. If $octet is specified, the padding octets each have that value, otherwise they are blankinitialized to 0.

$size must be a non-negative integer.

If the value of $in is the empty sequence, the function returns an empty sequence.

Error Conditions

[bin:negative-size] is raised if $size is negative.

[bin:octet-out-of-range] is raised if $octet lies outside the range 0 - 255.

Notes

Padding with a non-zero octet value can also be accomplished by the XPath expressions:

bin:join(($in,bin:from-octets((1 to $pad-length) ! $pad-octet)))  [XPath 3.0]
bin:join(($in,bin:from-octets(for $ i in (1 to $pad-length) return $pad-octet)))  [XPath 2.0]

6.7 bin:find

Summary

Returns the first location in $in of $search, starting at the $offset octet.

Signature

bin:find($in as xs:base64Binary?,
$offset as xs:integer,
$search as xs:base64Binary) as xs:integer?

Rules

The function returns the first location of the binary search sequence in the input, or if not found, the empty sequence.

If $search is empty $offset is returned.

The search sequence cannot be 'empty'.

The value of $offset must be a non-negative integer.

The $offset is zero based.

The returned location is zero based.

If the value of $in is the empty sequence, the function returns an empty sequence.

Error Conditions

[bin:index-out-of-range] is raised if $offset is negative or $offset is larger than the size of the binary data of $in.

[bin:negative-offset] is raised if $offset is negative.

[bin:offset-beyond-end] is raised if $offset extends beyond the binary data of $input.

[bin:empty-search-item] is raised if $search is empty binary data.

Notes

Finding all the matches can be accomplished with simple recursive application:

<xsl:function name="bin:find-all" as="xs:integer*">
     <xsl:param name="data" as="xs:base64Binary?"/>
     <xsl:param name="offset" as="xs:integer"/>
     <xsl:param name="pattern" as="xs:base64Binary"/>
     <xsl:sequence
         select="if(bin:length($pattern) = 0) then ()
         else let $found := bin:find($data,$offset,$pattern) return
         if($found) then ($found,
             if($found + 1 lt bin:length($data)) then bin:find-all($data,$found + 1,$pattern) else ())
             else ()"/>
</xsl:function>

7 Text decoding and encoding

7.1 bin:decode-string

Summary

Decodes binary data as a string in a given encoding.

Signatures

bin:decode-string($in as xs:base64Binary?) as xs:string?
bin:decode-string($in as xs:base64Binary?, $encoding as xs:string) as xs:string?
bin:decode-string($in as xs:base64Binary?,
$encoding as xs:string,
$offset as xs:integer) as xs:string?
bin:decode-string($in as xs:base64Binary?,
$encoding as xs:string,
$offset as xs:integer,
$size as xs:integer) as xs:string?

Rules

If $offset and $size are provided, the $size octets from $offset are decoded. If $offset alone is provided, octets from $offset to the end are decoded, otherwise the entire octet sequence is used.

The $encoding argument is the name of an encoding. The values for this attribute follow the same rules as for the encoding attribute in an XML declaration. The only values which every implementation is required to recognize are utf-8 and utf-16.

If $encoding is ommitted, utf-8 encoding is assumed.

The values of $offset and $size must be non-negative integers.

If the value of $in is the empty sequence, the function returns an empty sequence.

$offset is zero based.

Error Conditions

[bin:index-out-of-range] is raised if $offset is negative or $offset + $size is larger than the size of the binary data of $in.

[bin:negative-offset] is raised if $offset is negative.

[bin:negative-size] is raised if $size is negative.

[bin:offset-beyond-end] is raised if $offset + $size - 1 extends beyond the binary data of $in.

[bin:unknown-encoding] is raised if $encoding is invalid or not supported by the implementation.

[bin:conversion-error] is raised if there is an error or malformed input during decoding the string. Additional information about the error may be passed through suitable error reporting mechanisms - this is implementation-dependant.

Examples

Testing whether $data variable starts with binary content consistent with a PDF file:

bin:decode-string($data, 'UTF-8', 0, 4) eq '%PDF'

The first four characters of a PDF file are '%PDF'.

7.2 bin:encode-string

Summary

Encodes a string into binary data using a given encoding.

Signatures

bin:encode-string($in as xs:string?) as xs:base64Binary?
bin:encode-string($in as xs:string?, $encoding as xs:string) as xs:base64Binary?

Rules

The $encoding argument is the name of an encoding. The values for this attribute follow the same rules as for the encoding attribute in an XML declaration. The only values which every implementation is required to recognize are utf-8 and utf-16.

If $encoding is ommitted, utf-8 encoding is assumed.

If the value of $in is the empty sequence, the function returns an empty sequence.

Error Conditions

[bin:unknown-encoding] is raised if $encoding is invalid or not supported by the implementation.

[bin:conversion-error]is raised if there is an error or malformed input during encoding the string. Additional information about the error may be passed through suitable error reporting mechanisms - this is implementation-dependant.

8 Packing and unpacking of encoded numeric values

8.1 Number 'endianness'

Packing and unpacking numeric values can be performed in 'most-significant-first' ('big-endian') or 'least-significant-first' ('little-endian') octet order. The default is 'most-significant-first'. The functions have an optional parameter $octet-order whose string value controls the order. Least-significant-first order is indicated by any of the values least-significant-first, little-endian or LE. Most-significant-first order is indicated by any of the values most-significant-first, big-endian or BE.

8.2 Integer representation

Integers within binary data are represented, or assumed to be represented, as an integral number of octets. Integers where $length is greater than 8 octets (and thus not representable as a long) might be expected in some situations, e.g. encryption. Whether the range of integers is limited to ±2^63 may be implementation-dependant.

8.3 Representation of floating point numbers

Care should be taken with the packing and unpacking of floating point numbers (xs:float and xs:double). The binary representations are expected to correspond with those of the IEEE single/double-precision 32/64-bit floating point types [IEEE 754-1985]. Consequently they will occupy 4 or 8 octets when packed.

Positive and negative infinities are supported. INF maps to 0x7f80 0000 (float), 0x7ff0 0000 0000 0000 (double). -INF maps to 0xff80 0000 (float), 0xfff0 0000 0000 0000 (double).

Negative zero (0x8000 0000 0000 0000 double, 0x8000 0000 float) encountered during unpacking will yield negative zero forms (e.g. -xs:double(0.0)) and negative zeros will be written as a result of packing.

[XML Schema 1.1 Part 2] provides only one form of NaN which corresponds to a 'quiet' NaN with zero payload of [IEEE 754-1985] with forms 0x7fc0 0000 (float), 0x7ff8 0000 0000 0000 (double). These are the bit forms that will be packed. 'Signalling' NaN values (0x7f80 0001 -> 0x7fbf ffff or 0xff80 0001 -> 0xffbf ffff, 0x7ff0 0000 0000 0001 -> 0x7ff7 ffff ffff ffff or 0xfff0 0000 0000 0001 -> 0xfff7 ffff ffff ffff) encountered during unpacking will be replaced by 'quiet' NaN. Any low-order payload in a unpacked quiet NaN is also zeroed.

8.4 bin:pack-double

Summary

Returns the 8-octet binary representation of a double value.

Signatures

bin:pack-double($in as xs:double) as xs:base64Binary
bin:pack-double($in as xs:double, $octet-order as xs:string) as xs:base64Binary

Rules

Most-significant-octet-first number representation is assumed unless the $octet-order parameter is specified. Acceptable values for $octet-order are described in 8.1 Number 'endianness'.

The binary representation will correspond with that of the IEEE double-precision 64-bit floating point type [IEEE 754-1985]. For more details see 8.3 Representation of floating point numbers.

Error Conditions

[bin:unknown-significance-order] is raised if the value $octet-order is unrecognized.

8.5 bin:pack-float

Summary

Returns the 4-octet binary representation of a float value.

Signatures

bin:pack-float($in as xs:float) as xs:base64Binary
bin:pack-float($in as xs:float, $octet-order as xs:string) as xs:base64Binary

Rules

Most-significant-octet-first number representation is assumed unless the $octet-order parameter is specified. Acceptable values for $octet-order are described in 8.1 Number 'endianness'.

The binary representation will correspond with that of the IEEE single-precision 32-bit floating point type [IEEE 754-1985]. For more details see 8.3 Representation of floating point numbers.

Error Conditions

[bin:unknown-significance-order] is raised if the value $octet-order is unrecognized.

8.6 bin:pack-integer

Summary

Returns the twos-complement binary representation of an integer value treated as $size octets long. Any 'excess' high-order bits are discarded.

Signatures

bin:pack-integer($in as xs:integer, $size as xs:integer) as xs:base64Binary
bin:pack-integer($in as xs:integer,
$size as xs:integer,
$octet-order as xs:string) as xs:base64Binary

Rules

Most-significant-octet-first number representation is assumed unless the $octet-order parameter is specified. Acceptable values for $octet-order are described in 8.1 Number 'endianness'.

Specifying a $size of zero yields an empty binary data.

Error Conditions

[bin:unknown-significance-order] is raised if the value $octet-order is unrecognized.

[bin:negative-size] is raised if $size is negative.

Notes

If the integer being packed has a maximum precision of $size octets, then signed/unsigned versions are not necessary. If the data is considered unsigned, then the most significant bit of the bottom $size octets has a normal positive (2^(8 *$size - 1)) meaning. If it is considered to be a signed value, then the MSB and all the higher order, discarded bits will be '1' for a negative value and '0' for a positive or zero. If this function were to check the 'sizing' of the supplied integer against the packing size, then any values of MSB and the discarded higher order bits other than 'all 1' or 'all 0' would constitute an error. This function does not perfom such checking.

8.7 bin:unpack-double

Summary

Extract double value stored at the particular offset in binary data.

Signatures

bin:unpack-double($in as xs:base64Binary, $offset as xs:integer) as xs:double
bin:unpack-double($in as xs:base64Binary,
$offset as xs:integer,
$octet-order as xs:string) as xs:double

Rules

Extract the double value stored in the 8 successive octets from the $offset octet of the binary data of $in.

Most-significant-octet-first number representation is assumed unless the $octet-order parameter is specified. Acceptable values for $octet-order are described in 8.1 Number 'endianness'.

The value of $offset must be a non-negative integer.

The $offset is zero based.

The binary representation is expected to correspond with that of the IEEE double-precision 64-bit floating point type [IEEE 754-1985]. For more details see 8.3 Representation of floating point numbers.

Error Conditions

[bin:index-out-of-range] is raised if $offset is negative or $offset + 8 (octet-length of xs:double) is larger than the size of the binary data of $in.

[bin:negative-offset] is raised if $offset is negative.

[bin:offset-beyond-end] is raised if $offset + 7 (octet-length of xs:double - 1) extends beyond the binary data of $in.

[bin:unknown-significance-order] is raised if the value $octet-order is unrecognized.

8.8 bin:unpack-float

Summary

Extract float value stored at the particular offset in binary data.

Signatures

bin:unpack-float($in as xs:base64Binary, $offset as xs:integer) as xs:float
bin:unpack-float($in as xs:base64Binary,
$offset as xs:integer,
$octet-order as xs:string) as xs:float

Rules

Extract the float value stored in the 4 successive octets from the $offset octet of the binary data of $in.

Most-significant-octet-first number representation is assumed unless the $octet-order parameter is specified. Acceptable values for $octet-order are described in 8.1 Number 'endianness'.

The value of $offset must be a non-negative integer.

The $offset is zero based.

The binary representation is expected to correspond with that of the IEEE single-precision 32-bit floating point type [IEEE 754-1985]. For more details see 8.3 Representation of floating point numbers.

Error Conditions

[bin:index-out-of-range] is raised if $offset is negative or $offset + 4 (octet-length of xs:float) is larger than the size of the binary data of $in.

[bin:negative-offset] is raised if $offset is negative.

[bin:offset-beyond-end] is raised if $offset + 3 (octet-length of xs:float - 1) extends beyond the binary data of $in.

[bin:unknown-significance-order] is raised if the value $octet-order is unrecognized.

8.9 bin:unpack-integer

Summary

Returns a signed integer value represented by the $size octets starting from $offset in the input binary representation. Necessary sign extension is performed (i.e. the result is negative if the high order bit is '1').

Signatures

bin:unpack-integer($in as xs:base64Binary,
$offset as xs:integer,
$size as xs:integer) as xs:integer
bin:unpack-integer($in as xs:base64Binary,
$offset as xs:integer,
$size as xs:integer,
$octet-order as xs:string) as xs:integer

Rules

Most-significant-octet-first number representation is assumed unless the $octet-order parameter is specified. Acceptable values for $octet-order are described in 8.1 Number 'endianness'.

The values of $offset and $size must be non-negative integers.

$offset is zero based.

Specifying a $size of zero yields the integer 0.

Error Conditions

[bin:index-out-of-range] is raised if $offset is negative or $offset + $size is larger than the size of the binary data of $in.

[bin:negative-offset] is raised if $offset is negative.

[bin:negative-size] is raised if $size is negative.

[bin:offset-beyond-end] is raised if $offset + $size - 1 extends beyond the binary data of $in.

[bin:unknown-significance-order] is raised if the value $octet-order is unrecognized.

Notes

For discussion on integer range see 8.2 Integer representation.

8.10 bin:unpack-unsigned-integer

Summary

Returns an unsigned integer value represented by the $size octets starting from $offset in the input binary representation.

Signatures

bin:unpack-unsigned-integer($in as xs:base64Binary,
$offset as xs:integer,
$size as xs:integer) as xs:integer
bin:unpack-unsigned-integer($in as xs:base64Binary,
$offset as xs:integer,
$size as xs:integer,
$octet-order as xs:string) as xs:integer

Rules

Most-significant-octet-first number representation is assumed unless the $octet-order parameter is specified. Acceptable values for $octet-order are described in 8.1 Number 'endianness'.

The values of $offset and $size must be non-negative integers.

The $offset is zero based.

Specifying a $size of zero yields the integer 0.

Error Conditions

[bin:index-out-of-range] is raised if $offset is negative or $offset + $size is larger than the size of the binary data of $in.

[bin:negative-offset] is raised if $offset is negative.

[bin:negative-size] is raised if $size is negative.

[bin:offset-beyond-end] is raised if $offset + $size - 1 extends beyond the binary data of $in.

[bin:unknown-significance-order] is raised if the value $octet-order is unrecognized.

Notes

For discussion on integer range see 8.2 Integer representation.

9 Bitwise operations

9.1 bin:or

Summary

Returns the "bitwise or" of two binary arguments.

Signature

bin:or($a as xs:base64Binary?, $b as xs:base64Binary?) as xs:base64Binary?

Rules

Returns "bitwise or" applied between $a and $b.

If either argument is the empty sequence, an empty sequence is returned.

Error Conditions

[bin:differing-length-arguments] is raised if the input arguments are of differing length.

9.2 bin:xor

Summary

Returns the "bitwise xor" of two binary arguments.

Signature

bin:xor($a as xs:base64Binary?, $b as xs:base64Binary?) as xs:base64Binary?

Rules

Returns "bitwise exclusive or" applied between $a and $b.

If either argument is the empty sequence, an empty sequence is returned.

Error Conditions

[bin:differing-length-arguments] is raised if the input arguments are of differing length.

9.3 bin:and

Summary

Returns the "bitwise and" of two binary arguments.

Signature

bin:and($a as xs:base64Binary?, $b as xs:base64Binary?) as xs:base64Binary?

Rules

Returns "bitwise and" applied between $a and $b.

If either argument is the empty sequence, an empty sequence is returned.

Error Conditions

[bin:differing-length-arguments] is raised if the input arguments are of differing length.

9.4 bin:not

Summary

Returns the "bitwise not" of a binary argument.

Signature

bin:not($in as xs:base64Binary?) as xs:base64Binary?

Rules

Returns "bitwise not" applied to $in.

If the argument is the empty sequence, an empty sequence is returned.

9.5 bin:shift

Summary

Shift bits in binary data.

Signature

bin:shift($in as xs:base64Binary?, $by as xs:integer) as xs:base64Binary?

Rules

If $by is positive then bits are shifted $by times to the left.

If $by is negative then bits are shifted -$by times to the right.

If $by is zero, the result is identical to $in.

If |$by| is greater than the bit-length of $in then an all-zeros result, of the same length as $in, is returned.

|$by| can be greater than 8, implying multi-byte shifts.

The result always has the same size as $in.

The shifting is logical: zeros are placed into discarded bits.

If the value of $in is the empty sequence, the function returns an empty sequence.

Notes

Bit shifting across byte boundaries implies 'big-endian' treatment, i.e. the leftmost (high-order) bit when shifted left becomes the low-order bit of the preceding byte.

Examples
bin:shift(bin:hex("000001"), 17) → bin:hex("020000")

A References

ASN.1
OSI networking and system aspects – Abstract Syntax Notation One (ASN.1) – see ASN.1 encoding rules: Specification of Basic Encoding Rules (BER), Canonical Encoding Rules (CER) and Distinguished Encoding Rules (DER) . ITU-T X.690 (07/2002)
EXPath File
File Module. Christian Grün and Matthias Brantner, editors. EXPath Candidate Module. 14 June 2012.
F&O 3.0
XPath and XQuery Functions and Operators 3.0. Michael Kay, editor. W3C Candidate Recommendation 21 May 2013.
IEEE 754-1985
IEEE Standard for Binary Floating-Point Arithmetic. See http://standards.ieee.org/reading/ieee/std_public/description/busarch/754-1985_desc.html
QT3
XML Query Test Suite. W3C 21 June 2013.
Test-suite
The test suite for this module, using QT3 format, is in the EXPath repository http://github.com/expath/expath-cg in the directory tests/qt3/binary/
XML Schema 1.1 Part 2
W3C XML Schema Definition Language (XSD) 1.1 Part 2: Datatypes. David Peterson et al, editors. W3C Recommendation 5 April 2012.

B Summary of error conditions

bin:differing-length-arguments
The arguments to a bitwise operation are of differing lengths.
bin:index-out-of-range
Attempting to retrieve data outside the meaningful range of a binary data type.
bin:negative-offset
Attempting to retrieve data before the start of a binary data type.
bin:offset-beyond-end
Attempting to retrieve data beyond the end of a binary data type.
bin:negative-size
Size of binary portion, required numeric size or padding is negative.
bin:empty-search-item
Binary search item is empty.
bin:octet-out-of-range
Attempting to pack binary value with octet outside range.
bin:non-numeric-character
Wrong character in binary 'numeric constructor' string.
bin:unknown-encoding
The specified encoding is not supported.
bin:conversion-error
Error in converting to/from a string.
bin:unknown-significance-order
Unknown octet-order value.