Archive versions

Archive versions#

The structure of QIIME 2 archives has evolved as QIIME 2 has been developed. This section describes each historical version of the QIIME 2 Archive format, and may be useful to interface developers whose code depends on guarantees made by that format (source code).

Version-agnostic format guarantees#

Though there is significant variability in the format of QIIME 2 Archives across archive versions, all archive versions share some common traits.

These shared characteristics, defined in the _Archive class in qiime2/core/archive/archiver.py, must be consistent across all formats over time as they allow archive versions to be checked, and archives with different formats to be dispatched to the appropriate version-specific parsers.

All QIIME 2 Archives have:

  • a directory named with the Archive UUID, directly under the archive root at /<UUID>/, and

  • a file /<UUID>/VERSION within that directory, formatted as shown below.

Fig. 8 illustrates the Archive file system.

../../_images/format-agnostic-archive-structure.svg

Fig. 8 Box and arrow diagram of the guaranteed components of an archive.#

The VERSION file format is described above.

Archive Version 0#

Version 0 was the original QIIME 2 Archive format, and there aren’t many V0 Archives “in the wild”. V0 Archives were produced by alpha versions of the QIIME 2 framework, and were superseded in framework version 2.0.6 on 24 October 2016.

  • Result data files are written in the directory /<UUID>/data

  • Result UUID, semantic type, and format information are saved in /<UUID>/metadata.yaml.

  • The ArchiveFormat class in v0.py offers convenience methods for loading and parsing metadata.yaml files.

Fig. 9 illustrates the format of a Version 0 Archive.

../../_images/v0-archive-format.svg

Fig. 9 Box and arrow diagram of a v0 archive.#

Archive Version 1#

Released in QIIME 2 version 2.0.6, commit bdc8aed, Version 1 Archives introduce decentralized provenance tracking to QIIME 2. ArchiveFormat V1 inherits all traits of v0, modifying its __init__() and write() methods only to add provenance capture.

Note

All ArchiveFormat versions subclass their predecessor. For example, the ArchiveFormat in v1.py inherits from the ArchiveFormat in v0.py, etc. This makes it easier for humans to interpret the version history.

Provenance data is stored in the directory /<UUID>/provenance/. Specifically, metadata.yaml, action.yaml and VERSION files are captured for the current Result and each of its ancestors. Each Result’s action.yaml file and associated data artifacts (e.g. sample metadata) are stored in an action directory alongside that Result’s VERSION and metadata.yaml. Considered together, we can describe these as “provenance files”. This structure is illustrated in Fig. 10.

../../_images/v1-archive-provenance-format.svg

Fig. 10 Box and arrow diagram of the provenance files in a v1 archive. The blue “multiple files” icon represents all provenance files associated with a single action (e.g. the current action, or one of its ancestors).#

The structure of V1 Archives as a whole is illustrated in Fig. 11.

../../_images/v1-v4-archive-format.svg

Fig. 11 Box and arrow diagram of a v1 archive.#

Provenance files for the current Result are stored in /<UUID>/provenance/. Provenance files for each ancestor Result are stored in directory at /<root_UUID>/provenance/artifacts/<ancestor_UUID>/.

Note

V0 Archives do not capture provenance data. As a result, if a V0 artifact is an ancestor to a V1 (or greater) artifact, it is possible for the action.yaml to list Artifact UUIDs which are not present in the artifacts directory.

Archive Version 2#

Released across QIIME 2 versions 2017.9 (changelog) and 2017.10 (changelog), the directory structure of this format is identical to v1, but the action.yaml file has changed.

In commit 4389a0b the Version 2 ArchiveFormat adds an output-name key to the action section of action.yaml (unless the action type is import), assigning it the output name registered to the relevant action. Prior to this change, if one action returned multiple artifacts of the same artifact class, it was not possible to differentiate between them using provenance alone.

In commit e072706, it adds provenance support for Pipelines, adding the alias-of key to the action section of action.yaml.

Archive Version 3#

Released in QIIME 2 version 2017.12 (changelog), commit 684b8b7, the directory structure of this format is identical to v1 and v2.

With this release, QIIME 2 Actions are able to take variadic arguments, allowing users to pass collections of Artifacts (Lists and Sets). A YAML representer has been added so that action.yaml can represent Sets of Artifact inputs. These will show up in action.yaml as custom !set tags.

Archive Version 4#

Released in QIIME 2 version 2018.4 (changelog), commit 00a294c, this format adds citations to the directory format, adds a transformers section to action.yaml, and aligns the structure of environment:framework (also in action.yaml) to match the structure of environment::plugins::<some_plugin>.

Whenever an Action is run, its registered citations are captured. When saved, they are written to a citations.bib file inside the Archive’s provenance directory. Citations for all of the current Result’s ancestors are stored in their respective directories (e.g. /<root_UUID>/provenance/artifacts/<ancestor_UUID>/citations.bib).

../../_images/v4-archive-provenance-format.svg

Fig. 12 Box and arrow diagram of the provenance files in a v4 archive.#

The overall directory structure remains identical to a v1 archive (Fig. 10).

Result-specific citation tags are also written to the transformers and environment sections of the action.yaml files, for the current Result and for all ancestors with registered citations. A new custom !cite '<citation key>' tag is use to support this in YAML.

A transformers section is added between the action and environment sections of action.yaml. Because Pipelines do not use transformers, transformers will be recorded only for Methods, Visualizers, and when importing data (such as with qiime tools import). It looks like the following:

transformers:
inputs:
    demultiplexed_seqs:
    -   from: SingleLanePerSamplePairedEndFastqDirFmt
        to: SingleLanePerSamplePairedEndFastqDirFmt
output:
-   from: q2_types.feature_data._transformer:DNAIterator
    to: DNASequencesDirectoryFormat
    plugin: !ref 'environment:plugins:types'

environment::framework was previously only a version string, and is now structured identically to each plugin action’s software_entry, with version, website, and citation sections:

framework:
    version: 2019.10.0
    website: https://qiime2.org
    citations:
    - !cite 'framework|qiime2:2019.10.0|0'
plugins:
    fragment-insertion:
        version: 2019.10.0
        website: https://github.com/qiime2/q2-fragment-insertion
        citations:
        - !cite 'plugin|fragment-insertion:2019.10.0|0'
        ...

Archive Version 5#

Released in QIIME 2 version 2018.11 (changelog) commit f95f324, this format version adds archive checksums to the directory structure.

A new, md5sum-formatted checksum file has been added at /<root_UUID>/checksums.md5, with one md5sum and one filename on each line. For a more detailed specification, see the QIIME 2 Pull Request #414.

checksums.md looks like the following:

5a7118c14fd1bacc957ddf01e61491b7  VERSION
333fd63a2b4a102e58e364f37cd98b74  metadata.yaml
4373b96f26689f78889caeb1fbb94090  data/faith_pd-cat1.jsonp

...

7a40cff7855daffa28d4082194bdf60e  provenance/artifacts/f6105891-2c00-4886-b733-6dada99d0c81/metadata.yaml
ae0d0e26da5b84a6c0722148789c51e0  provenance/artifacts/f6105891-2c00-4886-b733-6dada99d0c81/action/action.yaml

Fig. 13 illustrates the V5 Archive.

../../_images/v5-archive-format.svg

Fig. 13 Box and arrow diagram of a v5 archive.#

Archive Version 6#

Released in QIIME 2 version 2023.5 (changelog) commit 85f7111, this version adds execution_context to the execution section of action.yaml and adds support for input, parameter, and output Collections in action.yaml.

The execution_context goes at the bottom of the execution section of action.yaml and gives some details about how the Action that produced this Result was executed. It indicates whether the Action was executed synchronously, asynchronously, or using Parsl. If Parsl was used it indicates what type of Parsl executor was used to run it. If more execution contexts are added in the future, the possibilities here may expand.

execution:
    uuid: 8b1ddcc2-b4e7-4c9f-9338-a7785a47862e
    runtime:
        start: 2025-09-18T11:45:15.273922-07:00
        end: 2025-09-18T11:45:15.659510-07:00
        duration: 385588 microseconds
    execution_context:
        type: parsl/synchronous/asynchronous
        parsl_type: (if type is parsl) <Type of executor used>

Support for input, parameter and output Collections of key: value pairs was also added in v6.

Input Collections go under the inputs section of action.yaml and take the form of a list of yaml key: value pairs where the key is a given Artifact’s key in the Collection and the value is its UUID.

action:
    type: <type>
    plugin: <plugin>
    action: <action>
    inputs:
        - <input collection name>:
            - '<key1>': <uuid1>
            - '<key2>': <uuid2>
            ...

Parameter Collections go under the parameters section of action.yaml and are formatted identically to input Collections only the values will be of whatever type the Parameter Collection is not UUIDs.

action:
    type: <type>
    plugin: <plugin>
    action: <action>
    inputs: <inputs>
    parameters:
        - <parameter collection name>:
            - '<key1>': <value1>
            - '<key2>': <value2>
            ...

If a Result is part of an Output Collection, the output-name section of its action.yaml will contain two additional pieces of information. The key of this Result in the Collection, and the index of this Result in the Collection of the form x/y where x is this Result’s index and y is the size of the Collection.

action:
    type: <type>
    plugin: <plugin>
    action: <action>
    inputs: <inputs>
    parameters: <parameters>
    output-name:
    - <output collection name>
    - <key of this element in the output collection>
    - <index of this element in the collection out of total elements in the collection e.g. 1/100 for the first element in a collection of 100>

Archive Version 7#

Starting with QIIME 2 version 2025.4, QIIME 2 Archive versions 7+ utilize semantic versioning. What this means is each new QIIME 2 Archive version is broken up into a major version and a minor version. Major version updates (i.e. version 6.x -> 7.x) include a significantly different internal structure, such as new files within the zip archive. Backward compatibility across major versions is not guaranteed (such that a version 7.0 archive will not be compatible with older software versions that didn’t yet know about that version). Minor version updates (i.e. 7.n -> 7.n+1) include minor differing internal structure and/or features/capabilities. These will be backwards compatible to the nearest major version (such that a version 7.1 or 7.2 archive will be compatible with other software versions as long as they knew about 7.0).

Archive Version 7.0#

Released in QIIME 2 version 2025.4 (changelog) commit 0d2c8ec, this version adds:

  • support for creation of Annotations (of sub-type Note) to a QIIME 2 Result;

  • an all-new conda-env.yaml file containing all environment dependencies a Result was created from;

  • updated checksum calculations from md5 to sha512;

  • and total file size calculation for all files within the data directory as a new entry under the top level/provenance level metadata.yaml files. All of the new additions to the Archive file structure are outlined below.

Annotations are a way of attaching additional information to a QIIME 2 Result. In version 7.0, Note has been added as a supported Annotation sub-type. Notes can either contain inline text or a filepath whose contents will be written to the Note.

The updated structure of a QIIME 2 Result includes a new top-level annotations directory, where all Annotations will be written. Each Note that is created and written to this directory will have the following structure:

annotations/
    8b1ddcc2-b4e7-4c9f-9338-a7785a47862e/
        checksums.sha512
        metadata.yaml
        note.txt
    217d5eb4-febf-48e8-83ac-1a4dbebe8cf5/
        checksums.sha512
        metadata.yaml
        note.txt

Each sub-directory underneath the annotations directory corresponds to the UUID associated with each Note Annotation that was added to the Result. In each UUID’s subdirectory, a Note will contain the following files:

  • A note.txt file that contains either the inline text or the contents of the filepath provided upon creation of the Note.

  • A metadata.yaml file that contains specific details about the Annotation (which will be laid out in detail below).

  • A checksums.sha512 file with the sha512 calculated checksum for each file within the directory (metadata.yaml and note.txt).

The contents of the metadata.yaml file are as follows:

id: 8b1ddcc2-b4e7-4c9f-9338-a7785a47862e
name: <note-name>
type: Note
created_at: <yyyy-mm-ddThh:mm:ss.ms>
root_result_uuid: d594e658-65fd-4658-8de1-88097ce43abd
referenced_result_uuid: d594e658-65fd-4658-8de1-88097ce43abd

id corresponds to the Annotation’s UUID, the root_result_uuid corresponds to the UUID of the Result the Annotation is attached to, and the referenced_result_uuid corresponds to the UUID of the Result the Annotation is in reference to. Note that the current implementation of Annotations doesn’t support referencing a Result separate from the one the Annotation is attached to, but these separate fields have been created to allow for this functionality to be supported in future Archive versions.

In addition to Annotations, a new conda-env.yaml file (located under the provenance directory) has been added within the Archive structure. This file contains all of the dependencies present in the conda environment where the Result in question was created from. The contents of this file are as follows (with this example showing a very minimal environment):

dependencies:
    - numpy=1.26.4=py310h4bfa8fc_0
    - pandas=2.2.2=py310hbf2a7f0_1
    - python=3.10.14=h00d2728_0_cpython
    - q2-metadata=2025.10.0.dev0+1.g61b847b=py310hd4cd7ad_0
    - q2-mystery-stew=2025.10.0.dev0+1.g25270e3=py310h3cfa9b4_0
    - q2-types=2025.10.0.dev0+5.g4a8aaa7=py310h974e487_0
    - q2cli=2025.10.0.dev0+2.g2976552=py310h68e8629_0
    - q2galaxy=2025.10.0.dev0=py310hac6da46_0
    - q2templates=2025.10.0.dev0+1.g695609d=py310h88e6d8d_0
    - qiime2=2025.10.0.dev0+6.g65a1488=py310hf606c39_0

Each entry matches the pattern: <package-name>=<package-version>=<build-number>.

Archive Version 7.1#

Released in QIIME 2 version 2025.10 (changelog) commit 7d2a9b1, this version adds Signatures, a new Annotation sub-type that provides support for cryptographically signing a QIIME 2 Result.

A new Signature is created by using the fingerprint of an existing GnuPG keypair to cryptographically sign a Result, thus providing a verifiable identity associated with that Result.

The structure of a Signature Annotation within a Result is as follows:

annotations/
    217d5eb4-febf-48e8-83ac-1a4dbebe8cf5/
        checksums.sha512
        metadata.yaml
        signature.gpg

Each Signature (separated by their associated UUID sub-directories) contains the following files:

  • A signature.gpg file that contains the cryptographic signature created with the provided GnuPG keypair.

  • A metadata.yaml file that contains specific details about the Signature (laid out in detail below).

  • A checksums.sha512 file with the sha512 calculated checksum for each file within the directory (metadata.yaml and signature.gpg).

The contents of the metadata.yaml file are as follows:

id: 217d5eb4-febf-48e8-83ac-1a4dbebe8cf5
name: <signature-name>
type: Signature
created_at: <yyyy-mm-ddThh:mm:ss.ms>
root_result_uuid: 046a982d-dbfa-41d1-8596-f767c71f4673
referenced_result_uuid: 046a982d-dbfa-41d1-8596-f767c71f4673
algorithm: <GnuPG keypair algorithm>
checksum_digest: <sha512sum>
signer_name: <signer-name>
signer_email: <signer-email>
fingerprint: <GnuPG keypair fingerprint>

The Signature-specific fields are defined as follows:

  • algorithm refers to the encryption algorithm used when generating the GnuPG keypair used for creating the Signature.

  • checksum_digest refers to the calculated sha512 sum of the root-level checksums.sha512 file, used for Signature verification.

  • signer_name and signer_email refer to the GnuPG uid credentials associated with the keypair used for creating the Signature.

  • fingerprint refers to the fingerprint associated with the GnuPG keypair used for creating the Signature.

More details on Signature creation and verification, along with information on installing and using GnuPG for this purpose, can be found in How to sign and verify Results.

Archive Version 7.2#

Archive Version 7.2 development is currently being planned.