Using Software Heritage for technical writing

Software Heritage actively archives software from several sources such as…​

The service will periodically capture a snapshot of the same software project (which we have access to, among other things as we’ll see later in the post).

Furthermore, it can save source code that is managed by different version control software such as Git, Mercurial, and Subversion. Software Heritage is also offering its services with its API over HTTPS which is nice if you want to create some neat little scripts or integrate it in your software. Most of the functionality are already available with its website which I covered in a later section. But first, we’ll have to learn an important concept with Software Heritage: its identifier system to access the archive in the first place.

Dialog on archive statistics

As of 2023-05-11, Software Heritage contains at least 230 million projects with 3.2 billion commits. It is only expected to go up throughout the years. GitHub takes the majority of the sources as the archive have 175 million projects from it.

Does that include forks and everything?

Also, that’s a bit scary to think about GitHub having 75% of the archive. What about the second largest?

I’m fairly sure it does include forks in the archive which makes the actual number of projects a lot less but it is still an impressive count considering the wide coverage of sources they monitor. They only archive public repos or if the user opt in of the archival integration such as in GitHub.

As for the second largest source, it seems to come from GitLab instances totalling…​ 4 million projects. That’s at least 1%.

Well, that’s at least…​ unfortunate.

The main intention of the project is to provide a centralized archive for identifying and referencing software. The primary way of using such service is with an identifier system like DOI for digital objects, ISBN for books, and ISWC for music. Software Heritage uses its own identifier system called SoftWare Heritage persistent IDentifier or SWHID for short. The identifier follows a certain format.

The following examples should be enough to show what they look like.

As you can tell from the table, SWHID also offers some control of granularity of what parts of software we want to refer: from individual files and directories, from a certain point in the history of the project, and from a certain point of time of capture.

What about pointing to specific lines of code? Didn’t the preamble mentioned something like "granularity down to the lines of code"?

You’ll see it later.

An interesting property with SWHIDs is that they are intrinsic identifiers: meaning you get the object alongside the identifier. Unlike DOIs and ISBNs where objects are arbitrarily assigned by a central authority, SWHIDs are computationally generated from the object. This means SWHIDs are deterministic and we can do a reverse lookup with the object. In fact, it can be computed with objects locally in your machine.

Due to its intrinsic nature of SWHIDs and with the ability to refer various parts of a software, we’re also slowly unraveling the fact that Software Heritage archive itself is essentially a gigantic Merkle tree where it contains several objects. Let’s go back to the previous table of SWHIDs again and see what those are.

Similarities with Git

Wait…​ this sounds similar to the Git internals.

That’s because it is using a similar data model as Git with the graph objects and even the object identifier being a hex-encoded SHA1 hash.

Does this mean it is compatible with Git then?

Yes but it is more coincidental than anything. This is especially clearer once you noticed the service supports importing software from version control software other than Git. Just don’t expect that to work every time.

While the previously shown SWHIDs is enough and working as intended, there are some lack of information with the identifier alone. From the identifier system, one cannot easily infer certain information that we often needed such as the URL of the repository and the path relative to the repository. This is also reflected in the website interface if you’ve visited the links where it just strictly presents the software artifact (e.g., content, directory, revision).

This is because of the data model of the archive being a gigantic Merkle tree where objects may be shared among multiple projects. This makes certain tasks to be tedious such as identifying whether the artifact belong from a canonical repository or one of its many forks which is also included in the archive.

Because of this, SWHIDs may also have a semicolon-delimited (;) list of qualifiers that adds contextual information.

Each qualifier may mean different things which is documented nicely in its website. Let’s take the previous table and add more contextual information with it.

If you click each of the link, the website interface is more complete compared to the previous table of SWHIDs.

Side-to-side comparison of the website interface for swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2 without and with qualifiers

This practice of adding contextual information is recommended as documented from its FAQ. More specifically, the contextual information has to be as full as possible which you can easily get the identifier with all relevant qualifiers in its archive website interface which we’ll cover next. You can see more of them from Guidelines for referencing SWHIDs.

So what happens when I give a qualifier with a wrong value such as the origin qualifier that points to an non-existent origin or an anchor qualifier that points to an invalid SWHID?

Why don’t you try those out yourself? Here’s a list of them just for starters.

• A swh:cnt: with invalid origin

• A swh:dir: with invalid anchor

• A swh:cnt: with invalid path

You could also mix and match qualifiers that are not supposed to appear in certain object types such as the lines qualifier in non-content objects.

Throughout the Software Heritage ecosystem, there are tools that make use of the service. Its main interface is on the archive website interface is what you’re likely to use the most. The workflow from the website interface is pretty simple: you search for the origin of the software, enter the corresponding object, and specify what you want to refer to. ^[2]

The most important thing to note with this website is using it as a resolver for SWHIDs that is similarly used with DOIs. You’ve already seen its usage with the links from the previous tables such as in Examples of SWHIDs and Previous SWHIDs with contextual information. Using it as a resolver is simple: just append the identifier on the root endpoint of the service.

https://archive.softwarearchive.org/$SWHID

With the user-facing side of the website, what you’ll see first is a search interface. Take note the quality of the search results is not perfect nor usable if you’re not aware of the quirks of its search engine. For example, merely entering the name of the software is not typically enough for searching.

The search result for the query "linux kernel"

Even searching with metadata doesn’t help.

The search result for the query "linux kernel" with metadata

It’s pretty obvious that it doesn’t have enough quality results. Instead, I recommend to enter the origin URL that you’re searching for (e.g., https://github.com/torvalds/linux). If there is an exact match of the given origin, the website will directly go to the page of the software artifact with that origin. This is especially nice for the sources it already monitors such as GitHub, GitLab instances, and Gitea instances. This even works for package indices such as Pypi and npm (e.g., https://pypi.org/project/swh.core, https://www.npmjs.com/package/vue). For more details, there is a dedicated page on what sources are being monitored which you can infer what URLs can be resolved in this way.

Once you get into the software artifact of your choosing (e.g., directory, file, revision, snapshot), you can get the identifier with the permalink tab on the side of the website.

Other than the website, there are tools available to easily make use of the service. The ecosystem of Software Heritage is somewhat limiting as Software Heritage itself is relatively young but it does have nice tools to begin with. Let’s take a closer look at them.

Furthermore, there are initiatives to integrate it with projects such as with Guix and peer-to-peer access with IPFS.

While using SWHIDs is a done-and-forget procedure (for the most part), there is a set of guidelines to make usage of them a bit easier.

Linking SWHIDs could be tedious when writing documents. In Asciidoctor, there are features where this makes it easier. Specifically, we’re talking about storing the identifiers in document attributes.

:swh-system76-firmware-license-core-identifier: swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2
:swh-system76-firmware-license: swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2;origin=https://github.com/pop-os/system76-firmware;lines=471-538

link:https://archive.softwarearchive.org/{swh-system76-firmware-license}[{swh-system76-firmware-license-core-identifier}]

In my opinion, this is still tedious since we have to store two attributes that would need separate changes where it should be only one. Fortunately, Asciidoctor can be extended to introduce new syntax which I’ve previously shown how Asciidoctor can be extended. We can apply a similar solution here.

For our initial version of the new syntax, it looks like the following.

swh:$SWHID[$CAPTION]

It is an inline macro that accepts an SWHID and can accept a caption as the link text. Take note the caption is optional with the core identifier being the default caption. The following listing should show a complete list of use cases we considered for this macro.

// Should produce a link at https://archive.softwareheritage.org/$SWHID with
// '$SWHID_CORE_IDENTIFIER' as the link text.
swh:swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2;origin=https://github.com/pop-os/system76-firmware;lines=471-538[]

// Similar as above but with the link text replaced with 'replacing the caption'.
swh:swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2;origin=https://github.com/pop-os/system76-firmware;lines=471-538[replacing the caption]

// For aesthetic purposes, you could also use the `swh` macro with the `swh:`
// cut off from the SWHID.
swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2;origin=https://github.com/pop-os/system76-firmware;lines=471-538[]

The inline macro should produce a link target to the default SWHID resolver at https://archive.softwareheritage.org. Anyways, here’s the code for the swh Asciidoctor extension.

# frozen_string_literal: true

class SWHInlineMacro < Asciidoctor::Extensions::InlineMacroProcessor
  use_dsl

  named :swh
  name_positional_attributes 'caption'

  def process(parent, target, attrs)
    doc = parent.document

    # We're only considering `swh:` starting with the scheme version. Also, it
    # looks nice aesthetically.
    swhid = target.start_with?('swh:') ? target : %(swh:#{target})
    swhid_core_identifier = (swhid.split ';').at 0

    text = attrs['caption'] || swhid_core_identifier
    target = %(https://archive.softwareheritage.org/#{swhid})

    doc.register :links, target
    create_anchor parent, text, type: :link, target: target
  end
end

You cannot make use of the extension as it is not registered within the Asciidoctor registry yet. Let’s make the file that does that.

# frozen_string_literal: true

require 'asciidoctor'
require 'asciidoctor/extensions'

require_relative './asciidoctor/custom_extensions/swhid_link_inline_macro'

Asciidoctor::Extensions.register do
  inline_macro SWHInlineMacro
end

Now with the extension in place, you can use it with Asciidoctor like with the following listing.

asciidoctor -r ./lib/asciidoctor-custom-extensions.rb sample.adoc

Voila! Now you have an nicer way of linking them SWHIDs with the archive. This extension should be usable for all backends since it is a simple shorthand for linking SWHIDs to the archive.