See a typo? Have a suggestion? Edit this page on Github
Back in 2015, there were two proposals made for securing package distribution in Haskell. The Stackage team proposed and implemented a solution using HTTPS and Git, which was then used as the default in Stack. Meanwhile, the Hackage team moved ahead with hackage-security. Over the past few weeks, I've been working on moving Stack over to hackage-security (more on motivation below). The current status of the overall hackage-security roll-out is:
One upside to this is more reliable package index download time. We have had complaints from some firewalled users of slow Git clone time, so this is a good thing. We're still planning on maintaining the Git-based package indices for people using them (to my knowledge they are still being used by Nix, and all-cabal-metadata is still used to power a lot of the information on stackage.org).
However, there's one significant downside I've encountered in the current implementation that I want to discuss.
Quick summary of how hackage-security works: there is a
file, the contents of which I'll discuss momentarily. This is the file
which is downloaded by Stack/cabal-install when you "update your
index." It is signed by a cryptographic algorithm specified within the
hackage-security project, and whenever a client does an update, it
must verify the signature. In theory, when that signature is verified,
we know that the contents of the
01-index.tar file are unmodified.
Within this file are two (relevant) kinds of files: the
for every upload to Hackage (including revisions), and
containing metadata about the package tarballs
themselves. Importantly, this includes a SHA256 checksum and the size
of the tarball. Using these already-validated-to-be-correct JSON
files, we can download and verify a package tarball, even over an
The alternative Git-based approach that the Stackage team proposed has an almost-identical JSON file concept in the all-cabal-hashes repo. Originally, these were generated by downloading tarballs from https://hackage.haskell.org (note the HTTPS). However, a number of months back it became known that the connection between the CDN in front of Hackage and Hackage itself was not TLS-secured, and therefore reliance on HTTPS was not possible. We now rely on the JSON files provided by hackage-security to generate the JSON files used in the Git repo.
With that background, the bug is easy to describe: sometimes the
.json files are missing from the
01-index.tar file. This was
originally opened in April 2016
(for Americans: on tax day no less), and then
I rediscovered the issue three weeks ago
when working on Stack.
Over the weekend, another
.json file went missing, resulting in
the FP Complete mirror not receiving updates
manually updated the list of missing index files.
Due to the inability to securely generate the
.json file in the
all-cabal-hashes Git repo without the file existing upstream, that
file is now missing in
all-cabal-hashes, causing downstream issues
to the Nix team.
There are a number of outcomes to be aware of from this issue:
all-cabal-hashes. I can't speak to the Nix team internal processes, and cannot therefore assess how big an impact that is.
Overall, I'm still very happy that we've moved Stack over to hackage-security: