I created my first NixOS package a couple of months ago. The package is called Terraspace. I needed it for some work-related stuff, and I didn't find it in the nixpkgs repo, so I took the opportunity to become a nixpkgs contributor. I had some problems while creating it that I had to debug alone, so I'll recreate the process of packaging it to help anyone stuck on a similar set of problems. I'll also write a cookbook that I use when creating a nix package.
The great thing about nix and NixOS is the ability to reproducibly package niche programs for your system, and have that config stored in a VCS, and I think that all nix users should be familiar with it without needing to learn how to breathe nix. It took me more than a year to grasp how powerful this is. I'm writing this post in the hope that it will help a novice nix user with things I had to figure out through troubleshooting.
The cookbook
- Look up a similar package in the nixpkgs repo! This will save you lots of time, and will give you a great starting point.
- Search the NixOS wiki. This can give you some useful points, but it can also fail you epically (as you will see later).
Most of the time, the source of the package you want to build will be on GitHub.
These kind of packages are built by fetching a wanted release fron GitHub and then building it (probably by following the instructions found in the README.md).
The exeptions to the above instruction are the pip
, gem
, etc. packages.
You can build those following the above instructions, or by using nix built-in modules for those kinds of packages.
Building Terraspace
Terraspace is a ruby package, so googling "nixos create ruby package" yields this, so I decided to follow it.
Following the NixOS wiki
The wiki gives us a shell.nix
:
with import <nixpkgs> {};
stdenv.mkDerivation {
name = "env";
buildInputs = [
ruby.devEnv
git
sqlite
libpcap
postgresql
libxml2
libxslt
pkg-config
bundix
gnumake
];
}
and a couble of commands:
$ nix-shell
$ bundle install # generates Gemfile.lock
$ bundix # generates gemset.nix
so let's create a Gemfile
:
source "https://rubygems.org"
gem "terraspace", '~> 2.2.7'
and do what the wiki tells us to.
Running bundle install
and bundix
generates a Gemfile.lock
and gemset.nix
files respectively.
[nix-shell:~/.local/dev/nix-tinkering/terraspace]$ ls
Gemfile Gemfile.lock gemset.nix shell.nix
Next, we create the default.nix
file.
Luckily, the wiki gives us a pretty good start, and, with some adjustments, we get:
{ stdenv, bundlerEnv, ruby }:
let
gems = bundlerEnv {
name = "terraspace-env";
inherit ruby;
gemdir = ./.;
};
in stdenv.mkDerivation {
name = "terraspace";
src = ./.;
buildInputs = [gems ruby];
installPhase = ''
mkdir -p $out/{bin,share/terraspace}
cp -r * $out/share/terraspace
bin=$out/bin/terraspace
cat > $bin <<EOF
#!/bin/sh -e
exec ${gems}/bin/bundle exec ${ruby}/bin/ruby $out/share/terraspace/terraspace "\$@"
EOF
chmod +x $bin
'';
}
NixOS wiki failing us
Running the nix-build -E '((import <nixpkgs> {}).callPackage (import ./default.nix) { })'
we get this:
[nix-shell:~/.local/dev/nix-tinkering/terraspace]$ nix-build -E '((import <nixpkgs> {}).callPackage (import ./default.nix) { })'
...
building '/nix/store/zw4qcxiiz4zx35w877mm3pnl93fknrks-nokogiri-1.15.2.gem.drv'...
trying https://rubygems.org/gems/nokogiri-1.15.2.gem
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4501k 100 4501k 0 0 1522k 0 0:00:02 0:00:02 --:--:-- 1522k
error: hash mismatch in fixed-output derivation '/nix/store/zw4qcxiiz4zx35w877mm3pnl93fknrks-nokogiri-1.15.2.gem.drv':
specified: sha256-W8ppa5KDrXzpe5wN/fAppiwm6S859ECmV5Xjd9RPEZo=
got: sha256-INyAC4++TE9LWxZOaqOrgqNxvLJ+toXBZpYcNN2KItc=
error: 1 dependencies of derivation '/nix/store/i507v7y7qfn5bsvw7l29zq99z7c4mhz9-ruby2.7.7-nokogiri-1.15.2.drv' failed to build
error: 1 dependencies of derivation '/nix/store/fl4db3lr6v8vks68j599nqfhd3cyhhlk-terraspace-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/lkfx6q1jyv6wzsp24hxdyrcchf25bvcj-terraspace.drv' failed to build
This errors tend to show up.
Nothing big, we just need to replace the sha
in the gemset.nix
file.
After replacing it we get:
[nix-shell:~/.local/dev/nix-tinkering/terraspace]$ nix-build -E '((import <nixpkgs> {}).callPackage (import ./default.nix) { })'
...
error: builder for '/nix/store/3acagl668bb5bg8dpmam63xkvm782g61-ruby2.7.7-nokogiri-1.15.2.drv' failed with exit code 1;
last 10 log lines:
> from extconf.rb:1034:in `<main>'
>
> To see why this extension failed to compile, please check the mkmf.log which can be found here:
>
> /nix/store/ygjqya5igj2d77ik6p6bsfdw2asprr6b-ruby2.7.7-nokogiri-1.15.2/lib/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/nokogiri-1.15.2/mkmf.log
>
> extconf failed, exit code 1
>
> Gem files will remain installed in /nix/store/ygjqya5igj2d77ik6p6bsfdw2asprr6b-ruby2.7.7-nokogiri-1.15.2/lib/ruby/gems/2.7.0/gems/nokogiri-1.15.2 for inspection.
> Results logged to /nix/store/ygjqya5igj2d77ik6p6bsfdw2asprr6b-ruby2.7.7-nokogiri-1.15.2/lib/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/nokogiri-1.15.2/gem_make.out
For full logs, run 'nix log /nix/store/3acagl668bb5bg8dpmam63xkvm782g61-ruby2.7.7-nokogiri-1.15.2.drv'.
error: 1 dependencies of derivation '/nix/store/d1clnrg5mfzsi79v9kl264cxg0a5qcmg-terraspace-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/d7snzgwfizvix2y5ql4pygccjm4n5nr4-terraspace.drv' failed to build
Seems like we have a problem with nokogiri...
Looking at Gemfile.lock
we see:
nokogiri (1.15.2-x86_64-linux)
racc (~> 1.4)
racc (1.7.1)
Googling "nokogiri" gets us to the rubygems site.
Here we find an error.
It seems like bundle install
didn't add mini_portile2
as a nokogiri dependency.
Also, it has 1.15.2-x86_64-linux
as a version, which seems weird, let's fix this.
This is what the nokogiri section of the Gemfile.lock
should look like:
nokogiri (1.15.2)
racc (~> 1.4)
mini_portile2 (~> 2.8.2)
racc (1.7.1)
mini_portile2 (2.8.2)
We remove the old gemset.nix
file and run bundix
and try to build terraspace again and get:
[nix-shell:~/.local/dev/nix-tinkering/terraspace]$ nix-build -E '((import <nixpkgs> {}).callPackage (import ./default.nix) { })'
...
/nix/store/vlpslz0zpqgdn9yp019vva1jgr0rlky8-terraspace
Success! Let's run it!
[nix-shell:~/.local/dev/nix-tinkering/terraspace]$ result/bin/terraspace -v
Traceback (most recent call last):
/nix/store/yy9sbr2sd4qfn5fdygcqkmibscbcknhq-ruby-2.7.7/bin/ruby: No such file or directory -- /nix/store/vlpslz0zpqgdn9yp019vva1jgr0rlky8-terraspace/share/terraspace/terraspace (LoadError)
Ehh... Seems like there's something wrong with our installPhase
script.
The best way to fix this (in my opinion) is to find a simple enough script in the nixpkgs repo, but since this is an analysis of a packaging process, we'll try to fix this manually.
ℹ️
This
is a good example of the same thing we are trying to do.
It uses a makeWrapper
function defined here which, among other things, does what we are about to do.
We would use makeWrapper
like this:
mkdir -p $out/bin
makeWrapper ${rubyEnv}/bin/terraspace $out/bin/terraspace
and this would make our build work.
The light at the end of a tunnel
Taking a look at the generated result, we see this:
[nix-shell:~/.local/dev/nix-tinkering/terraspace/result/bin]$ cat terraspace
#!/nix/store/96ky1zdkpq871h2dlk198fz0zvklr1dr-bash-5.1-p16/bin/sh -e
exec /nix/store/l9l60bs44jgn59gya59pip2h4rbln66g-terraspace-env/bin/bundle exec /nix/store/yy9sbr2sd4qfn5fdygcqkmibscbcknhq-ruby-2.7.7/bin/ruby /nix/store/vlpslz0zpqgdn9yp019vva1jgr0rlky8-terraspace/share/terraspace/terraspace "$@"
There are three different nix-store paths here: /nix/store/l9l60bs44jgn59gya59pip2h4rbln66g-terraspace-env
, /nix/store/yy9sbr2sd4qfn5fdygcqkmibscbcknhq-ruby-2.7.7
and /nix/store/vlpslz0zpqgdn9yp019vva1jgr0rlky8-terraspace
.
The second one is a nix-store path for ruby 2.7.7, and the last one is the one we got running nix-build
.
Taking a look at /nix/store/l9l60bs44jgn59gya59pip2h4rbln66g-terraspace-env
, we see:
[nix-shell:/nix/store/l9l60bs44jgn59gya59pip2h4rbln66g-terraspace-env]$ ls
bin lib
[nix-shell:/nix/store/l9l60bs44jgn59gya59pip2h4rbln66g-terraspace-env/bin]$ ls -la | grep terraspace
-r-xr-xr-x 2 root root 1242 Jan 1 1970 terraspace
-r-xr-xr-x 2 root root 1266 Jan 1 1970 terraspace-bundler
[nix-shell:/nix/store/l9l60bs44jgn59gya59pip2h4rbln66g-terraspace-env/bin]$ ./terraspace --version
2.2.7
This suggests that our terraspace binary was built in the bundlerEnv
part of the build script, and we just have to expose it in the installPhase
:
mkdir -p $out/bin
bin=$out/bin/terraspace
cat > $bin <<EOF
#!/bin/sh -e
exec ${gems}/bin/terraspace "\$@"
EOF
chmod +x $bin
Building it with the new installPhase
we get:
[nix-shell:~/.local/dev/nix-tinkering/terraspace]$ nix-build -E '((import <nixpkgs> {}).callPackage (import ./default.nix) { })'
...
/nix/store/4xrb026k46ql5zxqbfxm753szmclc8qa-terraspace
[nix-shell:~/.local/dev/nix-tinkering/terraspace]$ result/bin/terraspace -v
2.2.7
And the thing works this time.
Final remarks
I hope this post was helpful to someone stuck with building a nix package. I tried to illustrate some problems I encountered while building terraspace and how I solved them. The nixpkgs repo is a huge help when writing a package, don't be afraid to look packages up in it.