[logseq-plugin-git:commit] 2025-12-11T14:50:38.767Z
This commit is contained in:
@@ -1,96 +0,0 @@
|
||||
---
|
||||
created: 2024-01-17T10:28:08 (UTC -05:00)
|
||||
tags: [nixos]
|
||||
source: https://johns.codes/blog/efficient-nix-derivations-with-file-sets
|
||||
author: John Murray
|
||||
---
|
||||
|
||||
- Efficient Nix Derivations with File Sets
|
||||
|
||||
> ## Excerpt
|
||||
> Using nix's new file set API to make efficient derivations
|
||||
|
||||
---
|
||||
Table of Contents
|
||||
- [What are file sets](https://johns.codes/blog/efficient-nix-derivations-with-file-sets#what-are-file-sets)
|
||||
- [A Real Example](https://johns.codes/blog/efficient-nix-derivations-with-file-sets#a-real-example)
|
||||
- [Wrap up](https://johns.codes/blog/efficient-nix-derivations-with-file-sets#wrap-up)
|
||||
|
||||
If you are using Nix to build your own packages you will eventually come across something like
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>stdenv</span><span>.</span><span>mkDerivation</span><span> {</span></code></p><p><code><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>"my awesome pkg"</span><span>;</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span>buildPhase</span><span> </span><span>=</span><span> </span><span>''</span></code></p><p><code><span> # I have no idea if this actually works (the gcc call bit)</span></code></p><p><code><span> # but the vibe is what I care about</span></code></p><p><code><span> gcc main.c -o my_program</span></code></p><p><code><span> mkdir -p $out</span></code></p><p><code><span> cp my_program $out</span></code></p><p><code><span> ''</span><span>;</span></code></p><p><code><span>}</span></code></p><p></p>
|
||||
```
|
||||
|
||||
The issue with the above is setting `src = ./.`, which makes **ALL** of the current directory an input to the derivation. So if you have a readme file in this folder, changing that will cause this derivation to rebuild.
|
||||
|
||||
The build only really cares about `main.c` in this case so what can we do to fix this?
|
||||
|
||||
Back in the dark dark times of pre-Nixos 23.11, there were ways to do this kind of filtering but IMO they were kinda confusing and I never quite got it to work right so I just stuck with `src = ./.`
|
||||
|
||||
Now with 23.11 we have [filesets](https://www.tweag.io/blog/2023-11-28-file-sets/), which makes filtering and adding files much simpler.
|
||||
- What are file sets
|
||||
|
||||
I would recommend checking out the [docs](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset) or [official tutorial](https://nix.dev/tutorials/file-sets), but for a TLDR, you can do the following
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>let</span></code></p><p><code><span> </span><span>fs</span><span> </span><span>=</span><span> </span><span>pkgs</span><span>.</span><span>lib</span><span>.</span><span>fileset</span><span>;</span></code></p><p><code><span> </span><span>baseSrc</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>unions</span><span> [ </span><span>./Makefile</span><span> </span><span>./src</span><span> ];</span></code></p><p><code><span> </span><span>filterMarkdownFiles</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>fileFilter</span><span> (</span><span>file</span><span>: </span><span>hasSuffix</span><span> </span><span>".md"</span><span> </span><span>file</span><span>.</span><span>name</span><span>) </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span>removedMarkedDown</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>difference</span><span> </span><span>baseSrc</span><span> </span><span>filterMarkdownFiles</span><span>;</span></code></p><p><code><span>in</span></code></p><p><code><span>stdenv</span><span>.</span><span>mkDerivation</span><span> {</span></code></p><p><code><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>"my awesome pkg"</span><span>;</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>toSource</span><span> {</span></code></p><p><code><span> </span><span>root</span><span> </span><span>=</span><span> </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span>fileset</span><span> </span><span>=</span><span> </span><span>removedMarkedDown</span><span>;</span></code></p><p><code><span> };</span></code></p><p><code><span> </span><span>buildPhase</span><span> </span><span>=</span><span> </span><span>''</span></code></p><p><code><span> # call make or w/e you want</span></code></p><p><code><span> ''</span><span>;</span></code></p><p><code><span>}</span></code></p><p></p>
|
||||
```
|
||||
|
||||
Now with this setup, we have a "base" file set of `[ ./Makefile ./src ]` then with `fs.difference` we can remove all files that are markdown with the `filterMarkdownFiles` filter.
|
||||
- A Real Example
|
||||
|
||||
Recently I started messing around with the language [Roc](https://www.roc-lang.org/). If you haven't heard of it, Roc is a new functional language heavily inspired by Elm. It's fast but also very nice to use (though some rough edges since it's pre-0.1.0).
|
||||
|
||||
One of its interesting ideas is that Roc needs your app to pick what [platform](https://www.roc-lang.org/platforms) to run on. A platform would be written in something like rust, zig, c, etc. The platform provides roc APIs for things like managing memory, making network requests, printing to stdout, and other IO-like actions. Right now the two most widely used are a [cli platform](https://github.com/roc-lang/basic-cli) and [webserver platform](https://github.com/roc-lang/basic-webserver)
|
||||
|
||||
This is really neat but brings an issue for developing a platform along with the roc code needed to define the platform API. You need to compile the "platform code" (ie rust + some c), do w/e linking is needed for that, then distribute that with the roc source code. The roc cli will do this for you when developing but it doesn't work as nicely to compile just the platform with nix.
|
||||
|
||||
For example, [this](https://github.com/roc-lang/roc/tree/main/examples/platform-switching/rust-platform) is one of the sample platforms
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>❯ exa --tree --level 2</span></code></p><p><code><span>.</span></code></p><p><code><span>├── Cargo.lock</span></code></p><p><code><span>├── Cargo.toml</span></code></p><p><code><span>├── host.c</span></code></p><p><code><span>├── main.roc</span></code></p><p><code><span>├── rust-toolchain.toml</span></code></p><p><code><span>└── src</span></code></p><p><code><span> ├── glue.rs</span></code></p><p><code><span> ├── lib.rs</span></code></p><p><code><span> └── main.rs</span></code></p><p></p>
|
||||
```
|
||||
|
||||
To build this platform you need to run a `cargo build --lib ...` on the rust code, compile the `host.c` file, link those two object files together, and then finally distribute the linked object file with the roc code.
|
||||
|
||||
so after all those steps, it should look something like
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>❯ exa --tree </span><span><</span><span>compiled folder</span><span>></span></code></p><p><code><span><</span><span>compiled folder</span><span>></span></code></p><p><code><span>├── linux-x64.o</span></code></p><p><code><span>└── main.roc</span></code></p><p></p>
|
||||
```
|
||||
|
||||
The naive approach would probably be something like
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>let</span></code></p><p><code><span>compiledC</span><span> </span><span>=</span><span> </span><span>mkDerivation</span><span> {</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span># compile the c ...</span></code></p><p><code><span>};</span></code></p><p><code><span>rustBuiltLib</span><span> </span><span>=</span><span> </span><span>buildRustPackage</span><span> {</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span># build the rust</span></code></p><p><code><span>};</span></code></p><p><code><span>in</span></code></p><p><code><span>llvmPkgs</span><span>.</span><span>stdenv</span><span>.</span><span>mkDerivation</span><span> </span><span>rec</span><span> {</span></code></p><p><code><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>"</span><span>${</span><span>pname</span><span>}</span><span>-</span><span>${</span><span>version</span><span>}</span><span>"</span><span>;</span></code></p><p><code><span> </span><span>srcs</span><span> </span><span>=</span><span> [</span></code></p><p><code><span> </span><span>rustBuiltLib</span></code></p><p><code><span> </span><span>compiledC</span></code></p><p><code><span> </span><span>./.</span><span> </span><span># for the roc code</span></code></p><p><code><span> ];</span></code></p><p><code><span> </span><span>sourceRoot</span><span> </span><span>=</span><span> </span><span>"."</span><span>;</span></code></p><p><code><span> </span><span>buildPhase</span><span> </span><span>=</span><span> </span><span>''</span></code></p><p><code><span> # link the rust and c files</span></code></p><p><code><span> # copy roc and linked object out</span></code></p><p><code><span> ''</span><span>;</span></code></p><p><code><span>}</span></code></p><p></p>
|
||||
```
|
||||
|
||||
while this works, it also sucks. **ANY** change to the files will cause all 3 derivations to be rebuilt.
|
||||
|
||||
Now with filesets we can be all cute and fancy
|
||||
|
||||
This example will have many parts omitted to keep the code easy to follow. To see the real code, look at my [roc2nix repo](https://github.com/JRMurr/roc2nix/blob/main/lib/platformBuilders/buildRustPlatform.nix) where this came from
|
||||
|
||||
First let's define a helper file for filtering files based on their extension
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>{</span><span>lib</span><span>}:</span></code></p><p><code><span># Note i generally don't like doing `with` at the top of a file</span></code></p><p><code><span># but since this will be only fileSet stuff it should be fine</span></code></p><p><code><span>with</span><span> </span><span>lib</span><span>.</span><span>fileset</span><span>;</span></code></p><p><code><span>let</span></code></p><p><code><span> </span><span># helper func to take in a list of allowed</span></code></p><p><code><span> </span><span># returns a function of `file => bool` to be used in a fileFilter.</span></code></p><p><code><span> </span><span># true if file has suffix, false if not</span></code></p><p><code><span> </span><span>fileHasAnySuffix</span><span> </span><span>=</span><span> </span><span>fileSuffixes</span><span>: </span><span>file</span><span>: (</span><span>lib</span><span>.</span><span>lists</span><span>.</span><span>any</span><span> (</span><span>s</span><span>: </span><span>lib</span><span>.</span><span>hasSuffix</span><span> </span><span>s</span><span> </span><span>file</span><span>.</span><span>name</span><span>) </span><span>fileSuffixes</span><span>);</span></code></p><p><code><span> </span><span># given a basePath src path, return a fileset of files in that path that are rust files, toml files, or cargo toml/lock</span></code></p><p><code><span> </span><span>rustFilter</span><span> </span><span>=</span><span> </span><span>basePath</span><span>: (</span></code></p><p><code><span> </span><span>let</span></code></p><p><code><span> </span><span>mainFilter</span><span> </span><span>=</span><span> </span><span>fileFilter</span></code></p><p><code><span> (</span><span>fileHasAnySuffix</span><span> [ </span><span>".rs"</span><span> </span><span>".toml"</span><span> ])</span></code></p><p><code><span> </span><span>basePath</span><span>;</span></code></p><p><code><span> </span><span>in</span></code></p><p><code><span> </span><span>unions</span><span> [ </span><span>mainFilter</span><span> (</span><span>basePath</span><span> </span><span>+</span><span> </span><span>"/Cargo.toml"</span><span>) (</span><span>basePath</span><span> </span><span>+</span><span> </span><span>"/Cargo.lock"</span><span>) ]</span></code></p><p><code><span> );</span></code></p><p><code><span> </span><span># given a basePath src path return a fileset with files ending with `.c`</span></code></p><p><code><span> </span><span>cFilter</span><span> </span><span>=</span><span> </span><span>basePath</span><span>: </span><span>fileFilter</span><span> (</span><span>fileHasAnySuffix</span><span> [ </span><span>".c"</span><span> ]) </span><span>basePath</span><span>;</span></code></p><p><code><span> </span><span># given a basePath src path return a fileset with files ending with `.roc`</span></code></p><p><code><span> </span><span>rocFilter</span><span> </span><span>=</span><span> </span><span>basePath</span><span>: </span><span>fileFilter</span><span> (</span><span>fileHasAnySuffix</span><span> [ </span><span>".roc"</span><span> ]) </span><span>basePath</span><span>;</span></code></p><p><code><span>in</span></code></p><p><code><span>{</span></code></p><p><code><span> </span><span>inherit</span><span> </span><span>rustFilter</span><span> </span><span>cFilter</span><span> </span><span>rocFilter</span><span>;</span></code></p><p><code><span>}</span></code></p><p></p>
|
||||
```
|
||||
|
||||
Now with that helper, we can do
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>let</span></code></p><p><code><span> </span><span>fs</span><span> </span><span>=</span><span> </span><span>lib</span><span>.</span><span>fileset</span><span>;</span></code></p><p><code><span> </span><span>languageFilters</span><span> </span><span>=</span><span> </span><span>import</span><span> </span><span>./languageFilters.nix</span><span> {</span><span>inherit</span><span> </span><span>lib</span><span>;};</span></code></p><p><code><span> </span><span>baseDir</span><span> </span><span>=</span><span> </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span>compiledC</span><span> </span><span>=</span><span> </span><span>mkDerivation</span><span> {</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>toSource</span><span> {</span></code></p><p><code><span> </span><span>root</span><span> </span><span>=</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> </span><span>fileset</span><span> </span><span>=</span><span> </span><span>languageFilters</span><span>.</span><span>cFilter</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> };</span></code></p><p><code><span> </span><span># compile the c ...</span></code></p><p><code><span> };</span></code></p><p><code><span> </span><span>rustBuiltLib</span><span> </span><span>=</span><span> </span><span>buildRustPackage</span><span> {</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>toSource</span><span> {</span></code></p><p><code><span> </span><span>root</span><span> </span><span>=</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> </span><span>fileset</span><span> </span><span>=</span><span> </span><span>languageFilters</span><span>.</span><span>rustFilter</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> };</span></code></p><p><code><span> </span><span># build the rust</span></code></p><p><code><span> };</span></code></p><p><code><span> </span><span>rocCode</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>toSource</span><span> {</span></code></p><p><code><span> </span><span>root</span><span> </span><span>=</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> </span><span>fileset</span><span> </span><span>=</span><span> </span><span>languageFilters</span><span>.</span><span>rocFilter</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> };</span></code></p><p><code><span>in</span></code></p><p><code><span>llvmPkgs</span><span>.</span><span>stdenv</span><span>.</span><span>mkDerivation</span><span> </span><span>rec</span><span> {</span></code></p><p><code><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>"</span><span>${</span><span>pname</span><span>}</span><span>-</span><span>${</span><span>version</span><span>}</span><span>"</span><span>;</span></code></p><p><code><span> </span><span>srcs</span><span> </span><span>=</span><span> [</span></code></p><p><code><span> </span><span>rustBuiltLib</span></code></p><p><code><span> </span><span>compiledC</span></code></p><p><code><span> </span><span>rocCode</span></code></p><p><code><span> ];</span></code></p><p><code><span> </span><span>sourceRoot</span><span> </span><span>=</span><span> </span><span>"."</span><span>;</span></code></p><p><code><span> </span><span>buildPhase</span><span> </span><span>=</span><span> </span><span>''</span></code></p><p><code><span> # NOTE: this link could be pulled into its own derivation for even better seperation</span></code></p><p><code><span> # I only just realized this while making this post...</span></code></p><p><code><span> $LD -r -L </span><span>${</span><span>rustBuildName</span><span>}</span><span>/lib </span><span>${</span><span>cBuildName</span><span>}</span><span>/</span><span>${</span><span>cHostDest</span><span>}</span><span> -lhost -o </span><span>${</span><span>host_dest</span><span>}</span></code></p><p><code><span> mkdir -p $out</span></code></p><p><code><span> cp </span><span>${</span><span>host_dest</span><span>}</span><span> $out/</span><span>${</span><span>host_dest</span><span>}</span></code></p><p><code><span> cp -r </span><span>${</span><span>rocCode</span><span>}</span><span>/. $out</span></code></p><p><code><span> ''</span><span>;</span></code></p><p><code><span>}</span></code></p><p></p>
|
||||
```
|
||||
|
||||
Now this is about as good as you can get. The rust and c builds have no dependency on each other so you are free to modify the c without needing to rebuild the rust.
|
||||
|
||||
If you only change the roc code, you won't need to do any build (other than linking but in this example that could also be pulled out....).
|
||||
- Wrap up
|
||||
|
||||
Huge shoutout to [Silvan Mosberger](https://github.com/infinisil) from [Tweag](https://www.tweag.io/) for bringing file sets to the main Nix library. He was sponsored through my current employer [Antithesis](https://antithesis.com/) to develop this feature!
|
||||
|
||||
I hope filesets make more people take a stab at making their own `*2nix` builders, or just make their own builds more efficent.
|
||||
|
||||
I had a lot of fun working on [roc2nix](https://github.com/JRMurr/roc2nix/), it was my first time making a "real" nix library and I learned a lot along the way (not just file sets). If you are interested in learning more about it check out the repo or let me know in the comments and I might make a separate blog diving into that (and hopefully some blogs on roc itself).
|
||||
@@ -1,213 +0,0 @@
|
||||
---
|
||||
created: 2023-12-10T12:53:42 (UTC -05:00)
|
||||
tags:
|
||||
- nixos
|
||||
source: https://gist.github.com/misuzu/89fb064a2cc09c6a75dc9833bb3995bf
|
||||
author: misuzu
|
||||
---
|
||||
|
||||
- Install NixOS on Oracle Cloud over Ubuntu 18.04
|
||||
|
||||
> ## Excerpt
|
||||
> Install NixOS on Oracle Cloud over Ubuntu 18.04. GitHub Gist: instantly share code, notes, and snippets.
|
||||
|
||||
---
|
||||
- Install NixOS on Oracle Cloud over Ubuntu 18.04 (make sure to use Ubuntu 18.04 or this may not work)
|
||||
|
||||
```shell
|
||||
# install useful tools
|
||||
sudo apt-get update
|
||||
sudo apt-get install --no-install-recommends -y nano mc git
|
||||
|
||||
# prepare /boot
|
||||
sudo umount /boot/efi
|
||||
sudo mv /boot /boot.bak
|
||||
sudo mkdir /boot/
|
||||
sudo mount /dev/sda15 /boot
|
||||
sudo mv /boot/* /boot.bak/efi/
|
||||
|
||||
# use swap file
|
||||
sudo dd if=/dev/zero of=/swapfile bs=1M count=1024 status=progress
|
||||
sudo chmod 600 /swapfile
|
||||
sudo mkswap /swapfile
|
||||
sudo swapon /swapfile
|
||||
|
||||
# install nix
|
||||
sh <(curl -L https://nixos.org/nix/install)
|
||||
. $HOME/.nix-profile/etc/profile.d/nix.sh
|
||||
nix-channel --add https://nixos.org/channels/nixos-21.11 nixpkgs
|
||||
nix-channel --update
|
||||
|
||||
# install nixos-generate-config and nixos-install
|
||||
nix-env -f '<nixpkgs>' -iA nixos-install-tools
|
||||
|
||||
# generate config
|
||||
sudo `which nixos-generate-config` --root /
|
||||
|
||||
# remove lxc mounts
|
||||
sudo nano /etc/nixos/hardware-configuration.nix
|
||||
# set hostname, add users and ssh-keys, enable openssh
|
||||
sudo nano /etc/nixos/configuration.nix
|
||||
|
||||
# build config
|
||||
nix-env -p /nix/var/nix/profiles/system -f '<nixpkgs/nixos>' -I nixos-config=/etc/nixos/configuration.nix -iA system
|
||||
|
||||
# prepare target
|
||||
sudo chown -R 0.0 /nix
|
||||
sudo touch /etc/NIXOS
|
||||
sudo touch /etc/NIXOS_LUSTRATE
|
||||
echo etc/nixos | sudo tee -a /etc/NIXOS_LUSTRATE
|
||||
|
||||
# install NixOS
|
||||
sudo NIXOS_INSTALL_BOOTLOADER=1 /nix/var/nix/profiles/system/bin/switch-to-configuration boot
|
||||
|
||||
sudo reboot
|
||||
```
|
||||
- Recommended configuration options
|
||||
|
||||
```nix
|
||||
{
|
||||
# Oracle Cloud uses EFI boot
|
||||
boot.loader.systemd-boot.enable = true;
|
||||
boot.loader.efi.canTouchEfiVariables = true;
|
||||
|
||||
# Kernel cmdline from Ubuntu config
|
||||
boot.kernelParams = [
|
||||
"console=ttyS0"
|
||||
"console=tty1"
|
||||
"nvme.shutdown_timeout=10"
|
||||
"libiscsi.debug_libiscsi_eh=1"
|
||||
];
|
||||
|
||||
# Load graphics driver in stage 1
|
||||
boot.initrd.kernelModules = [ "bochs_drm" ];
|
||||
|
||||
# swap file is recommended
|
||||
swapDevices = [
|
||||
{
|
||||
device = "/swapfile";
|
||||
priority = 0;
|
||||
}
|
||||
];
|
||||
}
|
||||
```
|
||||
- Repartitioning target system from kexec image
|
||||
|
||||
Create `kexec.nix` file with following contents (do not add any packages to `environment.systemPackages` or it won't boot on 1GB system):
|
||||
|
||||
```nix
|
||||
{ config, pkgs, ... }:
|
||||
{
|
||||
imports = [
|
||||
# this will work only under qemu, uncomment next line for full image
|
||||
# <nixpkgs/nixos/modules/installer/netboot/netboot-minimal.nix>
|
||||
<nixpkgs/nixos/modules/installer/netboot/netboot.nix>
|
||||
<nixpkgs/nixos/modules/profiles/qemu-guest.nix>
|
||||
];
|
||||
|
||||
# stripped down version of https://github.com/cleverca22/nix-tests/tree/master/kexec
|
||||
system.build = rec {
|
||||
image = pkgs.runCommand "image" { buildInputs = [ pkgs.nukeReferences ]; } ''
|
||||
mkdir $out
|
||||
cp ${config.system.build.kernel}/${config.system.boot.loader.kernelFile} $out/kernel
|
||||
cp ${config.system.build.netbootRamdisk}/initrd $out/initrd
|
||||
nuke-refs $out/kernel
|
||||
'';
|
||||
kexec_script = pkgs.writeTextFile {
|
||||
executable = true;
|
||||
name = "kexec-nixos";
|
||||
text = ''
|
||||
#!${pkgs.stdenv.shell}
|
||||
set -e
|
||||
${pkgs.kexectools}/bin/kexec -l ${image}/kernel --initrd=${image}/initrd --append="init=${builtins.unsafeDiscardStringContext config.system.build.toplevel}/init ${toString config.boot.kernelParams}"
|
||||
sync
|
||||
echo "executing kernel, filesystems will be improperly umounted"
|
||||
${pkgs.kexectools}/bin/kexec -e
|
||||
'';
|
||||
};
|
||||
kexec_tarball = pkgs.callPackage <nixpkgs/nixos/lib/make-system-tarball.nix> {
|
||||
storeContents = [
|
||||
{
|
||||
object = config.system.build.kexec_script;
|
||||
symlink = "/kexec_nixos";
|
||||
}
|
||||
];
|
||||
contents = [ ];
|
||||
compressCommand = "cat";
|
||||
compressionExtension = "";
|
||||
};
|
||||
kexec_tarball_self_extract_script = pkgs.writeTextFile {
|
||||
executable = true;
|
||||
name = "kexec-nixos";
|
||||
text = ''
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
ARCHIVE=`awk '/^__ARCHIVE_BELOW__/ { print NR + 1; exit 0; }' $0`
|
||||
tail -n+$ARCHIVE $0 | tar x -C /
|
||||
/kexec_nixos $@
|
||||
exit 1
|
||||
__ARCHIVE_BELOW__
|
||||
'';
|
||||
};
|
||||
kexec_bundle = pkgs.runCommand "kexec_bundle" { } ''
|
||||
cat \
|
||||
${kexec_tarball_self_extract_script} \
|
||||
${kexec_tarball}/tarball/nixos-system-${kexec_tarball.system}.tar \
|
||||
> $out
|
||||
chmod +x $out
|
||||
'';
|
||||
};
|
||||
|
||||
boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" ];
|
||||
boot.kernelParams = [
|
||||
"panic=30" "boot.panic_on_fail" # reboot the machine upon fatal boot issues
|
||||
"console=ttyS0" # enable serial console
|
||||
"console=tty1"
|
||||
];
|
||||
boot.kernel.sysctl."vm.overcommit_memory" = "1";
|
||||
|
||||
environment.systemPackages = with pkgs; [ cryptsetup ];
|
||||
environment.variables.GC_INITIAL_HEAP_SIZE = "1M";
|
||||
|
||||
networking.hostName = "kexec";
|
||||
|
||||
services.getty.autologinUser = "root";
|
||||
services.openssh = {
|
||||
enable = true;
|
||||
challengeResponseAuthentication = false;
|
||||
passwordAuthentication = false;
|
||||
};
|
||||
|
||||
documentation.enable = false;
|
||||
documentation.nixos.enable = false;
|
||||
fonts.fontconfig.enable = false;
|
||||
programs.bash.enableCompletion = false;
|
||||
programs.command-not-found.enable = false;
|
||||
security.polkit.enable = false;
|
||||
security.rtkit.enable = pkgs.lib.mkForce false;
|
||||
services.udisks2.enable = false;
|
||||
i18n.supportedLocales = [ (config.i18n.defaultLocale + "/UTF-8") ];
|
||||
|
||||
users.users.root.openssh.authorizedKeys.keys = [
|
||||
# add your ssh key here
|
||||
"ssh-ed25519 ...."
|
||||
];
|
||||
}
|
||||
```
|
||||
|
||||
Build kexec image (you'll need nix/NixOS installed on your machine):
|
||||
|
||||
```shell
|
||||
nix-build '<nixpkgs/nixos>' -A config.system.build.kexec_bundle -I nixos-config=./kexec.nix
|
||||
```
|
||||
|
||||
Copy tarball to remote machine using `scp` and reboot into kexec image:
|
||||
|
||||
```shell
|
||||
scp ./result ubuntu@somehost:/tmp/kexec
|
||||
ssh ubuntu@somehost -t sudo /tmp/kexec
|
||||
# wait for machine to boot and then connect
|
||||
ssh root@somehost
|
||||
```
|
||||
|
||||
Repartition your drive, format, mount file systems, create swap(file) and activate it as soon as possible. Check [manual](https://nixos.org/nixos/manual/index.html#sec-installation-partitioning) for more info.
|
||||
@@ -1,129 +0,0 @@
|
||||
---
|
||||
created: 2024-01-17T10:25:58 (UTC -05:00)
|
||||
tags: [nixos]
|
||||
source: https://www.stackbuilders.com/blog/self-contained-scripts-with-nix/
|
||||
author: Written by:
|
||||
---
|
||||
|
||||
- Stack Builders - Self-contained Scripts With Nix
|
||||
|
||||
> ## Excerpt
|
||||
> The blog post discusses the benefits of Nix in creating reproducible environments where developers can focus on developing new features and delivering value to the users. More specifically, it’s focused on the idea of self-contained scripts, which is one of the most basic use cases for Nix where we can see the benefits it brings to the table and the endless possibilities of the tool.
|
||||
|
||||
---
|
||||
The blog post discusses the benefits of Nix in creating reproducible environments where developers can focus on developing new features and delivering value to the users. More specifically, it’s focused on the idea of self-contained scripts, which is one of the most basic use cases for Nix where we can see the benefits it brings to the table and the endless possibilities of the tool.
|
||||
|
||||
---
|
||||
|
||||
**_“Nix is a tool that takes a unique approach to package management and system configuration.”_** \- nixos.org
|
||||
|
||||
The goal here is pretty simple: help teams create reproducible environments where they can focus on creating value through developing new features.
|
||||
|
||||
Reproducible environments abstract some of the complexity of setting up a project, which is usually detailed in the first page of its repository, and explains how to set up the machine before doing any work. This configuration might very well conflict with that of other projects, be outdated, or simply not work on a particular architecture or OS (Operating System).
|
||||
|
||||
With this in mind, we are introducing Nix in our projects without developers even noticing it - their workflow stays the same after the introduction of Nix. This incremental approach increases the abstraction level with every step, improving the parity between the environments that an application goes through in the development process.
|
||||
|
||||
Nix has different use cases, but in the context of this blog post, we would like to focus on self-contained scripts as a quick introduction to some of the benefits offered by Nix, which revolve around the ideas of isolation and reproducibility.
|
||||
- Introduction to Nix
|
||||
|
||||
Before we go any further, let's get started with a brief introduction of Nix, which usually refers to a bunch of things like the package manager, the language, or the Nixpkgs (Nix packages) library. These concepts will be described in the following sections:
|
||||
- Nix Package Manager
|
||||
|
||||
The Nix package manager is fully functional and has a few benefits over common ones like apt or Homebrew. For example, being multi-platform, supporting side-by-side installation of multiple versions of a package or being hermetic, which prevents packages from breaking during installation and upgrades.
|
||||
|
||||
While most operating systems expect packages in /usr/local/bin or other global paths, Nix places them in the Nix store, which usually lives under /nix/store, and then plays around with the PATH variable to make those tools accessible from everywhere on the system.
|
||||
|
||||
A package example could be: `/nix/store/<ID>-firefox-33.1`, where ID is a cryptographic hash unique for this particular version of the Firefox package that captures all its dependencies. This enables Nix to do a few things:
|
||||
- Hold multiple versions of the same package as every little change to the package will generate a completely different hash, preventing collisions.
|
||||
- Perform atomic installs and updates, where nothing is overridden. Nix will only change paths when the package is fully installed. This no-override feature facilitates that different packages use different versions of their dependencies and simplifies rollbacks as nothing gets deleted.
|
||||
- Nix language
|
||||
|
||||
The Nix language is used to declare packages and configurations to be built by Nix. It is also purely functional and lazily evaluated.
|
||||
- Nixpkgs library
|
||||
|
||||
Finally, Nixpkgs is a [repository](https://github.com/NixOS/nixpkgs) that holds every package available in the Nix ecosystem. The branches are called channels which group different packages and versions together based on their stability, amount of packages, etc. Those packages are defined using the Nix language and have been built from source with a full dependency tree. After being built, resulting binaries are hosted on a [binary cache](https://cache.nixos.org/) allowing end users to download and use them without the need of rebuilding them locally.
|
||||
- Self-contained scripts
|
||||
|
||||
After this introduction, let’s dive into the main topic of this post and one of the most basic use cases for Nix, self-contained scripts.
|
||||
|
||||
Scripts are useful to automate tasks that otherwise would have to be executed by hand. These scripts can be written in a variety of languages, but even the most basic Bash scripts depend on certain commands to be present at run-time. Some of these basic utilities are available in most operating systems, however, differences in versions or flavors of the same tool (GNU vs non-GNU tools) present a challenge for maintainers who want to create portable scripts.
|
||||
|
||||
As you might imagine, this looks like a recipe for disaster, and the most common way to solve this problem is by either adding a lot of complexity to a script to try to handle all different edge cases or relying on extensive documentation that is overlooked most of the time. To make things worse, even if we are able to capture the full dependency tree of the script and pin every package to a specific version, the next user will have a hard time replicating that environment, as the average package manager is not flexible enough to pinpoint an exact version for every package.
|
||||
|
||||
The end result of all this scripting is a very fragile piece of code which “works on my machine”, at least until you update one of its mutable dependencies, breaking it for every user. Let’s see how self-contained scripts with Nix can help us to avoid this issue with an example of a regular Bash script:
|
||||
|
||||
```text
|
||||
#!/usr/bin/env bash
|
||||
echo "---AWS CLI---"
|
||||
echo "Version:" "$(aws --version)"
|
||||
echo "Location:" "$(which aws)"
|
||||
echo "---jq---"
|
||||
echo "Version:" "$(jq --version)"
|
||||
echo "Location:" "$(which jq)"
|
||||
```
|
||||
|
||||
Now check the Nix version of it:
|
||||
|
||||
```text
|
||||
#!/usr/bin/env nix-shell
|
||||
#!nix-shell -i bash -p awscli2 jq
|
||||
#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-22.11.tar.gz
|
||||
echo "---AWS CLI---"
|
||||
echo "Version:" "$(aws --version)"
|
||||
echo "Location:" "$(which aws)"
|
||||
echo "---jq---"
|
||||
echo "Version:" "$(jq --version)"
|
||||
echo "Location:" "$(which jq)"
|
||||
```
|
||||
|
||||
Assuming you have [Nix installed](https://nixos.org/download.html), you can run the above script as you would do with any other bash script and it will use its own, self-contained dependencies to execute.
|
||||
|
||||
This is possible thanks to the first three lines, which individually do the following:
|
||||
- `#!/usr/bin/env nix-shell` is a shebang, which tells the system to run the script with nix-shell, one of Nix’s tools
|
||||
- `#!nix-shell -i bash -p awscli2 jq` is doing a few things:
|
||||
- `i bash` sets the real interpreter, which is the one ultimately running the script
|
||||
- `p awscli2 jq` sets which packages will be included in the shell
|
||||
- The last line is pinning a specific nixpkgs channel, in this case, nixos-22.11
|
||||
|
||||
After this, we have our usual bash code which will make use of every package available in the system, but will give priority to those specified in the nix-shell. This is a source of “impurity”, but our code only uses these two packages (awscli2 and jq), which are properly isolated from the rest, including their own dependencies.
|
||||
|
||||
Another point worth mentioning is the ability to pin a specific revision of Nixpkgs, as with the current example, package versions will change over time as the channel is updated. To do that, we would replace the last line with:
|
||||
|
||||
```text
|
||||
#!nix-shell -I
|
||||
|
||||
nixpkgs=https://github.com/NixOS/nixpkgs/archive/3c75992f01290979c9c1f997e40efa77845bef1a.tar.gz
|
||||
```
|
||||
|
||||
Compared to the line in the example script, this one has a specific commit hash, preventing packages from changing over time unless we manually update the channel. The downside of this approach is that old package versions may be removed from the remote binary cache, forcing us to compile them locally from source, causing extensive build times. Depending on the use case we might choose one option or the other.
|
||||
|
||||
The other major language for scripting, Python, is also compatible with this kind of shell:
|
||||
|
||||
```text
|
||||
#!/usr/bin/env nix-shell
|
||||
#!nix-shell -i python -p python37 pythonPackages.prettytable
|
||||
#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-22.11.tar.gz
|
||||
import prettytable
|
||||
# Print a simple table
|
||||
t = prettytable.PrettyTable(["N", "N^2"])
|
||||
for n in range(1, 10): t.add_row([n, n * n])
|
||||
print(t)
|
||||
```
|
||||
|
||||
In this case, we are including a specific version of python (3.7), and some python packages (prettytable) which we would normally install with pip. This way we skip the usual steps of:
|
||||
- Using pyenv to select python 3.7
|
||||
- Creating a virtual environment with venv
|
||||
- Activating this virtual environment
|
||||
- Installing dependencies with pip
|
||||
- Running the script
|
||||
- Deactivating the environment and changing python version again
|
||||
- Conclusions
|
||||
|
||||
As we can see, introducing these self-contained scripts in our workflows requires very little effort from the development team, which only needs to have Nix installed, but brings lots of benefits in terms of reproducibility, isolation and ease of use. We can tweak these parameters to accommodate the project’s needs, for example, by being more strict with the pinning or allowing a higher level of impurity. Most advanced use cases of Nix, like development environments, bring the same benefits to the table but instead of just a script, they encapsulate an entire setup, from dependencies to environment variables and everything in between, helping developers reduce the overhead of configuring and switching between different projects while ripping the benefits of reproducibility, isolation and portability.
|
||||
|
||||
Published on Mar. 21, 2023
|
||||
|
||||
|
||||
- Subscribe to the Blog
|
||||
|
||||
Join our community of avid readers and stay informed with the latest articles, tips, and insights delivered straight to your inbox. Don't miss out on valuable content – subscribe now and be part of the conversation!
|
||||
Reference in New Issue
Block a user