[logseq-plugin-git:commit] 2025-12-11T14:50:38.767Z
This commit is contained in:
@@ -1,9 +0,0 @@
|
||||
---
|
||||
tags:
|
||||
- Hover
|
||||
- directions
|
||||
---
|
||||
[[Hover - Creating an SPF record]]
|
||||
|
||||
|
||||
![[Pasted image 20240119104604.png]]
|
||||
@@ -1,67 +0,0 @@
|
||||
---
|
||||
created: 2024-01-19T10:43:45 (UTC -05:00)
|
||||
tags:
|
||||
- Hover
|
||||
source: https://help.hover.com/hc/en-us/articles/115006406047-Creating-an-SPF-record
|
||||
author:
|
||||
---
|
||||
|
||||
- Creating an SPF record – Hover Help Center
|
||||
|
||||
---
|
||||
Having an SPF record will help protect your domain against spoofing. It also helps prevent your outgoing messages from being marked as spam by receiving mail servers.
|
||||
- [SPF basics](https://help.hover.com/hc/en-us/articles/115006406047-Creating-an-SPF-record#h_01EV7KH3T1W7KVC543YF6W595A)
|
||||
- [Adding an SPF record](https://help.hover.com/hc/en-us/articles/115006406047-Creating-an-SPF-record#h_01G0FP8WPC1J4SASBV304DHYGQ)
|
||||
- [Removing an SPF record](https://help.hover.com/hc/en-us/articles/115006406047-Creating-an-SPF-record#h_01G0FP93A5TDJH6RCH9E4T1ENJ)
|
||||
- [SPF configuration examples](https://help.hover.com/hc/en-us/articles/115006406047-Creating-an-SPF-record#h_01G0FP8NNM27GJYHTW4VEWPMRG)
|
||||
- SPF Basics
|
||||
|
||||
SPF (Sender policy framework) is a spam protection method based on the authorization of the email sender. The SPF record is simply a TXT record added to the DNS settings of your domain that defines which mail servers are authorized to send emails.
|
||||
|
||||
You can only have one SPF record per domain. Having more will cause DNS conflicts and in turn, the SPF records will not resolve.
|
||||
|
||||
_Note: The Hover platform has a 255 character limit. This means the SPF record will need to be shorter than 255 characters._
|
||||
|
||||
An SPF record consists of several parts. As an example, here is the most widely used SPF record at Hover.
|
||||
|
||||
```
|
||||
<span>v=spf1 include:_spf.hostedemail.com include:hover.com ~all</span>
|
||||
```
|
||||
|
||||
<table><tbody><tr><td>SPF part</td><td>Relevance</td></tr><tr><td>v=spf1</td><td><p>Defines this as an SPF record. Required to be at the beginning of the value for each SPF record.</p></td></tr><tr><td> a</td><td><p><span>Defines the DNS A record of the domain as a valid sending source. <br><em>Note: The above example does not include an a mechanism</em>.</span></p></td></tr><tr><td> mx</td><td><p><span>Defines the DNS MX record of the domain as a valid sending source.<br><em>Note: The above example does not include an mx mechanism.</em><br></span></p></td></tr><tr><td> -all</td><td><p>Specifies that all emails sent through a different server than those already listed as "ok" will return a code of “hard fail.” The email will not be delivered and generate a bounce-back email.</p></td></tr><tr><td>~all</td><td><p>If you want to allow an email sent through a different mail server to be delivered, you can force a “soft fail”. With the ~all in place, the email will be tagged as suspicious but will still be delivered to the recipient.</p></td></tr></tbody></table>
|
||||
|
||||
[Back to top](https://help.hover.com/hc/en-us/articles/115006406047-Creating-an-SPF-record#h_01G0FRACWPTM31VA6H142G17YH)
|
||||
- Adding an SPF record
|
||||
|
||||
1. Sign in to your [Hover control panel](https://hover.com/signin) using your chosen method of 2FA.
|
||||
|
||||
2. From the domains Overview page, choose the **DNS** tab.
|
||||
|
||||
3. Select **Add a record**.
|
||||
|
||||
4. Select **TXT** from the Type dropdown menu, followed by the Hostname **@**.
|
||||
The Content is the SPF record. Below is the most widely used SPF record at Hover.
|
||||
|
||||
```
|
||||
v=spf1 include:_spf.hostedemail.com include:hover.com ~all
|
||||
```
|
||||
|
||||
|
||||
5. Click **Add record** to save the changes.
|
||||
|
||||
[Back to top](https://help.hover.com/hc/en-us/articles/115006406047-Creating-an-SPF-record#h_01G0FRACWPTM31VA6H142G17YH)
|
||||
- Removing an SPF record
|
||||
|
||||
There may be times you are required to remove an SPF record.
|
||||
|
||||
1. From the domains Overview page, choose the **DNS** tab.
|
||||
2. Locate the SPF record and select the green **X** to the far right of the record.
|
||||
|
||||
3. From the popup, select **Delete**.
|
||||
|
||||
[Back to top](https://help.hover.com/hc/en-us/articles/115006406047-Creating-an-SPF-record#h_01G0FRACWPTM31VA6H142G17YH)
|
||||
- SPF configuration examples
|
||||
|
||||
<table><tbody><tr><td>Desired outcome</td><td>Example</td></tr><tr><td>Specify a single IPv4 address that can send emails</td><td><pre>v=spf1 ip4:204.200.197.197 -all</pre><p>This would allow mail to be sent from a mail server at the IP address 204.200.197.197 only. </p><p>Mail sent from mail servers on any other IP address would not be delivered, and the sender would receive a bounceback.</p></td></tr><tr><td>Specify a range of IPv4 addresses that can send emails</td><td><pre>v=spf1 ip4:192.168.0.1/16 -all</pre><p>Allows mail to be sent from any IP address between 192.168.0.1 and 192.168.255.255.</p></td></tr><tr><td>Specify a mail server that can send</td><td><pre>v=spf1 mx:mx1.domain.com -all</pre><p>Would allow mail to be sent from a mail server named mx1.domain.com. Mail from any other mail server would not be delivered, and the sender would receive a bounce message.</p></td></tr><tr><td>Specify multiple items in one SPF record</td><td><pre>v=spf1 a mx ip4:204.200.197.197 mx:mx1.domain.com -all</pre><p>Allows an IP address and a mail server to send out emails.</p></td></tr><tr><td>Make domain unable to send any emails</td><td><pre>v=spf1 -all<br><br></pre><p>All emails will be bounced back to the sender.</p></td></tr><tr><td>Specify a single IPv6 address that can send</td><td><pre>v=spf1 ip6:1080::8:800:200C:417A -all</pre><p>Would allow mail to be sent from a mail server at the IP address 1080::8:800:200C:417A only. </p></td></tr><tr><td>Specify a range of IPv6 addresses that can send</td><td><pre>v=spf1 ip6:1080::8:800:200C:417A/96 -all</pre><p>Allows mail to be sent from any IPv6 address between 1080::8:800:0000:0000 and 1080::8:800:FFFF:FFFF.</p></td></tr><tr><td>Specify another domain that can send email from the domain</td><td><pre>v=spf1 include:anotherdomain.com -all</pre><div><p>Allows mail to be sent from another specific domain on behalf of the domain, for example, anotherdomain.com, which has this SPF record in its DNS records.</p><p><em>Note: For this to work, anotherdomain.com must have a valid SPF record in its own DNS records.</em></p></div></td></tr></tbody></table>
|
||||
|
||||
[Back to top](https://help.hover.com/hc/en-us/articles/115006406047-Creating-an-SPF-record#h_01G0FRACWPTM31VA6H142G17YH)
|
||||
@@ -1,15 +0,0 @@
|
||||
#selfhosted
|
||||
|
||||
DYN DNS SERVERs
|
||||
|
||||
- Servers
|
||||
|
||||
https://www.duckdns.org/
|
||||
|
||||
https://desec.io/
|
||||
- sync service
|
||||
|
||||
https://www.dnsomatic.com/
|
||||
|
||||
|
||||
UNIFI sends the updates via service to dnsomatic, it fwds to others
|
||||
@@ -1,110 +0,0 @@
|
||||
---
|
||||
created: 2024-01-19T13:12:25 (UTC -05:00)
|
||||
tags: [Ubiquiti]
|
||||
source: https://community.ui.com/questions/SSH-authorizedkeys-USG/f73c36ff-e01c-4ca1-9868-584f31cdb310
|
||||
author:
|
||||
---
|
||||
|
||||
- SSH authorized_keys USG | Ubiquiti Community
|
||||
|
||||
> ## Excerpt
|
||||
> can just, for the life of me not get this to work. I don't have a system json entry in my config.gateway.json... and when i try to add the section [and I add it correctly as JSON object] it just gets the USG stuck in provisioning mode when it tries to pull down...I have tried inserting the code with necessary trailing comma - at the start of the file [after opening bracket... but fails.
|
||||
|
||||
---
|
||||
can just, for the life of me not get this to work. I don't have a system json entry in my config.gateway.json... and when i try to add the section \[and I add it correctly as JSON object\] it just gets the USG stuck in provisioning mode when it tries to pull down...I have tried inserting the code with necessary trailing comma - at the start of the file \[after opening bracket... but fails.
|
||||
|
||||
```
|
||||
{
|
||||
"firewall": {
|
||||
"name": {
|
||||
"WAN_LOCAL": {
|
||||
"rule": {
|
||||
"4": {
|
||||
"action": "accept",
|
||||
"description": "SSH to WAN",
|
||||
"destination": {
|
||||
"address": "*redacted*",
|
||||
"port": "22"
|
||||
},
|
||||
"protocol": "tcp"
|
||||
},
|
||||
"50": {
|
||||
"action": "accept",
|
||||
"description": "Allow L2TP",
|
||||
"destination": {
|
||||
"port": "500,1701,4500"
|
||||
},
|
||||
"protocol": "udp"
|
||||
},
|
||||
"51": {
|
||||
"action": "accept",
|
||||
"description": "Allow ESP",
|
||||
"protocol": "esp"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"vpn": {
|
||||
"pptp": {
|
||||
"remote-access": {
|
||||
"authentication": {
|
||||
"local-users": {
|
||||
"username": {
|
||||
"user1": {
|
||||
"password": "*redacted*"
|
||||
}
|
||||
}
|
||||
},
|
||||
"mode": "local"
|
||||
}
|
||||
}
|
||||
},
|
||||
"ipsec": {
|
||||
"auto-firewall-nat-exclude": "disable",
|
||||
"ipsec-interfaces": {
|
||||
"interface": [
|
||||
"eth0"
|
||||
]
|
||||
},
|
||||
"nat-networks": {
|
||||
"allowed-network": {
|
||||
"0.0.0.0/0": "''"
|
||||
}
|
||||
},
|
||||
"nat-traversal": "enable"
|
||||
},
|
||||
"l2tp": {
|
||||
"remote-access": {
|
||||
"authentication": {
|
||||
"local-users": {
|
||||
"username": {
|
||||
"user1": {
|
||||
"password": "*redacted*"
|
||||
}
|
||||
}
|
||||
},
|
||||
"mode": "local"
|
||||
},
|
||||
"client-ip-pool": {
|
||||
"start": "192.168.1.200",
|
||||
"stop": "192.168.1.254"
|
||||
},
|
||||
"dhcp-interface": "eth0",
|
||||
"dns-servers": {
|
||||
"server-1": "8.8.8.8",
|
||||
"server-2": "8.8.4.4"
|
||||
},
|
||||
"ipsec-settings": {
|
||||
"authentication": {
|
||||
"mode": "pre-shared-secret",
|
||||
"pre-shared-secret": "*redacted*"
|
||||
},
|
||||
"ike-lifetime": "3600"
|
||||
},
|
||||
"mtu": "1492"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -1,24 +0,0 @@
|
||||
---
|
||||
title: Set UniFI BGP
|
||||
updated: 2022-09-09 14:26:31Z
|
||||
created: 2022-09-09 14:25:27Z
|
||||
latitude: 40.73565700
|
||||
longitude: -74.17236670
|
||||
altitude: 0.0000
|
||||
---
|
||||
|
||||
show ip bgp
|
||||
configure
|
||||
set protocols bgp 64512 parameters router-id 192.168.1.1
|
||||
set protocols bgp 64512 neighbor 192.168.1.221 remote-as 64512
|
||||
set protocols bgp 64512 neighbor 192.168.1.222 remote-as 64512
|
||||
set protocols bgp 64512 neighbor 192.168.1.223 remote-as 64512
|
||||
set protocols bgp 64512 neighbor 192.168.1.224 remote-as 64512
|
||||
set protocols bgp 64512 neighbor 192.168.1.225 remote-as 64512
|
||||
set protocols bgp 64512 neighbor 192.168.1.226 remote-as 64512
|
||||
set protocols bgp 64512 neighbor 192.168.1.227 remote-as 64512
|
||||
show protocols bgp
|
||||
commit
|
||||
save
|
||||
exit
|
||||
show ip bgp
|
||||
@@ -1,125 +0,0 @@
|
||||
- Restore access to a unifi controller
|
||||
|
||||
When you are *unable to login* to the unifi controller or forgot admin password, you can **restore access** using SSH and manipulating **mongodb** directly.
|
||||
- Warning
|
||||
|
||||
Do not uninstall unifi controller - most of the data is not stored in mongodb. In case you thought a mongodb backup would be sufficient, you may have fucked up already, just like me. However I managed to write this "tutorial" for anyone to not run into the same trap.
|
||||
|
||||
**Apparently this guide no longer works with recent unifi controller versions (starting nov/dec 2022)**. Since I no longer use unifi hardware in my home system, I can not update the guide myself. In case you've gotten here to recover your data, you're likely doomed. But giving it a try won't hurt anyway, therefore: good luck.
|
||||
- Steps
|
||||
- 1. Generate password
|
||||
|
||||
Use [quickhhash.com](https://quickhash.com/) to generate a new password. Use `sha512 / crypt(3) / $6$` with the any salt you like (I used `9Ter1EZ9$lSt6` in the example below, but it really doesn't matter).
|
||||
|
||||
I have generated a dummy password for you if you want to leave this step out. It is `Ch4ngeM3VeryQu!ck`:
|
||||
```
|
||||
$6$9Ter1EZ9$4RCTnLfeDJsdAQ16M5d1d5Ztg2CE1J2IDlbAPSUcqYOoxjEEcpMQag41dtCQv2cJ.n9kvlx46hNT78dngJBVt0
|
||||
```
|
||||
- 2. SSH to controller
|
||||
|
||||
SSH to the server running the unifi controller. In my case it's running on a raspberry pi.
|
||||
|
||||
```
|
||||
|
||||
wget https://fastdl.mongodb.org/linux/mongodb-linux-arm64-ubuntu1604-3.4.24.tgz
|
||||
tar -zxvf mongodb-linux-arm64-ubuntu1604-3.4.24.tgz
|
||||
|
||||
mongodb-linux-aarch64-ubuntu1604-3.4.24/bin/mongo -port 27117
|
||||
```
|
||||
```
|
||||
|
||||
|
||||
### 3. Connect to mongodb
|
||||
|
||||
By default unifi comes with mongodb running on port `27117`. To connect to it, use the `mongo` cli tool. Make sure it is installed.
|
||||
|
||||
Connect using the following command:
|
||||
|
||||
```bash
|
||||
mongo --port 27117
|
||||
```
|
||||
|
||||
When connected to mongo, execute the following commands to switch the database and verify the installation
|
||||
|
||||
```
|
||||
use ace;
|
||||
show collections;
|
||||
```
|
||||
|
||||
It should show a list of collections, e.g. `account, admin, alarm, broadcastgroup, ...`.
|
||||
|
||||
### 4. Fix
|
||||
|
||||
It is very likely that you got here because of power/data loss. You want to check if admins are still in the database.
|
||||
To do so, execute the following command in the mongo cli:
|
||||
|
||||
```
|
||||
db.admin.find()
|
||||
```
|
||||
|
||||
If the result is blank or you don't remember your password, there's two ways. Make sure to replace variables before executing commands.
|
||||
|
||||
#### 4.1. Change password of *existing user*
|
||||
|
||||
```
|
||||
db.admin.update({ name: "<YOUR-NAME-GOES-HERE>" }, { $set: { "x_shadow": "<PASSWORD-HASH-FROM-STEP-1-GOES-HERE>" } });
|
||||
```
|
||||
|
||||
#### 4.2. Create a *new user*
|
||||
|
||||
```
|
||||
db.admin.insert({ "email" : "<YOUR-EMAIL-GOES-HERE>", "last_site_name" : "default", "name" : "<YOUR-NAME-GOES-HERE>", "time_created" : NumberLong(100019800), "x_shadow" : "<PASSWORD-HASH-FROM-STEP-1-GOES-HERE>" })
|
||||
```
|
||||
|
||||
### 5. Get admin id
|
||||
|
||||
```
|
||||
db.admin.find()
|
||||
```
|
||||
|
||||
Will output something like this:
|
||||
|
||||
```
|
||||
> db.admin.find()
|
||||
{ "_id" : ObjectId("5d0a2e7e8f01c49af4cbe3cd"), "email" : "...", ... }
|
||||
```
|
||||
|
||||
Take the contents of `_id`, in this case it is `5d0a2e7e8f01c49af4cbe3cd`. You should remember it for the next steps.
|
||||
|
||||
### 6. Fix permissions
|
||||
|
||||
You will need to attach the admin role using db.privilege to the newly created user. The privilege belongs to an admin and a site_id.
|
||||
|
||||
Make sure to get your site_ids using the following command:
|
||||
|
||||
```
|
||||
db.site.find()
|
||||
```
|
||||
|
||||
It will show something like this:
|
||||
|
||||
```
|
||||
> db.site.find()
|
||||
{ "_id" : ObjectId("5d07b088280f9002d7676c87"), "name" : "super", "key" : "super", "attr_hidden_id" : "super", "attr_hidden" : true, "attr_no_delete" : true, "attr_no_edit" : true }
|
||||
{ "_id" : ObjectId("5d07b088280f9002d7676c88"), "name" : "default", "desc" : "Default", "attr_hidden_id" : "default", "attr_no_delete" : true }
|
||||
```
|
||||
|
||||
Once you know the ids of your sites, you can continue with creating privilege entries. You will need the **admin id** from [step 5](#5-Get-admin-id).
|
||||
|
||||
Use the following command for **each site** you got from `db.site.find()`
|
||||
|
||||
```
|
||||
db.privilege.insert({ "admin_id" : "<ADMIN-ID-GOES-HERE>", "permissions" : [ ], "role" : "admin", "site_id" : "<SITE-ID-GOES-HERE>" });
|
||||
```
|
||||
|
||||
Optionally verify that all privileges have been created using the following command:
|
||||
|
||||
```
|
||||
> db.privilege.find()
|
||||
{ "_id" : ObjectId("5d0bb7573d70717df47d5af6"), "admin_id" : "5d0a2e7e8f01c49af4cbe3cd", "permissions" : [ ], "role" : "admin", "site_id" : "5d07b088280f9002d7676c87" }
|
||||
{ "_id" : ObjectId("5d0bb7573d70717df47d5af7"), "admin_id" : "5d0a2e7e8f01c49af4cbe3cd", "permissions" : [ ], "role" : "admin", "site_id" : "5d07b088280f9002d7676c88" }
|
||||
```
|
||||
- 7. Test
|
||||
|
||||
Now you're all set. You eventually want to restart the unifi controller using `service unifi restart`.
|
||||
You can login now. Good Luck.
|
||||
@@ -1,19 +0,0 @@
|
||||
|
||||
|
||||
- Update DynDNS
|
||||
`update dns dynamic interface eth0`
|
||||
|
||||
`show dns dynamic status`
|
||||
> interface : eth0
|
||||
> ip address : 74.64.197.78
|
||||
> host-name : all.dnsomatic.com
|
||||
> last update : Wed Mar 8 14:01:43 2023
|
||||
> update-status: good
|
||||
>
|
||||
> interface : eth0
|
||||
> ip address : 74.64.197.78
|
||||
> host-name : fbleagh.ignorelist.com
|
||||
> last update : Wed Mar 8 14:01:43 2023
|
||||
> update-status: good
|
||||
|
||||
|
||||
@@ -1,96 +0,0 @@
|
||||
---
|
||||
created: 2024-01-17T10:28:08 (UTC -05:00)
|
||||
tags: [nixos]
|
||||
source: https://johns.codes/blog/efficient-nix-derivations-with-file-sets
|
||||
author: John Murray
|
||||
---
|
||||
|
||||
- Efficient Nix Derivations with File Sets
|
||||
|
||||
> ## Excerpt
|
||||
> Using nix's new file set API to make efficient derivations
|
||||
|
||||
---
|
||||
Table of Contents
|
||||
- [What are file sets](https://johns.codes/blog/efficient-nix-derivations-with-file-sets#what-are-file-sets)
|
||||
- [A Real Example](https://johns.codes/blog/efficient-nix-derivations-with-file-sets#a-real-example)
|
||||
- [Wrap up](https://johns.codes/blog/efficient-nix-derivations-with-file-sets#wrap-up)
|
||||
|
||||
If you are using Nix to build your own packages you will eventually come across something like
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>stdenv</span><span>.</span><span>mkDerivation</span><span> {</span></code></p><p><code><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>"my awesome pkg"</span><span>;</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span>buildPhase</span><span> </span><span>=</span><span> </span><span>''</span></code></p><p><code><span> # I have no idea if this actually works (the gcc call bit)</span></code></p><p><code><span> # but the vibe is what I care about</span></code></p><p><code><span> gcc main.c -o my_program</span></code></p><p><code><span> mkdir -p $out</span></code></p><p><code><span> cp my_program $out</span></code></p><p><code><span> ''</span><span>;</span></code></p><p><code><span>}</span></code></p><p></p>
|
||||
```
|
||||
|
||||
The issue with the above is setting `src = ./.`, which makes **ALL** of the current directory an input to the derivation. So if you have a readme file in this folder, changing that will cause this derivation to rebuild.
|
||||
|
||||
The build only really cares about `main.c` in this case so what can we do to fix this?
|
||||
|
||||
Back in the dark dark times of pre-Nixos 23.11, there were ways to do this kind of filtering but IMO they were kinda confusing and I never quite got it to work right so I just stuck with `src = ./.`
|
||||
|
||||
Now with 23.11 we have [filesets](https://www.tweag.io/blog/2023-11-28-file-sets/), which makes filtering and adding files much simpler.
|
||||
- What are file sets
|
||||
|
||||
I would recommend checking out the [docs](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset) or [official tutorial](https://nix.dev/tutorials/file-sets), but for a TLDR, you can do the following
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>let</span></code></p><p><code><span> </span><span>fs</span><span> </span><span>=</span><span> </span><span>pkgs</span><span>.</span><span>lib</span><span>.</span><span>fileset</span><span>;</span></code></p><p><code><span> </span><span>baseSrc</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>unions</span><span> [ </span><span>./Makefile</span><span> </span><span>./src</span><span> ];</span></code></p><p><code><span> </span><span>filterMarkdownFiles</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>fileFilter</span><span> (</span><span>file</span><span>: </span><span>hasSuffix</span><span> </span><span>".md"</span><span> </span><span>file</span><span>.</span><span>name</span><span>) </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span>removedMarkedDown</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>difference</span><span> </span><span>baseSrc</span><span> </span><span>filterMarkdownFiles</span><span>;</span></code></p><p><code><span>in</span></code></p><p><code><span>stdenv</span><span>.</span><span>mkDerivation</span><span> {</span></code></p><p><code><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>"my awesome pkg"</span><span>;</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>toSource</span><span> {</span></code></p><p><code><span> </span><span>root</span><span> </span><span>=</span><span> </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span>fileset</span><span> </span><span>=</span><span> </span><span>removedMarkedDown</span><span>;</span></code></p><p><code><span> };</span></code></p><p><code><span> </span><span>buildPhase</span><span> </span><span>=</span><span> </span><span>''</span></code></p><p><code><span> # call make or w/e you want</span></code></p><p><code><span> ''</span><span>;</span></code></p><p><code><span>}</span></code></p><p></p>
|
||||
```
|
||||
|
||||
Now with this setup, we have a "base" file set of `[ ./Makefile ./src ]` then with `fs.difference` we can remove all files that are markdown with the `filterMarkdownFiles` filter.
|
||||
- A Real Example
|
||||
|
||||
Recently I started messing around with the language [Roc](https://www.roc-lang.org/). If you haven't heard of it, Roc is a new functional language heavily inspired by Elm. It's fast but also very nice to use (though some rough edges since it's pre-0.1.0).
|
||||
|
||||
One of its interesting ideas is that Roc needs your app to pick what [platform](https://www.roc-lang.org/platforms) to run on. A platform would be written in something like rust, zig, c, etc. The platform provides roc APIs for things like managing memory, making network requests, printing to stdout, and other IO-like actions. Right now the two most widely used are a [cli platform](https://github.com/roc-lang/basic-cli) and [webserver platform](https://github.com/roc-lang/basic-webserver)
|
||||
|
||||
This is really neat but brings an issue for developing a platform along with the roc code needed to define the platform API. You need to compile the "platform code" (ie rust + some c), do w/e linking is needed for that, then distribute that with the roc source code. The roc cli will do this for you when developing but it doesn't work as nicely to compile just the platform with nix.
|
||||
|
||||
For example, [this](https://github.com/roc-lang/roc/tree/main/examples/platform-switching/rust-platform) is one of the sample platforms
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>❯ exa --tree --level 2</span></code></p><p><code><span>.</span></code></p><p><code><span>├── Cargo.lock</span></code></p><p><code><span>├── Cargo.toml</span></code></p><p><code><span>├── host.c</span></code></p><p><code><span>├── main.roc</span></code></p><p><code><span>├── rust-toolchain.toml</span></code></p><p><code><span>└── src</span></code></p><p><code><span> ├── glue.rs</span></code></p><p><code><span> ├── lib.rs</span></code></p><p><code><span> └── main.rs</span></code></p><p></p>
|
||||
```
|
||||
|
||||
To build this platform you need to run a `cargo build --lib ...` on the rust code, compile the `host.c` file, link those two object files together, and then finally distribute the linked object file with the roc code.
|
||||
|
||||
so after all those steps, it should look something like
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>❯ exa --tree </span><span><</span><span>compiled folder</span><span>></span></code></p><p><code><span><</span><span>compiled folder</span><span>></span></code></p><p><code><span>├── linux-x64.o</span></code></p><p><code><span>└── main.roc</span></code></p><p></p>
|
||||
```
|
||||
|
||||
The naive approach would probably be something like
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>let</span></code></p><p><code><span>compiledC</span><span> </span><span>=</span><span> </span><span>mkDerivation</span><span> {</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span># compile the c ...</span></code></p><p><code><span>};</span></code></p><p><code><span>rustBuiltLib</span><span> </span><span>=</span><span> </span><span>buildRustPackage</span><span> {</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span># build the rust</span></code></p><p><code><span>};</span></code></p><p><code><span>in</span></code></p><p><code><span>llvmPkgs</span><span>.</span><span>stdenv</span><span>.</span><span>mkDerivation</span><span> </span><span>rec</span><span> {</span></code></p><p><code><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>"</span><span>${</span><span>pname</span><span>}</span><span>-</span><span>${</span><span>version</span><span>}</span><span>"</span><span>;</span></code></p><p><code><span> </span><span>srcs</span><span> </span><span>=</span><span> [</span></code></p><p><code><span> </span><span>rustBuiltLib</span></code></p><p><code><span> </span><span>compiledC</span></code></p><p><code><span> </span><span>./.</span><span> </span><span># for the roc code</span></code></p><p><code><span> ];</span></code></p><p><code><span> </span><span>sourceRoot</span><span> </span><span>=</span><span> </span><span>"."</span><span>;</span></code></p><p><code><span> </span><span>buildPhase</span><span> </span><span>=</span><span> </span><span>''</span></code></p><p><code><span> # link the rust and c files</span></code></p><p><code><span> # copy roc and linked object out</span></code></p><p><code><span> ''</span><span>;</span></code></p><p><code><span>}</span></code></p><p></p>
|
||||
```
|
||||
|
||||
while this works, it also sucks. **ANY** change to the files will cause all 3 derivations to be rebuilt.
|
||||
|
||||
Now with filesets we can be all cute and fancy
|
||||
|
||||
This example will have many parts omitted to keep the code easy to follow. To see the real code, look at my [roc2nix repo](https://github.com/JRMurr/roc2nix/blob/main/lib/platformBuilders/buildRustPlatform.nix) where this came from
|
||||
|
||||
First let's define a helper file for filtering files based on their extension
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>{</span><span>lib</span><span>}:</span></code></p><p><code><span># Note i generally don't like doing `with` at the top of a file</span></code></p><p><code><span># but since this will be only fileSet stuff it should be fine</span></code></p><p><code><span>with</span><span> </span><span>lib</span><span>.</span><span>fileset</span><span>;</span></code></p><p><code><span>let</span></code></p><p><code><span> </span><span># helper func to take in a list of allowed</span></code></p><p><code><span> </span><span># returns a function of `file => bool` to be used in a fileFilter.</span></code></p><p><code><span> </span><span># true if file has suffix, false if not</span></code></p><p><code><span> </span><span>fileHasAnySuffix</span><span> </span><span>=</span><span> </span><span>fileSuffixes</span><span>: </span><span>file</span><span>: (</span><span>lib</span><span>.</span><span>lists</span><span>.</span><span>any</span><span> (</span><span>s</span><span>: </span><span>lib</span><span>.</span><span>hasSuffix</span><span> </span><span>s</span><span> </span><span>file</span><span>.</span><span>name</span><span>) </span><span>fileSuffixes</span><span>);</span></code></p><p><code><span> </span><span># given a basePath src path, return a fileset of files in that path that are rust files, toml files, or cargo toml/lock</span></code></p><p><code><span> </span><span>rustFilter</span><span> </span><span>=</span><span> </span><span>basePath</span><span>: (</span></code></p><p><code><span> </span><span>let</span></code></p><p><code><span> </span><span>mainFilter</span><span> </span><span>=</span><span> </span><span>fileFilter</span></code></p><p><code><span> (</span><span>fileHasAnySuffix</span><span> [ </span><span>".rs"</span><span> </span><span>".toml"</span><span> ])</span></code></p><p><code><span> </span><span>basePath</span><span>;</span></code></p><p><code><span> </span><span>in</span></code></p><p><code><span> </span><span>unions</span><span> [ </span><span>mainFilter</span><span> (</span><span>basePath</span><span> </span><span>+</span><span> </span><span>"/Cargo.toml"</span><span>) (</span><span>basePath</span><span> </span><span>+</span><span> </span><span>"/Cargo.lock"</span><span>) ]</span></code></p><p><code><span> );</span></code></p><p><code><span> </span><span># given a basePath src path return a fileset with files ending with `.c`</span></code></p><p><code><span> </span><span>cFilter</span><span> </span><span>=</span><span> </span><span>basePath</span><span>: </span><span>fileFilter</span><span> (</span><span>fileHasAnySuffix</span><span> [ </span><span>".c"</span><span> ]) </span><span>basePath</span><span>;</span></code></p><p><code><span> </span><span># given a basePath src path return a fileset with files ending with `.roc`</span></code></p><p><code><span> </span><span>rocFilter</span><span> </span><span>=</span><span> </span><span>basePath</span><span>: </span><span>fileFilter</span><span> (</span><span>fileHasAnySuffix</span><span> [ </span><span>".roc"</span><span> ]) </span><span>basePath</span><span>;</span></code></p><p><code><span>in</span></code></p><p><code><span>{</span></code></p><p><code><span> </span><span>inherit</span><span> </span><span>rustFilter</span><span> </span><span>cFilter</span><span> </span><span>rocFilter</span><span>;</span></code></p><p><code><span>}</span></code></p><p></p>
|
||||
```
|
||||
|
||||
Now with that helper, we can do
|
||||
|
||||
```
|
||||
<p><code></code></p><p><code><span>let</span></code></p><p><code><span> </span><span>fs</span><span> </span><span>=</span><span> </span><span>lib</span><span>.</span><span>fileset</span><span>;</span></code></p><p><code><span> </span><span>languageFilters</span><span> </span><span>=</span><span> </span><span>import</span><span> </span><span>./languageFilters.nix</span><span> {</span><span>inherit</span><span> </span><span>lib</span><span>;};</span></code></p><p><code><span> </span><span>baseDir</span><span> </span><span>=</span><span> </span><span>./.</span><span>;</span></code></p><p><code><span> </span><span>compiledC</span><span> </span><span>=</span><span> </span><span>mkDerivation</span><span> {</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>toSource</span><span> {</span></code></p><p><code><span> </span><span>root</span><span> </span><span>=</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> </span><span>fileset</span><span> </span><span>=</span><span> </span><span>languageFilters</span><span>.</span><span>cFilter</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> };</span></code></p><p><code><span> </span><span># compile the c ...</span></code></p><p><code><span> };</span></code></p><p><code><span> </span><span>rustBuiltLib</span><span> </span><span>=</span><span> </span><span>buildRustPackage</span><span> {</span></code></p><p><code><span> </span><span>src</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>toSource</span><span> {</span></code></p><p><code><span> </span><span>root</span><span> </span><span>=</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> </span><span>fileset</span><span> </span><span>=</span><span> </span><span>languageFilters</span><span>.</span><span>rustFilter</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> };</span></code></p><p><code><span> </span><span># build the rust</span></code></p><p><code><span> };</span></code></p><p><code><span> </span><span>rocCode</span><span> </span><span>=</span><span> </span><span>fs</span><span>.</span><span>toSource</span><span> {</span></code></p><p><code><span> </span><span>root</span><span> </span><span>=</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> </span><span>fileset</span><span> </span><span>=</span><span> </span><span>languageFilters</span><span>.</span><span>rocFilter</span><span> </span><span>baseDir</span><span>;</span></code></p><p><code><span> };</span></code></p><p><code><span>in</span></code></p><p><code><span>llvmPkgs</span><span>.</span><span>stdenv</span><span>.</span><span>mkDerivation</span><span> </span><span>rec</span><span> {</span></code></p><p><code><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>"</span><span>${</span><span>pname</span><span>}</span><span>-</span><span>${</span><span>version</span><span>}</span><span>"</span><span>;</span></code></p><p><code><span> </span><span>srcs</span><span> </span><span>=</span><span> [</span></code></p><p><code><span> </span><span>rustBuiltLib</span></code></p><p><code><span> </span><span>compiledC</span></code></p><p><code><span> </span><span>rocCode</span></code></p><p><code><span> ];</span></code></p><p><code><span> </span><span>sourceRoot</span><span> </span><span>=</span><span> </span><span>"."</span><span>;</span></code></p><p><code><span> </span><span>buildPhase</span><span> </span><span>=</span><span> </span><span>''</span></code></p><p><code><span> # NOTE: this link could be pulled into its own derivation for even better seperation</span></code></p><p><code><span> # I only just realized this while making this post...</span></code></p><p><code><span> $LD -r -L </span><span>${</span><span>rustBuildName</span><span>}</span><span>/lib </span><span>${</span><span>cBuildName</span><span>}</span><span>/</span><span>${</span><span>cHostDest</span><span>}</span><span> -lhost -o </span><span>${</span><span>host_dest</span><span>}</span></code></p><p><code><span> mkdir -p $out</span></code></p><p><code><span> cp </span><span>${</span><span>host_dest</span><span>}</span><span> $out/</span><span>${</span><span>host_dest</span><span>}</span></code></p><p><code><span> cp -r </span><span>${</span><span>rocCode</span><span>}</span><span>/. $out</span></code></p><p><code><span> ''</span><span>;</span></code></p><p><code><span>}</span></code></p><p></p>
|
||||
```
|
||||
|
||||
Now this is about as good as you can get. The rust and c builds have no dependency on each other so you are free to modify the c without needing to rebuild the rust.
|
||||
|
||||
If you only change the roc code, you won't need to do any build (other than linking but in this example that could also be pulled out....).
|
||||
- Wrap up
|
||||
|
||||
Huge shoutout to [Silvan Mosberger](https://github.com/infinisil) from [Tweag](https://www.tweag.io/) for bringing file sets to the main Nix library. He was sponsored through my current employer [Antithesis](https://antithesis.com/) to develop this feature!
|
||||
|
||||
I hope filesets make more people take a stab at making their own `*2nix` builders, or just make their own builds more efficent.
|
||||
|
||||
I had a lot of fun working on [roc2nix](https://github.com/JRMurr/roc2nix/), it was my first time making a "real" nix library and I learned a lot along the way (not just file sets). If you are interested in learning more about it check out the repo or let me know in the comments and I might make a separate blog diving into that (and hopefully some blogs on roc itself).
|
||||
@@ -1,213 +0,0 @@
|
||||
---
|
||||
created: 2023-12-10T12:53:42 (UTC -05:00)
|
||||
tags:
|
||||
- nixos
|
||||
source: https://gist.github.com/misuzu/89fb064a2cc09c6a75dc9833bb3995bf
|
||||
author: misuzu
|
||||
---
|
||||
|
||||
- Install NixOS on Oracle Cloud over Ubuntu 18.04
|
||||
|
||||
> ## Excerpt
|
||||
> Install NixOS on Oracle Cloud over Ubuntu 18.04. GitHub Gist: instantly share code, notes, and snippets.
|
||||
|
||||
---
|
||||
- Install NixOS on Oracle Cloud over Ubuntu 18.04 (make sure to use Ubuntu 18.04 or this may not work)
|
||||
|
||||
```shell
|
||||
# install useful tools
|
||||
sudo apt-get update
|
||||
sudo apt-get install --no-install-recommends -y nano mc git
|
||||
|
||||
# prepare /boot
|
||||
sudo umount /boot/efi
|
||||
sudo mv /boot /boot.bak
|
||||
sudo mkdir /boot/
|
||||
sudo mount /dev/sda15 /boot
|
||||
sudo mv /boot/* /boot.bak/efi/
|
||||
|
||||
# use swap file
|
||||
sudo dd if=/dev/zero of=/swapfile bs=1M count=1024 status=progress
|
||||
sudo chmod 600 /swapfile
|
||||
sudo mkswap /swapfile
|
||||
sudo swapon /swapfile
|
||||
|
||||
# install nix
|
||||
sh <(curl -L https://nixos.org/nix/install)
|
||||
. $HOME/.nix-profile/etc/profile.d/nix.sh
|
||||
nix-channel --add https://nixos.org/channels/nixos-21.11 nixpkgs
|
||||
nix-channel --update
|
||||
|
||||
# install nixos-generate-config and nixos-install
|
||||
nix-env -f '<nixpkgs>' -iA nixos-install-tools
|
||||
|
||||
# generate config
|
||||
sudo `which nixos-generate-config` --root /
|
||||
|
||||
# remove lxc mounts
|
||||
sudo nano /etc/nixos/hardware-configuration.nix
|
||||
# set hostname, add users and ssh-keys, enable openssh
|
||||
sudo nano /etc/nixos/configuration.nix
|
||||
|
||||
# build config
|
||||
nix-env -p /nix/var/nix/profiles/system -f '<nixpkgs/nixos>' -I nixos-config=/etc/nixos/configuration.nix -iA system
|
||||
|
||||
# prepare target
|
||||
sudo chown -R 0.0 /nix
|
||||
sudo touch /etc/NIXOS
|
||||
sudo touch /etc/NIXOS_LUSTRATE
|
||||
echo etc/nixos | sudo tee -a /etc/NIXOS_LUSTRATE
|
||||
|
||||
# install NixOS
|
||||
sudo NIXOS_INSTALL_BOOTLOADER=1 /nix/var/nix/profiles/system/bin/switch-to-configuration boot
|
||||
|
||||
sudo reboot
|
||||
```
|
||||
- Recommended configuration options
|
||||
|
||||
```nix
|
||||
{
|
||||
# Oracle Cloud uses EFI boot
|
||||
boot.loader.systemd-boot.enable = true;
|
||||
boot.loader.efi.canTouchEfiVariables = true;
|
||||
|
||||
# Kernel cmdline from Ubuntu config
|
||||
boot.kernelParams = [
|
||||
"console=ttyS0"
|
||||
"console=tty1"
|
||||
"nvme.shutdown_timeout=10"
|
||||
"libiscsi.debug_libiscsi_eh=1"
|
||||
];
|
||||
|
||||
# Load graphics driver in stage 1
|
||||
boot.initrd.kernelModules = [ "bochs_drm" ];
|
||||
|
||||
# swap file is recommended
|
||||
swapDevices = [
|
||||
{
|
||||
device = "/swapfile";
|
||||
priority = 0;
|
||||
}
|
||||
];
|
||||
}
|
||||
```
|
||||
- Repartitioning target system from kexec image
|
||||
|
||||
Create `kexec.nix` file with following contents (do not add any packages to `environment.systemPackages` or it won't boot on 1GB system):
|
||||
|
||||
```nix
|
||||
{ config, pkgs, ... }:
|
||||
{
|
||||
imports = [
|
||||
# this will work only under qemu, uncomment next line for full image
|
||||
# <nixpkgs/nixos/modules/installer/netboot/netboot-minimal.nix>
|
||||
<nixpkgs/nixos/modules/installer/netboot/netboot.nix>
|
||||
<nixpkgs/nixos/modules/profiles/qemu-guest.nix>
|
||||
];
|
||||
|
||||
# stripped down version of https://github.com/cleverca22/nix-tests/tree/master/kexec
|
||||
system.build = rec {
|
||||
image = pkgs.runCommand "image" { buildInputs = [ pkgs.nukeReferences ]; } ''
|
||||
mkdir $out
|
||||
cp ${config.system.build.kernel}/${config.system.boot.loader.kernelFile} $out/kernel
|
||||
cp ${config.system.build.netbootRamdisk}/initrd $out/initrd
|
||||
nuke-refs $out/kernel
|
||||
'';
|
||||
kexec_script = pkgs.writeTextFile {
|
||||
executable = true;
|
||||
name = "kexec-nixos";
|
||||
text = ''
|
||||
#!${pkgs.stdenv.shell}
|
||||
set -e
|
||||
${pkgs.kexectools}/bin/kexec -l ${image}/kernel --initrd=${image}/initrd --append="init=${builtins.unsafeDiscardStringContext config.system.build.toplevel}/init ${toString config.boot.kernelParams}"
|
||||
sync
|
||||
echo "executing kernel, filesystems will be improperly umounted"
|
||||
${pkgs.kexectools}/bin/kexec -e
|
||||
'';
|
||||
};
|
||||
kexec_tarball = pkgs.callPackage <nixpkgs/nixos/lib/make-system-tarball.nix> {
|
||||
storeContents = [
|
||||
{
|
||||
object = config.system.build.kexec_script;
|
||||
symlink = "/kexec_nixos";
|
||||
}
|
||||
];
|
||||
contents = [ ];
|
||||
compressCommand = "cat";
|
||||
compressionExtension = "";
|
||||
};
|
||||
kexec_tarball_self_extract_script = pkgs.writeTextFile {
|
||||
executable = true;
|
||||
name = "kexec-nixos";
|
||||
text = ''
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
ARCHIVE=`awk '/^__ARCHIVE_BELOW__/ { print NR + 1; exit 0; }' $0`
|
||||
tail -n+$ARCHIVE $0 | tar x -C /
|
||||
/kexec_nixos $@
|
||||
exit 1
|
||||
__ARCHIVE_BELOW__
|
||||
'';
|
||||
};
|
||||
kexec_bundle = pkgs.runCommand "kexec_bundle" { } ''
|
||||
cat \
|
||||
${kexec_tarball_self_extract_script} \
|
||||
${kexec_tarball}/tarball/nixos-system-${kexec_tarball.system}.tar \
|
||||
> $out
|
||||
chmod +x $out
|
||||
'';
|
||||
};
|
||||
|
||||
boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" ];
|
||||
boot.kernelParams = [
|
||||
"panic=30" "boot.panic_on_fail" # reboot the machine upon fatal boot issues
|
||||
"console=ttyS0" # enable serial console
|
||||
"console=tty1"
|
||||
];
|
||||
boot.kernel.sysctl."vm.overcommit_memory" = "1";
|
||||
|
||||
environment.systemPackages = with pkgs; [ cryptsetup ];
|
||||
environment.variables.GC_INITIAL_HEAP_SIZE = "1M";
|
||||
|
||||
networking.hostName = "kexec";
|
||||
|
||||
services.getty.autologinUser = "root";
|
||||
services.openssh = {
|
||||
enable = true;
|
||||
challengeResponseAuthentication = false;
|
||||
passwordAuthentication = false;
|
||||
};
|
||||
|
||||
documentation.enable = false;
|
||||
documentation.nixos.enable = false;
|
||||
fonts.fontconfig.enable = false;
|
||||
programs.bash.enableCompletion = false;
|
||||
programs.command-not-found.enable = false;
|
||||
security.polkit.enable = false;
|
||||
security.rtkit.enable = pkgs.lib.mkForce false;
|
||||
services.udisks2.enable = false;
|
||||
i18n.supportedLocales = [ (config.i18n.defaultLocale + "/UTF-8") ];
|
||||
|
||||
users.users.root.openssh.authorizedKeys.keys = [
|
||||
# add your ssh key here
|
||||
"ssh-ed25519 ...."
|
||||
];
|
||||
}
|
||||
```
|
||||
|
||||
Build kexec image (you'll need nix/NixOS installed on your machine):
|
||||
|
||||
```shell
|
||||
nix-build '<nixpkgs/nixos>' -A config.system.build.kexec_bundle -I nixos-config=./kexec.nix
|
||||
```
|
||||
|
||||
Copy tarball to remote machine using `scp` and reboot into kexec image:
|
||||
|
||||
```shell
|
||||
scp ./result ubuntu@somehost:/tmp/kexec
|
||||
ssh ubuntu@somehost -t sudo /tmp/kexec
|
||||
# wait for machine to boot and then connect
|
||||
ssh root@somehost
|
||||
```
|
||||
|
||||
Repartition your drive, format, mount file systems, create swap(file) and activate it as soon as possible. Check [manual](https://nixos.org/nixos/manual/index.html#sec-installation-partitioning) for more info.
|
||||
@@ -1,129 +0,0 @@
|
||||
---
|
||||
created: 2024-01-17T10:25:58 (UTC -05:00)
|
||||
tags: [nixos]
|
||||
source: https://www.stackbuilders.com/blog/self-contained-scripts-with-nix/
|
||||
author: Written by:
|
||||
---
|
||||
|
||||
- Stack Builders - Self-contained Scripts With Nix
|
||||
|
||||
> ## Excerpt
|
||||
> The blog post discusses the benefits of Nix in creating reproducible environments where developers can focus on developing new features and delivering value to the users. More specifically, it’s focused on the idea of self-contained scripts, which is one of the most basic use cases for Nix where we can see the benefits it brings to the table and the endless possibilities of the tool.
|
||||
|
||||
---
|
||||
The blog post discusses the benefits of Nix in creating reproducible environments where developers can focus on developing new features and delivering value to the users. More specifically, it’s focused on the idea of self-contained scripts, which is one of the most basic use cases for Nix where we can see the benefits it brings to the table and the endless possibilities of the tool.
|
||||
|
||||
---
|
||||
|
||||
**_“Nix is a tool that takes a unique approach to package management and system configuration.”_** \- nixos.org
|
||||
|
||||
The goal here is pretty simple: help teams create reproducible environments where they can focus on creating value through developing new features.
|
||||
|
||||
Reproducible environments abstract some of the complexity of setting up a project, which is usually detailed in the first page of its repository, and explains how to set up the machine before doing any work. This configuration might very well conflict with that of other projects, be outdated, or simply not work on a particular architecture or OS (Operating System).
|
||||
|
||||
With this in mind, we are introducing Nix in our projects without developers even noticing it - their workflow stays the same after the introduction of Nix. This incremental approach increases the abstraction level with every step, improving the parity between the environments that an application goes through in the development process.
|
||||
|
||||
Nix has different use cases, but in the context of this blog post, we would like to focus on self-contained scripts as a quick introduction to some of the benefits offered by Nix, which revolve around the ideas of isolation and reproducibility.
|
||||
- Introduction to Nix
|
||||
|
||||
Before we go any further, let's get started with a brief introduction of Nix, which usually refers to a bunch of things like the package manager, the language, or the Nixpkgs (Nix packages) library. These concepts will be described in the following sections:
|
||||
- Nix Package Manager
|
||||
|
||||
The Nix package manager is fully functional and has a few benefits over common ones like apt or Homebrew. For example, being multi-platform, supporting side-by-side installation of multiple versions of a package or being hermetic, which prevents packages from breaking during installation and upgrades.
|
||||
|
||||
While most operating systems expect packages in /usr/local/bin or other global paths, Nix places them in the Nix store, which usually lives under /nix/store, and then plays around with the PATH variable to make those tools accessible from everywhere on the system.
|
||||
|
||||
A package example could be: `/nix/store/<ID>-firefox-33.1`, where ID is a cryptographic hash unique for this particular version of the Firefox package that captures all its dependencies. This enables Nix to do a few things:
|
||||
- Hold multiple versions of the same package as every little change to the package will generate a completely different hash, preventing collisions.
|
||||
- Perform atomic installs and updates, where nothing is overridden. Nix will only change paths when the package is fully installed. This no-override feature facilitates that different packages use different versions of their dependencies and simplifies rollbacks as nothing gets deleted.
|
||||
- Nix language
|
||||
|
||||
The Nix language is used to declare packages and configurations to be built by Nix. It is also purely functional and lazily evaluated.
|
||||
- Nixpkgs library
|
||||
|
||||
Finally, Nixpkgs is a [repository](https://github.com/NixOS/nixpkgs) that holds every package available in the Nix ecosystem. The branches are called channels which group different packages and versions together based on their stability, amount of packages, etc. Those packages are defined using the Nix language and have been built from source with a full dependency tree. After being built, resulting binaries are hosted on a [binary cache](https://cache.nixos.org/) allowing end users to download and use them without the need of rebuilding them locally.
|
||||
- Self-contained scripts
|
||||
|
||||
After this introduction, let’s dive into the main topic of this post and one of the most basic use cases for Nix, self-contained scripts.
|
||||
|
||||
Scripts are useful to automate tasks that otherwise would have to be executed by hand. These scripts can be written in a variety of languages, but even the most basic Bash scripts depend on certain commands to be present at run-time. Some of these basic utilities are available in most operating systems, however, differences in versions or flavors of the same tool (GNU vs non-GNU tools) present a challenge for maintainers who want to create portable scripts.
|
||||
|
||||
As you might imagine, this looks like a recipe for disaster, and the most common way to solve this problem is by either adding a lot of complexity to a script to try to handle all different edge cases or relying on extensive documentation that is overlooked most of the time. To make things worse, even if we are able to capture the full dependency tree of the script and pin every package to a specific version, the next user will have a hard time replicating that environment, as the average package manager is not flexible enough to pinpoint an exact version for every package.
|
||||
|
||||
The end result of all this scripting is a very fragile piece of code which “works on my machine”, at least until you update one of its mutable dependencies, breaking it for every user. Let’s see how self-contained scripts with Nix can help us to avoid this issue with an example of a regular Bash script:
|
||||
|
||||
```text
|
||||
#!/usr/bin/env bash
|
||||
echo "---AWS CLI---"
|
||||
echo "Version:" "$(aws --version)"
|
||||
echo "Location:" "$(which aws)"
|
||||
echo "---jq---"
|
||||
echo "Version:" "$(jq --version)"
|
||||
echo "Location:" "$(which jq)"
|
||||
```
|
||||
|
||||
Now check the Nix version of it:
|
||||
|
||||
```text
|
||||
#!/usr/bin/env nix-shell
|
||||
#!nix-shell -i bash -p awscli2 jq
|
||||
#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-22.11.tar.gz
|
||||
echo "---AWS CLI---"
|
||||
echo "Version:" "$(aws --version)"
|
||||
echo "Location:" "$(which aws)"
|
||||
echo "---jq---"
|
||||
echo "Version:" "$(jq --version)"
|
||||
echo "Location:" "$(which jq)"
|
||||
```
|
||||
|
||||
Assuming you have [Nix installed](https://nixos.org/download.html), you can run the above script as you would do with any other bash script and it will use its own, self-contained dependencies to execute.
|
||||
|
||||
This is possible thanks to the first three lines, which individually do the following:
|
||||
- `#!/usr/bin/env nix-shell` is a shebang, which tells the system to run the script with nix-shell, one of Nix’s tools
|
||||
- `#!nix-shell -i bash -p awscli2 jq` is doing a few things:
|
||||
- `i bash` sets the real interpreter, which is the one ultimately running the script
|
||||
- `p awscli2 jq` sets which packages will be included in the shell
|
||||
- The last line is pinning a specific nixpkgs channel, in this case, nixos-22.11
|
||||
|
||||
After this, we have our usual bash code which will make use of every package available in the system, but will give priority to those specified in the nix-shell. This is a source of “impurity”, but our code only uses these two packages (awscli2 and jq), which are properly isolated from the rest, including their own dependencies.
|
||||
|
||||
Another point worth mentioning is the ability to pin a specific revision of Nixpkgs, as with the current example, package versions will change over time as the channel is updated. To do that, we would replace the last line with:
|
||||
|
||||
```text
|
||||
#!nix-shell -I
|
||||
|
||||
nixpkgs=https://github.com/NixOS/nixpkgs/archive/3c75992f01290979c9c1f997e40efa77845bef1a.tar.gz
|
||||
```
|
||||
|
||||
Compared to the line in the example script, this one has a specific commit hash, preventing packages from changing over time unless we manually update the channel. The downside of this approach is that old package versions may be removed from the remote binary cache, forcing us to compile them locally from source, causing extensive build times. Depending on the use case we might choose one option or the other.
|
||||
|
||||
The other major language for scripting, Python, is also compatible with this kind of shell:
|
||||
|
||||
```text
|
||||
#!/usr/bin/env nix-shell
|
||||
#!nix-shell -i python -p python37 pythonPackages.prettytable
|
||||
#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-22.11.tar.gz
|
||||
import prettytable
|
||||
# Print a simple table
|
||||
t = prettytable.PrettyTable(["N", "N^2"])
|
||||
for n in range(1, 10): t.add_row([n, n * n])
|
||||
print(t)
|
||||
```
|
||||
|
||||
In this case, we are including a specific version of python (3.7), and some python packages (prettytable) which we would normally install with pip. This way we skip the usual steps of:
|
||||
- Using pyenv to select python 3.7
|
||||
- Creating a virtual environment with venv
|
||||
- Activating this virtual environment
|
||||
- Installing dependencies with pip
|
||||
- Running the script
|
||||
- Deactivating the environment and changing python version again
|
||||
- Conclusions
|
||||
|
||||
As we can see, introducing these self-contained scripts in our workflows requires very little effort from the development team, which only needs to have Nix installed, but brings lots of benefits in terms of reproducibility, isolation and ease of use. We can tweak these parameters to accommodate the project’s needs, for example, by being more strict with the pinning or allowing a higher level of impurity. Most advanced use cases of Nix, like development environments, bring the same benefits to the table but instead of just a script, they encapsulate an entire setup, from dependencies to environment variables and everything in between, helping developers reduce the overhead of configuring and switching between different projects while ripping the benefits of reproducibility, isolation and portability.
|
||||
|
||||
Published on Mar. 21, 2023
|
||||
|
||||
|
||||
- Subscribe to the Blog
|
||||
|
||||
Join our community of avid readers and stay informed with the latest articles, tips, and insights delivered straight to your inbox. Don't miss out on valuable content – subscribe now and be part of the conversation!
|
||||
@@ -1,25 +0,0 @@
|
||||
---
|
||||
tags:
|
||||
- nixos
|
||||
---
|
||||
https://github.com/nix-community/home-manager/issues/1107
|
||||
|
||||
|
||||
```nix
|
||||
{ config, pkgs, ... }:
|
||||
|
||||
let
|
||||
pkgsUnstable = import <unstable> {};
|
||||
in
|
||||
|
||||
{
|
||||
nixpkgs.overlays = [
|
||||
(final: previous: {
|
||||
skim = pkgsUnstable.skim;
|
||||
})
|
||||
];
|
||||
|
||||
programs.home-manager.enable = true;
|
||||
programs.skim.enable = true;
|
||||
}
|
||||
```
|
||||
@@ -1,14 +0,0 @@
|
||||
---
|
||||
tags:
|
||||
- nixos
|
||||
---
|
||||
|
||||
GIT PULL
|
||||
`mr update`
|
||||
|
||||
git fetch
|
||||
`mr fetch`
|
||||
`mr -d ~/projects/nomad fetch`
|
||||
|
||||
git add *
|
||||
`mr -d ~/projects/nomad run git add --all`
|
||||
@@ -1,7 +0,0 @@
|
||||
---
|
||||
tags:
|
||||
- nixos
|
||||
---
|
||||
```bash
|
||||
docker login ghcr.io -u sstent -p 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
|
||||
```
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
tags: [[nixos]]
|
||||
---
|
||||
|
||||
wsl --import NixOS_2305_2 .\NixOS_2305\ Downloads\nixos-wsl.tar.gz --version 2
|
||||
|
||||
- Start a shell with GIT
|
||||
```
|
||||
sudo su -
|
||||
|
||||
nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/release-23.05.tar.gz -p git -p bitwarden-cli
|
||||
```
|
||||
- login to bitwarden
|
||||
|
||||
```
|
||||
bw config server https://vault.fbleagh.duckdns.org
|
||||
|
||||
BW_SESSION=`bw login stuart.stent@gmail.com 'RadiantlyIdeologyAskew!StagnatePortholePogo?' --raw` && export BW_SESSION
|
||||
|
||||
```
|
||||
- Export Secret Zero
|
||||
|
||||
```
|
||||
bw get attachment ssh_host_ed25519_key --itemid 90113413-e518-4853-9ddd-45a45b3d882d
|
||||
bw get attachment ssh_host_ed25519_key.pub --itemid 90113413-e518-4853-9ddd-45a45b3d882d
|
||||
sudo cp ssh_host_ed25519_key* /etc/ssh/
|
||||
sudo chmod 0644 /etc/ssh/ssh_host_ed25519_key.pub
|
||||
sudo chmod 0600 /etc/ssh/ssh_host_ed25519_key
|
||||
```
|
||||
- Clone the repo
|
||||
|
||||
```
|
||||
git clone https://sstent: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@github.com/sstent/vmimages.git
|
||||
```
|
||||
- generate host ed25519 key
|
||||
|
||||
```
|
||||
sudo ssh-keygen -q -N "" -t ed25519 -f /etc/ssh/ssh_host_ed25519_key
|
||||
nix-shell -p ssh-to-age --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'
|
||||
cp /mnt/c/Users/stuar/masteragekey ~/.config/sops/age/keys.txt
|
||||
|
||||
sops updatekeys hosts/wsl2/secrets.yaml
|
||||
```
|
||||
- generate user ed25519 key
|
||||
```
|
||||
ssh-keygen -q -N "" -t ed25519
|
||||
cat ~/.ssh/id_ed25519_key.pub | ssh-to-age
|
||||
```
|
||||
- backup hostkeys to tar.gz in base64
|
||||
```
|
||||
sudo tar -cz /etc/ssh/ssh_host_ed25519_key* | base64 -w0
|
||||
```
|
||||
Paste result in Keypass
|
||||
- restore hostkeys
|
||||
- GO3
|
||||
```
|
||||
echo H4sIAAAAAAAAA+3TW4+aQBQAYJ75FbybrVxmUB6aFHDKooF1FS/lxTgrixWUFUYEfn13SPvQdtcmjTFper6EGcJMhpM550TsqVsUW/6stlnBVtFGxVgxVklUC1civ9JluZ3l32dZV3qCgrQexljFqirICtI1LEjytQK45FSwdS5JQp5l7NK+P63/o+44iziuLz2MiT+d3kvjiTs3AyKNyJd2VaSaldKD36xtJQ+XCdvsiMlZDi7pfsZfCd0bp/DR/MHig3fmI2ue1JiFC/Gx9oNZ9cA32fZzv/aWi26CvPQ5UkpZ34765dGtUtNyq8zA88+9oN/RHTTihwztxiXr+6K2HHH706l/e6jIg7ajJcsHQUd+8eIk0g12RNPznKJznSelmSXRcF0vwklp1GHYbdxqQztBXM+N3qIbHEledzJx58xMsnN2WjpbsnHTy2ic8/jsw7Ck2sSk+xRRzbPM2PsottdJ/MH7V33z/EcX+v/Dy4le4x9tjyP0bv8rGvql/zFSEfT/Lbzm/e57yqW2brW2z9Nw4Cp+QDD/5g7fLv63al/i9/Tp8LXKitsXMwAAAAAAAAAAAAAAAAAAAAAAAADAf+gbTLOLhgAoAAA= | base64 -d | tar -tzv
|
||||
```
|
||||
- STUPC
|
||||
```
|
||||
sudo su -
|
||||
echo H4sIAAAAAAAAA+3TX4+aQBAAcJ/5FLxfrLvyR3y4pIusFBEOBVR4MaKeepyisOgdn763pk2attqkMSZN55ewS9jNMtmZWbJ5oyjW/Jmus4JNl4umouD2NF2+124EfVAROs/o1xmpklTDsoIxUhW5JdcQllUJ1UR0qwCuKQs2y0WxlmcZu7bvT+v/qDqnU9NyxSePur7/RfSG1ogEVLRpdF4VEkl/TXZuNevgPJ6kbPFCCaebyjHZhvyVJtt2GQ/IdzofnBMfWTVvrlg8FgbvbhC+PfFNHX1UVLqtDwOtZQfuJG7YD4ehMYu7ctogxljVel40OrQamxeTn9HbzHxyWG3TlS1k5MdT//ZQgQfdeY6fmwtPLudMXUeKYrNk28/MxFsNDD1KtbiyUNg2N7bZ744rh2bdrlcVewcvdlqu7qm7F9LJJtCIm+SntL0e58cH24l4fMaud0ykIQmlIQ6pw0bYdcjAMnQSmvrp8VE4Xy11jcvXfpf8L6/0/6d9mdziH+cel+WL/Y9l9FP/KzJWof/v4SPv9W8pF3nddqRzn7/GhoXdgCr8m3Wh+H9X+yK/p88+K71Ofez371PEAAAAAAAAAAAAAAAAAAAAAAAAAAD/sa+dz1X7ACgAAA==| base64 -d | tar -xzvl -C /
|
||||
|
||||
exit
|
||||
```
|
||||
|
||||
|
||||
cd vmimages
|
||||
sudo nixos-rebuild switch --flake .#StuPC-WSL
|
||||
- login as me
|
||||
|
||||
```
|
||||
su - sstent
|
||||
```
|
||||
- wierd fix
|
||||
```
|
||||
mkdir -p ~/.local/state/nix/profiles
|
||||
```
|
||||
|
||||
|
||||
```
|
||||
echo H4sIAAAAAAAAA+3S3YqbUBAHcK99Cl+g7Tl+Jhe9UGM0aWLWmKjxpsRoappqWD2uG5++e4RCKbQXhd1S+P8uZg7MMAzDed+25YdL/rnIZU2jU+E1kBc6ITxTQxszoao65rFGDYGqimFQxdBVXSBU1okiSORVtvlF17JjI0lC27KiZn/qK5r2LRZ6W+84y3EXvrR5cPww9KSH7SIyd470yTmMVTFTrG9Z7Q9HmzZpcmX5V8fkLFd7yqo9fzpZNe3SwPzB4mHd88iGk/yFpbEY3P3d/nnDm2zbkyeb29Xd1Yxcr7Ffn5p5fIiq6cTS5X30PNiGN/fUYpUMfMbSeswUmx2raS82wc9T/3aoyJeeaZlxp8F8XU1Sd3lbLc+G2y9nemdmxPTzpGThJa7q6PEWL85Zr1+CdFMoYbg9yPm9jdIyfrlN7zXpNvHWq/a0P5cqO6/5fjPPH3I36nLH6o5J+XTqnY/ieE7Hn/3+1P/6OwAAAAAAAAAAAAAAAAAAAAAAwH/kO6Ga0/sAKAAA | base64 -d | tar -xzvl -C /home/sstent/
|
||||
```
|
||||
|
||||
|
||||
|
||||
cd vmimages
|
||||
home-manager switch --flake .
|
||||
- Clone the repo into user dir
|
||||
|
||||
```
|
||||
git clone https://sstent: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@github.com/sstent/vmimages.git
|
||||
```
|
||||
@@ -1,82 +0,0 @@
|
||||
---
|
||||
tags:
|
||||
- nixos
|
||||
---
|
||||
|
||||
wsl --import NixOS_2305_2 .\NixOS_2305\ Downloads\nixos-wsl.tar.gz --version 2
|
||||
|
||||
- Start a shell with GIT
|
||||
```
|
||||
sudo su -
|
||||
|
||||
nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/release-23.05.tar.gz -p git -p bitwarden-cli
|
||||
```
|
||||
- login to bitwarden
|
||||
|
||||
```
|
||||
bw config server https://vault.fbleagh.duckdns.org
|
||||
|
||||
BW_SESSION=`bw login stuart.stent@gmail.com 'RadiantlyIdeologyAskew!StagnatePortholePogo?' --raw` && export BW_SESSION
|
||||
|
||||
```
|
||||
- Export Secret Zero
|
||||
|
||||
```
|
||||
bw get attachment ssh_host_ed25519_key --itemid 90113413-e518-4853-9ddd-45a45b3d882d
|
||||
bw get attachment ssh_host_ed25519_key.pub --itemid 90113413-e518-4853-9ddd-45a45b3d882d
|
||||
cp ssh_host_ed25519_key* /etc/ssh/
|
||||
chmod 0644 /etc/ssh/ssh_host_ed25519_key.pub
|
||||
chmod 0600 /etc/ssh/ssh_host_ed25519_key
|
||||
```
|
||||
- Clone the repo
|
||||
|
||||
```
|
||||
git clone https://sstent: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@github.com/sstent/vmimages.git
|
||||
```
|
||||
- IF NEW HOST - generate host ed25519 key
|
||||
- NIXOS REBUILD
|
||||
```
|
||||
cd vmimages
|
||||
sudo nixos-rebuild switch --flake .#StuPC-WSL
|
||||
```
|
||||
- populate User Keys
|
||||
|
||||
```
|
||||
|
||||
bw get attachment id_rsa --itemid 3e48341a-fccb-492b-9e67-1955d5936407 --output /home/sstent/.ssh/id_rsa
|
||||
bw get attachment id_rsa.pub --itemid 3e48341a-fccb-492b-9e67-1955d5936407 --output /home/sstent/.ssh/id_rsa.pub
|
||||
|
||||
bw get attachment id_rsa_git --itemid 9b057d35-86c5-4d41-895c-35c2cf5e259b --output /home/sstent/.ssh/id_rsa_git
|
||||
bw get attachment id_rsa_git.pub --itemid 9b057d35-86c5-4d41-895c-35c2cf5e259b --output /home/sstent/.ssh/id_rsa_git.pub
|
||||
|
||||
bw get attachment id_ed25519 --itemid 267e8af2-1caa-4323-8dd2-bc7f3fd8e8b6 --output /home/sstent/.ssh/id_ed25519
|
||||
bw get attachment id_ed25519.pub --itemid 267e8af2-1caa-4323-8dd2-bc7f3fd8e8b6 --output /home/sstent/.ssh/id_ed25519.pub
|
||||
|
||||
chown sstent:users /home/sstent/.ssh/*
|
||||
chmod 0600 /home/sstent/.ssh/*
|
||||
ls -lan /home/sstent/.ssh/
|
||||
```
|
||||
- login as me
|
||||
|
||||
```
|
||||
su - sstent
|
||||
```
|
||||
- wierd fix
|
||||
```
|
||||
mkdir -p ~/.local/state/nix/profiles
|
||||
```
|
||||
|
||||
|
||||
```
|
||||
echo H4sIAAAAAAAAA+3S3YqbUBAHcK99Cl+g7Tl+Jhe9UGM0aWLWmKjxpsRoappqWD2uG5++e4RCKbQXhd1S+P8uZg7MMAzDed+25YdL/rnIZU2jU+E1kBc6ITxTQxszoao65rFGDYGqimFQxdBVXSBU1okiSORVtvlF17JjI0lC27KiZn/qK5r2LRZ6W+84y3EXvrR5cPww9KSH7SIyd470yTmMVTFTrG9Z7Q9HmzZpcmX5V8fkLFd7yqo9fzpZNe3SwPzB4mHd88iGk/yFpbEY3P3d/nnDm2zbkyeb29Xd1Yxcr7Ffn5p5fIiq6cTS5X30PNiGN/fUYpUMfMbSeswUmx2raS82wc9T/3aoyJeeaZlxp8F8XU1Sd3lbLc+G2y9nemdmxPTzpGThJa7q6PEWL85Zr1+CdFMoYbg9yPm9jdIyfrlN7zXpNvHWq/a0P5cqO6/5fjPPH3I36nLH6o5J+XTqnY/ieE7Hn/3+1P/6OwAAAAAAAAAAAAAAAAAAAAAAwH/kO6Ga0/sAKAAA | base64 -d | tar -xzvl -C /home/sstent/
|
||||
```
|
||||
|
||||
|
||||
|
||||
cd vmimages
|
||||
home-manager switch --flake .
|
||||
- Clone the repo into user dir
|
||||
|
||||
```
|
||||
git clone https://sstent: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@github.com/sstent/vmimages.git
|
||||
```
|
||||
@@ -1,171 +0,0 @@
|
||||
---
|
||||
Article Title: Nix from the bottom up
|
||||
Original Link: http://www.chriswarbo.net/projects/nixos/bottom_up.html
|
||||
wallabag_link: https://wallabag.fbleagh.duckdns.org/view/75
|
||||
tags:
|
||||
- nixos
|
||||
created_at: 2024-01-26T13:30:37+0000
|
||||
published_at:
|
||||
---
|
||||
This page describes what [Nix](https://nixos.org/manual/nix/stable/introduction) fundamentally _is_, at its core; what it is actually doing when it runs; and why it makes the choices that it does. This won’t tell you what commands you need to run, or which of the Nixpkgs override functions you should call to swap out that broken dependency you’ve found; but hopefully it will give you a clearer picture of what’s happening, what’s possible, and maybe steer you in the right direction.
|
||||
|
||||
To begin, I like to say that the _point_ of Nix is the following:
|
||||
|
||||
**GET SPECIFIED OUTPUTS**
|
||||
|
||||
The key, of course, is to understand what I mean by these words. That’s what the rest of this page is about!
|
||||
|
||||
- What is an output?
|
||||
|
||||
This is pretty easy: an “output” is just a file or folder. Nix puts all outputs in the “Nix store”, which is usually the folder `/nix/store`. For example, my phone has an output with the path `/nix/store/sqm5miynrjc5sw0zbnkvr9281xp043iw-firefox-120.0`: it is a folder which happens to contain a copy of the Firefox browser (containing an executable at `bin/firefox`, an icon image at `share/icons/hicolor/64x64/apps/firefox.png`, etc.).
|
||||
- How do we specify an output?
|
||||
|
||||
There are two ways to specify an output: using its hash, or using a derivation.
|
||||
- Outputs specified by hash
|
||||
|
||||
These use two pieces of information: a name (for human readability), and a hash of the file/folder contents. These are combined into the output’s path, using the pattern `<store>/<hash>-<name>`; e.g. `/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh`. This is a neat trick: paths are the standard way to refer to files/folders, so putting hashes in output paths ensures every reference specifies the exact content that it’s referring to!
|
||||
|
||||
There are two downsides to this approach:
|
||||
- There is no feasible way to invert a hash, so there’s no way for Nix to create such outputs if they are not found.
|
||||
- The only way to know a hash is to calculate it from existing data (or have someone else do that and tell you the result), so this approach cannot specify _new_ outputs that don’t yet exist.
|
||||
|
||||
This approach to specifying outputs is mostly useful as a cache/database of things we already have; when we want a “content address” for files and folders. For example, if we’re going to compile multiple copies of some source code then giving identical versions the same path avoids redundant compilation; whilst those with differing content should be kept at distinct paths so we can compile each individually.
|
||||
- Outputs specified by derivation
|
||||
|
||||
A “derivation” is a text file in the Nix store, following the same `<store>/<hash>-<name>` pattern as the previous approach, but whose name always ends in `.drv`. These files must contain certain fields, including a “system” (such as `x86_64-linux`), the path of an executable (like `/bin/sh`), a list of string arguments to give to that executable (say, `["-c", "echo RUNNING >&2 && echo $message > $out"]`) and a key/value map of strings to use as environment variables (e.g. `("message", "hello")`). Together these fields are essentially specifying an `exec` syscall; and that’s exactly how Nix will use them, to _create_ outputs that don’t already exist. This overcomes the first problem identified with the hash-only approach.
|
||||
|
||||
Derivation files must specify at least one output, with each having a name and a path. Like with the previous approach, these paths have the form `<store>/<hash>-<name>`, although this time the hash is calculated from the fields of the derivation file: that way, we don’t have to know the _contents_ of an output ahead of time, only _a command to create it_; hence overcoming the second problem.
|
||||
|
||||
Derivations may optionally specify “inputs”, which are just references to other outputs, specified either by a path (which contains a hash), or as a named output of some other derivation (specified by the path to its `.drv` file, which again contains a hash). This way, `.drv` files can refer to each other to form a directed acyclic graph; and since each path contains a hash, these dependencies form a Merkle tree which completely specifies the entire dependency graph of each derivation. This is similar to how the ID of a git commit depends on its content, and that content includes the commit IDs of its parents; hence giving every git commit a tamper-proof specification of its entire history.
|
||||
- How do we get an output?
|
||||
|
||||
Now we know how outputs are specified, it should be pretty clear how Nix can get them (I already said, it can run the specified executable!). The algorithm Nix follows is sketched below, and is known as “building” or “realising” an output.
|
||||
|
||||
All outputs have a filesystem path, either specified directly (when we’re relying on the hash that occurs in the path) or written in a specified `.drv` file. The first thing Nix will do is check if the specified output’s path already exists on the filesystem: if it does, no further action is needed and we’re finished!
|
||||
|
||||
If the specified output _doesn’t_ already exist on our system, Nix can query _other_ systems to see if they have it. These are called “binary caches” or “remote stores” (depending on how they’re set up), and are usually part of our system config (e.g. a default Nix installation will use `cache.nixos.org` as a binary cache). If one of these returns a hit for the specified output path, Nix will copy its contents to the local filesystem, and is then finished. This lets us decouple _specification_ from _implementation_: for example, the Nixpkgs project tends to specify derivations with large dependency graphs, with inputs being as detailed and fine-grained as the particular patches to apply to the source of the GCC compiler that is used to build the shell that is used to run the tests of the library that… and so on! However, we don’t need to _run_ all of those steps, since Nix will just fetch the output we asked for from a cache!
|
||||
|
||||
If there were no cache hits, and the output is only specified by its hash, then Nix must abort at this point, since there’s no way to re-create the desired file/folder given only its hash.
|
||||
|
||||
The interesting case happens when outputs of a _derivation_ are not found, since we must run the derivation’s command. Before that, Nix must get all of the derivation’s inputs (each of which, remember, is some other output): that’s right, this procedure is recursive!
|
||||
|
||||
Once all of the inputs exist, Nix will run the derivation’s executable with the given arguments and environment (this usually happens in a sandbox, depending on the system configuration, but _that’s not important_; the idea of Nix is **orthogonal** to the idea of containers, although they complement each other well)!
|
||||
|
||||
When the executable has finished, Nix will check whether all of the output paths now exist on the filesystem: if not, it aborts with an error message. Otherwise, Nix is now finished.
|
||||
|
||||
**Note:** We do not need to trust remote systems when copying outputs specified by hash (since Nix will verify the hash after copying). However, copying the output of a derivation requires some trust (either in the remote system itself, or a key that’s signed its contents). This is because verifying the contents would require us to run the command ourselves, and compare the result; yet the entire point of a cache is to avoid having to run things ourselves!
|
||||
- Examples
|
||||
- Specifying with a hash
|
||||
|
||||
We can use `nix-store --add` to put an existing file/folder into our Nix store. This returns its path, which we can use to access it like any other file:
|
||||
|
||||
$ echo 'hello' > greeting.txt
|
||||
$ nix-store --add greeting.txt
|
||||
/nix/store/5cil4z0s59ii1splw7bhxf230bfdxfq5-greeting.txt
|
||||
$ rm greeting.txt
|
||||
$ cat /nix/store/5cil4z0s59ii1splw7bhxf230bfdxfq5-greeting.txt
|
||||
hello
|
||||
|
||||
Since the path contains a hash of the content, adding the same content (with the same name `greeting.txt`) will always give the same path; yet different content will give an utterly different path:
|
||||
|
||||
$ echo 'goodbye' > greeting.txt
|
||||
$ nix-store --add greeting.txt
|
||||
/nix/store/q4x6d9w3x7wx1d2rx18n28sfbss4b9nw-greeting.txt
|
||||
$ cat /nix/store/q4x6d9w3x7wx1d2rx18n28sfbss4b9nw-greeting.txt
|
||||
goodbye
|
||||
$ echo 'hello' > greeting.txt
|
||||
$ nix-store --add greeting.txt
|
||||
/nix/store/5cil4z0s59ii1splw7bhxf230bfdxfq5-greeting.txt
|
||||
$ cat /nix/store/5cil4z0s59ii1splw7bhxf230bfdxfq5-greeting.txt
|
||||
hello
|
||||
- Specifying a derivation
|
||||
|
||||
Derivations are a great idea, but they’re pretty tedious to write by hand (especially calculating all the required hashes). Instead, Nix comes with a simple scripting language (the “Nix expression language”) for generating `.drv` files for us. Here’s a Nix expression for a simple derivation, which I’ll save in a file called `example.nix` (note that this isn’t an output or `.drv` file, so it doesn’t need to live in the Nix store):
|
||||
|
||||
derivation {
|
||||
name = "myName";
|
||||
system = "aarch64-linux";
|
||||
builder = "/bin/sh";
|
||||
args = [ "-c" "echo RUNNING >&2 && echo $message > $out" ];
|
||||
message = "hello";
|
||||
}
|
||||
|
||||
This expression is calling a built-in function called `derivation` with a single argument: a “set” (think JSON object) containing a bunch of “attributes” (keys). Note that we’re not specifying any output names, so the `derivation` function will default to using a single output called `out`, which will also appear as an environment variable (hence why our Bash snippet is writing to `$out`). If you’re not on an `aarch64-linux` machine, you can replace that string with the expression `builtins.currentSystem` instead!
|
||||
|
||||
We can evaluate this Nix expression to produce a `.drv` file by using the `nix-instantiate` command:
|
||||
|
||||
$ nix-instantiate example.nix
|
||||
warning: you did not specify '--add-root'; the result might be removed by the
|
||||
garbage collector
|
||||
/nix/store/i762zk23lrfsz8fjfd4lbjh48073hmlh-myName.drv
|
||||
$ cat /nix/store/i762zk23lrfsz8fjfd4lbjh48073hmlh-myName.drv
|
||||
Derive([("out","/nix/store/zcgax4c4wfvby6p06dwjl8cc4dvkvypr-myName","","")],[],
|
||||
[],"aarch64-linux","/bin/sh",["-c","echo RUNNING >&2 && echo $message > $out"],
|
||||
[("builder","/bin/sh"),("message","hello"),("name","myName"),("out",
|
||||
"/nix/store/zcgax4c4wfvby6p06dwjl8cc4dvkvypr-myName"),("system","aarch64-linux")
|
||||
])
|
||||
|
||||
Unfortunately Nix was created before the ubiquity of JSON, so its `.drv` format may look unfamiliar. Thankfully the `nix derivation show` command will translate it for us (albeit with a warning message, which we can ignore!):
|
||||
|
||||
$ nix derivation show /nix/store/i762zk23lrfsz8fjfd4lbjh48073hmlh-myName.drv
|
||||
warning: The interpretation of store paths arguments ending in `.drv` recently
|
||||
changed. If this command is now failing try again with
|
||||
'/nix/store/i762zk23lrfsz8fjfd4lbjh48073hmlh-myName.drv^*'
|
||||
{
|
||||
"/nix/store/i762zk23lrfsz8fjfd4lbjh48073hmlh-myName.drv": {
|
||||
"args": [
|
||||
"-c",
|
||||
"echo RUNNING >&2 && echo $message > $out"
|
||||
],
|
||||
"builder": "/bin/sh",
|
||||
"env": {
|
||||
"builder": "/bin/sh",
|
||||
"message": "hello",
|
||||
"name": "myName",
|
||||
"out": "/nix/store/zcgax4c4wfvby6p06dwjl8cc4dvkvypr-myName",
|
||||
"system": "aarch64-linux"
|
||||
},
|
||||
"inputDrvs": {},
|
||||
"inputSrcs": [],
|
||||
"name": "myName",
|
||||
"outputs": {
|
||||
"out": {
|
||||
"path": "/nix/store/zcgax4c4wfvby6p06dwjl8cc4dvkvypr-myName"
|
||||
}
|
||||
},
|
||||
"system": "aarch64-linux"
|
||||
}
|
||||
}
|
||||
|
||||
Hopefully you can see how each part of a derivation I described above appears in this file (although we’ve not specified any inputs, for simplicity). We can “realise” the outputs of this derivation using the `nix-store --realise` command:
|
||||
|
||||
$ nix-store --realise /nix/store/i762zk23lrfsz8fjfd4lbjh48073hmlh-myName.drv
|
||||
this derivation will be built:
|
||||
/nix/store/i762zk23lrfsz8fjfd4lbjh48073hmlh-myName.drv
|
||||
building '/nix/store/i762zk23lrfsz8fjfd4lbjh48073hmlh-myName.drv'...
|
||||
RUNNING
|
||||
warning: you did not specify '--add-root'; the result might be removed by the
|
||||
garbage collector
|
||||
/nix/store/zcgax4c4wfvby6p06dwjl8cc4dvkvypr-myName
|
||||
$ cat /nix/store/zcgax4c4wfvby6p06dwjl8cc4dvkvypr-myName
|
||||
hello
|
||||
|
||||
Notice that the text `RUNNING` appears, which shows that the `/bin/sh` command was indeed executed; and sure enough the output contains the text `hello` which we specified for the `message` environment variable.
|
||||
|
||||
Whilst `nix-instantiate` and `nix-store --realise` are useful to make it clear what’s going on (generating a `.drv` then getting its outputs), in practice we can just use the simpler `nix-build` command:
|
||||
|
||||
$ nix-build example.nix
|
||||
/nix/store/zcgax4c4wfvby6p06dwjl8cc4dvkvypr-myName
|
||||
|
||||
Note that this time we did not get a `RUNNING` message, or anything mentioning “building” or “garbage collectors”. That’s because the output already exists, so Nix did not need to run anything!
|
||||
|
||||
We can make life _even easier_ by renaming our file to `default.nix`: that way, we can run `nix-build` without having to specify any filename!
|
||||
|
||||
$ mv example.nix default.nix
|
||||
$ nix-build
|
||||
/nix/store/zcgax4c4wfvby6p06dwjl8cc4dvkvypr-myName
|
||||
- Going forward
|
||||
|
||||
I’ve described the most basic, fundamental workings of Nix, which hopefully gives you a solid understanding to build upon. However, not only is there much more to the Nix tool itself, but there’s now an entire “ecosystem” built around it, such as Nixpkgs, NixOS, NixOps, Cachix, etc. New features are pushing what’s possible, and simplifying what’s useful.
|
||||
|
||||
If you want to start playing around, I highly recommend using the `nix repl` command, looking through [the Nix manual](https://nixos.org/manual/nix/stable/language/), and following some of the excellent documentation and resources people have put online!
|
||||
@@ -1,31 +0,0 @@
|
||||
---
|
||||
tags:
|
||||
- nixos
|
||||
- overlays
|
||||
---
|
||||
|
||||
- https://github.com/Misterio77/nix-starter-configs/tree/main/standard
|
||||
- can aoveralys be applied flake-wide?
|
||||
- in nixos config `nixpkgs.overlays = builtins.attrValues outputs.overlays;`
|
||||
- in HM `overlays = builtins.attrValues outputs.overlays;`
|
||||
- Shouldn't this autoinherit all overlays?
|
||||
|
||||
```
|
||||
mkPkgs = pkgs: extraOverlays:
|
||||
|
||||
import pkgs {
|
||||
|
||||
inherit system;
|
||||
|
||||
config.allowUnfree = true; # forgive me Stallman senpai
|
||||
|
||||
overlays = extraOverlays ++ (lib.attrValues self.overlays);
|
||||
|
||||
};
|
||||
|
||||
pkgs = mkPkgs nixpkgs [self.overlay];
|
||||
|
||||
pkgs' = mkPkgs nixpkgs-unstable [];
|
||||
```
|
||||
THis looks like it might be the right mixer
|
||||
https://github.com/jboyens/dotfiles/blob/54b33e81daa6a30f101301222757fff87acbb52b/flake.nix#L73:~:text=%23%20flake.nix%20---%20the,133
|
||||
@@ -1,26 +0,0 @@
|
||||
---
|
||||
title: PythonPackage in Nix
|
||||
date: 2024-01-22 18:17:05
|
||||
tags:
|
||||
- nixos
|
||||
---
|
||||
pkgs.python3.override
|
||||
|
||||
https://discourse.nixos.org/t/how-to-use-build-a-python-package-from-a-local-nixpkgs-repo/31919
|
||||
|
||||
https://discourse.nixos.org/t/how-to-add-custom-python-package/536
|
||||
|
||||
|
||||
```nix
|
||||
pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
|
||||
(
|
||||
python-final: python-prev:
|
||||
{
|
||||
bindep = python-final.callPackage ./packages/python3/bindep { };
|
||||
ansible-builder = python-final.callPackage ./packages/python3/ansible-builder { };
|
||||
}
|
||||
# infinite recursion
|
||||
# (mapModules ./packages/python3 (p: python-final.callPackage (lib.debug.traceVal p) { }))
|
||||
)
|
||||
];
|
||||
```
|
||||
@@ -1,31 +0,0 @@
|
||||
---
|
||||
created: 2024-01-26T15:43:17 (UTC -05:00)
|
||||
tags:
|
||||
- nixos
|
||||
source: https://serverfault.com/questions/890886/recover-from-nixos-rebuild-test-without-a-reboot
|
||||
author: |-
|
||||
aij
|
||||
20311 silver badge77 bronze badges
|
||||
---
|
||||
|
||||
- Recover from `nixos-rebuild test` without a reboot - Server Fault
|
||||
|
||||
> ## Excerpt
|
||||
> Is there a way to revert to the previous (or at least "current") configuration after nixos-rebuild test without rebooting?
|
||||
|
||||
Per the nixos-rebuild manual:
|
||||
|
||||
test
|
||||
Build and activate the new
|
||||
|
||||
---
|
||||
```
|
||||
# List previous generations
|
||||
sudo nix-env --list-generations -p /nix/var/nix/profiles/system
|
||||
|
||||
# Live switch to any generation
|
||||
sudo nix-env --switch-generation 12345 -p /nix/var/nix/profiles/system
|
||||
sudo /nix/var/nix/profiles/system/bin/switch-to-configuration switch
|
||||
```
|
||||
|
||||
Source: [https://github.com/NixOS/nixpkgs/issues/24374](https://github.com/NixOS/nixpkgs/issues/24374)
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
tags:
|
||||
- nixos
|
||||
---
|
||||
```bash
|
||||
nix-shell -E '
|
||||
let
|
||||
pkgsA = (import (builtins.fetchTarball https://github.com/NixOS/nixpkgs/archive/141439f6f11537ee349a58aaf97a5a5fc072365c.tar.gz) {});
|
||||
pkgsB = (import (builtins.fetchTarball https://github.com/NixOS/nixpkgs/archive/7d7622909a38a46415dd146ec046fdc0f3309f44.tar.gz) {});
|
||||
in
|
||||
pkgsA.mkShell {
|
||||
buildInputs = [
|
||||
pkgsA.ktlint
|
||||
pkgsB.jq
|
||||
];
|
||||
}'
|
||||
```
|
||||
@@ -1,8 +0,0 @@
|
||||
---
|
||||
tags:
|
||||
- nixos
|
||||
---
|
||||
|
||||
|
||||
----
|
||||
[High-speed 10Gbps full-mesh network based on USB4 for just $47.98 – Fang-Pen's coding note](https://fangpenlin.com/posts/2024/01/14/high-speed-usb4-mesh-network/ )
|
||||
@@ -1,19 +0,0 @@
|
||||
---
|
||||
tags:
|
||||
- windows
|
||||
---
|
||||
|
||||
|
||||
Including link: https://superuser.com/ \"The settings of the Windows 10 touch keyboard are found under the registry key HKEY_CURRENT_USER\SOFTWARE\Microsoft\TabletTip\1.7.
|
||||
|
||||
Value EnableCompatibilityKeyboard is 1 for enabling switching to the full layout.
|
||||
Value KeyboardLayoutPreference controls the default opened layout, your value might be 1.
|
||||
|
||||
Value DisableNewKeyboardExperience is 1 for the old keyboard style.
|
||||
|
||||
Value UserKeyboardScalingFactor is the touch keyboard size, a number between 20 (smaller) to 200 (larger).
|
||||
|
||||
All the values are of type DWORD 32.
|
||||
|
||||
The first point might be what you are looking for, but the others may also be useful.\"
|
||||
https://superuser.com/questions/1787479/how-to-enable-full-touch-keyboard-layout-on-windows-10-touch-screen-machine#:~:text=The%20settings%20of,also%20be%20useful.
|
||||
@@ -1,9 +0,0 @@
|
||||
---
|
||||
title: Flemmar
|
||||
date: 2024-01-22 11:15:47
|
||||
tags: [[aar]]
|
||||
---
|
||||
|
||||
- https://github.com/Flemmarr/Flemmarr/blob/master/config/flemmarr/config.yml
|
||||
- export current configs?
|
||||
- requires https://github.com/Flemmarr/Flemmarr/pull/14
|
||||
@@ -1,8 +0,0 @@
|
||||
---
|
||||
title: Lidarr
|
||||
date: 2024-01-22 11:17:15
|
||||
tags: [[aar]]
|
||||
---
|
||||
|
||||
- https://github.com/Lidarr/Lidarr
|
||||
http://lidarr.audio/
|
||||
Reference in New Issue
Block a user