# Standup for Ocean Sprint 2023 Projects link to the content: https://pad.lassul.us/os23-projects # Day 5 - final stand-off @lassulus - talking about mergebot with Raito - we went through a lot of sections of the future design documentation @Mic92 - tpm/yubikey support in sops - people bugged me with sops "when tpm2" - he did it this morning - PR scraping of nixpkgs: - state: finished (≥ 200K PRs) !!!! - "just upload it again on github" - Mic92: please people, mirror it for me - With git, we can do even more! TODO @Raito: - Woke up late because of the very late party until 3.30AM - Work meeting - Sat down with flokli and discussed future of various things surrounding nixpkgs - Sat down with Lassulus for the design documentation - reviewer automatic selection - how to have an activity log - how to have a """"""""""""trusted set of maintainers""""""""""""""""" that can have more privileges with the mergebot - how to have a web UI to see things related to mergebot administration - Sat down with Julien and found the secret bug for his TPM2 bug in UEFI: he forgot to use `OVMFFull` instead of `OVMF` which does not initialize TPM2 in UEFI @Tom: - String in Nix - what is proved is that it requires foundational factoring - enough progress for working builds with all the refactored code - don't have the original result aimed for - significant progress and paths to success are now opened - nerdsnipped a certain number of people to be interested into solving Nix codebase issues - Swimming - thank you - Mic92 & hsjobeki is prized as the one who attended to every day - Foundation will start **FUNDING** potentially **EFFORTS** on **Nix codebase**, please join us! (hint hint nudge nudge) @phaer - Confirmed & annouced the first nixos.at meetup since 6 months (26th of October) in **Vienna** - Wooooooooo - Went through @chaoflows dream2nix & python setup - Took the first look into pyproject.nix & dream2nix since nixcon #ngi :sweat_smile: - pyproject.nix is a new parser for pyproject.toml - connection between parser builtins in Nix and pyproject.nix @blitz: - Tom's emotional support - agreed that it would be much easier if the C++ was refactored in Nix - Systemd closure minimization - still in discussion: https://github.com/NixOS/nixpkgs/pull/261798 - lanzaboote janitor - when your boot items looks sane and don't confuse users and ordered normally @alejandrosame - Created draft PR for release notes generator script: https://github.com/NixOS/nixpkgs/pull/262296 - insane stuff — this is a Nix static generator for Markdown files @hsjobeki - Nodejs Builder - Made (almost) everything configurable. - Building more complex stuff. Trying to package "prettier" which is relatively complex. - Prettier as an target example @Enzime - got QEMU building with Apple Virtualization support - tried running a macOS VM inside - QEMU asserts and panics @tfc/jacek: - spent the morning helping a team of summer-of-nixers that @fricklerhandwerk connected with me. Turns out some projects use `e2fsprogs` for a library called libcom_err, and the libkrb5 package also uses this language, but provides its own hardcopy of it which is extremely old. So there are to competing libcom_err.so libraries... Apparently libkrb5 can be configured with `--with-system-et` that would fix that, but as a global change this introduces circular deps and doesn't work. So they use this as a local override to their package's input deps. I'm happy I could help. - will continue running linux-builder in a wrapper script instead nix-darwin's version. However, connecting to builders that run on localhost with custom SSH ports seems difficult. @duijf - Got a bit burnt out with the testing of my sandboxing stuff. - I'm on a macOS… - Ended up chatting with flokli about macOS Virtualization.Framework - Playing with macOS Virtualization.Framework @SomeoneSerge - Side-tracked scraping [Hercules CI](https://hercules-ci.com/github/SomeoneSerge/nixpkgs-cuda-ci) in order to figure out how many cpu-hours it takes on average to build a (subset of) nixpkgs revision. Motivation: reference for an OpenCollective application - Considered reaching out to Robert Hensing - It is already enough to use the Swagger to get data out - But if it's needed, reach out to Robert in the future potentially - A copy of all the build logs - Tom: "I dugged into that too" - Kranzes: "ongoing optimization are being done that will change the performance characteristics about Hercules CI soon" - [GPUs](https://github.com/NixOS/nixpkgs/pull/256230) still need reviewers! - https://github.com/NixOS/nixpkgs/pull/261720 needs approvals or a merge too - Discovered that `nix bundle --bundler github:NixOS/bundlers#toReport .#cudaPackages.cuda-samples` is not a satisfiable way to inspect licenses (yet), because before you can see the license report, you have to accept the `allowUnfree = true` license - I'm now unblocked with the ingestion benchmark for tvix-castore, but I need time @m1-s - continued/finished work on trying to integrate external network traffic with nixos test driver. final solution does not require any changes to the nixos test driver. will write down what I learned. - wooooooooooo @lheckemann: - Slept in and missed the morning dip :( - Switched to systemd stage 1, struggled with systemd services once again and found some bugs - wooooooooooooooo - Butterflied around providing little bits of help - Next Munich NixOS meetup soon?? - Merged some PRs @nikstur: - Replaced some more activationScripts @flokli: - started bumping nixos config: test nsncd PR from NinjaTrappeur/picnoir - debugged some overlay not applying defaults from the module system, results inconclusive - found a aarch64-linux compilation failure in nsncd and reported back - got bitten by systemd credential chmod to ACL change *again*, poked garage people and opened an [issue](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/658) - was annoyed about copying back and forth too much stuff, might tackle nix daemon protocol next to make that faster (we have unix socket support!) let me know who's interested - tvix: helped Someone debug criterion benches - tvix: started sketching out some more output path calculation tests to get more coverage for CAHash refactor (hello.outPath works though, so maybe it's fine ?!) - gerrit-queue PR merged, sent CL for TVL repo - finally brought diving equipment to servicing guy, who will get spare parts this evening `(ಥ⌣ಥ)` @alejandrosame & @fricklerhandwerk: - Move reference manuals to nix.dev - Discussed PR and next steps for Nix Manual - 50% for Nixpkgs and NixOS Manual @fricklerhandwerk: - Yesterday I talked to Jonas, Domen, and Tom about solving long-standing issues with money - Refined a funded project proposal with Tom for the Nix code base @zupo: - tesh: a last Ocean Sprint's, tesTING shell sessions in README files - testing documentation quick start - a lot of nice features and inspired from existing art (mdsh, etc.) - fixed tesh multiline support on NixOS: https://github.com/OceanSprint/tesh/issues/48 - getting close with migrating tesh to flakes - still a few things to flesh out before production usage @Jezen: - systemd secure thing - abandoned parser - systemd-analyze security already provides a json output - and I use this - I managed to get the client and server to communicate - Progress @sandydoo: - deploying flakestry on fly.io - Mic92: "it's super flakey all the time" @Chloe: - work crap today @Kranzes: - helped with Mic with the sops-nix age plugins # Day 4 lassulus: - fucked up dnssec on lassul.us - pad.lassul.us was down - except for smart people who uses resolved without DNSSEC validation like Raito - more nixpkgs-merge-bot stuff - started a design document to submit to community for the nixpkgs-merge-bot before doing anything production-related - green light for dry run mode from community (no drama anymore) - faster evaluation Mic92: - when lassul.us was force pushing all the time to the mergebot - I had to fix the buildbot - On nix-community, there are github bugs and things are very slow on the gh org - I wrote a scraper for all nixpkgs PRs which I don't know why but Raito knows - Is this allowed? "It's allowed for archived reasons" - We also the GDPR validation from our GDPR person (IANAL) Arian: - Growpart now isch guut: t - APPROVED - only to merge Raito: - QMP API (thanks to Jacek): - So, the NixOS test framework cannot assert things like "wait for reboot", "wait for shutdown", "wait for panic" - It makes it hard to test very early stuff, e.g. UEFI firmware crashing, "this image won't boot" or unexpected issues - The QEMU Machine API (QMP) offers some facilities to tap into internal information about QEMU, including VM state - Pairing it with https://github.com/qemu/qemu/blob/master/docs/specs/pvpanic.txt — you can ask things like "did my VM panic?" if you wire it to your firmware / kernel / whatever layer - But also, it is very useful to say: "I will wait for a reboot" (from, e.g. UEFI firmware) - Here's an example of usage in Lanzaboote: https://github.com/nix-community/lanzaboote/pull/229 - Some stuff is hard to achieve because QMP API is inherently async, NixOS test framework is sync, we need to do some smart bookkeeping of the events to avoid just waiting infinitely for events and hanging the test - Work on @Linus's https://github.com/NixOS/nixpkgs/pull/249556/files weird `assert /boot present failed` bug - Deep dive into installer tests legacyness - Impossible to get a proper shell because of how the NixOS test framework behaves with installer tests - Stuff just ended up hanging for no reason - I ragequitted - Work on @nikstur's asking "How can I mount thing beneath something else" for upgrading the /etc racelessly - https://lwn.net/Articles/927491/ - *wears the Linux hat* Since Linux 6.5, we have https://kernelnewbies.org/LinuxChanges#Linux_6.5.Allow_to_move_mounts_beneath_top_mount - But I thought util-linux 2.39 had it in `mount --beneath` - Unfortunately: <some screenshot of discussion with Christian Brauner and Daan de Meyer> - Anyway, we have https://github.com/brauner/move-mount-beneath we can use in NixOS Enzime: - Work on running macOS inside QEMU on M1 Macs - Creator of firecracker wrote a patch to make this work™ - Tried to build the patch - It builds but could not get it to test yet - They didn't implement enough to run the macOS installer yet - Usecase: in the future, to be able to legally use Rosetta, you have to use Apple's virtualization framework, we can use Rosetta inside NixOS and that can accelerate x86_64 emulation on aarch64 - Add cross architecture support to `linux-builder` e.g. `x86_64-linux` NixOS VM on `aarch64-darwin` - linux-builder is a Linux VM for macOS so you can build x86_64-linux VMs and run them - Review some PRs hsjobeki: - Some documentation fixes (with Tom help). - `?` operator has Complexity O(log(n)) - Discovered that `genericClosure` also accepts lists -> Now Documented it. - `genericClosure` accepts lists which could be way more performant on huge depdency graphs (nodejs). - generic closure takes a start set with a key and creates more (…) list of sets, it stops when no new item are produced - you can use it to remove the cycles in a dependency graph, because in nodejs, they could be cycles - we only include the nodes that are connected in the graph, e.g. dev dependencies - Improvement already built into it. - Node builder is working™ quite well. Working on exposing interfaces now. alejandrosame (Release Editor for 23.11 & 24.05): - Current release notes can be autogenerated from split files. Working on adding CI checks to eventually prevent contributors from editing the release notes directly. - To make people stop to the big file - Nobody should touch the big file anymore - Now, CI check to make it automatically generated Valentin: - Made progress on ripping out the Nixpkgs and NixOS manuals into nix.dev and still render them with the same styling to avoid disruption - It works in principle but still a lot of work left to get it into a state where it can be evolved phaer: - quite sleep-deprived/hangover - continued to work on buildbot. Only really got started with buildbot-nix-specific things today. - small fixes to nixos-anywhere docs - quickstart - answered github issues, i.e. feedback on https://github.com/NixOS/nixpkgs/pull/255023/files Someone: - "[GPUs in the sandbox](https://github.com/NixOS/nixpkgs/pull/256230)" needs reviews - Extra thoughts: could try setting up PCI passthrough for NixOS tests - Onboarding with flokli (goal: benchmarks; progress: stuck with infra) - onboarding on TVL monorepo (depot) and learning how to use Gerrit (I assume) - how to add benchmarks to Tvix? - a benchmark to evaluate the impact of different string implementations w.r.t. to nixpkgs evaluation - a benchmark for ingesting stuff in the CA store - not yet written but here are the goals - some annoying technical issues to get started - then surfing Flokli: - Refactor `CAHash` code in `nix-compat` crate, and dealing with the fallout in `store_path::utils` - Goal: make output path calculation more understandable, merge some of the separate `text` hashing codepaths - Validate mental model of CAHashes with @Ericson2314 and @tomberek. - TL;DR: hash mode recursive is a bad name - Start writing up some design doc with edef regarding tvix-castore verified streaming of blobs - allows more granular substitution, and faster detection of bad data - Rebased WIP `drvvis` CL on top of latest tvix changes. - two interactive graphical visualizers for derivation spiderwebs Gabriel: - Remote signing support for secure boot in Lanzaboote - Solved a dynamic dispatch problem in Rust. - Parsing some ASN.1 stuff in Rust for EC crypto stuff which is somehow not yet implemented in Rust. - Raged against ASN.1. - Cool discussion with Tom on accelerators with FPGA (not related to NixOS tho) - I think it's related to NixOS - In Minecraft, you have an equivalent to electronics, called Redstone - You want to simulate Redstone really fast, so there's a JIT-optimizing Minecraft server (MCHPRS) - I want to replace this JIT with a FPGA to reconfigure automatically the FPGA live - Gabriel already built an almost RISC-V CPU in Minecraft - Climbing 🧗!!! - Hopefully some more ~~work~~ hacking during the night. sandydoo: - Sniped by @zupo: worked on a PR for pre-commit-hooks.nix that automatically adds packages for every enabled commit hook to the shell. - Lets you set up your linters/formatters once and get a pre-configured shell for your favourite editor. - <3 - It's hard because you need a hashmap of linters → package name representations - DavHau: May I point out that mkShell are not composable? But if you were using dream2nix, you could have composable shells! - Tvix-related: - Explored various filesystem visualization approaches: graphs vs trees vs sunburst charts. Perhaps we need a mix of approaches: sunburst charts for filesystem exploration and a graph view for dependencies. <iframe width="100%" height="784" frameborder="0" src="https://observablehq.com/embed/96a8efe9ad43f55f@366?cells=chart"></iframe> <iframe width="100%" height="1235" frameborder="0" src="https://observablehq.com/embed/96a8efe9ad43f55f@366?cells=radialChart"></iframe> tom: - still working on the abstraction to allow for easier experimentation with alternative string representations. Things compile, but don't link `:(` WorkingInProgress™ - discussed CA hash naming schemes and semantics with flokli - builtins.genericClosure, acceleration, MPI support in builders - MPI: Message Passing Interface, set up Nix in such a way that instead of communicating to a single remote builder that you configure statically in advance - You can ask a slurm cluster to allocate a certain number of remote builders - And then you can run workloads on a certain number of machines that will share their computation and progress towards it (job style stuff in High Performance Computing) - Maybe the solution is to relax the sandbox and use build hooks - Reconsider maybe the derivation model is sufficient for this idea - You have a definition of a worker that is not just "execute this process", it is a bit more collaborative, so you can use it for training (machine learning), e.g. if memory is not enough, you can spread it across all the resources you have instead of buying more RAM - Flokli: you should have both — spreading your problems and collect your outputs - Important reference work https://github.com/jbedo/static-nix/blob/main/slurm-submit.patch - O(log(n)) for lookup operator - If someone wants to do C++ debugging, it is happening right now in this corner - Curious if we do a similar transformation (ropes) to Tvix nikstur: - Developed plan to: - remove activationScripts in the boot flow with the removal of the scripted initrd - Raito: Ambitious - enable perlless activation for now when the systemd initrd is enabled - Would work with https://switch-to-configuration.pl (should be the only thing you use with Perl) - remove as many activationScripts from nixpkgs quickly as long as they can be easily replace via systemd-tmpfiles, preStart or dedicated services - Niklas, the janitor - Fixed a few issues for perlless activation - Trying to figure out how to work with switch-to-configuration - Mount tucking to atomically replace the /etc directory on `switch-to-configuration switch` Jacek/tfc: - Attempted to create a wrapper around nix that launched the linux-builder on macOS so that macOS users don't need nix-darwin. - Didn't work well because there seems to be no nice way to tell the --builders ssh-ng://bla thingy a port number. - Patching nix to accept an SSH port number now, fixing other things on the way, PR open. JulienMalka: - systemd-boot backend (`systemd-boot-builder.py`) - adapted to conform to bootspec (RFC125), no one did that before (Raito hates Python scripts) - I work on the boot.initrd.secrets true encryption feature - For this, you need to boot a stub that supports loading systemd-credentials - e.g. systemd-stub and soon lanzastub - Now, there is a feature to enable systemd-stub in your NixOS system - Now, systemd-credentials are generated and get picked up by the stub in the ESP and can use the TPM2 to decrypt the secrets - TL;DR: it boots with systemd-stub, picks up the credentials in the ESP, decrypts them with a TPM2 and pass them to the systemd service manager and you can use them with `LoadCredentialEncrypted` or `LoadCredential` - All is left are systemd bugs now duijf: - systemd sandboxing - I wanted to open a PR, found a small bug in my thing, wrote a test on it, still working on that and polishing - Went climbing, was fun, hardcore climber, Linus too apparently - Did life admin, as always it happens to everyone of us marijanp: - dig into nix ctrl-c issue '#7245' - Ctrl-C doesn't work with Nix (the worst offense in a program) - There's a reproducer in the issue, got a stacktrace, attached gdb and looking at the code - Basically, during the fetchTree function, it spawns threads or whatever, there's some thread to acquire lock, and deadlock ensues - There's one `while (true) { }`, and it checks whether it's interrupted by the user and it uses an atomic variable and the lock is not acquired lheckemann: - Opened a PR which duplicates existing functionality for Niklas - Raito: Weird flex but ok - What does it do? Limiting the set of things your system can depend on (`system.forbiddenDependenciesRegex`) - Usecase: forbid Perl from system. - Converted Julian to the light side - Emacs is more readable in light theme in sunlight - Flokli: I want something to switch to light mode and dark mode based on some events - Emailed the vfs subsystem maintainer for access to incomplete patches that do magic - Christian Brauner (the VFS subsystem maintainer) - Want to write his first kernel patch - Less significant contributions to Niklas's and Julian's work - Frustrated Raito - Worked on a blog post about building bootable Debian and Ubuntu images in Nix - Raito: <3 - Climbed some walls - Probably forgot some things Julian: - I used this stuff to build appliance images - I managed to defeat to have 2 systemd in my closure (bootstrap stuff) for a benefit of 10MB - I also defeated a 2nd OpenSSL that leaks in the bootstrap for a benefit of 6MB - While my systemd was compiling, I was trying to fix the lanzaboote PR part and pinning EDK2 part m1-s: - How the NixOS manual is built - Split in multiple pages for better SEO Jezen: did stuff related to systemd-analyze security Jean-François: I realized I need push-based metrics inside the nix-daemon, you mentioned it before with Open Telemetry, I am switching my stuff to OTEL now instead of Prometheus Chloe: Worked with syd on Nix CI, secrets manager based on agenix for deployment usage - It leveraged systemd with systemd-credentials - Except, systemd-credentials doesn't have all the nice features - It keeps the lifetime of the secret tied to the service - Called: secrix Zupo: - I find 37 minutes without anyone interrupting me (!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!) - during which I submitted the PR to search.nixos.org to remove nix-env instructions - maniac applause from a certain sprinter, not named for GDPR reasons - fixed local development for search.nixos.org with Raito in that same PR - Nix CI survey? edef: - Design docs for verified streaming blob - I want send a middle level of the BLAKE3 hash - We can only send the leaf nodes, that's already a fair amount of data (3% of the full data) - We want larger chunks - Reimplementing the BLAKE3 internal plumbing to get this feature - Started the Zstandard stuff yesterday - Will continue it later - Discovered that XZ blocks are just one big block instead of multiple blocks, so multi-threaded decompression cannot be used # Day 3 Morning stuff and boat. # Day 2 ## Diving - xx? try scuba divers (...) - lots of fish, **no casualities** - 4 regular divers (flo, flokli, sander, DavHau) - 32m, 60min - 2x Octopus - 3x Sepia - Seahorse - 2 small "caves" and "blue hole" ## Work Arian: … :sleeping: DavHau: package crab.fit frontend and backend -> still wonky -> nextjs server MUST write to the store - next.js needs a cache directory and it has to be in the cwd of the source code hsjobeki: finish the nodejs PR in dream2nix. Basic builder works. - nodejs builder for dream2nix, still WIP and very dumb and it works hsjobeki: was introduced to vim. because pros most use vim. thx enzime. - used VSCode before Gabriel & Ryan: validation of elliptic curve public keys for remote signing support in lanzaboote for HSM use cases - kept working on the same thing as day 1, remote signing for secureboot - reading the specs for different cryptographic formatting & APIs - didn't pay for the 100EUR spec and used the website that should not be named to use it - started implementation flokli + edef: finished up the NAR parser, now docs and unit tests, merge imminent - reviewed by Raito (nerdsnipped) - parse NAR files even faster (!!!!!!!!!!!!!) - cursed flokli: cleaned up (ca)store golang bindings, separared from .proto files, so we have space for more bindings - big mess up with Go/JavaScript bindings stuff (no js yet but soon inchallah) edef + tomberek: talked about zstd chunking & indexing - cache chunking to the current store - folding this into edef's work into how to chunk the cache.nixos.org cache - why? currently, it costs ~7K EUR/mo to host, it is a lot of duplicated data, we will fix it. - for this, we need data on how to compress it optimally and we need S3 metadata edef + zimbatm: set up cache.nixos.org S3 inventory dumps (in Parquet format!) - it unblocks on the previous point regarding S3 deduplication target nikstur + blitz: Perlless activation has been achieved! Now it's polishing time. nikstur converts activation scripts to systemd services. blitz is trying to fix the systemdMinimal sadness. - it works now! - we have tests! - we have Perl scripts in activation (make-etc.pl, etc.), we remove them - we just don't do nonsense - switch-to-configuration removed - we have now a "perlless-profile.nix" and "appliance-profile.nix" - "what is it written in now? systemd is written in nothing!" tomberek: - Strings in Nix. Use an implementation that allows for efficient substring and concatenation. Looking to provide initial proof of benfit via benchmars. Also interested in seeing if this can help in tvix. - evaluation optimization - similar problems to cache.nixos.org - optimize the fuck out of it - ulterior motive for the above is to implement an efficient linear parser of arbitrary grammars (lockfiles, etc) - Packrat parser in Nix as a builtin - "lassulus: jq in nix" - all the parsing of lockfiles is going to be faster etc. - there's a `builtins.fromYAML` in nix — is this a good replacement? - it's not fast enough yet and not viable as a replacement for `builtins.fromYAML` - discussions about cache chunking in the current store and to provide large-scale access to the cache during the "reduce the costs" effort Someone: the "GPUs in the sandbox" PR is now open for reviews; added a NixOS test for the hook; pinged peoples about the CUDA EULA; chatted with Tom about streaming metrics (video frames are "metrics" too, aren't they?) from inside the sandbox in realtime - no action taken yet; first-time dive, 7m, ~40min! - make official a "patchelf" exception to make it legal to redistribute binaries that links aaginst CUDA and make data science nicer for everyone - flake issues on allowUnfree, no mechanism for the user to commit to accept things regarding certain software like CUDA - maybe we should have nix do the licensing or policy checking Valentin and Alejandro: nix.dev madness - Trying to add nix/nixpkgs/nixos manuals in nix.dev. Found several problems with Sphinx and local reload and dev mode. - Julien: what about the issue that some people may feel like nix.dev is a non-official resource whereas *.nixos.org is - Valentin started a prototyping a rabbit hole idea out of frustration: static website generation using NixOS modules. - styx just doesn't cut it Ilan: age-plugin-tpm: 0.1.0 -> m1-s: continue work from day 1, extending nixos test driver vlan to allow for external access jacek/tfc: NixOS integration tests start just fine on macOS with some minor changes - `ulimit -n` is 256 by default on macOS. One change increases that in the config.system.build.vm derivation - works! - requiredSystemFlags "kvm" and "nixos-tests" are not available on macOS - patched these out in case hostsystem is darwin - certain processes inside the NixOS VMs crash (e.g. udevadm on avx instructions, which apple's hw accel does not support). Without HW-accel there's less crashes, so the right qemu parameters need to be figured out. (i'm running on x86_64-darwin) Raito: - I booted with TPM2 now, will document how to do this Laurens: - I continued on the sandboxing stuff, changing up the API (systemd-hardening) - Push as a PR JulienMalka: - Upstreaming features into systemd-stub and systemd-boot - So we can rely on upstream features rather than lanzastub - What's the upside? We share the maintenance cost with systemd folks and are friends with them - "It's also C instead of Rust, so better" Jonas: - Handling the fallout after some controversy - Giving access to edef to the S3 bucket so she can be unblocked on analysis Jean-François: - Prometheus & metrics to Nix: added more operations - Refactoring in C++ is hell, stuck in refactoring hell Paul: - didn't die, but me (Raito) neither - He will try on Buildbot for Gitea, it's a headache, it was a hard day Enzime: - went diving and didn't die too Jezen: - yesterday, Haskell programs for systemd-analyze security for a dashboard and explain it to people - today, went diving Mic92 & lassulus: Ignited new drama in the community 🎉🎉🎉 # Day 1 ## Perlless Activation @nikstur @lheckemann @blitz https://pad.lassul.us/perlless-activation intro: try to remove perl from activation the `make-etc.pl` and `update-users-groups.pl` have to go I want to make switch to configuration optional Why? For appliances image, they don't need rebuilding in an appliance, I also don't want Perl anywhere. Also (Arian) : boot times are very slow, this will make them faster. For /etc, we are mounting an overlayfs, for the user groups, we use systemd-sysusers and for the files, we use the systemd-tempfiles. --- * lassulus did https://github.com/lassulus/sonos-play * Mic92 & lassulus did https://github.com/Mic92/nixpkgs-merge-bot * This bot allows maintainers to merge PRs that touch their packages. It means that now maintainers can react to user feedback and self-service. That should scale much better since there are 3030 maintainers, and only 203 committers. * limited to packages in ./pkgs/by-name * soon: nixos modules, tests ; it works in our staging "environment" * arian and nikstur did [nixos/grow-partition: Resize partition online instead of in initrd](https://github.com/NixOS/nixpkgs/pull/261449) * I want systemd-based initramfs in the next NixOS release * We have a lot of scripts in the initrd, a lot of them were written ~5-7 years ago * We just do online resizing instead of just doing it in the initrd * It works for both (MBR and GPT) but systemd-repart is cooler for GPT * m1-s extended nixos test driver to allow external network connections to be integrated with the VM network * Usecase: in some tests scenarios, you want to execute part of the compute load on actual hardware * and the other part on virtualization and the external network card is the "bridge" to achieve this * It's not impure w.r.t to sandbox but it's kind of impure * Gabriel & Ryan: validation of elliptic curve public keys for remote signing support in lanzaboote for HSM use cases * If you have SecureBoot on your laptop, you have your signing keys on your laptop, that's not really secure * You have a secure device where you store the signing keys and you sign your stub and bootloader on that remote device, like a hardware security module, a TPM or whatever * For that, one of the missing tech part is validating EC public keys * Reviewed 1 CL * @SomeoneSerge is wrapping up [GPUs in the sandbox](https://github.com/NixOS/nixpkgs/pull/256230), currently trying to add Blender tests. They would like to try and make flakes respect licenses in the next days * Scan derivation being built if it got a special marker in the required system features attribute and if it has this, it will expose the files required to access CUDA / graphical stuff APIs * Usecase: I want GPU tests in nixpkgs as a single passthru tests, simple things like "assert pytorch with cuda support runs on a host with CUDA device and torch CUDA is indeed available" * It is something that has been broken many times in the past because we do patchelf fuckery and it would be nice to test this * You can put your high performance computing using GPU inside nix-builds now * @phaer started to look into buildbot & gitea integration; also still has a nicdoc PR open. Happy to do more pairing in the next days * Only supports GitHub for now * Buildbot is an open source CI to define your jobs and stuff in Python scripts, you can have dynamic in a lot of places * How does it compare to Dagger (:sic:) ? It's less licensing violations * The main difference (Julien) is that Buildbot is packaged in nixpkgs instead of Dagger * Isn't Buildbot ancient? There's no other good solution (Mic92) * Enzime and Linus started working on a PoC to show that `trusted-users` in Nix can bypass sudo * Trusted users gives root equivalent access on NixOS systems * I don't think anybody did actually a demo of how to use that feature to get root access * "You don't need SSH access on your machine, it's good" (lassulus) * We would like to pester people to implement a more restricted version of trusted users to sidestep that problem * "Are you going to use that exploit to deploy the fix? No." (Tom) * Enzime, tfc and marijan worked on changing the default for `max-jobs` in Nix to `auto` * Stuff started at Nix Camp * The reason for why it is important is that the current default is 1, anyone who doesn't use NixOS or Darwin, will get only one thread * It's about concurrent derivations * It's really bad if you have thousands derivations to build * Working on the C++ codebase of Nix and tfc/marijan are helping him to tackle this * tomberek and Enzime worked on adding the `--bundler` flag to `nix build` * I would like to express multiple transformation backends, e.g. containers, identity transformation * and you still would like to keep the classical Nix CLI semantics * Do we do a subcommand or do we do something else? * Enzime and DavHau: release v010 of nix-portable: static nix that just works in any environment * They released a new version! No release for like 2 years * Static Nix that works in any environment without having to reconfigure anything * They just did the maintenance thingie * DavHau and Ilan (Kranzes): package crab.fit with dream2nix: FOSS meeting scheduling app with google integration * soon on when.lassul.us * zimbatm - public fastly logs - https://github.com/NixOS/nixos-org-configurations/pull/286 * If you want to know what's going on with the binary cache * If you want to run some analysis, you can access it! * It's anonymized, only the ASN is known (sorry for Ryan) * Jeff: prometheus exporter in nix-daemon * We added a Prometheus exporter so you can get metrics out of the nix-daemon, e.g. builds, etc. * I have to create the PR to make it accepted to Nix * Why? We want to graph anything that is happening, currently, we know very few things about what's going on your nix-daemon * "Yes… !" (Linus) * * "Does it use the cgroups features? Not yet" (Kranzes) * @sandydoo and @domen were working on a project for the last months related to flakes, hub, registry, open source… it will be announced soon (!) * Alejandro & Valentin: * Working on pinning manuals to releases and moving them to nix.dev * Kept working on release notes (he's the release editor for NixOS 23.11 and 24.05) * Ryan: * NixCon NA * Fundraising in the NixOS ecosystem and project * zstd decompression (cache.nixos.org) * Reviewed a lot of CL (change requests) from flokli for Tvix * Ilan * Killed a Flamingo ## tvix * @sandydoo is working on visualizing the tvix store with d3.js * Why visualizing the store? * In Tvix, you have things that can be shared as subtrees, to understand duplication dynamics * Figure out what are nice visualization strategies * @flokli: merged the ATerm parser in nix-compat, fixed the MacOS build for tvix (again), started to do some cross compilation fixes. Want to work on verified Blob streaming protocol and maybe some more smaller fixes (race in import, pagination for (optional) PathInfoService listing). * A .drv file (no syntax highlighting) uses something called ATerm as a syntax * We now have a parser for this in Tvix, so you can write treesitter grammar and all nice stuff! * security framework thingie about macOS * Verified streaming: * You are watching a movie from Netflix, you want to ensure that every frame is part of the actual movie * It's about guarantees about the contents while you stream and not only at the end * Tom managed to find a race in the importer and more misc fixes - JulienMalka: infra work + familiarization with TPM stuff - Infrastructure work for NixOS project (non-critical infrastructure team) - Now Vaultwarden is now available and we ditch the commercial Bitwarden (which was super expensive!) - Now working on TPM2 and find the happy story to unlock their disk with TPM - Computer broken in the process but repaired it ## nodejs * Johannes (hsjobeki) working on a final build system. Design Document @ https://viewer.diagrams.net/?tags=%7B%7D&highlight=0000ff&edit=_blank&layers=1&nav=1#G1q28YLaWeg2Zot-eWjbeZUwKjcoPyrfO- * I am facing an old enemy ("nodejs build systems") * I reimplemented or I have a new build architecture that I implemented in dream2nix * so you can use module system to override anything in the build system and anything in the build system is exposed as an API (mainly for experts) * Regular users should have little or no configuration to make for their projects * Big vision: package every Node.js package from source * Package managers can include multiple package managers, focuses on NPM ## lighthouse-flake * David (DavHau), Johannes (hsjobeki), Marijan (marijanp) packaged google [lighthouse using dream2nix ](https://github.com/nix-community/dreampkgs/blob/main/packages/lighthouse/default.nix) * Flake-Parts module that gives you a build check where you specify performance metrics * Lighthouse is this thing which does web performance checks (SEO, load times, etc.) * With a FP module, you can define the criteria you care about * And then you get a checks output that runs the checks according to your criteria * Packaging Lighthouse took some time (next section) * Marijan (marijanp) did an initial implementation of [lighthouse-flake](https://github.com/marijanp/lighthouse-flake) ## systemd sandboxing options * duijf implemented * I want to make it easier to implement systemd sandboxing on systemd services * By default, NixOS doesn't enable anything, runs everything as root, allow everything * I'm not changing the defaults, but I want to make it easier to have nice things * I came up with a list of stuff that is nice to have enabled by default * I added a NixOS module and Ryan gave me some feedback on the API design * I am still working on that and it works on my homelab setup * `systemd-analyze security` answered 1.6 (!!!!!!!!!) * Chatting with Syd to also have this for the nix-daemon to have sandboxing for that by default * If anyone knows about hardening nix-daemon, please chat with them!