Planet Scheme

November 19, 2019

Programming Praxis

Modified Coin Change

We solved the standard coin change problem in two previous exercises. The particular problem given here is to find the minumum number of coins that can be used to create a target amount of money, given an unlimited number of coins of various denominations, and is normally solved by a dynamic programming algorithm.

Today’s task is a variant on that problem: find the minimum number of coins that can be used to create a target amount of money, given an unlimited number of coins of denomination that are powers of 5.

Your task is to write a program to solve the modified coin change problem. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at November 19, 2019 10:00 AM

November 17, 2019

GNU Guix

Running a Guix Xfce Desktop on CentOS 7

This tutorial will show how to run a fully fledged Xfce desktop environment installed with Guix on top of an existing GNU/Linux distribution. This guide uses CentOS 7 as the base operating system and assumes that Xorg is already configured and running on VT2 under a different user account.

We will borrow Xorg and xinit from the host distribution and run Guix Xfce on virtual terminal 4 as user alice. No system-wide configuration files need to be touched (apart from the Guix install), but we do make a couple of changes for convenience.

From scratch to Xfce

If Guix is not already installed, go grab the installation script and run it as sudo bash guix-install.sh.

The script creates /gnu/store/ and /var/guix/ and configures a system service for guix-daemon. By default the daemon runs from the root users Guix; we won't be using the root account in this guide, so let's start by making the guix-daemon service refer to our local user alice instead.

sudo sed -i 's/root/alice/' /etc/systemd/system/guix-daemon.service

Now every time Alice runs 'guix pull', the daemon gets updated too. If you installed Guix just now, make sure to run guix pull before proceeding further.

Next we'll add some lines to Alices .bash_profile to set up PATH and related variables:

~/.bash_profile:

GUIX_PROFILE="${HOME}/.guix-profile"
[[ -L "${GUIX_PROFILE}" ]] && . "${GUIX_PROFILE}/etc/profile"

export PATH="${HOME}/.config/guix/current/bin:${PATH}"
export INFOPATH="${HOME}/.config/guix/current/share/info:${INFOPATH}"
export MANPATH="${HOME}/.guix-profile/share/man:/usr/share/man"

export XDG_CONFIG_DIRS="${HOME}/.desktop-profile/etc/xdg:${HOME}/.guix-profile/etc/xdg"
export XDG_DATA_DIRS="${HOME}/.desktop-profile/share:${HOME}/.guix-profile/share"

This will look familiar if you have used Guix on a foreign distribution before. The XDG_ variables tell desktop environments where to look for installed programs and things like autostart files: we want minimal interference from the host system, so we "hard code" them to refer to just our Guix profiles.

We will install Xfce and related programs to a separate Guix profile that can be updated and rolled back independently of the main user profile. That allows us to distinguish between "stable desktop environment" and "end user packages". To keep things manageable, we create a manifest for the desktop profile that can be kept in version control, and which allows us to reproduce the exact same environment in the future (even on a different computer!).

~/desktop-manifest.scm:

(specifications->manifest
 '("xfce" "xfce4-session" "xfconf" "xfce4-battery-plugin"
   "pulseaudio" "xfce4-volumed-pulse" "xfce4-notifyd"
   ;; Helpful graphical programs.
   "mousepad" "orage"
   ;; System configuration utilities.
   "xbacklight" "pavucontrol" "stow"
   ;; For HTTPS access.
   "nss-certs"
   ;; These utilities are provided by the host, but we want the Guix versions
   ;; because they are likely better integrated and up to date.
   "fontconfig" "bash-completion" "gnupg" "man-db" "git"))

Create the initial profile generation:

guix package -p ~/.desktop-profile -m ~/desktop-manifest.scm

That installs a union of all packages listed in the manifest to ~/.desktop-profile, and creates a script we will use to "activate" it later. To update this profile, simply invoke the same command again after running guix pull or modifying the manifest.

Before Xfce can be started, we need to create a configuration file for the X server to ensure the host executable is used, and we will tell it to to stay on virtual terminal 4. We also create a .xinitrc script that automatically starts Xfce every time xinit is invoked.

~/.xserverrc:

exec /usr/bin/Xorg -novtswitch -nolisten tcp "$@" vt$XDG_VTNR

~/.xinitrc:

#!/bin/sh

# Get the default xinit configuration for CentOS.
. /etc/X11/xinit/xinitrc-common

exec startxfce4

.xinitrc needs to be executable:

chmod +x ~/.xinitrc

Now let's activate the desktop profile and start the X server, using ":1" as DISPLAY (remember that we have another X server running on VT2, occupying the default ":0" display).

GUIX_PROFILE=~/.desktop-profile
source ~/.desktop-profile/etc/profile
xinit -- :1

Cool, we're in Xfce! Let's open a terminal and install a browser & some fonts:

guix install icecat font-liberation font-dejavu

To make the newly installed fonts available right away we need to invoke fc-cache:

fc-cache -rv

Finally, we'll configure the shell to source scripts installed by Guix so that bash completions and similar work, by adding these lines at the end of .bashrc:

~/.bashrc:

# Source the Guix shell configuration directories, for vte.sh and bash completions.
GUIX_PROFILES=("${HOME}/.desktop-profile"
               "${HOME}/.guix-profile"
               "${HOME}/.config/guix/current")
for profile in "${GUIX_PROFILES[@]}"; do
    for dir in "${profile}/etc/bash_completion.d" "${profile}/etc/profile.d"; do
        if [[ -d "${dir}" ]]; then
            for f in "${dir}"/*; do
                . $f
            done
        fi
    done
done

Phew! It took some work, but by now you should have a working Xfce desktop environment, with bash completions and all. If you are content with starting it manually, skip to "final tweaks" below. Otherwise, read on.

(If you do not have a working desktop after following these steps, please email guix-devel@gnu.org so we can adjust the tutorial!)

Starting Xfce automatically on boot

We can configure our login shell to run xinit every time we log in to VT4 by adding these lines at the end of ~/.bash_profile:

# Start Xorg on display :1 when logged in to VT4, unless DISPLAY is already set.
if [[ -z "${DISPLAY}" && "${XDG_VTNR}" == 4 ]]; then
    GUIX_PROFILE="${HOME}/.desktop-profile"
    source "${HOME}/.desktop-profile/etc/profile"
    exec xinit -- :1
fi

To avoid the need for typing username and password at the console, instruct the getty service for TTY4 to automatically log in user 'alice':

/etc/systemd/system/getty@tty4.service.d/override.conf:

[Unit]
After=graphical.target

[Service]
# Delay for a few seconds, to ensure the Xorg server on VT2 starts first.
ExecStartPre=/bin/sleep 3
ExecStart=
ExecStart=-/sbin/agetty --autologin alice --noclear %I $TERM
Restart=on-success

Now just switching to VT4 will start Xfce! To do this when the system boots, simply enable the getty@tty4 service:

sudo systemctl enable getty@tty4.service

Final tweaks

Some issues were found during usage of the Xfce environment. Launching programs from the file manager failed because gio-launch-desktop was unavailable, and xfce4-terminal complained that the shell function __vte_prompt_command was not found.

These problems will be fixed in Guix eventually, but for now we'll work around them by adding the glib:bin and vte packages to our manifest:

~/desktop-manifest.scm:

(specifications->manifest
 '("xfce" "xfce4-session" "xfconf" "xfce4-battery-plugin"
   ...
   "glib:bin"                  ;for 'gio-launch-desktop'
   "vte"))                     ;for vte.sh, required by xfce4-terminal

We also found that closing the lid would not send the system to sleep, even though xfce4-power-manager --dump showed no problems. To work around it, we told systemd to ignore any "inhibitors" and take care of lid handling itself:

/etc/systemd/logind.conf:

HandleLidSwitch=suspend
LidSwitchIgnoreInhibited=yes

Additionally it is strongly recommended to enable the name service cache daemon if not already running. On CentOS this can be done by:

sudo yum install nscd

Bonus section: Installing programs with a custom build of Qt

One additional issue was that Qt programs did not work due to the stock CentOS kernel being too old. Specifically it lacks the renameat2() system call. Luckily Qt can be configured not to use it. A patch has been submitted to Guix, but since we are in a hurry, we will add a procedure to our manifest so we can use Qt programs (here wpa-supplicant-gui) until the Guix fix is merged:

~/.desktop-manifest.scm:

(use-modules (guix packages)
             (guix utils)
             (gnu)
             (gnu packages admin)
             (gnu packages qt))

(define qtbase/fixed
  (package/inherit
   qtbase
   (arguments
    (substitute-keyword-arguments (package-arguments qtbase)
      ((#:phases phases)
       `(modify-phases ,phases
          (add-after 'unpack 'disable-renameat2
            (lambda _
              ;; Mimic the '-no-feature-renameat2' configure flag.
              (substitute* "src/corelib/configure.json"
                (("config\\.linux && tests\\.renameat2")
                 "false"))
              #t))))))))

(define with-fixed-qt
  ;; This procedure recursively rewrites any references to 'qtbase'
  ;; with our patched version.
  (package-input-rewriting `((,qtbase . ,qtbase/fixed))))

(packages->manifest
 (append (list (with-fixed-qt wpa-supplicant-gui))
         (map specification->package
              '("xfce" "xfce4-session" "xfconf" "xfce4-battery-plugin"
                ...))))

...and now wpa_gui works after installing the new manifest!

Acknowledgements

Special thanks to Ocean Space Acoustics AS for sponsoring this work.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Marius Bakke at November 17, 2019 07:00 PM

November 15, 2019

Programming Praxis

Tri-Partitions

We have a fun little task for your Friday lunch hour:

Given an array of integers, rearrange the integers so no two consecutive integers have a sum that is divisible by 3, or indicate that such an arrangement is not possible.

Your task is to write a program to rearrange an array of integers as described above. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at November 15, 2019 10:00 AM

November 12, 2019

GNU Guix

Spreading the news

Like most free software projects, Guix has no shortage of communication channels: there’s this blog, the NEWS file for release notes, a bunch of mailing lists, an IRC channel, there’s also an unofficial sub-Reddit and certainly more. Yet, as developers, we often find it hard to communicate important changes to our users. Starting from a few weeks ago, guix pull --news tells users what’s new, and it already feels very helpful! This post is about our motivations and the implementation of this new feature.

Getting the word out

Developers keep adding crazy features, fixing bugs, and generally improving things. But how good is it if users aren’t aware of these new things? As an example, since June, our build farm has been offering lzip-compressed binaries, which results in better performance when installing software. But to take advantage of that, users need to be aware of its existence, and they need to upgrade their Guix daemon. Likewise, how do we get people to learn about the new guix deploy command that’s now available at their fingertips, about security issues that were fixed, about important infrastructure changes, new options added to existing commands, and so forth?

Our (frustrating!) experience has been that release notes, blog posts, and mailing list announcements aren’t quite enough to get the word out. There’s always people who’ll miss important info and realize when it’s already late, sometimes too late. Hence this simple idea: wouldn’t it be nice if important information would reach users right in their terminal?

guix pull news

Alright, that’s not exactly a novel idea! In Debian for example, apt-listchanges shows news at the level of individual packages, taken from the NEWS.Debian or changelog.Debian files that package maintainers update. In addition, apt dist-upgrade and similar commands typically display dialog boxes and menus when special actions need to be taken when upgrading. That’s more or less what we’re looking for.

The situation in Guix is a little different: all of it lives in a single Git repository that contains the core, the command-line interfaces, as well as package definitions and operating system service definitions. The guix pull command is sort-of equivalent to apt update, except that it updates not only the set of available packages but also the guix tools themselves and all the operating system interfaces. Under the hood, guix pull essentially does git pull and consequently updates all of this. Guix very much follows a “rolling release” model.

For some time already, guix pull has been able to extract information about new and upgraded packages and to present it to the user. Our goal was to complement it with high-level information about important changes written with users in mind. Clearly, showing the Git commit log is not an option: commit logs are meant for developers and there’s roughly a thousand commits per month—way too much information. We needed high-level news entries, explicitly written for users.

The end result is this: guix pull now displays, in addition to a summary of the new and upgraded packages, the headlines of applicable news entries contributed by developers. Users can view the details by running guix pull --news:

'guix pull' displaying news.

Users can no longer miss the news, for the benefit of both users and developers!

Under the hood

How does this all work? There were several goals and constraints. First, like commit logs, our high-level news entries should be anchored in the Git history. Second, unlike commit logs, it should be possible to amend them—to fix typos, provide additional info, and so on. Third, the project has been paying a lot of attention to internationalization, with translations available for user interface messages, for package descriptions, and for the user manual—it’s one of these things that helps free software reach out to more people; thus, we naturally wanted news to be internationalized. Last, since Guix supports third-party “channels”, which are extensions of the official guix channel, why not provide channel authors access to that news feature?

With all these things in mind, we designed a simple news format. In essence, channel authors, including Guix developers, can provide a news file that looks like this:

(channel-news
 (version 0)

 (entry (commit "3e962e59d849e4300e447d94487684102d9d412e")
        (title (en "@command{guix graph} now supports package
transformations")
               (de "@command{guix graph} unterstützt nun Paketumwandlungen"))
        (body
         (en "The @command{guix graph} command now supports the common package
transformation options (see @command{info \"(guix) Package Transformation
Options\"}).  This is useful in particular to see the effect of the
@option{--with-input} dependency graph rewriting option.")
         (de "Der Befehl @command{guix graph} unterstützt nun die mit anderen
Befehlen gemeinsamen Umwandlungsoptionen (siehe @command{info \"(guix.de)
Paketumwandlungsoptionen\"}).  Sie helfen insbesondere dabei, die Wirkung der
Befehlszeilenoption @option{--with-input} zum Umschreiben des
Abhängigkeitsgraphen zu sehen.")))

 (entry (commit "49af34cfac89d384c46269bfd9388b2c73b1220a")
        (title (en "@command{guix pull} now honors
@file{/etc/guix/channels.scm}")
               (es "Ahora @command{guix pull} tiene en cuenta
@file{/etc/guix/channels.scm}"))
        (body
         (en "The @command{guix pull} command will now read the
@file{/etc/guix/channels.scm} file if it exists and if the per-user
@file{~/.config/guix/channels.scm} is not present.  This allows administrators
of multi-user systems to define site-wide defaults.")
         (es "Ahora la orden @command{guix pull} lee el fichero
@file{/etc/guix/channels.scm} si existe y el fichero personalizable
@file{~/.config/guix/channels.scm} no está presente. Esto permite a quienes
administran sistemas con múltiples usuarias definir valores predeterminados
en el sistema."))))

Each news entry refers to a commit, the commit that introduced the change it documents, and it has a title and body. Those can use Texinfo markup for rich formatting, and translations can be provided directly within the news file.

When guix pull --news runs, it determines which news entries are applicable given the user’s previous Guix instance. The (guix channels) module provides a simple programming interface for that:

(use-modules (guix channels) (srfi srfi-1))

(channel-news-for-commit (first %default-channels)
                         "66b707a7d2325daadeed5ca913637eea3a2628e7"
                         "ac19950507e941b6263f62f4ee4e8934c1b1598e")
⇒ (#<<channel-news-entry> commit: "3e962e59d849e4300e447d94487684102d9d412e" tag: #f title: (("en" . "@command{guix graph} now supports package\ntransformations") …) body: …> #<<channel-news-entry> commit: "49af34cfac89d384c46269bfd9388b2c73b1220a" tag: #f title: (("en" . "@command{guix pull} now honors\n@file{/etc/guix/channels.scm}") …) body: …>)

One thing is quite unusual (one might say “weird” :-)) about this news format: it refers to commit IDs “in-band”. In other words, unlike Git commit logs, which are “out-of-band”, the news file is contained inside the repository that it refers to. Technically, it means that before pushing a news entry, one must make sure it refers to the right commit ID (the news format allows you to refer to tags as well, but one may not want to create tags for every “news-worthy” change). Likewise, rebasing might invalidate commit IDs that appear in the news file. So this whole “in-band” log has drawbacks, but the big win is that it allows us to amend news entries to fix typos, add translations, and so on.

In other news…

Since it was applied a bit more than a month ago, we’ve already put the news mechanism to good use on quite a few occasions: giving users instructions on how to deal with locales after the last glibc upgrade, giving them upgrade info for CVE-2019-18192, telling them about new command-line options, and more.

In parallel, given that reading the mailing lists is akin to “drinking from a fire hose” as they say, Christopher Baines has been thinking about how to provide regular development updates to interested users and developers. Chris announced last week a prototype of a “Guix Weekly News” web site that would aggregate information about package updates automatically extracted from the Guix Data Service, along with manually written updates. It would seem that this service could readily grab info from channel news as well.

What about you, what do you expect in terms of news distribution? Join us on the mailing list and on IRC and let us know!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Ludovic Courtès at November 12, 2019 02:00 PM

Programming Praxis

Reverse A Linked List

We’ve done this before, but I saw questions asked about this task on three different beginner-programming forums this weekend (it must that that time in the semester when teachers introduce linked lists), and it never hurts to do some basic drill.

Your task is to write a program that reverses a linked list. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at November 12, 2019 10:00 AM

November 08, 2019

Programming Praxis

Union Of Two Bags

Today’s exercise is somebody’s homework:

A bag is a list of key/count pairs: for instance, a bag containing three of item E, three of item R, and three of item A can be written E3R3A3. The union of two bags is a single bag containing a list of key/count pairs in which each item in each of the two bags has its count combined in a single item: for instance, the union of bags E3R3A3 and B3R3F3 is E3R6A3B3F3. The order of items in the bags is not significant.

Your task is to write three programs to compute the union of two bags, with time complexities O(mn), O((m+n) log (m+n)), and O(m+n), where m and n are the sizes of the two bags. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at November 08, 2019 10:00 AM

November 05, 2019

Programming Praxis

Encrypting Ed(1)

When I introduced the topic crypt(1) last week my actual goal was to write an editing script using ed on an encrypted file; that’s when I realized that crypt(1) was missing and that ed no longer had an -x flag. I didn’t want anything fancy, just basic encryption. I knew the old crypt(1) was no good (actually, it was no good when it was introduced forty years ago) but I was surprised that Unix and Linux had not kept up with some kind of encryption. Humbug!

Your task is to enhance ed by adding the -x encryption flag. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at November 05, 2019 10:00 AM

November 04, 2019

GNU Guix

Managing Servers with GNU Guix: A Tutorial

The outcome of this year's GSoC is a Guile-based programming interface named guix deploy for automatically creating, upgrading, and changing the configurations of machines running the Guix System. The tool is comparable to Ansible or NixOps, but makes use of the system configuration facilities provided by Guix. A post from earlier this summer described an early version of the programming interface, but we're already a few months into autumn, so it's time for guide on how you can use guix deploy in production.

Simple case: managing a home server

If the machine you need to manage is already running the Guix System, it shouldn't be too hard to incorporate guix deploy into your workflow. All that's needed is the <operating-system> declaration you've been passing to guix system reconfigure and some information about the machine (specifically its IP address and architecture). The guix deploy command is invoked with the filename of a "deployment specification" as an argument, whose contents should look something like this:

;; Module imports
(use-modules (gnu) (guix))
(use-service-modules networking ssh)
(use-package-modules bootloaders)

;; Operating system description
(define os
  (operating-system
    (locale "en_US.utf8")
    (timezone "America/New_York")
    (keyboard-layout (keyboard-layout "us" "altgr-intl"))
    (bootloader (bootloader-configuration
                 (bootloader grub-bootloader)
                 (target "/dev/sda")
                 (keyboard-layout keyboard-layout)))
    (file-systems (cons* (file-system
                          (mount-point "/")
                          (device "/dev/sda1")
                          (type "ext4"))
                         %base-file-systems))
    (host-name "alyssas-home-server")
    (users (cons* (user-account
                   (name "alyssa")
                   (comment "Alyssa")
                   (group "users")
                   (home-directory "/home/alyssa")
                   (supplementary-groups
                    '("wheel" "netdev" "audio" "video")))
                  %base-user-accounts))
    (sudoers-file (plain-file "sudoers" "\
root ALL=(ALL) ALL
%wheel ALL=NOPASSWD: ALL\n"))
    (services (append
               (list (service openssh-service-type
                              (openssh-configuration
                               (permit-root-login #t)))
                     (service dhcp-client-service-type))
               %base-services))))

;; List of machines to deploy
(list (machine
       (operating-system os)
       (environment managed-host-environment-type)
       (configuration (machine-ssh-configuration
                       (host-name "alyssa-p-hacker.tld")
                       (system "i686-linux")
                       (identity "/path/to/ssh-key")))))

Even if Scheme isn't your forté, parts of this should look familiar if you've used Guix before. The "operating system description" section in particular is something you might use with guix system reconfigure. What's new is the last part: We construct a list containing one machine of the managed-host-environment-type, for which we've specified that os is the operating-system declaration that we want to install on it, and that we can connect to it using the parameters specified by the machine-ssh-configuration.

Let's take a step back for a moment and explain what a machine is. guix deploy aims to support a number of different use-cases, which we abstract as "environment types". We'll see other environment types later in this article, but the general idea is that these environments specify how resources should be "provisioned" or created. For example, an environment type designed for working with a Virtual Private Server (VPS) provider might make calls the provider's API to request a virtual machine before installing the machine's operating-system declaration on it.

The environment type used in this example, managed-host-environment-type, is intended for machines that are already running Guix System and are accessible over SSH. It expects that the configuration field of the machine be an instance of machine-ssh-configuration, whose available fields are described in the manual. This gives guix deploy the information it needs to connect to the machine's SSH daemon.

Running guix deploy with this file would build the "operating system closure" of os -- a bundle of the packages, configuration files, and other dependencies necessary to realize that configuration -- for the architecture specified by system (in this case i686-linux), send it over SSH to alyssa-p-hacker.tld, and then remotely "activate" the configuration by creating a new system generation and upgrading running services. Sweet! Upgrading our single server setup has been reduced to an endeavour involving just over a dozen keystrokes.

More advanced case: managing a virtual private server deployment

One server not cutting it for you? guix deploy can still help. Suppose we run a web service that we'd like to split up across multiple machines for performance reasons.

(define %forum-server-count 4)

(define (forum-server n)
  (operating-system
    (host-name (format #f "forum-server-~a" n))
    ...
    (services (append (list (service httpd-service-type
                                     (httpd-configuration
                                      ...)))
                      %base-services))))

(map (lambda (n)
       (machine
        (system (forum-server n))
        (environment digital-ocean-environment-type)
        (configuration (digital-ocean-configuration
                        (region "nyc3")
                        (size "s-1vcpu-1gb")
                        (enable-ipv6 #t)))))
     (iota %forum-server-count))

This example isn't as concrete as the first one; I'm intentionally omitting parts of the configuration to make the example clearer. Here, we automate the creation of %forum-server-count Digital Ocean "droplets" in their NYC3 region by creating a list of 4 machines.

Assuming that the environment variable GUIX_DIGITAL_OCEAN_TOKEN is properly set, running guix deploy with this file will do much of the same as the previous example. The difference is that four virtual machines will be automatically created on Digital Ocean.

One important thing to note about the digital-ocean-environment-type is that, currently, it does not automatically clean up unused virtual machines. If you change something in the deployment specification and run guix deploy again, the virtual machines from the previous deployment will remain until you destroy them yourself.

A quick peek into the internals of digital-ocean-environment-type

It would be an overstatement to say that the process of implementing a new environment type is easy, but a fair amount of the work has already been done for you. We'll use the definition of digital-ocean-environment-type as an example.

(define digital-ocean-environment-type
  (environment-type
   (machine-remote-eval digital-ocean-remote-eval)
   (deploy-machine      deploy-digital-ocean)
   (roll-back-machine   roll-back-digital-ocean)
   (name                'digital-ocean-environment-type)
   (description         "Provisioning of \"droplets\": virtual machines
 provided by the Digital Ocean virtual private server (VPS) service.")))

The environment-type record specifies a small amount of metadata (name and description), as well as the names of three procedures: one for remotely evaluating a G-Expression on the host (machine-remote-eval), one for deploying an operating-system declaration to the host, and one for rolling the host back one generation.

This might sound like a lot, but the pattern for these high-level environment types is to somehow obtain a machine running Guix System, set up an SSH daemon, and then delegate to managed-host-environment-type. digital-ocean-remote-eval is a pretty good example of this:

(define (digital-ocean-remote-eval target exp)
  "Internal implementation of 'machine-remote-eval' for MACHINE instances with
an environment type of 'digital-ocean-environment-type'."
  (mlet* %store-monad ((name (droplet-name target))
                       (network -> (droplet-public-ipv4-network name))
                       (address -> (hash-ref network "ip_address"))
                       (ssh-key -> (digital-ocean-configuration-ssh-key
                                    (machine-configuration target)))
                       (delegate -> (machine
                                     (inherit target)
                                     (environment managed-host-environment-type)
                                     (configuration
                                      (machine-ssh-configuration
                                       (host-name address)
                                       (identity ssh-key)
                                       (system "x86_64-linux"))))))
    (machine-remote-eval delegate exp)))

As you can see, you could reasonably go about implementing an environment type without ever having to learn what a G-Expression is. Here, droplet-name derives the name of the droplet from the machine's operating-system declaration, the information necessary to connect to the droplet is found using droplet-public-ipv4-network, and that's used to create machine of managed-host-environment-type.

In conclusion

I sincerely hope that guix deploy proves to be a useful to anyone dealing with system administration or software development. Transactional upgrades should provide peace of mind to those managing servers (it's worth noting that few existing tools are capable of recovering from failed deployments), and I believe that procedurally-generated deployment configurations could very well be the future of distribution for software such as web services: Imagine if setting up a Mastodon instance were as easy as downloading a Scheme file and handing it off to guix deploy. The ease of writing code that generates code isn't the only benefit of using Guile for something like this. Guile is a general-purpose programming language, so more advanced tooling can reasonably be built atop guix deploy. A GTK or Emacs DevOps interface, perhaps? (If that idea sounds outlandish, consider that the latter has already happened for the package management interface.)

It's been a great summer working alongside everyone in the Guix community. guix deploy is brand new (and a little unstable!), but we've had enthusiastic adoption by several on the mailing lists who were quick to report any issues they found. I'd like to thank everyone on the #guix IRC channel and the mailing lists who got me up to speed with the code, answered my questions, gave feedback when I submitted my patches, and put guix deploy under the pressure of use in production. And of course, I want to thank my mentors Christopher Lemmer Webber and David Thompson. I had to make some hard design decisions, but this was made easier thanks to the guidance of two experienced Guix veterans.

Oh, and this isn't a goodbye. I really feel I've found my place as a Guix contributor, and I can't wait to see what the future will bring for guix deploy. Catch ya on the mailing lists!

Editor's note

Thank you for all of your hard work, Jakob!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Jakob L. Kreuze at November 04, 2019 03:00 PM

November 01, 2019

Programming Praxis

Crypt(1), Again

In the previous exercise I tried to write a replacement for the old Unix crypt(1) program, but never did figure out how to enter a password in Chez Scheme. So today I have the program in C.

To answer some questions that came up in the previous exercise: Yes, I know about ccrypt. Yes, I know that I should not rely on any cryptographic code I write. Yes, the original Unix crypt wasn’t very secure, though at least I can claim that my crypt is better now than the Unix crypt was at the time.

You can see my crypt program on the next page.

by programmingpraxis at November 01, 2019 09:00 AM

October 30, 2019

Arthur A. Gleckler

Scheme Workshop 2019

ICFP 2019 logo, based on the Ampelmännchen

Scheme Workshop 2019

When working on software projects, it's easy to forget that one's collaborators are real people, not just disembodied voices somewhere in cyberspace. That's one of the reasons I like to attend the Scheme and Functional Programming Workshop each year. The talks are fascinating, and it's great to spend time with real people doing interesting things with Scheme. This year's workshop, like many in the past, was held in conjunction with the International Conference on Functional Programming.

For many of the workshop's twenty-plus years, I couldn't get away from work to attend it, so I only experienced it by reading the proceedings. Because many people miss it but would like to see the talks, I've been volunteering as videographer since 2016. I'm an amateur, but using a camera, a lapel microphone, a screen capture device, and a Scheme program (as yet unpublished) that wraps around FFmpeg, I'm able to produce videos with good sound quality that show the on-screen demos and slides clearly. Here are the YouTube playlists for the past four years:

  • 2016 (Nara, Japan)
  • 2017 (Oxford, England)
  • 2018 (St. Louis, Missouri, USA)
  • 2019 (Berlin, Germany)

I enjoyed all the talks this year, but I especially recommend two. First, Paulette Koronkevich's Reigniting Fuse, an Online Partial Evaluator for Scheme (YouTube). In this short talk, she does a great job explaining online partial evaluation, an old idea in static analysis that deserves more attention.

Second, I recommend Andy Keep's Clever Representations & Small Optimizations in Chez Scheme (YouTube). This was the afternoon keynote of the conference, so it was a bit longer, which gave Andy time to give a detailed explanation of how Chez Scheme's compiler and runtime system produce such fast code.

After the workshop, I explored Berlin for a few days, then attended Summer BOB 2019 (YouTube), a mini-conference that explores technologies beyond the mainstream and asks the question, What happens when we use what's best for a change? There were many excellent talks, including A Functional Reboot for Deep Learning (YouTube), a thoughtful talk by Conal Elliott about finding the essence of deep learning. I'm a novice at deep learning, so I'm not sure how to evaluate the strong criticisms he made of current practice, but since he did co-invent the functional reactive programming paradigm, it seems like he might have good ideas. He's definitely a clear thinker.

Attending the Scheme Workshop is one of the highlights of my year every year. I hope that the feeling of excitement and intellectual exploration from the workshop comes through in the videos.

Footnotes

[camera] The camera was donated by my friend Philip Greenspun, author of Greenspun's tenth rule. Thanks, Phil.

by Arthur A. Gleckler at October 30, 2019 12:00 PM

October 29, 2019

Programming Praxis

Crypt(1)

I was surprised to learn recently that modern Linux systems do not provide a crypt command, which was common in older Unix systems. So I decided to write one.

The interface to crypt is simple: it reads from standard input and writes to standard output, requesting a key from the console using a method that doesn’t echo the key to the screen. The program has no command-line arguments.

The cipher is symmetric, so encryption and decryption use the same key; the ciphering algorithm I choose is <a href="“>RC4-drop512, which isn’t exactly state-of-the-art encryption, but is simple and good enough for casual use. The base algorithm is RC4; dropping the first 512 characters of the generated keystream eliminates a potential weakness in the bas RC4 key scheduling algorithm.

Your task is to write a crypt program as described above. When you are finished, you are welcome to read a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at October 29, 2019 09:00 AM

October 25, 2019

GNU Guix

Guix Profiles in Practice

Note: An updated version of this article is available in the brand new cookbook.

Guix provides a very useful feature that may be quite foreign to newcomers: profiles. They are a way to group package installations together and all users on the same system are free to use as many profiles as they want.

Whether you're a developer or not, you may find that multiple profiles bring you great power and flexibility. While they shift the paradigm somewhat compared to traditional package managers, they are very convenient to use once you've understood how to set them up.

If you are familiar with Python's virtualenv, you can think of a profile as a kind of universal virtualenv that can hold any kind of software whatsoever, not just Python software. Furthermore, profiles are self-sufficient: they capture all the runtime dependencies which guarantees that all programs within a profile will always work at any point in time.

Multiple profiles have many benefits:

  • Clean semantic separation of the various packages a user needs for different contexts.

  • Multiple profiles can be made available into the environment either on login or within a dedicated shell.

  • Profiles can be loaded on demand. For instance, the user can use multiple shells, each of them running different profiles.

  • Isolation: Programs from one profile will not use programs from the other, and the user can even install different versions of the same programs to the two profiles without conflict.

  • Deduplication: Profiles share dependencies that happens to be the exact same. This makes multiple profiles storage-efficient.

  • Reproducible: when used with declarative manifests, a profile can be fully specified by the Guix commit that was active when it was set up. This means that the exact same profile can be set up anywhere, anytime, with just the commit information. See section “Reproducible profiles” below.

  • Easier upgrades and maintenance: Multiple profiles make it easy to keep package listings at hand and make upgrades completely friction-less.

Concretely, here follows some typical profiles:

  • The dependencies of a project you are working on.

  • Your favourite programming language libraries.

  • Laptop-specific programs (like powertop) that you don't need on a desktop.

  • TeXlive (this one can be really useful when you need to install just one package for this one document you've just received over email).

  • Games.

Let's dive in the set up!

Basic setup with manifests

A Guix profile can be set up via a so-called manifest specification that looks like this:

(specifications->manifest
  '("package-1"
    ;; Version 1.3 of package-2.
    "package-2@1.3"
    ;; The "lib" output of package-3.
    "package-3:lib"
    ; ...
    "package-N"))

See (guix) Invoking guix package for the syntax details.

We can create a manifest specification per profile and install them this way:

GUIX_EXTRA_PROFILES=$HOME/.guix-extra-profiles
mkdir -p "$GUIX_EXTRA_PROFILES"/my-project # if it does not exist yet
guix package --manifest=/path/to/guix-my-project-manifest.scm --profile="$GUIX_EXTRA_PROFILES"/my-project/my-project

Here we set an arbitrary variable GUIX_EXTRA_PROFILES to point to the directory where we will store our profiles in the rest of this article.

Placing all your profiles in a single directory, with each profile getting its own sub-directory, is somewhat cleaner. This way, each sub-directory will contain all the symlinks for precisely one profile. Besides, "looping over profiles" becomes obvious from any programming language (e.g. a shell script) by simply looping over the sub-directories of $GUIX_EXTRA_PROFILES.

Note that it's also possible to loop over the output of

guix package --list-profiles

although you'll probably have to filter out ~/.config/guix/current.

To enable all profiles on login, add this to your ~/.bash_profile (or similar):

for i in $GUIX_EXTRA_PROFILES/*; do
    profile=$i/$(basename "$i")
    if [ -f "$profile"/etc/profile ]; then
        GUIX_PROFILE="$profile"
        . "$GUIX_PROFILE"/etc/profile
    fi
    unset profile
done

Note to Guix System users: the above reflects how your default profile ~/.guix-profile is activated from /etc/profile, that latter being loaded by ~/.bashrc by default.

You can obviously choose to only enable a subset of them:

for i in "$GUIX_EXTRA_PROFILES"/my-project-1 "$GUIX_EXTRA_PROFILES"/my-project-2; do
    profile=$i/$(basename "$i")
    if [ -f "$profile"/etc/profile ]; then
        GUIX_PROFILE="$profile"
        . "$GUIX_PROFILE"/etc/profile
    fi
    unset profile
done

When a profile is off, it's straightforward to enable it for an individual shell without "polluting" the rest of the user session:

GUIX_PROFILE="path/to/my-project" ; . "$GUIX_PROFILE"/etc/profile

The key to enabling a profile is to source its etc/profile file. This file contains shell code that exports the right environment variables necessary to activate the software contained in the profile. It is built automatically by Guix and meant to be sourced. It contains the same variables you would get if you ran:

guix package --search-paths=prefix --profile=$my_profile"

Once again, see (guix) Invoking guix package for the command line options.

To upgrade a profile, simply install the manifest again:

guix package -m /path/to/guix-my-project-manifest.scm -p "$GUIX_EXTRA_PROFILES"/my-project/my-project

To upgrade all profiles, it's easy enough to loop over them. For instance, assuming your manifest specifications are stored in ~/.guix-manifests/guix-$profile-manifest.scm, with $profile being the name of the profile (e.g. "project1"), you could do the following in Bourne shell:

for profile in "$GUIX_EXTRA_PROFILES"/*; do
  guix package --profile="$profile" --manifest="$HOME/.guix-manifests/guix-$profile-manifest.scm"
done

Each profile has its own generations:

guix package -p "$GUIX_EXTRA_PROFILES"/my-project/my-project --list-generations

You can roll-back to any generation of a given profile:

guix package -p "$GUIX_EXTRA_PROFILES"/my-project/my-project --switch-generations=17

Required packages

Activating a profile essentially boils down to exporting a bunch of environmental variables. This is the role of the etc/profile within the profile.

Note: Only the environmental variables of the packages that consume them will be set.

For instance, MANPATH won't be set if there is no consumer application for man pages within the profile. So if you need to transparently access man pages once the profile is loaded, you've got two options:

  • Either export the variable manually, e.g.

    export MANPATH=/path/to/profile${MANPATH:+:}$MANPATH
  • Or include man-db to the profile manifest.

The same is true for INFOPATH (you can install info-reader), PKG_CONFIG_PATH (install pkg-config), etc.

Default profile

What about the default profile that Guix keeps in ~/.guix-profile?

You can assign it the role you want. Typically you would install the manifest of the packages you want to use all the time.

Alternatively, you could keep it "manifest-less" for throw-away packages that you would just use for a couple of days. This way makes it convenient to run

guix install package-foo
guix upgrade package-bar

without having to specify the path to a profile.

The benefits of manifests

Manifests are a convenient way to keep your package lists around and, say, to synchronize them across multiple machines using a version control system.

A common complaint about manifests is that they can be slow to install when they contain large number of packages. This is especially cumbersome when you just want get an upgrade for one package within a big manifest.

This is one more reason to use multiple profiles, which happen to be just perfect to break down manifests into multiple sets of semantically connected packages. Using multiple, small profiles provides more flexibility and usability.

Manifests come with multiple benefits. In particular, they ease maintenance:

  • When a profile is set up from a manifest, the manifest itself is self-sufficient to keep a "package listing" around and reinstall the profile later or on a different system. For ad-hoc profiles, we would need to generate a manifest specification manually and maintain the package versions for the packages that don't use the default version.

  • guix package --upgrade always tries to update the packages that have propagated inputs, even if there is nothing to do. Guix manifests remove this problem.

  • When partially upgrading a profile, conflicts may arise (due to diverging dependencies between the updated and the non-updated packages) and they can be annoying to resolve manually. Manifests remove this problem altogether since all packages are always upgraded at once.

  • As mentioned above, manifests allow for reproducible profiles, while the imperative guix install, guix upgrade, etc. do not, since they produce different profiles every time even when they hold the same packages. See the related discussion on the matter.

  • Manifest specifications are usable by other guix commands. For example, you can run guix weather -m manifest.scm to see how many substitutes are available, which can help you decide whether you want to try upgrading today or wait a while. Another example: you can run guix pack -m manifest.scm to create a pack containing all the packages in the manifest (and their transitive references).

  • Finally, manifests have a Scheme representation, the <manifest> record type. They can be manipulated in Scheme and passed to the various Guix APIs.

It's important to understand that while manifests can be used to declare profiles, they are not strictly equivalent: profiles have the side effect that they "pin" packages in the store, which prevents them from being garbage-collected and ensures that they will still be available at any point in the future.

Let's take an example:

  1. We have an environment for hacking on a project for which there isn't a Guix package yet. We build the environment using a manifest, and then run guix environment -m manifest.scm. So far so good.

  2. Many weeks pass and we have run a couple of guix pull in the mean time. Maybe a dependency from our manifest has been updated; or we may have run guix gc and some packages needed by our manifest have been garbage-collected.

  3. Eventually, we set to work on that project again, so we run guix environment -m manifest.scm. But now we have to wait for Guix to build and install stuff!

Ideally, we could spare the rebuild time. And indeed we can, all we need is to install the manifest to a profile and use GUIX_PROFILE=/the/profile; . "$GUIX_PROFILE"/etc/profile as explained above: this guarantees that our hacking environment will be available at all times.

Security warning: While keeping old profiles around can be convenient, keep in mind that outdated packages may not have received the latest security fixes.

Reproducible profiles

To reproduce a profile bit-for-bit, we need two pieces of information:

  • a manifest,
  • a Guix channel specification.

Indeed, manifests alone might not be enough: different Guix versions (or different channels) can produce different outputs for a given manifest.

You can output the Guix channel specification with guix describe --format=channels. Save this to a file, say channel-specs.scm.

On another computer, you can use the channel specification file and the manifest to reproduce the exact same profile:

GUIX_EXTRA_PROFILES=$HOME/.guix-extra-profiles
GUIX_EXTRA=$HOME/.guix-extra

mkdir "$GUIX_EXTRA"/my-project
guix pull --channels=channel-specs.scm --profile "$GUIX_EXTRA/my-project/guix"

mkdir -p "$GUIX_EXTRA_PROFILES/my-project"
"$GUIX_EXTRA"/my-project/guix/bin/guix package --manifest=/path/to/guix-my-project-manifest.scm --profile="$GUIX_EXTRA_PROFILES"/my-project/my-project

It's safe to delete the Guix channel profile you've just installed with the channel specification, the project profile does not depend on it.

Special thanks

Chris Marusich and Simon Tournier for their thorough feedback.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Pierre Neidhardt at October 25, 2019 12:15 PM

Programming Praxis

IBAN

The International Bank Account Number (IBAN) is an internationally agreed standard of identifying bank accounts with a hash number to reduce errors. The first two characters of an IBAN are the two-letter country code, the next two characters are a two-digit hash, and the remaining characters identify the bank routing number and depositor account number. For instance, here is a fictition British IBAN: GB82 WEST 1234 5698 7654 32. The full IBAN standard specifies the range of valid bank account numbers and depositor account numbers for each country; we are interested only in the hash code.

In the code shown above, the hash code consists of the two digits 82, which are validated as follows:

1) Move the first four characters from the beginning of the string to the end: WEST 1234 5698 7654 32GB 82.

2) Replace letters with two-digit numbers according to the scheme A = 10, …, Z = 35:
3214 2829 1234 5698 7654 3216 1182.

3) Treating the string as a large integer, divide by 97 and calculate the remainder.

4) If the remainder is not 1, the IBAN is not valid.

To generate a hash code, perform the same procedure as above with the two hash digits initially set to 00, then subtract the hash code from 98 and insert it in the IBAN.

Your task is to write functions to validate existing IBAN numbers and generate hash codes for new IBAN numbers. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at October 25, 2019 09:00 AM

October 22, 2019

Programming Praxis

A Silly Task

Today’s task is silly:

Your task is to write a program that, when executed, writes to the output in words the number of characters in the program. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at October 22, 2019 09:00 AM

October 18, 2019

Programming Praxis

Anagrams, Again

The previous exercise showed a probabilistic method for determining if two strings are anagrams, and some users pointed out collisions in the method that falsely concluded two strings were anagrams when in fact they were not.

Your task is to write a program that recognizes anagrams without possibility of error. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at October 18, 2019 09:00 AM