Planet Scheme

March 22, 2019

Programming Praxis

Data Laundry, Revisited

Data laundry is the act of cleaning data, as when it arrives in one format and must be translated to another, or when external data must be checked for validity. Some weeks, as this week, data laundry occupies a significant portion of my time at work, so it’s an exercise worth examining. We looked at data laundry in two previous exercises. Today’s exercise in data laundry comes to us from Stack Overflow:

I have a file like this:

AAKRKA HIST1H1B AAGAGAAKRKATGPP
AAKRKA HIST1H1E RKSAGAAKRKASGPP
AAKRLN ACAT1 LMTADAAKRLNVTPL
AAKRLN SUCLG2 NEALEAAKRLNAKEI
AAKRLR GTF2F1 VSEMPAAKRLRLDTG
AAKRMA VCL NDIIAAAKRMALLMA
AAKRPL WIZ YLGSVAAKRPLQEDR
AAKRQK MTA2 SSSQPAAKRQKLNPA

I would like to kind of merge 2 lines if they are exactly the same in the 1st column. The desired output is:

AAKRKA HIST1H1B,HIST1H1E AAGAGAAKRKATGPP,RKSAGAAKRKASGPP
AAKRLN ACAT1,SUCLG2 LMTADAAKRLNVTPL,NEALEAAKRLNAKEI
AAKRLR GTF2F1 VSEMPAAKRLRLDTG
AAKRMA VCL NDIIAAAKRMALLMA
AAKRPL WIZ YLGSVAAKRPLQEDR
AAKRQK MTA2 SSSQPAAKRQKLNPA

Sometimes there could be more than two lines starting with the same word. How could I reach the desired output with bash/awk?

Your task is to write a program to solve this simple task of data laundry. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at March 22, 2019 04:00 AM

March 19, 2019

Programming Praxis

Target Sum Subarray

Given an array of non-negative integers, write a program that finds the contiguous portion of the array that sums to a given target, or report that there is no such subarray. For instance, in the array 1, 4, 20, 3, 10, 5, target 33 exists in the subarray 20, 3, 10, in the array 1, 4, 0, 0, 3, 10, 5, target 7 exists in the subarray 4, 0, 0, 3, and in the array 1, 4, target 3 does not exist in any subarray.

Your task is to write a program that finds a target sum in an array. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at March 19, 2019 09:00 AM

March 15, 2019

GNU Guix

Documentation video creation

Over the last few months, I have been working as an Outreachy intern with the GNU Guix crowd to develop videos presenting and documenting the project. My goal in this round as an Outreachy intern for the December 2018 to March 2019 period consists of creating introductory documentation videos about different topics for people who would like to use GNU Guix, admins and/or those who would like to join Guix community and don’t know where to start. Even interested or having a clear documentation, they might feel overwhelmed by it. I experienced this issue in the past with people in another context.

My main tasks consist of creating a workflow for automating as much as possible the process of creating the videos, as well as, of course, creating the videos themselves. Creating the videos is not that easy as it might seem, I have to design them (I cannot automate that part), let the audio match the video, and matching the exact timing is quite difficult. Something very important that I should mention is that the workflow currently allows translations to other languages.

It is a work in progress for too many reasons, specially because it keeps being improved all the time.

Also, I had to study tools deeply both for the creation of the workflow and the videos because I did not know them beforehand or I knew just the basics.

After trying several approaches for the workflow, the current one consists of creating “pieces of videos” and gluing them together in the end.

These “pieces of videos” may consist of:

  • Slide videos: they contain only a sequence of one or more slides.
  • Command line session videos: they contain only Guix or shell commands and their output, without showing any slide at all.
Workflow for creating each slide video.

slide

The inputs are SVG files and audio files. First, SVGs are converted to PNGs (“the slides”). Then, a text file having the order in which each slide will appear and the duration of the audio that matches it is created. An audio text file containing all the audio files sorted to have a complete audio file is created too. Lastly, with the slides' text file that has the reference to the slide files and the glued audio file the final slide video is made.

Workflow for creating each command line session video.

cli

The input is a session text file that has commands or meta-commands that are used to simulate, for example, the typing of a command, or the printing of it’s output. This file is passed to a Guile script that is in charge of executing the commands defined in the input text file and take text snapshots at a fixed time interval. Then, all these files are converted to postscript format. After that, they are transformed to SVG format. Finally, the process is repeated and the audio and the slides are glued to have final command line session video.

Workflow for creating the final video.

gluing

Slide videos and command line videos are a “bunch of videos” that need to be glued into the final one. They are sorted, and using the same tool for video creation our final introductory video is created.

The code for this video creation workflow is available on Savannah. Enjoy!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Laura Lazzati at March 15, 2019 11:00 AM

Programming Praxis

The Henry Used Car Dealership

Today’s exercise is simple, but leads me to ask a question. Here is the exercise, taken directly from /r/learnprogramming:

Draw the hierarchy chart and then plan the logic for a program needed by the sales manager of The Henry Used Car Dealership.

  1. The program will determine the profit on any car sold. Input includes the sale price and actual purchase price for a car.
  2. The output is the profit, which is the sale price minus the purchase price.
  3. Use three modules. The main program declares global variables and calls housekeeping, detail, and end-of-job modules.
  4. The housekeeping module prompts for and accepts a sale price. The detail module prompts for and accepts the purchase price, computes the profit, and displays the result. The end-of-job module displays the message Thanks for using this program.

I have written the actual program, but I am struggling with the hierarchy chart. Can someone give me any guidance on this one?

Is this for real? Separate modules for each statement? Why does the detail module get input, compute profit and display the result? If you have to have separate modules to ask for the purchase price and say thank you, why not split getting the sale price, computing the profit, and printing the result into separate modules? Is this how programming students are taught these days? To take a fundamentally simple program and make it ridiculously complex by splitting it into modules? What would David Parnas say about this modularization?

Your task is to write a program to compute profit for The Henry Used Car Dealership; you might also help me understand the way this program is intended to be modularized. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at March 15, 2019 09:00 AM

March 12, 2019

Programming Praxis

Recursion

Although we sometimes discuss rather more advanced topics, one focus of this blog will always be on student programmers, so I monitor several internet sites that cater to questions from learners. I have seen several posts in the last week or so about recursion, so I assume that we have reached the point in the semester where programming students are learning about recursion. Most programming professors seem to think that the factorial function (what is the value of n factorial) and the fibonacci function (what is the nth fibonacci number) are good demonstrations of recursion; personnally, I think it is better to discuss recursion in terms of binary trees.

Your task is to write versions of the factorial and fibonacci functions that use recursion to calculate their result. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at March 12, 2019 09:00 AM

March 08, 2019

Programming Praxis

Frequency Counts

Today we have a tricky exercise from Geeks For Geeks:

Given an unsorted array of n integers from 1 to n, possibly with some items appearing multiple times and others missing from the array, count the frequency of all items that are present. Your solution must run in O(n) time and take O(1) extra space.

Your task is to write a program to count frequencies as described above. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at March 08, 2019 10:00 AM

March 05, 2019

Programming Praxis

Business Card Turing Machine

We have something unusual today: a turing machine built from a business card that I learned about on Reddit. I immediately stopped and built my own, and had a lot of fun, and I suspect many of my readers will feel the same way.

Your task is to build your own turing machine. No coding, just playing. Feel free to discuss the exercise in the comments below.

by programmingpraxis at March 05, 2019 10:00 AM

March 01, 2019

Programming Praxis

Jumble

Last week I gave a rather silly solution to the Scrabble problem. Today’s exercise is my penance for that silliness.

As I’ve mentioned previously, my day job is on a team of programmers that supports our enterprise-wide computer system. I sit in a cube farm, where there is neither audible nor visual privacy. We recently hired a new programmer to replace a retiring team member, and he has a daily calendar on his desk that provides a jumbled series of letters that you have to rearrange to form a word. Yesterday’s puzzle was L T E A D E; most of the puzzles I solve in a few seconds, but that one took several minutes. The calendar appears to have a flaw: the solutions, one day after the next, are in alphabetical order, so I know before I start that the first letter will be E.

Your task is to write a program that solves jumbles. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at March 01, 2019 10:00 AM

February 26, 2019

Programming Praxis

ATM Machine

Today’s exercise asks you to simulate an ATM machine:

  1. Prompt for userid and password. If userid and password are not in the database, reprompt.
  2. Display balance.
  3. Present a menu asking to deposit, withdraw or exit. If user selects withdraw or deposit, perform the indicated transaction and goto Step 2. Deny withdrawals that would overdraw the account.
  4. Exit the program.

Your task is to write a program to simulate an ATM machine. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at February 26, 2019 10:00 AM

February 22, 2019

Programming Praxis

Scrabble

Today’s task is based on the game of Scrabble.

Your task is to write a program to find the dictionary words of length 2 to 7 that can be formed from the letters Q O F T H E A. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at February 22, 2019 10:00 AM

February 21, 2019

GNU Guix

Guix Days: Bootstrapping ARM

During the Guix Days before FOSDEM, some of us discussed bootstrapping on ARM architectures. We focused on how to port Mes to ARM. This post consists of notes from that discussion.

Recap: i686/x86_64 Reduced Binary Seed

We started our discussion by reviewing the current status for i686-linux and x86_64-linux. Jan (janneke) Nieuwenhuizen gave a similar summary a few days later at FOSDEM, and you can read the slides online.

Previously, the size of Guix's binary seed totaled about 250 MB. Now, on the core-updates branch, it's been reduced to about 130 MB. This is nearly a 50% reduction in size, which is great progress! Using this 130 MB reduced binary seed, it's currently possible to bootstrap Guix for both i686-linux and x86_64-linux.

To bootstrap x86_64-linux, we actually "cheat" and bootstrap from the i686-linux bootstrap binaries. This is possible because an x86_64-linux system can natively run i686-linux executables, and also because Guix automatically cross-compiles for the host platform during the bootstrap process. In other words, on an x86_64-linux system, Guix uses the i686-linux bootstrap binaries to build a cross-compilation toolchain, which it then uses to build a final, normal x86_64-linux toolchain. Guix then uses this final toolchain to build everything else for the x86_64-linux system.

That's great news for owners of i686 and x86_64 machines! But what about ARM? Although we could cross-compile the bootstrap binaries for ARM from an x86_64 machine, this isn't great because it would increase the number of things a person or organization would have to verify in order to audit the system. Perhaps more importantly, it would force owners of ARM machines to implicitly trust an x86_64 machine. The dominant vendors of CPUs implementing the x86_64 architecture, Intel and AMD, both include a management engine in many of their products, which represents a serious risk to user freedom.

In the short term, cross-compilation is better than nothing, but in the long term, we'd prefer to bootstrap ARM without cross-compiling from another architecture. Concretely, we'll need to complete at least the following tasks.

TODO: Implement a Mes backend for ARM

We need to implement a new Mes backend for an ARM architecture. We should choose an ARM instruction set that can work on a variety of ARM platforms with minimal fuss. The following candidates were suggested:

  • ARMv4. We would need to avoid unaligned memory accesses because they can behave in different ways depending on the CPU. This is the latest version of ARM that GCC 2.95 supports, which matters because GCC 2.95 is the latest version that the Mes C Library supports.

  • ARMv7. If we avoid extensions and unaligned memory accesses, it might still work for our needs. It was mentioned in the session that TinyCC will probably work with either ARMv4 or ARMv7. In TinyCC, as a first step, it's probably fine to depend on ARMv7 since it's the most common recent architecture version (and it is more forgiving). Later, if we remove unaligned accesses, it will also work on ARMv4 and thus on basically all ARM CPUs.

After the session concluded, Danny Milosavljevic committed some changes to the wip-arm branch of mes which enabled many of the tests to pass - but some tests still fail, and you can help finish the work!

TODO: Port mescc-tools to ARM, also

The mes project depends upon the mescc-tools project, which also must be ported. The mescc-tools project contains an M1 macro assembler, which would need to be extended to support ARM branches. Currently, ARM branches are very broken.

TODO: Improve Guix integration and merge core-updates

Even if we had a new Mes backend and mescc-tools for ARM, there would still be more to do. The Guix integration is not quite complete - the core-updates branch still needs to be merged with master, and we'll need to fix any problems that arise. Even on i686-linux, the bottom of the bootstrap path is incomplete. Preliminary Guix code exists on the wip-bootstrap branch to achieve a scheme-only bootstrap, but help would be welcome!

You can help!

In summary, you can help the Mes project by doing any of the following things:

  • Help implement an ARMv7 (or ARMv4) backend for Mes! This entails machine code and assembly. It should be fun for anyone who wants to play around close to the metal.

  • Help port mescc-tools to ARM. This entails writing an assembler in M1 macro code and probably goes hand-in-hand with work on Mes itself.

  • Help firm up core-updates and merge it to master! This involves Guix package definitions and troubleshooting build failures. It should be fun for anyone who wants to learn more about the bigger picture of how Guix bootstraps all of its software from the new reduced binary seed.

  • Help complete the i686-linux and x86_64-linux bootstrap. You can hack on the bleeding edge scheme code in the wip-bootstrap branch, or maybe you can help extend the bootstrap path all the way down to approximately 500 bytes of auditable assembly code!

There's still plenty of meaty work left to be done! If you're interested, get in touch and we'll help you get started.

Think Big: Bootstrapping without an OS

In addition to the immediate tasks necessary for porting Mes to ARM, we also took some time to think about the long term hopes and dreams of the bootstrappable project.

We discussed how in the long term, in parallel with the aforementioned tasks, it should be possible to investigate how to bootstrap an entire system without relying on a OS or even a kernel running on the machine. For example, one can imagine loading the transitive closure of source (including a tiny, human-readable machine code program to kick off the entire process) into a computer as a kind of "firmware image". When the computer runs, it would execute this "firmware image" and eventually produce a fully bootstrapped system.

Think Bigger: Bootstrapping Hardware

We also briefly talked about how even after we achieve full source software bootstrap, we will still need to tackle the problem of "hardware bootstrap". It isn't clear what form this will eventually take, but surely free hardware design will play an important role in ensuring that we can trust our hardware, too.

About Bootstrappable Builds and Mes

Software is bootstrappable when it does not depend on a binary seed that cannot be built from source. Software that is not bootstrappable - even if it is free software - is a serious security risk for a variety of reasons. The Bootstrappable Builds project aims to reduce the number and size of binary seeds to a bare minimum.

GNU Mes is closely related to the Bootstrappable Builds project. Mes aims to create an entirely source-based bootstrapping path for the Guix System and other interested GNU/Linux distributions. The goal is to start from a minimal, easily inspectable binary (which should be readable as source) and bootstrap into something close to R6RS Scheme.

Currently, Mes consists of a mutual self-hosting scheme interpreter and C compiler. It also implements a C library. Mes, the scheme interpreter, is written in about 5,000 lines of code of simple C. MesCC, the C compiler, is written in scheme. Together, Mes and MesCC can compile a lightly patched TinyCC that is self-hosting. Using this TinyCC and the Mes C library, it is possible to bootstrap the entire Guix System for i686-linux and x86_64-linux.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Chris Marusich at February 21, 2019 11:00 PM

February 19, 2019

Programming Praxis

The Trapped Knight

Over at Numberphile, Neil Sloane discusses a knight moving on an infinite chessboard with squares numbered in a square spiral (note that, starting from 1, the squares on the southeast diagonal are successive squares of odd numbers 1² = 1, 3² = 9, 5² = 25, 7² = 49, and so on):

37 36 35 34 33 32 31
38 17 16 15 14 13 30
39 18  5  4  3 12 29
40 19  6  1  2 11 28 .
41 20  7   8 9 10 27 .
42 21 22 23 24 25 26 .
43 44 45 46 47 48 49 50

The knight starts from square 1 and always moves to the smallest-numbered square not previously visited. Thus, from 1, the knight can move to squares 10, 12, 14, 16, 18, 20, 22 or 24, of which the smallest unvisited square is 10. From 10, the knight can then move to squares 1, 3, 23, 29, 47, 49, 51 or 53; the smallest of those, 1, has already been visited, so the knight moves to square 3. And so on. The beginning of the knight’s tour visits squares 1, 10, 3, 6, 9, 4 and so on (A316667).

Your task is to write a program to determine if the knight’s tour is infinite or if the knight becomes trapped with no remaining unvisited squares; if the knight becomes trapped, determine the length of his tour and the square on which he remains. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at February 19, 2019 10:00 AM

February 15, 2019

Programming Praxis

Peaks

Given an array of integers, a peak is a subarray of minimal length in which the integer values increase and then decrease. For instance, in the array [5,5,4,5,4] there is a peak [4,5,4] and in the array [6,5,4,4,4,4,4,5,6,7,7,7,7,7,6] there is a peak [6,7,7,7,7,7,6]. Note that [4,5,6,7,7,7,7,7,6] shows a pattern of increasing and decreasing values, but it is not minimal because the first two items can be removed and the remaining subarray remains a peak. The array [5,5,5,5,4,5,4,5,6,7,8,8,8,8,8,9,9,8] has two peaks, [4,5,4] and [8,9,9,8].

Your task is to write a program that finds all of the peaks in an array. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at February 15, 2019 10:00 AM

February 12, 2019

Programming Praxis

Consecutive Sums

Given the positive integer 15, there are three ways to take consecutive positive integers that sum to 15: 1+2+3+4+5, 4+5+6 and 7+8.

Your task is to write a program that, given a positive integer, finds all the ways that consecutive positive integers can sum to the target integer. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

by programmingpraxis at February 12, 2019 10:00 AM

February 07, 2019

GNU Guix

QA on non-Intel at Guix Days

During the second day of Guix Days (a FOSDEM fringe event) we split up into smaller working groups based on our areas of interest. I led a group which aimed to address some of the package issues which exist on non-Intel architectures. Of course not everyone has access to an ARM board, but with the qemu-binfmt-service service it is possible to use QEMU and the binfmt_misc functionality of the Linux kernel to emulate these systems. Many have reported that this system emulation is comparable in speed to many of the available ARM boards on the market. Yet another possibility would be to do the hacking on an x86_64 system and, when we had a working prototype, to test it with QEMU or on actual ARM hardware.

Our group decided to tackle Go, which was lacking support in Guix on armhf and aarch64. Upon checking the build logs from Cuirass and the source code for Go we determined that Go did indeed require the gold linker from the GNU Binutils. We didn't want to modify the copy of Binutils in Guix since it is part of our bootstrap story, so we quickly put together a new package definition which added the configure flag to enable gold.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     (substitute-keyword-arguments (package-arguments binutils)
       ((#:configure-flags flags)
        `(cons "--enable-gold=default" ,flags))))))

This was an obvious first step, and one which we knew would fail. Had it been this easy gold would have been enabled back in 2012 when it was first added. Our error came in the form of one of the binaries not being able to link against libstdc++.so, which is in the gcc:lib output. This was quickly added and we were off and building again.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     (substitute-keyword-arguments (package-arguments binutils)
       ((#:configure-flags flags)
        `(cons "--enable-gold=default" ,flags))))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

Once again this failed. What were we missing? The correct paths were included, the file was indeed in the gcc:lib output. We inspected the original binutils package again noticed that it was built against a static libgcc, so of course it wouldn't find the shared library. In order to work quickly we copied the configure flags rather than inheriting them from binutils and tried our build again.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     (substitute-keyword-arguments (package-arguments binutils)
       ((#:configure-flags flags)
        `(cons* "--enable-gold=default"
                "--enable-new-dtags"
                "--with-lib-path=/no-ld-lib-path"
                "--enable-install-libbfd"
                "--enable-deterministic-archives"))))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

This time we made it through the full build phase and we knew we were almost there. Our enthusiasm was quickly dampened when we got the error during the tests: unable to find the 'dc' program. What is this dc program? This isn't any package any of us had heard of before. It definitely wasn't packaged in Guix. A quick apt-cache search dc search in Ubuntu showed they didn't have package either. A second search of Ubuntu, apt-file search dc | grep '/bin/dc' quickly showed us it was in the bc package, and soon we were building binutils-gold again.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     (substitute-keyword-arguments (package-arguments binutils)
       ((#:configure-flags flags)
        `(cons* "--enable-gold=default"
                "--enable-new-dtags"
                "--with-lib-path=/no-ld-lib-path"
                "--enable-install-libbfd"
                "--enable-deterministic-archives"))))
    (native-inputs
     `(("bc" ,bc)))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

Approaching the end of the check phase we soon ran into another error, there was an unpatched /bin/sh somewhere in the source code which was generated during the check phase. Based on the build logs we were able to track down approximately where the code should be, so we downloaded the source tar xf $(guix build --source binutils) and started looking. There were many obvious /bin/sh calls which we cross-referenced with the build logs and the patch-source-shebangs phase, and this left us with some code in gold/Makefile.in, which by default is not included in the patch-source-shebangs and would need to be fixed manually.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (add-after 'patch-source-shebangs 'patch-more-shebangs
           (lambda _
             (substitute* "gold/Makefile.in"
               (("/bin/sh") (which "sh")))
             #t)))
       ,@(substitute-keyword-arguments (package-arguments binutils)
         ((#:configure-flags flags)
          `(cons* "--enable-gold=default"
                  "--enable-new-dtags"
                  "--with-lib-path=/no-ld-lib-path"
                  "--enable-install-libbfd"
                  "--enable-deterministic-archives")))))
    (native-inputs
     `(("bc" ,bc)))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

One more build cycle later and we did it! /gnu/store/…-binutils-gold-2.31.1 existed! We now did two things, we copied our patch over to an aarch64 build machine and we started cleaning up our package definition on our x86_64 build machine, where we knew we had a working package definition.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (add-after 'patch-source-shebangs 'patch-more-shebangs
           (lambda _
             (substitute* "gold/Makefile.in"
               (("/bin/sh") (which "sh")))
             #t)))
       ,@(substitute-keyword-arguments (package-arguments binutils)
         ((#:configure-flags flags)
          `(cons* "--enable-gold=default"
                  (delete "LDFLAGS=-static-libgcc" ,flags))))))
    (native-inputs
     `(("bc" ,bc)))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

Fortunately for us the changes in the code worked on x86_64 and we still got a working binutils-gold output. On our aarch64 side the build was progressing nicely and everything seemed fine... until we suddenly were presented with big red errors about unrelocatable code. How could it? Everything was working so well! Undeterred, we built the source again, this time targeting armhf and were unfortunately presented with similar errors. Deciding to address the test failures later (It's ARM! It's not as well tested as other architectures! Right?) we disabled the tests and unsurprisingly binutils-gold built on both aarch64 and armhf.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (add-after 'patch-source-shebangs 'patch-more-shebangs
           (lambda _
             (substitute* "gold/Makefile.in"
               (("/bin/sh") (which "sh")))
             #t)))
       ,@(substitute-keyword-arguments (package-arguments binutils)
         ((#:tests? _ #f) #f)
         ((#:configure-flags flags)
          `(cons* "--enable-gold=default"
                  (delete "LDFLAGS=-static-libgcc" ,flags))))))
    (native-inputs
     `(("bc" ,bc)))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

Now for the real test. Due to bootstrapping issues with Go and aarch64, aarch64 uses Go@1.4 built for armhf. Go@1.11 failed to build until now because it was missing the gold linker. Surely using the gold linker would be a good test if our package worked. Since Go for aarch64 is 'more complex' due to the bootstrapping using armhf's Go, we decided to test armhf first. binutils-gold was added and our build started.

    (native-inputs
     `(("go" ,go-1.4)
+      ,@(match (%current-system)
+          ((or "armhf-linux" "aarch64-linux")
+           `(("gold" ,binutils-gold)))
+          (_ `()))
       ,@(package-native-inputs go-1.4)))

First build, success! /gnu/store/…-go-1.11.5 exists! OK, but does it actually work? guix build syncthing --system=armhf-linux. /gnu/store/…-syncthing-1.0.0 exists too! A quick check of guix refresh --list-dependent go@1.4 showed that we had unlocked 176 new packages for armhf. Even better, since they had all failed by default due to go@1.11 failing to build, for each package that did build meant one fewer package which failed to build which should take a big bite out of our build failures.

Our next test was syncthing for aarch64. /gnu/store/…-go-1.11.5 exists! /gnu/store/…-syncthing-1.0.0 ... does not. "unknown architecture 'armv7-a'." It seems that Go is confused which architecture it is building for. Unfortunately we were reaching the end of our time for hacking, so that will have to wait for another day. All that was left now was the test failures on binutils-gold for the ARM systems. Some attempts at cargo-culting other code failed (per-architecture tests we had and overriding flags in substitute-keyword-arguments we had, but not together), but after some attempts we were able to create a working package definition we were happy with.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (add-after 'patch-source-shebangs 'patch-more-shebangs
           (lambda _
             (substitute* "gold/Makefile.in"
               (("/bin/sh") (which "sh")))
             #t)))
       ,@(substitute-keyword-arguments (package-arguments binutils)
         ; Upstream is aware of unrelocatable test failures on arm*.
         ((#:tests? _ #f)
          (if (any (cute string-prefix? <> (or (%current-target-system)
                                               (%current-system)))
                   '("i686" "x86_64"))
              '#t '#f))
         ((#:configure-flags flags)
          `(cons* "--enable-gold=default"
                 (delete "LDFLAGS=-static-libgcc" ,flags))))))
     (native-inputs
      `(("bc" ,bc)))
     (inputs
      `(("gcc:lib" ,gcc "lib")))))

This patch was pushed to the master branch as 28317d499034b00cf1f08a9efd39bd2bc3425b19, and the commit following uses it as a native-input for Go@1.9 and Go@1.11. Go@1.4 was added in June 2016 and Go@1.6 that August, with our first go packages being added in October 2017. That same October Go@1.4 had support limited to Intel and armhf and in October 2018, in an effort to work toward a resolution, a patch was added to have aarch64 use Go@1.4 built for armhf for it's bootstrap path. Basically since the addition of the Go language support into Guix there was not a time when it was usable on armhf or aarch64. Hopefully we will soon finish getting full Go support on aarch64 and we can move all 352 dependents of Go@1.4 from "failing" to "succeeding" and have these architectures better supported.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

by Efraim Flashner at February 07, 2019 08:30 AM