Enable LXD GPU pass-through in Ubuntu 18.04 (Bionic Beaver)

Problem

One of the biggest challenge (aside from upgrading nvidia & cuda) in upgrading from Ubuntu 16.04 to Ubuntu 18.04 is enabling GPU passthrough in LXD Containers. Due to a bug in parsing logic, it cannot properly parse Blacklisted line in nvidia-410 drivers.

Solution

The simplest solution is to upgrade lxd to 3.0.2. To do that you have to enable pre-released updates (bionic-proposed).

Enable Pre-released updates from Ubuntu Software

Then just upgrade lxd:

sudo apt upgrade lxd

Reboot and you are done.

Why GPU Passthrough?

I have explained how to enable GPU Passthrough from LXD container but would you like to know what exactly am I doing with it?

I am using it to fine-tune Deep Learning algorithms from LXD containers. Having them in containers makes it very easy to move them to a different server with better GPU(s) and well as to Cloud.

How to run Gogs behind HAProxy serving https (SSL/TLS) connections

I run Gogs in a LXD container which runs behind HAProxy in another LXD container. HAProxy handles SSL/TLS connection (SSL Termination).

Note: Gogs is a Git service like GitHub or GitLabs. It is written in Go and normally used to host your own git server along with GitHub like interface and functionality (for free).

As you know, Gogs normally starts on non-standard ports so it doesn’t need super-user access. The following is a sample configuration ( custom/conf/app.ini ) for Gogs running internally on HTTP port 3000 while being served by a regular HTTPS URL (running on regular HTTPS port where SSL/TLS termination is provided by HAProxy):

[server]
DOMAIN           = gogs-domain.com
HTTP_PORT = 3000
ROOT_URL = https://gogs-domain.com/
DISABLE_SSH = false
SSH_PORT = 3333 START_SSH_SERVER = false OFFLINE_MODE = false

LXD: How to easily apt update and upgrade all running LXD containers

It is easy to create large number of LXD / LXC containers as it consumes minimal resources. Coupled with btrfs, it consumes minimal hard-disk space also due to de-duplication built-in within btrfs file system. Soon you would find yourself spending a significant amount of time updating (sudo apt update) and upgrading (sudo apt -y upgrade) all these containers (don’t forget: sudo apt -y autoremove). Here is a simple bash-fu magic which you can run from host to update all the containers:
lxc -c ns --format csv ls|grep RUNNING|cut -f1 -d',' | xargs -I'{}' sh -c "echo Updating '{}';lxc exec '{}' -- apt -qq update;lxc exec '{}' -- apt -qq -y upgrade;lxc exec '{}' -- apt -qq -y autoremove"
The code has been tested on Ubuntu 16.04 but it is expected to work on any Ubuntu system starting from 16.04. For previous version(s) use apt-get instead of apt.