Enable LXD GPU pass-through in Ubuntu 18.04 (Bionic Beaver)

One of the biggest challenge (aside from upgrading nvidia & cuda) in upgrading from Ubuntu 16.04 to Ubuntu 18.04 is enabling GPU passthrough in LXD Containers. Due to a bug in parsing logic, it cannot properly parse Blacklisted line in nvidia-410 drivers.

Only solution is to upgrade lxd to 3.0.2. To do that you have to enable pre-released updates (bionic-proposed).

Enable Pre-released updates from Ubuntu Software

Then just upgrade lxd:

sudo apt upgrade lxd

Reboot and you are done.

How to rapidly test alternative ideas in Java during development

Problem

Java is a compiled language and in any non-trivial project you use multiple libraries which are neatly assembled for you when you run the application with your build system like gradle. This makes it hard and slow to test alternative ideas as you have to compile and run the project each time which is inconvenient and slow even in the best configuration.

Solution

Groovy is an interpreted language (can be compiled too for performance) and is a super-set of Java. It is ideally suited for testing alternative approaches while development. However, how do you include your runtime classpath so that it can access all your classes as well as third party libraries that you use?

There is a simple solution. You can add a task to your gradle build file. Here is a sample build file (build.gradle):

plugins {
    ...
    id 'groovy'
}
mainClassName = 'com.taragana.App'
dependencies {
    ...
    compile 'org.codehaus.groovy:groovy-all:2.4.14'
}   
repositories {
    jcenter()
}
task(console, dependsOn: 'classes', type: JavaExec) {
   main = 'groovy.ui.Console'
   classpath = sourceSets.main.runtimeClasspath
}

This build file contains the task console which can be run with:

gradle console

In the console you can import and access any of your existing class files as well as third party libraries that you included in the application.

Value Addition

This is also an instant test environment that you can use to test your tests before codifying them in test classes. This saves tremendous amount of time not only in coding but also in developing test cases.

Guide: HAProxy HTTP Basic Authentication for specific sites (SSL Termination)

Objective

I use HAProxy to serve multiple SSL/TLS enabled sites with HAProxy doing SSL termination. The sites serve regular HTTP while users see proper HTTPS sites (with free certificates from LetsEncrypt). My objective was to provide HTTP Basic Authentication as a second layer of protection for certain applications like NextCloud (DropBox clone) or Gitea (GitHub clone).

Challenges

I wanted to provide HTTP Basic Auth over specific services (not all which is much easier) which didn’t natively support them like Gitea. Secondly, I wanted it to be transparent to the underlying Application. Thirdly, I didn’t want to provide plain-text passwords in haproxy.cfg.

Solution

After few iterations I arrived at a simple solution to the problem. The steps are:

  1. Create users with encrypted passwords
  2. Add users to haproxy.cfg
  3. Force authentication for specific sites on frontend
  4. Remove authorization on backend

Create users with encrypted passwords

To create encrypted passwords, you need a tool called mkpasswd which is available with whois, so you need to install it first (one time activity):

sudo apt install whois

Create password as shown below (replace Password with your actual Password):

mkpasswd -m sha-512 Password

Copy the encrypted password generated by the tool (mkpasswd).

Add users to haproxy.cfg

You can add multiple user lists as well as user groups (beyond the scope of this guide) to haproxy. Let’s create an user list named AuthUsers (as an example):

userlist AuthUsers
        user Username1 password $6$d./LYD0vplX$XoPWiTQfhNt4g4NRcU/toFiV89xhW524abcdfg
        user Username2 password $6$d./LYD0vplX$XoPWiTQfhNt4g4NRcU/toFefghxhW524abcdfg

Replace Username1, Username2 with your actual user names and the corresponding encrypted password as the last argument in the line.

You can add as many users as you want.

Force authentication for specific sites only

Let’s say we want to force authentication for these two sites (in frontend section):

acl host_example1 hdr(host) -i example1.com
acl host_example1 hdr(host) -i example2.com

Below this we force them to be authenticated:

acl authorized http_auth(AuthUsers)
http-request auth realm Example1 if host_example1 !authorized
http-request auth realm Example2 if host_example2 !authorized

Use backend only when properly authenticated:

use_backend example1 if host_example1 authorized
use_backend example2 if host_cexample2 authorized

Remove authentication header from backend

HAProxy for some strange reason sends this Authorization header to backend which sends certain servers in a loop. it is advisable to remove it.

backend example1
http-request set-header X-Client-IP %[src]
server example1 example1:3000 check
http-request del-header Authorization

backend example2
http-request set-header X-Client-IP %[src]
server example2 example2:3000 check
http-request del-header Authorization

Now restart the haproxy server and voila!

How to run Gogs behind HAProxy serving https (SSL/TLS) connections

I run Gogs in a LXD container which runs behind HAProxy in another LXD container. HAProxy handles SSL/TLS connection (SSL Termination).

Note: Gogs is a Git service like GitHub or GitLabs. It is written in Go and normally used to host your own git server along with GitHub like interface and functionality (for free).

As you know, Gogs normally starts on non-standard ports so it doesn’t need super-user access. The following is a sample configuration ( custom/conf/app.ini ) for Gogs running internally on HTTP port 3000 while being served by a regular HTTPS URL (running on regular HTTPS port where SSL/TLS termination is provided by HAProxy):

[server]
DOMAIN           = gogs-domain.com
HTTP_PORT = 3000
ROOT_URL = https://gogs-domain.com/
DISABLE_SSH = false
SSH_PORT = 3333 START_SSH_SERVER = false OFFLINE_MODE = false

How to provide Secure Isolated (Restricted) SFTP Access for file transfer

Often we need to provide secure but restricted sftp access to our clients to upload or download large files.

Regular sftp accounts allow the users to view login names of other users/clients and in many default installations even allow them to browse through the directories and files. In addition, they can also view your system files and any unprotected files and directories including much of your system settings under etc.

When you provide sftp access, you also provide ssh access (sftp is a subsystem of ssh) and as such users can login.

We address these issues by enclosing the user’s home directory in chroot jail, which makes it impossible for the user to break out of his home directory and view other’s files and directories.

Follow the steps below to configure your system.

As root (or sudo) modify /etc/ssh/sshd_config:Change Subsystem sftp line to:

Subsystem sftp internal-sftp

Add this to the end of the file:

Match Group sftp
    ChrootDirectory %h
    X11Forwarding no
    AllowTcpForwarding no
    ForceCommand internal-sftp

Save the changes and restart OpenSSH:

sudo apt install whois

Create a system group for users with restricted sftp access:

addgroup --system sftp

Now you can add users with addsftpuser and delete with delsftpuser from GitHub project restricted-sftp.

Clone the repository with:

git clone https://github.com/angsuman/restricted-sftp.git

Copy the files addsftpuser and delsftpuser to your ~/bin or any other directory in your PATH like /usr/bin.

Run it with by specifying the login name of the sftp user:

addsftpuser new-login

The only downside is that the user cannot upload/download files and directories under his home directory directly (as it is owned by root) but can do under Files sub-directory. There you have full access and can upload/download/delete/rename files and directories. To create additional directories under home, edit the addftpuser script and create them following the same procedure as Files.

Nirvana Satkam

॥ निर्वाण षटकम्॥

मनोबुद्ध्यहङ्कार चित्तानि नाहं

 

न च श्रोत्रजिह्वे न च घ्राणनेत्रे ।

 

न च व्योम भूमिर्न तेजो न वायुः

 

चिदानन्दरूपः शिवोऽहम् शिवोऽहम् ॥१॥

I am neither the mind, nor the intellect, nor the ego, nor the mind-stuff ;
I am neither the body, nor the changes of the body ;
I am neither the senses of hearing, taste, smell, or sight,
Nor am I the ether, the earth, the fire, the air ;
I am Existence Absolute, Knowledge Absolute, Bliss Absolute—
I am He, I am He. (Shivoham, Shivoham).

न च प्राणसंज्ञो न वै पञ्चवायु


न वा सप्तधातुः न वा पञ्चकोशः ।


न वाक्पाणिपादं न चोपस्थपायु


चिदानन्दरूपः शिवोऽहम् शिवोऽहम् ॥२॥

I am neither the Prâna, nor the five vital airs ;
I am neither the materials of the body, nor the five sheaths ;
Neither am I the organs of action, nor object of the senses ;
I am Existence Absolute, Knowledge Absolute, Bliss Absolute—
I am He, I am He. (Shivoham, Shivoham).

न मे द्वेषरागौ न मे लोभमोहौ

 

मदो नैव मे नैव मात्सर्यभावः ।

 

न धर्मो न चार्थो न कामो न मोक्षः

 

चिदानन्दरूपः शिवोऽहम् शिवोऽहम् ॥३॥

 

I have neither aversion nor attachment, neither greed nor delusion;
Neither egotism nor envy, neither Dharma nor Moksha;
I am neither desire nor objects of desire ;
I am Existence Absolute, Knowledge Absolute, Bliss Absolute—
I am He, I am He. (Shivoham, Shivoham).

न पुण्यं न पापं न सौख्यं न दुःखं

 

न मन्त्रो न तीर्थो न वेदो न यज्ञ ।

 

अहं भोजनं नैव भोज्यं न भोक्ता

 

चिदानन्दरूपः शिवोऽहम् शिवोऽहम् ॥४॥

I am neither sin nor virtue, neither pleasure nor pain ;
Nor temple nor worship, nor pilgrimage nor scriptures,
Neither the act of enjoying, the enjoyable nor the enjoyer ;
I am Existence Absolute, Knowledge Absolute, Bliss Absolute—
I am He, I am He. (Shivoham, Shivoham).

न मृत्युर्न शङ्का न मे जातिभेदः

 

पिता नैव मे नैव माता न जन्मः ।

 

न बन्धुर्न मित्रं गुरुर्नैव शिष्यं

 

चिदानन्दरूपः शिवोऽहम् शिवोऽहम् ॥५॥

I have neither death nor fear of death, nor caste ;
Nor was I ever born, nor had I parents, friends, and relations ;
I have neither Guru, nor disciple ;
I am Existence Absolute, Knowledge Absolute, Bliss Absolute—
I am He, I am He. (Shivoham, Shivoham).

अहं निर्विकल्पो निराकाररूपो

 

विभुत्वाच्च सर्वत्र सर्वेन्द्रियाणाम् ।

 

न चासङ्गतं नैव मुक्तिर्न मेयः

 

चिदानन्दरूपः शिवोऽहम् शिवोऽहम् ॥६॥

I am untouched by the senses, I am neither Mukti nor knowable ;
I am without form, without limit, beyond space, beyond time ;
I am in everything ; I am the basis of the universe ; everywhere am I.
I am Existence Absolute, Knowledge Absolute, Bliss Absolute—
I am He, I am He. (Shivoham, Shivoham).