Maximum number of Files in a Directory

Different file systems in different Operating Systems has different limitations in the number of files that can be in a directory as determined by the underlying architecture.. Here is an overview of maximum of number of files that you can have in a directory and associated problems, if any, like performance.

FAT32

  • Maximum number of files: 268,173,300
  • Maximum number of files per directory: 216 – 1 (65,535)
  • Maximum file size: 2 GiB – 1 without LFS, 4 GiB – 1 with

NTFS

  • Maximum number of files: 232 – 1 (4,294,967,295)
  • Maximum file size
    • Implementation: 244 – 26 bytes (16 TiB – 64 KiB)
    • Theoretical: 264 – 26 bytes (16 EiB – 64 KiB)
  • Maximum volume size
    • Implementation: 232 – 1 clusters (256 TiB – 64 KiB)
    • Theoretical: 264 – 1 clusters

ext2

  • Maximum number of files: 1018
  • Maximum number of files per directory: ~1.3 × 1020 (performance issues past 10,000)
  • Maximum file size
    • 16 GiB (block size of 1 KiB)
    • 256 GiB (block size of 2 KiB)
    • 2 TiB (block size of 4 KiB)
    • 2 TiB (block size of 8 KiB)
  • Maximum volume size
    • 4 TiB (block size of 1 KiB)
    • 8 TiB (block size of 2 KiB)
    • 16 TiB (block size of 4 KiB)
    • 32 TiB (block size of 8 KiB)

ext3

  • Maximum number of files: minimum (volume Size / 213, number of Blocks)
  • Maximum file size: same as ext2
  • Maximum volume size: same as ext2

ext4

  • Maximum number of files: 232 – 1 (4,294,967,295)
  • Maximum number of files per directory: unlimited
  • Maximum file size: 244 – 1 bytes (16 TiB – 1)
  • Maximum volume size: 248 – 1 bytes (256 TiB – 1)

Possible causes & solutions for MongoDB not running

Is your MongoDB properly running? A simple way to check it would be with:

service mongod status

On my CentOS 5.x it gave:

mongod dead but subsys locked

There are two possible causes I could find. First is related to disk space. If you do not have enough space for dbpath (database directory) then you may get this message.

To solve it you can add following to the config file provided when running mongod --config mongod.conf

For MongoDB 3.x (latest version)

storage:
   mmapv1:
      smallFiles: true

For version 2.6+

storage:
   smallFiles: true

For version 2.4 and less

smallfiles = true

Then just execute mongod to accept your config file (here it assumes that location of the config is /etc/mongodb.conf):

mongod -f /etc/mongodb.conf

Documentation for smallfiles parameter:

Set to true to modify MongoDB to use a smaller default data file size. 
Specifically, smallfiles reduces the initial size for data files and
limits them to 512 megabytes. The smallfiles setting also reduces the
size of each journal files from 1 gigabyte to 128 megabytes.
Alternatively you can run mongod with --smallfiles parameter like:
mongod --smallfiles

On my machine the problem was different. It was expecting a directory /var/lib/mongod with proper permissions, which was not there. The solution is simple:

sudo mkdir /var/lib/mongod
sudo chown -R mongod /var/lib/mongod

However how did I find out about the problem(s) in the first place. Again the solution is very simple. I looked in the log file:

vi /var/log/mongodb/mongodb.log

Always start with the log file.

Finally you can verify that your system is running properly with:

sudo service mongod status

After fixing it gave:

mongod (pid 26194) is running...

Now that’s solved, let’s get back to mongodb coding..

Ubuntu: How to Change Host name and Propagate to DHCP Server

How to change Hostname on Ubuntu

First find your current host name by typing:

hostname

Open /etc/hostname and change the host name specified there to the new host name.

sudo vi /etc/hostname

Now open /etc/hosts and locate the old host name in the file and change it to the new host name.

sudo vi /etc/hosts

Now restart hostname with:

sudo service hostname restart

How to propagate Hostname Change to DHCP Server and other machines

The chosen host name should be propagated to the DHCP server so that any other computer can refer to this computer by your new chosen hostname. To do that run the following commands:

sudo dhcclient -r ; sudo dhclient

Now you should be able to access this machine by its newly chosen host name from any other machine in the same network and served the same DHCP server.

An exception would be if a machine is configured statically or not configured to use the name server associated with this dhcp server.

Most routers today comes with DHCP server and an associated Name Server which can be configured as shown above.

Suggestions for Raspberry Pi Demo & Tutorial for Engineering Students

Next week I am giving a demo of Raspberry Pi and a Tutorial to Engineering students, primarily from Electronics & Computer Science department. I am collecting suggestions on what to show them in this seminar. Currently I have thought about the following items

  • How to use Raspberry Pi to create a mini FM Radio Station as a cool example to get them hooked on Pi
  • How to write simple Java Programs with built-in IDE to show the capability of Pi

What else do you think I should show and tell?

Safest way to Update and Upgrade Raspbian Wheezy (Raspberry Pi OS)

There are plethora of instructions on the web to update and upgrade Raspbian and they work too, under normal conditions. However as Raspberry Pi is increasingly being used for Mission Critical tasks, it is important to adopt a procedure which won’t leave you with broken packages or worst, a un-bootable Operating System with your data on it. The following is a simple procedure, with explanations which will make sure that your Pi remains working.

First let’s update and upgrade the Pi with a single line command, run from terminal:

sudo apt-get update -y && sudo apt-get upgrade -y && echo success

This will update and upgrade your Pi and if successful will print success at the end.

Now reboot the Pi with:

sudo reboot

Normally Pi would reboot properly 99% of the time. If it refuses to boot then you can select the old kernel while booting to get back an usable version of Raspbian OS.

After this only two more steps remain. First is to dist-upgrade the Raspbian OS:

sudo apt-get dist-upgrade -y && sudo apt-get autoremove -y && echo success

You  may wonder why I didn’t do it the very first time. While dist-upgrade is better than plain upgrade as it can intelligently install packages based on dependency, upgrade retains the older kernel and gives you a way to back out safely, unlike dist-upgrade. The procedure outlined above protects your OS from cases where the upgrade is broken for any reason.

Now all that is left is to reboot again to make sure everything is OK.:

sudo reboot

You may notice that I didn’t ask you to rpi-update, an instruction you will find at many places online. This is because you do not need rpi-update, the necessary firmware updates are already installed when you upgraded Raspbian. In fact, and I speak from experience, rpi-update can break your running system. Let’s leave that bleeding edge to who really needs it. If you are reading this article, then it can safely be said you don’t.

I used sudo apt-get autoremove to remove any extra packages.

The && before echo success makes sure success is printed only when the previous command succeeds.

Disk space is premium in SD Card. So it may not be out of place to check for available disk space before running update, upgrade or dist-upgrade:

df -h

To save some space clean up the downloaded debian archives with:

sudo apt-get clean

Now you understand why automating Raspbian updates / upgrades is a bad idea, unless you want to use the Raspberry pi only for testing purposes.

Advanced Note on why rpi-update was not suggested:

It’s current Raspberry Pi practice not to use this [ rpi-update ] unless you REALLY need it as it can bring in a non-working kernel (i.e. the latest bleeding edge version, which is not guaranteed to work).

My thoughts are that it is not actually required once you have done the apt-get update/upgrade cycle.

James, Raspberry Pi Foundation (Trading)

How to remove Squeak and Scratch from Raspberry Pi: Raspbian Jessie and Wheezy Distro

Raspbian Jessie and Wheezy distribution includes Squeak and Scratch. While these can be helpful educational tools for students which provides a gentle introduction to programming, it doesn’t have much use for others. The following instructions will remove both Squeak and Scratch from your Raspberry Pi (Raspbian distribution). It saves 92.7 MB and 1.15 MB respectively in Jessie and another 10.2 MB with subsequent autoremove, so over 100 MB space saved.

root@puma:/home/pi# apt-get remove --purge scratch*
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'nuscratch' for regex 'scratch*'
Note, selecting 'squeak-plugins-scratch-dbg' for regex 'scratch*'
Note, selecting 'squeak-plugins-scratch' for regex 'scratch*'
Note, selecting 'python3-pifacedigital-scratch-handler' for regex 'scratch*'
Note, selecting 'libdirectory-scratch-perl' for regex 'scratch*'
Note, selecting 'libdirectory-scratch-structured-perl' for regex 'scratch*'
Note, selecting 'scratch' for regex 'scratch*'
Package 'libdirectory-scratch-perl' is not installed, so not removed
Package 'libdirectory-scratch-structured-perl' is not installed, so not removed
Package 'squeak-plugins-scratch-dbg' is not installed, so not removed
The following packages were automatically installed and are no longer required:
  fonts-roboto gtk2-engines-clearlookspix liblightdm-gobject-1-0
  libxfce4ui-1-0 libxfce4util-bin libxfce4util-common libxfce4util6
  libxfconf-0-2 lightdm-gtk-greeter pimixer pipanel squeak-vm wiringpi
  xfce-keyboard-shortcuts xfconf
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  nuscratch* python3-pifacedigital-scratch-handler* scratch*
  squeak-plugins-scratch*
0 upgraded, 0 newly installed, 4 to remove and 1 not upgraded.
After this operation, 92.7 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 136309 files and directories currently installed.)
Removing nuscratch (20160115) ...
Removing 'diversion of /usr/bin/scratch to /usr/bin/scratch.old by nuscratch'
Removing 'diversion of /usr/bin/squeak to /usr/bin/squeak.old by nuscratch'
Removing 'diversion of /usr/share/scratch/locale/ja_HIRA.po to /usr/share/scratch/locale/ja_HIRA.po.old by nuscratch'
Removing 'diversion of /usr/share/scratch/locale/ja.po to /usr/share/scratch/locale/ja.po.old by nuscratch'
Purging configuration files for nuscratch (20160115) ...
Removing python3-pifacedigital-scratch-handler (2.0.5-1) ...
Purging configuration files for python3-pifacedigital-scratch-handler (2.0.5-1) ...
Removing scratch (1.4.0.6~dfsg1-5) ...
Purging configuration files for scratch (1.4.0.6~dfsg1-5) ...
Removing squeak-plugins-scratch (1.4.0.2~svn.r83-2) ...
Processing triggers for hicolor-icon-theme (0.13-1) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for gnome-menus (3.13.3-6) ...
Processing triggers for desktop-file-utils (0.22-1) ...
Processing triggers for mime-support (3.58) ...
Processing triggers for shared-mime-info (1.3-1) ...
root@puma:/home/pi# apt-get remove --purge squeak*
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'squeak-plugins-scratch-dbg' for regex 'squeak*'
Note, selecting 'squeak-vm' for regex 'squeak*'Mathematica
Note, selecting 'squeak-plugins-scratch' for regex 'squeak*'
Package 'squeak-plugins-scratch' is not installed, so not removed
Package 'squeak-plugins-scratch-dbg' is not iAfter this operation, 1,151 kB disk space will be freed.nstalled, so not removed
The following packages were automatically installed and are no longer required:And 
  fonts-roboto gtk2-engines-clearlookspix liblightdm-gobject-1-0
  libxfce4ui-1-0 libxfce4util-bin libxfce4util-common libxfce4util6
  libxfconf-0-2 lightdm-gtk-greeter pimixer pipanel wiringpi
  xfce-keyboard-shortcuts xfconf
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  squeak-vm*
0 upgraded, 0 newly installed, 1 to remove and 1 not upgraded.
After this operation, 1,151 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 135057 files and directories currently installed.)
Removing squeak-vm (1:4.10.2.2614-1.1+b1) ...
Processing triggers for shared-mime-info (1.3-1) ...
Processing triggers for gnome-menus (3.13.3-6) ...
Processing triggers for desktop-file-utils (0.22-1) ...
Processing triggers for mime-support (3.58) ...
Processing triggers for gnome-icon-theme (3.12.0-1) ...
Processing triggers for man-db (2.7.0.2-5) ...

Don’t forget to run clean and autoremove:

apt-get clean
apt-get autoremove

Now you are done!

The commands were run as root. However you can do the same from pi login by prefixing sudo at the beginning of each command. Here is a single-line version of the above using sudo:
sudo apt-get -y remove --purge scratch* squeak*; sudo apt-get clean; sudo apt-get autoremove

How to remove Wolfram Language and Mathematica from Raspberry Pi: Raspbian Jessie and Wheezy distro

Raspbian Jessie and Wheezy distribution includes Wolfram Language and Mathematica. While this can be helpful for students, it doesn’t have much use for others. The following instructions will remove both Wolfram and Mathematica from your Raspberry Pi (Raspbian distribution). It saves 658MB in Jessie.

sudo apt-get remove wolfram-engine
Reading package lists... Done
Building dependency tree 
Reading state information... Done
The following packages will be REMOVED:
 wolfram-engine
0 upgraded, 0 newly installed, 1 to remove and 59 not upgraded.
After this operation, 658 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 150191 files and directories currently installed.)
Removing wolfram-engine (10.3.1+2016012407) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for gnome-menus (3.13.3-6) ...
Processing triggers for desktop-file-utils (0.22-1) ...
Processing triggers for mime-support (3.58) ...
Processing triggers for shared-mime-info (1.3-1) ...
Processing triggers for hicolor-icon-theme (0.13-1) ...

Then there is one final step:

sudo rm /usr/share/raspi-ui-overrides/applications/wolfram-language.desktop /usr/share/raspi-ui-overrides/applications/wolfram-mathematica.desktop

And you are done!

PS. You can read more about these free (not for commercial use) packages here.

Code: How to reboot TP-Link Router

TP-Link router may need to be occasionally rebooted as it sometimes hangs in my experience. Here is a script I developed long back to boot such errant server from command line:

wget -qO- --user=admin --password=admin-password http://192.168.1.1/userRpm/SysRebootRpm.htm?Reboot=Reboot

Notes:

  • You will need wget installed on your system.
  • You may need to change the IP Address and Password.

If you want (let me know) I can give the script to check periodically and reboot TP-Link Router as needed.

BitBucket vs. GitHub vs. ProjectLocker: Where would you setup your closed source project for free and why

Each of the Source Code repositories come with a free tier but each has their own restrictions

BitBucket from Atlassian is free for small team up to 5 users or you can self-host for up to 10 members for one time fee of 10$. With the free plan in cloud, you can have unlimited public and private repositories.

GitHub doesn’t allow private repositories at this time. While you may host closed source project there, I wouldn’t recommend it.

ProjectLocker provider a free tier: 1 user, 1 project and up to 50 MB of storage.

All three supports Git which is probably the most widely used Source Control system these days.

Of the three BitBucket stands out in hosting closed source private repositories for free. Overall their pricing is most lucrative in the long run.
GitHub, while not free, has the most traction.

Mini-Story: Deathly Silence of Linux (after rm -rf /*)

Silence of Linux (after rm -rf /* )

To be fair, rm -rf /* was taking way too long with several millions of files in deep hierarchy. So I ran Delete (Java based recursive File & Directory Delete Tool).

Finally all data that could be deleted was deleted. I could’ve run shred, I could’ve run dd on the drive itself ( dd if=/dev/urandom of=/dev/sda ), I chose the simpler path.
-bash-3.2# clear
-bash: /usr/bin/clear: No such file or directory
-bash-3.2# ls
-bash: /bin/ls: No such file or directory
-bash-3.2# echo *
boot dev media misc net proc sys var
-bash-3.2# echo boot/*
boot/*
-bash-3.2# echo dev/*
dev/initctl dev/pts dev/shm
-bash-3.2# echo misc/*
misc/*
-bash-3.2# echo net/*
net/*
-bash-3.2# echo net/*
net/*

Only one thing left to do:

-bash-3.2# reboot
-bash: reboot: command not found

I still have the final weapon:

-bash-3.2# echo 1 > /proc/sys/kernel/sysrq
-bash-3.2# echo b > /proc/sysrq-trigger

This is pretty much the same as pressing the reset button on the server (if equipped). No daemons will be shut down gracefully, no filesystem sync will occur, and you may get the wrath of a fsck (or worse, a non-booting server) upon reboot. But what do I care?

R.I.P.