Author: Yazeed

I am getting old, my brain aint all that bright no more, for example i am so used to using

mysqldump –opt -u root –password=”LaooIa12@Hsu” mytodolist | gzip > mytodolist.bin.gz

So to compress the database while dumping it without having it hit the disk first in what is called piping.

the problem is that i didn’t think of checking whether pbzip2 supports this so that i can use all 6 CPU cores (12 vCPUs)

So, turns out that pbzip2 (parallel BZIP) does work

mysqldump –opt -u root –password=”LaooIa12@Hsu” mytodolist | pbzip2 -vc > mytodolist.bin.bz2

Now, i can get it done much much faster, 6-12 times faster

In short, it is the average space between the processor’s logic gates (transistors).

It makes all the difference in speed, and a considerable difference in power consumption.

For example, i ran a certain task on both of the following processors

E3300 which is a low cost celeron processor with a lithography of 45nm and (1M Cache, 2.50 GHz, 800 MHz FSB)
Q6600 Which is a much more expensive (at the time when both were purchased) with a lithography of 65nm and (8M Cache, 2.40 GHz, 1066 MHz FSB)

When comparing a single core’s throughput, the cheap celeron processor beat the quad core by a very considerable number, much higher than the difference in clock speed, The actual numbers would need me to explain many factors such as the nature of the millions of records that needed processing, how they were processed, how jobs were distributed between computers, how the random sample is guaranteed to be random and so on, and i don’t think this is very relevant to you.

So, lithography is something you should really consider when buying a processor, the lower the better, my laptop’s I7 is built with a lithography of 22nm, this is the best number as of 2013.

My mission: at the office there are WINDOWS computers that don’t run 24/7, they are on a few hours a day, and i would like to make those computers do some processing for my programs at night and when they are not in use. to run my programs they need to be running linux, so here is my plan.

One very simple way is to install a second hard drive on those computers and boot from it (Because i dont want to touch drives that have people’s data. But this is alot of work, and i don’t want to end up having people accidentally booting the second drive or even have an extra spinning disk in the computer that does absolutly nothing while people are working with windos.

So, My plan is to get cheap 4GB flash pen drives and boot all those computers from them.

Step 1: get debian live (http://live.debian.net/) iso file

I am downloading debian-live-7.0.0-amd64-standard.iso.log But if you want a desktop GUI environment like gnome or KDE you should get a different file.

Step 2: Dump the ISO file onto the flash stick, on windows this can simply be done with
Step 3: Change some stuff to make the live image persistent and not read only (Could be step 2, or 3 if we mount the USB and edit directly)
Step 4: Deploy

Simple steps to do so, Mind you, i created the following torrents to download all files, the first 3 (3 of 32 and 3 of 64) torrent files are simply the same ones as the ones posted on debian.org, the rest i am managing, if you want to use jigdo instead here are the simple steps.

apt-get update
apt–get install jigdo-file

wget http://cdimage.debian.org/debian-cd/7.1.0/amd64/jigdo-dvd/debian-7.1.0-amd64-DVD-1.jigdo
wget http://cdimage.debian.org/debian-cd/7.1.0/amd64/jigdo-dvd/debian-7.1.0-amd64-DVD-1.template

jigdo-lite debian-7.1.0-amd64-DVD-1.jigdo

And in no time you will have the file, you can resume any time by issuing the command again

Happy downloading

The following is the shortest tutorial that should get you up and running…

1- install rtorrent

apt-get install rtorrent

2- download the torrents

wget http://cdimage.debian.org/debian-cd/7.0.0/amd64/bt-dvd/debian-7.0.0-amd64-DVD-1.iso.torrent

3- start rtorrent

rtorrent

Hit enter on the screen you are on, then hit tab, you should see a list of the torrent files you downloaded, type the first letter of your file then tab again, the program will either show you your file , or reduce the list to all the files that have that same prefix, add one more letter then hit tab again, keep going until you have your file.

Once done, use the arrow keys to select your torrent file, then hit ctrl+s to Start the torrent.

That’s all you need to know !

Using the top command, you can find out how much time your processor spends waiting for input and output from disk, the IO wait field, but how can we know what applications are responsible for that IO wait.

to examin disk io and iowait more closely, we need some tools, the following is a list of tools that can show you exactly what is causing all the IO wait so that you can start from there.

Sometimes, apps that use the disk with many random reads should go to the secondary disk. but to find that application, take a look at the following tools, i will be explaining each and every one of them as we go.

Here are a bunch of programs on linux that will tell you just that

• iotop – apt-get install iotop – simple top-like I/O monitor
• iostat – apt-get install sysstat – shows and monitors activity per disk – The sysstat package contains the sar, mpstat and iostat commands for Linux.
• isag – apt-get install isag – Interactive System Activity Grapher for sysstat
• dstat – apt-get install dstat – versatile resource statistics tool
• vmstat 1
• iostat 1
• lsof
• strace -e trace=open
• strace -e trace=open -p
• In ps auxf you’ll also see which processes are are in uninterruptible disk sleep (D) because they are waiting for I/O.
• fatrace – Amazing tool showing you which files are being accessed or written on the hard drive

examples

iostat -c -d -x -t 2 100

I got myself a new C850B797 toshiba laptop with an i7-3630QM processor and 16GB of ram (coz it’s cheap), the C850B797 is the grey edition in the C850 series, the processor is a 22nm Lithography and everything is very good as far as i can see, but then i realized that the laptop had one very serious issue, i am limited to 100mbit Ethernet, and i really do move plenty of files around all the time (many gigabytes), so i needed to get gigabit, but how ?

Usb speed is 480mbit/second, much of that goes overhead, so the Ethernet controller would probably be USB3 which would block my only usb port, so the answer is to get a USB3 hub and connect the USB gigabit ethernet to it, but it turns out that the hub followed by gigabit eithernet is already on the market from one of 2 providers.

The first, the one i got is the StarTech ST3300U3S USB hub and Ethernet controller, that uses ASIX AX88179 for gigabit Ethernet after a VIA VL811 USB3 hub, the question is, will i be getting the speeds i am hoping for from this ?

I will be running tests concerning reliability of the combo (since both are connected in serial) and let you guys know.

1- StarTech ST3300U3S USB hub and Ethernet controller – Power adapter included = $55, Also happens to be the same exact thing as the SYBA inf@zone (infozone) with the same chips and the same plastic casing, the SYBA infozone will sell at $33 if you buy 3 together, i found 3 for $98.99 on ebay, making it much cheaper than the $66 each for the same exact thing from StarTech, SYBA labels it as SY-HUB50056.)

AX88179 — USB3.0 to 10/100/1000M Gigabit Ethernet Controller
VIA VL811 – SuperSpeed USB 3.0 Hub Controller

The other options on the market are…
2- Kanex DualRole Gigabit Ethernet with 3-Port USB Hub (USB3GBITX) – Power adapter not included – $40
AX88179 — USB3.0 to 10/100/1000M Gigabit Ethernet Controller
No idea about the hub chip yet.

Please bare in mind that this same combo but with USB 2 and a 100Mbit eithernet adapter should not cost more than $10, it is the USB 3 and gigabit Ethernet that are expected to be this expensive.

Via also produces the VL812 which is almost identical to VL811

Disk Spin down (Tested with Bullseye 2022)

Even though everything concerning block devices in linux has shifted to unique identifiers, hdparm has not, and will still use the old /dev/sdx system

To control disk spindown, and to manually issue commands, you will need to have the package installed

apt-get install hdparm

There is a probelm with disk spindown via hdparm, the problem is that you must address a disk as /dev/sdc , which changes in the case of USB media and other disks, even when you add slaves,

hdparm -Y /dev/sdb will spin a disk down instantly

hdparm -S 240 /dev/sdb will set this disk to sleep when idle for 20 minutes (5 second units here)

or adding at the bottom of the file /etc/hdparm.conf a section such as

/dev/sdc {
spindown_time = 240
}

to make those changes persistent across reboots.

To check the status of a disk, here is what you do

hdparm -C /dev/sde

You could get one of the following results
When spun down…
drive state is: standby
When active
drive state is: active/idle

Don’t make your disks spin-down too often, 20 minutes is good for me almost in all circumstances.

If the disks don’t spin down, chances are that selftest is enabled…

Check if it is enabled with

smartctl -a /dev/sdb
if it reads
Auto Offline Data Collection: Enabled.

then you need to disable it with
smartctl --offlineauto=off /dev/sdb

then wait for them to finish (if a test is running) then spin down.

1- Installed the debian base system, debian wheezy (7) (But should work with debian squeeze (6))
2- apt-get install ssh openssh-server fail2ban
3- INSTALLED THE HARDWARE 8 CAMERA DVR CARD into the PCI slot
4- Modified the network interfaces to use a static IP, and pointed the DMZ of the router to that IP
5- installed some software (Even though some of them would have been installed anyway when i install zoneminder), but i do it anyway just in case they are optional in zoneminder

apt-get update
apt-get install apache2 php5-mysql libapache2-mod-php5 mysql-server
apt-get install ffmpeg libarchive-tar-perl libarchive-zip-perl libdate-manip-perl libdevice-serialport-perl
apt-get install libjpeg62 libmime-perl libstdc++6 libwww-perl zlib1g
apt-get install zoneminder

I can see close to the end of operation

ZoneMinder is stopped
invoke-rc.d: initscript zoneminder, action “status” failed.
Starting ZoneMinder: success

I hope it’s nothing too serious

Now, check if your card is supported, you issue lspci -v as root… Since my 8 port card is supported, i see eight stanzas of this

I got eight of the following labeled 03:02.0 03:02.1 03:02.2 03:02.3 03:02.4 03:02.5 03:02.6 03:02.7

03:02.2 Multimedia video controller: Techwell Inc. TW6816 multimedia video controller (rev 10)
Subsystem: Techwell Inc. TW6816 multimedia video controller
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 3
Memory at febfe800 (32-bit, non-prefetchable) [size=1K]
Capabilities: [44] Power Management version 2

If you dont see your card’s devices, well, your out of luck, maybe you should try a newer kernel, i am using 3.2 from debian wheezy (7)

Now, to the second test

ls -ls /dev/video*

NO love, nothing, If i were lucky, i would get something like

ls -ls /dev/video*
0 crw-rw----+ 1 root video 81,  0 30. Jul 15:18 /dev/video0
0 crw-rw----+ 1 root video 81,  2 30. Jul 15:18 /dev/video1
0 crw-rw----+ 1 root video 81,  4 30. Jul 15:43 /dev/video2
0 crw-rw----+ 1 root video 81,  6 30. Jul 15:43 /dev/video3
0 crw-rw----+ 1 root video 81,  8 30. Jul 15:17 /dev/video4
0 crw-rw----+ 1 root video 81, 10 30. Jul 15:17 /dev/video5
0 crw-rw----+ 1 root video 81, 12 30. Jul 15:17 /dev/video6
0 crw-rw----+ 1 root video 81, 14 30. Jul 15:17 /dev/video7

But i did not, so i need the driver to my techwell card.

Simple, there is a nice person who made a driver already, let’s see how we can install it

First, we need GIT so that we can download the driver

apt-get install git
apt-get install linux-headers-$(uname -r)
apt-get install make sudo

(without installing sudo, the compile gives the error /bin/sh: 1: sudo: not found which is because the guy probably wrote the driver with ubuntu in mind)

git clone git://gitorious.org/tw68/tw68-v2.git
cd tw68-v2
make
make install

So, i was expecting 8 cameras, i got the following

ls -ls /dev/video*
0 crw-rw---T 1 root video 81, 0 May 12 23:12 /dev/video0
0 crw-rw---T 1 root video 81, 2 May 12 23:12 /dev/video1
0 crw-rw---T 1 root video 81, 4 May 12 23:12 /dev/video2
0 crw-rw---T 1 root video 81, 6 May 12 23:12 /dev/video3

probably a driver issue, oh well, lets see what we can do with 4 then i will see about this issue once i have 4 working:D

ln -s /etc/zm/apache.conf /etc/apache2/conf.d/zoneminder.conf

adduser www-data video

/etc/init.d/apache2 restart

Now, visiting 192.168.2.101/zm/ allows me to add cameras, the matter is very intuitive from that point on, all you need is a browser, remember you will need to password protect things too :D so that people from the outside can not look at your cameras.
	

This post is somewhat old, and kept here for historical reasons, if you want to run LXC containers on Debian Bookworm (12), I have composed a much more useful post here

Yes, Wheezy is out to the public, and openVZ is out of Wheezy, so what to do.

Basically, what i am doing now is investigating the alternative LXC, i have no time to learn right now, so i am going to have to do this fast.

I have a gut feeling that LXC is better than openVZ, after all, it is in the mainline kernel, and it is supposed to be marvelously easy to install, so let me start working on this with everyone here.

NOTES: if you want to give away LXC containers to people, you will need to use AppArmor with it, here, i run my containers, so i will not be installing AppArmor in this tutorial, but maybe soon i will add a tutorial for the AppArmor part.

So, LXC here we come, to completely replace openVZ, with something more open (Sorry Parallels Virtuozzo, welcome IBM), something that can keep up with the kernel and not keep us behind.

I will be turning this post into a tutorial on installing and running LXC on debian wheezy (7) with memory allocation to containers and with the kernel that shipped with wheezy, i should be done creating this tutorial in a few days, and it will remain an incremental effort where i will be adding more and more as i learn about this.

NOTES: memory allocation is not compiled with the kernel by default but disabled, you enable it by adding a parameter to grub. (Not anymore, now memory allocation works out of the box)

1- Install base system of wheezy (debian 7)

2- Install some stuff i can never do without

apt-get update

apt-get upgrade

apt-get install ssh openssh-server fail2ban

fail2ban is a very important application that will prevent outsiders from bruit force cracking your server, it is very important, without it you will be hacked sooner or later (especially if you are in a datacenter), hackers look for servers to send spam from all the time.

Now, we need to specify a hostname for this machine (the LXC HOST), i want to call mine server5.example.com

echo server5.example.com > /etc/hostname

/etc/init.d/hostname.sh start

hostname

hostname -f

apt-get install ntp ntpdate

Now, we need to setup networking for LXC, every physical NIC (Network adapter) will need a bridge.

To create a bridge, you need to install

apt-get install bridge-utils

Then your /etc/network/interfaces file must look like this

------------------------------------------------
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
  auto lo
  iface lo inet loopback
# The primary network interface
  #allow-hotplug eth0
  #iface eth0 inet dhcp
#Bridge setup
auto br0
  iface br0 inet static
  bridge_ports eth0
  bridge_fd 0
  address 192.168.2.121
  netmask 255.255.255.0
  gateway 192.168.2.1
  dns-nameservers 8.8.8.8
------------------------------------------------

apt-get install lxc

You will be presented with the following prompt, i myself accept the default /var/lib/lxc

Please specify the directory that will be used to store the Linux Containers. If unsure, use /var/lib/lxc (default). LXC directory:

mkdir /cgroup

Add the following line in /etc/fstab using a text editor:

cgroup /cgroup cgroup defaults 0 0

mount -a

Now, to make sure everything is working like it should

lxc-checkconfig

------------------- OUTPUT OF lxc-checkconfig ----------------START

Kernel config /proc/config.gz not found, looking in other places...
Found kernel config file /boot/config-3.2.0-4-amd64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig.

------------------- OUTPUT OF lxc-checkconfig ------------------END

And on the host machine, you need to enable IP forwarding befor you fire up any of those LXC containers

 echo 1 > /proc/sys/net/ipv4/ip_forward

But to make that peppermint you need to edit the file /etc/sysctl.conf where we can add a line containing net.ipv4.ip_forward = 1

/etc/sysctl.conf:

net.ipv4.ip_forward = 1

You might find that the entry is already there but with the value 0, in that case just flip the zero to a 1, or you might find it there but commented out, in that case, delete the # that precedes that line to enable it.

To enable the changes made in sysctl.conf (And you don’t if you already executed the echo 1 statement above) you will need to run the command:

sysctl -p /etc/sysctl.conf

Now that LXC is officially installed, there is more than 1 way to create containers, debootstrap is one of them (you will need to install it, and the container config will need to be done manually by adding a few lines into a file you create inside the container area), while i will use the LXC way by using the application lxc-create you are free to use any tool, including importing containers from vmware (copying vmware containers will work).

Also worth mentioning, i use apt-cacher so when i am asked about the urls of the distro, i simply modify it to read http://192.168.2.133:3142/ftp.us.debian.org/debian/ which is how i accerss apt-cacher to speed up things and not re-download everything every time.

So, lets start

lxc-create -t debian -n vm33

On a newer releast (7.7), the above gave me an error, so the following was the error and the solution (needed command)

MIRROR=http://ftp.us.debian.org/debian lxc-create -n vm10 -t debian -- -r wheezy

Or if you want to use apt-cacher

MIRROR=http://192.168.10.237:3142/ftp.us.debian.org/debian lxc-create -n vm10 -t debian -- -r wheezy

1- Preseed file anyone? Enter (optional) preseed file to use: <== leave this one empty

2- Chose the distro (debian wheezy for me)

3- 64 or 32, i use 64

4-
Archives.

[*] Debian Security

[*] Debian Updates

[*] Debian Backports

[ ] Debian Proposed Updates

5- Mirror.

i modify this to read http://192.168.2.133:3142/ftp.us.debian.org/debian/ in order to use my apt-cacher, you can put any mirror here, or leave the default one (http://ftp.debian.org/debian/ Mirror Security http://security.debian.org/ and Mirror Backports) provided for you. Archive areas Main, Packages (leave blank or specify the packages you want, you can install them later with apt-get), then the root password

You must keep in mind that even after you see the message ‘debian’ template installed ‘vm33’ created, the config file for vm33 is not really ready, you need to enable networking in it manually. so, let’s edit the file /var/lib/lxc/vm33/config and add networking support

vi /var/lib/lxc/vm33/config

NOTE: THE BELOW IS FOR TYPICAL SETUPS, FOR HETZNER DATACENTER, PLEASE SEE THE POST ON LXC NETWORK SETUP WITH HETZNER.

then add the lines right before #Capabilities and after the lines of ## Container

lxc.network.type = veth

lxc.network.flags = up

lxc.network.link = br0

lxc.network.name = eth0

lxc.network.ipv4 = 192.168.2.125/24

Also, before we start the container, there are a few things we need to do…

there seems to be an issue with the ssh keys, so what we will do around this issue is copy the keys from the host, (We will generate new ones for the conatiner later)

EXECUTE ON HOST

cp /etc/ssh/ssh_host_dsa_key /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_dsa_key
cp /etc/ssh/ssh_host_dsa_key.pub /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_dsa_key.pub
cp /etc/ssh/ssh_host_ecdsa_key /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_ecdsa_key
cp /etc/ssh/ssh_host_ecdsa_key.pub /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_ecdsa_key.pub
cp /etc/ssh/ssh_host_rsa_key /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_rsa_key
cp /etc/ssh/ssh_host_rsa_key.pub /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_rsa_key.pub

Then, they won’t work without proper permissions

chmod 0600 /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_dsa_key.pub
chmod 0600 /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_ecdsa_key 
chmod 0600  /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_rsa_key

Now i reboot the server just to be on the safe side, then i do the following

lxc-start -n vm33 -d

lxc-info -n vm33

When you run the command for information, you should see the word RUNNING and a pid.

Just SSH to the host !

Now if you want to create new host keys for SSH just do the following

delete the files

/var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_dsa_key.pub
/var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_ecdsa_key
/var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_rsa_key

execute

dpkg-reconfigure openssh-server

—————————————

Making LXC auto start at the system boot
The old Way – create a symbolic link, should still work, but i have not tried

ln -s /var/lib/lxc/vm34/config /etc/lxc/auto/vm34_config

The new way that provides better control of the order they are started in.
Set lxc.start.auto == 1 in the config

Then, the following will tell the system what containers to start first, and when