QWORQS

The things i add here are primarily for my reference, but they could save you a lot of time, so here i am only covering the practical side, wikipedia can help you with the theory. things here are sort of in random order, so feel free to use your browser’s search facility.

I have the following positioning systems/chips/boards, so most of the stuff you will find here will relate to them

* V.KEL VK2828U7G5LF TTL Ublox GPS module with antenna: supports GPS, GALILEO, SBAS (WAAS, EGNOS, MSAS, GAGAN)
* K-172 USB GPS USB Receiver Dongle Adapter Smart Antenna Module For Gmouse Glonass, also running u-blox, Support NMEA 0183 and ublox binary protocol.
* The ADAFRUIT ultimate breakout board with the MTK3339 chip (Not U-BLOX), but rather mediatek

A-GPS and AssistNow:

The long story short: A-GPS seems to be when the satellite data is available in advance, whether you allow your system to download it from the internet as it goes, or you download it in advance and make it available to the system, it boils down to telling your system where the satellites are and their relevant data rather than wait for your device to download that data from the satellites at very low speeds (Sat data is downloaded at a maximum of 50 bits per second, so getting the data over the internet or from the SD card in your raspberry pi or arduino etc is much much faster).

AssistNow is how the U-Blox A-GPS works

1- A-GPS only work out of the box with u-center, the software by u-blox, on your raspberry PI or arduino, it will not work out of the box , to make it work you need to look online for software that does that (It’s on github)

the online edition is when the data is downloaded from the internet in real time (when it is needed), the AssistNow  offline is when you download “AlmanacPlus® Differential Almanac Correction Data” from the internet and store it with your device (On whatever is driving the device, be it a PC, laptop, raspberry pi, arduino etc…)

2- The adafruit GPS raspberry pi board does A-GPS in a different way, which works out of the box, they add a battery to the board, and once data is downloaded once, it stays on the board for as long as there is a battery 😉

 

 

I have this bad habit of copying things so many times when modifying them, when that is a large database, we talking many gigas, so here is a script to find those duplicate files among many hard drives and telling you which ones are duplicates, moving and deleting and symbolic linking is done manually after.

1- this script is PHP-CLI, so make sure that is installed on your computer
2- this script runs the find command, make sure it can execute that program
3- you run the script with the path parameter, but will need to edit the script to change the 1GB size i have hard coded

What this script does is

1- find files with size greater than 1GB (find /hds -size +1G)
2- Store the files in database with size
3- retrieve the files ordered by size
4- if 2 files have exactly the same size, calculates MD5sum for the first MB of the file
5- If the MD5 of the first MB of the files are the same, calculate the whole MD5
6- If they turn out to be duplicates, they are printed to the command line

About this tutorial

Despite being lengthy, this tutorial is in fact easy and fast, I have split it to parts so that you can get down to business instantly if you need to.

Worth mentioning is that i think this simple procedure presents itself as rocket science, it is not, so advise you to dive in (experimenting on a separate computer first may be a good idea), again i assure you it is VERY STRAIGHT FORWARD, the length is because i am elaborating to make it easy.

Disclaimer

This is an effort to put all the information i need about bcache in one place for my referance and your benefit, but please beware, bcache should be run with backup (You will have to come up with things as raid will render the cache redundant for example and rsync for big files might make your CPU do a lot of work), in any case, i am not responsible and will not be held liable for any damage you may endure.

SSDs are the future

When it comes to SSDs, I would say they have come a long way in terms of price, and one day they will be replacing hard drives, I have no doubt about that, there is no advantage in a hard drive that an SSD can’t eventually match (You might argue that TBs written, maybe, but have you tried to check the reliability of a hard drive stressed to the level needed to achieve those TBs written ?).

What is bcache for

Spinning hard drives are fast beasts when it comes to sequential reads, but when it comes to random reads where the head has to go seek the data, they become very very slow, you can be reading at 200MB/s and suddenly drop to 2MB/s, While SSDs do not suffer this much from random reads, slower than sequential, yes, but nothing close to the gap you see in spinning disks, in a spinning disk, the speed difference can be 100 fold OR MORE.

History (Windows)

The earliest attempt that i can remember was Intel robson (2005), Intel robson or intel turbo memory was a feature in the Core 2 CPUs, but i don’t think it made it up to the Core I, it was not very popular and for a good reason, at the extra cost, OEMs could add more ram, not only would it be better for marketing, it also made more sense, as Windows was already introducing memory cache for disks with windows Vista.

Some time later, microsoft came up with Microsoft ReadyBoost (With windows Vista), readyboost relied on fast pen drives to cache the data from the spinning disk, it was not a very popular feature at the time for many reasons, the drawbacks is that they had to design it to be pulled out without affecting data integrity, making restrictions on the writing speed (Writethrough, can’t writeback), and still it was doing the stuff that RAM did perfectly. not to mention that affordable pen drives were not that fast to begin with.

Caching today.

As it is today, caching still makes sense, I would argue it makes more sense than ever, spinning hard disk drives are still much cheaper than SSDs, A good SSD, A 1 TB SSD from samsung is at around $340 for the EVO, and 460 for the pro (Jul 2017), compare that to the spinning disk, with a price tag averaging $40, and you will know that the difference is still around 10 fold, even more if you go up in size, So what do we do ?

The answer is cache the disk. now is a better time to use caching with super fast SSDs that employ wear leveling and are connected in a more stable and persistant connection (SATA inside the computer).

SSD caching On Windows.

On windows, the answer may be ISR (Intel Smart Responce), I have not tried it myself, but i have heard many good things about it, you get into your bios and set the disks are R.A.I.D, then use the Intel Management Engine software to cahce the spinning disk on the SSD, that simple.

I could almost swear INTEL had a software solution for this that was a bit pricy, but i can’t seem to find it, i remember watching a video about it many years ago.

In any case, I am not very experienced with windows, so I will just leave it here.

SSD disk caching on Linux

On Linux, there are many solutions, the one that i will be showing you how to use right now is bcache, because it is fast, efficient, and works on block devices.

So, I am assuming you have installed debian stretch (9), and you have logged in, and you have networking et al running, now, let us get to installing bcache, mind you, bcache has been part of the linux kernel since jessie or even before, so all you need is bcache-tools, in Jessie, you had to compile those with a few lines, in stretch, there is a package for it.

** BCACHE **

To help avoid the confusion, you can use your big hard disk before attaching an SSD, you can then, whenever you want, attach an SSD to it to start the performance gain.

Installing bcache tools in Debian Jessie (8)

** IF YOU ARE INSTALLING ON JESSIE, BCACHE TOOLS WERE NOT PACKAGED FOR JESSIE**

apt-get install git make gcc pkg-config uuid openssl util-linux uuid-dev libblkid-dev

git clone https://github.com/g2p/bcache-tools.git
cd /usr/src
cd bcache-tools
make
make install

** END OF FOR JESSIE **

Installing bcache tools in Debian Stretch (9)

apt-get install bcache-tools

Planning how to setup the drives

In this article, i will be setting up 2 separate disks that are not system disks, one is a 4TB spinning disk, the other is a 1TB SSD, there are a few rules that you need to be aware of though

1- You can cache as many backing devices as you wish with one SSD
2- You can not cache one backing device with more than one SSD

3- There are memory requirements for bcache, so for example dropping the disks in a 486 computer with 256mb ram and using iscsi is not a good idea .

My setup

The backing device is your large spinning disk, the caching device is the SSD

My backing device is a 4TB hard drive that is connected as /dev/sde
My caching device is a 1TB samsung 850evo (alignment considerations here since it is a tlc disk (the pro is MLC, works like a regular with no alignment issues)), connected as /dev/sdc

Setting up the backing device (sde), mounting and populating it with data

You may want to start with the following command to clear any existing filesystem from the drives (Change SDE with your own drive designation)

wipefs -a /dev/sde

Now, let’s format SDE as backing, and SDC as caching

1- Run parted for backing device

parted /dev/sde
mklabel gpt
mkpart primary ext4 0% 100%

2- Make it a bcache backing partition

Using make-bcache, you will use the -B switch to tell the system that this is the backing device, meaning the spinning disk

make-bcache -B /dev/sde1

output from the above will be something like

UUID:                   19d92bc8-8f49-479a-9480-33ca659b91b2
Set UUID:               0e3f386a-ec62-42b9-b0f3-025a09253946
version:                1
block_size:             1
data_offset:            16

3- Format it as ext4 or whatever filesystem you fancy

mkfs.ext4 /dev/bcache0

4- Mounting it like you would mount any other partition

mount /dev/bcache0 /hds/bcache0

5- If you like, you can now copy your data to it and get things ready before installing the caching device (before attaching the SSD as cache).

as i prefer to copy all the files to the spinning disk before attaching the SSD, since when we copy sequential, the SSD does not cache anyway, but the things it does cache are not the things we will use frequently, So i copy my files to it first, then i attach the SSD.

Setting up the caching device (sdc), then attaching it to the backing device

1- Create a partition on caching device (you chose the size you want to use as cache), but i would recommend that if you want to use the whole disk that you leave 10% unpartitioned for over-provisioning.

wipefs -a /dev/sdc

parted /dev/sdc
mklabel gpt
mkpart primary ext4 0% 90%

Using make-bcache, you will use the -C switch to tell the system that this is the caching device, meaning the solid state disk (SSD)

make-bcache -C /dev/sdc1

output from the above will be something like

UUID: eeda3570-eb1b-4983-8c53-76322a654585
Set UUID: 92dbf6ca-0f0b-44d5-b70e-8f1e7919838d
version: 0
nbuckets: 1716964
block_size: 1
bucket_size: 1024
nr_in_set: 1
nr_this_dev: 0
first_bucket: 1

Now, even if this is not for a technical purpose, just to give you the feel of this, try running the command below, the command should result in “no cache” because we did not attach a cache to it yet

cat /sys/block/bcache0/bcache/state

DO NOT Format the caching partition as ext4

this time, we won’t be formatting it in ext4 like the backing device above (think about it, the OS should see the backing device, and at some abstraction layer not even know about this one, so why would it have a file system other than the one that bcache itself understands), we will simply be attaching it to the disk.

Attaching the caching device

If you take a look at the result from make-bcache -C command, you will notice a Set UUID, we will need this unique ID to tell bcache what SSD to connect to what cache, the only cache we have so far is bcache0 as you can see from above, here is how we attach it.

echo 92dbf6ca-0f0b-44d5-b70e-8f1e7919838d > /sys/block/bcache0/bcache/attach

Now, if we run the command above again

cat /sys/block/bcache0/bcache/state

It should read “Clean” or “Dirty” instead of “no cache” (I would bet it reads clean at this stage), Depending on whether something has been written to it and still not in the backing device, or clean otherwise.

Setup all done, unless you want to fine tune it for your purpose, then read on.

Tuning the cache.

1- Caching mode

to inspect what caching mode we are using now

cat /sys/block/bcache0/bcache/cache_mode

Which will probably result in

[writethrough] writeback writearound none

By default, the system uses writethrough (better data integrity), but if you are like me, and have made 100% sure the electric won’t ever go down, or if you backup the data in real time, you might want to switch to writeback, writeback gives much faster write operations which is not necessarily a requierment for all applications.

echo writeback > /sys/block/bcache0/bcache/cache_mode

2- sequential read cutoff

The other thing you might wish to tune is the size of the sequential read/write cutoff, we want a size short enough to be worth caching, by default, it is 4MB, so that everything under 4MB sequential will be cached, I personally like to take that down to 1MB judging by the fact that files larger than 1MB do read pretty fast directly from the disk ! but surely, this will depend on your application and on experimentation with your application.

cache 1 megabyte and smaller

echo 1M > /sys/block/bcache0/bcache/sequential_cutoff

cache everything (special value, not the same mathematical logic of less than)

echo 0 > /sys/block/bcache0/bcache/sequential_cutoff

back to caching 4 mega bytes and smaller (default)

echo 4M > /sys/block/bcache0/bcache/sequential_cutoff

3- Percentage of dirty data to allow on SSD.

I personally like it the way it is (10% of the SSD’s size), but you can change that, and sometimes you have to temporarily change that for certain purposes)

Flush all dirty data to disk as soon as you can

echo 0 > /sys/block/bcache0/bcache/writeback_percent

Allow 10% dirty data

echo 10 > /sys/block/bcache0/bcache/writeback_percent

the first (Value 0) is very usefull when you want to disconnect the cache, to disconnect you want the dirty_data to be 0 on the SSD, so you can start by issuing the first line above, then as soon as all the data is flushed to the backing device, you can disconnect the SSD like i will be showing you further down.

Manipulating the setup

Sometimes, you want to change your SSD with a larger or smaller or newer one, other times, you want to disconnect it and use the backing device without a cahce, other times, you want to use the same caching device to cache more disks, here i will show you how

Assuming you want to disconnect the SSD, for this to happen, you will need to go through a couple of steps, first, make sure there is no dirty data, and second, detach it from the backing device

For the first step, we should inform bcache that we don’t want any dirty data, by default, bcache allows for 10% of the size of the SSD to be dirty data, we need to make that ZERO percent

echo 0 > /sys/block/bcache0/bcache/writeback_percent

remember, if you reattach or otherwise, you should set it back to ten percent in the same way

echo 10 > /sys/block/bcache0/bcache/writeback_percent

Monitoring cache and cache performance

1- How much dirty data is on the SSD, Assuming that “/sys/block/bcache0/bcache/state” reads dirty, you can see how much data is dirty with the command.

cat /sys/block/bcache0/bcache/dirty_data

2- Caching statistics

tail /sys/block/bcache0/bcache/stats_total/*

At first, i checked the MD5 of the DVD which was as follows
b894f1e8ebf3065a488b6c01a742cf4e = debian-9.0.0-amd64-DVD-1.iso

The lowdown: Bottom line before you waste your time, the trick is disabling secure boot in the bios, if that does not work, read the article, it is faster if you see my trial and error than doing it yourself.

Following that, i DD the DVD onto a flash stick, Trying to install with network mirror, during the “Select and install software” stage, i get the following error on a beautiful red screen

Installation step failed
An installation step failed. You can try to run the failing item again from the menu, or skip it and choose something else. The failing step is: Select and install software

And a continue button that takes me to the list of steps/stages

In any case, the next time, i tried without a network mirror, and that stage completed successfully, but what do you know, a new error popped up, this time concerning the inability to install grub on the partition.

trying the above with a USB DVD gave the same exact errors, and thinking it might be a USB issue, i installed an internal SATA hard drive, but still same error.

So the only way to continue at this stage (Where both grub and LILO would not install is to continue without installing a boot loader.

So, considering the circumstance, i had 2 options, fix it manually, or try to figure the problem out, figuring out the problem would seem wiser, since We don’t want to have this problem every time we come across an installation scenario

Note: Disabling hardware turned out to be irrelevant, its all about the secure boot.
So i entered into UEFI (Delete at post), and disabled everything that may be causing the problem, for example, i disabled the azilla audio, I disabled the USB 3 Controller (Asmedia), and so forth, then i noticed that there is a secure boot option that had 2 options within, the first would be Windows and the other would be non windows.

So i switched that to non-windows (Even though the windows option worked fine on Jessie, and then went back to the USB flash stick, booted the system.

So this time, the installer is telling me that (SCREENSHOT)

So the answer to whether or not i should force the UEFI installer or not was YES, force it

As for whether or not to use a network mirror, for the sake of this article, i decided to experiment with installing with network mirror yet, then if that does not work, run the installer without a network mirror, the reason is that the secure boot option, if it were to fix anything, is most likely going to fix the failing to write boot manager, so to have this complete, we will need to start with the one more likely to fail

So running the installer with network mirror came back with the same error as anticipated, So i ran the installer again, but with no network mirror, and now the error is “Unable to install GRUB in dummy, executing “grub-install dummy failed”

So, now that all those options have failed, I am up for one last thing to try, I will boot from a linux stick, then issue the following command

wipefs -a /dev/sda

Now let us try again !

Now, a new error, the installer claims that a 40GB SSD is too small to be partitioned (Maybe because 64GB of ram will make a swap file larger than the disk) , so i rebooted again to the debian USB stick, but this time, to delete any existing partitions on the disk (Even though wipefs should have handled that)

So, because parted was not available, i used fdisk, followed by “o” followed by create a new partition on the whole disk

Next step, i took the 40GB SSD, reduced the ram on an HP 8300 USFF PC to 4GB, installed the disk and installed debian stretch (Using network mirror !), all went perfectly, now we would like to examine if it would boot the other computer, it did not,

Solution:
so i installed the OS on a very old core2 duo laptop, then it worked on the asusp9x79 after i enabled legacy mode rather than UEFI in the bios, I also made sure the boot disk was on the first sata port (SDA), and thats that.

I anticipate that this problem will also apply if you try to upgrade Jessie to Stretch, so BEFORE YOU UPGRADE, switch that UEFI secure boot option to non windows (Or disabled) depending on your motherboard.

I upgrading my I7 930 to an I7 980 worth it ?

The answer is YES, the I7 980 at the time of writing cost me $135 used on ebay.

there are 2 extra cores in the CPU, so it is like having an extra dual core computer to share the RAM and any other resources, even without wasting more electrical power.

It is also the newer Gulftown architecture (32nm Lithography), compared to the older Bloomfield (45nm Lithography)

So how much computing power did i gain for this database server

i7 930: 3GHz * 8 = 24 total at turbo speed
i7 980: 3.6GHz * 12 = 43.2 total at turbo speed

That is 180% of the original
almost double the computing power for 100 dollars !

To make this more interesting, many are reporting the with the new bios update, the GA-X58A-UD3R will allow for 48GB of ram, not 24, I will be trying that soon.

Let me start with the warning, DO NOT FLASH THE ORIGINAL TPLINK FIRMWARE BACK BEFORE FINDING OUT IF YOU NEED TO STRIP THE FIRST NUMBER OF BYTES OUT, YOU WILL BRICK YOUR ROUTER, IF YOU SEE

Failed to erase block

IF YOU DO, YOU NEED TO MAKE SURE YOU GET IT right after that, or you will start to get.

segmentation fault

on every and any command you run, at this stage, when you see this error after the failed to erase block error, your router is bricked, reboot it and it will not boot back up. if this happens, you might (depending on the model you have) have to connect your router to a com port and debrick it.

Now that you know, let’s see how to deal with this

If your firmware download file name has the word boot in it’s file name, you will need to strip out the first 0x20200 bytes (131584 bytes), if you fail to do this, meaning you fail to either identify if you need to do this, or fail to trim them, you will brick your router in the next step, to make matters both, TP-LINK is no longer including the word boot in firmwares with boot.

so to trim

dd if=orig.bin of=tplink.bin skip=257 bs=512

this command means read the file into another, skipping the first 257×512 = 131584 bytes (Which is the number you find above)

Now, you can flash it with the following command

mtd -r write /tmp/tplink.bin firmware

And if it says all is ok, you can now reboot your router into the new firmware

 

 

Converting DVD files to MP4

Video files on a DVD are usually in the video_ts directory and have the extension VOB.

On Windows, the easiest way to do this is by copying the files in the video directory on the DVD to a folder in your hard drive, then running the commands below, then deleting the source VOB files,

NOTE to ignore: surely, you can do it by adding absolute paths, but that would force me to explain much more here. with find the directory and copy the path then append it here then copy the names and append them to the path and all that good stuff, so why not just make it simple with copy then delete the source.

Assuming you have ffmpeg installed and in your path, you would execute the command like so, concatenating all the video files in the DVD

cd c:\directory_where_you_put_the_vob_files
ffmpeg -i "concat:VTS_01_1.VOB|VTS_01_2.VOB|VTS_01_3.VOB|VTS_01_4.VOB|VTS_01_5.VOB" outfile.mp4

Another note to ignore, On a linux box (The above works on both linux and windows), you can concatinate the VOB files then pipe them into ffmpeg like so

cat VTS_01_1.VOB VTS_01_2.VOB VTS_01_3.VOB VTS_01_4.VOB VTS_01_5.VOB | ffmpeg -i - outfile.mp4

So, Microsoft decided to change things a bit on how this can be done. To make a long story short, this is no longer inside “Advanced options” But rather in a different link names “Restart Options” on that same page

So open your settings app, then visit Updates and security, then windows update, then restart options (And active hours if needed)

Now, You can move your update hours to your weekend if that need to be the time for restarts

This problem is one i face often, because of how older versions functioned, the answers online no longer apply, online, you will find that

ntfsfix /dev/sdc2

should do the trick, in reality, it will not as you will see the following error

Mounting volume... OK
Processing of $MFT and $MFTMirr completed successfully.
Checking the alternate boot sector... OK
NTFS volume version is 3.1.
NTFS partition /dev/sdc1 was processed successfully.

The solution in reality is asking ntfs-3g’s mount to remove the hiberfile

WHAT YOU NEED – YOU WILL LOSE THE HIBERFILE

mount -t ntfs-3g -o remove_hiberfile /dev/sdc2 /hds/intelssd

Without the remove_hiberfile instruction, you will probably get an error message such as

Windows is hibernated, refused to mount.
Failed to mount '/dev/sdc2': Operation not permitted
The NTFS partition is in an unsafe state. Please resume and shutdown
Windows fully (no hibernation or fast restarting), or mount the volume
read-only with the 'ro' mount option.

Where you can actually mount it as read only if you do not want to write to it with the line

 mount -o ro /dev/sdc1 /hds/intelssd

apt-get install ufw

ufw allow ssh

but what do you know, i got an error

ERROR: initcaps
[Errno 2] modprobe: ERROR: could not insert 'ip6_tables': Unknown symbol in module, or unknown parameter (see dmesg)
ip6tables v1.4.21: can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

Well, that was not so nice, so i went into the file /etc/default/ufw and disabled IPv6, and now it works

So, before we enable it, i enabled SSH access otherwise i will be locked out

ufw allow ssh

now, enable it with

ufw enable

Now, enabled a few more ports for website or SIP for example

I have a website under development, so i will want to open port 80 and 443 for my ip only ! I would like to delete those rules as my IP changes

ufw allow from 213.139.53.70
# and When my ip changes
ufw status numbered
ufw delete 4

Then add the new IP like in the first line