UPDATE: This post has been added at easywebdns : recovering ext3 hard disks

So, here are the tools you need

1- Checking a linux partition for bad sectors / Bad Blocks without deleting the data

badblocks -o /path/somefiletowriteto.txt /dev/sdb1

this will do a check and write the bad block numbers to a text file.

Please note that 6TB and 8TB hard drives will not work out of the box with badblocks program, you will probably get a message informing you that the boundary must fit a 32bit integer (Value too large for defined data type), in that case, you can always increase the block size with the (-b 4096) switch, this will give you 8 times more block addressing space (Since the default is 512 bytes.

If you want it to test the disks hard but you are OK with deleting the data (Say after you dd_rescue), then try the -w switch

badblocks -w -o /path/somefiletowriteto.txt /dev/sdb1

Or, to see information while it scans

badblocks -wsv /dev/sdd > /root/250bad-blocks.txt

Notice that the -w has to come before the -o since the -o must be followed by a file name.

2- fsck /dev/sdb1 , the all famous, need i say more ?

3- tune2fs

tune2fs -O ^has_journal /dev/sdb1

remove the Journal (Converting ext3 to ext2 file system)

4- Mounting a disk: mount -t ext3 /dev/sdb1 /adirectory

5- debugfs <- use it in Read only more, this thing is like brain surgery, 1 mistake and your FS is playing chess in heaven with all the other dead disks

6- dd_rescue /dev/sdb1 /directory/mydisk.img

7- Running fsck on a disk image

fsck -y /path/mydisk.img

8- Mount the image

mount /pathto/backup.img /mountpoint/data

this is a nice link for you http://www.linuxjournal.com/article/193 that already has info about the stuff i mentioned here

Those are probably the most common, will add more when i remember them

This tutorial was first written for Lenny, then tested on squeeze, wheezy… up to bullseye

In August 2021, I have revised this for Debian Buster (10) and Debian Bullseye (11).

If you have installed Debian Lenny with file sharing (in the lenny installer), you will need to add users to that installation and specify folders that need to be shared, here i will show you how to add a user and share a folder with that user.

If you have not installed file sharing while installing Debian lenny, you need to install them, the easiest way to do that is

apt-get install samba

Or if you like

run the command

aptitude

using the arrow keys and the enter key, expand “TASKS” then place the bar over File Server, then the plus sign to select it (Shift +), right after hit “g” then “g” on your keyboard to install them.

Once done, add the following lines to smb.conf

[public]
   comment = Public Folder
   path = /public
   writable = yes
   guest ok = yes
   guest only = yes
   force create mode = 775
   force directory mode = 775
[private]
   comment = Private Folder
   path = /private
   writable = yes
   guest ok = no
   valid users = @smbshare
   force create mode = 770
   force directory mode = 770
   inherit permissions = yes

Now to adding users, from the following 3 commands, After the second and after the third command, you will be asked to chose a password for the user joe

useradd yazeed -m -G users
passwd yazeed
smbpasswd -a yazeed

Now with this out of the way, you can also edit the following fields in the file /etc/samba/smb.conf

The following need to be edited

netbios name : Should become the name of the computer on the network, in windows, this is the machine’s name
hosts allow : If your network is like mine where PCs take IP addresses of the form 192.168.2.x, then this one should be 127. 192.168.2. where 127 is for the local machine, the following part of the IP is the part of the IP that prefixes the IP of all network machines that should be able to access this file server (usually you will change the 2 with a 0 or 1)
interfaces : 127.0.0.1/8 192.168.2.0/24, like above, if your IP subnet is not 192.168.2.x, change it here (usualy you will change the 2 with a 0 or 1)
remote announce : if your subnet does not start with 192.168.2. then change it to your own (usually you will change the 2 with a 0 or 1)
remote browse sync : if your subnet does not start with 192.168.2. then change it to your own (usualy you will change the 2 with a 0 or 1)

Now scroll down, you will see a sample folder, edit that to the folder you want to share, and copy it over and over again for any other folders that you want to share, You are done, now you should be able to open those folders from the network.

This article is work in progress, i have started the ddrescue and waiting for it to finish before i go on with this post.

One thing you should notice that this is the GNU DDRESCUE, from the package gddrescue, not the old script dd_rescue that is a wrapper around the dd program.

It has been some time since i found out that dd_rescue has been replaced by the newer rewritten ddrescue from the gddrescue package, to be more specific, since i posted this back in march 2011.

So, now i have yet another disk that is busted, with 3 partitions, but not like that one, this one simply has so many bad sectors, it’s a 2TB western digital caviar black that is causing me trouble

So i couldn’t find a 2TB caviar black, so to be on the safe side, i got a 3TB western digital green, partitioned it and formatted it like i describe here.

So, now that i have a hard drive that needs rescuing, lets revise what we need to do

1- Install the new ddrescue tool gddrescue
apt-get install gddrescue
2- Run ddrescue, make sure to use a file to resume in case we get interrupted (sometimes saves days of rescue can be lost and need to be done again, if they have not been damaged with disk deterioration that is)

ddrescue /dev/sdb /hds/3tb/2tb.img /root/resumelog.log

If you lose power, or get interrupted, or need to restart your computer, ddrescue will resume ONLY if you use the same exact line above once again, it will then use the log file to append to the existing output file.

Now, we have an image, we can now mount that image and take a look, so we mount the image on a loop

You could have partitions on the original disk, in my case i had 3 EXT2 partitions, the data i need is on the third partition

So i enter parted (debian package), and did the following

Using /hds/3tb/2tb.img
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit
Unit?  [compact]? B
(parted) print
Model:  (file)
Disk /hds/3tb/2tb.img: 2000398934016B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start          End             Size            Type     File system  Flags
 1      32256B         400077619199B   400077586944B   primary  ext2
 2      400077619200B  700144058879B   300066439680B   primary  ext2
 3      700144058880B  2000396321279B  1300252262400B  primary  ext2

So now i know that the partition i want (third) starts at 700144058880, that’s all i need to know to mount this as a loop device
I am mounting loop device simply because i want to run fsck (disk check) on the partition before actually mounting that partition.

First, what is the next free / available loop device number ?

losetup -f

this should produce output such as

/dev/loop1

So now we know that we need to mount this on loop1 since it is the next available (not in use) loop device.

losetup /dev/loop1 /hds/3tb/2tb.img -o 700144058880

So, now loop 1 has my partition, you should omit the -y if you want to manually agree to every repair fsck wants to make

fsck.ext2 -y /dev/loop1

Great, now we should have a clean file system to mount
Even though we can mount the attached loop directly, i will demonstrate how to mount the loop and how to mount the image on a loop directly. i am doing this so that this tutorial would have the complete command referance of what tools and command parameters you might need.

First methode, the detach / release the loop device then mount it again in one go, this is done as follows
(-d means detach)

losetup -d /dev/loop1

Then, we attach with the foolowing command, notice how we used the starting ofset exactly like we did when attaching to a loop device.

Your mount command here

The other way is simply to mount our already attached loop device as follows

mount -t ext2 /dev/loop1 /hds/img

Now, we can mount this partition

mount -o loop,offset=700144058880 harddrive.img /hds/img

Or if you like, you can mount it read only

mount -o ro,loop,offset=700144058880 harddrive.img /hds/img

mount | grep /hds/3tb/2tb.img
/hds/3tb/2tb.img on /hds/img type ext2 (ro,loop=/dev/loop1,offset=700144058880)

My web server got hacked today, i know because my datacenter contacted me today telling me that there is a bruit force attack originating from my server to another server on a different network, so what is happening is that my server got hacked, then the hacker is using the server she hacked to hack other servers by sending FTP requests.

So, how come i got hacked when i am so obsessed with security, well, in reality, this is just an intermediate machine that i used to run a certain script that would move my mail server, and i did not (and did not see the need) to secure it.

What i usually do to secure my server is simply install fail2ban, in this case i did not out of lazyness but here is how i got hacked and how fail2ban would have protected me.

Before i show you the log files, this whole problem would not happen if i had a strong password combined with fail2ban

In the complaining partie’s log files

Tue Jul 24 22:28:27 2012: user: hauvouuc service: ftp target: yyy.yyy.yyy.yyy source: xxx.xxx.xxx.xxx
Tue Jul 24 22:28:27 2012: user: pkmcndgq service: ftp target: yyy.yyy.yyy.yyy source: xxx.xxx.xxx.xxx
Tue Jul 24 22:28:27 2012: user: malumdvc1 service: ftp target: yyy.yyy.yyy.yyy source: xxx.xxx.xxx.xxx

In my log files (auth.log):

Many lines like the following right below each other

Jul 24 18:03:08 run sshd[14229]: pam_unix(sshd:auth): check pass; user unknown
Jul 24 18:03:08 run sshd[14229]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=9.12-14-84.ripe.coltfrance.com 
Jul 24 18:03:10 run sshd[14229]: Failed password for invalid user ts3 from 84.14.12.9 port 41014 ssh2
Jul 24 18:03:11 run sshd[14231]: Invalid user ts3 from 84.14.12.9

Anod some lines like this

Jul 25 15:30:46 run sshd[10728]: pam_unix(sshd:auth): check pass; user unknown
Jul 25 15:30:46 run sshd[10728]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=217.119.29.135 
Jul 25 15:30:48 run sshd[10728]: Failed password for invalid user public from 217.119.29.135 port 34292 ssh2
Jul 25 15:30:48 run sshd[10730]: Address 217.119.29.135 maps to gamma2-7.cust.smartspb.net, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Jul 25 15:30:48 run sshd[10730]: Invalid user public from 217.119.29.135

Thousands of lines like this one

Jul 24 14:12:38 run sshd[2025]: error: connect_to 213.186.33.207 port 80: failed.
Jul 24 14:12:39 run sshd[2025]: error: connect_to 192.168.10.24 port 2110: failed.
Jul 24 14:12:39 run sshd[2025]: error: connect_to 195.130.65.50 port 80: failed.

OR

Jul 24 06:41:19 run sshd[9824]: error: connect_to 213.186.33.207 port 80: failed.
Jul 24 06:41:19 run sshd[13434]: Failed password for invalid user test from 202.28.123.191 port 37830 ssh2
Jul 24 06:41:20 run sshd[9824]: error: connect_to 213.186.33.207 port 80: failed.

And more like this

Jul 24 08:19:18 run sshd[20882]: pam_unix(sshd:auth): check pass; user unknown
Jul 24 08:19:18 run sshd[20882]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=puck748.server4you.de 
Jul 24 08:19:21 run sshd[20882]: Failed password for invalid user kk from 85.25.235.73 port 49213 ssh2
Jul 24 08:19:21 run sshd[20884]: Invalid user css from 85.25.235.73

It is simple, here is what you need to know

You can format it EXT4, but ext2 and ext3 are also OK ! ext2 and ext3 allow up to 16TB disks, and file sizes of up to 2TB, ext4 allows much more.

Any linux kernel newer than 2.6.31 should work just fine with “Advanced format” drives using the exact same steps in this article.

MBR only supports 2TB drives, you need GPT, so let us get started

1- apt-get update
2- apt get install parted
3- parted /dev/sdc
4- mklabel gpt
5- Answer yes to: Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
6- mkpart primary ext4 0% 100% (to make a partition as big as the disk (will occupy starting from first megabyte (for alignment) to the end of disk))
7- quit

FYI, if you want multiple partitions, here are the 2 lines that should replace step 6
6- mkpart primary ext4 0% 40%
6- mkpart primary ext4 40% 100%

and remember to format both (sdc1 and sdc2) when you are done with parted

Now to formatting the drive

mkfs.ext4 /dev/sdc1

Before mounting it, i like ext4, but i don’t want a journaling OS on this drive that is not the system drive, so i will need do a few things to the drive first

Lazy writeback

tune2fs -o journal_data_writeback /dev/sdc1

No Journaling

tune2fs -O ^has_journal /dev/sdc1

Now to check what we have

dumpe2fs /dev/sdc1 |grep 'Filesystem features'

Or maybe if you want the whole thing on the screen

dumpe2fs /dev/sdc1 |more

if has_journal option exist when executing the first – you have journal on the file system

And there we are, Now we need to mount it at boot time by adding it to fstab, to do that, we will need the disk’s unique ID !

8- Now executing the following command will give you the unique ID of this new partition for use with fstab (The disk list we will edit below in step 10)
blkid /dev/sdc1
9- create the directory where you want to mount your hard disk, for example
mkdir /hds
mkdir /hds/3tb
10- Now, we add the following line to fstab, notice that noatime increases performance, but some applications might need or rely on it. postfix does not and i have verified that.

UUID=b7a491b1-a690-468f-882f-fbb4ac0a3b53       /hds/3tb            ext4     defaults,noatime                0       1

defaults and noatime are but only a couple of options, here are more options that you can add
nofail = If the disk is not present, continue booting
nobootwait = Limit the amount of time you plan to wait
noauto = Don’t mount it until I issue a “mount /dev/sdb1”, or mount “/hds/thisdisk” command

11- Now execute
mount -a

You are done,. if you execute
df -h
You should see your 2+TB hard drive in there !

To make sure the drive is aligned correctly, i like to write a file on it and see how fast that goes… so let us use a 2GB file

dd if=/dev/zero of=/hds/WD2000_3/deleteme.img bs=1M count=2000

Outcome came out (for a western digital black 2TB)
First run: 2097152000 bytes (2.1 GB) copied, 5.94739 s, 353 MB/s
Consecutive runs: 2097152000 bytes (2.1 GB) copied, 11.1405 s, 188 MB/s
Outcome came out for a western digital green 3TB
First run: 2097152000 bytes (2.1 GB) copied, 8.32337 s, 252 MB/s
Consecutive runs: 2097152000 bytes (2.1 GB) copied, 14.376 s, 146 MB/s

the consecutive runs give close results, what i printed here is the average

My old tablet (HP tc4200) had problems with the wireless adpater , A broadcom BCM4309

To find out what the Broadcom wireless adapter model is i issued

lspci -vvnn | grep 14e4

For yours, you may need to check with this website here as you may or may not need the sta or the b43legacy driver, in general here are the popular models

STA – BCM4311, BCM4312, BCM4313, BCM4321, BCM4322, BCM43224, BCM43225, **BCM43227, **BCM43228

b43 – BCM4306/3, BCM4311, BCM4312, BCM4318, BCM4320

b43legacy – BCM4301, BCM4306, BCM4306/2

http://www.linuxwireless.org/en/users/Drivers/b43#Supported_devices

The, now that i know… i edited /etc/apt/sources.list and added the contrib and non-free repositories

then

apt-get update

apt-get install firmware-b43-installer b43-fwcutter

And what do you know, just reboot and it works

Inspecting Postfix’s email queue.

This post explains how to view messages in the postfix queue, another post on this blog explains how to delete or selectively delete from the postfix queue

1- Postfix maintains two queues, the pending mails queue, and the deferred mail queue,
the differed mail queue has the mail that has soft-fail and should be retried (Temporary failure),
Postfix retries the deferred queue on set intervals (configurable, and by default 5 minutes)

In any case, the following commands should be useful

1- Display a list of queued mail, deferred and pending

mailq

or

postqueue -p

To save the output to a text file you can run

mailq > myfile.txt

or

postqueue -p > myfile.txt

the above commands display all queued messages (Not the message itself but the sender and recipients and ID), The ID is particularly useful if you want to inspect the message itself.

2- View message (contents, header and body) in Postfix queue

Assuming the message has the ID XXXXXXX (you can see the ID form the QUEUE)

postcat -vq XXXXXXXXXX

Or to save it in a file

postcat -vq XXXXXXXXXX > themessage.txt

3- Tell Postfix to process the Queue now

postqueue -f

OR

postfix flush

4- Delete queued mail

Delete all queued mail

postsuper -d ALL

Delete differed mail queue messages

(The ones the system intends to retry later)

postsuper -d ALL deferred

Delete from queue selectively

To delete from the queue all emails that have a certain address in them, we can use this program (perl script)…

NOTE: This perl script seems to be free, and is all over the internet, i could not find out where it originates or who wrote it.

1- Download this file, unzip, and upload the file to your server, then from your bash command line, Change Directory to wherever you uploaded this file, for example cd /root (Just an example, You can upload it wherever you wish)

NOTE: A second script here works differently, i have not yet tested it, download it here

Now, from within that directory, execute…

./postfix-queue-delete.pl anyaddress@example.com

Any mail that has this email address in it’s IN or OUT list will be deleted

The script uses the postqueue -p then looks for your string, once found, it deletes the email by ID, this means that this script can delete messages using any text that appears when you run mailq (or postqueue -p), so if you run it with the parameter joe all mail with addresses such as joefriend@example.com and

Other moethods exist, like executing directly

mailq | tail +2 | grep -v '^ *(' | awk  'BEGIN { RS = "" } { if ($8 == "email@address.com" && $9 == "") print $1 } ' | tr -d '*!' | postsuper -d -

——————————–

Sample Messages in a differed mail queue

——————————–

SOME282672ID 63974 Mon Nov 29 05:12:30 someaddresss@yahoo.com
(temporary failure. Command output: maildrop: maildir over quota.)
localuser@exmple.com

———————————-

SOME282672ID 9440 Wed Jun 30 05:30:11 MAILER-DAEMON
(SomeHostName [xxx.xxx.xxx.xxx] said: 452  Mailbox size limit exceeded (in reply to RCPT TO command))
username@example.org

———————————-

SOME282672ID 4171 Thu Nov 25 13:22:03 MAILER-DAEMON
(host inbound.somedomain.net [yyy.yyy.yyy.yyy] refused to talk to me: 550 Rejected: 188.xx.179.46, listed at http://csi.cloudmark.com/reset-request for remediation.)
someuser@example.com

———————————

SOME282672ID 37031 Thu Nov 25 08:53:36 someuser@example.net
(Host or domain name not found. Name service error for name=example.com type=MX: Host not found, try again)
someuser@example.com 

Section 1: My hard drive has bad sectors / Blocks / area

Do i need to change it ?
Not necessarily, but If it is in warranty, and they allow you to replace it, a new one is not a bad idea, otherwise read on

it all depends on whether the bad sectors are expanding or not, if they are not, they are probably caused by shock to the hard drive, usually, it is enough to mark them as bad using “chkdsk /r” on windows and leave the drive working.

To find out if your bad sectors are Spreading or not spreading, do a “chkdsk /r” four times, make sure the same number appears in the second and third and fourth time (Forget the first time), then, if the second is different but the third and fourth are the same, then do the test 2 more times, and make sure you get the same number of bad sectors for trials 3, 4, 5, 6, if so, your bad sectors are not spreading.

You did not mention backup in the answer before, do we need to backup ?
People would typically ask you to backup just in case, i say you should always have backup of your most important files, non spreading sectors of the hard drive, in my humble experience do not contribute negatively to reliability, so my answer is, backup should be done regardless

How do i know how many bad sectors are marked on an NTFS hard drive ?
There is a tool called nfi.exe that comes with a bundle Microsoft makes available here http://support.microsoft.com/kb/253066/en-us this tool is part of (OEM Support Tools), it can tell you everything about a disk formatted in NTFS

On a linux box, there is a simple way to search for some text inside a file using regular expressions in the command line

So, the short answer, in Linux, you can simply execute this line

grep -R -B3 -A4 "MYTEXT" /var/ > /root/findingres.txt

this will show you 3 lines before the text and 4 lines after the string and the file name, it will search in the /var/ folder, and will store the searching results to /root/findingres.txt

The above is a simple example, you can use regular expressions to find more complicated stuff

Also, if you want to search the entier hard drive for a string, and you want to search only in one type of file that you want to specify using a file extension you can use the following

grep -R --include=*.txt "MYTEXT" /etc/

Now, if you want to search for a file by file name,

find / -type f -name "myfile.txt"

would look for an exact file name, if you are looking to find a file using wildcards, for example, any file that ends in a certain extension you would

find / -type f -iname "*.psd"

If you want to include directories in your search, remove the -type flag

When i get delivery problems like the message i got today from a microsoft exchange server

I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can delete your own text from the attached returned message.

host mail1.example.com[xxx.xxx.xxx.xxx] said: 550 5.7.1 Message rejected due to content restrictions (in reply to end of DATA command)

I usually run the test on the following locations

1- the road runner website allows you to check cloudmark… run a scan with
http://postmaster.rr.com/amIBlockedByRR

tells me about
Return Path Reputation Network Blacklist :
Spamhaus (ZEN) :
Road Runner Blocklist :
FBL Enrollment :
Road Runner Internal Name-Based Block :
Cloudmark Sender Intelligence :
Return Path Sender Score Reputation Score :
historical Road Runner Internal IP Address-Based block records

Now, MXtoolbox

http://www.mxtoolbox.com/blacklists.aspx

for

AHBL
BACKSCATTERER
BARRACUDA
BURNT-TECH
CASA-CBL
CASA-CBLPLUS
CBL
IMP-SPAM
INPS_DE
ivmSIP
ivmSIP24
LASHBACK
MAILSPIKE-BL
MAILSPIKE-Z
NIXSPAM
NOMOREFUNN
PSBL
RATS-Dyna
RATS-NoPtr
RATS-Spam
REDHAWK
SEM-BACKSCATTER
SEM-BLACK
SORBS-DUHL
SORBS-SPAM
SORBS-WEB
SPAMCANNIBAL
SPAMCOP
SWINOG
TRUNCATE
UCEPROTECTL1
UCEPROTECTL2
UCEPROTECTL3
WPBL

If i can not see anything in the blacklists / blocklists above, i will generally contact the server administrator