Freeing up space on a Raspberry Pi

Partway through a big compile, the process died with “No space left on device”. Aaarrgh! What to do?

The Pi had an 8GB card and check with df -h showed this:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.2G  6.8G   34M 100% /

So that’s me stuffed. Or is it?

Option 1

Use a bigger memory card; 16GB, 32GB … Yeah, great. I’m 30% through a compile based on a complicated setup using lots of downloads and additions. Either I’d have to start again from square one or image the existing card and copy it onto the new one. Helluva hassle. Besides, first I’d have go out and buy a new card. Not an option.

Option 2

Install Raspbian Lite. It’s Raspbian without all the extras.

This is what I should have done at the outset. Hindsight’s a great thing, but I’d still be faced with starting from square one. What else?

Option 3

Expand the filesystem. Run the command sudo raspi-config. You’ll get this:

Select Advanced Options:

Choose Expand Filesystem. After a reboot you’ll hopefully you’ll have more space. But not in my case. So that left …

 

Option 4

Delete unneeded programs.

The beauty of Linux is it’s easy to reinstall stuff later, so I tried deleting things I didn’t need and charting my progress.Here are the results.

Command

Result

sudo apt-get purge wolfram-engine
677M recovered
sudo apt-get purge libreoffice*
250M recovered
sudo apt-get purge minecraft-pi*
5M recovered
sudo apt-get purge sonic-pi*
134M recovered

I finishing up with:

sudo apt-get clean
sudo apt-get autoremove

which freed up another 100M+ and re-ran df -h:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.2G  5.6G  1.3G  82% /

That’s 1.3GB freed up. Almost 20% of my 8GB card. Result!

Now back to that compile …

 

Share this ...
Share on FacebookTweet about this on TwitterPin on PinterestShare on Google+Share on TumblrShare on LinkedInDigg thisShare on RedditShare on StumbleUponEmail this to someonePrint this page

SMARTen up!

How do you know if your hard disk is about to die?

A couple of months ago, a writer friend called me about a problem with her computer. The sort of problem that drives you nuts; an intermittent one.

Sometimes the machine would be slow to boot. V-e-r-y slow. Sometimes it would freeze while she was working, then resume. Other times it would behave perfectly normally. But the bad times were getting more frequent and she’d reached the stage where she no longer trusted the machine.

What was it? What could she do?

She’d called in a geek – the wheeled variety (Kiwis will know who I mean) – who performed some tests, did some checks, presented a bill and declared the machine was fine. Only it wasn’t.

Someone recommended “their guy” who charged in, did some stuff, uttered some techno-babble and charged out again. (As well as charging, in the other sense.)

He went away, but the problems didn’t.

So she called me.

Oh great. Two “experts” had failed. What chance did I have?

But in talking to her and her husband about the problems – something neither of my precedessors had done – I began to see a pattern in the randomness, booted the machine, hit F2, and within two minutes had the solution.

The machine was an HP. Like many “brand” computers, HPs contain a set of hardware diagnostic tools available from the boot menu. All I did was kick them off.

A typical short hard drive check takes around two minutes. And, as I’d guessed, two minutes later the diagnositcs reported the hard drive was failing.

HPfail

The machine was a little over a year old, still under warranty, and the faulty drive was replaced within a week.

 

Behind the scenes

Hard drives die in one of two ways. Around 40% go suddenly and without warning. The remainder suffer lingering deaths from mechanical wear and drive surface degradation, sometimes giving out warnings – like my friend’s – in the form of sluggish response and erratic performance. And, if you know where to look, you can see and even log their decline.

Behind the scenes, that HP diagnostics program ran a SMART analysis of the hard disk. SMART stands for Self-Monitoring, Analysis, and Reporting Technology, and is built in to all hard disk and solid-state drives. It tries to aniticpate failure by running a series of electrical and mechanical tests and recording the results. Some tests are more useful than others, but by looking at past failures and their frequency, it can provide you with a vital clue that a drive’s on its way out.

Some motherboards display a SMART drive status when they boot. Some don’t. Plus, there are many different types of drive and types of connection – USB, Firewire, ATA, SATA, SCSI, SSA, RAID, etc. That “low-levelness” is something operating systems like Windows struggle with. What’s more, SMART is only a “sort of” standard. Most drive manufacturers follow the basic implementation, but only some aspects are cross-compatible.

 

Linux SMARTs

As usual, Linux users have the edge here. Installing SMART is simply a matter of installing Smartmontools:

sudo apt-get install smartmontools

This provides two utilities — smartctl and smartd – a monitoring and control program and a disk monitoring daemon.

 

To get information about the disk and see whether it supports SMART:

sudo smartctl -i /dev/sda

where sda is the drive concerned. (Use lsblk to see what drives are attached to the machine.)

This will give you a summary of your drive. Look for the lines:

SMART support is: Available - device has SMART capability.
SMART support is: Enabled

 

If SMART’s not enabled, enable it with:

sudo smartctl -s on /dev/sda

 

To get a quick health status report:

sudo smartctl -H /dev/sda

which should show something like this:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

If this shows FAILING, back up the data immediately!

 

To get a full drive report:

sudo smartctl -a /dev/sda

There are two options for testing a drive – short and longt. A short test typically takes around two minutes. Long tests take considerably longer – two to six hours is not uncommon – but both tests will tick away in the background and still allow you to use your machine.

 

To see roughly how long each test will take, run the full report

sudo smartctl -a /dev/sda

and scroll down to a section under the line

=== START OF READ SMART DATA SECTION ===
 where you'll find something like this:Short self-test routine
 Short self-test routine
 recommended polling time: ( 1) minutes.
 Extended self-test routine
 recommended polling time: ( 333) minutes.

 

To run either test, use the -t option:

sudo smartctl -t short /dev/sda
sudo smartctl -t long /dev/sda

Running a test will give you a completion time:

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
...
Testing has begun.
Please wait 2 minutes for test to complete.
Test will complete after Tue Jun 21 21:33:38 2016

 

To abort the test use:

sudo smartctl -X

You can see how the time is going with the date command:

 

To see the results of the test:

sudo smartctl -l selftest /dev/sda

or run the full report again.

 

If you prefer a GUI front end for SMART, install GsmartControl:

sudo apt-get install gsmartcontrol

 

Next time, I’ll show you how to automate drive testing using smartd.

 

 

Share this ...
Share on FacebookTweet about this on TwitterPin on PinterestShare on Google+Share on TumblrShare on LinkedInDigg thisShare on RedditShare on StumbleUponEmail this to someonePrint this page

Death of a hard drive

The first sign you’re likely to get of a failing hard drive is a SMART warning when you boot your computer. Something like this …

fig1

Which is exactly what I got recently on a relatively new 1TB drive.

SMART stands for Self-Monitoring, Analysis, and Reporting Technology and there’s really not much you can do except be grateful for the warning, get a replacement drive and start moving data off the failing one as soon as possible.

SMART records all sorts of information about your HDD, some of which is used to predict future drive fails. The most common is the Reallocated Sectors Count fault — which is what my dying drive has.

fig2

What that means is that the hard drive has found a read, write or verification error on a part of the disk, marked the dodgy sector as “reallocated” and moved the data to a working sector. As you can see above, this has happened 3,995 times so far. As a predictor of impending failure, it’s a good one.

The BIOS warning doesn’t give you much information about precisely what’s going wrong with your disk. To find that, you’ll need to add some software appropriate to your operating system.

The best package I’ve come across for Linux is simply called Disks. It’s included in Debian, Ubuntu, Fedora, Red Hat Enterprise Linux and CentOS, and typically filed under the Utilities menu. If it’s not already installed, just grab it with your usual package manager.

fig3

Of course, SMART can’t anticipate all hard drive failures. In the case of a sudden failure, your only realistic recourse is to your backups. If you don’t have any, and the data’s critical, you can get the drive rebuilt, but it’s going to cost in time and $$$s. A quick local Google found one firm with a turnaround of 5-10 working days at a cost of about $600. Considering the price of a new 1TB drive is only about $100, I’m happy to heed the warning!

 

Share this ...
Share on FacebookTweet about this on TwitterPin on PinterestShare on Google+Share on TumblrShare on LinkedInDigg thisShare on RedditShare on StumbleUponEmail this to someonePrint this page

Free file recovery

Deleted some vital files or trashed your hard drive? Have I got a couple of fantastic free tools for you!

photorecPhotoRec recovers lost files from hard disks, CDs and DVDs, USB stick and camera memory cards. Don’t be fooled by its name — it recovers way more than just photos. 440 different file types to be exact.

Photorec runs under the following operating systems:

  • DOS/Windows 9x
  • Windows NT 4/2000/XP/2003/Vista/2008/7
  • Linux
  • FreeBSD, NetBSD, OpenBSD
  • Sun Solaris
  • Mac OS X

and can recover data from the following file systems:

  • FAT
  • NTFS
  • exFAT
  • ext2/ext3/ext4 filesystem
  • HFS+

Testdisklogo_clear_100

PhotoRec’s companion program, TestDisk, is designed to recover lost partitions and make non-booting disks bootable again. Whether caused by faulty software, viruses or human error, TestDisk makes partition table recovery easy.

In addition, TestDisk can:

  • Fix partition table, recover deleted partition
  • Recover FAT32 boot sector from its backup
  • Rebuild FAT12/FAT16/FAT32 boot sector
  • Fix FAT tables
  • Rebuild NTFS boot sector
  • Recover NTFS boot sector from its backup
  • Fix MFT using MFT mirror
  • Locate ext2/ext3/ext4 Backup SuperBlock
  • Undelete files from FAT, exFAT, NTFS and ext2 filesystem
  • Copy files from deleted FAT, exFAT, NTFS and ext2/ext3/ext4 partitions

TestDisk runs under the following operating systems:

  • DOS (either real or in a Windows 9x DOS-box),
  • Windows (NT4, 2000, XP, 2003, Vista, 2008, Windows 7 (x86 & x64),
  • Linux,
  • FreeBSD, NetBSD, OpenBSD,
  • SunOS and
  • MacOS X

 

Both programs have extensive documentation and walk-throughs, and both are open source software and licensed under the GNU General Public License, meaning they’re free to use and free to copy.

 

 

Share this ...
Share on FacebookTweet about this on TwitterPin on PinterestShare on Google+Share on TumblrShare on LinkedInDigg thisShare on RedditShare on StumbleUponEmail this to someonePrint this page

Secure data deletion

In my last blog I detailed why it wasn’t necessary to take a hammer to your hard drive to protect your data. So what can you do if you’re sending an old PC off for recycling and don’t want Joe Random looking over your supposedly deleted files?

.

Whole disk deletion

The simplest whole-disk solution is DBan — Darik’s Boot and Nuke — “a self-contained boot disk that automatically deletes the contents of any hard disk that it can detect.”

DBANDBan is an open source program that securely erases hard disks by overwriting them with with random garbage. It can be run from a CD, DVD or USB stick and can even be configured to automatically wipe every disk that it finds on a system or network. Download it here.

.

File-by-file deletion

Linux users have a built-in command-line tool called shred. It overwrites the specified file(s) with random junk — 25 times by default.

Here’s how to use it:

shred secrets.txt
Will shred the contents of secrets.txt but it leaves the file in place! While this is a good way of checking what shred does, you probably really want to …

shred -u secrets.txt
… remove the file after you’ve shredded it. For extra security you can …

shred -u -n 100 secrets.txt
… tell it to overwrite the file 100 times instead of the default 25, and even …

shred -u -n 100 -z secrets.txt
… overwrite the file with zeros on its last pass. This disguises the fact that there was ever any file there at all!

Note that shred does however come with a couple of caveats. The man shred command will give you the full details, but essentially it assumes that the file system overwrites data in place. That’s the usual way of doing things, but Linux has a wide variety of possible file systems and they don’t all work the same way! Still, shred works just fine with the default ext3 file system used on most distributions.

.

The full kit

The Secure-Delete toolkit provides a suite of tools to:

  • securely wipe files
  • wipe free disk space
  • wipe swap space and computer memory

All work in a similar fashion; writing and rewriting random data, then a set of special cryptographic values, followed by more random data. In addition, the file tool also randomly renames and truncates the file.

Secure-delete may not be installed by default, so use your package manager to add it. Debian / Ubuntu / Mint users can just do:
sudo apt-get install secure-delete

Here’s a quick run-down of the SD tools and how to use them:

.

srm (secure remove) :
wipes files or directories currently on your hard disk. The algorithm used is based on this paper by local boy Peter Gutmann.

To wipe a file:
srm filename.txt

To wipe a directory:
srm -r folder_name

.

sfill (secure free space wiper) :
wipes the free space areas on your disk. If you haven’t used secure deletion tools before, chances are there’s still a lot of recoverable data in regions where files have been unsecurely deleted. sfill will clean this up!

Clean up your home folder:
sfill /home/yourname

.

smem (secure memory wipe) :
deletes data stored in your computer’s memory. Why? Because data held in SDRAM doesn’t “fade away” and can be easily recovered!

To wipe memory:
smem

Note: a full smem run can take some time! Try
smem -l
or
smem -ll for a quicker (though less secure) run.

.

sswap (secure swap space wipe) :
does a secure wipe of your swap partition.

Find your swap partition:
cat /proc/swaps
Disable swap:
sudo swapoff /dev/swap_partition
Securely wipe it:
sudo sswap /dev/swap_partition
Re-enable swap:
sudo swapon /dev/swap_partition

 

Share this ...
Share on FacebookTweet about this on TwitterPin on PinterestShare on Google+Share on TumblrShare on LinkedInDigg thisShare on RedditShare on StumbleUponEmail this to someonePrint this page