<Long & OT> Data protection and recovery techniques

Good Afternoon,

Recently there was discussion on backup strategies.  Here are my current
methods.

I use three data protection strategies. The first is possible because of
the way I design my computers. I use a small solid state drive for the
OS and for all of the applications I use. Once everything is installed
and all the settings massaged I make an ISO image of the drive and save
it on my server. Then I clone a drive from that ISO and put it into a
drawer. If I ever have a major OS crash I can pull that drive out of
storage, install it into the box, and be running again in a few minutes.

The second method I use for data protection requires a versioning
application. I use Subversion. When a set of folders with their
attendant files is ready I create a Subversion repository to hold them.
This repository is a database of files. Files can be of any type I have
tried: C code, text files, CAD files, SPICE files, circuit board
layouts, bills of material, etc. While I am working I commit my changes
many times a day. If I need to recover a file from a week ago I can type
svn update XXX and roll back my file system to when the XXX snapshot was
current. I can also check out the repository to another computer and
work on it. The first thing I do, when I start working on any of my
computers, is type svn update. This pulls any changes from the
repository and brings my files up to date. Then I write code, prose, or
CAD committing the changes as I work. By typing svn update I bring each
of my laptops and work computers up to the same revision number.

The third method of data protection is more common. I make copies of my
files. I have found backup schemes which use compression or encryption
get in my way. When I want to recover something it is not normally the
entire system; I only need a small subset of the whole. So my backups
consist of exact copies of files in the folder structure I built. If I
need to, I can move entire chunks of the file hierarchy to recover from
a major crash. For the most sensitive data I make a copy on the same
computer but on a different drive and again on a separate machine.

The tool I use for this third method of data protection is a command
built into Linux but available to Windows and Mac users too. Rsync lets
me ‘push’ data from where it is to a remote location or ‘pull’ from a
remote location to a local file. Rsync works across a network or across
the Internet so you can have your files anywhere you have access. Linux
provides me with another valuable command called cron. It is used to
perform a task periodically, hours, days, weeks, months, etc. Once I set
up a cron job I don’t need to worry about backing up my files. It all
happens behind the scenes at whatever time of day I choose.

The first time you run rsync on a set of folders it copies the entire
set. The next time it runs only the modified files are copied. Normal
backup commands make compressed copies of all of the files in the chosen
set of folders. Over time they build up a large set of files which are
not readily accessible by humans taking up an inordinate amount of
space. Rsync creates a mirror image of what your file system looks like
right now. If you need one data file from the middle of that pile of
files you can find it using your normal file system search mechanism.
You do not need to decompress or decrypt anything. Everything is in
front of you, easy to read.

I know this may seem Linux specific but it is not. That is just where I
learned how to do this. Subversion, rsync, and a cron equivalent are
available for both Linux and Windows systems and I suspect there are Mac
versions too. The modern Macintosh OS is a flavor of Linux if you
scratch beneath the surface. My first method of data protection is
specific to how I build computers. Keep the OS and applications on one
small drive while keeping all of the user created and collected data on
a second, larger drive (or in my case across the network on the file
server). The OS and application set do not change very rapidly so you
can make a copy for later. The data changes on a regular basis so you
need another method to preserve it.

Hopefully, these suggestions will help you keep your most precious data
intact.

73 & GL with any hardware crashes,

    Kevin.  KD5ONS

-

______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html
Message delivered to [hidden email]

[Phil Kane-2]

On 6/13/2020 12:11 PM, kevinr wrote:

> The tool I use for this third method of data protection is a command
> built into Linux but available to Windows and Mac users too.

My regular backup program is NovaBACKUP PC from NovaStor.  It is a
commercial program that I have been using for many years on all my
computers, making a daily incremental backup  and a weekly full backup
using external drives for storage of the compressed files.  It has a GUI
that permits restoration (both overwrite and write elsewhere) of files
and folders.  It has saved my backsplash several times in the recent
past.    There is a linux version called NovaBACKUP Data Center that is
distro-independent.  (No, I don't own stock in NovaStor.)

73 de K2ASP - Phil Kane
Elecraft K2/100   s/n 5402

From a Clearing in the Silicon Forest
Beaverton (Washington County) Oregon
______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html
Message delivered to [hidden email]

[Jim Miller]

Backblaze here

On Jun 13, 2020, at 6:11 PM, Phil Kane <[hidden email]> wrote:

On 6/13/2020 12:11 PM, kevinr wrote:

> The tool I use for this third method of data protection is a command
> built into Linux but available to Windows and Mac users too.

My regular backup program is NovaBACKUP PC from NovaStor.  It is a
commercial program that I have been using for many years on all my
computers, making a daily incremental backup  and a weekly full backup
using external drives for storage of the compressed files.  It has a GUI
that permits restoration (both overwrite and write elsewhere) of files
and folders.  It has saved my backsplash several times in the recent
past.    There is a linux version called NovaBACKUP Data Center that is
distro-independent.  (No, I don't own stock in NovaStor.)

73 de K2ASP - Phil Kane
Elecraft K2/100   s/n 5402

From a Clearing in the Silicon Forest
Beaverton (Washington County) Oregon
______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html
Message delivered to [hidden email]
______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html
Message delivered to [hidden email]

[k6dgw]

And, do any of you ever check the backups to make sure they really have
anything there?  In the olden days of reel-to-reel tapes, the number of
backup tapes in the library that were actually blank often exceeded the
number of valid tapes in the library.  Lots of scrambling when you
needed that tape.

73,
Fred ["Skip"] K6DGW
Sparks NV DM09dn
Washoe County

On 6/13/2020 3:31 PM, Jim Miller wrote:

> Backblaze here
>
> On Jun 13, 2020, at 6:11 PM, Phil Kane <[hidden email]> wrote:
>
> On 6/13/2020 12:11 PM, kevinr wrote:
>
>> The tool I use for this third method of data protection is a command
>> built into Linux but available to Windows and Mac users too.
> My regular backup program is NovaBACKUP PC from NovaStor.  It is a
> commercial program that I have been using for many years on all my
> computers, making a daily incremental backup  and a weekly full backup
> using external drives for storage of the compressed files.  It has a GUI
> that permits restoration (both overwrite and write elsewhere) of files
> and folders.  It has saved my backsplash several times in the recent
> past.    There is a linux version called NovaBACKUP Data Center that is
> distro-independent.  (No, I don't own stock in NovaStor.)
>
> 73 de K2ASP - Phil Kane
> Elecraft K2/100   s/n 5402
>
>  From a Clearing in the Silicon Forest
> Beaverton (Washington County) Oregon
> ______________________________________________________________
> Elecraft mailing list
> Home: http://mailman.qth.net/mailman/listinfo/elecraft
> Help: http://mailman.qth.net/mmfaq.htm
> Post: mailto:[hidden email]
>
> This list hosted by: http://www.qsl.net
> Please help support this email list: http://www.qsl.net/donate.html
> Message delivered to [hidden email]
> ______________________________________________________________
> Elecraft mailing list
> Home: http://mailman.qth.net/mailman/listinfo/elecraft
> Help: http://mailman.qth.net/mmfaq.htm
> Post: mailto:[hidden email]
>
> This list hosted by: http://www.qsl.net
> Please help support this email list: http://www.qsl.net/donate.html
> Message delivered to [hidden email]

______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html
Message delivered to [hidden email]

[M0XDF]

Backblaze and Time Machine.
73 de David, M0XDF
--
It is almost impossible to carry the torch of truth through a crowd without
singeing somebody's beard. -George Christopher Lichtenberg, scientist and
philosopher (1742-1799)

> On 13 Jun 2020, at 23:40, Fred Jensen <[hidden email]> wrote:
>
> And, do any of you ever check the backups to make sure they really have anything there?  In the olden days of reel-to-reel tapes, the number of backup tapes in the library that were actually blank often exceeded the number of valid tapes in the library.  Lots of scrambling when you needed that tape.
>
> 73,
> Fred ["Skip"] K6DGW
> Sparks NV DM09dn
> Washoe County
>
> On 6/13/2020 3:31 PM, Jim Miller wrote:
>> Backblaze here
>>
>> On Jun 13, 2020, at 6:11 PM, Phil Kane <[hidden email]> wrote:
>>
>> On 6/13/2020 12:11 PM, kevinr wrote:
>>
>>> The tool I use for this third method of data protection is a command
>>> built into Linux but available to Windows and Mac users too.
>> My regular backup program is NovaBACKUP PC from NovaStor.  It is a
>> commercial program that I have been using for many years on all my
>> computers, making a daily incremental backup  and a weekly full backup
>> using external drives for storage of the compressed files.  It has a GUI
>> that permits restoration (both overwrite and write elsewhere) of files
>> and folders.  It has saved my backsplash several times in the recent
>> past.    There is a linux version called NovaBACKUP Data Center that is
>> distro-independent.  (No, I don't own stock in NovaStor.)
>>
>> 73 de K2ASP - Phil Kane
>> Elecraft K2/100   s/n 5402
>>
>> From a Clearing in the Silicon Forest
>> Beaverton (Washington County) Oregon
>> ______________________________________________________________
>> Elecraft mailing list
>> Home: http://mailman.qth.net/mailman/listinfo/elecraft
>> Help: http://mailman.qth.net/mmfaq.htm
>> Post: mailto:[hidden email]
>>
>> This list hosted by: http://www.qsl.net
>> Please help support this email list: http://www.qsl.net/donate.html
>> Message delivered to [hidden email]
>> ______________________________________________________________
>> Elecraft mailing list
>> Home: http://mailman.qth.net/mailman/listinfo/elecraft
>> Help: http://mailman.qth.net/mmfaq.htm
>> Post: mailto:[hidden email]
>>
>> This list hosted by: http://www.qsl.net
>> Please help support this email list: http://www.qsl.net/donate.html
>> Message delivered to [hidden email] <mailto:[hidden email]>
>
> ______________________________________________________________
> Elecraft mailing list
> Home: http://mailman.qth.net/mailman/listinfo/elecraft <http://mailman.qth.net/mailman/listinfo/elecraft>
> Help: http://mailman.qth.net/mmfaq.htm <http://mailman.qth.net/mmfaq.htm>
> Post: mailto:[hidden email] <mailto:[hidden email]>
>
> This list hosted by: http://www.qsl.net <http://www.qsl.net/>
> Please help support this email list: http://www.qsl.net/donate.html <http://www.qsl.net/donate.html>
> Message delivered to [hidden email] <mailto:[hidden email]>

______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html
Message delivered to [hidden email]

[Phil Kane-2]

On 6/13/2020 3:40 PM, Fred Jensen wrote:

> And, do any of you ever check the backups to make sure they really have
> anything there? 

Yes.  Each morning I check to see if the backup ran overnight and if the
file size is appropriate for what it should be.  If it's Zero I start
looking why.

73 de K2ASP - Phil Kane
Elecraft K2/100   s/n 5402

From a Clearing in the Silicon Forest
Beaverton (Washington County) Oregon
______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html
Message delivered to [hidden email]

[Bill Frantz]

Being a Mac guy, I use "Time Machine". I do test it every once
in a while when I recover a file, but having been in the
computer industry for my career, I am generally careful enough
that I don't have to recover files. (Knock on wood.)

I have the largest disk I could find at Costco as a backup disk
sitting on my desk.

The real test comes when I buy an new computer and restore the
entire backup to the new machine. That has worked through
several new computers. The one time it didn't work, the old
backup was so many no-longer-supported levels back, that the new
machine didn't recognize it. However, with Time Machine, if you
open the backup folder on the backup disk, you can dig down to a
complete file system image that can just be copied. I like
backup systems that are simple and don't try to do irreversible magic.

The other dimension of backup is several offsite disks. One is
at a house nearby, and another is on the other coast. Whenever I
travel to those locations, I make a backup. If everthing here
goes up in smoke, I do have some recourse.

73 Bill AE6JV

-----------------------------------------------------------------------
Bill Frantz        | Can't fix stupid, but   | Periwinkle
[phone]      | duct tape can muffle the| 150 Rivermead
Road #235
www.pwpconsult.com | sound... - Bill Liebman | Peterborough, NY 03458

______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html
Message delivered to [hidden email]

[W2xj]

I backup into the cloud. When I get a new Mac, it restores automatically.

Sent from my iPad

> On Jun 13, 2020, at 10:46 PM, Bill Frantz <[hidden email]> wrote:
>
> Being a Mac guy, I use "Time Machine". I do test it every once in a while when I recover a file, but having been in the computer industry for my career, I am generally careful enough that I don't have to recover files. (Knock on wood.)
>
> I have the largest disk I could find at Costco as a backup disk sitting on my desk.
>
> The real test comes when I buy an new computer and restore the entire backup to the new machine. That has worked through several new computers. The one time it didn't work, the old backup was so many no-longer-supported levels back, that the new machine didn't recognize it. However, with Time Machine, if you open the backup folder on the backup disk, you can dig down to a complete file system image that can just be copied. I like backup systems that are simple and don't try to do irreversible magic.
>
> The other dimension of backup is several offsite disks. One is at a house nearby, and another is on the other coast. Whenever I travel to those locations, I make a backup. If everthing here goes up in smoke, I do have some recourse.
>
> 73 Bill AE6JV
>
> -----------------------------------------------------------------------
> Bill Frantz        | Can't fix stupid, but   | Periwinkle
> [phone]      | duct tape can muffle the| 150 Rivermead Road #235
> www.pwpconsult.com | sound... - Bill Liebman | Peterborough, NY 03458
>
> ______________________________________________________________
> Elecraft mailing list
> Home: http://mailman.qth.net/mailman/listinfo/elecraft
> Help: http://mailman.qth.net/mmfaq.htm
> Post: mailto:[hidden email]
>
> This list hosted by: http://www.qsl.net
> Please help support this email list: http://www.qsl.net/donate.html
> Message delivered to [hidden email]

______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html
Message delivered to [hidden email]

[Peter W2IRT]

I have a paid Dropbox account, supplemented by Microsoft OneDrive, folders for which reside on my D: drive. All my data (including pictures, videos, music, documents, and work projects) go in there AND get backed up to a local NAS for redundancy. The OS partition (C : drive) including all software, settings, download, temporary and archive folders are backed up using a differential scheme to a NAS device daily, and a full backup to a Passport drive weekly. I use Acronis True Image Home for backup/restore software.

All my logs are backed up multiple places for safety, however due to the structure of both my DX logger and N1MM+ Contest Logger, they have to be run from a non-cloud folder, so extra care is taken in both cases.

I'm less worried about the OS and software; those can be reinstalled easily enough over the course of a few days (albeit with lots of swearing involved), but the data, all of it irreplaceable, has to be stored in Dropbox (meaning a copy exists not just in the cloud but on the hard drive of every computer I own that's connected to the account).

 - pjd

-----Original Message-----
From: [hidden email] <[hidden email]> On Behalf Of W2xj
Sent: Sunday, June 14, 2020 6:28 AM
To: Bill Frantz <[hidden email]>
Cc: [hidden email]
Subject: Re: [Elecraft] <Long & OT> Data protection and recovery techniques

I backup into the cloud. When I get a new Mac, it restores automatically.

Sent from my iPad

> On Jun 13, 2020, at 10:46 PM, Bill Frantz <[hidden email]> wrote:
>
> Being a Mac guy, I use "Time Machine". I do test it every once in a
> while when I recover a file, but having been in the computer industry
> for my career, I am generally careful enough that I don't have to
> recover files. (Knock on wood.)
>
> I have the largest disk I could find at Costco as a backup disk sitting on my desk.
>
> The real test comes when I buy an new computer and restore the entire backup to the new machine. That has worked through several new computers. The one time it didn't work, the old backup was so many no-longer-supported levels back, that the new machine didn't recognize it. However, with Time Machine, if you open the backup folder on the backup disk, you can dig down to a complete file system image that can just be copied. I like backup systems that are simple and don't try to do irreversible magic.
>
> The other dimension of backup is several offsite disks. One is at a house nearby, and another is on the other coast. Whenever I travel to those locations, I make a backup. If everthing here goes up in smoke, I do have some recourse.
>
> 73 Bill AE6JV
>
> -----------------------------------------------------------------------
> Bill Frantz        | Can't fix stupid, but   | Periwinkle
> [phone]      | duct tape can muffle the| 150 Rivermead Road #235
> www.pwpconsult.com | sound... - Bill Liebman | Peterborough, NY 03458
>
> ______________________________________________________________
> Elecraft mailing list
> Home: http://mailman.qth.net/mailman/listinfo/elecraft
> Help: http://mailman.qth.net/mmfaq.htm
> Post: mailto:[hidden email]
>
> This list hosted by: http://www.qsl.net Please help support this email
> list: http://www.qsl.net/donate.html Message delivered to
> [hidden email]

______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html Message delivered to [hidden email]

______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html
Message delivered to [hidden email]

[k6dgw]

 From history:  First job after discharge from the US Air Force in 1972
was in the Systems & Data Processing department at the County of
Sacramento.  Please note the date, we were still using 80-col punched
cards on an IBM 360-40 with 2314 disk drives and reel-to-reel tape
drives.  The tape library was in a vault where one wall was used for
system back-up tapes.  I was working on the Assessor's systems, one of
which was a multiple linear regression program for appraising similar
properties in suburban areas.  I was the only one in the department
fluent in FORTRAN IV, one of the reasons I got the job.

The Assessor asked me one day how his data were protected.  After
telling him, "Magnetic tape copies made whenever your parcel file [6
tape volumes] is mounted with write rings,"  I wondered, so I wrote a
little program to mount the file [sans write rings] and gather some
statistics.  The first 2 volume sets we ran were blank, so we went to
the end of the backup chain and tried the oldest one.  Not surprisingly,
it too was blank.  Checking some of the other systems [e.g. Tax
Collection, Welfare], we found:

1.  Blank back-up sets

2.  Back-up sets being recycled faster than the processing cycles that
created the data

3.  Two systems with intact, recoverable back-up sets

I just asked the original question because the "art of back-up" is often
the epitome of "Fire and Forget."  You might want to carve out a little
time and see if the recovery part of "backup and recovery" actually
recovers anything. [:=)

73,
Fred ["Skip"] K6DGW
Sparks NV DM09dn
Washoe County

On 6/14/2020 11:51 AM, Peter Dougherty wrote:

> I have a paid Dropbox account, supplemented by Microsoft OneDrive, folders for which reside on my D: drive. All my data (including pictures, videos, music, documents, and work projects) go in there AND get backed up to a local NAS for redundancy. The OS partition (C : drive) including all software, settings, download, temporary and archive folders are backed up using a differential scheme to a NAS device daily, and a full backup to a Passport drive weekly. I use Acronis True Image Home for backup/restore software.
>
> All my logs are backed up multiple places for safety, however due to the structure of both my DX logger and N1MM+ Contest Logger, they have to be run from a non-cloud folder, so extra care is taken in both cases.
>
> I'm less worried about the OS and software; those can be reinstalled easily enough over the course of a few days (albeit with lots of swearing involved), but the data, all of it irreplaceable, has to be stored in Dropbox (meaning a copy exists not just in the cloud but on the hard drive of every computer I own that's connected to the account).
>
>   - pjd
>
> -----Original Message-----
> From: [hidden email] <[hidden email]> On Behalf Of W2xj
> Sent: Sunday, June 14, 2020 6:28 AM
> To: Bill Frantz <[hidden email]>
> Cc: [hidden email]
> Subject: Re: [Elecraft] <Long & OT> Data protection and recovery techniques
>
> I backup into the cloud. When I get a new Mac, it restores automatically.
>
> Sent from my iPad
>
>> On Jun 13, 2020, at 10:46 PM, Bill Frantz <[hidden email]> wrote:
>>
>> Being a Mac guy, I use "Time Machine". I do test it every once in a
>> while when I recover a file, but having been in the computer industry
>> for my career, I am generally careful enough that I don't have to
>> recover files. (Knock on wood.)
>>
>> I have the largest disk I could find at Costco as a backup disk sitting on my desk.
>>
>> The real test comes when I buy an new computer and restore the entire backup to the new machine. That has worked through several new computers. The one time it didn't work, the old backup was so many no-longer-supported levels back, that the new machine didn't recognize it. However, with Time Machine, if you open the backup folder on the backup disk, you can dig down to a complete file system image that can just be copied. I like backup systems that are simple and don't try to do irreversible magic.
>>
>> The other dimension of backup is several offsite disks. One is at a house nearby, and another is on the other coast. Whenever I travel to those locations, I make a backup. If everthing here goes up in smoke, I do have some recourse.
>>
>> 73 Bill AE6JV
>>
>> -----------------------------------------------------------------------
>> Bill Frantz        | Can't fix stupid, but   | Periwinkle
>> [phone]      | duct tape can muffle the| 150 Rivermead Road #235
>> www.pwpconsult.com | sound... - Bill Liebman | Peterborough, NY 03458
>>

______________________________________________________________
Elecraft mailing list
Home: http://mailman.qth.net/mailman/listinfo/elecraft
Help: http://mailman.qth.net/mmfaq.htm
Post: mailto:[hidden email]

This list hosted by: http://www.qsl.net
Please help support this email list: http://www.qsl.net/donate.html
Message delivered to [hidden email]