Setting Up a Twitch Studio - Backup Overview and Strategy from hairylarry's blog
There are two types of backups. File backups and system backups. File backups preserve a set of files, for instance my folder /Video/twitch/2021-08-08 which contains the original recording of my twitch stream, Openshot project folders and .osp files that allow me to reload the project into Openshot. Rendered videos that are the actual mp4 files that get uploaded to the internet. Audio files, mp3s extracted from the video files, normalized, and ready to be uploaded to the internet. And other assets, my screenshot, my web size screenshot, and a text file, notes.txt.
If I lose my data on my hard drive, whether the entire folder or just a screenshot, I can go to the backup and recover this file. That is the whole purpose of backups.
System backups allow you to restore your entire working environment, the OS, the drivers, the software installs, the data, etc. I don't do system backups. If I have a catastrophic failure I do a new install and then restore my data from backup.
Some businesses cannot live with the cash flow loss due to excessive downtime. They will have system backups and redundant sustems ready to be swapped in if necessary. Their hard drives will be mirrored and every time they write a bit it will be written to two drives. They will have daily, hourly, or real time offsite backups just in case the catastrophic data loss take the form of a tsunami or a tornado.
Needless to say this kind of full system backup is difficult to install and requires constant monitoring and maintenance. It is not really realistic for an artist's, writer's or gamer's creative project. So I am not going into system backups any further. There are software and hardware turnkey systems that perform both system backups and file backups. If this is what you need and you can afford it then this is what you should do.
Just like web promotion can take the other 100% of your time, backups can take the other 100% of your time. So try to be realistic.
File backups breakdown further into two different categories, active data sets and static data sets.
The data set described above is a static data set. The files are created in the course of a day or two and then it is unlikely they will ever be changed again.
My webdev folder and my Documents folder are active data sets. Each day, week, or other random interval I write to them. I change existing files and I create new files. It's a constant churn, hence active.
Backing up static data sets to optical discs like CDs, DVDs, or Blu-Ray works great. You work on a project. You finish the project. You write it to an optical disc. And you're good. I do this with my Something Blue radio production. When I'm done with each show I have three audio discs and a data disc. The data disc is my static backup.
I also have a backup server that is great for static data sets and not so good for active data sets. I use rsync, a command line file transfer program. I run it once a day on every home directory. If a file is new it gets copied to the backup server. If a file is changed it gets copied to the backup server. As long as this is working correctly I never lose a data file.
Which is great! But not so good for active data sets. Because if I have to recover an active data set and I go to this backup server I get all my files back. But I also get all my deleted files back. This is a feature, not a bug. This type of backup does not delete files so if I accidentally delete a file and I don't notice it right away I can still recover it from my backup server.
But it is also not so great. Because if I want to restore an active data set I want to restore all the files and none of the deleted files so I have the exact same data set that I lost.
The other thing not so great about my rsync data server is if I make a bad edit to a file and I don't notice it until tomorrow I then have two copies of the file with the bad edit and no copies the way the file was before I messed it up.
The solution to this problem is called versioning. A backup system that keeps older versions of the file as well as the current version. Computer geeks have been working this problem for a while and they are able to keep many versions of a file without using excessive disk space by keeping only the changed parts in what are called diff files. Computer programmers rely on versioning because computer programs are easy to break and sometimes a programmer needs to find the code that was in the program three versions back.
The most popular versioning program for computer programs is called git and was written by Linus Torvalds, the original author and maintainer of the Linux kernel. git can be used for other projects that require versioning as well.
I am installing a software called Restic on another backup server. This server will have mirrored drives in what is called RAID 1 so every bit will be written to two drives. Restic will handle both the exact mirroring of active data sets and the versioning. Perfect for active data sets.
Which is another thing about backups. As your projects change your backup needs change. So besides monitoring your backups to make sure they are working good and that you have plenty of disk space sometimes you have to reevaluate your backup needs and change or add to what you are already doing.
A moving target. Complicated. And tricky.
So, first line of defense. Include backups in your workflow.
After I perform a twitch stream there are two copies of the stream. One on the twitch website and one on the hard drive of the computer in my twitch studio. The first thing I do is create the folder for video post production on my primary work computer and then I copy the video file from my computer in my twitch studio onto my primary work computer. This is where I am going to do my video post production, where I have Openshot, VLC, GIMP, and other tools installed. So no matter how bad a mistake I make while extracting my videos from this file I do not mess up the original video file. I am working with a backup copy. As part of my workflow I make a backup copy. It's the first thing I do. And I never forget to do it or find out to late that my automated backup system jammed and didn't back it up.
If you only have one system, install another hard drive for your post production work so that you will have two copies of your original file on two different hard drives. So, your backup is integrated into your workflow and that drives equipment purchase. See how quickly we get back to the fun stuff.
Up above I talked about my optical backups for my Something Blue shows and I mentioned three audio CDs. My show is an hour long and the station needs it delivered in two parts so they can include a station break at the half hour. So I write Part 1 to a CD and I write Part 2 to another CD. I used to carry these CDs to the station but now I use Exact Audio Copy to turn each CD into a single mp3 file and I deliver these files to the station on the internet.
Then I create a CD with the whole show. I rip it with Exact Audio Copy to a single mp3 file that I upload to MixCloud. MixCloud has widgets so I can then embed the show on the Something Blue website, sbblues.com, or include it anywhere that I can post embeds. I also use this copy for syndication.
So these three CDs are audio backups of my finished product and they are made as part of the workflow. I also make three 29 second spots for KASU to air and to help promote the show on the internet. And I make my From The Archives Of Something Blue podcast that airs on Arkansas Roots on Wednesdays and Saturdays and that I also post to my funkwhale, my hairylarry.rocks blog, and to the Something Blue website.
There are, again, additional assets, a picture to promote the show and a picture to promote the podcast as well as the text files with all my spoken parts.
So after I'm done with all that and the show is done and uploaded and the podcast and promo spots are uploaded I make an optical backup of the folder to a CD or DVD. From this backup I could recreate the show or even change it if necessary, even if I lost the hard drive with the shows data.
And that brings me to synchronization, the third strategy. I have three systems with all the programs installed necessary to produce Something Blue. My main workstation, my office workstation, and our photo server. Network shares are great but they are not as quick and easy as working from the local hard drive so I use a program called SyncThing that works on Linux and Windows to synchronize my somethingblue folders on all three systems. So every time I make a change to a file or add a file it is automatically updated on three different systems. This is easy to set up and check, works in the background, and adds to my convenience as well as providing backup copies of my changes.
If I accidentally delete a file, synchronization will delete the file in all synchronized folders, so like all the other backup strategies discussed there are advantages and disadvantages. I think synchronization is best used along with some other file backup strategy. Since I installed SyncThing they have added versioning so If it works for you this should solve the problem of accidental deletions.
And the final strategy is cloud backup. There are companies, like Carbonite, that handle your backups for you over the internet. This is a one stop solution that handles file backups with versioning. There is a per computer monthly fee on their personal accounts that is not high for what they provide but if you are in a multi computer shop, like mine, these fees may add up. They also rely on your bandwidth instead of local hardware. So if you've got plenty of bandwidth, like most twitch streamers, and you have only one or two computers I recommend that you try them out. My brother works from a single laptop and uses Carbonite for his only backup strategy and he has recovered lost or damaged files with it. He also likes the fact that when he upgrades his computer all he has to do is install Carbonite and then let his computer churn away for a day or two while carbonite restores all of his work files in the correct folders and it's just like he's sitting down at his old computer, except faster.
So we have covered four strategies, system backup, file backup, synchronization, and cloud backup. The first three take work to setup and maintain. If cloud backup works for you it is the simplest solution. So if you do your production work on a single computer and you have good bandwidth start there.
If the cloud doesn't work for you start with optical backups of your production folders as part of your workflow. If you have an older computer it will work fine as a backup server using rsync or a file transer program with a GUI to backup your home folders daily. You will probably need to buy a hard drive. I buy 4TB HGST drives on ebay for about $65. It doesn't really matter if your backup drives are fast as long as they are reliable. So spend an extra few dollars for new drives even though I have had good luck with used HGST drives too.
And make sure that you don't have to spend too much time on your backups or they won't get done. Integrate manual backups to optical discs into your workflow and automate everything else.
After all, we're not here to make backups. We're here to create! Just don't forget that when it comes to computers failure is not if but when and make sure your backup strategies will allow you to recover your precious work.
If I lose my data on my hard drive, whether the entire folder or just a screenshot, I can go to the backup and recover this file. That is the whole purpose of backups.
System backups allow you to restore your entire working environment, the OS, the drivers, the software installs, the data, etc. I don't do system backups. If I have a catastrophic failure I do a new install and then restore my data from backup.
Some businesses cannot live with the cash flow loss due to excessive downtime. They will have system backups and redundant sustems ready to be swapped in if necessary. Their hard drives will be mirrored and every time they write a bit it will be written to two drives. They will have daily, hourly, or real time offsite backups just in case the catastrophic data loss take the form of a tsunami or a tornado.
Needless to say this kind of full system backup is difficult to install and requires constant monitoring and maintenance. It is not really realistic for an artist's, writer's or gamer's creative project. So I am not going into system backups any further. There are software and hardware turnkey systems that perform both system backups and file backups. If this is what you need and you can afford it then this is what you should do.
Just like web promotion can take the other 100% of your time, backups can take the other 100% of your time. So try to be realistic.
File backups breakdown further into two different categories, active data sets and static data sets.
The data set described above is a static data set. The files are created in the course of a day or two and then it is unlikely they will ever be changed again.
My webdev folder and my Documents folder are active data sets. Each day, week, or other random interval I write to them. I change existing files and I create new files. It's a constant churn, hence active.
Backing up static data sets to optical discs like CDs, DVDs, or Blu-Ray works great. You work on a project. You finish the project. You write it to an optical disc. And you're good. I do this with my Something Blue radio production. When I'm done with each show I have three audio discs and a data disc. The data disc is my static backup.
I also have a backup server that is great for static data sets and not so good for active data sets. I use rsync, a command line file transfer program. I run it once a day on every home directory. If a file is new it gets copied to the backup server. If a file is changed it gets copied to the backup server. As long as this is working correctly I never lose a data file.
Which is great! But not so good for active data sets. Because if I have to recover an active data set and I go to this backup server I get all my files back. But I also get all my deleted files back. This is a feature, not a bug. This type of backup does not delete files so if I accidentally delete a file and I don't notice it right away I can still recover it from my backup server.
But it is also not so great. Because if I want to restore an active data set I want to restore all the files and none of the deleted files so I have the exact same data set that I lost.
The other thing not so great about my rsync data server is if I make a bad edit to a file and I don't notice it until tomorrow I then have two copies of the file with the bad edit and no copies the way the file was before I messed it up.
The solution to this problem is called versioning. A backup system that keeps older versions of the file as well as the current version. Computer geeks have been working this problem for a while and they are able to keep many versions of a file without using excessive disk space by keeping only the changed parts in what are called diff files. Computer programmers rely on versioning because computer programs are easy to break and sometimes a programmer needs to find the code that was in the program three versions back.
The most popular versioning program for computer programs is called git and was written by Linus Torvalds, the original author and maintainer of the Linux kernel. git can be used for other projects that require versioning as well.
I am installing a software called Restic on another backup server. This server will have mirrored drives in what is called RAID 1 so every bit will be written to two drives. Restic will handle both the exact mirroring of active data sets and the versioning. Perfect for active data sets.
Which is another thing about backups. As your projects change your backup needs change. So besides monitoring your backups to make sure they are working good and that you have plenty of disk space sometimes you have to reevaluate your backup needs and change or add to what you are already doing.
A moving target. Complicated. And tricky.
So, first line of defense. Include backups in your workflow.
After I perform a twitch stream there are two copies of the stream. One on the twitch website and one on the hard drive of the computer in my twitch studio. The first thing I do is create the folder for video post production on my primary work computer and then I copy the video file from my computer in my twitch studio onto my primary work computer. This is where I am going to do my video post production, where I have Openshot, VLC, GIMP, and other tools installed. So no matter how bad a mistake I make while extracting my videos from this file I do not mess up the original video file. I am working with a backup copy. As part of my workflow I make a backup copy. It's the first thing I do. And I never forget to do it or find out to late that my automated backup system jammed and didn't back it up.
If you only have one system, install another hard drive for your post production work so that you will have two copies of your original file on two different hard drives. So, your backup is integrated into your workflow and that drives equipment purchase. See how quickly we get back to the fun stuff.
Up above I talked about my optical backups for my Something Blue shows and I mentioned three audio CDs. My show is an hour long and the station needs it delivered in two parts so they can include a station break at the half hour. So I write Part 1 to a CD and I write Part 2 to another CD. I used to carry these CDs to the station but now I use Exact Audio Copy to turn each CD into a single mp3 file and I deliver these files to the station on the internet.
Then I create a CD with the whole show. I rip it with Exact Audio Copy to a single mp3 file that I upload to MixCloud. MixCloud has widgets so I can then embed the show on the Something Blue website, sbblues.com, or include it anywhere that I can post embeds. I also use this copy for syndication.
So these three CDs are audio backups of my finished product and they are made as part of the workflow. I also make three 29 second spots for KASU to air and to help promote the show on the internet. And I make my From The Archives Of Something Blue podcast that airs on Arkansas Roots on Wednesdays and Saturdays and that I also post to my funkwhale, my hairylarry.rocks blog, and to the Something Blue website.
There are, again, additional assets, a picture to promote the show and a picture to promote the podcast as well as the text files with all my spoken parts.
So after I'm done with all that and the show is done and uploaded and the podcast and promo spots are uploaded I make an optical backup of the folder to a CD or DVD. From this backup I could recreate the show or even change it if necessary, even if I lost the hard drive with the shows data.
And that brings me to synchronization, the third strategy. I have three systems with all the programs installed necessary to produce Something Blue. My main workstation, my office workstation, and our photo server. Network shares are great but they are not as quick and easy as working from the local hard drive so I use a program called SyncThing that works on Linux and Windows to synchronize my somethingblue folders on all three systems. So every time I make a change to a file or add a file it is automatically updated on three different systems. This is easy to set up and check, works in the background, and adds to my convenience as well as providing backup copies of my changes.
If I accidentally delete a file, synchronization will delete the file in all synchronized folders, so like all the other backup strategies discussed there are advantages and disadvantages. I think synchronization is best used along with some other file backup strategy. Since I installed SyncThing they have added versioning so If it works for you this should solve the problem of accidental deletions.
And the final strategy is cloud backup. There are companies, like Carbonite, that handle your backups for you over the internet. This is a one stop solution that handles file backups with versioning. There is a per computer monthly fee on their personal accounts that is not high for what they provide but if you are in a multi computer shop, like mine, these fees may add up. They also rely on your bandwidth instead of local hardware. So if you've got plenty of bandwidth, like most twitch streamers, and you have only one or two computers I recommend that you try them out. My brother works from a single laptop and uses Carbonite for his only backup strategy and he has recovered lost or damaged files with it. He also likes the fact that when he upgrades his computer all he has to do is install Carbonite and then let his computer churn away for a day or two while carbonite restores all of his work files in the correct folders and it's just like he's sitting down at his old computer, except faster.
So we have covered four strategies, system backup, file backup, synchronization, and cloud backup. The first three take work to setup and maintain. If cloud backup works for you it is the simplest solution. So if you do your production work on a single computer and you have good bandwidth start there.
If the cloud doesn't work for you start with optical backups of your production folders as part of your workflow. If you have an older computer it will work fine as a backup server using rsync or a file transer program with a GUI to backup your home folders daily. You will probably need to buy a hard drive. I buy 4TB HGST drives on ebay for about $65. It doesn't really matter if your backup drives are fast as long as they are reliable. So spend an extra few dollars for new drives even though I have had good luck with used HGST drives too.
And make sure that you don't have to spend too much time on your backups or they won't get done. Integrate manual backups to optical discs into your workflow and automate everything else.
After all, we're not here to make backups. We're here to create! Just don't forget that when it comes to computers failure is not if but when and make sure your backup strategies will allow you to recover your precious work.
Links
Groups
The Wall