Welcome to the ShiftyBlog!

Posted on Wed 24 February 2016 in General

Well, welcome back to the ShiftyBlog. I've decided to move away from the terrible cluster of hell that is WordPress, and have decided to play around with a new, flat-file format(Pelican, for anybody wondering).

Don't worry, my guides for the Amazon Unlimited Plex Server still exist, and are undergoing a bit of a rewrite to make them more up to date and to play nicely with Markdown formatting. They'll be up shortly.

A whole bunch of stuff has happened in terms of servers and swapping stuff around, so I'll actually be following my own guides again to get set up, meaning I can fix stuff on the way.

Since it's flat-file HTML, you can just ctrl+s the page if you're nervous of it going down again (it won't!), but be sure to check back for any updates!

Thanks guys,

Love Makeshift <3


Unlimited Plex Media Server (using Amazon's Unlimited Storage Backend) Pt1

Posted on Thu 25 February 2016 in Plex

Note: This is a project, rather than a tutorial, and such may not be particularly succinct or easy to follow. Please read it all the way through before starting as there may be entire sections that you can skip. Sub-note: I do not condone piracy. This tutorial series is designed to allow you to download legal movies, TV shows and audiobooks and stream them.

Today, we’ll be putting together a Plex Media Server with an Amazon unlimited size cloud drive.

Cloud Storage used in ways it was not intended

Everybody has heard of and probably used cloud storage. Dropbox, Mega, SkyDrive, whatever. Very recently I found out about Amazon’s contender: Amazon Cloud. I hadn’t heard of it before, probably because it skimmed its way under the radar here in the UK as it doesn’t offer anything particularly crazy. In the US however, they offer an “Unlimited Everything” package, unlimited file storage for only $60 a year.

As a person with near enough 5TB of media and on a network that doesn’t allow port forwarding (Student life!), this caused a pretty severely raised eyebrow. If I could store and access all of my media on the go from a reliable source, not only could I cut down on my drives at home (All of which are fast approaching “Warning” on SMART due to 20k+ hours of uptime), but I would be significantly more comfortable on trains, especially with the PlexSync(assuming I ever actually bother getting a PlexPass). Not to mention it means I could just set up my dedicated box (which is already a torrent server) to upload straight to Amazon and have it accessible to all of my devices.

So let’s go through the steps of trying to get a cloud based media server running.

Requirements

An Amazon account: Thankfully, if you’re not willing to shell out $60 for an unlimited everything package just yet, Amazon offers 5GB for free for you to play with before you upgrade, so I’ll be using that. This is going to be our cloud storage. Note that this must be an American Amazon account, and a UK account will just redirect to the UK site. Make sure you register and sign up to cloud drive with a US account.

A Dedicated Hosted Server: Yeah, this is where most people fall short. Any VPS or hosted/local server will do as long as you can forward ports and it has a decent download and upload speed. This is going to be the “head” for our storage: We’ll trick it into thinking the Amazon storage is local, then use it to stream data to our clients. Effectively, it’s a proxy. I get most of mine from OVH, and you can get fairly cheap ones from the OVH company Kimsufi. Bear in mind that if you set your Plex to convert your files, you will not only need a fast internet connection to stream it from Amazon, but a fast CPU to convert it on the fly.

Some Linux Knowledge: For this project I’m assuming the dedicated server is an Ubuntu based server. In theory it should work fine with any Debian based distribution, and with some modifications of the programs in use it could even work with Windows. I am by no means a Linux pro, so if you see any improvements to be made, please inform me.

A PLEX account: PLEX is the easiest way I’ve found to quickly stream media to pretty much any platform, so I’ll be using that to try and view my library online.

Where do we start?

Being that this is actually the second iteration of this project, I’ve found much less frustrating ways to get it running. We'll be starting with Plex itself. I quite like putting everything under the "Plex" user for neatness sake, though the Plex user isn't actually accessible by default - We'll fix that later.

First up, let’s grab some Plex and set it up. SSH into your server and get ready for some terminal action. If you’re using Putty, under the “Connection” tab you may want to enable keepalives so we don’t randomly get kicked off while waiting for something.

Setting up Plex

For now, let’s jump into our home directory, download and install plex(Latest version at time of posting). This should be fairly easy.

cd ~
wget https://downloads.plex.tv/plex-media-server/0.9.15.6.1714-7be11e1/plexmediaserver_0.9.15.6.1714-7be11e1_amd64.deb
sudo dpkg -i plexmediaserver_0.9.15.6.1714-7be11e1_amd64.deb

If we navigate to http://serveripaddress:32400/web we should get the Plex login, fantastic! If you like, you can login, but unfortunately we can’t yet configure Plex due to being a client. Before we bother with that, let’s make our Plex user a bit more usable.

First, we need to make plex an account we can login to so we can test stuff out. We then need to make it a sudoer so we can actually do things, and we can also disable its SSH access to make it a little more secure.

sudo su
passwd plex
adduser plex sudo

This allows us to jump over to the root user and then update the Plex user password, and makes them a sudoer. Let’s disable their SSH login.

nano /etc/ssh/sshd_config

Scroll down to the bottom and type

DenyUsers plex

Save, and run service ssh restart That should disable the plex account from being logged in via SSH. To get to it, we now need to su over to it. Let’s do that now.

su plex Now, the plex user technically does have a home directory, it’s just in an awkward place. To make life a little bit easier when trying to work with files, let’s just make a folder for plex to play with in the home directory. Note that “~” will NOT resolve to this directory, this is just for playing with.

sudo mkdir /home/plex/
sudo chown plex /home/plex/

Because it will be used later as our cloud drive storage location, let’s just quickly make an amazon folder.

cd /home/plex
mkdir amazon

Since we’re in our pretend home directory, we can start setting up acd_cli, which is the application we’ll be using to access our cloud drive.

acd_cli

We really should get some test data on our Amazon Cloud. I used a YouTube music video I had sitting around. You can start uploading your entire media library if you like, but for now I’m being conservative. On a desktop machine, uploading single files is trivial, merely go to your Amazon Cloud Dashboard and drag a video file or two onto the browser window.

Next, we should make our media visible on our dedicated server. For this we’ll be using an open source app called acd_cli by a user on GitHub called yadayada. Effectively, it’s a command line interface that "symlinks" your cloud drive to an accessible part of the filesystem, so you have read/write access.

The easiest way to get acd_cli installed is by starting with easy install from python-setuptools. Let’s grab a bunch of other dependencies at the same time, as well as “pip”, a package manager for python.

sudo apt-get install python3-setuptools python3-appdirs python3-dateutil python3-requests python3-sqlalchemy
sudo easy_install3 -U pip

Once that’s done, we can grab acd_cli from source with all its dependencies

sudo pip3 install –upgrade git+https://github.com/yadayada/acd_cli.git

We can ensure all that worked by running acd_cli sync

You should have a request for cookies. Hit A to accept all the incoming cookies. After a few moments you’ll have a fantastic replica of the Amazon website in text.

Since text isn’t my favourite way to browse, and probably isn’t yours either, hit Q then Y to quit, and take note of the link it printed.

On your computer, navigate to https://tensile-runway-92512.appspot.com/ , which will direct you to Amazon to login. This step is to authorise the application to your Amazon account.

After authorising, it should have downloaded a file called “oauth_data” for you. SFTP into your dedicated server and drop this file into /var/lib/plexmediaserver/.cache/acd_cli (/var/lib/plexmediaserver is the real home directory for plex, by the way). This may take a little bit of effort, since you can’t login as plex and your other user probably doesn’t have permission into Plex’s home folder, but as long as it makes its way into the folder it’s fine. Personally I SFTP'd it into my home directory, then used

sudo mv oauth_data /var/lib/plexmediaserver/.cache/acd_cli/
sudo chown plex:plex /var/lib/plexmediaserver/.cache/acd_cli/oauth_data

Note that if you copy it using root, you’ll need to chown the file to plex so it can read it.

If you exited acd_cli to do the copying, run acd_cli sync again or just press enter if you hadn’t. It should correctly sync.

Now we can actually mount the drive using acd_cli

acd_cli mount -ao /home/plex/amazon This may give you a couple of “fusermount” permission errors. This is because plex doesn’t have write permission for a couple of things, but we can actually ignore this error and cd into our amazon folder, where we should now have a mounted drive! If you're wondering, the ao means "allow other users". This is insanely handy later.

So here is where we reach our first caveat - Yes, you have mounted your Amazon drive, and you can even read and write files to it and check out how cool it looks in the web UI. Unfortunately, large files don't work so well.

Thankfully, we have a fix for that later.

Configuring Plex

Since we’re not local to our Plex installation, Plex web isn’t going to give us half the settings we want. We can make ourselves tunnelled so Plex thinks we’re local using Putty.

Open up Putty, enter all your normal connection settings, then go down to the SSH menu, and open Tunnels. In source port, enter 32400 and in destination enter localhost:32400. Once you login, you should have a working tunnel.

In your browser, navigate to http://localhost:32400/web . You may need to log in again, but once you do you can click the settings button in the top right. Set it up how you like. You may need to add your login once more, You may also want to tick “Update my library automatically”.

Once done, hit the home button at the top left and find your server in the navigation pane on the left. Click the little + button to add a new library. In my case, I knew my test video's name was formatted weirdly so I picked Home Video, and added Amazon as a library. Plex will do a quick scan and then huzzah, you should be able to stream media!

Note that if you have long loading times you may have left transcoding on. On a slow server, transcoding will take a long time.

If you add more media to cloud drive, you will have to manually run “acd_cli sync” to update your cloud drive, then do a library refresh on Plex. We will automate this in future.

What now?!

In part two, we explore adding a torrent server into the mix, automatically uploading the torrents into the cloud, and deleting them from the local server. In parts three and four, we’ll explore automated ways of grabbing torrents, automatically uploading into different folders (and different libraries) for types of media, setting up crons for syncing, and boot-time mounting and starting of scripts.


Unlimited Plex Media Server (using Amazon's Unlimited Storage Backend) Pt2

Posted on Thu 25 February 2016 in Plex

Update: It may be that acd_cli has been updated to be able to manage larger files. I haven't confirmed this, so I'm not going to update this yet.

Note: This blog does not condone piracy and no pirated content was used in the creation of this post or the series. We only condone downloading and hosting of freely available series and movies when you have the appropriate rights.

Alright, we have our cloud based Plex server, so let’s work out how we’re going to be putting media on it. I don’t really fancy downloading it locally then uploading it to Amazon Cloud, so we’re going to set our dedicated server up to be able to download torrents and automatically upload them to Amazon Cloud for us.

Assumptions made in this post: You’ve followed part 1 of this series. You’re logged in as the Plex user. rTorrent = The torrent server back-end ruTorrent – The torrent server front-end we’ll be using Note: If you already run apache, bear in mind the script we will be using may affect some of your settings, and you should back them up first.

Using Our Dedicated Server to Download and Upload to Amazon Cloud

Using an absolutely fantastic script by Kerwood on Github, we can have rTorrent, ruTorrent and a bunch of plugins installed for us. Grab his script, make it executable and run it.

wget https://raw.githubusercontent.com/Kerwood/rtorrent.auto.install/master/Rtorrent-Auto-Install-3.0.2-Debian-Wheezy
chmod +x Rtorrent-Auto-Install-3.0.2-Debian-Wheezy
sudo ./Rtorrent-Auto-Install-3.0.2-Debian-Wheezy

At the time of writing, 3.0.2 is the most up to date.

Follow the on-screen instructions, install all of the plugins (because why not) and spend an hour or so watching it compile. Make sure Putty has keep-alive turned on so it doesn’t timeout, and for the love of God run it in a screen so you can do other things at the same time.

Make sure to press 3 and download all the plugins, to exit that screen press 0. Press 0 to begin the compilation and installation.

It is advised to use Apache to stay in sync with this guide. We will be doing a lot of modification of config files later on and I’m not sure how well that works with nginx.

By default, the script will have you downloads folder as /home/username/Downloads. We’ll be modifying that in a minute anyway.

cd into /home/plex/Downloads and then make two folders inside, toupload and downloading

mkdir toupload
mkdir downloading

toupload is where finished downloads will save to be automatically uploaded to clouddrive. Downloading will contain then currently downloading torrents.

Once rTorrent is definitely finished installing, you can head over to http://yourip/rutorrent

First, we want to make absolutely and utterly certain that we do not upload any unfinished torrents to Amazon, as that would be annoying. So open up rutorrent in your browser and open up the cog at the top. You may want to modify the settings to your liking, but I’m assuming leaving things in home/Download is fine for you.

Under the “Downloads” tab, change your default directory for downloads to ~/Downloads/downloading. This is so we don’t mix up our finished and unfinished torrents.

Under the “Autotools” tab, tick the “AutoMove” checkbox and select the “toupload” folder as our finished torrent location.

Feel free to test at this point to ensure your downloads sit in the downloading folder when expected, and move to toupload when finished.

If that all works, we can continue building our script! Acd_clid doesn’t currently support uploading from the FUSE mount, so we need a script that does it for us.

So what does our script need to do?

  • Detect changes in our toupload folder
  • Upload changed files
  • Delete the files from the toupload folder

So first off we’re going to have to be able to detect the changes. There’s a great little package that can do that:

sudo apt-get install inotify-tools This is the script that I threw together that seems fairly effective:

1
2
3
4
5
6
7
8
#!/bin/bash
FOLDER="/home/plex/toupload"
inotifywait -m $FOLDER -e create -e moved_to |
        while read path action file; do
                echo "The file '$file' appeared in directory '$path' via '$action'"
                acd_cli ul $FOLDER/"$file" /Media/Unsorted/
                rm -rf $FOLDER/"$file"
        done

Simply create a script under /home/plex with nano autoupload.sh, paste in the script and modify it to your liking. You will need to change the cd and inotifywait commands to your download location. If you’ve been following it like mine, it should be /home/plex/toupload.

Use chmod +x autoupload.sh to give it execute permission. We’ll then need to open up a screen. If you’re using su to use the plex user, you’ll need to run the command script /dev/null first, otherwise screen won't open (trust me).

Run screen, then start the script with ./autoupload.sh

Later on, you can hold down Ctrl and press A, then release both and press D to detach from the screen and leave the script running in the background. For now, I advise keeping it up and giving it a quick test by downloading something with rTorrent.

Bear in mind that if you restart your server, you will have to manually start acd_cli and the upload script, as this guide does not deal with automatic starting yet.

You should now have an auto-uploading torrent server which uploads into clouddrive! You can change the script to upload into whatever folder you like. At the moment mine uploads into /Media/, and unfortunately it doesn’t separate into TV or Movies. But we’ll sort that out in our next tutorial when we start setting up automatic movie and TV series downloading (Yup, that’s next! And I've tested it and everything, it works).

So what do we need right now?

Well, at the moment, acd_cli doesn’t ever actually sync our FUSE mount. Sure, it uploads, but Plex doesn’t get updated with new content. It won’t be sorted too well, but we can still stream files.

So let’s make our local drive update using crontab.

First of all, we need to get into crontab. Type crontab -e and press “2” for nano.

This post won’t go over how crontab works, all you need to know is that if you add this to the crontab file, acd_cli will sync every minute:

* * * * * /usr/local/bin/acd_cli sync

It should now sync automatically if you add something to the drive!

Awesome! What now?

Well, at this point we’ve got a pretty decently working server. We can download torrents, they upload to clouddrive then delete themselves locally, then the server will refresh, PleX will update, and we can watch things!

It isn’t sorted so well at the moment, but in the next section when we set up automatic downloading we’ll be able to set up uploading into different folders.

For now, have fun!