Working end-to-end script automation for downloads, filing, metadata

Ask for help and report issues not specific to either the Mac OS X or GTK+ versions of Transmission
Post Reply
Fever1223
Posts: 1
Joined: Mon Jul 03, 2017 8:55 pm

Working end-to-end script automation for downloads, filing, metadata

Post by Fever1223 »

I hope that people find this collection of tools useful. I think it makes the whole experience as transparent as possible with as little user interaction as anyone could reasonably expect.

I’ve been enjoying using Transmission for some years now, and like the presentation of each show with its accompanying metadata for the shows I watch, typically via plex. But what I really want is:
  • Automatic downloading of each episode of the shows that interest me.
  • Convert the episode from whatever format it presents in to .mp4, ideally without transcoding, with an h264 video stream and an aac or ac3 audio stream.
  • Renaming the resultant downloaded files to the form “Showname S##E## Episode Title.mp4”.
  • Adding the following metadata items: show name, season number, episode number, episode title, track number (same as episode number), show synopsis, and artwork.
  • Move the video file with embedded metadata to the appropriate directory for Plex to see.
  • Make it possible for me to drag-and-drop the resultant shows into iTunes for potential transfer to my phone so that I can watch things while I’m traveling (about one week out of three) with all the appropriate metadata so that it looks more or less like purchased iTunes Music Store content as far as the UI is concerned.
The following set of tools accomplishes this perfectly. For the last few weeks, things just “show up” in Plex when they’re ready and I can watch them without having to think about putting them there. And I can import anything I gather by this means into iTunes so that when I’m sitting on the airplane or in my hotel room during a trip, I have all the content I want available to enjoy, with all the metadata that makes the presentation feel professional and convenient. These directions are specific to MacOS – Unix guys, you’re on your own, but it shouldn’t be hard to make the necessary changes to file locations and so on.
  1. Start by going ShowRSS.info and create an account, and build a custom RSS feed for all the shows you want to watch. Start with just a couple of shows – no need to go crazy right at first. The site lets you arrange your preferences as to SD, HD, or “don’t care”, and whether you prefer magnet links or actual torrent files. Once you have that all set up correctly, you will want to make note of your custom feed’s URL, found on the bottom of the “my feed” page.
  2. Open the terminal (assuming you’re using the bash shell – just type bash at the command prompt if you normally use the c-shell or one of the others out there) and:

    1. Install command line tools:

      Code: Select all

      xcode-select --install
    2. Install homebrew, a useful package manager for MacOS.

      Code: Select all

      /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    3. Install or update python (you’re going to need 2.7.x or better for this)

      Code: Select all

      brew install python
      pip install --upgrade pip 
      pip install --upgrade setuptools
    4. Install flexget

      Code: Select all

      pip install --upgrade flexget
    5. Create a launch agent to launch flexget at startup:
      Type:

      Code: Select all

      nano ~/Library/LaunchAgents/com.flexget.plist
      and insert:

      Code: Select all

      <?xml version="1.0" encoding="UTF-8"?> 
      <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> 
      <plist version="1.0"> 
      <dict> 
      	<key>Label</key> 
      	<string>com.flexget</string> 
      	<key>ProgramArguments</key> 
      	<array> 
      		<string>/usr/local/bin/flexget</string> 
      		<string>--cron</string> 
      		<string>execute</string> 
      	</array> 
      	<key>Nice</key> 
      	<integer>1</integer> 
      	<key>StartInterval</key> 
      	<integer>900</integer> 
      	<key>RunAtLoad</key> 
      	<true/> 
      </dict> 
      </plist>
      Once that’s finished, type the following command in the terminal:

      Code: Select all

      launchctl load -w /Users/USERNAME/Library/LaunchAgents/com.flexget.plist
    6. Create an appropriate configuration file for flexget. In my case,

      Code: Select all

      nano ~/.flexget/config.yml
      and then add

      Code: Select all

      tasks:
        # downloading task and remove finished torrents
        # called via cron every 30 minutes.
        download-showrss:
          rss:
            url: PUT IN YOUR CUSTOM SHOWRSS.INFO FEED ADDRESS HERE.
          all_series: yes
          transmission: yes
          clean_transmission:
            host: localhost
            port: 9091
            username: username
            password: password
            delete_files: yes
            transmission_seed_limits: yes
    7. Just a couple more things. Install transmission:

      Code: Select all

      brew install transmission
      ln -sfv /usr/local/opt/transmission/*.plist ~/Library/LaunchAgents
      (make sure that, as you did in the flexget plist above, the RunAtLoad key is set to true).

      Now you’ll want to edit the settings.json for Transmission, at ~/Library/Application\ Support/transmission-daemon/settings.json.
      This part can get a little complicated, so be careful, but what you’re look for is at the part of the alphabetized list of parameters beginning with rpc. I set
      “rpc-enabled” to “true” (lets you control remotely),
      “rpc-whitelist” to "127.0.0.*,192.168.1.*" (from your home network),
      "script-torrent-done-enabled" to “true” (running a script when each torrent finishes),
      and "script-torrent-done-filename" to "/usr/local/bin/conv" (specifically, this script).

      You can also adjust the seeding ratio to suit your own notions of karma here. Do this with the transmission-daemon off – check with ps ax | grep transmission and kill the number of the process, if any, that appears. Once the changes are made, restart the transmission-daemon process by typing transmission-daemon.
    8. Install tvnamer, AtomicParsley, and ffmpeg

      Code: Select all

      brew install tvnamer
      brew install atomicparsley
      brew install ffmpeg
  3. If all this has worked right, you are able to go to your browser either on the machine you’re using or any other machine on your local network, and type in {IP address of machine I’m installing all this stuff on:9091} and see a transmission web interface. In my case typing “MyMacMiniName:9091” in Safari brings this up.
  4. Test the use of transmission by finding a magnet link for something and seeing if it downloads. Ctrl-click or right-click on the little magnet symbol, select copy link, and paste it in the window that appears when you click on the open torrent symbol on the transmission web interface. You should be getting a download to the directory you specify in your transmission-daemon preferences, which you can adjust to the /Users/<username>/Downloads for the script below.
  5. Configure tvnamer: When you install tvnamer, there should be a file in your home directory named .tvnamer.json (the leading dot will make it invisible to your GUI but the terminal should reveal it with ls –a. Check the line that begins

    Code: Select all

    "filename_with_episode": 
    to be sure it ends with

    Code: Select all

    "%(seriesname)s S%(seasonno)02dE%(episode)s %(episodename)s%(ext)s", 
    This is my personal format preference for show titles, but feel free to edit this to suit your own preferences, so long as you make the similar, appropriate changes to the script below.
  6. The tvnamer utility installs the api for theTVDB. This is useful, because it means you can write the following function to grab a show’s overview, or synopsis:

    Code: Select all

    nano /usr/local/bin/getoverview.py
    and then in the nano editor, type:

    Code: Select all

    #!/usr/bin/python
    # getoverview.py
    SINGLE_QUOTE_MAP = {
            0x2018: 39,
            0x2019: 39,
            0x201A: 39,
            0x201B: 39,
            0x2039: 39,
            0x203A: 39,
    }
    
    DOUBLE_QUOTE_MAP = {
            0x00AB: 34,
            0x00BB: 34,
            0x201C: 34,
            0x201D: 34,
            0x201E: 34,
            0x201F: 34,
    }
    
    def convert_smart_quotes(str):
            return str.translate(DOUBLE_QUOTE_MAP).translate(SINGLE_QUOTE_MAP)
    
    import sys, tvdb_api
    cmdargs = str(sys.argv)
    t = tvdb_api.Tvdb()
    episode = t [str(sys.argv[1])] [float(sys.argv[2])] [float(sys.argv[3])]
    value = episode['overview']
    value = convert_smart_quotes(value)
    print value
    The show overviews frequently include "curly" quotes, which look nice, but they tend to break things when put into bash script arguments, so I just remap them all to the boring ordinary quotes.
  7. The last piece of the puzzle is the convert script. This is the code for that:

    Code: Select all

    #!/bin/bash
    #Based in part on code convert-copy.sh by Rahul Jiresal on github
    
    if [  "$#" -eq 1 ]; then
    	TR_TORRENT_DIR=$HOME/Downloads/
    	TR_TORRENT_NAME=$1 #This is here to let the script run from the downloads folder when testing on a file passed as the first argument
    fi 
    
    TR_DOWNLOADS="$TR_TORRENT_DIR/$TR_TORRENT_NAME"
    
    LOG_DIR="$HOME/Library/Application Support/Transmission/Logs"
    mkdir -p "$LOG_DIR"
    LOGFILE=$LOG_DIR/$TR_TORRENT_NAME.log
    if [ ! -e "$LOGFILE" ] ; then
    	touch "$LOGFILE"
    fi
    
    function logthis {
    	echo "`date '+%d %b %Y %H:%M:%S'`    $1" >> "$LOGFILE"
    }
    
    function process_file() {
    	filename=$(basename "$1")
    	extension="${filename##*.}"
    	filename="${filename%.*}"
    	cd $HOME/Downloads > /dev/null 2>&1
    	current_time=$(date "+%Y.%m.%d-%H.%M.%S")
    	mkdir "$current_time" #creates working folder for 
    	if [ -n "$(ffmpeg -i "$1" 2>&1 >/dev/null | grep "h264")" ]; #ffmpeg writes its output to stderr for some reason
    		then 
    			video_codec="copy"; #if it's already h264 then just copy it through
    		else 
    			video_codec="h264"; #otherwise, transcode it to h64
    	fi
    	if [ -n "$(ffmpeg -i "$1" 2>&1 >/dev/null | grep "aac")" ]; 
    		then 
    			audio_codec="copy"; #if it's already aac then copy it through
    		elif [ -n $(ffmpeg -i "$1" 2>&1 >/dev/null | grep "ac3") ]; 
    		then
    			audio_codec="copy"; #ac3 is good also
    		else 
    			audio_codec="aac"; #otherwise, transcode to aac
    		fi
    	ffmpeg -i "$1" -hide_banner -loglevel quiet -y -c:v $video_codec -c:a $audio_codec "$current_time/$filename.mp4"
    	logthis "Conversion of $1 to .mp4 format complete"
    	filename=$(tvnamer $current_time/"$filename.mp4" | grep "New filename: " | cut -d:  -f2- | cut -c 2-)
    	logthis "File has been renamed to: $filename" #tvnamer changes the filename and gives output as to what it did - this gets searched to find the new filename
    	regex="(.*) S([0-9]{2})E([0-9]{2}) (.*)\.mp4" #this regex breaks the new filename into four pieces for later processing
    		    if [[ "$filename" =~ $regex ]]; then
    			show_name=${BASH_REMATCH[1]}
    			season_no=$(echo ${BASH_REMATCH[2]} | sed 's/^0*//') #strips leading zeroes
    			episode_no=$(echo ${BASH_REMATCH[3]} | sed 's/^0*//')
    			episode_title=${BASH_REMATCH[4]}
    			logthis "Show is $show_name, season $season_no, episode $episode_no, titled $episode_title"
    			descr="$(/usr/local/bin/getoverview.py "$show_name" $season_no $episode_no)" #gets the episode description
    			theargs="AtomicParsley \"$current_time/$filename\" -H \"$show_name\" -U $season_no -N $episode_no -S \"TV Show\" --tracknum $episode_no --title \"$episode_title\" --artwork ~/Pictures/\"$show_name $season_no.jpg\" --description \"$descr\" --overWrite" #note the ~/Pictures/ directory - edit to where you keep your show art. 
    			eval $theargs    
    	    fi
    	source_dir=$(printf %q "$current_time/$filename") #use of printf to escape nonstandard characters - a show with an apostrophe or colon in the title
    	final_dir=$(printf %q "/Volumes/Vault/$show_name/")
    	logthis "Copying $source_dir $final_dir"
    	final="cp $source_dir $final_dir"
    	eval $final 	#these eval statements are left over from debugging when I had echo in their place
    	rm -fr "$current_time"
    }
    
    logthis "Working on the new download $TR_DOWNLOADS"
    
    if [ -d "$TR_DOWNLOADS" ]; then
    	IFS=$(echo -en "\n\b") #handles directories or single files
    	for f in $(find "$TR_DOWNLOADS"); do
    		if ! [ -d "$f" ]; then
    			case "$f" in
    			*.mkv | *.mp4 | *.avi | *.m4v ) #do something if it's a video file
            			process_file "$f"
            			;;
    			esac #otherwise never mind
    		fi
    	done;
    	unset IFS
    else
    	process_file "$TR_DOWNLOADS"
    fi
    
  8. To schedule things properly, set up a chronologically scheduled job with the crontab:

    Code: Select all

    MAILTO=""
    10-59/30 * * * * /usr/local/bin/flexget -l /var/log/flexget.log execute --tasks=download-showrss
    The odd syntax in the second line is my way of getting cron to run every half hour on the tens, to give the torrents for shows a few minutes to appear in various feeds.
  9. Remember to make your scripts executable with

    Code: Select all

    chmod +x /usr/local/bin/getoverview.py
    chmod +x /usr/local/bin/conv
The way this is supposed to work, ideally, is that your showRSS feed keeps an eye out for new shows continuously. Flexget pops up at 10 and 40 past the hour to see if there’s anything new. If there is, it sends the magnet link from the RSS feed to the transmission-daemon process running in the background, and checks if there’s anything it needs to clean up from torrents that have already finished downloading. As soon as transmission-daemon is done with a download, it calls the conv script, and passes the $TR_TORRENT_NAME variable to it for its use. The conv script takes care of renaming, transcoding if necessary, adding the metadata, and moving the file to the correct location.


I think this represents pretty much what I was hoping for in the way of an end-to-end workflow. There are still some things I’d like to fix:
  1. As written, I have a directory of square formatted TV artwork for each season and show in my ~/Pictures folder. Ideally, I’d prefer to just access squaredtvart.tumblr.com via the tumblr api directly. I’m still working on that part.
  2. AtomicParsley is the only tool I can find that will let me add the artwork. I can add all the other metadata with ffmpeg, but the addition of the artwork eludes me. It’d be nice not to have to download an additional piece of software which is not under active development to do just one thing, but I don’t see a workaround currently.
  3. I expect that tvnamer will break in October 2017 when theTVdb.com deprecates version 1 of their API in favor of version 2. If I, or someone, manages to work out how to do api calls like this easily, I can readily picture some new python helper functions to do the work of tvnamer. Or perhaps its developers will update it before then.
  4. There’s a few nuisance bugs still in there – I don’t test to see if the destination directory exists yet before copying the file – but I’ll get that going pretty soon.
Post Reply