best way to configure storage multi tier

Ask for help and report issues not specific to either the Mac OS X or GTK+ versions of Transmission
Post Reply
nackstein
Posts: 2
Joined: Wed Dec 01, 2021 11:09 am

best way to configure storage multi tier

Post by nackstein »

hello everyone,
I just killed a 11 month old HDD by uploading 57 TB of torrents... I suspect the workload was too much for a consumer HDD (external 2.5").
I'm going to upgrade the setup and use a 2 tier storage + backup:
tier 1: 2 TB SSD for most accessed torrent
tier 2: 5 TB HDD for least accessed torrent
backup: 5 TB HDD

for the backup I just schedule an rsync and shutdown the USB HUB port to put the drive offline. no problem here.
What I ask your support for is what is the best way to implement a 2 tier storage. Some Ideas I came up till now:

2 transmission-daemon instances:
the first instance on port 9091 will ssed exclusively on SSD and will have about 5000 kB/s upload limit
the second instance on port 9092 will seed exclusively from HDD and will have about 500 kB/s upload limit
I will prepare a script to move torrent data from SSD to HDD, add the torrent to second instance, remove the torrent from first instance.
should work but seems a little overcomplicated to manage.

1 transmission-daemon instance:
I will ssed from SSD and HDD and will move to HDD those torrent that are less required. the problem is that if I wrongly select the torrent to move
or situation changes, I could end up with high traffic on HDD, I can't put a limit on bandwidth for torrents on HDD.
I could prepare a script to cap every single torrent on HDD but seems unfair, for example 10 kB/s for each torrent on HDD.


for both solution I'm asking if you have some formula that can use stastistics from the command transmission-remote -t xxx -i
something like: (uploaded GB) / (seeding time) / (time from latest activity)
put everything in order by the above formula and move the latest accessed torrents when space on SSD is scarse.

the above formula is far from optimal because 1 uploaded piece can take to 0 the time from latest activity.
ideally I would need to record activity distribution and keep track of traffic day by day and than make some calculus on that data but so far
it's not available.

any idea is really appreciated
nackstein
killemov
Posts: 533
Joined: Sat Jul 31, 2010 5:04 pm

Re: best way to configure storage multi tier

Post by killemov »

Use a single instance. Guesstimate which torrents are most popular to start with and put them on the SSD and put the rest on your HDD. You can then use the relocate feature within transmission OR use symbolic linking for the slow content. Let the the upload run for a couple of days and reassess the popularity of your torrents and relocate (or copy) accordingly. Then increase the control interval to a couple of weeks and then months. At some point all torrents do seem to peter out and sometimes they experience a massive uptick. Popular season torrents of a TV show when a new season is announced or about to start for example.

Oh, and just using the number of uploaded bytes as your metric should be fine. But you have to keep the historic data for that yourself. You could create a script to query the RPC to generate a listing of all upload stats and dump the resulting json file in a database that can handle that. (PostgreSQL for example.)
nackstein
Posts: 2
Joined: Wed Dec 01, 2021 11:09 am

Re: best way to configure storage multi tier

Post by nackstein »

thanks for your feedback. This help me pondering things.
Post Reply