Transmission deleting directories full of downloaded files?

Ask for help and report issues with the GTK+ version of Transmission
CrunchBangUser
Posts: 8
Joined: Sun May 29, 2016 12:17 am

Transmission deleting directories full of downloaded files?

Postby CrunchBangUser » Sun May 29, 2016 1:43 am

Hello all... this is my first post. I did search for prior posts about the problem I'm experiencing ...I was unable to find a solution, other than removing Transmission completely and re-installing it, which doesn't really explain the problem, or give confidence that it's a meaningful fix.

I've been using Transmission on a few different Linux PCs/distros for a couple years and generally love it. My main torrent PC is an ancient Dell Inspirion netbook w/a lowly Pentium M CPU and less than a half GB of RAM. This PC was rescued from a recycling bin, an external 1TB USB HD was added and the minimalist CrunchBang Linux distro installed (now BusenLabs Linux: https://www.bunsenlabs.org/) and this little netbook has been running Transmission 2.52 (23304) great ...a bigger/faster/hungrier PC doesn't DL/UL torrents any better. Thus, this netbook is the one I leave on 24/7 and it routinely hosts a few hundred torrents at a time, including "longshots" some will probably never finish, but once in a while one does. :-) I realize this isn't the latest version of Transmission, but the CPU limits me to a 32-bit non-PAE OS and this is what got installed with it.

I recently started experiencing the problem where Transmission directories spontaneously disappear (the folder where both the torrent description file and the actual file itself are stored: to stay organized, I put each running torrent in its own folder within a "TRANSMISSION_EXTERNAL" folder (to denote it lives on the external 1TB USB HD). At some point earlier today, inexplicably more than 90% of the directories within the TRANSMISSION_EXTERNAL folder simply vanished. Of course, the listing for each affected torrent in Transmission now displays in red text, and when I look, all I see is the "No data found! Ensure your drives are connected or use 'Set Location. To re-download, remove the torrent and re-add it." error.

Set Location is useless, as is Verify Local Data, since the directory that contained both the torrent description file and the file data appear to be gone (I've checked the trash and the entire external HD w/no luck finding any of the files, so they've probably been mistakenly deleted by Transmission and now unrecoverable). Of more than 500 torrents/directories, only a couple dozen survived the purge ...all happen to have been added recently, though many recently-added ones are among the disappeared. Among the now-missing are 53 torrents that were 100% complete, totaling 55GB of data, now apparently sadly lost forever.

I did find prior posts describing this problem at these links:

viewtopic.php?t=7380

https://bugs.launchpad.net/ubuntu/+sour ... bug/700214

...but as mentioned, there doesn't seem to be much explanation. Thus far, I haven't (yet) noticed this problem on my Ubuntu desktop PC running Transmission, but my confidence in Transmission is shaken... today I lost hundreds of GB of data, much of it older, esoteric public domain stuff I've been patiently DLing over time and happily seeding to others. Foolishly, I didn't have a backup of this in-progress torrent data... the netbook just ran Transmission (it's only task) 24/7 w/o complaint so I got lazy and didn't pause occasionally to make a backup of the whole thing.

Can anyone shed some light on this? I accept that more than 500 torrent description files are now gone, along with 53 completed files and hundreds of other files in various percentages of completion. It's very disheartening, and my confidence in Transmission is not what it was. The similar bug reports linked above date back to 2008/2009 and involved other OSes (Ubuntu and MacOS) so it would seem that this problem has been around for a while and isn't limited to Linux users.

I'd feel a lot better continuing to use Transmission if someone can tell me this is/was a known bug and has been identified and conclusively fixed. It goes w/o saying that I'd love it if there were some way to recover the lost data (I haven't deleted the hundreds of torrent listings now displaying in red) but won't hold my breath on that.

Thanks in advance for any help/info/feedback you can provide. :-)

x190
Posts: 5094
Joined: Sun Nov 30, 2008 4:59 am

Re: Transmission deleting directories full of downloaded files?

Postby x190 » Sun May 29, 2016 4:52 am

? Are you talking about 'Black Magic', and after 2 years? Transmission only deletes what you tell it to delete. Did you check your external drive mount point? If it became unmounted (which appears likely - "No data found! Ensure your drives are connected or use 'Set Location. To re-download, remove the torrent and re-add it."), then you may find a similarly named volume at your mount point. Unless you moved or deleted it or you directed Transmission to move or delete it, your data will be exactly where it always was.
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

CrunchBangUser
Posts: 8
Joined: Sun May 29, 2016 12:17 am

Re: Transmission deleting directories full of downloaded files?

Postby CrunchBangUser » Sun May 29, 2016 4:37 pm

Not sure what you mean by 'Black Magic.'

As mentioned, I keep my transmission files (both torrent pointers and the file data they eventually download and upload) in a directory called TRANSMISSION_EXTERNAL on an external 1TB USB hard drive... it's the only external hard drive on the netbook. The drive has not been unmounted, nor did any part of the path to the torrent directories change at all... I'm familiar with getting the "No data found..." error ***IF*** a directory is accidentally renamed or moved of course, but that's NOT what's happening here.

What DID happen is that there WERE a few hundred directories containing torrent pointers and file data inside the TRANSMISSION_EXTERNAL directory a couple days ago and everything was working fine, as it had been for years. Then yesterday, for some UNKNOWN (to me) REASON, there are only about 30 directories left inside the TRANSMISSION_EXTERNAL directory, resulting in hundreds of torrents now displaying in red with the "No data found..." error.

I did search the whole drive looking to see if I could find any of the missing directories/files and came up with nothing: hundreds of directories, torrent pointers and data files in various stages of completion appear to have vanished. I'm hoping someone can explain this to me. Thanks.

x190
Posts: 5094
Joined: Sun Nov 30, 2008 4:59 am

Re: Transmission deleting directories full of downloaded files?

Postby x190 » Sun May 29, 2016 6:46 pm

#1 Despite your protest, your drive did apparently become unmounted at some point, at least temporarily.

#2 I have asked you to find and go to the mount point for that drive and look for a directory named the same as your 'external 1TB USB hard drive' with perhaps an '_1' appended. You may find your missing files there.

and the minimalist CrunchBang Linux distro installed (now BusenLabs Linux: https://www.bunsenlabs.org/) and this little netbook has been running Transmission 2.52 (23304) great ...a bigger/faster/hungrier PC doesn't DL/UL torrents any better. Thus, this netbook is the one I leave on 24/7 and it routinely hosts a few hundred torrents at a time, including "longshots" some will probably never finish, but once in a while one does. :-) I realize this isn't the latest version of Transmission, but the CPU limits me to a 32-bit non-PAE OS and this is what got installed with it.

You may need to check with the package maintainer. Source code is open source.

CrunchBangUser, when did you last take a close look at your (loaded - see your Message Log) settings.json file?

https://trac.transmissionbt.com/wiki/Ed ... gFiles#RPC
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

CrunchBangUser
Posts: 8
Joined: Sun May 29, 2016 12:17 am

Re: Transmission deleting directories full of downloaded files?

Postby CrunchBangUser » Mon May 30, 2016 3:41 pm

My only protest was in regards to magic being involved. ;-)

Per your suggestion, I did check the mount point for the 1TB external drive and there's only that one drive directory there (the drive/mount directory name is a string of letters/numbers). Sadly, there isn't another directory there with a '_1' appended to the end. As mentioned, I did search the entire drive for some of the missing files w/o luck. Of course, I also viewed the TRANSMISSION_EXTERNAL directory with 'view hidden files' activated, but the missing dirs were nowhere to be seen. Today I'll briefly mount the drive on a Mac and on a Windows PC to see if anything looks different (won't hold my breath).

I'm not quite clear on what you'd have me check/do re: the package maintainer or settings.json file. Though I've been using PCs and the net since the late 70s (BBSes and Usenet), I'm a relatively recent convert to Linux and haven't (had the need yet) to get into the guts of it, since it usually just runs fine. I use Ubuntu and/or Mint on my other/main PCs... this netbook running Crunchbang is left alone to seed for weeks at a stretch without me touching it, so I'm less familiar with tweaking CrunchBang (I was just happy to find a contemporary distro available as 32-bit non-PAE so I could use this old netbook). I can tell you that one day there were hundreds of sub-directories (one for each torrent in-progress) in the TRANSMISSION_EXTERNAL folder and a day later, there were only about 30 sub-directories there. I did not touch the netbook in between, nor did it get power cycled... at most, it might have lost its net connection briefly which happens from time to time.

Seeing that something similar may have happened to others using Transmission, I thought maybe it was a known thing by now. I'm both happy and grateful in advance to follow any instructions which might provide clues, but ask please that you don't assume I've got a whole lot of Linux smarts since I've not yet gathered much. Thanks! :-)

Added later: When mounted on a Mac or Windows PC, the situation is the same: only 30 directories (each for a separate torrent) where there were/should be hundreds.

HOWEVER, I would like to point out that there's about 200GB of space unaccounted for on the 1TB drive: it says 748GB is available for use, and the CURRENT (30 torrents left) contents of the TRANSMISSION_EXTERNAL directory are only 50GB in size... leaving a 200GB gap between what's being used and what's available (there are no other dirs/files on the drive of any size... just a few small things in the trash dir, etc...). Perhaps just wishful thinking on my part, but I can't help but wonder if that 200GB is the shadow left behind by the hundreds of missing torrents... either they're still somehow in the drive's filesystem (but unfindable, at least w/my modest Linux skills) or have been deleted but the space they took hasn't been released or something like that.

PS: To avoid any unnecessary writing to the drive, I've stopped using it/the netbook for P2P of course.

x190
Posts: 5094
Joined: Sun Nov 30, 2008 4:59 am

Re: Transmission deleting directories full of downloaded files?

Postby x190 » Mon May 30, 2016 5:04 pm

I'm not quite clear on what you'd have me check/do re: the package maintainer or settings.json file. Though I've been using PCs and the net since the late 70s (BBSes and Usenet), I'm a relatively recent convert to Linux and haven't (had the need yet) to get into the guts of it, since it usually just runs fine.

I think what you're saying is that we're both getting too damn old, however, we both speak English, so let's press on. Can we get off this irrelevant 'blame Transmission' stuff and just stick to the facts?

I'm really getting concerned about your apparent lack of knowledge about Transmission's RPC feature, because, if you do not have that feature properly secured, anybody with your IP address (which is everyone you share data with) can load the Web Interface in their browser and delete torrents in your list, including the data, at will. RPC = Remote Program Control and must be secured by password and/or (preferably 'and') whitelisting of specific addresses. Normally, only 127.0.0.1 is whitelisted, but either you, or your package supplier, could have changed those settings.

Please find that settings.json file I mentioned and check your settings.

For the GTK+ client, all settings are kept in $HOME/.config/transmission
https://trac.transmissionbt.com/wiki/Ed ... gFiles#RPC

As for your external drive, I would suggest taking it to a computer shop. They may well be able to recover your missing data, at which point, a drive reformatting might be in order.
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

CrunchBangUser
Posts: 8
Joined: Sun May 29, 2016 12:17 am

Re: Transmission deleting directories full of downloaded files?

Postby CrunchBangUser » Mon May 30, 2016 6:47 pm

You're right: I never heard of Transmission's Remote Program Control function until just now when you mentioned it. As a matter of course, I don't necessarily make it a point to review/understand every available feature in a program or operating system before I start using it ...if that were the case, I'd still be using Wordstar on an Osborne 1. ;) Assuming that the "Web" tab in Transmission's preferences *is* for RPC, it looks secured in that it's *not* enabled (web access) and the only IP addy that's allowed is 127.0.0.1. Those are all default settings untouched by me... as expected, the defaults wouldn't leave Transmission open to hijacking by third parties.

I copied the settings.json file, editing out only some filenames and the RPC password. Here it is for your review:

"alt-speed-down": 50,
"alt-speed-enabled": false,
"alt-speed-time-begin": 540,
"alt-speed-time-day": 127,
"alt-speed-time-enabled": false,
"alt-speed-time-end": 1020,
"alt-speed-up": 50,
"bind-address-ipv4": "0.0.0.0",
"bind-address-ipv6": "::",
"blocklist-enabled": false,
"blocklist-updates-enabled": true,
"blocklist-url": "http://www.example.com/blocklist",
"cache-size-mb": 4,
"compact-view": true,
"dht-enabled": true,
"download-dir": "/media/24B816F4560311B9/TRANSMISSION_EXTERNAL",
"download-queue-enabled": true,
"download-queue-size": 700,
"encryption": 1,
"idle-seeding-limit": 30,
"idle-seeding-limit-enabled": false,
"incomplete-dir": "/home/s/downloads",
"incomplete-dir-enabled": false,
"inhibit-desktop-hibernation": true,
"lpd-enabled": true,
"main-window-height": 703,
"main-window-is-maximized": 0,
"main-window-width": 615,
"main-window-x": 407,
"main-window-y": 39,
"message-level": 2,
"open-dialog-dir": "/media/24B816F4560311B9/TRANSMISSION_EXTERNAL",
"peer-congestion-algorithm": "",
"peer-limit-global": 999,
"peer-limit-per-torrent": 100,
"peer-port": 51413,
"peer-port-random-high": 65535,
"peer-port-random-low": 49152,
"peer-port-random-on-start": false,
"peer-socket-tos": "default",
"pex-enabled": true,
"port-forwarding-enabled": true,
"preallocation": 1,
"prefetch-enabled": 1,
"queue-stalled-enabled": true,
"queue-stalled-minutes": 30,
"ratio-limit": 2,
"ratio-limit-enabled": false,
"recent-download-dir-1": "/media/24B816F4560311B9/TRANSMISSION_EXTERNAL/FILENAME1”,
"recent-download-dir-2": "/media/24B816F4560311B9/TRANSMISSION_EXTERNAL/FILENAME2”,
"recent-download-dir-3": "/media/24B816F4560311B9/TRANSMISSION_EXTERNAL/FILENAME3”,
"recent-download-dir-4": "/media/24B816F4560311B9/TRANSMISSION_EXTERNAL/FILENAME4",
"rename-partial-files": true,
"rpc-authentication-required": false,
"rpc-bind-address": "0.0.0.0",
"rpc-enabled": false,
"rpc-password": "PASSWORD_EDITED_OUT",
"rpc-port": 9091,
"rpc-url": "/transmission/",
"rpc-username": "",
"rpc-whitelist": "127.0.0.1",
"rpc-whitelist-enabled": true,
"scrape-paused-torrents-enabled": true,
"script-torrent-done-enabled": false,
"script-torrent-done-filename": "/home/s",
"seed-queue-enabled": false,
"seed-queue-size": 10,
"show-backup-trackers": false,
"show-extra-peer-details": false,
"show-filterbar": true,
"show-notification-area-icon": true,
"show-options-window": true,
"show-statusbar": true,
"show-toolbar": true,
"show-tracker-scrapes": false,
"sort-mode": "sort-by-progress",
"sort-reversed": false,
"speed-limit-down": 100,
"speed-limit-down-enabled": false,
"speed-limit-up": 1,
"speed-limit-up-enabled": false,
"start-added-torrents": false,
"statusbar-stats": "total-ratio",
"torrent-added-notification-enabled": true,
"torrent-complete-notification-enabled": true,
"torrent-complete-sound-command": "canberra-gtk-play -i complete-download -d
'transmission torrent downloaded'",
"torrent-complete-sound-enabled": true,
"trash-can-enabled": true,
"trash-original-torrent-files": false,
"umask": 18,
"upload-slots-per-torrent": 14,
"user-has-given-informed-consent": true,
"utp-enabled": true,
"watch-dir": "/home/s/downloads",
"watch-dir-enabled": false

Given my lack of experience with Transmission, I don't want to over-impose on your time and will probably just cut my losses and chalk up the hundreds of now-vanished torrents to bad luck. I suppose that since they're still listed in Transmission's active torrent list (though displayed in red/not working) I can go find them again and start over, perhaps on my Ubuntu PC... I've never noticed Transmission do anything odd like this there (so far) though one of the two similar cases of suddenly/randomly disappearing directories/files that I linked to in my original post did happen to be on a PC running Transmission under Ubuntu.

x190
Posts: 5094
Joined: Sun Nov 30, 2008 4:59 am

Re: Transmission deleting directories full of downloaded files?

Postby x190 » Mon May 30, 2016 7:02 pm

I'm willing to bet your files are all intact on your external drive, and in the trash, probably along with with everything else you thought you deleted since acquiring the drive, hence the 200+GB missing space. Just search for its location using your favorite search engine (or ask the manufacturer/supplier). Here's some links to get you started.

http://ubuntuforums.org/showthread.php?t=1122670

"sudo find / -type d -name *Trash*" (for searching your external drive - change "/" to <fullpathtoexternaldrive>).

http://ubuntuforums.org/showthread.php?t=898573

Your RPC settings appear to be okay, but please do find the GTK "Message Log" to help inform yourself in the future (I use a Mac, so can't be specific here).*

* With GTK+ client, select Help -> Message Log from main menu. On start-up this log will show which settings.json file has been loaded, allowing you to confirm your settings.

"peer-limit-global": 999, (too high - reduce to below 800)
"blocklist-enabled": false, (you should use, at least, a basic level 1 list)
"peer-port-random-on-start": false, (good idea to use 'true' here)
Last edited by x190 on Mon May 30, 2016 7:40 pm, edited 1 time in total.
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

CrunchBangUser
Posts: 8
Joined: Sun May 29, 2016 12:17 am

Re: Transmission deleting directories full of downloaded files?

Postby CrunchBangUser » Mon May 30, 2016 7:19 pm

As I was able to find those settings in preferences, I've changed the settings for max peers, random port at start and blocklist enabled per your instructions on both the netbook and my other/main Ubuntu PC... thanks.

I'll move the drive over to my other PC and see if I can somehow locate that missing 200GB worth of stuff, maybe with some file utility or something (years ago when I was a longtime Mac guy myself, I recall using something called Alsoft Disk Warrior and was amazed at the recovered data... perhaps I can find something equivalent/contemporary). Thanks. :)

x190
Posts: 5094
Joined: Sun Nov 30, 2008 4:59 am

Re: Transmission deleting directories full of downloaded files?

Postby x190 » Mon May 30, 2016 7:43 pm

My last post has been edited.

Alsoft Disk Warrior and was amazed at the recovered data... perhaps I can find something equivalent/contemporary). Thanks.

Or just learn how to access the trash on that drive (which is likely a 'hidden directory'). For your internal drive, it would be likely '/home/username/.local/share/Trash', and could be something similar on your external. Maybe search according to what filesystem it uses.
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

CrunchBangUser
Posts: 8
Joined: Sun May 29, 2016 12:17 am

Re: Transmission deleting directories full of downloaded files?

Postby CrunchBangUser » Tue May 31, 2016 12:25 am

Searching through the trash on the 1TB drive is among the first things I did: there's a hidden directory called ".Trash-1000" and within it, three subdirectories called:

"expunged" (which is empty)
"files" (which contains a half-dozen recently deleted items, but not the hundreds of missing torrent sub-directories)
"info" (which contains a half dozen little text/info files... one for each deleted item currently in "files").

Dredging up a bit from my time w/Unix boxes in the mid-90s, I did open a terminal window and ls -al the 1TB drive just to be sure that the Linux GUI *was* showing me all the hidden files (it seems to be). Up at the same (top level) directory of the 1TB drive, there's also a hidden directory called "$RECYCLE.BIN" (which contains a dir that contains a 129 byte text file called "desktop.ini" and another called "System Volume Information" which contains a 76 byte file (of indeterminate type) called "IndexerVolumeGuid." Everything else visible on the drive was made by me: empty dirs called "Tors to load" and "Tors finished", and of course the "TRANSMISSION_EXTERNAL" directory that Transmission uses for torrents and files.

I freely stipulate that this problem of disappearing directories (and the torrents/files contained within) is *not* necessarily the fault of Transmission. It's certainly possible that the operating system itself had some sort of hiccup which moved or deleted those directories while Transmission was using them, after which Transmission rightly reported errors ("no data found"). The fact that similar things have happened to others using Transmission (with other OSes) could be anecdotal.

I'll google around tomorrow to see what I can do within Linux to try to find/see/recover whatever's taking up that 200GB of space on the drive. Thanks for your help x190... even if the files are never recovered and I don't grok why they disappeared, this has still been educational.

x190
Posts: 5094
Joined: Sun Nov 30, 2008 4:59 am

Re: Transmission deleting directories full of downloaded files?

Postby x190 » Tue May 31, 2016 2:42 am

Search terms: trash file +external drive +linux

As admin:
sudo ls -la /media/24B816F4560311B9/.Trashes

Ubuntuforum links above show how to get more info on this issue (du etc.).
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

CrunchBangUser
Posts: 8
Joined: Sun May 29, 2016 12:17 am

Re: Transmission deleting directories full of downloaded files?

Postby CrunchBangUser » Fri Jun 10, 2016 5:22 pm

Still working on this... will report back when there's news. Thanks. :)

x190
Posts: 5094
Joined: Sun Nov 30, 2008 4:59 am

Re: Transmission deleting directories full of downloaded files?

Postby x190 » Fri Jun 10, 2016 6:57 pm

CrunchBangUser wrote:Still working on this... will report back when there's news. Thanks. :)

Please do! Your info may be very helpful to others.
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

CrunchBangUser
Posts: 8
Joined: Sun May 29, 2016 12:17 am

Re: Transmission deleting directories full of downloaded files?

Postby CrunchBangUser » Mon Jul 11, 2016 9:22 pm

Sorry for the delayed follow-up... been offline for a bit.

Still no luck finding/recovering the lost files/folders. Someone elsewhere suggested that plugging the drive into any late-model Windows machine would probably prompt an automatic error recovery process, so I put the drive on a Win10 laptop (pretty sure I tried that before IIRC) and no such process happened/was suggested.

While on the Win10 laptop, I did re-confirm that the TRANSMISSION_EXTERNAL folder (where the torrents and torrent data are supposed to be) and the .Trash-1000 folder (which only contains recently/purposely deleted items; not yet emptied) together take up a total of 11GB on the 1TB drive. Asking for properties on the whole drive itself reports that 239GB are in use... so, more than 200GB of space is unaccounted for (the missing data I'm guessing).

I still haven't used/messed with the drive in an effort to avoid corrupting the missing data and continue to be open to advice on how I might recover the data. I have plugged the drive into Win/Mac/Linux PCs ...none enable me to see the missing data. I was hoping maybe to find an Ubuntu Linux utility that might help, but have come up empty thus far.

Added later: well, I sort of ran out of patience (and have other, more vital stuff needing my time) so I put the drive back on the netbook and re-started Transmission, even though only about 30 of the hundreds of partially-completed torrents will run now. I know there's a chance that re-starting downloads may overwrite some of the lost data, but I figure completing some of the torrents is better than completing none, and some of the remaining/working 30 are quite rare, so they're worth seeding for as long as possible.

If anyone can shed some light on how to try to recover the lost data, I'D STILL LOVE TO HEAR ABOUT IT! Thanks!


Return to “GTK+ Support”