Corrupt files, did not know and verifying

[SandJ]

I realised a file I had downloaded some time ago still had an upload ratio of 0.0 and I was trying to work out why. At one point I looked in Properties and saw I only had 80-something% of the file available despite it saying 100% on the front screen. So I verified the file it and the proportion available went up. Verified it again and got the popup box saying it was downloaded.

So I checked the properties of the other files I had in Transmission and find of the 23 files I have, another 5 were in the same state: marked as 100% but the properties said I did not have 100%. And I am referring to the "Have:" information, not the "Downloaded:" information. Verifying them resulted in the last few % of errors (or whatever the wording was) disappearing and a corresponding "Torrent Complete" popup.

The only thing I use torrent for are Linux distribution .iso files. Among those with errors were one from April 2012 and, most annoyingly, one from April 2010. I have been cutting disks and bootable USB sticks from these ISOs which Transmission said were 100% complete and getting poor success at booting from them. I have wondered why some distros don't work when they should; I had come to the conclusion Linux distros are just generally unreliable. Never getting the 64 bit version of Ubuntu 10.04 working was a particular annoyance and I recently struggled with a 64-bit Linux Mint for some ages. I think I may have wasted many hours - certainly evenings and probably entire weekends - as a consequence of this.

What is the point of Transmission reporting a file is downloaded and at 100% when parts of it are corrupt or missing? Is this a failing of the torrent concept or of Transmission or something else?



BTW, the Captcha for registration is a right royal pain in the arse. Do you really need so many PC / food items in that list? And having the too-many-logins automatically enabled is also annoying. And Cupertino? Is that even real? Ha ha bloody ha. How about MugOfEspresso or BaconSarnieTown? You're supposed to make it hard for bots, not for people who are registering for support and are already annoyed.

[SandJ]

BTW, it is Transmission 2.82 (14160) on Xubuntu 13.04 and these files were downloaded between 04/11/2012 and the last few weeks. When the files were downloaded this PC may have been running Xubuntu 12.04, 13.04 or 13.10 according to whatever state of upgrade it was at at the time.

[SandJ]

Here is a patch that you could try: https://trac.transmissionbt.com/ticket/4649#comment:39

Ah. That made interesting reading. So it's not just me then.

Also, don't linux isos have checksums available that can be tested?

They do, but I had assumed (yes, I am old enough to know better) I had assumed that a piece of software written for downloading stuff in pieces and stitching them together might have been relied upon to do just that. Especially since it is provided as the default BitTorrent client in at least one distro. I intentionally started using Transmission specifically for Linux .isos in the belief that it was a superior and reliable alternative to FTP.

I don't suppose this will go down very well, but I should like to express my disappointment that it should report 100% in the compact view and transfer a file to the Complete folder, when said file is not complete.

[SandJ]

So I checked the properties of the other files I had in Transmission and find of the 23 files I have, another 5 were in the same state: marked as 100% but the properties said I did not have 100%.

Can you elaborate on this? Are these multifile torrents with 1 or more files (originally) unchecked?

When I upgrade on this machine, I copy the partition and upgrade the copy. The old partition gets left until I need the space. That means I can do a binary file compare between the downloaded files from back then with now.

Comparing the ~/Downloads/Torrents/Complete directory in my current Xubuntu 13.10 partition with the still existing Xubuntu 12.04 partition, I can see some differences in some of the files. Bearing in mind the Xubuntu 13.10 files have all been through the verify process and been amended:

~/Downloads/Torrents/Complete/kubuntu-13.04-desktop-amd64.iso
From 0A3C0000 to 0A414FFF is all zeroes in the older file.
From 16480000 to 164D4FFF is all zeroes in the older file.
From 25080000 to 251EEFFF is all zeroes in the older file.
From 39900000 to 39A43FFF is all zeroes in the older file.
From 39F00000 to 39F23AAF is all zeroes in the older file.
After that it gets messy as the newer file is mostly zeroes anyway with occasional strings in it, whereas the older file is all zeroes, but we are right near the end of the file anyway.

~/Downloads/Torrents/Complete/linuxmint-14.1-mate-dvd-64bit.iso
From 10D00000 to 11FFEFFF is all zeroes in the older file.
From 16D80000 to 16FFEFFF is all zeroes in the older file.
From 2B4C0000 to 2BF8BFFF is all zeroes in the older file.
Then match up to the end at 3C7FFFFF

~/Downloads/Torrents/Complete/ubuntu-12.04.2-alternate-amd64.iso
From 17F00000 to 17FFEFFF is all zeroes in the older file.
Then match up to the end at 2C0FFFFF

~/Downloads/Torrents/Complete/ubuntu-13.04-desktop-amd64.iso
From 06280000 to 06E7CFFF is all zeroes in the older file.
From 12380000 to 13FFEFFF is all zeroes in the older file.
From 1E280000 to 1E2B6FFF is all zeroes in the older file.
From 29580000 to 297FEFFF is all zeroes in the older file.
Then match up to the end at 310FFFFF

(Blimey. That's enough. That was boring.)

What I can see is that:
- two .iso files downloaded before 18/02/2013 12:19:40 were OK
- seven .iso files downloaded between 18/02/2013 12:23:11 and 07/08/2013 11:01:19 are all partly blocks of zeroes
- five .iso files downloaded since 10/10/2013 12:53:13 are OK.

However, when I ran the verify yesterday, I can see there are another 6 .iso files I have downloaded since October 2013 that were all fixed by the verify because they have a time stamp for that period of 3 hours when I ran the verify process. Those files were downloaded since the start of March 2014.

In the morning I'll boot up into the old Xubuntu 12.04 partition and see what Transmission says about the state of the files.

[SandJ]

Are you trying to download isos which may be modifying pre-existing files and while perhaps also seeding from another torrent?

Unless I misunderstand you, I don't think so. I download an iso and then it just sits there indefinitely. I think twice I've gone to download an iso I already have and I just get a message like "Torrent already exists" and nothing is downloaded.

[SandJ]

In the morning I'll boot up into the old Xubuntu 12.04 partition and see what Transmission says about the state of the files.

It was showing files as 100%. Here is one of them:

Properties of an iso in X12.04:
Properties of same iso in X13.10 (which has been verified and fixed, hence extra content has been downloaded):

[SandJ]

How do you know for sure the file had undiscovered bad pieces immediately after download to 100%, and while still on the original partition and at the original location?

It is still in its original 12.04 partition and location. I copied the partition it is on to create a newer partition to upgrade to 13.04. The same blocks of zeroes exist on the 13.04 partition that I copied to make the current 13.10 partition, which is the one where I ran the verify.

When you tried to re-verify the file, did you do so using the original torrent file? Did a .resume file come into play? Even changed mtimes could be a factor.

I'll need to test that on the 12.04 partition.

If you feel some form of trac ticket #4649 is a good idea as a user option, then please add your voice there.

Since I started on this exercise because of a file I had downloaded within the last few weeks which, when verified, downloaded more bits, then I reckon the problem occurred for me in 2012, 2013 and in 2014 on 3 different partitions (yes the same disk, but I've had no other signs of any disk corruption such as not in my 1.8 Gb email system which gets written to daily and has also been dragged from partition to partition). I reckon this problem has existed for some time and still happening and there seems to be one common factor.

What I will do one night is set up shedloads of downloads and see what happens.

I'll add to the trac system, thank you for your help.