Unwanted Files Mess

[Rolcol]

Well... when Transmission needs the piece to verify the files you have selected, it allocates the file to place the piece in the appropriate place. Ticket 532 looks like it would fix the problem by not having to create the "container" for the piece in the first place. When the .torrent file is first created, pieces are not created at file boundaries. A single piece can span multiple files and the entire thing needs to be downloaded to assure that the data is correct. Say, for example, that you download a big file within a torrent and the file right after that is some sort of metadata. It's possible that that last file is fully contained in the last piece for the file you wanted. It's going to be created because Transmission needs a container for the piece.

Complaining that the developers are not doing anything to fix the problem is the wrong stance to take. Help contribute to the code pool if it bothers you enough... More important things are being taken care of first.

[Jordan]

Further testing. The problem lies in the leading piece and is repeatable with a multi-file torrent. For the leading piece instead of simply d/ling the partial piece as it does for the trailing piece, Transmission AUTO-SELECTS the entire file (in the Inspector) causing THE FULL FILE SIZE TO BE ALLOCATED altho' only the partial piece is d/led.

On large multi-file torrents, this can waste tens of GBs of disk space. In addition, one cannot at a later time complete these AUTO-SELECTED files.

I will create a trac ticket, but only if devs are actively developing for OS X. RSVP.

Sure I'm a bit upset, but only because there has been tons and tonnes and a whole bunch of complaints about this and nobody does anything. Devs, where to store these .part files is really only a side issue to the AUTO-SELECTION (LEADING PIECE) problem.

What colorful language.

Okay. As you already understand -- but to get other readers up to speed -- what's happening is that a file you've selected has a piece that overlaps with another file that you don't want. BitTorrent clients need to download all of a piece in order to checksum the piece, so Transmission downloads part of that other file.

Here's what you're not seeing: by default, Transmission's fully preallocates wanted files to reduce disk fragmentation, but sparsely preallocates unwanted-but-overlapping files. Because of the sparse preallocation, these unwanted file fragments take up less than one piece's worth of space on disk. Unfortunately this doesn't work for Mac users because -- unlike every other modern operating system -- OS X still ships with a 20th century filesystem that doesn't support sparse files. Why Apple is still using HFS+ instead of adopting ZFS or better is beyond me, but in practical terms it means Transmission will need to adopt something like #532 so that file fragments are done at the application level instead of the filesystem level.

Unless I'm misunderstanding your report, all of this is covered in 514 and 532. If there's something new that I've missed, I'm happy to listen.

[Jordan]

Unfortunately this doesn't work for Mac users because -- unlike every other modern operating system -- OS X still ships with a 20th century filesystem that doesn't support sparse files.

Perhaps, but nevertheless, there's a major problem with this statement. For the trailing partial piece, only the partial piece size is allocated. The problem lies with the leading partial piece. For the leading partial piece ONLY, Transmission does allocate the full file size (1 GB plus in my current example) and then of course only uses <4 MB. So, I'd be looking for something besides Apple and OS X to blame. Why not A: create file and then B: download partial piece and never mind any kind of allocation as appears to already happen for the trailing partial piece.

Sparse files are created by making an empty file, then seeking where in that file the block should be written, then writing the data. On filesystems that support file sparse files, the nonexistent data leading up to that spot will take up 0 bytes on the disk. On those that don't, the nonexistent data leading up to that spot will take up as much disk space as if the data existed anyway.

In the case of the trailing partial piece, that leading-up-to offset is 0 bytes because the piece fragment occurs at the start of the file. Naturally, that 0-byte offset uses 0 bytes HFS+ and ext3 and NTFS.

In the case of the leading partial piece, that leading-up-to offset is (filesize - part-of-a-piece) bytes because the piece fragment occurs at the end of the file. This is where sparse allocation comes in: that space takes up 0 bytes on modern filesystems but unfortunately (filesize - part-of-a-piece) on HFS+ because HFS+ doesn't support sparse files

In practical terms, it means for Mac Transmission users to see this feature, Transmission will have to code file fragments at the app level instead of relying on a modern filesystem to be in place. In other words, something like #532.

[Longinus00]

Why Apple is still using HFS+ instead of adopting ZFS or better is beyond me...

> Apple can currently just take the ZFS CDDL code and incorporate it
> (like they did with DTrace), but it may be that they wanted a "private
> license" from Sun (with appropriate technical support and
> indemnification), and the two entities couldn't come to mutually
> agreeable terms.

I cannot disclose details, but that is the essence of it.

http://mail.opensolaris.org/pipermail/z ... 33125.html

I think the netapp patent infringement lawsuit was still ongoing at the time. Now with the patent lawsuit gutted and oracle at the reins of zfs maybe things will change.

Actually, last I heard apple was hiring FS engineers so maybe they'll do something in house or expand on another project like llvm/clang.

[finny388]

I am using fat32 to be compatible both with Windows and linux.

so I've got the same issue but I agree I'm using an old filesystem

would it be hard to do the auto-delete of unwanted files once the torrent is finished downloading as suggested above?

in the mean time I'm doing it manually

thanks