File allocation problem when doing selective download

Ask for help and report issues with the Windows version of Transmission
kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

File allocation problem when doing selective download

Postby kIssGlaY » Tue Apr 05, 2016 1:55 am

Assume the torrent contain file: 1, 2, 3, 4.
If only download file 3, and pieces cross boundary of file 3 to file 2 and 4, then the file 2 will be fully allocated rather then a sparse file, but the file 4 as a single piece. Maybe this's because the block getting allocated is at the end of file 2 but header of file 4. But if the file 2 is large enough, like above 20G, transmission would freeze for a long time until the file is fully allocated.
Donno if it's a bug or not.
Last edited by kIssGlaY on Tue Apr 05, 2016 5:01 am, edited 1 time in total.

x190
Posts: 5095
Joined: Sun Nov 30, 2008 4:59 am

Re: File allocation problem when doing selective download

Postby x190 » Tue Apr 05, 2016 4:12 am

Take a look at your settings.json file.

https://trac.transmissionbt.com/wiki/EditConfigFiles

Your described behavior is same as OS X (and normal), but *might* be adjustable on Windows.

https://trac.transmissionbt.com/ticket/532
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

Re: File allocation problem when doing selective download

Postby kIssGlaY » Tue Apr 05, 2016 4:54 am

"preallocation" is set to 1.
Only the unselected file right before the selected one will be fully allocated rather than a sparse, all other files(the selected ones) are allocated as expected as sparse.
So if want to get rid of the client freeze for this situation, the workaround is: if want to download only file 3 from 1, 2, 3, 4, you should select 1, 2, 3 and start, wait for 1, 2, 3 to be created (as sparse of course), and then deselect 1 and 2.
Maybe this is the same behavior as on other platforms.

x190
Posts: 5095
Joined: Sun Nov 30, 2008 4:59 am

Re: File allocation problem when doing selective download

Postby x190 » Tue Apr 05, 2016 2:16 pm

@mike.dld: Care to comment here? Even though, afaik, OS X doesn't do sparse files, the before and after behavior is the same, although not the rest of what kIssGlaY seems to be saying.
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

mike.dld
Transmission Developer
Posts: 249
Joined: Wed Dec 25, 2013 10:56 pm

Re: File allocation problem when doing selective download

Postby mike.dld » Tue Apr 05, 2016 2:39 pm

I'll need to take a look later. Theoretically, there should be no difference in how files are deselected and space should be allocated sparsely or in full depending on preallocation setting. For now, the question is how do you check that space for file was allocated in full? File properties dialog I presume (which has "Size:" and "Size on disk:" fields)?

kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

Re: File allocation problem when doing selective download

Postby kIssGlaY » Tue Apr 05, 2016 3:25 pm

mike.dld wrote:I'll need to take a look later. Theoretically, there should be no difference in how files are deselected and space should be allocated sparsely or in full depending on preallocation setting. For now, the question is how do you check that space for file was allocated in full? File properties dialog I presume (which has "Size:" and "Size on disk:" fields)?

Yes, this is one thing I checked, the "Size on disk" of that ".part" file is equal to the actual size of the file.
What's more important is the client hang/freeze (remote client and webui have no response, no uploading / downloading) writing this file and from system monitor tool, observed the disk activity is all of writing that file at full rate of the hard disk, and once done writing, remote client and webui is back to normal. This is exactly the same behavior as doing a full preallocation (preallocation set to 2). For example, an unselected file of 30G would freeze the client for approx. 10 minutes.
BTW, the unofficial transmission-qt also got this issue.

x190 wrote:@mike.dld: Care to comment here? Even though, afaik, OS X doesn't do sparse files, the before and after behavior is the same, although not the rest of what kIssGlaY seems to be saying.

The unselected file right after the selected one is also NOT a sparse file, but the size of this file(for both "size" and "size on disk") is only of the size of the piece block (couple MiB or KiB).
But as for the unselected file right before the selected one, it is the size of the whole file, not just of the piece block.

x190
Posts: 5095
Joined: Sun Nov 30, 2008 4:59 am

Re: File allocation problem when doing selective download

Postby x190 » Wed Apr 06, 2016 1:17 am

The unselected file right after the selected one is also NOT a sparse file, but the size of this file(for both "size" and "size on disk") is only of the size of the piece block (couple MiB or KiB).
But as for the unselected file right before the selected one, it is the size of the whole file, not just of the piece block.
Top

Yes, precisely what OS X does, but it never did support sparse files. Maybe Transmission is missing some file checkout voodoo? Did you ever ask rberber (rb07) about this?

http://www.codeproject.com/Articles/530 ... on-Windows

...or maybe it's all just an illusion?

http://www.flexhex.com/docs/articles/sp ... phtml#msdn
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

Re: File allocation problem when doing selective download

Postby kIssGlaY » Wed Apr 06, 2016 2:25 pm

The freeze of the client when writing these files is exactly as same as on the cygwin build of transmission daemon. Cygwin build doesn't support sparse file at all, so every single file created freeze the webui and remote client, and the time of freezing depends on file size, and this's a known issue on these builds.

The official build only freeze under situation I mentioned, and these files are definitely non-sparse. Because the block to be written is exactly the at the end of the unselected file, not middle or anywhere else, maybe windows does this ignoring the flag, if there's really no neglect in programming itself.

cfpp2p
Posts: 281
Joined: Sat Aug 08, 2009 3:14 pm

Re: File allocation problem when doing selective download

Postby cfpp2p » Thu Apr 07, 2016 5:14 pm

Cygwin build doesn't support sparse file at all, so every single file created freeze the webui and remote client, and the time of freezing depends on file size, and this's a known issue on these builds.

There is a way of patching a cygwin transmission build and that patching, combined with patch of https://trac.transmissionbt.com/ticket/532, such that with a transmission cygwin build the freeze can be minimized to practically nil for most torrents.
peace doesn't come without a price...
there is nothing truer on Earth or of the Universe

mike.dld
Transmission Developer
Posts: 249
Joined: Wed Dec 25, 2013 10:56 pm

Re: File allocation problem when doing selective download

Postby mike.dld » Fri Apr 08, 2016 8:23 am

inout.c, line ~93:

Code: Select all

          const int prealloc = file->dnd || !doWrite
                             ? TR_PREALLOCATE_NONE
                             : tor->session->preallocationMode;

The logic traces back to the very introduction of preallocation in r7051...

x190
Posts: 5095
Joined: Sun Nov 30, 2008 4:59 am

Re: File allocation problem when doing selective download

Postby x190 » Fri Apr 08, 2016 3:34 pm

Please see file-win32.c. Since FSCTL_SET_ZERO_DATA does not appear to be set after FSCTL_SET_SPARSE, then would that be the source of this problem?

An application may decide to make a normal file as Sparse. This is allowed, but then, it's also the responsibility of the application to ensure that it scans the files for regions of zero that it needs to explicitly mark as Sparse.

When FSCTL_SET_SPARSE is set, we already have a handle to the file. Now should we not just use FSCTL_SET_ZERO_DATA to include the whole file (uint64_t size)?

Sparse regions are marked using the FSCTL command code FSCTL_SET_ZERO_DATA. This code basically tells the file system to mark the specified range as zeroes, which internally is just stored as a range within the file system meta-data and does not explicitly allocate any space for it.


Otherwise, you are left with the whims of the OS.
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

mike.dld
Transmission Developer
Posts: 249
Joined: Wed Dec 25, 2013 10:56 pm

Re: File allocation problem when doing selective download

Postby mike.dld » Fri Apr 08, 2016 4:23 pm

I was pointing to the fact that DND files aren't created as sparse in the first place. IIUC, FSCTL_SET_ZERO_DATA is to be used when space used by existing sparse file needs to be deallocated.

x190
Posts: 5095
Joined: Sun Nov 30, 2008 4:59 am

Re: File allocation problem when doing selective download

Postby x190 » Fri Apr 08, 2016 10:34 pm

mike.dld wrote:I was pointing to the fact that DND files aren't created as sparse in the first place.

Okay, but shouldn't they be (when they are required for these end pieces), if that is the setting (sparse)?
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

Re: File allocation problem when doing selective download

Postby kIssGlaY » Wed Apr 11, 2018 3:29 pm

This issue is still there in 2.93.

BTW, this issue was fixed 2 years ago in the discontinued unofficial Transmission-QT after this thread was posted.


Return to “Windows Support”