preallocation setting problem on cygwin

Ask for help and report issues with the Windows version of Transmission
Post Reply
kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

preallocation setting problem on cygwin

Post by kIssGlaY »

Built on cygwin, no matter what value is assigned to preallocation (0, 1), transmission always does a full preallocation, which is the value 2. If there were a file in torrent having size up to 30G, transmission would hang lots of time to allocate the file.

Is there a way to let it use sparse file under windows? Does it support it through cygwin?
rb07
Posts: 1400
Joined: Sun Aug 24, 2008 3:14 am

Re: preallocation setting problem on cygwin

Post by rb07 »

kIssGlaY wrote:transmission always does a full preallocation, which is the value 2
How do you know this?

Pre-allocation, and full both create the file showing the full size. The difference is that pre-allocation doesn't use the space until it is needed, and as you point out, full takes a lot of time, sparse doesn't.
kIssGlaY wrote:Is there a way to let it use sparse file under windows?
Yes.

If you mean Windows, not Cygwin, I'm sure Transmission-Qt ported to Windows uses sparse allocation (I implemented that part).
kIssGlaY wrote:[Is] it support[ed] through cygwin?
In theory, yes.

Cygwin uses newlib (a variant of libc), and looking at the log left by configure (config.log) fallocate64() is not present, but posix_fallocate() is, so that last function is used. I haven't tested it, so perhaps there's a bug on newlib, or posix_fallocate() is not the real thing... hold on, I did test it a long time ago, with a torrent bigger than 1 TiB, which was more, much more, than the free space on my hard drive, so I think the answer is yes, it does work... so we are back to the top, are you sure sparse is not working for you?
kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

Re: preallocation setting problem on cygwin

Post by kIssGlaY »

How do you know this?

Pre-allocation, and full both create the file showing the full size. The difference is that pre-allocation doesn't use the space until it is needed, and as you point out, full takes a lot of time, sparse doesn't.
I've used Transmission-Qt with preallocation set to 0, and it didn't hang when initially creating the file downloaded, everything seemed normal. But as for transmission-daemon under cygwin, even preallocation set to 0, it did hang for certain period of time (depending on the bulk size it creates) -- webui or `transmission remote gui' can't connect to the daemon, until the file is allocated.
rb07
Posts: 1400
Joined: Sun Aug 24, 2008 3:14 am

Re: preallocation setting problem on cygwin

Post by rb07 »

Zero is no pre-allocation, my answer was about using pre-allocation with the daemon built on Cygwin. I didn't say anything about setting the option to zero, or testing that the file is created as the pieces are downloaded, which is random... and if the last piece comes first, then the whole file will be created as with option set to 2.

There are other factors that will make the daemon, or Transmission-Qt, seem to hang. One well know is the number of files on the torrent, if it is very large both take a large time to create the visible list; that has nothing to do with pre-allocation. Of course "very large" is relative to the machine the program is running, the monster torrent I mentioned (more than 1 TiB) had over 3,000 files (and even Windows Explorer has trouble displaying that many on one directory).

If you really want a meaningful test, don't open the Web client, or Transmission-Qt connected to the daemon, just observe the file(s) being created. The default setting of 1 should show the file(s) being created as soon as the daemon has something to store.
kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

Re: preallocation setting problem on cygwin

Post by kIssGlaY »

If you really want a meaningful test, don't open the Web client, or Transmission-Qt connected to the daemon, just observe the file(s) being created. The default setting of 1 should show the file(s) being created as soon as the daemon has something to store.
I've observed what you say. When set to 1, the file was created immediately as the daemon had something to store, but the time of the process varied depending on the single file size of which the daemon created. Let's say there's only 1 file in the torrent which is 8GB, and with preallocation set to 1, this process could last several minutes until the whole file size becoming 8GB (when the file was created, the size shown in the Explorer is `0', and after several minutes hang, the file size becoming 8G immediately, not gradually increasing during the period) and the client cant connect to the daemon when this process under way.

With `preallocation = 0', no different to `preallocation = 1'.
kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

Re: preallocation setting problem on cygwin

Post by kIssGlaY »

x190 wrote:Similar issue here:
viewtopic.php?f=2&t=14888#p65222
Not the same issue, the link posted is discussing native linux transmission on ext4, but the problem here is the transmission through cygwin emulation layer on ntfs. Native transmission-qt ported to windows has no such problem.
So dunno it's a cygwin glitch or what else. Or extra tweak needed in ./configure when building under cygwin to get rid of the full preallocation?
rb07
Posts: 1400
Joined: Sun Aug 24, 2008 3:14 am

Re: preallocation setting problem on cygwin

Post by rb07 »

kIssGlaY wrote:So dunno it's a cygwin glitch or what else.
No, my mistake, there is no sparse pre-allocation on Cygwin. I was reading the full pre-allocation code, which uses posix_fallocate() and its probably faster than the default option.

About my test, I probably don't remember correctly, since I tested a Cygwin build, and a MinGW build.
kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

Re: preallocation setting problem on cygwin

Post by kIssGlaY »

So `preallocation = 1' or `preallocation = 0' has no effect under cygwin? all three settings result in invoking the same core disk allocation code in cygwin?

Native minGW port seems working fine concerning preallocation setting.
rb07
Posts: 1400
Joined: Sun Aug 24, 2008 3:14 am

Re: preallocation setting problem on cygwin

Post by rb07 »

kIssGlaY wrote:`preallocation = 1' or `preallocation = 0' has no effect under cygwin?
I don't know about no-preallocation, if you read what I wrote its not easy to test (i.e. randomness).

"Has no effect" is a bad description, of course it has an effect, they are not the same (one creates and fills the full file with zeros, the other fills it randomly), but from a user's point of view maybe they seem the same.
kIssGlaY wrote:Native minGW port seems working fine concerning preallocation setting.
A MinGW build uses my code, which is the same used by Transmission-Qt.

There are a lot of advantages to using the Cygwin build (large file support, foreign language support, invalid characters support, performance is better -- all supported by Transmission-Qt for Windows, but those changes have not been incorporated into the project's source code, so you don't get them in MinGW), you just found one of the disadvantages (no pre-allocation) .
kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

Re: preallocation setting problem on cygwin

Post by kIssGlaY »

Actually, preallocation does work under cygwin, and only full preallocation will kick in, no matter what is assigned to `preallocation', because files created were always filled with `0' with its full size. For a torrent of 270G with seven bluray iso, it caused almost 40-50 minutes to allocate the required space by filling 0, and meanwhile with client and web ui hanging, the daemon can by no means be terminated until at least one file created is filled with 0. There seems no way to turn preallocation off.
rb07
Posts: 1400
Joined: Sun Aug 24, 2008 3:14 am

Re: preallocation setting problem on cygwin

Post by rb07 »

Sparse files are a Cygwin option, external to Transmission, you just have the add the option in /etc/fstab. Ref: The Cygwin Mount Table

If you know Cygwin you probably can guess that to apply the change you have to close all Cygwin processes (anything that uses the Cygwin dll), the first time the dll is loaded after the change will make all file creations, on the mount entry, sparse. If the file already existed, there is no change, it will not be converted into sparse.
kIssGlaY
Posts: 24
Joined: Sat Jun 29, 2013 11:13 am

Re: preallocation setting problem on cygwin

Post by kIssGlaY »

Thank you for the tip.

But still no lack after adding following to /etc/fstab, filling 0 by full size still, tested with `preallocation = 0|1'.

Code: Select all

w: /cygdrive/w ntfs binary,sparse,posix=0 0 0
v: /cygdrive/v ntfs binary,sparse,posix=0 0 0
Post Reply