Fragmentation of downloaded content

[Iznakurnog]

I'm using transmission (1.51 currently) at my NAS - Dlink DNS-323
I've recently found that all downloaded content comes nasty fragmented For instance I've never been able to copy any new file to PC at rates faster than 3-4 mbps. But when it is copied and dumped back to NAS inplace of old file coping speed (both up and down) become excellent (up to 30 mbps)
Coping files to new location is not very quick and convinient solution. vast majority of my files are quite large - 4-40 Gb. But AFAIK there are no any relevant utilities for defragmenting ext2 (I heard they only going to introduce something of that kind in ext4)

Is it possible to do something from transmission side ? Teach it to write pieces/blocks not by linear way but exactly to places where they should be in the final file. and avoid fragmentation entirely.

Appreciate you consulting in advance !

[Jordan]

See the `preallocation' entry in http://trac.transmissionbt.com/wiki/EditConfigFiles. That will do what you're looking for.

[Iznakurnog]

Superb ! That's exactly what I want Now turn it on. will try and test ! Preallocation : 2 - definitely better choice for me.
I would even suggest to make it default instead of 1. There are no detailed description for configuration parameters or guides for tuning.
so many of newbs (one was me) not even guess what is the root case of poor transfer rate and twitching of movies on theirs mediatanks ... they messing with NFS, medaistreaming and so on.

Well as far as solution already exists we probably can close the topic. Transmission is excellent and powerful client. Thanks for your great work !

[Iznakurnog]

I have tested preallocation:2
This definitely makes things better without any noticable drawbacks but fragmentation still retain.

coping rate of downloaded files is now 6.5 mbps instead of 8.5 for non fragmented content via 100 mb network
and only 9-10 mb with gigabit vs 30 mbps for non fragmented ....

I think fragmentation problem still makes a big point for future improvement of new data blocks allocation algorithm ....

[hdfan2]

I'm using T 1.51 on Popcorn Hour 110 NMT, ext3 filesystem, preallocation is set to 2. However, downloaded video files are still terribly fragmented, so that they are unwatchable (playback constantly stutters). It is definitely fragmentation, because after copying (not moving!) file elsewhere on the same disk it is played seamlessly. This issue is very common in our local PCH community, everyone using T suffer it. Maybe there should be some "forced" preallocation, which would fill every chunk with zeroes before downloading it (having file in 1-4 MB fragments is way better than in 4KB, and filling the whole file takes way TOO long, especially on such weak hardware). Thanks!

[Iznakurnog]

Do something to fix the problem pleeeeease .... All our local DNS-323+transmission community terribly suffer from it
preallocation:2 doesn't much help. On the other hand native torrent client for DNS-323 downloads without any fragmentation ! So it is possible ! But native client is very simple and has other disadvantages and I get used to transmission and it is good and best in all other respects.

Well if it's really hard to do someting with engine and preallocation algoritm what about implementing very simple solution like this:
optionally download to temporary folder and when finished copy content to final location for seeding ?

Thanx !!!

[wishyou]

I've got this exact same problem on my NAS, the Readynas NV+ from Netgear (Infrant).
I get terrible fragmentation on content downloaded by Transmission 1.52 running on this device and by that poor performance copying/streaming it afterward.

Copying content once locally gives me low fragmentation and the expected performance again; like other users report.

For me the preallocation-setting do nothing, 0,1 or 2 makes no difference whatsoever.
Is this a known issue? Or a filesystem problem with ext2/ext3? (I saw a bug regarding XFS about similar matter)

Wish

[Jordan]

I've got this exact same problem on my NAS, the Readynas NV+ from Netgear (Infrant).
I get terrible fragmentation on content downloaded by Transmission 1.52 running on this device and by that poor performance copying/streaming it afterward.

Copying content once locally gives me low fragmentation and the expected performance again; like other users report.

For me the preallocation-setting do nothing, 0,1 or 2 makes no difference whatsoever.
Is this a known issue? Or a filesystem problem with ext2/ext3? (I saw a bug regarding XFS about similar matter)

Wish

What's the proper OS call to make to preallocate a file on that system?

Also, could you post the section of config.log where Transmission is looking for fallocate(), posix_fallocate(), etc?

[super-poussin]

for nv+ :

Code: Select all

configure:21608: checking for fallocate
configure:21631: gcc -o conftest -g -O2 -g -O0 -std=gnu99 -ggdb3 -Wall -W -Wpointer-arith -Wformat-security -Wcast-align -Wundef -Wcast-align -Wstrict-prot
configure:21637: $? = 1
configure: failed program was:
| /* confdefs.h.  */
| #define PACKAGE_NAME "transmission"
| #define PACKAGE_TARNAME "transmission"
| #define PACKAGE_VERSION "1.52+"
| #define PACKAGE_STRING "transmission 1.52+"
| #define PACKAGE_BUGREPORT "http://trac.transmissionbt.com/newticket"
| #define PACKAGE "transmission"
| #define VERSION "1.52+"
- config.log 544/1939 28%
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_DLFCN_H 1
| #define TR_NIGHTLY_RELEASE 1
| #define STDC_HEADERS 1
| #define TIME_WITH_SYS_TIME 1
| /* end confdefs.h.  */
| #include <linux/falloc.h>
| int
| main ()
| {
| return fallocate(-1,0,0,0);
|   ;
|   return 0;
| }
configure:21654: result: no

Code: Select all

configure:21672: checking for posix_fallocate
configure:21728: gcc -o conftest -g -O2 -g -O0 -std=gnu99 -ggdb3 -Wall -W -Wpointer-arith -Wformat-security -Wcast-align -Wundef -Wcast-align -Wstrict-prototypes -Wmissing-dec
configure:21734: $? = 0
configure:21752: result: yes
configure:21775: checking for a BSD-compatible install
configure:21831: result: /usr/bin/install -c
configure:21842: checking whether make sets $(MAKE)
configure:21863: result: yes

[wishyou]

Also some system details:

This is a debian-sarge distro built for a sparc-processor.
I do not know what the proper preallocate call is, but from super-poussins config log it looks like the system supports posix-fallocate()

For the record, the files downloaded actually look complete in size after a short while, but extents reported from filefrag increases steadily as the download proceeds.

Regards,
Wish

[hdfan2]

Any news on this one? I think it's obvious that on NMT (and perhaps other platforms like NAS) posix_fallocate doesn't actually allocates clusters - just reserves the disk space for future. So the only way I see is filling the chunk with 0's before starting downloading it (at preallocation=2, of course, or maybe even 3). This will slow down download, but is way much better than filling the whole file at the beginning (this would lead to HUGE delays on weak platforms).

[Jordan]

If someone on the NMT cooks up a clean patch for this I'd very likely use it. I think that would waste a lot less time than me trying to write code that optimizes for a system I can't test on. (hmm... maybe if NMT were to send me a sample box...

[Iznakurnog]

Parts of Config.log for DNS-323

Code: Select all

 $ ./configure --disable-gtk --disable-nls --disable-cli --prefix=/ffp

## --------- ##
## Platform. ##
## --------- ##

hostname = som
uname -m = armv5tejl
uname -r = 2.6.12.6-arm1
uname -s = Linux
uname -v = #30 Mon Aug 18 14:19:14 CST 2008
 

Code: Select all

configure:21786: checking for fallocate
configure:21809: gcc -o conftest -g -O2 -g -O3 -funroll-loops  -std=gnu99 -ggdb3 -Wall -W -Wpointer-arith -Wformat-security -Wcast-align -Wundef -Wcast-align -Wstrict-prototypes -Wmissing-declarations -Wmissing-format-attribute -Wredundant-decls -Wnested-externs -Wunused-parameter -Wwrite-strings -Wextra -Wdeclaration-after-statement -Winit-self   conftest.c  >&5
conftest.c:24:26: error: linux/falloc.h: No such file or directory
conftest.c:27: warning: function declaration isn't a prototype
conftest.c: In function 'main':
conftest.c:28: warning: implicit declaration of function 'fallocate'
conftest.c:28: warning: nested extern declaration of 'fallocate'
configure:21816: $? = 1
configure: failed program was:
| /* confdefs.h.  */
| #define PACKAGE_NAME "transmission"
| #define PACKAGE_TARNAME "transmission"
| #define PACKAGE_VERSION "1.61"
| #define PACKAGE_STRING "transmission 1.61"
| #define PACKAGE_BUGREPORT "http://trac.transmissionbt.com/newticket"
| #define PACKAGE "transmission"
| #define VERSION "1.61"
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_DLFCN_H 1
| #define TR_STABLE_RELEASE 1
| #define STDC_HEADERS 1
| #define TIME_WITH_SYS_TIME 1
| /* end confdefs.h.  */
| #include <linux/falloc.h>
| int
| main ()
| {
| return fallocate(-1,0,0,0);
|   ;
|   return 0;
| }
configure:21835: result: no 

Code: Select all

configure:21854: checking for posix_fallocate
configure:21910: gcc -o conftest -g -O2 -g -O3 -funroll-loops  -std=gnu99 -ggdb3 -Wall -W -Wpointer-arith -Wformat-security -Wcast-align -Wundef -Wcast-align -Wstrict-prototypes -Wmissing-declarations -Wmissing-format-attribute -Wredundant-decls -Wnested-externs -Wunused-parameter -Wwrite-strings -Wextra -Wdeclaration-after-statement -Winit-self   conftest.c  >&5
conftest.c:54: warning: function declaration isn't a prototype
conftest.c:64: warning: function declaration isn't a prototype
/tmp/ccc53nir.o: In function `main':
/mnt/HD_a2/COMPILING/Trans161/transmission-1.61/conftest.c:65: undefined reference to `posix_fallocate'
collect2: ld returned 1 exit status
configure:21917: $? = 1
configure: failed program was:
| /* confdefs.h.  */
| #define PACKAGE_NAME "transmission"
| #define PACKAGE_TARNAME "transmission"
| #define PACKAGE_VERSION "1.61"
| #define PACKAGE_STRING "transmission 1.61"
| #define PACKAGE_BUGREPORT "http://trac.transmissionbt.com/newticket"
| #define PACKAGE "transmission"
| #define VERSION "1.61"
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_DLFCN_H 1
| #define TR_STABLE_RELEASE 1
| #define STDC_HEADERS 1
| #define TIME_WITH_SYS_TIME 1
| #define HAVE_STRLCPY 1
| #define HAVE_DAEMON 1
| #define HAVE_DIRNAME 1
| #define HAVE_BASENAME 1
| #define HAVE_DAEMON 1
| #define HAVE_STRCASECMP 1
| #define HAVE_LOCALTIME_R 1
| /* end confdefs.h.  */
| /* Define posix_fallocate to an innocuous variant, in case <limits.h> declares posix_fallocate.
|    For example, HP-UX 11i <limits.h> declares gettimeofday.  */
| #define posix_fallocate innocuous_posix_fallocate
| 
| /* System header to define __stub macros and hopefully few prototypes,
|     which can conflict with char posix_fallocate (); below.
|     Prefer <limits.h> to <assert.h> if __STDC__ is defined, since
|     <limits.h> exists even on freestanding compilers.  */
| 
| #ifdef __STDC__
| # include <limits.h>
| #else
| # include <assert.h>
| #endif
| 
| #undef posix_fallocate
| 
| /* Override any GCC internal prototype to avoid an error.
|    Use char because int might match the return type of a GCC
|    builtin and then its argument prototype would still apply.  */
| #ifdef __cplusplus
| extern "C"
| #endif
| char posix_fallocate ();
| /* The GNU C library defines this for functions which it implements
|     to always fail with ENOSYS.  Some functions are actually named
|     something starting with __ and the normal name is an alias.  */
| #if defined __stub_posix_fallocate || defined __stub___posix_fallocate
| choke me
| #endif
| 
| int
| main ()
| {
| return posix_fallocate ();
|   ;
|   return 0;
| }
configure:21939: result: no 

Is that helpful ? I am not able write patch cause i'm not programmer for linux but probably I could help testing it at my DNS-323

[hdfan2]

I've took a quick look through T sources, and the first (and probably wrong ) place to put the patch on is at the end of tr_peerMsgsAddRequest function in libtransmission/peer-msgs.c file. The patch itself looks like this:

Code: Select all

886a887,897
> 
>     if( preallocationMode == TR_PREALLOCATE_FULL )
>         while( length > 0 )
>         {
>             uint8_t buf[MAX_STACK_ARRAY_SIZE];
>             const size_t thisPass = MIN( length, sizeof( buf ) );
>             tr_ioWrite( msgs->torrent, index, offset, thisPass, buf );
>             i -= thisPass;
>             offset += thisPass;
>         }
> 

Of course, this is extremely draft version, but I'd like the T developers to comment, is this actually a place where we begin downloading a new chunk? If so (and the function tr_ioWrite actually writes data to the disk considering it can span over different files), I think I'll find a way to build this special version of T and test it on our hardware.

[Jordan]

The correct place to put the preallocation implementation is in libtransmission/fdlimit.c's preallocateFileFull() function...