transmission-daemon 2.22 on FreeBSD consumes all CPU

[blacklion]

I use transmission-daemon 2.22 on FreeBSD 8.2-STABLE.

I seeds ~360 torrents with total size of 691.1GiB.

Normally, transmission-daemon shows total upload speed about 2700-3000KiB/s (no limits are set) and consumes 6-10% (system+user) of one core of Core2Duo E4500 CPU, and system time is below 1%.

But sometimes, when speed is about 3500KiB/s it consumes 100% of one core and 45% of it is system time.

ktrace shows only gettimeofday() and writev() calls with absurdly low count (1024) at these moments. kevent() calls are very rare, comparing to "normal" behavior.

Userland profiler (gprof) shows, that "bandwidthPulse" consume a lot of CPU over all program life (it was runnging for about a hour and consumes CPU for last 15-20 minutes of that hour).

It seems, that sometimes transmission-daemon starts to call gettimeofday() too often and it is expensive syscall on FreeBSD. It looks like a bug.

It is not directly connected to upload speed: sometimes it uploads 3500KiB/s with only 10% of CPU, and sometimes it hogs all available CPU, (almost) stops to respond to remote GUI, etc. But I never seen situation when it hogs CPU at low upload speed.

I could provide any additional information on request -- ktrace output or statistics, gprof output, etc.

[Jordan]

blacklion, thanks for doing this profiling!

I'd be very interested in hearing how 2.30 beta 4 behaves in similar situations.

Your discussion of tr_bandwidthClamp() sounds a lot like the https://trac.transmissionbt.com/changeset/12170 commit that was added to lower the uTP CPU overhead.

[blacklion]

I'll give it a try. Nom I'm building it, and here are some build errors (I'll fix them by yself, I'm posting this JFYI):

Code: Select all

gmake[2]: Entering directory `/usr/local/build/ports/usr/ports/net-p2p/transmission-daemon/work/transmission-2.30b4/third-party/libutp'
  CXX    utp.o
In file included from utp.cpp:78:
utp_config.h:8:2: warning: #warning implement this in libtransmission
utp.cpp:104: error: 'in6_addr' does not name a type
utp.cpp: In member function 'byte PackedSockAddr::get_family() const':
utp.cpp:118: error: 'in6_addr' was not declared in this scope
utp.cpp:118: error: expected primary-expression before ')' token
utp.cpp:118: error: 'IN6_IS_ADDR_V4MAPPED' was not declared in this scope
utp.cpp: In constructor 'PackedSockAddr::PackedSockAddr(const SOCKADDR_STORAGE*, socklen_t)':
utp.cpp:134: error: 'sockaddr_in' was not declared in this scope
utp.cpp:135: error: expected initializer before '*' token
utp.cpp:142: error: 'sin' was not declared in this scope
utp.cpp:145: error: 'sockaddr_in6' was not declared in this scope
utp.cpp:146: error: expected initializer before '*' token
utp.cpp:147: error: 'union PackedSockAddr::<anonymous>' has no member named '_in6addr'
utp.cpp:147: error: 'sin6' was not declared in this scope
utp.cpp: In member function 'SOCKADDR_STORAGE PackedSockAddr::get_sockaddr_storage(socklen_t*) const':
utp.cpp:157: error: 'sockaddr_in' was not declared in this scope
utp.cpp:157: error: 'sin' was not declared in this scope
utp.cpp:157: error: expected primary-expression before ')' token
utp.cpp:164: error: 'sockaddr_in6' was not declared in this scope
utp.cpp:164: error: 'sin6' was not declared in this scope
utp.cpp:164: error: expected primary-expression before ')' token
utp.cpp:168: error: 'const union PackedSockAddr::<anonymous>' has no member named '_in6addr'
utp.cpp: In member function 'const char* PackedSockAddr::fmt(char*, size_t) const':
utp.cpp:186: error: 'in6_addr' was not declared in this scope
utp.cpp:186: error: expected primary-expression before ')' token
utp.cpp:186: error: 'const union PackedSockAddr::<anonymous>' has no member named '_in6addr'

Replacing include section in utp.h with

Code: Select all

#include <sys/types.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <stdlib.h>

helps.

[blacklion]

Ok, 2.30b4 is running with disabled uTP (to make comparsion fair), I'll report about results after several hours.

Two observation after first 10 minutes:

(1) As upload speed rises to 1500+KiB/s, remote GUI (even transmission-remote --list) starts to work very slow and unstable. CPU usage is about 6%, 3 threads.

(2) About 25% of my torrents receives "400 Bad Request" from tracker. Maybe it is coincident, that there are problems on my main tracker right now, when I changed version, and maybe not. I'll dump exchange between client and tracker.

[blacklion]

Nope, 2.22 doesn't have problems with tracker. New topic about this problem is started.

[blacklion]

Now I'm starting experiments with fixed 2.30b4.

[blacklion]

3 hours, massive upload, 13-15% of CPU, no anomalies. Looks good. It spends slightly more than "normal" 2.22, but doesn't start to consume ALL CPU.

[blacklion]

2.30b4 works for 10 hours with 12-14% of CPU.

But it is still very unresponsive for GUI ("transmission-remote --list" takes about 30 seconds, Transmission Remote GUI 3.1 sometimes complains about timeouts and reconnects), and it has some problems with trackers: always 2-3 torrents are in state "Tracker gave HTTP response code 0 (No Response)," when other torrents (tens of them) have proper answers from same trackers. Manual re-announce helps.

[Jordan]

I'm wondering if your profiling runs turned up anything else interesting.

My profile runs pre-2.30b4 found encryption and checksums to be the most expensive calls... neither of those can really be reduced, but encryption is optional and checksums are short-term. During a steady state without encryption, gettimeofday() still looks like the most expensive call (about 10% of runtime in my tests).

[blacklion]

I'll repeat profiling of 2.30b4 at this weekend.