Some cache stats?

[Astara]

Maybe this code could be added to the mainline in some form so some mainline
stats can be gathered.

Basically, it counts hits+misses and writes them out every 15 minutes. Every
hour would be fine in my case, but if someone wanted to test with 1 torrent on a high
speed download, given that a 300-800MB torrent can be downloaded (@~3MB/s...) in a few to several minutes, finer granularity might be wanted as well (like every 30seconds in that case). So
the time period might default to 60 minutes, but if _TRX_STAT_DELAY_ is set
in env, then use it as # of seconds between updates?

This is in cache.c: (line numbers and formatting may be different but shouldn't
be hard to adjust (I'd hope, if a prob can like produce a diff against cur release
or cur source...

Code: Select all

--- cache.c	2014-05-18 13:52:14.000000000 -0700
+++ cache.c	2014-06-15 17:09:08.000000000 -0700
@@ -20,6 +20,7 @@
 #include "torrent.h"
 #include "trevent.h"
 #include "utils.h"
+#include <unistd.h>
 
 #define MY_NAME "Cache"
 
@@ -191,8 +192,7 @@
 		err = flushContiguous(cache, runs[i].pos, runs[i].len);
 
 		for (j = i + 1; j < n; j++)
-			if (runs[j].pos > runs[i].pos)
-				runs[j].pos -= runs[i].len;
+			if (runs[j].pos > runs[i].pos) runs[j].pos -= runs[i].len;
 	}
 
 	return err;
@@ -321,18 +311,34 @@
 	return cacheTrim(cache);
 }
 
+int cache_hits, cache_misses;
+time_t logtime;
+# include <stdio.h>
+# include <fcntl.h>
+
 int
-tr_cacheReadBlock(tr_cache * cache,
-									tr_torrent * torrent,
+tr_cacheReadBlock(tr_cache * cache, tr_torrent * torrent,
 									tr_piece_index_t piece,
 									uint32_t offset, uint32_t len, uint8_t * setme) {
 	int err = 0;
 	struct cache_block *cb = findBlock(cache, torrent, piece, offset);
+	time_t now;
+
+	time(&now);
+	if (!logtime || now-logtime > 60*15 ) {
+		tr_logAddTorInfo(torrent, "[%ld], Cache H,M=%d,%d\n", 
+											now, cache_hits, cache_misses);
+		logtime=now;
+		//close(fileh);
+	}
 
-	if (cb)
+	if (cb) {
+		++cache_hits;
 		evbuffer_copyout(cb->evbuf, setme, len);
-	else
+	} else {
+		++cache_misses ;
 		err = tr_ioRead(torrent, piece, offset, len, setme);
+	}
 
 	return err;
 }

----
It's fairly trivial code... finding it's output in the log was a
matter of grepping for the lin -- but used perl to reduce it (as it comes out at the
end of a line):

[2014-06-14 16:35:13.754 PDT] Sample-linux-dist-tor.iso.xz [1402788913], Cache H,M=1992567,5111910

So wrote a perl script to reduce output of my logs and added option
to only put out results once/hour, 4x/day or 2x/day.
----cache_filter---

Code: Select all

#!/bin/bash
perl -we'use strict;
BEGIN {
  printf "%-9s  %-7s  %-7s  %-7s  ", "    DT", "Hits", "Misses", "Total";
  printf "%-7s  %-5s\n", "%Hits", "%Miss";
}
my $hr_re=qr{\d{4}\.\d{2}0};  #1/hour
my $Qs_re=qr{\d{4}\.(?:(?:0[06]0)|(?:1[28]0.))}; #quarter days
my $Hs_re=qr{\d{4}\.(?:00|12)0}; #half days
my $lastmatch;
while (<>) {
  m{^(\[[^\]]+\] ).*, (Cache H.*)} && do {
  my ($t,$stat)=($1,$2);
  $t =~ s/:\d\d\.\d\d\d [A-Z]{3}\]//; #now get rid of junk...
  $t =~ s/^\[\d{4}-//;
  $t =~ s/[:-]//g; $t=~s/ /./;
  my $lm = $lastmatch;
  if ($t =~ m{($Hs_re)}) {
    my $nm=$lastmatch=$1;
    $lm && $lm eq $nm and next;  # skip over too soon hits
    $stat =~ s/Cache H,M=//;
    $stat =~ m{^(\d+),(\d+)} && do {    # and show percentages
      my ($hits, $misses) = ($1, $2);
      my $total=$hits+$misses;
      printf "%s %-7d  %-7d  %-7d  ", $t, $hits, $misses, $total;
      if ($total) {
        printf("(%-6.2f  %6.2f)", 100.0*$hits/$total, 100.0*$misses/$total);
      }
      printf("\n");
    };
  } else { $lastmatch=$1; next}
  };
}
' $1

---
The above is in a shellscript (though work is in perl)
and I can run it on a log file via:
> cache_filter /var/log/transmission/daemon.log
DT Hits Misses Total %Hits %Miss
0812.1206 0 0 0
0813.0600 0 2800082 2800082 (0.00 100.00)
0813.0700 0 2957686 2957686 (0.00 100.00)

So for the latest 10 days... (aug 12-22):

Code: Select all

    DT     Hits     Misses   Total    %Hits    %Miss
0812.1206  0        0        0        
0813.0600  0        2800082  2800082  (0.00    100.00)
0813.0700  0        2957686  2957686  (0.00    100.00)
0813.0800  0        3115141  3115141  (0.00    100.00)
0813.0900  0        3273036  3273036  (0.00    100.00)
0813.1000  0        3431663  3431663  (0.00    100.00)
0813.1100  0        3591137  3591137  (0.00    100.00)
0813.1200  0        3750886  3750886  (0.00    100.00)
0813.1300  0        3909308  3909308  (0.00    100.00)
0813.1400  0        4066789  4066789  (0.00    100.00)
0813.1500  0        4224814  4224814  (0.00    100.00)
0813.1600  0        4384742  4384742  (0.00    100.00)
0813.1700  0        4541596  4541596  (0.00    100.00)
0813.1800  0        4696790  4696790  (0.00    100.00)
0813.1900  0        4856355  4856355  (0.00    100.00)
0813.2000  0        5014073  5014073  (0.00    100.00)
0813.2101  0        5172928  5172928  (0.00    100.00)
0813.2201  0        5333320  5333320  (0.00    100.00)
0813.2301  0        5494871  5494871  (0.00    100.00)
0814.0001  0        5662094  5662094  (0.00    100.00)
0814.0101  0        5827257  5827257  (0.00    100.00)
0814.0201  0        5990160  5990160  (0.00    100.00)
0814.0301  0        6152821  6152821  (0.00    100.00)
0814.0401  0        6315485  6315485  (0.00    100.00)
0814.0501  0        6479130  6479130  (0.00    100.00)
0814.0601  0        6642523  6642523  (0.00    100.00)
0814.0701  0        6803213  6803213  (0.00    100.00)
0814.0801  0        6963577  6963577  (0.00    100.00)
0814.0901  0        7125069  7125069  (0.00    100.00)
0814.1001  0        7283641  7283641  (0.00    100.00)
0814.1101  0        7440393  7440393  (0.00    100.00)
0814.1202  0        7594654  7594654  (0.00    100.00)
0814.1302  0        7747051  7747051  (0.00    100.00)
0814.1402  0        7903241  7903241  (0.00    100.00)
0814.1502  176570   8189357  8365927  (2.11     97.89)
0814.1602  176570   8387017  8563587  (2.06     97.94)
0814.1702  176570   8552971  8729541  (2.02     97.98)
0814.1802  176570   8717172  8893742  (1.99     98.01)
0814.1902  176570   8880289  9056859  (1.95     98.05)
0814.2002  176570   9044708  9221278  (1.91     98.09)
0814.2102  176570   9205987  9382557  (1.88     98.12)
0814.2202  176570   9369832  9546402  (1.85     98.15)
0814.2302  176570   9532492  9709062  (1.82     98.18)
0815.0002  205527   9705350  9910877  (2.07     97.93)
0815.0102  205527   9887064  10092591  (2.04     97.96)
0815.0202  205527   10049279  10254806  (2.00     98.00)
0815.0303  205527   10207031  10412558  (1.97     98.03)
0815.0403  205527   10363460  10568987  (1.94     98.06)
0815.0503  205527   10520387  10725914  (1.92     98.08)
0815.0603  205527   10679192  10884719  (1.89     98.11)
0815.0703  205527   10834626  11040153  (1.86     98.14)
0815.0803  205527   10989111  11194638  (1.84     98.16)
0815.0903  205527   11144314  11349841  (1.81     98.19)
0815.1003  205527   11299890  11505417  (1.79     98.21)
0815.1103  205527   11459430  11664957  (1.76     98.24)
0815.1203  205527   11617679  11823206  (1.74     98.26)
0815.1303  205527   11772829  11978356  (1.72     98.28)
0815.1403  205527   11929266  12134793  (1.69     98.31)
0815.1503  205527   12090035  12295562  (1.67     98.33)
0815.1603  205527   12247638  12453165  (1.65     98.35)
0815.1703  205527   12405248  12610775  (1.63     98.37)
0815.1804  250120   12570579  12820699  (1.95     98.05)
0815.1904  281533   12752753  13034286  (2.16     97.84)
0815.2004  281533   12915276  13196809  (2.13     97.87)
0815.2104  281533   13075067  13356600  (2.11     97.89)
0815.2204  281533   13238250  13519783  (2.08     97.92)
0815.2304  281533   13401445  13682978  (2.06     97.94)
0816.0004  281533   13567310  13848843  (2.03     97.97)
0816.0104  281533   13731373  14012906  (2.01     97.99)
0816.0204  281533   13895479  14177012  (1.99     98.01)
0816.0304  281533   14060308  14341841  (1.96     98.04)
0816.0404  281533   14218650  14500183  (1.94     98.06)
0816.0504  281533   14377049  14658582  (1.92     98.08)
0816.0604  281533   14539064  14820597  (1.90     98.10)
0816.0704  281533   14697839  14979372  (1.88     98.12)
0816.0804  281533   14856514  15138047  (1.86     98.14)
0816.0905  281533   15019803  15301336  (1.84     98.16)
0816.1005  281533   15184550  15466083  (1.82     98.18)
0816.1105  281533   15347745  15629278  (1.80     98.20)
0816.1205  281533   15511191  15792724  (1.78     98.22)
0816.1305  281533   15672750  15954283  (1.76     98.24)
0816.1405  322626   15832604  16155230  (2.00     98.00)
0816.1505  369028   16080038  16449066  (2.24     97.76)
0816.1605  369028   16244616  16613644  (2.22     97.78)
0816.1705  369028   16409368  16778396  (2.20     97.80)
0816.1805  369028   16573936  16942964  (2.18     97.82)
0816.1905  369028   16734520  17103548  (2.16     97.84)
0816.2005  369028   16897534  17266562  (2.14     97.86)
0816.2105  369028   17062075  17431103  (2.12     97.88)
0816.2205  369028   17225989  17595017  (2.10     97.90)
0816.2305  369028   17386065  17755093  (2.08     97.92)
0817.0006  369028   17544246  17913274  (2.06     97.94)
0817.0106  369028   17702214  18071242  (2.04     97.96)
0817.0206  369028   17859266  18228294  (2.02     97.98)
0817.0306  369028   18017833  18386861  (2.01     97.99)
0817.0406  369028   18172603  18541631  (1.99     98.01)
0817.0506  369028   18333460  18702488  (1.97     98.03)
0817.0606  369028   18492471  18861499  (1.96     98.04)
0817.0706  369028   18659765  19028793  (1.94     98.06)
0817.0806  369028   18823755  19192783  (1.92     98.08)
0817.0906  369028   18986082  19355110  (1.91     98.09)
0817.1006  369028   19148459  19517487  (1.89     98.11)
0817.1106  369028   19309604  19678632  (1.88     98.12)
0817.1206  369028   19473653  19842681  (1.86     98.14)
0817.1306  369028   19634794  20003822  (1.84     98.16)
0817.1406  369028   19792401  20161429  (1.83     98.17)
0817.1507  369028   19948057  20317085  (1.82     98.18)
0817.1607  369028   20109820  20478848  (1.80     98.20)
0817.1707  369028   20271541  20640569  (1.79     98.21)
0817.1807  369028   20430339  20799367  (1.77     98.23)
0817.1907  410923   20623456  21034379  (1.95     98.05)
0817.2007  410923   20784632  21195555  (1.94     98.06)
0817.2107  410923   20947461  21358384  (1.92     98.08)
0817.2207  410923   21107754  21518677  (1.91     98.09)
0817.2307  410923   21265632  21676555  (1.90     98.10)
0818.0007  410923   21420109  21831032  (1.88     98.12)
0818.0107  410923   21577187  21988110  (1.87     98.13)
0818.0207  410923   21736848  22147771  (1.86     98.14)
0818.0307  410923   21900815  22311738  (1.84     98.16)
0818.0407  410923   22066156  22477079  (1.83     98.17)
0818.0507  410923   22225327  22636250  (1.82     98.18)
0818.0608  410923   22382814  22793737  (1.80     98.20)
0818.0708  410923   22539394  22950317  (1.79     98.21)
0818.0808  410923   22696116  23107039  (1.78     98.22)
0818.0908  410923   22851771  23262694  (1.77     98.23)
0818.1008  410923   23009136  23420059  (1.75     98.25)
0818.1108  410923   23165336  23576259  (1.74     98.26)
0818.1208  454303   23342230  23796533  (1.91     98.09)
0818.1308  454303   23503918  23958221  (1.90     98.10)
0818.1408  454303   23663329  24117632  (1.88     98.12)
0818.1508  454303   23814764  24269067  (1.87     98.13)
0818.1608  454303   23971659  24425962  (1.86     98.14)
0818.1708  454303   24128539  24582842  (1.85     98.15)
0818.1808  454303   24286535  24740838  (1.84     98.16)
0818.1908  454303   24444339  24898642  (1.82     98.18)
0818.2008  454303   24605073  25059376  (1.81     98.19)
0818.2109  454303   24767293  25221596  (1.80     98.20)
0818.2209  454303   24930168  25384471  (1.79     98.21)
0818.2309  454303   25092581  25546884  (1.78     98.22)
0819.0009  454303   25255219  25709522  (1.77     98.23)
0819.0109  454303   25416760  25871063  (1.76     98.24)
0819.0209  454303   25574733  26029036  (1.75     98.25)
0819.0309  454303   25731611  26185914  (1.73     98.27)
0819.0409  454303   25888115  26342418  (1.72     98.28)
0819.0509  454303   26042256  26496559  (1.71     98.29)
0819.0609  454303   26198178  26652481  (1.70     98.30)
0819.0709  454303   26353457  26807760  (1.69     98.31)
0819.0809  454303   26510303  26964606  (1.68     98.32)
0819.0909  454303   26671372  27125675  (1.67     98.33)
0819.1009  454303   26828383  27282686  (1.67     98.33)
0819.1109  679433   27055414  27734847  (2.45     97.55)
0822.1500  1998483  39928649  41927132  (4.77     95.23)
0822.1600  1998483  40108694  42107177  (4.75     95.25)
0822.1700  1998483  40270326  42268809  (4.73     95.27)
0822.1800  1998483  40431023  42429506  (4.71     95.29)

hmmm...... not so great max of <5% hit rate over past 10 days.

But here's previous 2 months of log @ 2x/day:

Code: Select all

    DT     Hits     Misses   Total    %Hits    %Miss
0614.0000  254879   1342164  1597043  (15.96    84.04)
0614.1200  1886062  4224900  6110962  (30.86    69.14)
0615.0000  1992567  6325924  8318491  (23.95    76.05)
0615.1200  2033984  8269382  10303366  (19.74    80.26)
0616.0000  2055051  10193085  12248136  (16.78    83.22)
0616.1200  2055051  12082941  14137992  (14.54    85.46)
0618.1200  42432    2818192  2860624  (1.48     98.52)
0619.0001  572127   4752667  5324794  (10.74    89.26)
0619.1202  572127   6414917  6987044  (8.19     91.81)
0620.0007  64864    1970794  2035658  (3.19     96.81)
0620.1208  64864    3912078  3976942  (1.63     98.37)
0621.0009  105444   5865920  5971364  (1.77     98.23)
0622.0007  176735   1266609  1443344  (12.24    87.76)
0622.1208  439417   3323542  3762959  (11.68    88.32)
0623.0009  480345   5609442  6089787  (7.89     92.11)
0626.1200  565322   19415267  19980589  (2.83     97.17)
0627.0000  589676   21353400  21943076  (2.69     97.31)
0627.1201  589676   23228515  23818191  (2.48     97.52)
0628.0002  2954142  25860873  28815015  (10.25    89.75)
0628.1204  2954142  28228698  31182840  (9.47     90.53)
0629.0004  2996323  30224150  33220473  (9.02     90.98)
0629.1205  3950962  32566480  36517442  (10.82    89.18)
0630.0006  3950962  34734920  38685882  (10.21    89.79)
0630.1207  3999572  36626557  40626129  (9.84     90.16)
0701.0008  4020329  38540960  42561289  (9.45     90.55)
0701.1208  4020329  40430314  44450643  (9.04     90.96)
0702.0009  4041156  42315513  46356669  (8.72     91.28)
0705.1200  7665856  57646722  65312578  (11.74    88.26)
0706.0001  7793738  59797858  67591596  (11.53    88.47)
0706.1201  7793738  61831458  69625196  (11.19    88.81)
0707.0002  7814270  63762915  71577185  (10.92    89.08)
0707.1203  7859703  65707359  73567062  (10.68    89.32)
0708.0004  7926534  67663256  75589790  (10.49    89.51)
0708.1205  7950640  69572004  77522644  (10.26    89.74)
0709.0005  7971981  71482445  79454426  (10.03    89.97)
0709.1206  7971981  73371332  81343313  (9.80     90.20)
0710.0007  8035458  75317488  83352946  (9.64     90.36)
0710.1208  10357573  78011157  88368730  (11.72    88.28)
0711.0009  10668618  80612269  91280887  (11.69    88.31)
0711.1209  10668618  82554431  93223049  (11.44    88.56)
0715.0000  11277999  96250444  107528443  (10.49    89.51)
0715.1201  11730181  98442373  110172554  (10.65    89.35)
0716.0002  11751356  100435308  112186664  (10.47    89.53)
0716.1202  11751356  102342444  114093800  (10.30    89.70)
0717.0003  11786484  104241778  116028262  (10.16    89.84)
0717.1204  11880141  106183587  118063728  (10.06    89.94)
0718.0005  11880141  108081030  119961171  (9.90     90.10)
0718.1206  11880141  109948417  121828558  (9.75     90.25)
0719.0006  12066600  111961006  124027606  (9.73     90.27)
0719.1207  12066600  113854197  125920797  (9.58     90.42)
0720.0008  12066600  115714731  127781331  (9.44     90.56)
0720.1209  12066600  117586051  129652651  (9.31     90.69)
0722.0005  1598846  1073453  2672299  (59.83    40.17)
0722.1206  1619527  3776704  5396231  (30.01    69.99)
0724.0000  523886   1801736  2325622  (22.53    77.47)
0724.1201  523886   3744345  4268231  (12.27    87.73)
0725.0002  523886   5669154  6193040  (8.46     91.54)
0725.1202  523886   7578425  8102311  (6.47     93.53)
0726.0003  782424   9643736  10426160  (7.50     92.50)
0726.1204  782424   11522358  12304782  (6.36     93.64)
0727.0005  868881   13495034  14363915  (6.05     93.95)
0727.1206  868881   15304284  16173165  (5.37     94.63)
0728.0006  868881   17163350  18032231  (4.82     95.18)
0728.1207  1106254  19125347  20231601  (5.47     94.53)
0729.0008  1106254  20979416  22085670  (5.01     94.99)
0729.1209  1380523  22917698  24298221  (5.68     94.32)
0802.1200  0        1791925  1791925  (0.00    100.00)
0803.0001  63034    3826182  3889216  (1.62     98.38)
0803.1202  1823498  6162396  7985894  (22.83    77.17)
0804.0002  3140884  9100304  12241188  (25.66    74.34)
0804.1203  3140884  11392460  14533344  (21.61    78.39)
0805.0004  3182644  13595569  16778213  (18.97    81.03)
0805.1205  3182644  15736069  18918713  (16.82    83.18)
0806.0006  3182644  17687958  20870602  (15.25    84.75)
0806.1206  3182644  19574009  22756653  (13.99    86.01)
0807.0002  0        406449   406449   (0.00    100.00)
0807.1203  81168    2446469  2527637  (3.21     96.79)
0808.0003  81168    4428861  4510029  (1.80     98.20)
0808.1204  81168    6326287  6407455  (1.27     98.73)
0809.0005  845093   8792243  9637336  (8.77     91.23)
0810.0000  63204    2423213  2486417  (2.54     97.46)
0810.1201  192671   4432813  4625484  (4.17     95.83)
0811.0002  192671   6430238  6622909  (2.91     97.09)
0811.1203  360025   8454107  8814132  (4.08     95.92)
0812.0003  402191   10428886  10831077  (3.71     96.29)
0812.1204  402191   12349699  12751890  (3.15     96.85)

So... for LONG runs, measured in weeks, I got better numbers.

Note any jumps up might have been due to 1 torrent hogging the BW...
It happens during a D/L.

Hope this wasn't too boring...

Would be more useful if had more stats from standard
version from others...

A better stat script is probably needed... if you couldn't tell
it started out as a 1-liner in shell but quickly went to perl...oh well.

[Astara]

Why? It's a write cache. Maybe you could use your expertise to gather stats for #5423? Thanks!

???

why would you have a write cache? This is a read cache.
Those stats are for hits requesting a read of data that is possibly
already read.

This is purely a matter of transmissions internal read cache and whether or not it is successful in caching data.

The other bug you point at (not that I disagree with it's sentiment...) is inextricably interwoven with the OS's (or kernel's) caching mechanism.

Doing stats on it requires knowing and measuring the workload and the demands on resources *outside* of transmission -- since transmission shouldn't really labor to throw things out of memory if there is free memory. Vs. if the OS is tight on memory, then you'd want to free up inactive memory areas more quickly.

Doing the stats for workloads outside of transmission would likely be more intrusive. As it is, I think the memory sparing actions of the POSIX
calls should be configurable. If I have a system with 64G of memory, and most of it is free, then it makes no sense for transmission to tell the OS
to release it. OTOH, if I have a 16MB machine, I may want to preserve what little free memory I have for use by other processes.
In that case, having the procs being memory-sparring would be a good
bet.
It would have made more sense to require stats before putting
the slow-down code in, in the first place, but the authors have put other slowdown code in place to, primarily, it seems, to protect the Mac. Apparently it is not difficult to "overwhelm" a mac with i/o, for example. So the wait code was added to "benefit"[sic] all the
users... ;-(

Anway, not that I am not supportive, but first you'd have to more clearly tell *what* you wanted measured.

The read cache I displayed is likely already optimized.

It makes sense to require stats before it is turned on as you are.

But if it is a "write" cache, as you say, then why would I see any usage on my client ... it has a few hundred torrents seeding, vs. the occasional random download. So overall, the cache only makes sense for (or during) read to not have to pull something on disk again to send to someone else.

----------------------

By default, transmission shouldn't invalidate pages if the user has over "XXX" (5%? or 512MB free?) at the OS level. Where memory is tight,
then it makes sense to tell the kernel what to free first.

[Astara]

I will have to look at what you say more closely, BUT one thing I wanted to throw in -- THERE are no days when I have no downloads.

I have a large enough set of torrents in upload that I constantly
have an average of 1MB/s being uploaded.

The two copies of transmission I have running "concur"...
one will show 600-750, the other will show 250-350KB (the two copies
compete w/each other, but one ends up with about 25% of the BW vs. the other w/75%. Added together they are almost always in the 960-1100KB/s range.

So days I have no hits really mean nothing was in the cache . So are you saying transmission has NO read cache? If it sends a file to 1 person, and another wants the same "chunk" 10 minutes later, it has to read it off disk again?...

Hmmm....I'm thinking you may be right...that certainly would help explain the low cache hit numbers... ;-/

So transmission is REALLY inefficient -- i.e. it reads the stuff off disk each time a client asks for it and it makes it worse (likely?) by telling
the OS not to cache the read as well so your disk is certain to have to do another read?... Why wouldn't it run out of memory if it could? I hear of people running dedicated seed-boxes and getting very high transfer speeds -- but never hear of someone using TR -- that might explain it (talking, *at least*, 10x or greater than what I get @ home)...

Hey, the reason I wanted to measure it is I thought I wasn't getting
much re-use from memory! So it *is* noticeable.

I still don't see how curl could have developed a fix for the 1024 limit 7-8 years ago and it's still an issue in their lib.

I still don't know why I had the problems with all the torrents being
shut down (ran out of fd's).... When I look in /proc/<pid>/fd, I only see
about 500-600 open fd's at any one time -- not even close to the 1024 limit....had never hit that before several months ago...

Hey... thanks for the clue stick... sometimes the truth is too unbelievable for me to believe....

[Astara]

Hey... thanks for the clue stick... sometimes the truth is too unbelievable for me to believe....

Even your version of It?

My version is usually more believable because I gave the person the benefit of the doubt and it doesn't even enter my mind that they might have done "XXX", because NO ONE could do it that way!!! Then I find out they did... *arg*...

Well, I'm not making any blanket statements about details right now, as it's been awhile since I worked on memory management, but I've incorporated trac ticket #5423 patches in my own copy of T(urbo)T that should address some of those issues.

??turbotrx? Is that your name for your own version or is it posted somewhere?

Linux and OS X handle memory management differently and while developers can code in memory hints, in the end the OS just does its thing. Our devs

our devs?... xmission devs?

made choices based partly on user complaints and without hard data seem disinclined to change anything. My changes were made mainly in fdlimit.c[h] and torrent.c, so you can check out those files for yourself.

If it is trx, I can explain how that creates poor performance on linux.
The apple guys are used to being pushy to support their cult. So they will complain more
and get more attention. On top of that Apple's OS doesn't perform near as well as linux under multiple process/thread load. Heck, linux is ahead of windows these days from what I've read. So scaling things to Apple is similar to downscaling to lowest common denominator (that would be POSIX compliant latest draft ;^). (After 2001, POSIX stopped being a standards reporter (their mission _was_ "descriptive"), and became a "prescriptive" organization who tells others what to do. At that point they started making some real bozo standards decisions -- revisions since then have just gone down hill.
... )

I still don't see how curl could have developed a fix for the 1024 limit 7-8 years ago and it's still an issue in their lib.

Do you have documentation for this?

Was reading this, and toward the end it looked like they were working around the problem .. and that is from 2007.

http://comments.gmane.org/gmane.comp.we ... rary/14764

Anyway, shouldn't that be plenty, even for you?
----
Not for a well running machine. Trx consistently is the top cpu user and it shouldn't
even be close to its limits -- but doing things like opening and closing FD's from different
torrents -- shuffling them all the time and constantly reading in from disk... My upload
is only rated at 7-8Mb (though). If I had a 100Mb, transmission would be struggling to keep up. Explains, IMO, why it is not usually used on heavy-duty seed-boxes.

Both samba and Squid want and get 16K FD's so they can be efficient. Trx plays shuffle - ug.

When I look in /proc/<pid>/fd, I only see
about 500-600 open fd's at any one time -- not even close to the 1024

Looks like all is well down on the farm! [/quote]
'cept I ran out of FD's early w/that limit.

Seeing 5-700 FD's doesn't mean they are all in the 1st 1024.

From what read, it's because the [p]select call uses a bitmask, it is limited to the
the first 1024 contiguous fd's.

At least, on linux, it is suggested to use poll (as the message was hinting at from the curl reference), or on linux, 'epoll', then it can multiplex much more efficiently. Apparently
both of those calls have support for discontiguous ranges of FD's.

But if trx wants to be a good player, it should ask the OS what memory condition are, "currently", and adapt itself -- using more memory for buffering during time of low
memory pressure, and less when memory is tighter... then it would work in both situations.

Why I feel it is stupid for trx to just release memory w/no reason to back that up, is this:
> free -h
total used free shared buffers cached
Mem: 94G 93G 680M 0B 456K 80G
-/+ buffers/cache: 13G 81G
Swap: 8.0G 627M 7.4G
---
It's not always the case, but now about 81G is being used for the same types of data that trx forced out to make new room for. Yet because it is doing this, it use using the most CPU and IO of any process on the system. If it wasn't a busybody in
regards to focusing on keeping memory low, it wouldn't be wasting as much disk-i/o and cpu cycles! Ironic -- but this is what I mean by optimizing for MAC can hurt other platforms that don't have the same issues.

If trx kept more torrents in memory -- especially when there is more than one peer downloading, it would help when it makes sense, but not use more resources when it doesn't.

Does that make sense?

I.e. they (devs) should require numbers and studies BEFORE putting in "optimizations"[sic].

Premature optimization is on the 10-top bad sw practices of all time (not that I am not
guilty of such but I usually respond to changing data)...

That there is no data to support the optimization, is a reason *FOR* removing it as it is code that is not known to serve any purpose and contributes to code bloat and complexity. Putting in an "iffy" optimization and then telling others that it won't be removed w/o a study, is really poor development. Claiming others need to have proof of benefit before
a patch, which should have required the same proof (but didn't) is removed is a bit hypocritical IMO...

[Astara]

BTW, FWIW, I do keep track of the I/O going on using my "iomon" util (see attached pic).

Trying to cut/past it as text doesn't show it nearly as well.

Code: Select all

    Sat, Aug 23 19:16:58  Δt:5s  Load:0.02  Tasks: 640 Cpu: 10.8%  Case: 84℉
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
NetI:◄━━━━━━━━━━━━━━Receive & Transmit 2MB/s Scale ━━━━━━━━━━━━━(Tot: 1.4MB/s)━►
eth2:█████████████(996KB/s Trx)███████████                                1MB/s 
eth4:██(381KB/s Trx)                                                    397KB/s 
eth0:                                                                     38B/s 
eth1:                                                                     26B/s ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Device      :◄━━━━━━━━━━━━━Read & Write 1MB/s Scale ━━━━━━━━━━━━(Tot: 997KB/s)━►
Media       :███████████████████████████(995KB/s Rd)██████████████████████████  
Root        :                                                           1.9KB/s 
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Name     :PID  :◄━━━━━━━━━━━━Read & Write 1MB/s Scale ━━━━━━━━━━━━(Tot: 1MB/s)━►
transmiss:16001:████████████████████(811KB/s Rd)███████████████████     812KB/s 
transmiss:16007:██(242KB/s Rd)█                                         242KB/s 
Terminal :38408:                                                          4KB/s 
smbd     :32652:                                                        3.2KB/s 
nmbd     :7778 :                                                         818B/s  

[Astara]

?turbotrx? Is that your name for your own version or is it posted somewhere?

viewtopic.php?f=4&t=12523&p=57868&hilit=turbot#p57868
viewtopic.php?f=5&t=14231&p=63011&hilit=turbot#p63011

Get your fix:
Prairie Crocuses
https://trac.transmissionbt.com/ticket/5423
https://trac.transmissionbt.com/ticket/5486

Ahh... too bad the project leads couldn't support the users in having what they want. So many of these projects become about someone else using as their personal fiefdom.

optimizing for MAC can hurt other platforms that don't have the same issues.

Not necessarily so. They use different macros. Linux-posix/OS X-fcntl.

[/quote]
So if you agree with me, why did you start off your sentence with a negative? I said "can"... last I heard, that was identical in non-totality as 'not necessarily so'... ;-/

Don't tell me... "not necessarily so...?"

Another reason I wanted to test the caching was to find out how efficient
it was and how that might relate to #5553: regression in 2.82 -- open file descriptors no longer user configurable -- which they close out as a dup of some a bug about POSIX FD's.

Thing is, POSIX is a bug, so striving to follow it as your minimal behavior is itself a bug! Problem w/POSIX is it is no longer POSIX. It's like when PacBell bought up the AT&T name. Just because they buy a name doesn't mean they are AT&T. Political types wanting to establish larger fiefdoms thought it would be good to wipe out some of the standards they didn't like and replace then with their own version.

Had some BSD person who apparently didn't like some of the standards because they were written "descriptively" and didn't describe his wants. He told me that SysV lost and they won... so now they were free to undo all the standardization that came out of SysV and force everyone to go with BSD standards... I was a bit "shocked", hearing his "revelation", but it did explain some of the new POSIX directions and reduction of features that were coming out of POSIX -- which before, hadn't acted so illogically. Now, the new standards are no longer compatible w/they old, and they are prescribing correct implementations -- completely against their previous mission statement to be "observers" and document current practice.

Oh well, I rambled too much again...sigh. (no sleep again)...