TLS SNI problem on OS X

Ask for help and report issues with the Mac OS X version of Transmission
Crank
Posts: 11
Joined: Sun Mar 20, 2016 6:22 pm

TLS SNI problem on OS X

Post by Crank » Sun Mar 20, 2016 6:28 pm

Hi guys,

there is a harsh problem on OS X with ssl trackers using a SNI certificate (like cloudflare does). Some private trackers are using cloudflare for ddos protecting and as we know, cloudflare uses a special certificate system (SNI). Transmission can't handle SNI certificates, so every download from such a tracker ends with a "couldn't connect to tracker" response.
I upgraded my OpenSSL version to 1.0.2g, but nothing happens. What can I do if I want use transmission on trackers whith SNI certificates? I'm using Transmission 2.92 and tested on linux and windows with the same torrent, same tracker and same SNI-certificate. On both operating systems (windows and linux) there aren't any problems. It seems that only on OS X transmission can't handle these special certificates.
Transmission is a great client, so please tell me what I can do (or maybe you to fix this issue). If I disable tls and use the simple https-announce, it works on every system without problems.

Thanks in advance.

TheBang
Posts: 23
Joined: Thu Oct 25, 2007 2:09 am

Re: TLS SNI problem on OS X

Post by TheBang » Mon Mar 21, 2016 12:36 pm

I tested on my own tracker which is running HTTPS via SNI, and there were no problems.

There were, however, problems when using a 301 redirect to rewrite all HTTP traffic (including the announce URL) to HTTPS. This caused an error from the tracker "invalid info hash....". I'm pretty sure this happened because of a parsing error of the serialized torrent data, due to the change in the announce URL (one extra character, due to "http" changing to "https"). When I manually modified the torrent file to https for the announce URL (and incrementing the corresponding index by one), thus avoiding the HTTP to HTTPS redirect, it worked fine.

Crank
Posts: 11
Joined: Sun Mar 20, 2016 6:22 pm

Re: TLS SNI problem on OS X

Post by Crank » Mon Mar 21, 2016 12:50 pm

Did you test on OS X (10.11.3) and a cloudflare certificate? Nevertheless, I took a deeper look into the sourcecode of transmission. Transmission uses a very old cURL library: https://trac.transmissionbt.com/browser ... rl.4.dylib cURL supports SNI certificates on version 7.39 and above. The library in the transmission source is to old. I think in most cases Transmission uses the local cURL installation on the OS, except on OS X, where it doesn't work. To fix this issue, a cURL update (libcurl.4.dylib) would be enough I think. And by the way, it's not only me. I know some other guys with the same problems. OS X (10.11.2/10.11.3), transmission and SNI certificates doesn't work properly. I will create a bug report later.

mike.dld
Transmission Developer
Posts: 275
Joined: Wed Dec 25, 2013 10:56 pm

Re: TLS SNI problem on OS X

Post by mike.dld » Mon Mar 21, 2016 2:30 pm

Crank wrote:... Nevertheless, I took a deeper look into the sourcecode of transmission. Transmission uses a very old cURL library: https://trac.transmissionbt.com/browser ... rl.4.dylib cURL supports SNI certificates on version 7.39 and above. The library in the transmission source is to old. I think in most cases Transmission uses the local cURL installation on the OS, except on OS X, where it doesn't work. To fix this issue, a cURL update (libcurl.4.dylib) would be enough I think ...
T always uses system libcurl on OS X. There is no complete libcurl library on our SVN, just a stub library (taken from Xcode SDK for OS X ~10.6) to aid in resolving symbols during build. And since openssl API we use is stable enough (despite there being multiple forks recently) and didn't change in years, there is no point in updating this stub library and headers.

Note thought, that you updating openssl (BTW, there's no official way of doing so, so how did you update it exactly?) will have no effect on libcurl as the latter doesn't use openssl on 10.11 and 10.10 (and maybe even earlier versions I don't have at hand, since Apple probably moved to CommonCrypto API and Security framework sometime around 10.7 or 10.8):

Code: Select all

$ otool -L /usr/lib/libcurl.4.dylib
/usr/lib/libcurl.4.dylib:
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 8.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 57330.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1230.0.0)
	/System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.4.0)
	/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 6.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.0.0)
So, if you really want to update openssl used by T (not directly (which is another deal), but through libcurl) on OS X you'll need to replace system libcurl as well (on your own risk, I'm not advising anything here); or play with DYLD_LIBRARY_PATH; or build your own T linking statically to both openssl and libcurl.

Crank
Posts: 11
Joined: Sun Mar 20, 2016 6:22 pm

Re: TLS SNI problem on OS X

Post by Crank » Mon Mar 21, 2016 3:04 pm

What has cURL to do with OpenSSL? These are two different librarys and they are doing different jobs. I can access cloudflare protected sites in my browser without problems. It seems that only T has problems with it. And this pointed me to libcurl library. Maybe the local access to openssl and libcurl fails and T uses the files from its source? I'm not an expert in C++ coding, but there is a problem on OS X systems definitely (I'm not the only one). If I've some time I will try to replace the old libcurl version and compile T by myself.
If T would really use the local openssl and cURL on OS X, there would be no problem, because both versions are supporting SNI-Certs.

For updating openssl on OS X I used http://brew.sh/. After installing the new openssl I replaced the binary with the new one. cURL version on my system is 7.43, which is new enough.
Maybe there is a way to get some debug information, I searched but couldn't find information for debug mode on OS X. I think the tls handshake is failing, which results into the "cant connect to tracker" warning.

x190
Posts: 5094
Joined: Sun Nov 30, 2008 4:59 am

Re: TLS SNI problem on OS X

Post by x190 » Mon Mar 21, 2016 3:15 pm

The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

mike.dld
Transmission Developer
Posts: 275
Joined: Wed Dec 25, 2013 10:56 pm

Re: TLS SNI problem on OS X

Post by mike.dld » Mon Mar 21, 2016 3:41 pm

You're talking about trackers. T makes requests to trackers using libcurl (not directly using openssl for this on its own). Now, libcurl doesn't use openssl as I demonstrated. The questions is then, why did you try updating openssl and expect to see the difference?

It's not enough to tell that you have version 7.43. It may be any version, but if it doesn't use openssl then there's no point in stating that openssl supports SNI. Could you (in addition to what x190 suggests) execute `otool` the way I did above on both libcurl and Transmission binaries to make sure that they use the libraries you think they are using and not something else?

Code: Select all

$ otool -L /usr/lib/libcurl.4.dylib
...
$ otool -L /Applications/Transmission.app/Contents/MacOS/Transmission
...
Answering your other question, and reiterating what is already stated before, libraries (.dylib files) on SVN are not full libraries. They only export symbols, but don't implement any functionality, so there is no way those files are used on your system because then nothing would've worked at all.

Crank
Posts: 11
Joined: Sun Mar 20, 2016 6:22 pm

Re: TLS SNI problem on OS X

Post by Crank » Mon Mar 21, 2016 7:17 pm

Code: Select all

$ otool -L /usr/lib/libcurl.4.dylib
/usr/lib/libcurl.4.dylib:
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 8.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 57330.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1230.0.0)
	/System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.4.0)
	/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 6.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.0.0)

Code: Select all

$ otool -L /Applications/Transmission.app/Contents/MacOS/Transmission
/Applications/Transmission.app/Contents/MacOS/Transmission:
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 22.0.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	@loader_path/../Frameworks/Sparkle.framework/Versions/A/Sparkle (compatibility version 1.5.0, current version 1.5.0)
	@executable_path/../Frameworks/Growl.framework/Versions/A/Growl (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
	/usr/lib/libcurl.4.dylib (compatibility version 6.0.0, current version 6.1.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 57337.20.43)
	/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 157.0.0)
	/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1256.1.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1404.32.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1256.14.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 728.6.0)
	/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)
I will check the debugging with the environment variables now. If you want test the issue by yourself, I can provide you a torrent file if you want.

Crank
Posts: 11
Joined: Sun Mar 20, 2016 6:22 pm

Re: TLS SNI problem on OS X

Post by Crank » Mon Mar 21, 2016 7:43 pm

Hm, not sure how I'm able to set the "TR_CURL_VERBOSE" to get into the debug mode. Sorry. :/

x190
Posts: 5094
Joined: Sun Nov 30, 2008 4:59 am

Re: TLS SNI problem on OS X

Post by x190 » Mon Mar 21, 2016 10:22 pm

The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

Crank
Posts: 11
Joined: Sun Mar 20, 2016 6:22 pm

Re: TLS SNI problem on OS X

Post by Crank » Tue Mar 22, 2016 1:40 am

Thanks for help x190. The result is the following:

Code: Select all

*   Trying 104.24.106.127...
* Connected to xyz.com (104.24.106.127) port 443 (#0)
* Unknown SSL protocol error in connection to xyz.com:-9838
* Closing connection 0
CoreAnimation: warning, deleted thread with uncommitted CATransaction; set CA_DEBUG_TRANSACTIONS=1 in environment to log backtraces.
* Hostname xyz.com was found in DNS cache
*   Trying 104.24.106.127...
* Connected to xyz.com (104.24.106.127) port 443 (#1)
* Unknown SSL protocol error in connection to xyz.com:-9838
* Closing connection 1
Then I used the CA_DEBUG_TRANSACTIONS Flag with results into this:

Code: Select all

CoreAnimation: warning, deleted thread with uncommitted CATransaction; created by:
0   QuartzCore                          0x00007fff95d888b6 _ZN2CA11Transaction4pushEv + 318
1   QuartzCore                          0x00007fff95da3372 _ZN2CA11Transaction15ensure_implicitEv + 276
2   QuartzCore                          0x00007fff95d8ec85 _ZN2CA5Layer13thread_flags_EPNS_11TransactionE + 37
3   QuartzCore                          0x00007fff95d8ebd4 _ZN2CA5Layer4markEPNS_11TransactionEjj + 64
4   QuartzCore                          0x00007fff95d90ae3 _ZN2CA5Layer25set_needs_display_in_rectERK6CGRect + 333
5   QuartzCore                          0x00007fff95d90994 -[CALayer setNeedsDisplayInRect:] + 25
6   AppKit                              0x00007fff990b48f5 _NSBackingLayerSetNeedsDisplayInRect + 319
7   AppKit                              0x00007fff990b47b0 -[_NSBackingLayer setNeedsDisplayInRect:] + 61
8   QuartzCore                          0x00007fff95d90975 -[CALayer setNeedsDisplay] + 62
9   AppKit                              0x00007fff990b4763 -[NSView(NSInternal) _setLayerNeedsDisplayInViewRect:] + 606
10  AppKit                              0x00007fff990984ee NSViewSetNeedsDisplayInRect + 945
11  AppKit                              0x00007fff99098136 -[NSView setNeedsDisplayInRect:] + 48
12  AppKit                              0x00007fff99097f4d -[NSView setNeedsDisplay:] + 81
13  Transmission                        0x000000010dc9bbd0 Transmission + 293840
14  Foundation                          0x00007fff948fdc6f __NSThread__start__ + 1351
15  libsystem_pthread.dylib             0x00007fff88073c13 _pthread_body + 131

mike.dld
Transmission Developer
Posts: 275
Joined: Wed Dec 25, 2013 10:56 pm

Re: TLS SNI problem on OS X

Post by mike.dld » Tue Mar 22, 2016 8:53 am

As I presume you have libcurl installed via homebrew as well, you could try rebuilding it to use the new openssl (if you didn't do that yet) and make T use it, and see if that helps:

Code: Select all

$ brew reinstall curl --with-openssl
$ otool -L /usr/local/opt/curl/lib/libcurl.4.dylib
(confirm that it depends on /usr/local/opt/openssl/lib/... libraries)
$ env DYLD_LIBRARY_PATH=/usr/local/opt/curl/lib /Applications/Transmission.app/Contents/MacOS/Transmission

x190
Posts: 5094
Joined: Sun Nov 30, 2008 4:59 am

Re: TLS SNI problem on OS X

Post by x190 » Tue Mar 22, 2016 11:01 am

Doubt this has anything to do with Transmission --- Firefox doesn't like that IP either, nor does curl or openssl s_client (no certificate is presented). And something's missing from the logs because xyz.com (now xyz.xyz) doesn't resolve to 104.etc and neither one are trackers? Course my curl and openssl is pretty old, so good luck with the 'homebrew'.
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

Crank
Posts: 11
Joined: Sun Mar 20, 2016 6:22 pm

Re: TLS SNI problem on OS X

Post by Crank » Tue Mar 22, 2016 11:17 am

I will try to rebuild curl later, but I don't think this change anything.
x190 wrote:Doubt this has anything to do with Transmission --- Firefox doesn't like that IP either, nor does curl or openssl s_client (no certificate is presented). And something's missing from the logs because xyz.com (now xyz.xyz) doesn't resolve to 104.etc and neither one are trackers? Course my curl and openssl is pretty old, so good luck with the 'homebrew'.
Of course xyz.com isn't the real tracker address. For security reasons I changed this. As I wrote before, I can provide you a torrent file if you want (private message) for testing.

x190
Posts: 5094
Joined: Sun Nov 30, 2008 4:59 am

Re: TLS SNI problem on OS X

Post by x190 » Tue Mar 22, 2016 3:53 pm

Aaargh, late night and meaningless log posts! Issue could be between tracker server and cloudflare or an os x thing requiring tweaking obscure apache files.

For cloudflare support:
https://support.cloudflare.com/hc/en-us ... t-occurred

Search terms:
cloudflare +trackers +os x

If you think I can get it to work at all, please PM me a sample torrent.

EDIT:
If I disable tls and use the simple https-announce, it works on every system without problems.
Somewhere last night I read that you can compile this in/out of a certain library, but can't recall details. Where are you making these changes?

https://developer.apple.com/library/mac ... .1ssl.html
TLS extensions are only supported in OpenSSL 0.9.8 if they are explictly enabled at compile time
using for example the enable-tlsext switch.
https://www.webhostingtalk.com/showthread.php?t=1421536
https://curl.haxx.se/libcurl/c/CURLOPT_SSLVERSION.html
The stone age didn’t end because we ran out of stones.
---The Great Disruption - by Paul Gilding
https://paulgilding.com/
-------------------------------------------------

Post Reply