[0.60.x] Secure LDAP connection issues - Error: socket hang up

[tobru]

Description:

Connections to the LDAP server using TLS or StartTLS don't work.

Server Setup Information:

+------------------------------------------+
|              SERVER RUNNING              |
+------------------------------------------+
|                                          |
|  Rocket.Chat Version: 0.60.2             |
|       NodeJS Version: 8.9.3 - x64        |
|             Platform: linux              |
|         Process Port: 3000               |
|             Site URL: https://hidden     |
|     ReplicaSet OpLog: Disabled           |
|          Commit Hash: 2149a6c78d         |
|        Commit Branch: HEAD               |
|                                          |
+------------------------------------------+

• Deployment Method(snap/docker/tar/etc): Docker
• Number of Running Instances: 1

Steps to Reproduce:

The problem started to show up after upgrading from 0.59 to 0.60, it worked like a charm with 0.59.

Expected behavior:

Secure connections to LDAP work.

Actual behavior:

Connection doesn't work with the message "Error: socket hang up".

Relevant logs:

rocketchat_logger rocketchat_logger.js:278 LDAP ➔ Connection.error connection { Error: socket hang up
  at TLSSocket.onHangUp (_tls_wrap.js:1135:19)
  at Object.onceWrapper (events.js:313:30)
  at emitNone (events.js:111:20)
  at TLSSocket.emit (events.js:208:7)
  at endReadableNT (_stream_readable.js:1056:12)
  at _combinedTickCallback (internal/process/next_tick.js:138:11)
  at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  code: 'ECONNRESET',
  path: undefined,
  host: 'ldap.mydomain.com',
  port: 636,
  localAddress: undefined }
rocketchat_logger rocketchat_logger.js:278 LDAPHandler ➔ error { Error: socket hang up
  at TLSSocket.onHangUp (_tls_wrap.js:1135:19)
  at Object.onceWrapper (events.js:313:30)
  at emitNone (events.js:111:20)
  at TLSSocket.emit (events.js:208:7)
  at endReadableNT (_stream_readable.js:1056:12)
  at _combinedTickCallback (internal/process/next_tick.js:138:11)
  at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  code: 'ECONNRESET',
  path: undefined,
  host: 'ldap.mydomain.com',
  port: 636,
  localAddress: undefined }
Exception in callback of async function: Error: 140350016636736:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:1500:SSL alert number 40
140350016636736:error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:1217:



[Wed Jan 03 2018 09:00:31 GMT+0000 (UTC)] ERROR Error: 140350016636736:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:1500:SSL alert number 40
140350016636736:error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:1217:

I checked if connections to the LDAP server are working using OpenSSL, and they do:

$ openssl s_client -connect ldap.mydomain.com:636 | openssl x509 -noout -text
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL Wildcard, CN = *.mydomain.com
verify return:1
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            XXX
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=GB, ST=Greater Manchester, L=Salford, O=COMODO CA Limited, CN=COMODO RSA Domain Validation Secure Server CA
        Validity
            Not Before: Sep  6 00:00:00 2017 GMT
            Not After : Sep  6 23:59:59 2019 GMT
        Subject: OU=Domain Control Validated, OU=PositiveSSL Wildcard, CN=*.mydomain.com
[...]

Settings:

[AmShaegar13]

Using 0.60.2 it is working for me. I am using the same settings except for the CA Cert. I filled it because we used an internal CA to sign the LDAP cert.

[tobru]

Doing some more debugging doesn't reveal much more. Here are some more outputs I gathered:

$ openssl s_client -connect ldap.mydomain.com:636
CONNECTED(00000003)
[...]
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
---
SSL handshake has read 5378 bytes and written 459 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: XXX
    Session-ID-ctx: 
    Master-Key: XXX
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: XXX
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

So the server properly supports TLSv1.2 and the connection can be established.

Doing some tcpdumps when Rocket.Chat connects via LDAPS reveals this:

Secure Sockets Layer
    TLSv1.2 Record Layer: Handshake Protocol: Multiple Handshake Messages
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 4661
        Handshake Protocol: Server Hello
            Handshake Type: Server Hello (2)
            Length: 83
            Version: TLS 1.2 (0x0303)
            Random: XXX...
            Session ID Length: 32
            Session ID: XXX...
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
            Compression Method: null (0)
            Extensions Length: 11
            Extension: ec_point_formats (len=2)
            Extension: renegotiation_info (len=1)
        Handshake Protocol: Certificate
            Handshake Type: Certificate (11)
            Length: XXXX
            Certificates Length: XXXX
            Certificates (XXXX bytes)
    TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Handshake Failure)
        Content Type: Alert (21)
        Version: TLS 1.2 (0x0303)
        Length: 2
        Alert Message
            Level: Fatal (2)
            Description: Handshake Failure (40)

This corresponds to the errors seen in the logfile of Rocket.Chat. I was not yet able to get more information about the error out of the LDAP server (389DS), but I keep on trying.

In the meantime: Has anything changed in 0.60.x regarding TLS handling?

[tobru]

While travelling back in time and check old logfiles of the LDAP server (389DS) I could see connections from Rocket.Chat before upgrading to 0.60.x:

[27/Dec/2017:22:48:49.210441564 +0100] conn=2314393 fd=66 slot=66 SSL connection from XXX to YYY
[27/Dec/2017:22:48:49.233174192 +0100] conn=2314393 TLS1.2 256-bit AES-GCM
[27/Dec/2017:22:48:49.245526639 +0100] conn=2314393 op=0 BIND dn="XXX" method=128 version=3
[...]
[27/Dec/2017:22:48:50.313772638 +0100] conn=2314393 op=3 UNBIND
[27/Dec/2017:22:48:50.313800225 +0100] conn=2314393 op=3 fd=66 closed - U1

So according to this log lines TLS1.2 256-bit AES-GCM was used for the connection. It seems that since the upgrade this combination is not offered by Rocket.Chat anymore?

Just for the sake of completeness: I'm running Rocket.Chat in Docker, using the official Docker Image from here: https://hub.docker.com/r/rocketchat/rocket.chat/

[tobru]

Finally found the matching log entry:

[04/Jan/2018:09:56:47.751925124 +0100] conn=3108534 fd=72 slot=72 SSL connection from XXX to XXX
[04/Jan/2018:09:56:47.752378779 +0100] conn=3108534 op=-1 fd=72 closed - Cannot communicate securely with peer: no common encryption algorithm(s).

So there where definitively changes in either Rocket.Chat, the NodeJS TLS library or the LDAP library. I think it would be great to have an advanced configuration option to choose which cipher suites should be used.

[AmShaegar13]

Makes sense. Maybe this has something to do with the upgrade to node v8 or meteor v1.6. Sadly, I do not understand enough of meteor to judge what this change means: 7035293

Apparently, ldapjs was required as a meteor package in v1.0.0 and is now used as an npm package v1.0.1.

[tobru]

Yeah, I also guess that it has something to do with a newer version of NodeJS and/or a library. So the best thing IMHO would be the possibility to chose the available cipher suites.

[AmShaegar13]

But the previously working cipher is already included as you can see here:
https://nodejs.org/dist/latest-v8.x/docs/api/tls.html#tls_modifying_the_default_tls_cipher_suite

[tobru]

I see. No idea why this doesn't work anymore...

[rodrigok]

@tobru Which exactly version you was using? 0.59.(?)

[tobru]

I would say it was 0.59.1

[rodrigok]

Related to DevExpress/testcafe-hammerhead#1403

The problem was introduced by NodeJS 8.6, need to set ecdhCurve: 'auto' in TLS configuration

[rodrigok]

Related nodejs/node#16853

[tobru]

0.60.4-rc.0, especially commit 5d1b628, has a fix for this issue. Thank you very much @rodrigok for finding an fixing this nasty issue. I'm glad that Rocket.Chat has such a great community and people working on it 👍

[rodrigok]

Closed via #9343

[rodrigok]

@tobru Thanks, it's awesome to have a community that understand and help us to fix the issues.