SQL Anywhere 16: TLS handshake failed, error code 20

[Former Member]

I'm testing TLS connections on SQL Anywhere 16.0.0.2158 and am getting a TLS handshake failed, error code 20 error message.  I've read the following documents with no help:

http://dcx.sap.com/index.html#sa160/en/dbadmin/tls-j22-s-5729723.html

and

http://dcx.sap.com/index.html#sa160/en/dbadmin/gencert-ml-ref1.html

and

http://dcx.sap.com/index.html#sa160/en/dbadmin/ml-tls-s-6232604.html

here is the client log

Fri Sep 25 2015 14:52:03

14:52:03 Attempting to connect using:

UID=ficsro;PWD=********;DBN=fics;ServerName=billytest;CON=SQL_DBC_4c382e5200;ENC='TLS(tls_type=rsa;fips=n;trusted_certificate=C:\ssl\public.pem)';LOG=c:\ssl\ssl.log;LINKS='tcpip(HOST=web1)';CPOOL=NO

14:52:03 Attempting to connect to a running server...

14:52:03 Attempting TCPIP connection (address 192.168.4.112:2638 found in sasrv.ini cache)

14:52:03 Looking for server with name billytest

14:52:03 Trying to find server at cached address 192.168.4.112:2638 without broadcasting

14:52:03 Found database server billytest on TCPIP link

14:52:03 Connected using client address 192.168.5.150:52913

14:52:03 Connected to server over TCPIP

14:52:03 Connected to SQL Anywhere Server version 16.0.0.2158

14:52:03 Application information:

14:52:03 IP=192.168.5.150;HOST=staging11;OSUSER=estatuswebsvc;OS='Windows 2012R2 Build 9200 ';EXE=C:\ColdFusion11\estatuswebsvc\bin\coldfusion.exe;PID=0xab4;THREAD=0xaec;VERSION=16.0.0.2158;API=iAnywhereJDBC;TIMEZONEADJUSTMENT=-300

14:52:03 Connected to the server, attempting to connect to a running database...

14:52:03 The TLS handshake failed, error code 20

14:52:03 Communication function SQLPresSyncPoint code 8

14:52:03   unknown error 0

14:52:03 Client disconnected

14:52:03 Disconnected from server

here are the server startup options which startup just fine:

-c 128M

-ec none,simple,TLS(identity=c:\db\identity.pem;identity_password=fics)

-n billytest

-x tcpip

c:\db\fics.db

here is the test certificate info:

C:\Program Files\SQL Anywhere 16\Bin64>createcert -t rsa

SQL Anywhere X.509 Certificate Generator Version 16.0.0.2158

Warning: The certificate will not be compatible with older versions

of the software including version 12.0.1 prior to build 3994 and version 16.0

prior to build 1691. Use the -3des switch if you require compatibility.

Enter RSA key length (512-16384): 2048

Generating key pair...

Country Code: US

State/Province: TX

Locality: ADDISON

Organization: FICS,INC

Organizational Unit: FICS

Common Name: web1

Enter file path of signer's certificate:

Certificate will be a self-signed root

Serial number [generate GUID]:

Generated serial number: f3dde00072d04f319b17cd429769b75e

Certificate valid for how many years (1-100): 99

Certificate Authority (Y/N) [N]:

1.  Digital Signature

2.  Nonrepudiation

3.  Key Encipherment

4.  Data Encipherment

5.  Key Agreement

6.  Certificate Signing

7.  CRL Signing

8.  Encipher Only

9.  Decipher Only

Key Usage [1,3,4,5]: 3,4,5

Enter file path to save certificate: c:\db\public.pem

Enter file path to save private key: c:\db\private.pem

Enter password to protect private key: fics

Enter file path to save identity: c:\db\identity.pem

[Former Member · Accepted solution]

I think I see the issue.  Since you are using the newer software to generate you own self-signed certificate then

the issue would seem to be you need to add "7 - CRL Signing" which is now required.  See the paragraph on

    "Self-signed certificates must now have the Certificate Signing attribute set"

     http://dcx.sap.com/index.html#sa160/en/sachanges/newsa-sa-16-nagano-sp-enhancements.html

[Former Member · Accepted solution]

Dang, I actually looked at a previous posting, http://scn.sap.com/thread/3551445 and saw the piece about that but totally misread it.  I didn't key in on "Certificate Signing".  It's actually 6 - Certificate Signing.  See below:

Certificate Authority (Y/N) [N]: N

1.  Digital Signature

2.  Nonrepudiation

3.  Key Encipherment

4.  Data Encipherment

5.  Key Agreement

6.  Certificate Signing

7.  CRL Signing

8.  Encipher Only

9.  Decipher Only

Key Usage [3,4,5]: 3,4,5,6

• Self-signed certificates must now have the Certificate Signing attribute set   Self-signed certificates must now have the Certificate Signing attribute set when using the identity encryption option (for example, the -x mlsrvXX and -xs dbsrvXX options). To determine if a certificate has the Certificate Signing attribute set, use the viewcert utility and look for Certificate Signing in the Key Usage portion of the output. If your self-signed certificates do not have the Certificate Signing attribute set, then you must regenerate the certificates. 

Thanks for you assistance Nick. I was able to test this out and everything works as specified.

[former_member182948]

Hi Susan,

I think that you ran into the issue of the next KBA.

http://service.sap.com/sap/support/notes/2108057

"2108057 - The TLS handshake failed with OpenSSL"

Thanks

[Former Member]

This message was moderated.

[Former Member]

Susan,

You posted about your problems accessing this KBA on a new thread (out of context) so I am adding a note in this thread on your behalf.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

I can reach that KBA now so I am not able to tell if your access was broken earlier or not.  You do need to be able to connect with your SAP Support Id.

If you are using a Certificate that originally worked with prior version (before 12.0.1#3986 or 16.0.0#1695) or if you have had this certificate for a while, then the KBA could explain this behavior.

Otherwise your handshake error is a TLS based one and would need to be investigated.

[Former Member]

Sorry about that.  I thought I had clicked "Reply" below that last post.  Our production environment is currently using 12.0.1.3851 and are using the sajdbc.jar, dbjdbc12.dll and dbrsa12.dll with the JDBC 3.0 driver class sybase.jdbc.sqlanywhere.IDriver.  This is a 2003 server with Java JDK 1.6.x.  I'm migrating applications to 2012 R2, Java JDK 1.8 and want to use the latest JDBC driver with hopes that performance and stability will be better.  I just dropped in the 3 above mentioned files on the 2012 server and it times out every time.  I'm guessing that the .jar is not compatible with JDK 1.8?  What if i go with sajdbc4.jar that came with 16.0.1324?  Is this build prior to the RSA changes (what was it, a switch to AES?)

By the way, I appreciate your assistance.