`node/openssl/ssl.h` should compile without any additional defines #40575

mmomtchev · 2021-10-23T09:08:58Z

Version

17.0.1

Platform

5.11.0-38-generic #42~20.04.1-Ubuntu SMP

Subsystem

No response

What steps will reproduce the bug?

node/openssl/ssl.h is exported in the node-gyp bundle to be used by native modules that need to link against the openssl symbols in Node

Currently, when building Node.js itself some of these defines come from openssl.gyp: OPENSSL_API_COMPAT and OPENSSL_CONFIGURED_API

This means that a native addon must also define them and must do so in a manner compatible with Node.js across the eventual future versions

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior?

These defines should be present somewhere in an additional include file to be added to the node-gyp bundle distribution

What do you see instead?

No response

Additional information

No response

The text was updated successfully, but these errors were encountered:

mhdawson · 2021-10-28T22:11:52Z

@danbev thoughts on this?

danbev · 2021-10-29T04:26:41Z

@danbev thoughts on this?

I haven't looked at node-gyp at all during the OpenSSL work so if this is something that is not covered by the OpenSSL tests in test/addons (openssl-binding, openssl-client-cert-engine, or openssl-key-engine) then I've missed it I'm afraid.

I added OPENSSL_API_COMPAT to node.gypi to avoid warnings due to the usage of deprecated APIs. We could remove this and live with the warnings but since we need to keep backward compatibility with OpenSSL 1.1.1 for some time and it might become annoying seeing these.

mmomtchev · 2021-10-29T13:51:58Z

@danbev You should be able to include node/openssl/ssl.h from a standalone application from outside the Node.js tree

This is the test:

#include "openssl/ssl.h"

int main() {
	return 0;
}

If you have ever compiled a native module with 17.0.1 you will have this directory:

gcc -I${HOME}/.cache/node-gyp/17.0.1/include/node -o openssl openssl.c

${HOME}/Caches on macOS

Normally, the OpenSSL of node-gyp should be a drop-in replacement for a system-installed OpenSSL - it allows to build a native addon that uses OpenSSL without linking in a second copy in the Node executable - which might not work at all

d3x0r · 2021-11-07T18:40:35Z

On windows, and alpine linux docker image, updating to node 17...

After getting around OPENSSL_API_COMPAT by just defining it in CMakeList for it (and the binding.gyp), I ran into

  error "The requested API level higher than the configured API compatibility level"

which OPENSSL_CONFIGURED_API is undefined... so being 0, anything I set as a compatibility level fails.

adding this to the top of ....cmake-js\node-x64\v17.0.1\include\node\openssl\macros.h

#include "./configuration.h"

(which I don't think is a good or correct solution)

But even then I had to define the OPENSSL_MIN_API=30000 in my build configuration

Edit: Sorry for being terse and vague.

After tracking down this, and getting some hints, I made forward progress... there's still missing symbols (RSA_new() and RSA_free() for examples). There's a openssl/configuration.h file which is new in 17 that was not in 16. This configuration.h is not included by anything in openssl directory, and seems the user is supposed to include this file now too?

mscdex · 2022-02-05T02:00:39Z

Just chiming in that I'm also seeing this with node v17.x ("OPENSSL_API_COMPAT expresses an impossible API compatibility level").

Any (official) solutions to this? /cc @nodejs/crypto

d3x0r · 2022-02-05T04:43:28Z

Just chiming in that I'm also seeing this with node v17.x ("OPENSSL_API_COMPAT expresses an impossible API compatibility level").

Any (official) solutions to this? /cc @nodejs/crypto

"OPENSSL_API_COMPAT=10101"

and

#if NODE_MAJOR_VERSION >= 17
#  include <openssl/configuration.h>
#endif

before including anything from OpenSSL. If configuration.h is not defined than the configured version is undefined and becomes a '0' which everything is bigger than the configured level then.

mmomtchev · 2022-02-08T16:16:10Z

@danbev How about including #define OPENSSL_API_COMPAT=10101 in node.h?

danbev · 2022-02-23T07:19:16Z

@danbev How about including #define OPENSSL_API_COMPAT=10101 in node.h?

Sorry about the late reply. I'm currently on a leave of absence and will be back working in April, and would be happy to take a closer look at this then (unless someone else beats me to it).

mscdex · 2022-03-08T07:31:23Z

"OPENSSL_API_COMPAT=10101"

and
#if NODE_MAJOR_VERSION >= 17
#  include <openssl/configuration.h>
#endif
before including anything from OpenSSL. If configuration.h is not defined than the configured version is undefined and becomes a '0' which everything is bigger than the configured level then.

Unfortunately that doesn't seem to work for me, I still get the same error.

mscdex · 2022-03-09T03:55:15Z

Looks like part of the problem in my particular case is that homebrew on macos is using OpenSSL 1.1.1 with their node v17.x formula, which causes problems for addons because of the OpenSSL 3.0 headers that get pulled in by node-gyp. The addon thinks it's compiling for OpenSSL 3.0 but then during runtime you can get various errors from dyld depending on if you are using new OpenSSL APIs or OpenSSL APIs that are now aliases in 3.0.

danbev · 2022-05-10T12:18:22Z

I've taken a look at this and the following is what I've found.

Looking at the the error when running node-gyp build (with the addition of some
debugging of the macros involved to help troubleshoot this):

/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:83:9: note: ‘#pragma message: OPENSSL_API_COMPAT: 30000’
   83 | #pragma message "OPENSSL_API_COMPAT: " OPENSSL_MSTR(OPENSSL_API_COMPAT)
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:141:9: note: ‘#pragma message: OPENSSL_VERSION_MAJOR: 3’
  141 | #pragma message "OPENSSL_VERSION_MAJOR: " OPENSSL_MSTR(OPENSSL_VERSION_MAJOR)
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:142:9: note: ‘#pragma message: OPENSSL_VERSION_MINOR: 0’
  142 | #pragma message "OPENSSL_VERSION_MINOR: " OPENSSL_MSTR(OPENSSL_VERSION_MINOR)
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:143:9: note: ‘#pragma message: OPENSSL_VERSION: (3 * 1000 + OPENSSL_MINOR * 100)’
  143 | #pragma message "OPENSSL_VERSION: " OPENSSL_MSTR((OPENSSL_VERSION_MAJOR * 1000 + OPENSSL_MINOR * 100))
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:144:9: note: ‘#pragma message: OPENSSL_API_LEVEL: (30000)’
  144 | #pragma message "OPENSSL_API_LEVEL: " OPENSSL_MSTR(OPENSSL_API_LEVEL)
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:145:9: note: ‘#pragma message: OPENSSL_API_COMPAT: 30000’
  145 | #pragma message "OPENSSL_API_COMPAT: " OPENSSL_MSTR(OPENSSL_API_COMPAT)
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:146:9: note: ‘#pragma message: OPENSSL_CONFIGURED_API: OPENSSL_CONFIGURED_API’
  146 | #pragma message "OPENSSL_CONFIGURED_API: " OPENSSL_MSTR(OPENSSL_CONFIGURED_API)
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:148:4: error: #error "The requested API level higher than the configured API compatibility level"
  148 | #  error "The requested API level higher than the configured API compatibility level"
      |    ^~~~~
make: *** [example.target.mk:113: Release/obj.target/example/example.o] Error 1

Notice that OPENSSL_CONFIGURED_API has not been set which is the cause of
the error message above.

Now, macros.c has the following include:

#include <openssl/opensslconf.h>                                                
#include <openssl/opensslv.h>

In Node.js this is a "redirect" openssl header which choose different headers
depending of whether ASM of NO_ASM was configured:

#if defined(OPENSSL_NO_ASM)                                                        
# include "./opensslconf_no-asm.h"                                                 
#else                                                                              
# include "./opensslconf_asm.h"                                                    
#endif

And if we assume this using ASM this would end up in:

#elif defined(OPENSSL_LINUX) && defined(__x86_64__)                                
# include "./archs/linux-x86_64/asm/include/openssl/opensslconf.h

And this file does not include configuration.h.

In OpenSSL the following headers are generated:

$ find . -name '*.in'
./include/openssl/conf.h.in
./include/openssl/lhash.h.in
./include/openssl/crmf.h.in
./include/openssl/err.h.in
./include/openssl/safestack.h.in
./include/openssl/ess.h.in
./include/openssl/opensslv.h.in
./include/openssl/ocsp.h.in
./include/openssl/pkcs12.h.in
./include/openssl/asn1.h.in
./include/openssl/bio.h.in
./include/openssl/x509.h.in
./include/openssl/x509v3.h.in
./include/openssl/cmp.h.in
./include/openssl/pkcs7.h.in
./include/openssl/srp.h.in
./include/openssl/x509_vfy.h.in
./include/openssl/asn1t.h.in
./include/openssl/ct.h.in
./include/openssl/fipskey.h.in
./include/openssl/ui.h.in
./include/openssl/configuration.h.in
./include/openssl/crypto.h.in
./include/openssl/cms.h.in
./include/openssl/ssl.h.in
./include/crypto/bn_conf.h.in
./include/crypto/dso_conf.h.in

And notice that opensslconf.h is not among these.
What I think is happening is that we have an old opensslconf.h which may
have been generated with past versions but is not generated any more, instead
configuration.h is the file being generated.

If we replace this header,
/.cache/node-gyp/17.8.0/include/node/openssl/opensslconf.h, with the header
from OpenSSL 3.0:

$ cp opensslconf.h /home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/opensslconf.h

And the run the build again:

$ node-gyp build
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | linux | x64
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory '/home/danielbevenius/work/nodejs/node-gyp-openssl-issue/build'
  CC(target) Release/obj.target/example/example.o
In file included from /home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/././archs/linux-x86_64/asm/include/openssl/ssl.h:21,
                 from /home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/./ssl_asm.h:11,
                 from /home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/ssl.h:4,
                 from ../example.c:1:
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:83:9: note: ‘#pragma message: OPENSSL_API_COMPAT: 30000’
   83 | #pragma message "OPENSSL_API_COMPAT: " OPENSSL_MSTR(OPENSSL_API_COMPAT)
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:141:9: note: ‘#pragma message: OPENSSL_VERSION_MAJOR: 3’
  141 | #pragma message "OPENSSL_VERSION_MAJOR: " OPENSSL_MSTR(OPENSSL_VERSION_MAJOR)
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:142:9: note: ‘#pragma message: OPENSSL_VERSION_MINOR: 0’
  142 | #pragma message "OPENSSL_VERSION_MINOR: " OPENSSL_MSTR(OPENSSL_VERSION_MINOR)
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:143:9: note: ‘#pragma message: OPENSSL_API_LEVEL: (30000)’
  143 | #pragma message "OPENSSL_API_LEVEL: " OPENSSL_MSTR(OPENSSL_API_LEVEL)
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:144:9: note: ‘#pragma message: OPENSSL_API_COMPAT: 30000’
  144 | #pragma message "OPENSSL_API_COMPAT: " OPENSSL_MSTR(OPENSSL_API_COMPAT)
      |         ^~~~~~~
/home/danielbevenius/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:145:9: note: ‘#pragma message: OPENSSL_CONFIGURED_API: 30000’
  145 | #pragma message "OPENSSL_CONFIGURED_API: " OPENSSL_MSTR(OPENSSL_CONFIGURED_API)
      |         ^~~~~~~
  SOLINK_MODULE(target) Release/obj.target/example.node
  COPY Release/example.node
make: Leaving directory '/home/danielbevenius/work/nodejs/node-gyp-openssl-issue/build'
gyp info ok

Solution

So we should remove this template from Node.js:

$ git st .
On branch opensslconf
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	deleted:    deps/openssl/config/opensslconf.h
	deleted:    deps/openssl/config/opensslconf.h.tmpl
	deleted:    deps/openssl/config/opensslconf_no-asm.h

We can create the tar-headers file (by making some updates to the Makefile
to allow this to be built for a non-release):

$ make node-v17.9.1.tar-headers

First we can revert the headers:

$ node-gyp remove v17.8.0

And running node-gyp rebuild should fail with the above error message.

And then unzip the new headers to replace the headers used by node-gyp:

$ tar xvzf node-v17.9.1-headers.tar.gz -C ~/.cache/node-gyp/17.8.0 --strip 1

And this running node-gyp should succeed:

$ node-gyp build
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | linux | x64
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory '/home/danielbevenius/work/nodejs/learning-nodejs/node-gyp-openssl-issue/build'
make: Nothing to be done for 'all'.
make: Leaving directory '/home/danielbevenius/work/nodejs/learning-nodejs/node-gyp-openssl-issue/build'

There are addons in Node.js that include openssl/ssl.h but this error does not
happen for those and I've not had time to find out why this is the case.

This commit removes the header template files opensslconf.h as it is not a generated header anymore (configuration.h is the generated header now). The motivation for this is that not using opensslconf.h from the OpenSSL include directory means an addon that includes openssl/ssl.h will get the following error: ~/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:148:4: error: compatibility level" 148 | # error "The requested API level higher than the configured API compatibility level" For more details please see nodejs#40575. Fixes: nodejs#40575

This commit removes the header template files opensslconf.h as it is not a generated header anymore (configuration.h is the generated header now). The motivation for this is that not using opensslconf.h from the OpenSSL include directory means an addon that includes openssl/ssl.h will get the following error: ~/.cache/node-gyp/17.8.0/include/node/openssl/macros.h:148:4: error: compatibility level" 148 | # error "The requested API level higher than the configured API compatibility level" For more details please see #40575. PR-URL: #43035 Fixes: #40575 Reviewed-By: Michael Dawson <[email protected]>

BethGriggs added the openssl Issues and PRs related to the OpenSSL dependency. label Nov 16, 2021

paullouisageneau mentioned this issue Dec 2, 2021

libdatachannel does not compile because of an OpenSSL error with node 17 murat-dogan/node-datachannel#65

Closed

jwulf mentioned this issue Dec 14, 2021

Does not install with Node 17 camunda-community-hub/zeebe-client-node-js#247

Closed

mmomtchev mentioned this issue Jan 11, 2022

Native modules ABI compatibility problems on Node 17 with OpenSSL 3 / OpenSSL 1 #41410

Closed

d3x0r mentioned this issue May 2, 2022

OpenSSL include issues nodejs/node-gyp#2576

Closed

richardlau mentioned this issue May 4, 2022

End-of-Life dates of Node.js 16 and OpenSSL 1.1.1 do not align nodejs/TSC#1222

Closed

danbev self-assigned this May 10, 2022

danbev mentioned this issue May 10, 2022

deps: remove opensslconf template headers #43035

Closed

danbev closed this as completed in 556df3d May 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`node/openssl/ssl.h` should compile without any additional defines #40575

`node/openssl/ssl.h` should compile without any additional defines #40575

mmomtchev commented Oct 23, 2021

mhdawson commented Oct 28, 2021

danbev commented Oct 29, 2021

mmomtchev commented Oct 29, 2021 •

edited

Loading

d3x0r commented Nov 7, 2021 •

edited

Loading

mscdex commented Feb 5, 2022

d3x0r commented Feb 5, 2022 •

edited

Loading

mmomtchev commented Feb 8, 2022

danbev commented Feb 23, 2022

mscdex commented Mar 8, 2022 •

edited

Loading

mscdex commented Mar 9, 2022

danbev commented May 10, 2022

node/openssl/ssl.h should compile without any additional defines #40575

node/openssl/ssl.h should compile without any additional defines #40575

Comments

mmomtchev commented Oct 23, 2021

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior?

What do you see instead?

Additional information

mhdawson commented Oct 28, 2021

danbev commented Oct 29, 2021

mmomtchev commented Oct 29, 2021 • edited Loading

d3x0r commented Nov 7, 2021 • edited Loading

mscdex commented Feb 5, 2022

d3x0r commented Feb 5, 2022 • edited Loading

mmomtchev commented Feb 8, 2022

danbev commented Feb 23, 2022

mscdex commented Mar 8, 2022 • edited Loading

mscdex commented Mar 9, 2022

danbev commented May 10, 2022

Solution

`node/openssl/ssl.h` should compile without any additional defines #40575

`node/openssl/ssl.h` should compile without any additional defines #40575

mmomtchev commented Oct 29, 2021 •

edited

Loading

d3x0r commented Nov 7, 2021 •

edited

Loading

d3x0r commented Feb 5, 2022 •

edited

Loading

mscdex commented Mar 8, 2022 •

edited

Loading