Would you consider merging a patch that optionally splits pybind11 into declarations vs definitions in a backwards compatible opt-in manner? #2322

cirosantilli2 · 2020-07-23T16:31:09Z

Edit: PR at: #2445

Hello!

TL;DR: a split into declarations/definitions would save a lot of time/money for the gem5 project which I work on. I already have a prototype which reduces build time by 40% from 25 minutes to 15 minutes.

If I were to send a patch that does this split, in an opt-in way that does not affect existing users at all, would you consider merging it?

I would like to know if this is reasonable before I sink time into something that won't get reviewed.

The proposed implementation is: I would just check if a macro if defined before inclusion of pybind11.hpp, say PYBIND11_HPP.

If the macro is defined, only declarations will be enabled, and therefore you won't get object redefinitions. So usage would be in a project that wants to opt-in would be like this.

In one single file of the project which gets linked at the end:

pybind11.cpp

#include <pybind11/pybind11.h>

This will be the only file to contain definitions of non-templated stuff that does not need to be defined in headers.

Then, on every other file of the project, include as:

#define PYBIND11_HPP 1
#include <pybind11/pybind11.h>

which will only expose declarations and not definitions, therefore preventing a ton of object redefinitions.

More details about the gem5 project can be found at: #708 (comment) maybe this issue is a duplicate of that one, but I'm not sure, feel free to close if you think yes. I'm just waiting for another long build and decided to annoy you a bit :-) If you promise to review, I will send a patch.

The text was updated successfully, but these errors were encountered:

virtuald · 2020-07-25T16:23:25Z

FWIW, the way that I deal with pybind11's long build times in dev is by using ccache. Of course, that's less useful in CI.

How ugly would the resulting code be?

cirosantilli2 · 2020-07-25T16:58:05Z

Hi Dustin!

Yes ccache can help, and I already use it locally, and could be in theory setup for CI, but I still feel that the gains of the split would be significant. I just need to check out random branches/rebase on master in separate worktrees too often.

About code ugliness, for every file I would split all definitions that don't need to be in the header (basically non-templates) into declaration + definition, and then group all definitions together at the bottom of each header file, and surround them with a single ifdef. So "not too ugly" I believe.

https://gem5-review.googlesource.com/c/public/gem5/+/29132 shows a hasty prototype that physically splits into .hpp and .cpp. I would basically take that and put the .cpp back inside the corresponding .hpp in an ifdef.

virtuald · 2020-07-25T17:24:25Z

I would love a 40% speedup. I definitely think it should be opt-in, since one of the reasons pybind11 is so easy to use is the fact that it's header-only. However, for users with large projects this seems great. I also maintain a fork of pybind11 that I include in my robotpy-build tool, which includes patches that haven't yet been merged upstream. If this weren't accepted upstream, I would potentially be interested in using such a fork.

Are the resulting libraries smaller or bigger?

It seems if one did this, would want to modify the existing CI to build/test both the split/unsplit version. I suppose looking at the build times in CI would be a proof that the speedup is real?

A project that I'm mildly involved with uses a .h/.inl paradigm for making the header files less messy, but perhaps it would be a good approach for this?

I like this because it makes the headers easier to read, and the split feels natural. For this, you could use a similar approach at the end of each file, instead of having the definitions there?

class options {
... 
};

#ifndef PYBIND11_HPP
  #include "options.hpp.inl"
#endif

I don't have any actual say about merging anything here, but I think this is a great idea if there aren't downsides and it results in real speedups.

cirosantilli2 · 2020-07-25T20:17:28Z

Are the resulting libraries smaller or bigger?

I didn't note that when I last tested, I'd need to build again to check, but I don't see how they can possibly be larger since there will be less object redefinition flying around, if anything they would be smaller.

It seems if one did this, would want to modify the existing CI to build/test both the split/unsplit version. I suppose looking at the build times in CI would be a proof that the speedup is real?

It would be great to see on this pybind11 CI too. I'll patch whatever tests people want here. I wonder if you can see it on very small examples though: you'd need a certain number of object redefinitions to notice it. But I can guarantee that I did a gem5 build before and after that change alone, under the exact same conditions otherwise, and the build time fell by 40% (the project has several dozens of small autogenerated files that include pybind11).

YannickJadoul · 2020-07-25T22:01:33Z

Hi @cirosantilli2. We (i.e., the new group of maintainers that are currently trying to clean up the backlog of issues and PRs, etc) were discussing this yesterday, but somehow failed to actually post our conclusion here. But, here we go:

This seems quite interesting, indeed, a 40% speedup, though it probably depends on the amount of translation units you are compiling? To me, it's especially surprising because I would've guessed all the template instantiations are what make things slow?

https://gem5-review.googlesource.com/c/public/gem5/+/29132 shows a hasty prototype that physically splits into .hpp and .cpp. I would basically take that and put the .cpp back inside the corresponding .hpp in an ifdef.

This is already the first important step: the possibility to seen and play around with such a prototype.

From the project's perspective, there are two main concerns:

Maintainability is key (not exactly the same, but maybe related to @virtuald's "ugliness"?). pybind11 has the tendency to grow and become more complex in every direction, so we'd need to make sure this doesn't make things harder to add/maintain/be contributed. Duplication of code is also out of the question, of course.
pybind11 is advertised as header-only, which seems to be an important argument for adoption. Of course, your proposal still allows for this, but this is one of the things where we need to know if @wjakob wants to consider going there, as it does change/influence future directions of the library.

Related to the maintainability is how this fits into the build system and project structure. C++ sources should probably be in a src/ directory; do you have a plan or proposal for that? The good news is of course that we wouldn't be alone, as e.g. {fmt} has a similar approach.

The other thing is that we are still in the process of cleaning up old PRs and old issues. So if @wjakob doesn't veto this and we make things work work, be aware that it will take some time before things get merged and a new minor release gets created.

But yes, there is some interest, and while we currently can't promise anything before hearing from @wjakob, anything that can make this plan more concrete is welcome! :-)

YannickJadoul · 2020-07-25T22:04:54Z

The proposed implementation is: I would just check if a macro if defined before inclusion of pybind11.hpp, say PYBIND11_HPP.

My personal 2 cents and bike shedding: I'm not a big fan of that name. But that's of course some future discussion ;-)

cirosantilli2 · 2020-07-26T07:13:50Z

Hi @cirosantilli2. We (i.e., the new group of maintainers that are currently trying to clean up the backlog of issues and PRs, etc) were discussing this yesterday, but somehow failed to actually post our conclusion here. But, here we go:

Fantastic news! Is the discussion public?

This seems quite interesting, indeed, a 40% speedup, though it probably depends on the amount of translation units you are compiling?

Absolutely. The majority of small projects out there will not benefit at all from this.

But potentially some "big and important" ones will. gem5 is more or less in that category I'd say. But see also #708 (comment) quote on pytorch saying pybind11 is too slow for them.

As mentioned at #708 (comment) "our build has 1271 object files, and 365 of them are simple auto-generated C++ files that include pybind11 to contain parameters for different C++ classes, and those 365 files dominate build times."

To me, it's especially surprising because I would've guessed all the template instantiations are what make things slow?

I was surprised too. I only spent 4 hours splitting headers for the prototype after I analyzed the project with the methodology from https://stackoverflow.com/questions/2351148/explicit-template-instantiation-when-is-it-used/59614755#59614755 and it clearly pointed to pybind11 as being about 50% of the unlinked object sizes. I also had the list of top object sizes per object, but didn't write it down, I'll do it next time.

https://gem5-review.googlesource.com/c/public/gem5/+/29132 shows a hasty prototype that physically splits into .hpp and .cpp. I would basically take that and put the .cpp back inside the corresponding .hpp in an ifdef.

This is already the first important step: the possibility to seen and play around with such a prototype.

From the project's perspective, there are two main concerns:
1. Maintainability is key (not exactly the same, but maybe related to @virtuald's "ugliness"?). pybind11 has the tendency to grow and become more complex in every direction, so we'd need to make sure this doesn't make things harder to add/maintain/be contributed. Duplication of code is also out of the question, of course.

Yes, this would add a "split definitions into header/c++ overhead" like other fast compiling c++ projects need to do :-)

2. pybind11 is advertised as header-only, which seems to be an important argument for adoption. Of course, your proposal still allows for this, but this is one of the things where we need to know if @wjakob wants to consider going there, as it does change/influence future directions of the library.
Related to the maintainability is how this fits into the build system and project structure. C++ sources should probably be in a src/ directory; do you have a plan or proposal for that? The good news is of course that we wouldn't be alone, as e.g. {fmt} has a similar approach.

Let me know what setup you think is best and I'll do it. src/ sounds fine. I would just keep things in the same header file, but I don't mind.

The other thing is that we are still in the process of cleaning up old PRs and old issues. So if @wjakob doesn't veto this and we make things work work, be aware that it will take some time before things get merged and a new minor release gets created.

No problem at all. I intend to continue working on gem5 for the foreseeable future :-)

But yes, there is some interest, and while we currently can't promise anything before hearing from @wjakob, anything that can make this plan more concrete is welcome! :-)

My personal 2 cents and bike shedding: I'm not a big fan of that name. But that's of course some future discussion ;-)

No problem, you may call it WUBBA_LUBBA_DUB_DUB for all I care 😆

wjakob · 2020-07-26T07:37:46Z

Hi @cirosantilli2,

thank you for bringing up this point, it's definitely worth discussing it. In principle, separate linking of the .cpp portion could even be done by Cmake (pybind11_module), so many users would automatically benefit.

That said, before considering to merge such a change, it would be important to discuss what the change you mention implies (i.e. what truly needs to be moved). I imagine that it's mainly the initialization-related parts like creation of the 'internals' data structure and the method dispatcher that could benefit from being moved into a .cpp file. Most of the rest is templated and thus not admissible, and the benefit of moving methods of simple wrapper classes like py::list is perhaps not as clear due to the resulting indirect costs (relatively code size small gain, making code less readable, preventing inlining of these methods, etc.)

I would also like to see this cited 40% reduction on a benchmark that I can try on my end. Part of the problem here is that we currently don't have infrastructure to monitor build time and per-commit changes.

Finally, there is another important metric that I care about much more than build time, because that's an area where we are not doing super well at the moment: binary size of the resulting bindings. This aspect needs to be part of the discussion as well.

Thanks,
Wenzel

cirosantilli2 · 2020-07-26T15:37:43Z

Hi @cirosantilli2,

thank you for bringing up this point, it's definitely worth discussing it. In principle, separate linking of the .cpp portion could even be done by Cmake (pybind11_module), so many users would automatically benefit.

Fantastic, that would be even better.

That said, before considering to merge such a change, it would be important to discuss what the change you mention implies (i.e. what truly needs to be moved). I imagine that it's mainly the initialization-related parts like creation of the 'internals' data structure and the method dispatcher that could benefit from being moved into a .cpp file. Most of the rest is templated and thus not admissible,

Yes, templates simply cannot be moved.

At https://gem5-review.googlesource.com/c/public/gem5/+/29132 I just tried to (hastily) move everything that can be moved.

and the benefit of moving methods of simple wrapper classes like py::list is perhaps not as clear due to the resulting indirect costs (relatively code size small gain, making code less readable, preventing inlining of these methods, etc.)

I'll leave the criteria/benchmark selection to you, and I'll follow criteria/use benchmarks.

gem5 is not a good perf benchmark because the pybind11 is just a very shallow initialization wrapper.

I suspect this is the case for most projects, and people won't notice much slowdown due to removed inlining.

Not splitting everything has the downside that you will eternally need to correct people in pull requests :-)

Also worth noting that -flto can automatically inline functions these days: https://stackoverflow.com/questions/7046547/link-time-optimization-and-inline/61367211#61367211

I would also like to see this cited 40% reduction on a benchmark that I can try on my end. Part of the problem here is that we currently don't have infrastructure to monitor build time and per-commit changes.

Definitely, it would be amazing if others can reproduce, on Ubuntu 19.10 you can try:

sudo apt install libgoogle-perftools-dev libboost-all-dev m4 protobuf-compiler python-dev python-pip scons zlib1g-dev

git clone https://github.com/cirosantilli/gem5
cd gem5

# My prototype
git checkout pybind11-cpp-squash-2

# Time the build.
time scons --ignore-style -j `nproc` build/ARM/gem5.opt 

# One commit before my prototype on mainline gem5
git checkout HEAD~

mv build build-fast

time scons --ignore-style -j `nproc` build/ARM/gem5.opt

Let me know if there are any problems, I might have missed some dependency.

Quoted 40% on my 8 core Lenovo ThinkPad P51 Ubuntu 20.: https://github.com/cirosantilli/notes/blob/61919a4c1d789eee3461b4c6011b6bbfb9118c28/my-hardware.adoc#thinkpad-p51 using SSD.

Finally, there is another important metric that I care about much more than build time, because that's an area where we are not doing super well at the moment: binary size of the resulting bindings. This aspect needs to be part of the discussion as well.

I'll check this out and get back to you.

Thanks,
Wenzel

cirosantilli2 · 2020-07-26T16:56:09Z

I got a build error, let me investigate before others try it out.

YannickJadoul · 2020-07-26T16:56:22Z

Fantastic news! Is the discussion public?

No, it was a pretty informal "can we/how do we proceed with this?"

Absolutely. The majority of small projects out there will not benefit at all from this.

Actually, if some things from pybind11 don't need to be recompiled during development, that would still be nice. But OK, I understand.

Definitely, it would be amazing if others can reproduce, on Ubuntu 19.10 you can try:

To make it easier to try out, is there any chance you could turn this code into a PR, here? It doesn't need to be polished, but I think it might make things more concrete to move forward. That would make it easier to review as well. You do have our word we'll try to move forward on this and investigate the possibility (even though it might be slow progress, for now). If you don't have time or it's a lot more work to create a PR, that's fine as well, and we'll make do with the other change set, as well.

No problem, you may call it WUBBA_LUBBA_DUB_DUB for all I care

+1 for that name ;-)

cirosantilli2 · 2020-07-26T19:58:22Z

OK, I remember now, gerrit didn't like my patch because it was too large and I made some mess there, now pushed the working branch to GitHub, updated the above comment instructions.

On Ubuntu 20.04 I've tested just now, I sometimes saw an improvement of only 30% not 40%. But the build time varies a lot, so it's hard to be certain. I think watching YouTube videos might have something to do with it:

First old:

real    25m38.292s
user    182m43.304s
sys     6m26.731s

Second old:

real    35m16.948s
user    229m43.098s
sys     8m49.819s

First new

real    17m22.439s
user    124m33.609s
sys     5m44.999s

Second new

real    18m34.282s
user    123m47.526s
sys     5m41.309s

Edit: I tried again when not watching YouTube videos, and with my computer on a table that does not overheat instead of on my bed, and I got results more like the original (37%) speedup:

real    24m4.998s
user    178m30.085s
sys     6m10.131s


real    15m11.539s
user    107m29.313s
sys     4m48.169s

@wjakob

Finally, there is another important metric that I care about much more than build time, because that's an area where we are not doing super well at the moment: binary size of the resulting bindings. This aspect needs to be part of the discussion as well.

I checked now, the build-fast/ARM/gem5.opt final executable fell from 1.8G to 962M unstripped (uses -O2 -g), God knows why that executable is so large. Manually stripped with strip fell from 51M to 47M. With strip --strip-debug it falls from 61M to 58M, so it does seem that the humongous bulk was actually from debug info and not some gem5 madness.

@YannickJadoul

To make it easier to try out, is there any chance you could turn this code into a PR, here? It doesn't need to be polished, but I think it might make things more concrete to move forward. That would make it easier to review as well. You do have our word we'll try to move forward on this and investigate the possibility (even though it might be slow progress, for now). If you don't have time or it's a lot more work to create a PR, that's fine as well, and we'll make do with the other change set, as well.

Great, I'm convinced. Let me just get clearance with our legal (maybe I should have done that already 😜), it will be really simple since nothing sensitive in this project, and then I'll post something here.

cirosantilli2 · 2020-08-28T08:37:08Z

OK, legal person was on holiday, but everything is OK now, I'm giving it a shot. There are some functions that are marked as both PYBIND11_NOINLINE inline, is there a reason for this or just a bug?

YannickJadoul · 2020-08-28T09:57:03Z

Great! Curious to see how things will look :)

There are some functions that are marked as both PYBIND11_NOINLINE inline, is there a reason for this or just a bug?

As far as I know:

PYBIND11_NOINLINE means "Please compiler, make this a separate function, not inline, and don't blow up the binary size of pybind11 libraries."
And inline means "Don't bother with the One Definition Rule, though! This is in a header and thus I guarantee this is the same in every translation unit you'll compile."

cirosantilli2 · 2020-08-28T11:28:57Z

OK. I'm asking because when I try to split I get warnings, e.g.:

main.cpp

__attribute__ ((noinline)) inline int myfunc();

int myfunc() { return 2; }

and:

g++ -c -ggdb3 -O0 -std=c++11 -Wall -Wextra -pedantic -o main.o main.cpp

gives:

main.cpp:3:39: warning: inline function ‘int myfunc()’ given attribute noinline
    3 | __attribute__ ((noinline)) inline int myfunc();
      |

I get the sneaky suspicion that __attribute__ is ignored in definitions (and therefore everywhere in the existing code before my patch) which is why the warning does not happen.

Does anyone know the correct placement of those?

I believe inline has two effects: the hide linkage, but also heuristic suggestion of inlining. static should hide linkage only, but then it gives unused warnings on every includer that does not use it.

cirosantilli2 · 2020-08-28T12:01:40Z

OK, so if I do:

#  pragma GCC diagnostic ignored "-Wattributes"

__attribute__ ((noinline)) int myfunc();

inline int myfunc() { return 2; }

then there are no warnings, and pybind11 already does # pragma GCC diagnostic ignored "-Wattributes", so I think I'll move all inline to the definitions.

YannickJadoul · 2020-08-28T13:04:28Z

Not sure I'm a fan of this pragma for ignoring the warnings. It's probably better to adapt PYBIND11_NOINLINE to resolve to nothing whenever a not-headers-only build is being done? All of that can be fixed in incremental steps in the PR, though :-)

cirosantilli2 · 2020-08-28T16:40:23Z

Not sure I'm a fan of this pragma for ignoring the warnings. It's probably better to adapt PYBIND11_NOINLINE to resolve to nothing whenever a not-headers-only build is being done? All of that can be fixed in incremental steps in the PR, though :-)

I don't see a solution that maintains inline in header-only mode. The current code looks like:

__attribute__ ((noinline)) inline int myfunc() { return 2; }

and for some reason I don't understand, it does not give warnings, even without pragma. Might be a missing warning GCC bug.

To split that into the .cpp file, if first tried:

__attribute__ ((noinline)) inline int myfunc();

int myfunc() { return 2; }

but that gives an unignorable warning.

The only thing that works for me is:

# pragma GCC diagnostic ignored "-Wattributes"

__attribute__ ((noinline)) int myfunc();

inline int myfunc() { return 2; }

as it would produce an ignorable error.

But yes, I actually do need the PYBIND11_INLINE macro because the above symbol is not visible to other files in non-header-only mode, the inline has to be removed there.

New annoyance: every method needs to be marked inline?

I now noticed another annoyance.

Currently, every single non-template function in bypind11 has to be marked as inline to avoid symbol conflicts with multiple includes in a project.

This was not the case for class methods because when you do:

# pragma GCC diagnostic ignored "-Wattributes"

__attribute__ ((noinline)) int myfunc();

inline int myfunc() { return 2; }

struct MyClass {
    int MyClass::myfunc() { return 2; }
};

it does not expose a symbol.

However, if you do:

struct MyClass {
    int myfunc();
};

int MyClass::myfunc() { return 2; }

then it does expose a symbol.

To not expose the symbol, the only option I can find is:

struct MyClass {
    inline int myfunc();
};

int MyClass::myfunc() { return 2; }

I'm going to make a pull request later on splitting one or two files when I get it to build correctly so people can inspect it.

YannickJadoul · 2020-08-28T17:09:12Z

I don't see a solution that maintains inline in header-only mode. The current code looks like:

What do you mean? Why can't you adapt the #define of PYBIND11_NOINLINE in detail/common.h?

The current

#if defined(_MSC_VER)
#  define PYBIND11_NOINLINE __declspec(noinline)
#else
#  define PYBIND11_NOINLINE __attribute__ ((noinline))
#endif

could become something like

#ifdef PYBIND11_HPP
#  if defined(_MSC_VER)
#    define PYBIND11_NOINLINE __declspec(noinline)
#  else
#    define PYBIND11_NOINLINE __attribute__ ((noinline))
#  endif
#else
#  define PYBIND11_NOINLINE
#endif

or an additional thing like this

#ifdef PYBIND11_HPP
#  define PYBIND11_INLINE inline
#else
#  define PYBIND11_INLINE
#endif

or ... well, something like this, I meant. We need to see what we want to see in header-only mode and what we want to see in source-files mode. Maybe there's a better way to name all this, that doesn't cause the same amount of confusion as "noinline" + "inline".

Currently, every single non-template function in bypind11 has to be marked as inline to avoid symbol conflicts with multiple includes in a project.

I'm not entirely getting what you mean with "expose a symbol", but ...

I'm going to make a pull request later on splitting one or two files when I get it to build correctly so people can inspect it.

... yes, I think that's a good idea. That'll make things a lot easier to discuss :-)

As discussed at pybind#2322, the split is opt-in. If the PYBIND11_HEADER_ONLY is not defined, then the .cpp files are included in the .h files. PYBIND11_INLINE is introduced because if the header functions are marked inline, then that makes the symbol not show in the object file, so inline must be turned on or off depending on PYBIND11_HEADER_ONLY.

TODO For now, only pybind11.h and options.h have been split. If maintainers feel that this is going in the right direction, I will update this commit to split all files. As discussed at pybind#2322, the split is opt-in, and has been observed to speed up the compilation of the gem5 project by around 40% from 25 minutes to 15 minutes. If the PYBIND11_DECLARATIONS_ONLY is not defined, then the .cpp files are included in the .h files, and everything works header-only as was the case before this commit. If PYBIND11_DECLARATIONS_ONLY=1, then only declarations are made visible from, the header files, and the user must the user must compile the pybind11 .cpp files, also using PYBIND11_DECLARATIONS_ONLY=1, into objects and add link those into the final link. This commit also updates CMakeLists to automate building those object files. This commit has been tested as follows. First, the original build and pytest (without PYBIND11_DECLARATIONS_ONLY) work as before: mkdir build cmake .. make -j `nproc` make pytest TODO: if this commit gets traction, I will try to add a test for the PYBIND11_DECLARATIONS_ONLY version too, otherwise it will likely break. I'd just re-run the entire pybind with PYBIND11_DECLARATIONS_ONLY as well every time. The commit has also been tested with the following minimal manual example: class_test.cpp ``` struct ClassTest { ClassTest(const std::string &name) : name(name) {} void setName(const std::string &name_) { name = name_; } const std::string &getName() const { return name; } std::string name; }; struct ClassTestDerived : ClassTest { ClassTestDerived(const std::string &name, const std::string &name2) : ClassTest(name), name2(name2) {} std::string getName2() { return name + name2 + "2"; } std::string name2; }; namespace py = pybind11; PYBIND11_MODULE(class_test, m) { m.doc() = "pybind11 example plugin"; py::class_<ClassTest>(m, "ClassTest") .def(py::init<const std::string &>()) .def("setName", &ClassTest::setName) .def("getName", &ClassTest::getName) .def_readwrite("name", &ClassTest::name); py::class_<ClassTestDerived, ClassTest>(m, "ClassTestDerived") .def(py::init<const std::string &, const std::string &>()) .def("getName2", &ClassTestDerived::getName2) .def_readwrite("name", &ClassTestDerived::name); } ``` class_test_main.py ``` import class_test my_class_test = class_test.ClassTest('abc'); assert(my_class_test.getName() == 'abc') my_class_test.setName('012') assert(my_class_test.name == '012') my_class_test_derived = class_test.ClassTestDerived('abc', 'def'); assert(my_class_test_derived.getName2() == 'abcdef2') ``` without PYBIND11_DECLARATIONS_ONLY (`python3-config --cflags`) is opened up and hacked to point to the custom pybind11 source: ``` g++ \ -save-temps \ -I/usr/include/python3.8 \ -I/home/ciro/git/pybind11/include \ -Wno-unused-result \ -Wsign-compare \ -g \ -fdebug-prefix-map=/build/python3.8-fKk4GY/python3.8-3.8.2=. \ -specs=/usr/share/dpkg/no-pie-compile.specs \ -fstack-protector \ -Wformat \ -Werror=format-security \ -DNDEBUG \ -g \ -fwrapv \ -O3 \ -Wall \ -shared \ -std=c++11 \ -fPIC class_test.cpp \ -o class_test`python3-config --extension-suffix` \ `python3-config --libs` \ ; ./class_test_main.py ``` with PYBIND11_DECLARATIONS_ONLY: ``` g++ \ -DPYBIND11_DECLARATIONS_ONLY=1 \ -save-temps \ -I/usr/include/python3.8 \ -I/home/ciro/git/pybind11/include \ -Wno-unused-result \ -Wsign-compare \ -g \ -fdebug-prefix-map=/build/python3.8-fKk4GY/python3.8-3.8.2=. \ -specs=/usr/share/dpkg/no-pie-compile.specs \ -fstack-protector \ -Wformat \ -Werror=format-security \ -DNDEBUG \ -g \ -fwrapv \ -O3 \ -Wall \ -shared \ -std=c++11 \ -fPIC class_test.cpp \ /home/ciro/git/pybind11/build/libpybind11.a \ -o class_test`python3-config --extension-suffix` \ `python3-config --libs` \ ; ./class_test_main.py ```

cirosantilli2 · 2020-08-28T19:16:07Z

What do you mean? Why can't you adapt the #define of PYBIND11_NOINLINE in detail/common.h?

Ahh, sorry, I hadn't understood properly, yes disabling the PYBIND11_NOINLINE should remove the reliance on the -Wno-attributes, and everything cannot be inlined anyways.

I'm not entirely getting what you mean with "expose a symbol", but ...

What I mean by expose is that if you mark a global non-member function as inline, and then e.g. do nm myfile.o, it does not show up at all.

This is required if you want to have a project in which two cpp files include pybind11 in header-only mode, otherwise there would be multiple definition conflicts.

However, when using non-header-only, you cannot have inline, otherwise the cpp files that use pybind11 will not see any definition at all (the definition will be present in an object file generated from the pybind .cpp files).

Splitting the class methods as I'm doing also requires them to be marked as inline according to GCC.

cirosantilli2 · 2020-08-28T19:17:41Z

Pushed an initial pull request at: #2445

As mentioned there, only two files were split for now, this will make it easier to do an initial review, and will mean less re-work for me if something needs to change. I'll split all files and update the commit once that approach gets approved.

YannickJadoul · 2020-08-30T22:11:43Z

Pushed an initial pull request at: #2445

Great, thanks! Sorry for the delay; I'll try to get a look tomorrow.

cirosantilli2 · 2020-08-30T22:34:14Z

No worries, I also have to understand why the tests failed and fix them 👍 I'll do that soon.

TODO For now, only pybind11.h and options.h have been split. If maintainers feel that this is going in the right direction, I will update this commit to split all files. As discussed at pybind#2322, the split is opt-in, and has been observed to speed up the compilation of the gem5 project by around 40% from 25 minutes to 15 minutes. If the PYBIND11_DECLARATIONS_ONLY is not defined, then the .cpp files are included in the .h files, and everything works header-only as was the case before this commit. If PYBIND11_DECLARATIONS_ONLY=1, then only declarations are made visible from, the header files, and the user must the user must compile the pybind11 .cpp files, also using PYBIND11_DECLARATIONS_ONLY=1, into objects and add link those into the final link. This commit also updates CMakeLists to automate building those object files. This commit has been tested as follows. First, the original build and pytest (without PYBIND11_DECLARATIONS_ONLY) work as before: mkdir build cmake .. make -j `nproc` make pytest TODO: if this commit gets traction, I will try to add a test for the PYBIND11_DECLARATIONS_ONLY version too, otherwise it will likely break. I'd just re-run the entire pybind with PYBIND11_DECLARATIONS_ONLY as well every time. The commit has also been tested with the following minimal manual example: class_test.cpp ``` struct ClassTest { ClassTest(const std::string &name) : name(name) {} void setName(const std::string &name_) { name = name_; } const std::string &getName() const { return name; } std::string name; }; struct ClassTestDerived : ClassTest { ClassTestDerived(const std::string &name, const std::string &name2) : ClassTest(name), name2(name2) {} std::string getName2() { return name + name2 + "2"; } std::string name2; }; namespace py = pybind11; PYBIND11_MODULE(class_test, m) { m.doc() = "pybind11 example plugin"; py::class_<ClassTest>(m, "ClassTest") .def(py::init<const std::string &>()) .def("setName", &ClassTest::setName) .def("getName", &ClassTest::getName) .def_readwrite("name", &ClassTest::name); py::class_<ClassTestDerived, ClassTest>(m, "ClassTestDerived") .def(py::init<const std::string &, const std::string &>()) .def("getName2", &ClassTestDerived::getName2) .def_readwrite("name", &ClassTestDerived::name); } ``` class_test_main.py ``` import class_test my_class_test = class_test.ClassTest('abc'); assert(my_class_test.getName() == 'abc') my_class_test.setName('012') assert(my_class_test.name == '012') my_class_test_derived = class_test.ClassTestDerived('abc', 'def'); assert(my_class_test_derived.getName2() == 'abcdef2') ``` without PYBIND11_DECLARATIONS_ONLY (`python3-config --cflags`) is opened up and hacked to point to the custom pybind11 source: ``` g++ \ -save-temps \ -I/usr/include/python3.8 \ -I/home/ciro/git/pybind11/include \ -Wno-unused-result \ -Wsign-compare \ -g \ -fdebug-prefix-map=/build/python3.8-fKk4GY/python3.8-3.8.2=. \ -specs=/usr/share/dpkg/no-pie-compile.specs \ -fstack-protector \ -Wformat \ -Werror=format-security \ -DNDEBUG \ -g \ -fwrapv \ -O3 \ -Wall \ -shared \ -std=c++11 \ -fPIC class_test.cpp \ -o class_test`python3-config --extension-suffix` \ `python3-config --libs` \ ; ./class_test_main.py ``` with PYBIND11_DECLARATIONS_ONLY: ``` g++ \ -DPYBIND11_DECLARATIONS_ONLY=1 \ -save-temps \ -I/usr/include/python3.8 \ -I/home/ciro/git/pybind11/include \ -Wno-unused-result \ -Wsign-compare \ -g \ -fdebug-prefix-map=/build/python3.8-fKk4GY/python3.8-3.8.2=. \ -specs=/usr/share/dpkg/no-pie-compile.specs \ -fstack-protector \ -Wformat \ -Werror=format-security \ -DNDEBUG \ -g \ -fwrapv \ -O3 \ -Wall \ -shared \ -std=c++11 \ -fPIC class_test.cpp \ /home/ciro/git/pybind11/build/libpybind11.a \ -o class_test`python3-config --extension-suffix` \ `python3-config --libs` \ ; ./class_test_main.py ```

As discussed at pybind#2322, the split is opt-in, and has been observed to speed up the compilation of the gem5 project by around 40% from 25 minutes to 15 minutes. If the PYBIND11_DECLARATIONS_ONLY is not defined, then the .cpp files are included in the .h files, and everything works header-only as was the case before this commit. If PYBIND11_DECLARATIONS_ONLY=1, then only declarations are made visible from, the header files, and the user must the user must compile the pybind11 .cpp files, also using PYBIND11_DECLARATIONS_ONLY=1, into objects and add link those into the final link. This commit also updates CMakeLists to automate building those object files. This commit has been tested as follows. First, the original build and pytest (without PYBIND11_DECLARATIONS_ONLY) work as before: mkdir build cmake .. make -j `nproc` make pytest TODO: if this commit gets traction, I will try to add a test for the PYBIND11_DECLARATIONS_ONLY version too, otherwise it will likely break. I'd just re-run the entire pybind with PYBIND11_DECLARATIONS_ONLY as well every time. The commit has also been tested with the following minimal manual example: class_test.cpp ``` struct ClassTest { ClassTest(const std::string &name) : name(name) {} void setName(const std::string &name_) { name = name_; } const std::string &getName() const { return name; } std::string name; }; struct ClassTestDerived : ClassTest { ClassTestDerived(const std::string &name, const std::string &name2) : ClassTest(name), name2(name2) {} std::string getName2() { return name + name2 + "2"; } std::string name2; }; namespace py = pybind11; PYBIND11_MODULE(class_test, m) { m.doc() = "pybind11 example plugin"; py::class_<ClassTest>(m, "ClassTest") .def(py::init<const std::string &>()) .def("setName", &ClassTest::setName) .def("getName", &ClassTest::getName) .def_readwrite("name", &ClassTest::name); py::class_<ClassTestDerived, ClassTest>(m, "ClassTestDerived") .def(py::init<const std::string &, const std::string &>()) .def("getName2", &ClassTestDerived::getName2) .def_readwrite("name", &ClassTestDerived::name); } ``` class_test_main.py ``` import class_test my_class_test = class_test.ClassTest('abc'); assert(my_class_test.getName() == 'abc') my_class_test.setName('012') assert(my_class_test.name == '012') my_class_test_derived = class_test.ClassTestDerived('abc', 'def'); assert(my_class_test_derived.getName2() == 'abcdef2') ``` without PYBIND11_DECLARATIONS_ONLY (`python3-config --cflags`) is opened up and hacked to point to the custom pybind11 source: ``` g++ \ -save-temps \ -I/usr/include/python3.8 \ -I/home/ciro/git/pybind11/include \ -Wno-unused-result \ -Wsign-compare \ -g \ -fdebug-prefix-map=/build/python3.8-fKk4GY/python3.8-3.8.2=. \ -specs=/usr/share/dpkg/no-pie-compile.specs \ -fstack-protector \ -Wformat \ -Werror=format-security \ -DNDEBUG \ -g \ -fwrapv \ -O3 \ -Wall \ -shared \ -std=c++11 \ -fPIC class_test.cpp \ -o class_test`python3-config --extension-suffix` \ `python3-config --libs` \ ; ./class_test_main.py ``` with PYBIND11_DECLARATIONS_ONLY: ``` g++ \ -DPYBIND11_DECLARATIONS_ONLY=1 \ -save-temps \ -I/usr/include/python3.8 \ -I/home/ciro/git/pybind11/include \ -Wno-unused-result \ -Wsign-compare \ -g \ -fdebug-prefix-map=/build/python3.8-fKk4GY/python3.8-3.8.2=. \ -specs=/usr/share/dpkg/no-pie-compile.specs \ -fstack-protector \ -Wformat \ -Werror=format-security \ -DNDEBUG \ -g \ -fwrapv \ -O3 \ -Wall \ -shared \ -std=c++11 \ -fPIC class_test.cpp \ /home/ciro/git/pybind11/build/libpybind11.a \ -o class_test`python3-config --extension-suffix` \ `python3-config --libs` \ ; ./class_test_main.py ```

cirosantilli2 mentioned this issue Aug 28, 2020

Split pybind11 into headers and cpp files to speedup compilation #2445

Draft

EricCousineau-TRI mentioned this issue Dec 30, 2020

[FEAT] Should include more comprehensive benchmarking (primarily performance) #2760

Open

4 tasks

YannickJadoul mentioned this issue Jan 19, 2021

Move cpp_function to its own header file for further refactoring #2807

Open

rwgk mentioned this issue Feb 10, 2023

FWD pybind11 google/pybind11clif#2322

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Would you consider merging a patch that optionally splits pybind11 into declarations vs definitions in a backwards compatible opt-in manner? #2322

Would you consider merging a patch that optionally splits pybind11 into declarations vs definitions in a backwards compatible opt-in manner? #2322

cirosantilli2 commented Jul 23, 2020 •

edited

Loading

virtuald commented Jul 25, 2020

cirosantilli2 commented Jul 25, 2020 •

edited

Loading

virtuald commented Jul 25, 2020

cirosantilli2 commented Jul 25, 2020 •

edited

Loading

YannickJadoul commented Jul 25, 2020

YannickJadoul commented Jul 25, 2020 •

edited

Loading

cirosantilli2 commented Jul 26, 2020 •

edited

Loading

wjakob commented Jul 26, 2020

cirosantilli2 commented Jul 26, 2020 •

edited

Loading

cirosantilli2 commented Jul 26, 2020

YannickJadoul commented Jul 26, 2020

cirosantilli2 commented Jul 26, 2020 •

edited

Loading

cirosantilli2 commented Aug 28, 2020

YannickJadoul commented Aug 28, 2020 •

edited

Loading

cirosantilli2 commented Aug 28, 2020 •

edited

Loading

cirosantilli2 commented Aug 28, 2020

YannickJadoul commented Aug 28, 2020

cirosantilli2 commented Aug 28, 2020 •

edited

Loading

YannickJadoul commented Aug 28, 2020

cirosantilli2 commented Aug 28, 2020

cirosantilli2 commented Aug 28, 2020

YannickJadoul commented Aug 30, 2020

cirosantilli2 commented Aug 30, 2020

Would you consider merging a patch that optionally splits pybind11 into declarations vs definitions in a backwards compatible opt-in manner? #2322

Would you consider merging a patch that optionally splits pybind11 into declarations vs definitions in a backwards compatible opt-in manner? #2322

Comments

cirosantilli2 commented Jul 23, 2020 • edited Loading

virtuald commented Jul 25, 2020

cirosantilli2 commented Jul 25, 2020 • edited Loading

virtuald commented Jul 25, 2020

cirosantilli2 commented Jul 25, 2020 • edited Loading

YannickJadoul commented Jul 25, 2020

YannickJadoul commented Jul 25, 2020 • edited Loading

cirosantilli2 commented Jul 26, 2020 • edited Loading

wjakob commented Jul 26, 2020

cirosantilli2 commented Jul 26, 2020 • edited Loading

cirosantilli2 commented Jul 26, 2020

YannickJadoul commented Jul 26, 2020

cirosantilli2 commented Jul 26, 2020 • edited Loading

cirosantilli2 commented Aug 28, 2020

YannickJadoul commented Aug 28, 2020 • edited Loading

cirosantilli2 commented Aug 28, 2020 • edited Loading

cirosantilli2 commented Aug 28, 2020

YannickJadoul commented Aug 28, 2020

cirosantilli2 commented Aug 28, 2020 • edited Loading

YannickJadoul commented Aug 28, 2020

cirosantilli2 commented Aug 28, 2020

cirosantilli2 commented Aug 28, 2020

YannickJadoul commented Aug 30, 2020

cirosantilli2 commented Aug 30, 2020

cirosantilli2 commented Jul 23, 2020 •

edited

Loading

cirosantilli2 commented Jul 25, 2020 •

edited

Loading

cirosantilli2 commented Jul 25, 2020 •

edited

Loading

YannickJadoul commented Jul 25, 2020 •

edited

Loading

cirosantilli2 commented Jul 26, 2020 •

edited

Loading

cirosantilli2 commented Jul 26, 2020 •

edited

Loading

cirosantilli2 commented Jul 26, 2020 •

edited

Loading

YannickJadoul commented Aug 28, 2020 •

edited

Loading

cirosantilli2 commented Aug 28, 2020 •

edited

Loading

cirosantilli2 commented Aug 28, 2020 •

edited

Loading