I am writing down my findings about this process as I was only able to finish because of people sharing their experiences online. If anyone wants to do this or something similar in the future, hopefully one or two things listed here will be useful.
I intended to create Python bindings for the QtWinMigrate library to be able to use the 3ds Max windows as a parent for custom PyQt widgets. Blur did this for their Py3dsMax package and currently there is no equal functionality for PySide in the MaxPlus Python API.
I was hoping that the binding would probably work for PySide as well, which in the end of course was a bit naive. Nevertheless I decided to give it a try, as part of the work was already done and available online in the form of sip files that define the actual Python binding to the C++ code.
The following is a complete rundown of my personal struggles, problems and (partially hacky) solutions. It will probably not be the same for you, should you try repeat it due to differences in platform, environment and/or tools. So this is mainly intended to give an overview of needed steps and a list of potential problems and workarounds to overcome them.
Please keep in mind I am far from being in expert in any of this. There is probably a less error-prone way of building all this and I am pretty sure I missed something due to lack of understanding, but hey, information is quite rare about the whole thing and it's often like "lol just g++, ez lyfe bro" which is not particulary helpful for this specific application. Please be aware of the different licenses of the libraries and tools that are being used.
As a short overview this is what we have to do:
Build and install SIP
Build and install Qt
Build and install PyQt
Build and install QtWinmigrate
Build and install the QtWinmigrate Python bindings
We will have to use the same versions of the above tools as are being used in 3ds Max 2015. This is probably not crucial for all of them, but the following definitively have to match:
Python 2.7.3
Qt 4.8.5
Prerequisites
For compilation you need:
Python
The MSVC2010 compiler. I used Visual Studio 2012 Professional for this.
The MFC library. They come with Visual Studio Professional and above, but not with the Express versions.
We will use nmake (comes with Visual Studio) and qmake (comes with Qt) as our build tools.
Use the 3ds Max Python Installation
The easiest way to use the same Python version as in 3ds Max 2015 is to use the actual interpreter that comes with it. However this is not a good idea, as seemingly some of the tools in the build process are not made for paths with spaces in them, and the default C:\Program Files\Autodesk\3ds Max 2015\python has plenty of those.
Instead what you can safely do is copy and rename the C:\Program Files\Autodesk\3ds Max 2015\python directory to somewhere else, e.g. c:\python273_max2015.
For convenience you should then make this your default Python version. Simply modify your PATH variable to list the C:\python273_max2015 directory before any other Python version you may have installed. I recommend the handy Rapid Environment Editor for this, the default Windows interface for it is horrible.
Note: All commands below are executed within a VS2012 x64 Native Tools Command Prompt. It comes with Visual Studio and you can either launch it from the Windows Start Menu or by calling this in a cmd.exe:
"C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat" amd64
SIP
SIP is a tool to create Python bindings for theoretically any C++ library, but has been created specifically for PyQt. It needs the original C++ code and additional information defining what to expose in the binding which is given in form of a SIP file.
First download the latest SIP source files from here and unzip them.
To build and install SIP into our Python version do:
cd path\to\sip
python configure.py -p win32-msvc2010
This will configure the project for our Windows platform.
nmake
nmake install
You should now have a bunch of new directories and files:
C:\python273_max2015\sip.exe
C:\python273_max2015\Lib\site-packages\
C:\python273_max2015\Lib\site-packages\sip.pyd
C:\python273_max2015\Lib\site-packages\sipconfig.py
C:\python273_max2015\Lib\site-packages\sipdistutils.py
To check if everything worked do e.g.:
python
import sip
print (sip.wrapinstance)
Qt
Qt is a popular GUI and application framework written in C/C++.
It is pretty large on the filesystem and needs some preparations. Make sure to download version 4.8.5 for vs2010 of the source files from here. Install it to C:\Qt\4.8.5.
We then need to add a new environment variable that will be used to find your Qt location later on. You can use Rapid Environment Editor or do it from commandline like:
setx QTDIR C:\Qt\4.8.5
We also need to add nmake to the list of known commands. It is inside C:\Qt\4.8.5\bin, so add that to PATH.
Optional: Instead of nmake, we can also use jom as a build tool. It does the same thing, but using multiple cores to speed things up. This is recommended for building Qt, as it can otherwise easily take a few hours. Download jom here, extract and add its folder to PATH. Use it as a substitute for the nmake command, using the same arguments.
To build and install Qt do:
cd C:\Qt\4.8.5
configure -opensource -platform win32-msvc2010
jom
jom install
I ran into a problem associated with a _MSC_VER mismatch of some sort. This seems to be a bug in the make spec file for our platform. If it happens to you, modify this file: C:\Qt\4.8.5\mkspecs\win32-msvc2010\qmake.conf by changing this line:
QMAKE_COMPILER_DEFINES += _MSC_VER=1600 WIN32
to:
QMAKE_COMPILER_DEFINES += _MSC_VER=1700 WIN32
Your build should now succeed. Be patient, it will take some time.
PyQt
PyQt is a set of Python bindings to the Qt framework. Download version 4.11.3 here (other 4.x.x versions might work as well, but this is the one I used). Extract and install to C:\PyQt-win-gpl-4.11.3.
The build needs to find the make specs from an environment variable (at least in my case), so define it:
There are two configuration scripts. Call the recommended one (it does not use SIP's build system which will be removed in the next SIP version) and build the project. As it also takes some time, it makes sense to use jom again:
cd C:\PyQt-win-gpl-4.11.3
python configure-ng.py
jom
jom install
This should install PyQt into C:\python273_max2015\Lib\site-packages\PyQt.
To test it do:
python
from PyQt4 import QtGui
app = QtGui.QApplication([])
widget = QtGui.QLabel("Seems to work!")
widget.show()
QtWinMigrate
QtWinMigrate is part of the QtSolutions package, that extends Qt in various ways but is not part of the core distribution. It is used to port MFC applications to the Qt framework, especially in regards to user interfaces. You can download it here. Extract it to C:\qtwinmigrate.
We will need to build QtWinMigrate as a static library, so we can later link our binding against it, meaning we need a .lib file as result.
Build like:
cd C:\qtwinmigrate
configure -library
qmake
qmake will create a Makefile from the qtwinmigrate.pro that we can then build and install using:
nmake
nmake install
This should create various files in the \lib folder, including QtSolutions_MFCMigrationFramework-head.lib. That is the one we need. It has also been copied to C:\Qt\4.8.5.\lib, which we will reference to in a later step.
There is one more thing we have to do. We need to convert two of the header files, as this is what the SIP files need later. We use moc for this, a sort of compiler that comes with Qt and expands Qt specific macros in source code:
cd C:\qtwinmigrate\src
moc -o moc_qwinwidget.h qwinwidget.h
moc -o moc_qwinhost.h qwinhost.h
Just leave them here for now, we will reference this directory later.
Optional: To check if everything worked you can also build the examples which will give you .exe files that open small example tools. Just do:
nmake distclean
configure
qmake
nmake
The results will be in the \examples folder.
QtWinmigrate Python bindings
You can download existing SIP files for QtWinMigrate from here, thanks alot to the people that created and shared them. They are however not completely setup to work straight out of the box, so we will have to do some adaptions in related files.
This is the most nasty part of the whole process and the one that took me the most time. To start extract the SIP files to C:\PyQt-win-gpl-4.11.3\sip.
They need to include SIP files from PyQt and the links are relative to that location.
To build our custom SIP files, we will use SIP's build system. What, didn't you say above we shouldn't use that? True, but.. it worked for me, so stop asking intelligent questions. We need to create a small python script (see a more detailed explanation). Copy and save the following code to C:\PyQt-win-gpl-4.11.3\sip\configure.py:
I would like to explain some of the flags (-t WS_WIN and -t Qt_4_8_5), as they are there to fix some problems I encountered. When executing the configuration script without them I first got:
sip: Q_PID is undefined
which is fixed by specifying the platform using -t WS_WIN. Instead I then got:
sip: HWND is undefined
which is fixed by specifying the Qt version using -t Qt_4_8_5. Thanks to Phil from the PyQt mailing list for his help on this.
Now call:
python configure.py
This should have created the following files for you:
C:\PyQt-win-gpl-4.11.3\sip\Makefile
C:\PyQt-win-gpl-4.11.3\sip\sipAPIQtWinMigrate.h
C:\PyQt-win-gpl-4.11.3\sip\sipQtWinMigratecmodule.cpp
C:\PyQt-win-gpl-4.11.3\sip\sipQtWinMigrateQMfcApp.cpp
C:\PyQt-win-gpl-4.11.3\sip\sipQtWinMigrateQWinHost.cpp
C:\PyQt-win-gpl-4.11.3\sip\sipQtWinMigrateQWinWidget.cpp
C:\PyQt-win-gpl-4.11.3\sip\QtWinMigrate.sbf
Unfortunately, the Makefile needs some tweaking, or at least it needed for me. Sorry, this is where it gets a bit ugly.
We need to tell it where to find some of the Qt headers as well as the QtWinMigrate headers and .lib file.
If you don't do this you are most probably going to run into multiple missing file/definition and Linker errors (LNK2001, LNK2019, LNK1120)
I think there are multiple ways to add include and library paths in a Makefile, but this is how it worked for me.
Paths to search for header files to include are specified in CPPFLAGS with the -I flag. Add the following to the CPPFLAGS line (all in one line, space separated):
-IC:\python\python273_max2015\include
-IC:\Qt\4.8.5\include
-IC:\Qt\4.8.5\include\QtGui
-IC:\Qt\4.8.5\include\QtCore
-IC:\qtwinmigrate\src
I am not completely sure if the Python path is needed, but it doesn't hurt.
Paths to search for libraries are given as part of the LIBS line. You can either give multiple /LIBPATH: and then use relative filenames for the libraries or specify them with absolute file paths. We will do the latter, so extend the LIBS line with the following values (all in one line, space separated):
"C:\Qt\4.8.5\lib\QtCore4.lib"
"C:\Qt\4.8.5\lib\QtGui4.lib"
"C:\Qt\4.8.5\lib\QtSolutions_MFCMigrationFramework-head.lib"
Almost there. Save the Makefile and do:
nmake
The target path for the QtWinMigrate.pyd that you hopefully just created is not correct in the Makefile, so either add a \PyQt\ in between before calling
nmake install
or copy the file manually to C:\python273_max2015\Lib\site-packages\PyQt\QtWinMigrate.pyd.
Test if it worked by calling:
python
from PyQt4 import QtWinMigrate
print (QtWinMigrate.QWinWidget)
All that is left is to copy your now fully armed Python installation from c:\python273_max2015 back into the C:\Program Files\Autodesk\3ds Max 2015\python folder (better do a backup first in case something went wrong).
Done!
Details on how to actually use the library for parenting widgets is something I yet need to find out myself. Until then you should probably have a look at the way Blur did it for their Py3dsMax module (see the blurdev.gui package).
Phew, what a mess. Next: Do the same thing for PySide and Shiboken ;)