This is the second time I try to build the VTK in Mac. One month ago, I took a lot of time to find how to compile it with CMake and built it successfully, but it is much easier to forget the process. Here I record the entire compiling process.
First step is to download the VTK from the VTK official website. Then, unzip the package to a folder. Here I put it into the folder called “VTK-8.1.0”. By the way, you should also install CMake from the CMake website (Please ensure that the Qt and Xcode are installed in your Mac). Now, create another folder called “VTK_Build”. This is the target folder to save the files generated by CMake. Open CMake and set the “source code” and “binaries” folder:
Click “Configure” button, and set the generator for the source code, I just use the default selection. This means we will use make and make install to compile the code:
After click “Done”, CMake will configure the code:
Now, we create another folder called “VTK_Bin” which is used to install VTK libraries. In CMake, we need change two options:
CMAKE_INSTALL_PREFIX: /User/zhudongping/Documents/Code/VTK_Bin
VTK_Group_Qt: ON
Click “Configure” again, and CMake will tell you it cannot found the Qt5_DIR:
Set the Qt5_DIR: /Applications/Qt/5.10.0/clang_64/lib/cmake/Qt5 and click “Configure” again. you will see:
Now, you need click “Configure” again, CMake will finish the final configuration. 
Then, Click “Generate” in the panel to generate the binaries into “VTK_Build”. Use Mac terminal to find this folder and type make command. If you want to use multithreading, type make -j8 command. Mac will compile the files generated by CMake:
After the make command, you just need to type make install command to install the VTK libraries into “VTK_Bin” folder:
Now, we successfully build the source code of VTK 8.1.0:
For using VTK in Qt 5.10.0, Remember to put the following code into main function:
QSurfaceFormat::setDefaultFormat(QVTKOpenGLWidget::defaultFormat());