-
Notifications
You must be signed in to change notification settings - Fork 304
Building HPCC
Assuming you already have your git repository set up (see Git-step-by-step), you can build and install HPCC.
The build process requires some additional packages. We now utilize vcpkg for library and major build tool dependencies.
We force 'out-of-source' builds. The developers environment should have a build directory alongside the cloned HPCC-Platform git repository. Once vcpkg has generated it's build tools and libraries, we will have a vcpkg_installed directory alongside as well.
Ubuntu 24.04:
### install build utilities
sudo apt-get update && sudo apt-get install -y --no-install-recommends apt-transport-https autoconf autoconf-archive automake autotools-dev binutils-dev bison build-essential ca-certificates curl dirmngr flex git gnupg groff-base libtool pkg-config software-properties-common tar unzip uuid-dev zip mono-complete
Ubuntu 22.04:
### install build utilities
sudo apt-get update && sudo apt-get install -y --no-install-recommends apt-transport-https autoconf autoconf-archive automake autotools-dev binutils-dev bison build-essential ca-certificates curl dirmngr flex git gnupg groff-base libtool pkg-config software-properties-common tar unzip uuid-dev zip
## install mono for vcpkg
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
sudo sh -c 'echo "deb https://download.mono-project.com/repo/ubuntu stable-focal main" > /etc/apt/sources.list.d/mono-official-stable.list'
sudo apt-get update && sudo apt-get install -y mono-completeUbuntu 20.04:
### install build utilities
sudo apt-get update && sudo apt-get install -y --no-install-recommends apt-transport-https autoconf autoconf-archive automake autotools-dev binutils-dev bison build-essential ca-certificates curl dirmngr flex git gnupg groff-base libtool pkg-config software-properties-common tar unzip uuid-dev zip
### install mono for vcpkg
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
sudo sh -c 'echo "deb https://download.mono-project.com/repo/ubuntu stable-focal main" > /etc/apt/sources.list.d/mono-official-stable.list'
sudo apt-get update && sudo apt-get install -y mono-completeCentOS 7 (EOL on June 30, 2024):
### install build utilities
sudo yum update -y
sudo yum install -y centos-release-scl https://packages.endpointdev.com/rhel/7/os/x86_64/endpoint-repo.x86_64.rpm
sudo yum group install -y "Development Tools"
sudo yum install -y autoconf autoconf-archive automake curl git kernel-devel libtool perl-IPC-Cmd python3 tar unzip yum-utils zip
sudo yum -y clean all
sudo yum install -y devtoolset-11
### set devtoolset-11 for future shell environments
echo "source /opt/rh/devtoolset-11/enable" | sudo tee -a /etc/bashrc
source /opt/rh/devtoolset-11/enable
cd ~
### pkg-config
curl -o pkg-config-0.29.2.tar.gz https://pkg-config.freedesktop.org/releases/pkg-config-0.29.2.tar.gz && tar xvfz pkg-config-0.29.2.tar.gz
pushd pkg-config-0.29.2
./configure --prefix=/usr/local/pkg_config/0_29_2 --with-internal-glib
make -j$(nproc)
sudo make install
sudo ln -s /usr/local/pkg_config/0_29_2/bin/pkg-config /usr/local/bin/
sudo mkdir /usr/local/share/aclocal
sudo ln -s /usr/local/pkg_config/0_29_2/share/aclocal/pkg.m4 /usr/local/share/aclocal/
popd
### autoconf
curl -o autoconf-2.71.tar.gz http://ftp.gnu.org/gnu/autoconf/autoconf-2.71.tar.gz && gunzip autoconf-2.71.tar.gz && tar xvf autoconf-2.71.tar && pushd autoconf-2.71 && ./configure && make -j$(nproc) && sudo make install
### autoconf-archive
curl -o autoconf-archive-2021.02.19.tar.xz http://ftp.gnu.org/gnu/autoconf-archive/autoconf-archive-2021.02.19.tar.xz && xz -d -v autoconf-archive-2021.02.19.tar.xz && tar xvf autoconf-archive-2021.02.19.tar && pushd autoconf-archive-2021.02.19 && ./configure && make -j$(nproc) && sudo make install && popd
### automake
curl -o automake-1.16.5.tar.gz http://ftp.gnu.org/gnu/automake/automake-1.16.5.tar.gz && tar xvzf automake-1.16.5.tar.gz && pushd automake-1.16.5 && ./configure && make -j$(nproc) && sudo make install && popd
### libtool
curl -o libtool-2.4.6.tar.gz http://ftp.jaist.ac.jp/pub/GNU/libtool/libtool-2.4.6.tar.gz && tar xvfz libtool-2.4.6.tar.gz && pushd libtool-2.4.6 && ./configure --prefix=/usr/local/libtool/2_4_6 && make -j$(nproc) && sudo make install && popd
### symlinks
sudo ln -s /usr/local/libtool/2_4_6/bin/libtool /usr/local/bin/ && sudo ln -s /usr/local/libtool/2_4_6/bin/libtoolize /usr/local/bin/ && sudo ln -s /usr/local/libtool/2_4_6/include/libltdl /usr/local/include/ && sudo ln -s /usr/local/libtool/2_4_6/include/ltdl.h /usr/local/include/ && sudo ln -s /usr/local/libtool/2_4_6/lib/libltdl.a /usr/local/lib/ && sudo ln -s /usr/local/libtool/2_4_6/lib/libltdl.la /usr/local/lib/ && sudo ln -s /usr/local/libtool/2_4_6/lib/libltdl.so /usr/local/lib/ && sudo ln -s /usr/local/libtool/2_4_6/lib/libltdl.so.7 /usr/local/lib/ && sudo ln -s /usr/local/libtool/2_4_6/lib/libltdl.so.7.3.1 /usr/local/lib/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/libtool.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/ltargz.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/ltdl.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/lt~obsolete.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/ltoptions.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/ltsugar.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/ltversion.m4 /usr/local/share/aclocal/
### fetch mono
sudo yum-config-manager --add-repo http://download.mono-project.com/repo/centos/ && sudo yum clean all && sudo yum makecache && sudo rpm --import "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF" && sudo yum install -y mono-complete && sudo yum -y clean all
### extras with epel-release
sudo yum makecache && sudo yum install -y wget epel-release java-11-openjdk-devel python3-devel wget && sudo yum update -y && sudo yum install -y ccache R-core-devel && sudo yum -y clean all
### R dependencies
export Rcpp_package=Rcpp_0.12.19.tar.gz && export RInside_package=RInside_0.2.12.tar.gz && wget https://cran.r-project.org/src/contrib/Archive/Rcpp/${Rcpp_package} && wget https://cran.r-project.org/src/contrib/Archive/RInside/${RInside_package} && sudo R CMD INSTALL ${Rcpp_package} ${RInside_package} && rm -f ${Rcpp_package} ${RInside_package}On Mac (Snow Leopard):
sudo port install bison flex binutils openldap icu xalanc zlib boost openssl libarchive
On Windows:
The following Cmake flags are useful for developers building 7.0.0+:
- SPARK=ON enables building of the spark plugin package
- INCLUDE_SPARK=ON includes the spark plugin in a platform build
- SUPPRESS_SPARK=ON suppresses the building of a spark plugin in an INCLUDE_PLUGINS=ON build with the platform
- SPARK_URL select a specific version of spark to build against
- SPARK_HPCC_VERSION and WSCLIENT_VERSION select a specific version of spark-hpcc to fetch and build against
- SPARK_HPCC_JAR and WSCLIENT_JAR absolute file paths for using local jars instead of downloading from maven central
From 7.4 the build system has been changed to make it easier to build on windows. It is dependent on two other projects chocolatey and vcpkg for installing the dependencies:
Install chocolatey:
https://chocolatey.org/install
Use that to install bison/flex:
choco install winflexbison3
Install vcpkg:
git clone https://github.com/Microsoft/vcpkg
cd vcpkg
bootstrap-vcpkg
Use vcpkg to install various packages:
vcpkg install zlib
vcpkg install boost
vcpkg install icu
vcpkg install libxslt
vcpkg install tbb
vcpkg install cppunit
vcpkg install libarchive
vcpkg install apr
vcpkg install apr-util
You may need to force vcpkg to build the correct version, e.g. for 64bit:
vcpkg install zlib zlib:x64-windows
(See Git-step-by-step for full instructions)
git fetch upstream
git checkout master
git pull
git submodule update --init --recursive
Alternatively, for a quick build, you can simply do..
git clone [-b <branch name>] --recurse-submodules https://github.com/hpcc-systems/HPCC-Platform.git
Where [ ] denotes an optional argument.
The minimum version of Cmake required to build the HPCC Platform is 3.3.2 on Linux. If your system does not supply an adequate version of cmake in its repositories, you may download it here at cmake.org.
When building with CMake, it is recommended to create a separate directory for the built files. You can create it either inside the source tree or outside, depends much on personal preference. I'll assume you're creating it outside. If your source is under, say, ~/hpcc/src, create a build directory outside of the tree at ~/hpcc/build.
Now you need to run CMake to populate your build directory with Makefiles and special configuration to build HPCC, packages, tests, etc.
For release builds, do:
cd ~/hpcc/build
cmake ../src
For Plugins, do:
Make sure the build directory is cleared, target the normal platform
# cmake –D<Plugin Name>=ON ../HPCC-Platform
# make –j6 package
*these are the allowable names for plugin names;
CASSANDRAEMBED
REMBED
V8EMBED
MEMCACHED
PYEMBED
REDIS
JAVAEMBED
KAFKA
SQLITE3EMBED
MYSQLEMBED
If testing during development and you want to include plugins (except R) in the package, do: cd ~/hpcc/build cmake -DTEST_PLUGINS=ON ../src
If you're running a debug build, do:
cd ~/hpcc/build
cmake -DCMAKE_BUILD_TYPE:STRING=Debug ../src
To build the client tools only, do:
cd ~/hpcc/build
cmake -DCLIENTTOOLS_ONLY=1 ../src
To enable signing of the ecl standard library, insure you have a gpg private key loaded into your gpg keychain and do:
cd ~/hpcc/build
# Add -DSIGN_MODULES_KEYID and -DSIGN_MODULES_PASSPHRASE if applicable
cmake -DSIGN_MODULES=ON ../src
In some cases, users have found that when packaging for ubuntu, the dpkg-shlibdeps portion of the packaging adds an exceptional amount of time to the build process. This package will likely not run properly if installed on another system. Only use locally. To turn this off (and to create a package without dynamic dependency generation) do:
cd ~/hpcc/build
cmake -DUSE_SHLIBDEPS=OFF ../src
CMake will check for necessary dependencies like binutils, boost regex, cppunit, pthreads, etc. If everything is correct, it'll create the necessary Makefiles and you're ready to build HPCC.
####NOTE: We default to using libxslt in place of Xalan for xslt support. Should you prefer to use libxalan, you can specify -DUSE_LIBXALAN on the cmake command line.
It doesn't take too long to compile HPCC, but it's highly recommended that you use all your CPUs to do so. When building complex projects (where I/O plays an important part), it's often recommended to use 50% more threads than the number of CPUs you have.
So, on a quad-core machine, the command you should type is:
make -j6
Alternatively you can call a build system agnostic variant (works with make, ninja, XCode, Visual Studio etc.)
cmake --build .
That will make all binaries, libraries and scripts necessary to create the package.
The recommended way to install HPCC in your machine (even for testing) is to use distro packages. CMake has already detected your system, so it know whether to generate TGZ files, DEB or RPM packages.
Just type:
make package
Alternatively you can use the build system agnostic variant:
cmake --build . --target package
and it'll create the appropriate package for you to install. The package will be inside the build directory.
In my system (Ubuntu 11.04), it created a package called hpccsystems-platform-community_3.3.0-1trunk-Debugnatty_amd64.deb.
The HPCC package has dependencies built in, and your system might not have them all (you can have the source package in order to build, but not the library package installed by your package manager, for example).
Install the package:
sudo dpkg -i hpccsystems-platform-community_6.0.0-trunk0trusty_amd64.deb
(note that the name of the package you just built will depend on the branch you checked out, the distro, and other options).
Now fix any missing dependencies:
sudo apt-get -f install
(see here for Ubuntu based installation).
If your installation succeeded, you should see the service in /etc/init.d (or /etc/rc.d/init.d on RPM based distros) called hpcc-init. This service follows the standard Linux service interface, where you can use commands like:
sudo /etc/init.d/hpcc-init status
sudo /etc/init.d/hpcc-init start
sudo /etc/init.d/hpcc-init stop
Use hpcc-init start and make sure there are no error messages. If everything went ok, check that the cluster is up and running by visiting the admin page on your browser.
Building a development installation for quick startup and debugging is very similar to the normal build process. The only major difference are the flags passed to the invocation of cmake to build your make files, and instead of doing a make package you run make install and make configure. There are two key cmake variables that must be set, and some optional variables depending on the environment you're building in.
DESTDIR=/home/user/path
RUNTIME_USER=<my_user>
RUNTIME_GROUP=<my_alternate_group> (optional)
HOME_DIR=/path/to/alternate/home/directory (optional)
Destdir is used for the new root installation path of your build. For instance if you set DESTDIR=/home/user/runtime, then everything will be laid down there as though it was root. Then in order to start the platform, you simply need to call the hpcc-init script directly from /home/user/runtime/etc/init.d/hpcc-init. It is important that you do not set the DESTDIR environment variable in your shell. Allow cmake to handle DESTDIR, not the shell environment.
Runtime User is generally your user name. This will allow you to run the entire platform as your user instead of as root. Running as root will mess up your runtime environment as certain files will be owned by root on creation instead of ${USER}. Never run as root when doing a development build.
Runtime group is optional and is for environments like OSx where your Runtime User and Runtime Group aren't both set to your user name.
The Home directory is a useful variable if you have the home path for your environment set to a non-standard location such as /mnt/disk1/home/my_user. It is important to remember that the DESTDIR should most likely match this path in most situations. If the HOME_DIR is standard, this will be done by default for you. Otherwise look to manipulate this variable.
Example:
cmake -DDESTDIR=$HOME/runtime -DRUNTIME_USER=$USER ../HPCC-Platform/
make install #instead of make package, will also include configure target
- How to cross compile to other architectures
- More Windows details