-
Notifications
You must be signed in to change notification settings - Fork 304
Building HPCC
Assuming you already have your git repository set up (see Git-step-by-step), you can build and install HPCC.
The build process requires some additional packages:
- NodeJS: v16.x.x (Current LTS): see installing NodeJS for more information.
On Linux (Ubuntu 20.04):
sudo apt-get install cmake bison flex build-essential binutils-dev libldap2-dev libcppunit-dev libicu-dev libxslt1-dev zlib1g-dev libboost-regex-dev libarchive-dev python-dev libv8-dev default-jdk libapr1-dev libaprutil1-dev libiberty-dev libhiredis-dev libtbb-dev libxalan-c-dev libnuma-dev nodejs libevent-dev libatlas-base-dev libblas-dev python3-dev default-libmysqlclient-dev libsqlite3-dev r-base-dev r-cran-rcpp r-cran-rinside r-cran-inline libmemcached-dev libcurl4-openssl-dev pkg-config libtool autotools-dev automake libssl-dev
On Linux (Ubuntu 19.10):
sudo apt-get install cmake bison flex build-essential binutils-dev libldap2-dev libcppunit-dev libicu-dev libxslt1-dev zlib1g-dev libboost-regex-dev libarchive-dev python-dev libv8-dev default-jdk libapr1-dev libaprutil1-dev libiberty-dev libhiredis-dev libtbb-dev libxalan-c-dev libnuma-dev nodejs libevent-dev libatlas-base-dev libblas-dev python3-dev default-libmysqlclient-dev libsqlite3-dev r-base-dev r-cran-rcpp r-cran-rinside r-cran-inline libmemcached-dev libcurl4-openssl-dev pkg-config libtool autotools-dev automake libssl-dev
On Linux (Ubuntu 19.04):
sudo apt-get install cmake bison flex build-essential binutils-dev libldap2-dev libcppunit-dev libicu-dev libxslt1-dev zlib1g-dev libboost-regex-dev libarchive-dev python-dev libv8-dev default-jdk libapr1-dev libaprutil1-dev libiberty-dev libhiredis-dev libtbb-dev libxalan-c-dev libnuma-dev nodejs libevent-dev libatlas-base-dev libblas-dev python3-dev default-libmysqlclient-dev libsqlite3-dev r-base-dev r-cran-rcpp r-cran-rinside r-cran-inline libmemcached-dev libcurl4-openssl-dev pkg-config libtool autotools-dev automake libssl-dev
On Linux (Ubuntu 18.04):
sudo apt-get install cmake bison flex build-essential binutils-dev libldap2-dev libcppunit-dev libicu-dev libxslt1-dev zlib1g-dev libboost-regex-dev libarchive-dev python-dev libv8-dev default-jdk libapr1-dev libaprutil1-dev libiberty-dev libhiredis-dev libtbb-dev libxalan-c-dev libnuma-dev nodejs libevent-dev libatlas-base-dev libblas-dev python3-dev default-libmysqlclient-dev libsqlite3-dev r-base-dev r-cran-rcpp r-cran-rinside r-cran-inline libmemcached-dev libcurl4-openssl-dev pkg-config libtool autotools-dev automake libssl-dev
On Linux (Ubuntu 16.04):
sudo apt-get install cmake bison flex build-essential binutils-dev libldap2-dev libcppunit-dev libicu-dev libxslt1-dev zlib1g-dev libboost-regex-dev libssl-dev libarchive-dev python-dev libv8-dev default-jdk libapr1-dev libaprutil1-dev libiberty-dev libhiredis-dev libtbb-dev libxalan-c-dev libnuma-dev libevent-dev libatlas-base-dev libblas-dev libatlas-dev python3-dev libcurl4-openssl-dev libtool autotools-dev automake
Additional information for building versions prior to 7.0 on Ubuntu 18.04
get openssl-1.0.2o.tar.gz from https://www.openssl.org/source/
unpack, build, and install:
./config -fPIC shared
make
make install
This installs to /usr/local/ssl by default.
Build the platform with the following additional CMake options:
-DOPENSSL_CRYPTO_LIBRARY=/usr/local/ssl/lib/libcrypto.so -DOPENSSL_INCLUDE_DIR=/usr/local/ssl/include -DOPENSSL_LIBRARIES=/usr/local/ssl/lib/libssl.so -DOPENSSL_SSL_LIBRARY=/usr/local/ssl/lib/libssl.so
Regardless of which version of Centos you will be building on, it is suggested that you enable the EPEL repository
sudo yum install -y epel-release
CentOS 6.4:
sudo yum install gcc-c++ gcc make bison flex binutils-devel openldap-devel libicu-devel libxslt-devel libarchive-devel boost-devel openssl-devel apr-devel apr-util-devel hiredis-devel numactl-devel libmysqlclient-dev libevent-devel tbb-devel atlas-devel python34 R-core-devel R-Rcpp-devel R-inline R-RInside-devel nodejs libcurl-devel
CentOS 7:
sudo yum install gcc-c++ gcc make bison flex binutils-devel openldap-devel libicu-devel libxslt-devel libarchive-devel boost-devel openssl-devel apr-devel apr-util-devel hiredis-devel numactl-devel mariadb-devel libevent-devel tbb-devel atlas-devel python34 libmemcached-devel sqlite-devel v8-devel python-devel python34-devel java-1.8.0-openjdk-devel R-core-devel R-Rcpp-devel R-inline R-RInside-devel nodejs cmake3 rpm-build libcurl-devel
CentOS 8:
Go into /etc/yum.repos.d/CentOS-PowerTools.repo and set enable=0 to 1. Then do a sudo yum update before installing the development packages needed for the system.
sudo yum install gcc-c++ gcc make bison flex binutils-devel openldap-devel libicu-devel libxslt-devel libarchive-devel boost-devel openssl-devel apr-devel apr-util-devel hiredis-devel numactl-devel mariadb-devel libevent-devel tbb-devel atlas-devel libmemcached-devel sqlite-devel python2-devel python3-devel java-1.8.0-openjdk-devel R-core-devel R-Rcpp-devel R-inline R-RInside-devel nodejs cmake3 rpm-build libcurl-devel
The version of cmake that is in the Centos 6 and 7 repositories is too old, so you will need to install cmake3 and use that instead. Alternatively you can download the source from www.cmake.org and build and install from source - this is quite straightforward.
wget https://cmake.org/files/v3.16/cmake-3.16.3.tar.gz
tar xzf cmake-3.16.3.tar.gz
cd cmake-3.16.3
./bootstrap
make -j2
sudo make install
On Linux (Fedora 19):
sudo yum install gcc-c++ gcc make fedora-packager cmake bison flex binutils-devel openldap-devel libicu-devel xerces-c-devel xalan-c-devel libarchive-devel boost-devel openssl-devel apr-devel apr-util-devel
On Linux (Fedora 23):
sudo dnf install gcc-c++ gcc make fedora-packager cmake bison flex binutils-devel openldap-devel libicu-devel xerces-c-devel xalan-c-devel libarchive-devel boost-devel openssl-devel apr-devel apr-util-devel numactl-devel tbb-devel libxslt-devel nodejs
On Mac (Snow Leopard):
sudo port install bison flex binutils openldap icu xalanc zlib boost openssl libarchive
You can disable some functionality in order to reduce the list of required components, if necessary. Optional components, such as the plugins for interfacing to external languages, will be disabled automatically if the required libraries and headers are not found at build time.
Optional dependencies:
To build with support for all the plugins for third party embedded code, additional dependencies may be required. If not found, the system will default to skipping those components.
The following are steps for building the tag community_7.12.22 in an Archlinux environment
# update packages and install build dependencies
pacman -Syu --overwrite /usr/lib/libtraceevent.so.1
pacman -S base-devel
pacman -S devtools wget openldap nodejs npm \
icu libxslt libarchive boost openssl apr apr-util \
hiredis numactl mariadb libevent tbb libmemcached \
sqlite jdk8-openjdk r cmake cblas
# fetch source
cd
git clone -b community_7.12.22-1 --single-branch https://github.com/hpcc-systems/HPCC-Platform.git --recursive
# build
mkdir build
cd build
cmake ../HPCC-Platform -DCLIENTTOOLS_ONLY=ON
make -j4
# install to /opt/HPCCSystems/7.12.22/clienttools and add to PATH
sudo make install
export PATH=${PATH}:/opt/HPCCSystems/7.12.22/clienttools/bin
First insure that the R language is installed on your system. For Ubuntu use sudo apt-get install r-base-dev. For centos distributions use sudo yum install -y R-core-devel.
To install the prerequisites for building R support, use the following for all distros:
wget https://cran.r-project.org/src/contrib/00Archive/Rcpp/Rcpp_0.12.1.tar.gz
wget https://cran.r-project.org/src/contrib/00Archive/RInside/RInside_0.2.12.tar.gz
wget http://cran.r-project.org/src/contrib/inline_0.3.14.tar.gz
sudo R CMD INSTALL Rcpp_0.12.1.tar.gz RInside_0.2.12.tar.gz inline_0.3.14.tar.gz
To install the prerequisites for libcouchbase, go to http://developer.couchbase.com/documentation/server/current/sdk/c/start-using-sdk.html and install the libcouchbase repo for your distribution. Then add the libcouchbase libraries and headers to your system.
The following Cmake flags are useful for developers building 7.0.0+:
- SPARK=ON enables building of the spark plugin package
- INCLUDE_SPARK=ON includes the spark plugin in a platform build
- SUPPRESS_SPARK=ON suppresses the building of a spark plugin in an INCLUDE_PLUGINS=ON build with the platform
- SPARK_URL select a specific version of spark to build against
- SPARK_HPCC_VERSION and WSCLIENT_VERSION select a specific version of spark-hpcc to fetch and build against
- SPARK_HPCC_JAR and WSCLIENT_JAR absolute file paths for using local jars instead of downloading from maven central
From 7.4 the build system has been changed to make it easier to build on windows. It is dependent on two other projects chocolatey and vcpkg for installing the dependencies:
Install chocolatey:
https://chocolatey.org/install
Use that to install bison/flex:
choco install winflexbison3
Install vcpkg:
git clone https://github.com/Microsoft/vcpkg
cd vcpkg
bootstrap-vcpkg
Use vcpkg to install various packages:
vcpkg install zlib
vcpkg install boost
vcpkg install icu
vcpkg install libxslt
vcpkg install tbb
vcpkg install cppunit
vcpkg install libarchive
vcpkg install apr
vcpkg install apr-util
You may need to force vcpkg to build the correct version, e.g. for 64bit:
vcpkg install zlib zlib:x64-windows
(See Git-step-by-step for full instructions)
git fetch upstream
git checkout master
git pull
git submodule update --init --recursive
Alternatively, for a quick build, you can simply do..
git clone [-b <branch name>] --recurse-submodules https://github.com/hpcc-systems/HPCC-Platform.git
Where [ ] denotes an optional argument.
The minimum version of Cmake required to build the HPCC Platform is 3.3.2 on Linux. If your system does not supply an adequate version of cmake in its repositories, you may download it here at cmake.org.
When building with CMake, it is recommended to create a separate directory for the built files. You can create it either inside the source tree or outside, depends much on personal preference. I'll assume you're creating it outside. If your source is under, say, ~/hpcc/src, create a build directory outside of the tree at ~/hpcc/build.
Now you need to run CMake to populate your build directory with Makefiles and special configuration to build HPCC, packages, tests, etc.
For release builds, do:
cd ~/hpcc/build
cmake ../src
For Plugins, do:
Make sure the build directory is cleared, target the normal platform
# cmake –D<Plugin Name>=ON ../HPCC-Platform
# make –j6 package
*these are the allowable names for plugin names;
CASSANDRAEMBED
REMBED
V8EMBED
MEMCACHED
PYEMBED
REDIS
JAVAEMBED
KAFKA
SQLITE3EMBED
MYSQLEMBED
If testing during development and you want to include plugins (except R) in the package, do: cd ~/hpcc/build cmake -DTEST_PLUGINS=ON ../src
If you're running a debug build, do:
cd ~/hpcc/build
cmake -DCMAKE_BUILD_TYPE:STRING=Debug ../src
To build the client tools only, do:
cd ~/hpcc/build
cmake -DCLIENTTOOLS_ONLY=1 ../src
To enable signing of the ecl standard library, insure you have a gpg private key loaded into your gpg keychain and do:
cd ~/hpcc/build
# Add -DSIGN_MODULES_KEYID and -DSIGN_MODULES_PASSPHRASE if applicable
cmake -DSIGN_MODULES=ON ../src
In some cases, users have found that when packaging for ubuntu, the dpkg-shlibdeps portion of the packaging adds an exceptional amount of time to the build process. To turn this off (and to create a package without dynamic dependency generation) do:
cd ~/hpcc/build
cmake -DUSE_SHLIBDEPS=OFF ../src
CMake will check for necessary dependencies like binutils, boost regex, cppunit, pthreads, etc. If everything is correct, it'll create the necessary Makefiles and you're ready to build HPCC.
####NOTE: We default to using libxslt in place of Xalan for xslt support. Should you prefer to use libxalan, you can specify -DUSE_LIBXALAN on the cmake command line.
It doesn't take too long to compile HPCC, but it's highly recommended that you use all your CPUs to do so. When building complex projects (where I/O plays an important part), it's often recommended to use 50% more threads than the number of CPUs you have.
So, on a quad-core machine, the command you should type is:
make -j6
Alternatively you can call a build system agnostic variant (works with make, ninja, XCode, Visual Studio etc.)
cmake --build .
That will make all binaries, libraries and scripts necessary to create the package.
The recommended way to install HPCC in your machine (even for testing) is to use distro packages. CMake has already detected your system, so it know whether to generate TGZ files, DEB or RPM packages.
Just type:
make package
Alternatively you can use the build system agnostic variant:
cmake --build . --target package
and it'll create the appropriate package for you to install. The package will be inside the build directory.
In my system (Ubuntu 11.04), it created a package called hpccsystems-platform-community_3.3.0-1trunk-Debugnatty_amd64.deb.
The HPCC package has dependencies built in, and your system might not have them all (you can have the source package in order to build, but not the library package installed by your package manager, for example).
Install the package:
sudo dpkg -i hpccsystems-platform-community_6.0.0-trunk0trusty_amd64.deb
(note that the name of the package you just built will depend on the branch you checked out, the distro, and other options).
Now fix any missing dependencies:
sudo apt-get -f install
(see here for Ubuntu based installation).
If your installation succeeded, you should see the service in /etc/init.d (or /etc/rc.d/init.d on RPM based distros) called hpcc-init. This service follows the standard Linux service interface, where you can use commands like:
sudo /etc/init.d/hpcc-init status
sudo /etc/init.d/hpcc-init start
sudo /etc/init.d/hpcc-init stop
Use hpcc-init start and make sure there are no error messages. If everything went ok, check that the cluster is up and running by visiting the admin page on your browser.
Building a development installation for quick startup and debugging is very similar to the normal build process. The only major difference are the flags passed to the invocation of cmake to build your make files, and instead of doing a make package you run make install and make configure. There are two key cmake variables that must be set, and some optional variables depending on the environment you're building in.
DESTDIR=/home/user/path
RUNTIME_USER=<my_user>
RUNTIME_GROUP=<my_alternate_group> (optional)
HOME_DIR=/path/to/alternate/home/directory (optional)
Destdir is used for the new root installation path of your build. For instance if you set DESTDIR=/home/user/runtime, then everything will be laid down there as though it was root. Then in order to start the platform, you simply need to call the hpcc-init script directly from /home/user/runtime/etc/init.d/hpcc-init. It is important that you do not set the DESTDIR environment variable in your shell. Allow cmake to handle DESTDIR, not the shell environment.
Runtime User is generally your user name. This will allow you to run the entire platform as your user instead of as root. Running as root will mess up your runtime environment as certain files will be owned by root on creation instead of ${USER}. Never run as root when doing a development build.
Runtime group is optional and is for environments like OSx where your Runtime User and Runtime Group aren't both set to your user name.
The Home directory is a useful variable if you have the home path for your environment set to a non-standard location such as /mnt/disk1/home/my_user. It is important to remember that the DESTDIR should most likely match this path in most situations. If the HOME_DIR is standard, this will be done by default for you. Otherwise look to manipulate this variable.
Example:
cmake -DDESTDIR=$HOME/runtime -DRUNTIME_USER=$USER ../HPCC-Platform/
make install #instead of make package, will also include configure target
- How to cross compile to other architectures
- More Windows details